A target counting model training method based on enhanced local context supervision information

Abstract

This invention belongs to the field of remote sensing image processing technology and discloses a target counting model training method based on enhanced local contextual supervision information. The steps are as follows: First, a sample image set is acquired and divided into a training set, a validation set, and a test set. The sample image set includes multiple sample images containing target objects and the corresponding point annotations for each sample image. The training set is then trained using the target counting model to obtain the final target counting model. The target counting model is obtained by adding a local segmentation branch and a feature fusion module to a P2PNet. The target counting model of this invention combines small error with good generalization performance, accurately identifying the location and number of target objects, and solving the performance limitations of existing counting models caused by factors such as illumination, occlusion, and overlap of target objects.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a target counting model training method based on enhanced local contextual supervision information. Background Technology

[0002] Wheat is a crucial food crop in my country. Henan province alone accounts for one-quarter of China's wheat production, enough to feed approximately 400 million people. Therefore, maintaining consistently high wheat yields is of great significance for safeguarding my country's food security. During wheat growth, the number of wheat seedlings is a key factor limiting yield; excessively sparse or dense seedlings significantly impact yield. Therefore, timely and accurate seedling counts provide vital scientific basis for subsequent production stages such as emergence rate estimation, yield prediction, and grain quality assessment.

[0003] Traditional wheat seedling counting relies primarily on manual counting in the field, which suffers from high economic costs, labor-intensive processes, and low efficiency. Furthermore, the results are easily influenced by subjective factors. With the development of deep learning, using deep neural networks for automatic target object counting is becoming a new research hotspot. Compared to manual counting methods, using deep neural networks to analyze acquired wheat seedling images and automatically detect the number of seedlings can overcome time and space limitations and reduce reliance on agricultural experts, thereby improving labor efficiency.

[0004] Researchers have used deep learning techniques to count objects such as cells, people, pigs, and wheat ears. Among these, detection-based and density map regression methods have been widely applied. Detection-based counting methods primarily use detectors such as YOLO, SSD, and Faster R-CNN to detect objects in images and then obtain the number of objects. This method not only provides the counting results but also provides the location information of the objects through bounding boxes. However, these methods require bounding boxes as ground truth during the training phase. For wheat seedling images, which suffer from occlusion, overlap, and distortion, directly using bounding box annotations is difficult. Furthermore, using point annotations to generate pseudo-bounding maps is prone to errors, and subsequent processing is technically challenging. Density map regression-based counting methods use point annotations to generate density maps of the objects in the training samples, which serve as the model's learning objective. The number of objects is then obtained by integrating the density map predicted by the model. While this method has low annotation costs for wheat seedling counting, it cannot clearly identify the specific location of the seedlings, hindering its application in downstream tasks such as planting planning and farmland cultivation. It is also susceptible to perspective distortion, resulting in weak model robustness. Currently, P2PNet, proposed by Song et al., provides a new solution for object counting methods. P2PNet directly uses point annotations as the model's learning objective, predicting the point coordinates of all object objects to obtain the total number of objects. Compared to the two counting models mentioned above, P2PNet does not require bounding box annotations for object objects in the training samples, nor does it indirectly obtain the learning objective by generating pseudo-density maps or pseudo-boundary maps through point annotations. Furthermore, P2PNet can explicitly identify the location of object objects, better meeting the needs of downstream tasks.

[0005] However, the complex growing environment of wheat seedlings results in significant noise in wheat seedling images, leading to poor performance when existing target object counting methods are directly applied to wheat seedling counting. On one hand, withered leaves in the wheat field and different lighting angles cause shadows of varying directions and sizes in the wheat seedling images, introducing interference noise into the counting model and severely affecting its performance. On the other hand, occlusion of wheat seedlings by soil clods in the wheat field, as well as overlapping leaves when the seedlings are densely growing, lead to misjudgments by existing wheat seedling counting models. Summary of the Invention

[0006] In view of the problems and shortcomings of the existing technology, the purpose of this invention is to provide a target counting model training method based on enhanced local context supervision information.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of this invention provides a method for training a target counting model based on enhanced local context supervision information, comprising the following steps:

[0009] S1, Obtain a sample image set, which includes multiple sample images containing the target object and the corresponding point annotation results for each sample image; the point annotation results of the sample images are the location information corresponding to the annotation points of the target object; randomly divide the sample image set into a training set, a validation set, and a test set according to the proportion;

[0010] S2, the sample images in the training set are input into the pre-constructed target counting model for counting, and the target counting result of the sample images is obtained. The target counting result is the location information of the target object obtained based on the target counting model. A loss function is constructed based on the target counting result and the point annotation result of the sample images. Based on the loss function, backpropagation is used to update the parameters of the target counting model to obtain the trained target counting model. The target counting model is obtained by adding a local segmentation branch and a feature fusion module between the basic network of P2PNet and the point regression branch and the classification branch, based on the localization and classification backbone VGG16_bn.

[0011] S3, the trained target counting model is sequentially validated through the validation set and tested through the test set to obtain the optimal target counting model.

[0012] More preferably, the point-to-point P2PNet is a point-annotation-based counting model that annotates the location coordinates of the target object in the form of points, and then directly uses the annotation results as the model's learning objective. The original P2PNet includes a base network, a point regression branch, and a classification branch. Specifically, it first uses VGG16_bn as the base network to extract global features of the target object; then, the global features are simultaneously fed into the point regression branch and the classification branch to generate candidate points of the target object and the confidence score (i.e., classification result) corresponding to each candidate point; finally, the location coordinates of the target object are selected from the candidate points based on the classification results, and the total number of location coordinates is the count result of the target object.

[0013] Furthermore, in the original P2PNet model, the extracted global feature map F0 is directly fed into both the point regression branch and the classification branch, with the architecture of these two branches remaining consistent. Given a global feature map F0 with a scale of 256×H×W, where H and W represent height and width, respectively. First, F0 is processed twice with a 3×3 convolutional layer and ReLU activation function, resulting in a tensor with the same dimension of 256×H×W. Next, a 3×3 convolution is performed to change the number of channels from 256 to 2*Z, and the dimension of the output tensor becomes 2*Z×H×W. Finally, through dimension transposition and shape changes, a two-dimensional tensor with a scale of M×2 is output. Here, Z is a hyperparameter (generally set to 4 for best results), which determines the total number of output candidate point coordinates M and the total number of confidence scores M, where the total number M = H*W*Z. This two-dimensional tensor represents M binary classification results for the classification branch, i.e., classifying M point coordinates as target objects or non-target objects; for the point regression branch, it represents generating M point coordinate offsets, which are added to the predefined point coordinates in P2PNet to obtain the final M candidate point coordinates. During the inference phase, the final predicted results of the target object coordinates can be selected from these candidate points based on their corresponding confidence scores, and the number of these predicted coordinates is the result of the target object count.

[0014] More preferably, the target counting model of the present invention is an improved P2PNet, which inherits the basic network VGG16_bn, point regression branch and classification branch in the original P2PNet, and inserts the local segmentation branch and feature fusion module after the basic network and before the point regression branch and classification branch.

[0015] Furthermore, the base network VGG16_bn in the target counting network of the present invention still includes 13 convolutional blocks and 2 1×1 convolutional layers. The 13 convolutional blocks sequentially extract features from the input sample image, outputting C1, C2, C3, and C4 feature maps of different scales. The width and height of feature maps C1, C2, C3, and C4 are halved sequentially. To reduce the number of model parameters, the 2 1×1 convolutional layers perform channel compression on feature maps C3 and C4, respectively, obtaining Conv(C3) and Conv(C4), with the number of channels for Conv(C3) and Conv(C4) halved. Finally, Conv(C4) is upsampled using a 2x nearest neighbor interpolation method and added to Conv(C3) to obtain the global feature map F0. The generation process of F0 can be represented as follows:

[0016] F0 = Conv(C3) + Up(Conv(C4))

[0017] In the formula, Conv and Up represent convolution and upsampling, respectively.

[0018] Preferably, the local segmentation branch includes a local feature extraction module; the local feature extraction module is used to extract local features from the global feature map extracted by the basic network to obtain a local feature map; the local segmentation branch combines the point annotation results of the sample image to optimize the local feature map to obtain an optimized local feature map; the feature fusion module is used to fuse the optimized local feature map with the global feature map to obtain a fused feature map, and then use the fused feature map as input to both the point regression branch and the classification branch.

[0019] Preferably, the local feature extraction module includes three identical and sequentially connected dimensionality reduction convolutional modules and one 3×3 convolutional layer. The dimensionality reduction convolutional modules are used for channel compression of the input feature map. After channel compression, the global feature map F0 is input into the dimensionality reduction convolutional modules to obtain a first-order dimensionality reduction feature map with unchanged width and height and halved channels. The first-order dimensionality reduction feature map is then continuously input into two more dimensionality reduction convolutional modules for channel compression to obtain a second-order and a third-order dimensionality reduction feature map. After processing by the 3×3 convolutional layer, the third-order dimensionality reduction feature map yields a local feature map F1 with unchanged width and height and two channels. It is stipulated here that channels marked with 1 on the feature map correspond to feature maps representing local information; channels marked with 0 on the feature map correspond to feature maps representing non-local information, and are related to L in the following text. G The superscript of the loss function corresponds to this. Furthermore, the local feature map with two channels is composed of feature maps representing local information and feature maps representing non-local information, thus representing both high-level semantic features of the local area and the local contextual supervision information emphasized in this invention.

[0020] More preferably, the process of generating the local feature map can be represented as follows:

[0021] F1 = Conv(f(F0))

[0022] In the formula, f represents the dimensionality reduction convolution module processing three times consecutively; Conv represents the convolution processing.

[0023] More preferably, the dimensionality reduction convolutional module consists of two 3×3 convolutional layers alternating with two ReLU functions. The first convolutional layer undergoes non-linear activation with a ReLU function and is then connected to the second convolutional layer via a residual connection. Furthermore, the ReLU function can improve the non-linear expressive power of the network model, and the residual connection can reduce the risk of overfitting.

[0024] Preferably, the process of optimizing the local feature map by combining the local segmentation branch with the point annotation results of the sample image specifically involves:

[0025] (1) Generating a local segmentation map: First, based on the point annotation results of the sample image, generate a circular region with the coordinates of the annotation points as the center and σ as the radius; then, perform binarization processing on the pixels inside and outside the circular region, assigning a value of 1 to the pixel values ​​of the pixels inside the circular region, and otherwise assigning a value of 0; finally, obtain the local segmentation map G. The generation process of the local segmentation map G and the radius σ of the circular region is as follows:

[0026]

[0027]

[0028]

[0029] In the formula, p is the pixel position in the local segmentation image; p i Represents the coordinates of the i-th labeled point; P = {p i |i∈{1,…N}} represents the coordinates of all labeled points; R(w,h) represents the coordinates of all labeled points p. i The region is a rectangular area centered at a point with width w and height h (w and h are hyperparameters); |R(w, h)| is the number of markers contained within the rectangular area; a is any marker within the region R(w, h); K is the number of the K nearest markers to a (hyperparameter); d k，a Let be the Euclidean distance between the k-th labeled point and a.

[0030] It should be noted that the local segmentation map is a binary image, in which the area inside the circle with a pixel value of 1 is the region representing the information of the labeled points (i.e., the local context supervision information region), and the area outside the circle with a pixel value of 0 is the region representing the information of the non-labeled points.

[0031] (2) Generating the predicted segmentation map: The local feature extraction branch also includes a 3×3 convolutional layer; the local feature map F1 is first upsampled by 8 times and then input into the 3×3 convolutional layer for convolution processing to obtain the predicted segmentation map F. G The process of generating the predicted segmentation map is as follows:

[0032] F G =Conv(Up(F1))

[0033] In the formula, Up represents upsampling.

[0034] More preferably, the 8x upsampling process employs nearest neighbor interpolation, making the predicted segmentation map F... G The width and height of the feature map are consistent with those of the learning target (i.e., the local segmentation map G). The 3×3 convolutional layer is used to smooth the noise generated by upsampling, thereby obtaining a feature map with more stable mathematical properties.

[0035] (3) Optimization judgment: Based on the predicted segmentation map FG The loss function L is used to construct local segmentation branches with the local segmentation map G. G According to the loss function L G The target counting model is updated using backpropagation to obtain an updated target counting model; then, local feature extraction is performed again using the updated target counting model to obtain an optimized local feature map; the loss function L G Specifically:

[0036]

[0037]

[0038] In the formula, w is the weight; l is the superscript (value is 0 or 1); G l Let G be the tensor formed by the channel labeled l in the local segmentation graph; l |For tensor G l The sum of all numerical elements contained therein; |G| is the sum of all numerical elements in the local segmentation graph; F G l Let l be the tensor formed by the channel labeled l in the predicted segmentation map; mean is the average of all numerical elements contained in the tensor; γ is the hyperparameter.

[0039] It should be noted that the predicted segmentation map F generated by the local segmentation branch G The results are pixel-level binary classifications. To alleviate the sample imbalance problem between the foreground and background classes and reduce its impact on counting accuracy, this invention incorporates a loss function L for local segmentation branches. G .

[0040] Preferably, the process by which the feature fusion module fuses the optimized local feature map with the global feature map specifically involves:

[0041] (1) Generating local enhanced feature maps: The feature fusion module includes a softmax function and a repeat function; after the optimized local feature map is input into the feature fusion module, it is classified by a softmax function to obtain two tensors with a scale of H×W, one of which is a tensor representing local feature information and the other is a tensor representing non-local feature information; the tensor representing local feature information is copied by the repeat function and spliced ​​to obtain a local enhanced feature map with the same number of channels as the global feature map F0;

[0042] It should be noted that the classification processing of the softmax function can normalize the value of each tensor element to [0, 1], which represents the probability that the corresponding pixel is judged by the network as representing local feature information or representing non-local feature information. The closer the value is to 1, the greater the probability that the candidate point is judged as representing local feature information.

[0043] (2) Feature fusion: The local enhanced feature map and the global feature map F0 are multiplied element-wise to obtain the fused feature map F2. Furthermore, the fused feature map F2 integrates the local contextual supervision information and global feature information of the target object, further enhancing the network's ability to identify the target object, thereby effectively improving the model's accuracy in counting the target object.

[0044] Preferably, in step S2, the loss function consists of the loss function of the point regression branch, the loss function of the classification branch, and the loss function of the local segmentation branch, and the loss function L is specifically:

[0045] L = L CE +λ1L P +λ2L G

[0046]

[0047]

[0048]

[0049] In the formula, λ1 and λ2 are both hyperparameters; L P The loss function for the point regression branch; The coordinates p of the point label of the i-th target object i The coordinates of the successfully matched candidate points (i.e., the predicted coordinates of the target object); L CE The loss function for the classification branch; represents the confidence score, i.e., the probability of being predicted as the target; y represents the label (value is 0 or 1); L G Here, G represents the loss function for the local segmentation branch; w represents the weights; l represents the superscript (value is 0 or 1); l Let G be the tensor formed by the channel labeled l in the local segmentation graph; l |For tensor G l The sum of all numerical elements contained therein; |G| is the sum of all numerical elements in the local segmentation graph; F G l Let l be the tensor formed by the channel labeled l in the predicted segmentation map; mean is the average of all numerical elements contained in the tensor; γ is the hyperparameter.

[0050] It should be noted that the loss functions for the point regression branch and the classification branch are the same as the loss functions in the original P2PNet.

[0051] A second aspect of the present invention provides an image target counting method, the method comprising: acquiring an image to be identified, inputting the image to be identified into a target counting model, and obtaining a target counting result of the image to be identified; wherein the target counting model is a trained target counting model obtained by training any of the target counting model training methods described in the first aspect above.

[0052] Preferably, the image to be identified is an image of plant seedlings, and the target counting result is the location and number of plant seedlings. More preferably, the image to be identified is an image of wheat seedlings, and the target counting result is the location and number of wheat seedlings.

[0053] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step in the target recognition model training method as described in the first aspect above, and / or the image target counting method as described in any of the second aspects above.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any step in the target recognition model training method as described in the first aspect above, and / or the image target counting method as described in any of the second aspects above.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] (1) This invention proposes a target counting model, P2P_Seg, that enhances local contextual supervision information, addressing the performance limitations of existing counting models caused by factors such as illumination, occlusion, and overlap. The P2P_Seg target counting model of this invention introduces a local segmentation branch for wheat seedlings based on P2PNet to obtain more local contextual supervision information for wheat seedlings, and uses an element-wise dot product mechanism to fuse the local contextual supervision information with the global information extracted from the base network. The improvements to network structure and feature fusion in this invention enhance the model's feature extraction capability for wheat seedlings, strengthen its resilience against factors such as illumination, occlusion, and overlap, improve the model's robustness, and effectively avoid miscounting and undercounting of wheat seedlings. In one embodiment, the target counting model P2P_Seg obtained by the present invention has a MAE of 5.86 and an RMSE of 7.68, which are reduced by 0.74 and 1.78 respectively compared with the original P2PNet. Compared with other existing counting models CSRNet, CANet, SCAR, BL and DM-Count, P2P_Seg also has the smallest MAE and RMSE, and its counting performance is more accurate.

[0057] (2) The target counting model P2P_Seg of this invention can more accurately predict the number of wheat seedlings, solving the problem of time-consuming and laborious traditional manual seedling counting; at the same time, it can also predict the location of wheat seedlings, providing effective support information for downstream tasks such as planting planning and fertile land cultivation, which is more conducive to actual agricultural production.

[0058] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the process of acquiring the sample image set in an embodiment of the present invention;

[0060] Figure 2 This is a flowchart of the sample image preprocessing process in an embodiment of the present invention;

[0061] Figure 3 These are labeled images of wheat seedlings at different density levels in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the P2P_Seg network structure of the target counting model of this invention;

[0063] Figure 5 This is a schematic diagram of the network structure of the local feature extraction module of the present invention;

[0064] Figure 6 This is a schematic diagram of the network structure of the feature fusion module of the present invention;

[0065] Figure 7 The figure shows the counting results of the target counting model P2P_Seg of this invention and existing models CSRNet, CANet, SCAR, BL, DM-Count, and P2PNet on the same wheat seedling dataset. In the figure, column (a) is the labeled image, that is, the point labeling result, which is directly used as the true value of P2PNet and P2P_Seg; column (b) is the density image generated by the point labeling, which is used as the true value of the density map-based counting model; columns (c) to (g) are the counting results of CSRNet, CANet, SCAR, BL, and DM-Count, respectively; columns (h) and (i) are the counting results of P2PNet and P2P_Seg, respectively. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0067] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0068] Example 1

[0069] This embodiment provides a method for training a target counting model based on enhanced local context supervision information, including the following steps:

[0070] S1, Obtain the sample image set, such as Figure 1 As shown, the sample image set includes multiple sample images containing target objects and the corresponding point annotation results for each sample image; the point annotation results of the sample images are the location information corresponding to the annotation points of the target objects; the sample image set is randomly divided into a training set, a validation set, and a test set according to a certain proportion. The specific steps are as follows:

[0071] First, obtain the sample image. The specific process is as follows:

[0072] (1) Data Acquisition: In October 2021, a mobile device (HONOR 20PRO model, 48-megapixel camera, BSI CMOS sensor, f / 2.2 aperture) was used to photograph and sample wheat seedlings in the experimental field. The sampling standard was the target counting area marked by the red trapezoidal box. After sampling, a total of 317 wheat seedling images (4000 pixels × 3000 pixels resolution) were collected. After removing images with blurry quality and severe occlusion, 295 images were selected as the initial experimental images for this invention.

[0073] The experimental site was located at the Henan Provincial Modern Agricultural Research and Development Base, specifically at 35°0'28"N, 113°41'48"E, with an altitude of 97m. The experiment employed a completely randomized block design. Sowing was on October 15, 2021, and sampling was conducted on November 12, 2021. There were 400 plots, each with an area of ​​36m². 2 .

[0074] (2) Preprocessing: The preprocessing steps mainly include filling the non-target counting area outside the red trapezoidal frame with black and removing redundancy, such as Figure 2 As shown. To avoid the influence of wheat seedlings in non-target areas, a preprocessing tool is used to fill the non-target areas with black (see stage ①). To prevent the subsequent data augmentation random cropping strategy from resulting in large areas of non-target counting regions, thereby interfering with the counting results of the target region, non-target regions are redundantly removed to the greatest extent possible (see stage ②). After the above preprocessing steps, the final experimental image (i.e., sample image) of this invention is obtained.

[0075] Then, the point annotation results of the sample image are obtained. The specific process is as follows:

[0076] Due to the small size of wheat seedlings and their susceptibility to occlusion and overlap, bounding box annotation is very difficult. Therefore, a low-cost and convenient point annotation method (one point annotation represents the coordinates of the corresponding wheat seedling) is adopted. Simultaneously, an efficient online annotation tool developed based on HTML5, Javascript, and Python is used for dataset annotation. This tool supports both point and bounding box label formats, and can not only segment the image for labeling but also flexibly enlarge and label segmented regions. For denser, heavily occluded, and overlapping areas in the wheat seedling image, this tool can be used to enlarge them before annotation, effectively improving annotation speed and quality. The labeled area is the relatively distinctive root and stem part of the wheat seedling (the stem near the ground), which facilitates recognition by the network model.

[0077] The above method was used to annotate 295 initial experimental images, resulting in the annotation of 32,237 wheat seedlings. The image with the most annotations had 321 points, while the image with the fewest had 18 points, averaging approximately 109 annotations per image. Images of wheat seedlings annotated at different density levels are shown below. Figure 3 As shown.

[0078] Finally, the wheat seedling dataset (sample image set), consisting of 295 initial experimental images and their corresponding point annotations, was randomly divided into a training set, a validation set, and a test set in a ratio of 6:1:3; the training set, validation set, and test set contained 177, 29, and 89 wheat seedling images, respectively. The division results of the wheat seedling dataset are shown in Table 1.

[0079] Table 1. Dataset partitioning results for wheat seedlings

[0080] Dataset Number of images / pieces Total number of wheat seedlings / plants training set 177 19622 Validation set 29 2849 test set 89 9766 total 295 32237

[0081] It should be noted that before model training, random cropping and random rotation were used to augment the training sample images: each image was randomly cropped into four parts, each with a size of 128 pixels × 128 pixels; then, random rotation with a probability of 0.5 was used to augment the cropped images.

[0082] S2, input the sample images in the training set into the pre-constructed target counting model P2P_Seg for counting, and obtain the target counting result of the sample images. The target counting result is the location point information of the target object obtained based on the target counting model. Construct a loss function based on the target counting result and the point annotation result of the sample images. Based on the loss function, use backpropagation to update the parameters of the target counting model to obtain the trained target counting model.

[0083] The target counting model P2P_Seg is obtained by adding a local segmentation branch and a feature fusion module between the base network and the point regression and classification branches, based on the point-to-point network P2PNet with the localization and classification skeleton VGG16_bn as the base network. The local segmentation branch includes a local feature extraction module, which extracts local features from the global feature map extracted by the base network to obtain a local feature map. The local segmentation branch optimizes the local feature map by combining the point annotation results of the sample image to obtain an optimized local feature map. The feature fusion module fuses the optimized local feature map with the global feature map to obtain a fused feature map, which is then used as input to both the point regression and classification branches.

[0084] To train the network more effectively, loss functions Li for the point regression branch were designed for the point regression branch, classification branch, and wheat seedling local segmentation branch, respectively. P Loss function L for classification branch CE Loss function L for local segmentation branches G Three different loss functions are used. The Euclidean distance is used to optimize the point regression branch, and the cross-entropy loss function is used to represent the loss function of the classification branch. Therefore, the total loss function L is specifically:

[0085] L = L cE +λ1L P +λ2L G

[0086]

[0087]

[0088]

[0089] In the formula, λ1 is a hyperparameter, set to 0.002 in the experiment; λ2 is a hyperparameter, set to 0.005 in the experiment; L P The loss function for the point regression branch; The coordinates p of the point labeled with the i-th wheat seedling i The coordinates of the successfully matched candidate points (i.e., the predicted coordinates of the wheat seedlings); L CE The loss function for the classification branch; represents the confidence score, i.e., the probability of being predicted as wheat seedlings; y represents the label (value is 0 or 1); L G Here, G represents the loss function for the local segmentation branch; w represents the weights; l represents the superscript (value is 0 or 1); l Let G be the tensor formed by the channel labeled l in the local segmentation graph; l |For tensor G l The sum of all numerical elements contained therein; |G| is the sum of all numerical elements in the local segmentation graph; F G l Let l be the tensor formed by the channel superscripted l in the predicted segmentation map; mean is the average of all numerical elements contained in the tensor; γ is a hyperparameter, which is set to 2 in the experiment.

[0090] It should be noted that the machine used in the experiment was configured with Intel. (R) Core (TM)The system used an i7-10600 CPU at 2.90GHz and an NVIDIA GeForce RTX 3090 GPU with 24GB of VRAM. PyTorch was used as the deep learning framework, with 8 training batches, 1000 training epochs, and a fixed learning rate of 0.0001. The Adam algorithm was used for optimization. The base network was pre-trained on ImageNet and trained using an even smaller learning rate of 0.00001.

[0091] The specific steps are as follows:

[0092] S21, Constructing the Target Counting Model P2P_Seg: To reduce the impact of factors such as illumination, occlusion, and overlap on wheat seedling counting, this invention improves P2PNet by introducing a local segmentation branch for wheat seedlings to obtain local feature maps. These local feature maps are then integrated into the global feature map to enhance the local contextual supervision information of the wheat seedlings. A target counting model P2P_Seg for wheat seedlings with enhanced local contextual supervision information is proposed, and its overall structure is as follows: Figure 4 As shown (the specific improved structure will be detailed in the following steps).

[0093] S22, the process of inputting the sample images in the training set into the pre-built target counting model for counting is as follows:

[0094] S221, the global features of the wheat seedling sample images are extracted using the VGG16_bn basic network to obtain a 256×H×W global feature map F0 (see module ①).

[0095] S222, the wheat seedling local segmentation branch generates a local feature map F1. This local feature map is then optimized and judged based on the point annotation results of the sample image input to the target counting model to extract local contextual supervision information of the wheat seedling, thus obtaining an optimized local feature map (see module ②). The wheat seedling local segmentation branch, designed to extract local contextual supervision information of the wheat seedling root and stem region, serves two purposes: First, it focuses the model's attention on the point-annotated target region of the wheat seedling root and stem region, ignoring interference from noise such as shadows caused by lighting and field weeds. Second, it provides more contextual reference information and expands the model's recognition range when the wheat seedling annotation points are obscured by soil clods or other debris. The key technologies included in the wheat seedling local segmentation branch are the design of the wheat seedling local feature extraction module and the wheat seedling local segmentation map generation. Specifically:

[0096] (1) Generating Local Feature Map F1: The wheat seedling local feature extraction module is an important component of the wheat seedling local segmentation branch, aiming to generate the local feature map F1. The wheat seedling local feature extraction module designed in this invention is as follows: Figure 5As shown. The local feature extraction module includes three identical and sequentially connected dimensionality reduction convolutional modules and one 3×3 convolutional layer. The dimensionality reduction convolutional modules are used for channel compression of the input feature map. The 256×H×W global feature map F0 is input into the dimensionality reduction convolutional modules for channel compression, resulting in a 128×H×W first-order dimensionality reduction feature map with unchanged width and height and halved channels. The first-order dimensionality reduction feature map is then input into two more dimensionality reduction convolutional modules for channel compression, resulting in a 64×H×W second-order dimensionality reduction feature map and a 32×H×W third-order dimensionality reduction feature map. The third-order dimensionality reduction feature map is processed by the 3×3 convolutional layer to obtain a 2×H×W local feature map F1 with unchanged width and height and 2 channels. Here, it is stipulated that channels marked with 1 on the feature map correspond to feature maps representing local information; channels marked with 0 on the feature map correspond to feature maps representing non-local information, and are related to L in the following text. G The superscript of the loss function corresponds to this. The local feature map with two channels is composed of feature maps representing local information and feature maps representing non-local information. It represents both the high-level semantic features of the local area and the local contextual supervision information emphasized in this invention.

[0097] In summary, the generation process of the local feature map F1 can be represented as follows:

[0098] F1 = Conv(f(F0))

[0099] In the formula, f represents the dimensionality reduction convolution module processing three times consecutively; Conv represents the convolution processing.

[0100] The dimensionality reduction convolutional module consists of two alternating 3×3 convolutional layers and two ReLU functions. The first convolutional layer undergoes non-linear activation with a ReLU function and is then connected to the second convolutional layer via a residual connection. Furthermore, the ReLU function enhances the non-linear expressiveness of the network model, while the residual connection reduces the risk of overfitting.

[0101] It should be noted that the local feature map F1 serves two purposes. First, F1 is used to fuse with the global feature map F0, thereby achieving the fusion of local contextual supervision information and global information. Second, F1 is sequentially upsampled using 8x nearest neighbor interpolation and then processed by a 3×3 convolutional layer to generate the predicted segmentation map F. G This allows for the optimization of local segmentation branches of wheat seedlings during the network training phase.

[0102] (2) The process of optimizing and judging the local feature map by combining the local segmentation branch with the point annotation results of the sample image is as follows:

[0103] 1) Generating local segmentation maps: The wheat seedling local segmentation map is the result image from which local contextual supervision information of wheat seedlings is extracted from point labels. It serves as the learning target of the wheat seedling local segmentation branch. This allows the counting model to not only directly use point labels as the learning target, but also indirectly extract more local contextual information from point labels, making fuller use of the real values ​​and playing a stronger supervisory role on the network model.

[0104] Specifically, firstly, a circular region with the coordinates of the marked points as the center and σ as the radius is generated based on the point annotation results of the sample image; then, the pixels inside and outside the circular region are binarized, with the pixel value of the pixels inside the circular region assigned a value of 1, and the other assigned a value of 0; finally, a local segmentation map G is obtained. That is, given a wheat seedling image with N point annotations (the point annotations are located at the root of the wheat seedling, using P = {p...}... i Let |i∈{1,...,N}} represent the coordinates of all wheat seedlings in this image, where p i =(x i y i () represents the coordinates of the i-th wheat seedling, and N coordinates are generated with p as the coordinates. i A circular region with center σ and radius σ is defined, where pixels inside the region have a value of 1 and pixels outside the region have a value of 0, thus obtaining a local segmentation image G of the wheat seedling. The radius σ of the circular region determines the size of the target area at the root and stem of each wheat seedling. The generation process of the local segmentation image G and the radius σ of the circular region is as follows:

[0105]

[0106]

[0107]

[0108] In the formula, p is the pixel position in the local segmentation image; p i Represents the coordinates of the i-th labeled point; P = {p i |i∈{1,...,N}} represents the coordinates of all labeled points on the image; R(w,h) is the coordinates of all labeled points p. i The region is a rectangular area centered at a point with width w and height h (w and h are hyperparameters); |R(w, h)| is the number of markers contained within the rectangular area; a is any marker within the region R(w, h); K is the number of the K nearest markers to a (hyperparameter); d k，a Let be the Euclidean distance between the k-th labeled point and a.

[0109] It should be noted that the local segmentation image is a binary image, wherein the area within the circle with a pixel value of 1 is the target region of the wheat seedling root and stem that is of interest in this invention (i.e., the local context supervision information region), and the area outside the circle with a pixel value of 0 is the non-wheat seedling root and stem region.

[0110] 2) Generating the predicted segmentation map: The local feature extraction branch also includes a 3×3 convolutional layer; the local feature map F1 is first upsampled by 8 times and then input into the 3×3 convolutional layer for convolution processing to obtain the predicted segmentation map F. G The process of generating the predicted segmentation map is as follows:

[0111] F G =Conv(Up(F1))

[0112] In the formula, Up represents upsampling.

[0113] More preferably, the 8x upsampling process employs nearest neighbor interpolation, making the predicted segmentation map F... G The width and height of the feature map are consistent with those of the learning target (i.e., the local segmentation map G). The 3×3 convolutional layer is used to smooth the noise generated by upsampling, thereby obtaining a feature map with more stable mathematical properties.

[0114] 3) Optimization judgment: Based on the predicted segmentation map F G The loss function L is used to construct local segmentation branches with the local segmentation map G. G According to the loss function L G The target counting model is updated using backpropagation to obtain an updated target counting model; then, local feature extraction is performed again using the updated target counting model to obtain an optimized local feature map; the loss function L G Specifically:

[0115]

[0116]

[0117] In the formula, w is the weight; l is the superscript (value is 0 or 1); G l Let G be the tensor formed by the channel labeled l in the local segmentation graph; l |For tensor G l The sum of all numerical elements contained therein; |G| is the sum of all numerical elements in the local segmentation graph; F G l Let l be the tensor formed by the channel labeled l in the predicted segmentation map; mean is the average of all numerical elements contained in the tensor; γ is the hyperparameter.

[0118] It should be noted that the predicted segmentation map F generated by the local segmentation branch G The results are pixel-level binary classifications. To alleviate the sample imbalance problem between the foreground and background classes and reduce its impact on counting accuracy, this invention incorporates a loss function L for local segmentation branches. G .

[0119] S223, Feature Fusion Module (see...) Figure 6 The element-wise dot product mechanism of the model integrates global and local contextual information of the wheat seedlings to generate a fused feature map F2 (see module ③), specifically:

[0120] (1) Generating local enhanced feature maps: The feature fusion module includes a softmax function and a repeat function; after the optimized 2×H×W local feature map is input into the feature fusion module, it is classified by a softmax function to obtain two tensors with a scale of H×W, one of which is a tensor representing local feature information and the other is a tensor representing non-local feature information; the tensor representing local feature information is copied 256 times by the repeat function and spliced ​​to obtain a 256×H×W local enhanced feature map with the same number of channels as the global feature map F0;

[0121] It should be noted that the classification processing of the softmax function can normalize the value of each tensor element to [0, 1], which represents the probability that the corresponding pixel is judged by the network as representing local feature information or representing non-local feature information. The closer the value is to 1, the greater the probability that the candidate point is judged as representing local feature information.

[0122] (2) Feature fusion: The local enhanced feature map and the global feature map F0 are multiplied element-wise to obtain the fused feature map F2. Furthermore, the fused feature map F2 integrates the local contextual supervision information and global feature information of the target object, further enhancing the network's ability to identify the target object, thereby effectively improving the model's accuracy in counting the target object.

[0123] S224 predicts the candidate point coordinates of wheat seedlings and their corresponding confidence scores through point regression branch and classification branch respectively (see module ④).

[0124] Furthermore, the point regression branch predicts M candidate point coordinates, and correspondingly, the classification branch generates M confidence scores. During the training phase, the one-to-one matching strategy proposed by P2PNet is first used to perform one-to-one matching between the candidate point coordinates generated by the network and the labeled point coordinates; the N candidate point coordinates that successfully match the labeled point coordinates are the predicted wheat seedling location coordinates, and their corresponding confidence score labels are 1. The remaining candidate point coordinates are classified as background points, and the confidence score labels corresponding to these background points are 0.

[0125] S3, the trained target counting model is sequentially validated through the validation set and tested through the test set to obtain the optimal target counting model.

[0126] Performance testing:

[0127] 1. The impact of different local segmentation maps on target counting results

[0128] To investigate the impact of different local segmentation maps on target counting results, the inventors compared the target counting model obtained in Example 1 of this invention with the target counting model obtained in Comparative Example 1 on the same wheat seedling dataset using the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). MAE measures the network's counting accuracy; a smaller value indicates that the predicted number of wheat seedlings is closer to the true value. RMSE measures the network's stability; a smaller value indicates stronger stability and better robustness. The results are shown in Table 2.

[0129] It should be noted that the target counting model obtained in Comparative Example 1 of this invention is basically the same as that in Example 1, except that in step S222(2), the radius of the circular domain is no longer σ in Example 1, that is, it is no longer... To calculate the average, directly use The local segmentation map is obtained by using the radius of the circular domain, and then the target counting model is obtained.

[0130] Table 2. The Influence of Different Local Segmentation Maps on Counting Results for Wheat Seedlings

[0131] serial number MAE RMSE Example 1 6.56 8.08 Comparative Example 1 5.86 7.68

[0132] As shown in Table 2, using the local segmentation map of wheat seedlings obtained in Embodiment 1 of the present invention as the learning target of the local segmentation branch of wheat seedlings can achieve a more accurate counting effect. This indicates that the size of the target region of the root and stem of wheat seedlings (i.e., the local context supervision information region) is very important to the counting performance of the model.

[0133] 2. The impact of different target counting models on wheat seedling counting results

[0134] To explore the impact of different target counting models on the target counting results, the inventors compared the target counting model obtained in Example 1 of this invention with existing models CSRNet, CANet, SCAR, BL, DM-Count, and P2PNet on the same wheat seedling dataset in terms of MAE and RMSE performance. The results are shown in Table 3 and... Figure 7 As shown.

[0135] Table 3. The impact of different target counting models on wheat seedling counting results.

[0136] Model MAE RMSE CSRNet 26.98 31.71 CANet 34.25 41.19 SCAR 21.24 27.11 BL 6.62 9.45 DM-Count 6.54 9.97 P2PNet 6.60 9.46 P2P_Seg 5.86 7.68

[0137] As shown in Table 3, the target counting model P2P_Seg obtained in this invention has a MAE of 5.86 and an RMSE of 7.68, which are reduced by 0.74 and 1.78 respectively compared with the original P2PNet. Furthermore, compared with other counting methods, P2P_Seg also exhibits the smallest counting errors in both aspects. This indicates that the target counting model obtained in this invention improves the network's ability to identify wheat seedlings by enhancing local contextual supervision information, thereby reducing counting errors and improving the accuracy and stability of wheat seedling counting.

[0138] Figure 7 In the density map, darker colors indicate higher wheat seedling density. Visualizing the density map, existing models CSRNet, CANet, SCAR, BL, DM-Count, and P2PNet show poor counting results for wheat seedlings, with accuracy needing improvement. The generated density maps cannot directly identify the location of the wheat seedlings, failing to provide sufficient supporting information for downstream tasks. The last two columns show the predicted values ​​from P2PNet and P2P_Seg, respectively. Both output more intuitive wheat seedling coordinates. Because P2P_Seg introduces a local segmentation branch to enhance local contextual supervision, its predicted values ​​are closer to the true values ​​and have smaller counting errors when counting wheat seedlings affected by occlusion, overlap, and lighting (such as the images in rows 3-5). Furthermore, from rows 1-6, the wheat seedlings gradually become denser, and the images contain noise such as withered leaves and shadows caused by lighting, posing a significant challenge to existing counting network models in identifying wheat seedlings. However, the P2P_Seg proposed in this paper significantly improves the accuracy of wheat seedling counting by enhancing local contextual supervision information and focusing attention on the local root and stem parts of the wheat seedlings, allowing the network to ignore noise as much as possible. Furthermore, when processing wheat seedling images with varying densities, the target counting model P2P_Seg obtained in this invention achieves the best counting results, demonstrating good generalization performance.

[0139] Example 2

[0140] An image target counting method, the method comprising: acquiring an image to be identified, inputting the image to be identified into a target counting model, and obtaining a target counting result of the image to be identified; wherein the target counting model is a trained target counting model obtained by training the target counting model training method described in Example 1.

[0141] The image to be identified is a wheat seedling image, and the target counting result is the location and number of wheat seedlings.

[0142] Example 3

[0143] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the target counting model training method as described in Embodiment 1, or the image target counting method as described in Embodiment 2.

[0144] Example 4

[0145] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the target counting model training method as described in Embodiment 1, or the image target counting method as described in Embodiment 2.

[0146] In summary, this invention effectively overcomes the shortcomings of the prior art and has high industrial applicability. The above embodiments are intended to illustrate the substantive content of this invention, but are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the essence and scope of protection of this invention.

Claims

1. A method for training a target counting model based on enhanced local contextual supervision information, characterized in that, Includes the following steps: S1, Obtain a sample image set, which includes multiple sample images containing the target object and the corresponding point annotation results for each sample image; the point annotation results of the sample images are the location information corresponding to the annotation points of the target object; randomly divide the sample image set into a training set, a validation set, and a test set according to the proportion; S2, the sample images in the training set are input into the pre-constructed target counting model for counting, and the target counting result of the sample images is obtained. The target counting result is the location information of the target object obtained based on the target counting model. A loss function is constructed based on the target counting result and the point annotation result of the sample images. Based on the loss function, backpropagation is used to update the parameters of the target counting model to obtain the trained target counting model. The target counting model is obtained by adding a local segmentation branch and a feature fusion module between the basic network of P2PNet and the point regression branch and the classification branch, based on the localization and classification backbone VGG16. S3, the trained target counting model is validated sequentially through the validation set and tested through the test set to obtain the optimal target counting model; The local segmentation branch includes a local feature extraction module; the local feature extraction module is used to extract local features from the global feature map extracted by the base network to obtain a local feature map; the local segmentation branch combines the point annotation results of the sample image to optimize the local feature map to obtain an optimized local feature map; the feature fusion module is used to fuse the optimized local feature map with the global feature map to obtain a fused feature map, and then use the fused feature map as input to both the point regression branch and the classification branch; The feature fusion module is used to fuse the optimized local feature map with the global feature map in the following specific steps: Generating Local Enhanced Feature Maps: The feature fusion module includes a softmax function and a repeat function; after the optimized local feature map is input into the feature fusion module, it is classified by a softmax function to obtain two tensors of scale H×W, one of which represents local feature information and the other represents non-local feature information; the tensor representing local feature information is copied by the repeat function and concatenated to obtain a local enhanced feature map with the same number of channels as the global feature map; Feature fusion: The local enhanced feature map and the global feature map are multiplied element-wise to obtain the fused feature map.

2. The target counting model training method according to claim 1, characterized in that, The local feature extraction module includes three identical and sequentially connected dimensionality reduction convolutional modules and one 3×3 convolutional layer. The dimensionality reduction convolutional modules are used for channel compression of the input feature map. After the global feature map is input into the dimensionality reduction convolutional modules for channel compression, a first-order dimensionality reduction feature map with unchanged width and height and halved channels is obtained. The first-order dimensionality reduction feature map is then continuously input into two more dimensionality reduction convolutional modules for channel compression to obtain a second-order and a third-order dimensionality reduction feature map. After the third-order dimensionality reduction feature map is processed by the 3×3 convolutional layer, a local feature map with unchanged width and height and 2 channels is obtained.

3. The target counting model training method according to claim 2, characterized in that, The process of optimizing and judging the local feature map by combining the local segmentation branch with the point annotation results of the sample image is as follows: Generating a local segmentation map: First, based on the point annotation results of the sample image, generate a circular region with the coordinates of the annotated points as the center and σ as the radius; then, binarize the pixels inside and outside the circular region, assigning a value of 1 to the pixels inside the circular region, and otherwise assigning a value of 0; finally, obtain the local segmentation map G; the generation process of the local segmentation map G and the radius σ of the circular region is as follows: In the formula, p is the pixel position in the local segmentation image; p i Represents the coordinates of the i-th labeled point; P = {p i |i∈{1,...,N}} represents the coordinates of all labeled points; R(w,h) represents the coordinates of all labeled points p. i Let R(w, h) be a rectangular region centered at a point with width w and height h, where w and h are hyperparameters; |R(w, h)| is the number of labeled points contained within the rectangular region; a is any labeled point within the region R(w, h); K is the number of the K nearest labeled points to a, where K is a hyperparameter; d k，a Let a be the Euclidean distance between the k-th labeled point and a; Generating the predicted segmentation map: The local feature extraction branch also includes a 3×3 convolutional layer; the local feature map is first upsampled by 8 times and then input into the 3×3 convolutional layer for convolution processing to obtain the predicted segmentation map F. G ; Optimization judgment: Based on the predicted segmentation map F G The loss function L is used to construct local segmentation branches with the local segmentation map G. G According to the loss function L G The target counting model is updated using backpropagation to obtain an updated target counting model; then, local feature extraction is performed again using the updated target counting model to obtain an optimized local feature map; the loss function L G Specifically: In the formula, w represents the weight; l is a superscript with a value of 0 or 1; G l Let G be the tensor formed by the channel labeled l in the local segmentation graph; l |For tensor G l The sum of all numerical elements contained therein; |G| is the sum of all numerical elements in the local segmentation graph; F G I Let l be the tensor formed by the channel labeled l in the predicted segmentation map; mean is the average of all numerical elements contained in the tensor; γ is the hyperparameter.

4. The target counting model training method according to claim 3, characterized in that, In step S2, the loss function consists of the loss function of the point regression branch, the loss function of the classification branch, and the loss function L of the local segmentation branch. G Composition; the loss function L is specifically: L=L CE +λ1L P +λ2L G In the formula, λ1 and λ2 are both hyperparameters; L P The loss function for the point regression branch; L CE L is the loss function for the classification branch; G Let be the loss function for the local segmentation branch.

5. A method for counting targets in an image, characterized in that, The method includes: acquiring an image to be identified, inputting the image to be identified into a target counting model, and obtaining a target counting result of the image to be identified; wherein, the target counting model is a trained target counting model obtained by training using any of the target counting model training methods described in claims 1-4.

6. The image target counting method according to claim 5, characterized in that, The image to be identified is a plant seedling image, and the target counting result is the location and number of plant seedlings.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step in the target counting model training method as described in claims 1-4, and / or the image target counting method as described in claim 5 or 6.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements any step of the target counting model training method as described in claims 1-4, and / or the image target counting method as described in claim 5 or 6.

Images