Orchard pedestrian detection method, system, equipment and medium based on structural re-parameterization

Abstract

The present invention belongs to the technical field of pedestrian detection, and discloses a method, system, device and medium for orchard pedestrian detection based on structural re-parameterization. The present invention builds a target detection network model based on structural re-parameterization based on the improved DINO model. The structural re-parameterization method is used in the backbone network of the model. By re-parameterizing the model structure, the number of network parameters is greatly reduced under the premise that the detection accuracy is almost unchanged during the model deployment stage, thereby achieving the purpose of lightweight. In addition, a progressive bidirectional feature fusion network is introduced into the feature fusion module of the model to fuse high-order features to enhance the representation ability of the model. Different weights are added to the fusion of feature maps of different features within the module to distinguish the importance of different feature maps. The present invention also improves the model loss function metric, and uses the minimum point distance loss function for bounding box regression prediction.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and pedestrian detection, and in particular relates to an orchard pedestrian detection method, system, equipment and medium based on structural re-parameterization, which is particularly suitable for orchard pedestrian detection scenarios. Background Art

[0002] With the advancement of agricultural modernization and automation, the application of intelligent agricultural machinery in agricultural environments is becoming more and more extensive. However, in unstructured orchards, due to the particularity of the environment, the safety issues between agricultural machinery and pedestrians have gradually become prominent. Therefore, in order to ensure the safety of agricultural machinery operations in orchards, it is of great significance to propose a lightweight and high-precision orchard pedestrian detection method. Early orchard pedestrian detection methods mainly include one-dimensional fuzzy entropy segmentation and density peak cluster segmentation algorithms.

[0003] However, due to the complex environment in the orchard, pedestrian detection needs to consider many factors, such as occlusion, light intensity, etc. Due to the changeable posture and unpredictable movement trajectory of pedestrians, there may be overlaps with fruit trees, weeds and other backgrounds, similar colors, etc., which makes it difficult to detect. Therefore, the algorithm needs to be carefully adjusted and optimized to adapt to various situations. In addition, it is necessary to select a suitable artificial intelligence algorithm model for pedestrian detection in the orchard, and optimize the model to ensure the lightweight properties of the model, thereby improving the efficiency and detection accuracy of the model, so that the model can be transplanted to suitable hardware and used in actual orchards for pedestrian detection. At the same time, the applicability of different scenarios and the generalization ability of the model need to be considered. In addition to external factors, the complexity and robustness of the algorithm itself also need to be considered. When it is actually applied in the orchard scene, the algorithm needs to have a certain complexity and robustness so that it can work accurately in various situations.

[0004] The task of detecting pedestrians in an orchard is essentially a subtask of autonomous driving, which involves human safety. Therefore, the algorithm has higher requirements for lightness and robustness, because intelligent agricultural machinery needs to complete pedestrian detection in complex orchard scenes, and it is necessary to consider whether the camera resolution can meet the model requirements. In addition, the staff in the orchard often complete pruning, bagging, pollination and other tasks in different postures, and the existing algorithms are difficult to meet the above requirements.

[0005] Patent document 1 discloses a pedestrian detection method in an orchard environment based on improved YOLOv3. In this patent document, an improved K-means clustering method is used to extract anchor frames, and small target detection is performed through the YOLO-Z model, combined with the Kalman filter method to improve the missed detection rate of occluded objects. However, since the above model is improved based on YOLOv3, it is limited by the shortcomings of convolutional neural networks. Although the improved model has the advantages of small parameters and easy deployment, the local convolution operation based on the convolutional neural network makes the model weak in perception of long-distance pixel relationships and cannot handle global problems well. The detection accuracy is obviously not as good as other models based on Transformer blocks, such as the DINO model, the DETR model, etc.

[0006] References

[0007] Patent document 1 Chinese invention patent application, publication number: CN111626128A, publication date: 2020.09.04. Summary of the invention

[0008] In view of the problems of low accuracy and detection rate of existing algorithms for pedestrian detection in orchards, the present invention proposes an orchard pedestrian detection method based on structural reparameterization. This method is improved based on the DINO model. By using Transformer blocks, the detection accuracy and detection rate are improved, and the number of model parameters during deployment is reduced through the structural reparameterization method.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme:

[0010] The orchard pedestrian detection method based on structural reparameterization includes the following steps:

[0011] Step 1. Build a target detection network model based on structural reparameterization based on the improved DINO model;

[0012] The improved DINO model includes feature extraction module, attention module, feature fusion module and Transformer module;

[0013] The feature extraction module includes a convolution layer and four structure reparameterization modules, which are defined as structure reparameterization module 1, structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 respectively;

[0014] The four structural reparameterization modules are composed of different numbers of parallel multi-branch structures;

[0015] The Transformer module includes a position encoding layer, a Transformer encoder, and a Transformer decoder;

[0016] The signal processing flow in the improved DINO model is as follows:

[0017] The input image first enters the feature extraction module, and is preprocessed by the convolution layer. Four structure reparameterization modules are used to further extract features from the preprocessed feature map. The feature maps of structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 are used as the output of the feature extraction module.

[0018] Next, the feature maps of the three scales are enhanced through the attention module;

[0019] The feature map after feature enhancement is input into the feature fusion module for multi-scale feature fusion. After the feature maps with different channel numbers are fused to have the same channel number, the fused feature map is input into the Transformer module.

[0020] The position information is added to the fused feature map through the position encoding layer, and then input into the Transformer encoder. The hybrid query selection strategy is used to obtain the top K most likely positions of pedestrians, where K is a natural number.

[0021] The obtained output is input into the Transformer decoder, which outputs the final bounding box and predicted category.

[0022] Step 2. Train the improved DINO model, use the structure reparameterization method to reduce the number of parameters of the feature extraction module, and reparameterize the parallel multi-branch structure into a single-branch structure.

[0023] Step 3. Use the model after structural reparameterization as the final target detection network model for orchard pedestrian detection.

[0024] In addition, based on the above-mentioned orchard pedestrian detection method based on structural re-parameterization, the present invention also proposes a corresponding orchard pedestrian detection system based on structural re-parameterization, which adopts the following technical solutions:

[0025] An orchard pedestrian detection system based on structural re-parameterization is provided in the system, wherein a target detection network model based on structural re-parameterization is built based on an improved DINO model, and the improved DINO model includes:

[0026] Feature extraction module, attention module, feature fusion module and Transformer module;

[0027] The feature extraction module includes a convolution layer and four structure reparameterization modules, which are defined as structure reparameterization module 1, structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 respectively;

[0028] The four structural reparameterization modules are composed of different numbers of parallel multi-branch structures;

[0029] The Transformer module includes a position encoding layer, a Transformer encoder, and a Transformer decoder;

[0030] The signal processing flow in the improved DINO model is as follows:

[0031] The input image first enters the feature extraction module, and is preprocessed by the convolution layer. Four structure reparameterization modules are used to further extract features from the preprocessed feature map. The feature maps of structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 are used as the output of the feature extraction module.

[0032] Next, the feature maps of the three scales are enhanced through the attention module;

[0033] The feature map after feature enhancement is input into the feature fusion module for multi-scale feature fusion. After the feature maps with different channel numbers are fused to have the same channel number, the fused feature map is input into the Transformer module.

[0034] The position information is added to the fused feature map through the position encoding layer, and then input into the Transformer encoder. The hybrid query selection strategy is used to obtain the top K most likely positions of pedestrians, where K is a natural number.

[0035] The obtained output is input into the Transformer decoder, which outputs the final bounding box and predicted category.

[0036] After the training of the improved DINO model is completed, the trained model is used to reduce the number of parameters of the feature extraction module using the structural reparameterization method, and the parallel multi-branch structure is reparameterized into a single-branch structure to obtain the final target detection network model.

[0037] In addition, based on the above-mentioned method for detecting pedestrians in an orchard based on structural re-parameterization, the present invention further proposes a computer device, which includes a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, the steps of the above-mentioned method for detecting pedestrians in an orchard based on structural re-parameterization are implemented.

[0038] In addition, based on the above-mentioned orchard pedestrian detection method based on structural re-parameterization, the present invention also proposes a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the above-mentioned orchard pedestrian detection method based on structural re-parameterization.

[0039] The present invention has the following advantages:

[0040] As described above, the present invention relates to a method for detecting pedestrians in an orchard based on structural reparameterization, which is based on an improved DINO model to build a target detection network model based on structural reparameterization. Among them, the method of structural reparameterization is used in the backbone network of the model, that is, the feature extraction module. By reparameterizing the model structure, the network parameter amount is greatly reduced in the model deployment stage without changing the detection accuracy, thereby achieving the purpose of lightweight. In addition, a progressive bidirectional feature fusion network is introduced into the feature fusion module of the model to fuse high-order features to enhance the representation ability of the model. Different weights are added to the fusion of feature maps of different features within the module to distinguish the importance of feature maps of different scales. In addition, the present invention also improves the loss function metric of the model, uses the minimum point distance loss function for bounding box regression prediction, and improves the regression accuracy and detection accuracy of the bounding box. In addition, the present invention divides the actions, postures, distances and different degrees of occlusion of pedestrians in the data set into different subsets to enrich the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A model architecture diagram of a target detection network model constructed in an embodiment of the present invention;

[0042] Figure 2 A network structure diagram of a parallel multi-branch structure in an embodiment of the present invention;

[0043] Figure 3 Schematic diagram of a progressive bidirectional feature fusion network in an embodiment of the present invention;

[0044] Figure 4 It is a network structure diagram after the parallel multi-branch structure in the embodiment of the present invention is re-parameterized into a single-branch structure;

[0045] Figure 5 This is a network structure diagram of a channel attention module in an embodiment of the present invention;

[0046] Figure 6 A network structure diagram of a spatial attention module in an embodiment of the present invention;

[0047] Figure 7 This is a network structure diagram after the parallel multi-branch structure completes the ID subtask in an embodiment of the present invention;

[0048] Figure 8 This is a network structure diagram after the parallel multi-branch structure completes the CB subtask in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0050] Example 1

[0051] In order to reduce the potential safety hazards of unmanned agricultural machinery working in orchards to pedestrians, the present invention provides an orchard pedestrian detection method based on structural reparameterization. Pedestrians are detected by constructing a CNN-Transformer hybrid network model, and pedestrian features are extracted and fused using an improved DINO model. Multi-scale detection of pedestrians is achieved using a contrast denoising training method.

[0052] Specifically, the orchard pedestrian detection method based on structural re-parameterization in this embodiment includes the following steps:

[0053] Step 1. Build a target detection network model based on structural reparameterization based on the improved DINO model. The improved DINO model includes a feature extraction module, an attention module, a feature fusion module, and a Transformer module, such as Figure 1 shown.

[0054] The present invention improves the original DINO model in terms of feature extraction module, feature fusion module and loss function measurement.

[0055] The feature extraction module, that is, the backbone network of DINO, is a feature extraction module with structural reparameterization.

[0056] The feature extraction module includes a convolutional layer and four structure reparameterization modules, which are defined as structure reparameterization module one, structure reparameterization module two, structure reparameterization module three, and structure reparameterization module four respectively.

[0057] The four structural reparameterization modules are composed of different numbers of parallel multi-branch structures.

[0058] Specifically, the numbers of parallel multi-branch structures in the feature graphs of structure reparameterization module one, structure reparameterization module two, structure reparameterization module three, and structure reparameterization module four are 2, 4, 14, and 1, respectively.

[0059] like Figure 2 A network structure diagram of a single parallel multi-branch structure is shown. The parallel multi-branch structure includes three branches, namely a first branch, a second branch and a third branch, and the third branch is a residual branch.

[0060] Among them, a 1×1 convolution layer and a normalization layer are provided on the first branch; a 3×3 convolution layer and a normalization layer are provided on the second branch; and a normalization layer is provided on the third branch.

[0061] The input feature map is processed by the first branch, the second branch and the third branch respectively; the output feature maps of the first branch, the second branch and the third branch are added and passed through the activation function to serve as the output feature map of the parallel multi-branch structure.

[0062] The feature fusion module adopts a progressive bidirectional feature fusion network. Its overall structure and basic components are as follows: Figure 2 As shown in Figure 3, the bidirectional feature network consists of a top-down traditional feature fusion method and a bottom-up fusion path.

[0063] Among them, the top-down fusion method combines the deep semantic information of the feature map with the shallow texture information, and the bottom-up fusion path enhances the feature representation ability of the model and transmits part of the location information of the target.

[0064] In addition, the input nodes and output nodes at the same level are residually connected, which integrates more pedestrian features in the orchard without almost increasing the computational cost. The above two paths and the residual connection form a feature fusion unit, and the three feature fusion units are connected in series to form a progressive bidirectional feature fusion network in the present invention.

[0065] like Figure 3 As shown in the figure, the left node of each basic component unit in the feature fusion module is regarded as an input node, the middle node is regarded as a relay node, and the right node is regarded as an output node. Each line from top to bottom is a feature map of different scales, and an additional learnable weight is added to each fusion, which can ensure that the contribution of different feature maps to the fusion process is equally distributed to the greatest extent. The calculation formula relationship between the input and output feature maps is as follows:

[0066]

[0067] Among them, O represents the output feature map, I i represents the input feature map, ω i and ω j They represent the parameters learned through model training, and ε is 1×10 -4 To prevent the denominator from being 0 and ensure the stability of the calculated value, i represents the number of feature maps that the node needs to fuse, j represents the number of feature map weights, i,j∈N * , N * Is a positive integer.

[0068] exist Figure 2 In the example, taking the P6-level node in the first basic component unit as an example, the calculation formula between the input feature map and the output feature map at the P6 level is as follows:

[0069]

[0070]

[0071] in, It is the relay feature node of the top-down path of the P6 level feature map. is the output feature node in the bottom-up path of the P6 level feature map. Similarly, Represents the feature output node of the P5 level feature map, and Represents the input nodes of the feature maps of the P6 level and P7 level respectively. The Resize() function adjusts the feature maps of different resolutions to feature maps of the same size by upsampling or downsampling. DSConv() represents the depthwise separable convolution operation;

[0072] ω and ω indicate that the weights of each node are different under different computational convolution operations.

[0073] The feature fusion module aims to improve the fusion scale problem of the feature maps extracted by the feature extraction module in the process of pedestrian recognition in the orchard, and to improve the problem that the previous model fusion structure is too simple and the computational overhead is high.

[0074] After the improvement, the model is more suitable for detecting pedestrians in orchards under complex environments and at lower resolutions. In addition, the feature maps before and after weighting are connected in a residual manner to avoid feature loss caused by weighted fusion.

[0075] The following is combined with Figure 3 , the network structure and signal processing flow of the feature fusion module are explained in detail.

[0076] The feature fusion module is composed of three identical basic components in cascade. Each basic component includes five levels of nodes. Each level of nodes represents a feature map of different sizes, from large to small: P3, P4, P5, P6, P7;

[0077] Among them, the nodes at the P3, P4, and P5 levels are obtained by the output of the structure reparameterization module 2, the structure reparameterization module 3, and the structure reparameterization module 4 through the attention module, the nodes at the P6 level are obtained by the nodes at the P5 level after a 2x upsampling operation, and the nodes at the P7 level are obtained by the nodes at the P6 level after a 2x upsampling operation;

[0078] In each basic unit, the connection mode of nodes consists of three modes: top-down connection mode, bottom-up connection mode, and residual connection mode; the top-down connection mode consists of up-sampling connections, the bottom-up connection mode consists of down-sampling connections, and the residual connection is the connection path between the input node and the output node;

[0079] In the first basic unit, taking the P6-level feature map as an example, the signal processing flow of the feature fusion module is roughly as follows: the output feature map of the structure reparameterization module 4 is passed through the attention module and downsampling to obtain the P6-level input node By connecting to the node The feature map obtained after executing the Resize() operation is subjected to weighted convolution fusion calculation to obtain the relay node

[0080] Next, the output feature map node at the P5 level After executing the Resize() operation, and The nodes perform weighted convolution fusion calculations to obtain the final P6-level output node

[0081] The calculation of other nodes is similar.

[0082] The Transformer module consists of a position encoding layer, a Transformer encoder, and a Transformer decoder.

[0083] The position encoding layer in the original DINO model adds position information to the fused feature map. In the Transformer part, the feature map with position information is flattened and input into the encoder with self-attention. A hybrid query selection strategy is used in the output of the encoder to obtain the top K most likely locations of pedestrians and input them into the decoder.

[0084] In addition, the output of the encoder and the features learned in training are input into the decoder together. Finally, the output obtained in the above operation is used to obtain the refined anchor box and pedestrian features after decoding by the decoder, and the final bounding box and predicted category are output through the FFN prediction head. After a certain period of training, the trained improved DINO model is obtained.

[0085] The signal processing flow in the improved DINO model is as follows:

[0086] The input image first enters the feature extraction module, and is preprocessed by the convolution layer. Four structure reparameterization modules are used to further extract features from the preprocessed feature map. The feature maps of structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 are used as the output of the feature extraction module.

[0087] Next, the feature maps of the three scales are enhanced through the attention module;

[0088] The feature map after feature enhancement is input into the feature fusion module for multi-scale feature fusion. After the feature maps with different channel numbers are fused to have the same channel number, the fused feature map is input into the Transformer module.

[0089] The position information is added to the fused feature map through the position encoding layer, and then input into the Transformer encoder. The hybrid query selection strategy is used to obtain the top K most likely positions of pedestrians, where K is a natural number.

[0090] The obtained output is input into the Transformer decoder, which outputs the final bounding box and predicted category.

[0091] The attention module includes a channel attention module and a spatial attention module, which are used to enhance the information of the obtained feature map so that the model can pay more attention to the contour, posture, movement and other aspects of the pedestrian.

[0092] like Figure 5 As shown in the figure, the channel attention module performs maximum pooling and average pooling calculations on each channel of the input feature map, and inputs the pooled vector into the fully connected layer with parameter sharing. Through learning, the channel attention module can adaptively determine the important channels, and intersect the global maximum feature vector with the average feature vector to obtain the final attention weight vector; after activation by the activation function, each channel of the original feature map is multiplied by the vector to obtain the channel feature map;

[0093] like Figure 6 As shown in the figure, the spatial attention module obtains features of different context scales by performing maximum pooling and average pooling operations along the channel dimension, generates spatial attention weights through the convolution layer, and after activation by the activation function, applies the spatial attention weights to the original feature map to weight the spatial position to reduce the influence of non-important areas.

[0094] Step 2. Train the improved DINO model, use the structural reparameterization method to reduce the number of parameters of the feature extraction module, and reparameterize the parallel multi-branch structure into a single-branch structure.

[0095] Before model training, the acquired dataset is first segmented into subsets using a certain scheme. The dataset subset segmentation principles include the motion state of pedestrians, the behavior posture of pedestrians, the distance between pedestrians and unmanned agricultural machinery, and the degree of occlusion of pedestrians by fruit trees or weeds. In addition, the dataset is preprocessed and converted into the COCO format.

[0096] The dataset modified into COCO format is imported into the improved DINO model. During training, the image is first input into the feature extraction module. The size of each image is 3*720*480, where 3 channels represent RGB channels.

[0097] The present invention improves the original DINO model to address the problem of insufficient bounding box regression accuracy.

[0098] The original model used the generalized intersection-over-union (GIoU) ​​as the bounding box loss function. Although GIoU takes into account the overlapping area of ​​the bounding boxes and adds a penalty term, when the predicted box is inside the real box, the loss function cannot distinguish the relationship between the two.

[0099] In order to solve this limitation, the present invention proposes a loss function based on the minimum point distance intersection ratio. The principle of this loss function is to complete the regression of the loss function by minimizing the distance between the corner points of the predicted bounding box and the true bounding box. For the corner points, the upper left corner and the lower right corner of the bounding box are generally taken as the calculation basis.

[0100] The calculation formula for the squared Euclidean distance between the predicted box and the real box corner point is as follows:

[0101]

[0102] in, represents the square of the Euclidean distance between two points, Indicates the horizontal coordinate of a corner point of the real box, Represents the horizontal coordinate of a corner point of the prediction box, Indicates the ordinate of a corner point of the real box, Represents the ordinate of a corner point of the prediction box; n∈{1,2}, when n is 1, (x1,y1) represents the coordinates of the upper left corner of the bounding box. Similarly, when n is 2, (x2,y2) represent the coordinates of the lower right corner of the bounding box.

[0103] The formula for the minimum point distance loss function is as follows:

[0104]

[0105] Where h and w are the height and width of the input image, is the true bounding box, is the bounding box predicted by the model. and The ratio of the intersection to the union is the calculation formula of the intersection-union ratio. The ordinary intersection-union ratio loss function can only calculate the union area of ​​two bounding boxes and cannot distinguish the case where the two boxes have no overlapping area. hour, In this case, IoU cannot reflect the positional relationship of the two bounding boxes.

[0106] The more common loss functions include: GIoU, EIoU, CIoU, DIoU, etc., which calculate the regression of the bounding box by calculating the minimum circumscribed rectangular area, aspect ratio deviation and center point distance between the predicted box and the true box. All factors in these bounding box regression loss functions can be determined by the coordinates of the four corner points of the true box and the predicted box. The minimum point distance intersection proposed in this invention better integrates the advantages of the existing bounding box loss function and simplifies the calculation process.

[0107] During model training, the basic units in the feature extraction module are used Figure 2 The parallel multi-branch structure shown in the figure, in which all the components in the dotted box constitute the basic components of the feature extraction module.

[0108] Figure 1 The structural reparameterization module 1, the structural reparameterization module 2, the structural reparameterization module 3 and the structural reparameterization module 4 are respectively composed of 2, 4, 14 and 1 of the above-mentioned multi-branch structures.

[0109] In order to achieve lightweight model in the inference stage, the present invention applies the structural reparameterization method in the feature extraction module. Figure 2 The parallel multi-branch structure shown is equivalent to being transformed into Figure 4 Single branch structure shown.

[0110] The present invention divides the task of the structure reparameterization method into a CB subtask, an ID subtask and a PR subtask.

[0111] The trained improved model uses the structural reparameterization method to reduce the number of parameters of the feature extraction module, and reparameterizes the parallel multi-branch structure into a single-branch structure to achieve the purpose of lightweight model.

[0112] Specifically, the structure reparameterization method is divided into ID subtask, CB subtask and PR subtask:

[0113] The ID subtask reconstructs the 1×1 convolutional layer and the residual branch on the first branch of the parallel multi-branch structure into 3×3 convolutional layers respectively. The network structure after the ID subtask is as follows: Figure 7 shown.

[0114] The CB subtask calculates and fuses the normalization layers of each branch in the parallel multi-branch structure into a 3×3 convolutional layer. By calculating the mean, standard deviation, bias term, and proportional coefficient of the normalization layer, the two structures of the 3×3 convolutional layer and the normalization layer connected in series on each branch are fused into one structure. The network structure after the CB subtask is as follows: Figure 8 shown.

[0115] The PR subtask is to calculate the parallel multi-branch structure after executing the CB subtask and the ID subtask, and fuse the convolution kernel weights and bias items of different branches into a single branch structure, that is, re-parameterize the parallel multi-branch structure into a single branch structure. The network structure after the PR subtask is as follows: Figure 4 shown.

[0116] In the structural reparameterization method, the number of channels of the convolution kernel in each convolution layer of the feature extraction module is the same as the number of channels of the input feature map, and the number of convolution kernels determines the number of channels of the output feature map.

[0117] For the normalization layer, it contains 4 parameters: mean μ, standard deviation σ, bias term β and scale factor γ, where β and γ are learnable parameters obtained through iterative training process;

[0118] The calculation formula after performing the CB subtask for the convolutional layer and the normalization layer is as follows:

[0119]

[0120]

[0121] Where W i and They represent the weight of the i-th convolution kernel in the convolution layer before and after the CB subtask respectively; represents the bias term of the i-th convolution kernel in the convolution layer after executing the CB subtask; β i , γ i , σ i , μ i They represent the bias term, proportional coefficient, standard deviation and mean of the normalization layer in the ith channel respectively; ε is a constant value, which is 1×10 -4 , to ensure the stability of the calculation to prevent the denominator from being 0, i∈N * ;

[0122] For the PR subtask, each branch of the parallel multi-branch structure after executing the CB subtask is regarded as a convolution kernel of the same size for convolution operation. For the parallel multi-branch structure with N branches, the re-parameterized convolution layer weights and biases are as follows:

[0123]

[0124]

[0125] Among them, W inference and b inference Indicates the convolutional layer weights and bias terms after the parallel multi-branch structure is fused into a single-branch structure when the model is deployed. and b n merged Respectively represent the weight and bias of the convolutional layer after each branch in the parallel multi-branch structure executes the CB subtask and the ID subtask, n∈1,2,...,N

[0126] During model training, the attention module consists of Figure 5 The structure shown is Figure 6 The structures shown are connected in series. Figure 5 The input feature map F on the left is the output feature map of the structure reparameterization module 2, the structure reparameterization module 3, and the structure reparameterization module 4.

[0127] exist Figure 5 In the above example, the input feature map F is first calculated by parallel average pooling and maximum pooling operations, and the obtained vector is input into the parameter-sharing multi-layer perceptron. The attention weight vector is obtained by intersection operation, and the channel attention feature map F is obtained after activation function and residual link. C .exist Figure 6 In the input feature map F C is the output feature map of the above channel attention. The spatial attention is pooled along the channel dimension, and the spatial attention weight is generated through the convolution layer. After the activation function and residual connection, the spatial attention feature map F is obtained. S .

[0128] Step 3. Use the model after structural reparameterization as the final target detection network model for orchard pedestrian detection.

[0129] In addition, in order to verify the effectiveness of the method of the present invention, the lightweight model for detecting pedestrians in the orchard trained by the present invention achieved good results in pedestrian detection, and was compared with the training results of mainstream target detection models such as Faster R-CNN, YOLOX, Deformable DETR, and DCN. This example was experimented in the orchard pedestrian detection dataset proposed by Carnegie Mellon University (NREC), which is a mainstream image dataset for pedestrian detection in orchards. The dataset contains 95,924 images of two scenes, apple orchards and orange orchards. This example only uses apple orchards as the basis for training the model. The dataset contains 20,105 training set images, 9,781 validation set images, and 9,640 test set images, totaling 39,926 image samples. The dataset is divided into positive and negative samples in a ratio of 4:1, with a total of 31,426 positive samples and 8,500 negative samples. Among them, the positive samples of the training set, test set, and validation set are 15,535, 8,200, and 7,691, respectively, and the negative samples are 4,570, 1,981, and 1,949, respectively. In addition, the data size of the dataset is fixed to 720*480, providing a benchmark for model training and evaluation. This example improves the hyperparameter settings of the DINO model: the pixel size of the input image is adjusted to 720*480, and image enhancement methods such as mosaic, random rotation, and random scaling are performed; the training epoch is set to 24 rounds, and the category is adjusted to 2, including person and person-part categories; NVIDIA GeForce RTX3090 is used for model training and testing, and the batchsize is set to 4; an adaptive moment estimation optimizer with decoupled weight decay is used, and the initial learning rate and weight decay coefficient are set to 1×10 -4 and 1×10 -3 In addition, the number of layers of the four modules using the structural reparameterization method is set to 2, 4, 14, and 1 respectively; the encoder and decoder in the Transformer are set to 6 layers, and the number of predefined target queries in the model is 100. For other models, the conventional experimental scheme is followed, with Faster R-CNN set to 100 epochs, YOLOX set to 200 epochs, and the remaining target detection models set to 50 epochs.

[0130] In addition, this example conducts three random seed experiments, and takes the average of the three results as the final result for comparison of experimental data. The experimental data for comparing the performance of this example with other target detection models is shown in Table 1.

[0131] Table 1 Performance comparison of this example with other target detection models

[0132] Network Model mAP(%) mAP50(%) Detection rate (%) Parameter quantity (M) FLOPs(B) Faster R-CNN 31.2 51.6 35.5 41.353 0.197 YOLOX 28.3 51.2 38.2 - - Deformable DETR 35.5 57 52.9 40.099 0.184 DCN 36.4 52.6 42.6 41.934 0.17 DINO 39.7 56 64.9 47.542 0.261 Method of the present invention 43.9 63.2 67.4 28.392 0.101

[0133] It is not difficult to see from Table 1 that compared with the DINO model, the mAP of the method of the present invention is improved by 4.2%, the number of parameters is reduced by 40.2%, the detection rate is improved by 2.5%, and the FLOPs is reduced by 61.3%. Compared with the classic model based on convolutional neural network, the mAP value of the method of the present invention is improved by 12.7-15.6%, and the number of parameters is reduced by 31.3%. Compared with the CNN-Transformer hybrid model other than the DINO model, the mAP of the method of the present invention is improved by at least 6.3%, and the number of parameters is reduced by 13.542M, accounting for 32.29%. Through the above comparison, it is not difficult to find that the method of the present invention reduces the number of parameters of the model while slightly improving the accuracy, improves the detection rate of the model for pedestrians in the orchard, and achieves a good speed-accuracy trade-off.

[0134] In addition, the present invention also divides the data set according to different pedestrian postures, different occlusion levels, and different motion states of different pedestrians, and divides the different motion states of pedestrians into dynamic and static; the postures that are not common in the orchard, such as people falling, people lying under fruit trees, people lying flat on the road, and people sitting on the ground, are called "abnormal postures", and other postures are divided into classic postures, based on which the postures of different pedestrians are divided. In addition, the present invention also rates different occlusion states: unobstructed, partially obstructed, and severely obstructed. The rating is divided according to the visibility of the pedestrians, with the dividing points being 30% and 70%. Under different conditions, the detection effect of the model of the present invention is shown in Table 2.

[0135] Table 2 Detection effect of the model of the present invention under different conditions

[0136]

[0137] It can be seen from the above experiments that the present invention uses a structural reparameterization method to improve the model performance with improved accuracy while significantly reducing the number of parameters. Compared with the DINO model before the improvement, the model accuracy is improved by 4.2%, the model's detection rate for pedestrians in the orchard is improved by 2.5%, the number of parameters is reduced by 40.2%, and the detection rate is improved by 6.3% compared with the benchmark in this field.

[0138] The above test results show that the detection results of the improved DINO model proposed in the present invention exceed the benchmark data. Compared with other models, it is more suitable for displaying pedestrian detection in orchards and achieves accurate detection effects. Due to its lightweight properties, it is more suitable for transplantation on hardware. Therefore, the improved DINO model of the present invention has good generalization and robustness.

[0139] Example 2

[0140] This embodiment 2 describes an orchard pedestrian detection system based on structural re-parameterization, which is based on the same inventive concept as the orchard pedestrian detection method based on structural re-parameterization described in the above embodiment 1.

[0141] In the orchard pedestrian detection system based on structural re-parameterization in this embodiment, a target detection network model based on structural re-parameterization is built based on an improved DINO model. The improved DINO model includes:

[0142] Feature extraction module, attention module, feature fusion module and Transformer module;

[0143] The feature extraction module includes a convolution layer and four structure reparameterization modules, which are defined as structure reparameterization module 1, structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 respectively;

[0144] The four structural reparameterization modules are composed of different numbers of parallel multi-branch structures;

[0145] The Transformer module includes a position encoding layer, a Transformer encoder, and a Transformer decoder;

[0146] The signal processing flow in the improved DINO model is as follows:

[0147] The input image first enters the feature extraction module, and is preprocessed by the convolution layer. Four structure reparameterization modules are used to further extract features from the preprocessed feature map. The feature maps of structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 are used as the output of the feature extraction module.

[0148] Next, the feature maps of the three scales are enhanced through the attention module;

[0149] The feature map after feature enhancement is input into the feature fusion module for multi-scale feature fusion. After the feature maps with different channel numbers are fused to have the same channel number, the fused feature map is input into the Transformer module.

[0150] The position information is added to the fused feature map through the position encoding layer, and then input into the Transformer encoder. The hybrid query selection strategy is used to obtain the top K most likely positions of pedestrians, where K is a natural number.

[0151] The obtained output is input into the Transformer decoder, which outputs the final bounding box and predicted category.

[0152] After the training of the improved DINO model is completed, the trained model is used to reduce the number of parameters of the feature extraction module using the structural reparameterization method, and the parallel multi-branch structure is reparameterized into a single-branch structure to obtain the final target detection network model.

[0153] It should be noted that, in the orchard pedestrian detection system based on structural re-parameterization of this embodiment 2, the implementation process of the functions and effects of each functional module is specifically described in detail in the implementation process of the corresponding steps in the method in the above embodiment 1, and will not be repeated here.

[0154] Example 3

[0155] This embodiment 3 describes a computer device, which includes a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement the steps of the orchard pedestrian detection method based on structure re-parameterization in the above embodiment 1.

[0156] In this embodiment, the computer device is any device or apparatus with data processing capability, which will not be described in detail here.

[0157] Example 4

[0158] This embodiment 4 describes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the orchard pedestrian detection method based on structure re-parameterization in the above-mentioned embodiment 1.

[0159] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card, etc. equipped on the device.

[0160] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. The orchard pedestrian detection method based on structural re-parameterization is characterized by: The steps include: Step 1. Build a target detection network model based on structural reparameterization based on the improved DINO model; The improved DINO model includes feature extraction module, attention module, feature fusion module and Transformer module; The feature extraction module includes a convolution layer and four structure reparameterization modules, which are respectively defined as structure reparameterization module 1, structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4; The four structural reparameterization modules are composed of different numbers of parallel multi-branch structures; The Transformer module includes a position encoding layer, a Transformer encoder, and a Transformer decoder; The signal processing flow in the improved DINO model is as follows: The input image first enters the feature extraction module and is preprocessed by the convolution layer. Then, the four structure reparameterization modules are used to further extract features from the preprocessed feature map. The feature maps of structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 are used as the output of the feature extraction module. Next, the feature maps of the three scales are enhanced through the attention module; The feature map after feature enhancement is input into the feature fusion module for multi-scale feature fusion. After the feature maps with different channel numbers are fused to have the same channel number, the fused feature map is input into the Transformer module. The position information is added to the fused feature map through the position encoding layer, and then input into the Transformer encoder. The hybrid query selection strategy is used to obtain the top K most likely positions of pedestrians, where K is a natural number. The obtained output is input into the Transformer decoder, which outputs the final bounding box and predicted category. Step 2. Train the improved DINO model, use the structure reparameterization method to reduce the number of parameters of the feature extraction module, and reparameterize the parallel multi-branch structure into a single-branch structure. Step 3. Use the model after structural reparameterization as the final target detection network model to perform pedestrian detection in the orchard.

2. The orchard pedestrian detection method based on structure re-parameterization according to claim 1 is characterized in that: The numbers of parallel multi-branch structures in the feature graphs of the structure reparameterization module 1, the structure reparameterization module 2, the structure reparameterization module 3, and the structure reparameterization module 4 are 2, 4, 14, and 1, respectively; The parallel multi-branch structure includes a first branch, a second branch and a third branch; The first branch is provided with a 1×1 convolution layer and a normalization layer; the second branch is provided with a 3×3 convolution layer and a normalization layer; the third branch is a residual branch, and the third branch is provided with a normalization layer; The input feature map is processed by the first branch, the second branch and the third branch respectively; the output feature maps of the first branch, the second branch and the third branch are added and passed through the activation function to serve as the output feature map of the parallel multi-branch structure.

3. The orchard pedestrian detection method based on structure re-parameterization according to claim 2 is characterized in that: In step 2, the trained improved model uses a structural reparameterization method to reduce the number of parameters of the feature extraction module, and reparameterizes the parallel multi-branch structure into a single-branch structure; The structure reparameterization method is divided into ID subtask, CB subtask and PR subtask: The ID subtask will be parallelized on the first branch of the multi-branch structure The convolutional layer and residual branch are reconstructed as Convolutional layers; The CB subtask calculates and fuses the normalization layers of each branch in the parallel multi-branch structure into The convolution layer calculates the mean, standard deviation, bias term, and proportional coefficient of the normalization layer to connect the concatenated The convolutional layer and normalization layer structures are fused into one structure; The PR subtask is to calculate the parallel multi-branch structure after executing the CB subtask and the ID subtask, and fuse the convolution kernel weights and bias items of different branches into a single branch structure, that is, reparameterize the parallel multi-branch structure into a single branch structure; In the structural reparameterization method, the number of channels of the convolution kernel in each convolution layer of the feature extraction module is the same as the number of channels of the input feature map, and the number of convolution kernels determines the number of channels of the output feature map; For the normalization layer, there are 4 parameters: mean , Standard Deviation , bias term and the proportionality factor ,in, and It is a learnable parameter, obtained through iteration of the training process; The calculation formula for performing the CB subtask for the convolutional layer and the normalization layer is as follows: ； ； in and Respectively represent the convolutional layers before and after the CB subtask. The weight of the convolution kernel; Indicates the convolutional layer after executing the CB subtask. The bias term of the convolution kernel; , , , They represent the normalization layer in The bias term, scale factor, standard deviation and mean of each channel; is a constant value, ; For the PR subtask, each branch of the parallel multi-branch structure after executing the CB subtask is regarded as a convolution kernel of the same size for convolution operation. For the parallel multi-branch structure with N branches, the re-parameterized convolution layer weights and biases are as follows: ； ； in, and Indicates the convolutional layer weights and bias terms after the parallel multi-branch structure is fused into a single-branch structure when the model is deployed. and They represent the weight and bias of the convolutional layer after each branch in the parallel multi-branch structure executes the ID subtask and the CB subtask, respectively. .

4. The orchard pedestrian detection method based on structure re-parameterization according to claim 1 is characterized in that: The feature fusion module adopts a progressive bidirectional feature fusion network, and its network structure and signal processing flow are as follows: The feature fusion module is composed of three identical basic components in cascade. Each basic component consists of five levels of nodes. Each level of nodes represents a feature map of different sizes, from large to small: , , , , ; in, , , The nodes at the level are the outputs of the structure reparameterization module 2, the structure reparameterization module 3, and the structure reparameterization module 4 obtained through the attention module. The nodes of the level are The nodes of the level are obtained by 2 times upsampling operation. The nodes of the level are The nodes of the level are obtained by 2 times upsampling operation; In each basic unit, the connection mode of nodes consists of three modes, namely top-down connection mode, bottom-up connection mode, and residual connection mode; among them, the top-down connection mode is composed of up-sampling connection mode, the bottom-up connection mode is composed of down-sampling connection mode, and the residual connection is the connection path between the input node and the output node; The feature fusion module also adds a learnable weight mechanism for the fusion of each node to avoid the equivalent impact of feature graphs of different levels on the fusion process; the specific calculation formula is as follows: ； in, represents the output feature map, represents the input feature map, and They represent the parameters learned through model training, is a constant value used to ensure the stability of the calculated value. i represents the number of feature graphs that the node needs to fuse, and j represents the number of feature graph weights. , Is a positive integer.

5. The orchard pedestrian detection method based on structure re-parameterization according to claim 1 is characterized in that: The attention module includes a channel attention module and a spatial attention module; The channel attention module performs maximum pooling and average pooling calculations on each channel of the input feature map, and inputs the pooled vector into the parameter-sharing multi-layer perceptron. Through training, the channel attention module can adaptively determine the important channels, and intersect the global maximum feature vector with the average feature vector to obtain the final attention weight vector; after activation by the activation function, each channel of the original feature map is multiplied by the vector to obtain the channel feature map; The spatial attention module obtains features of different context scales by performing maximum pooling and average pooling operations along the channel dimension, generates spatial attention weights through the convolution layer, and after activation by the activation function, applies the spatial attention weights to the original feature map to weight the spatial position to reduce the influence of non-important areas.

6. The method for detecting pedestrians in an orchard based on structural reparameterization according to claim 1, characterized in that: In step 2, before model training, the acquired data set is first preprocessed; The preprocessing process includes dividing the acquired data set into different data set subsets according to the pedestrian's movements, postures, distances, and different degrees of occlusion, and converting the format of the preprocessed data set subsets into the COCO format.

7. The orchard pedestrian detection method based on structure re-parameterization according to claim 1 is characterized in that: In step 2, the loss function based on the minimum point distance intersection-over-union ratio is used during model training, and the calculation process is as follows: The calculation formula for the squared Euclidean distance between the predicted box and the real box corner point is as follows: ； in, represents the square of the Euclidean distance between two points, Indicates the horizontal coordinate of a corner point of the real frame, Represents the horizontal coordinate of a corner point of the prediction box, Indicates the ordinate of a corner point of the real box, Indicates the ordinate of a corner point of the prediction box; ,when When taking 1, Represents the coordinates of the upper left corner of the bounding box. Take 2, Respectively represent the coordinates of the lower right corner of the bounding box; the formula of the minimum point distance loss function is as follows: ； in and are the height and width of the input image, is the true bounding box, is the bounding box predicted by the model.

8. An orchard pedestrian detection system based on structural re-parameterization, characterized in that: Steps for implementing the method for detecting pedestrians in an orchard based on structural re-parameterization as described in any one of claims 1 to 7 above; In the orchard pedestrian detection system based on structural re-parameterization, a target detection network model based on structural re-parameterization is built based on the improved DINO model. The improved DINO model includes: Feature extraction module, attention module, feature fusion module and Transformer module; The feature extraction module includes a convolution layer and four structure reparameterization modules, which are defined as structure reparameterization module 1, structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 respectively; The four structural reparameterization modules are composed of different numbers of parallel multi-branch structures; The Transformer module includes a position encoding layer, a Transformer encoder, and a Transformer decoder; The signal processing flow in the improved DINO model is as follows: The input image first enters the feature extraction module, and is preprocessed by the convolution layer. Four structure reparameterization modules are used to further extract features from the preprocessed feature map. The feature maps of structure reparameterization module 2, structure reparameterization module 3, and structure reparameterization module 4 are used as the output of the feature extraction module. Next, the feature maps of the three scales are enhanced through the attention module; The feature map after feature enhancement is input into the feature fusion module for multi-scale feature fusion. After the feature maps with different channel numbers are fused to have the same channel number, the fused feature map is input into the Transformer module. The position information is added to the fused feature map through the position encoding layer, and then input into the Transformer encoder. The hybrid query selection strategy is used to obtain the top K most likely positions of pedestrians, where K is a natural number. The obtained output is input into the Transformer decoder, which outputs the final bounding box and predicted category. After the training of the improved DINO model is completed, the trained model is used to reduce the number of parameters of the feature extraction module using the structural reparameterization method, and the parallel multi-branch structure is reparameterized into a single-branch structure to obtain the final target detection network model.

9. A computer device comprising a memory and one or more processors; an executable code is stored in the memory; characterized in that: When the processor executes the executable code, it is used to implement the steps of the orchard pedestrian detection method based on structure re-parameterization as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon; characterized in that: When the program is executed by a processor, it is used to implement the steps of the orchard pedestrian detection method based on structure re-parameterization as described in any one of claims 1 to 7.

Images