Tree fall identification method, device, electronic device and program product based on deep learning

Abstract

The present application discloses a deep learning method, device, electronic device and program product for identifying fallen trees, wherein the method performs identification through a trained tree identification model. Among them, the backbone network and / or neck network of the tree identification model is provided with a C2fMSG module, which embeds the MSGhost obtained by multi-scale convolution in the lightweight network structure Ghost, which can not only greatly improve the recognition speed of fallen trees, but also facilitate the identification of fallen trees of different scales. Setting the W‑SPPELAN module after the last scale of the backbone network can help the model better integrate information from different scales, dynamically adjust the channel weights of different scales through self-learning, so as to enhance the model's ability to capture the effective features of fallen trees, thereby better coping with the problems of complex background environment and unclear identification features faced in identification, and improving the recognition accuracy of fallen trees.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a deep learning method for identifying fallen trees, a deep learning device for identifying fallen trees, an electronic device, and a computer program product. Background Art

[0002] With the acceleration of urbanization, greening has been given great attention as a key means to improve the ecological environment. Trees not only play an important role in beautifying the landscape, regulating the climate and purifying the air, but also reduce noise pollution. However, with the increase in the number of trees and changes in the natural environment, tree falling incidents have occurred frequently, posing a threat to the safety of life and property.

[0003] Traditional manual inspection and empirical judgment methods are inefficient and have limited coverage, making it difficult to meet real-time monitoring needs. In addition, trees in natural scenes have multi-scale characteristics with large differences in height and width, making it more difficult to identify fallen trees and low in accuracy. Summary of the invention

[0004] The present application provides a deep learning method for identifying fallen trees, a deep learning device for identifying fallen trees, an electronic device and a computer program product, which can reduce the difficulty of identifying fallen trees of different scales and improve the accuracy of the model for identifying fallen trees.

[0005] In a first aspect, the present application provides a deep learning method for identifying fallen trees, comprising:

[0006] The backbone network of the pre-trained tree recognition model extracts features from the image to be recognized to obtain image features; the image to be recognized includes roadside trees;

[0007] The neck network based on the tree recognition model fuses the image features to obtain the fused features;

[0008] The recognition network based on the tree recognition model recognizes the fused features and obtains the recognition result of the fallen trees in the image to be recognized;

[0009] The backbone network and / or the neck network is provided with a C2fMSG module, which is obtained by improving the C2f module by embedding a lightweight network structure with multi-scale convolutions. The C2fMSG module is used to extract and fuse multi-scale deep features.

[0010] In some embodiments, the C2fMSG module includes at least 2 levels of MSG Bottleneck structures, and the MSGBottleneck structure is obtained by improving the Bottleneck of the C2f module by embedding a lightweight network structure with multi-scale convolutions; for the first feature of the input C2fMSG module:

[0011] The first feature after the compression channel is input into the MSG Bottleneck structure to obtain multi-scale deep features;

[0012] The multi-scale deep feature is concatenated with the first feature after the channel compression to obtain a first concatenated feature;

[0013] The channels of the first concatenated feature are expanded to obtain a first output feature having the same scale as the first feature.

[0014] In some embodiments, the MSG Bottleneck structure includes at least 2 sequentially connected MSG Bottleneck layers; for inputting the second feature of each MSG Bottleneck layer:

[0015] Perform two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature;

[0016] Performing a depthwise convolution operation and a convolution operation on the second feature in sequence to obtain a second convolution feature;

[0017] The multi-scale feature is concatenated with the second convolutional feature to obtain the second output feature corresponding to the second feature. The multi-scale deep feature is obtained based on the fusion of the second output features of each MSG Bottleneck layer.

[0018] In some embodiments, the MSG Bottleneck layer includes a multi-scale convolution branch, which is used to perform two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature; the multi-scale convolution branch includes two MSGhosts stacked in sequence; for the third feature of each MSGhost input:

[0019] Perform at least two convolution operations with different convolution kernels on the third feature after the channel compression to obtain at least two corresponding third convolution features;

[0020] The third feature after the compression channel is concatenated with each third convolution feature to obtain a third output feature with the same scale as the third feature.

[0021] In some embodiments, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; a W-SPPELAN module is provided after the mth feature extraction module, and the W-SPPELAN module is used to fuse multi-scale feature information and dynamically adjust the channel weights of the fused feature information.

[0022] In some embodiments, for the fourth feature of each W-SPPELAN module input:

[0023] Input the fourth feature after the compression channel into the serial 3-level maximum pooling structure to obtain the pooling results corresponding to the maximum pooling structure of each level;

[0024] The fourth feature after the compression channel is concatenated with each pooling result to obtain a second concatenated feature;

[0025] Performing a normalization operation on each channel sub-feature of the second concatenated feature to determine a weight value of the sub-feature of each channel;

[0026] Perform weighted splicing on each sub-feature based on each weight value to obtain a third splicing feature;

[0027] Channel expansion is performed on the third concatenated feature to obtain a fourth output feature having the same scale as the fourth feature.

[0028] In some embodiments, the tree recognition model is trained based on regression loss and classification loss; the regression loss is:

[0029]

[0030]

[0031] in, is the regression loss; is the extended intersection-over-union loss; IoU is the intersection-over-union ratio of the predicted bounding box and the true bounding box, To control the degree of outlier suppression, is the intersection-and-join loss, is the center distance loss; is the width and height loss; b and Represent the center point of the real box and the center point of the predicted box respectively; Respectively represent the Euclidean distance between the center point of the real box and the predicted box; and Respectively represent the width and height of the minimum bounding rectangle.

[0032] In a second aspect, the present application provides a deep learning tree fall recognition device, comprising:

[0033] An extraction module is used to extract features of an image to be identified based on a backbone network of a pre-trained tree identification model to obtain image features; the image to be identified includes roadside trees;

[0034] A fusion module is used to fuse image features based on the neck network of the tree recognition model to obtain fusion features;

[0035] A recognition module is used to recognize fused features based on a recognition network of a tree recognition model to obtain a recognition result of fallen trees in an image to be recognized;

[0036] The backbone network and / or the neck network is provided with a C2fMSG module, which is obtained by improving the C2f module by embedding a lightweight network structure with multi-scale convolutions. The C2fMSG module is used to extract and fuse multi-scale deep features.

[0037] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0039] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0040] The first aspect of the present application has the beneficial effect compared with the prior art: a multi-scale convolution (Multi-Scale Ghost Module, MSGhost) block based on the C2f module, namely a C2fMSG module, is set in the backbone network and / or the neck network. Among them, MSGhost is a lightweight network structure embedded with multi-scale convolution, which can effectively extract and fuse effective image features of fallen trees of different scales while reducing computational complexity, which not only greatly improves the recognition speed of fallen trees, but also facilitates the recognition of fallen trees of different scales in the image. By extracting and fusing the image features of fallen trees of each scale through such a backbone network and / or neck network, it is possible to obtain fused features that fully characterize fallen trees of different scales, and recognition based on the fused features can improve the model's recognition efficiency and accuracy for fallen trees of different scales.

[0041] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 is a schematic diagram of the network structure of the tree recognition model provided in the embodiment of the present application;

[0044] Figure 2 is a schematic diagram of the network structure of another tree recognition model provided in an embodiment of the present application;

[0045] Figure 3 This is an image example obtained from the Trees Lodging dataset provided in the embodiment of the present application;

[0046] Figure 4 It is a flowchart of a tree fall identification method based on deep learning provided in an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of the network structure of the C2fMSG module provided in an embodiment of the present application;

[0048] Figure 6 It is a schematic diagram of the network structure of the MSGhost Bottleneck layer provided in an embodiment of the present application;

[0049] Figure 7 It is a schematic diagram of the network structure of MSGhost provided in an embodiment of the present application;

[0050] Figure 8 Schematic diagram of the network structure of the W-SPPELAN module provided in the embodiment of the present application;

[0051] Fig. 9 It is a structural schematic diagram of a tree fall identification device based on deep learning provided in an embodiment of the present application;

[0052] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0054] In the related technology, with the increase in the number of trees and changes in the natural environment, tree falling incidents occur frequently, posing a threat to the safety of life and property. However, traditional manual inspections and empirical judgment methods are inefficient and have limited coverage, making it difficult to meet real-time monitoring needs. In addition, trees in natural scenes have the characteristics of large differences in height and width, that is, scale diversity. For small trees, their proportion in the image is relatively small and they are easily submerged by background noise; for large trees, their local features may occupy most of the area in the image, resulting in other important information being ignored. In other words, trees of diverse scales increase the difficulty of identifying fallen trees and easily reduce the accuracy of identifying fallen trees.

[0055] In order to solve this problem, the present application proposes a tree recognition model, which includes a backbone network, a neck network and a detection network. Specifically, the backbone network is used to extract features from the input image to obtain image features; the neck network is used to fuse the image features to obtain fused features; the detection head is used to identify the fused features to obtain the recognition result of the fallen trees in the image to be identified.

[0056] In order to process images containing roadside trees of different scales, the present application sets a C2fMSG module in the backbone network. The C2fMSG module is improved by embedding a lightweight network structure (i.e., MSGhost) with multi-scale convolution to improve the C2f module. MSGhost can efficiently extract and fuse the key features of fallen trees of different scales while reducing the computational complexity. This structural design not only significantly improves the recognition speed of fallen trees, but also enhances the adaptability and recognition accuracy of the tree recognition model to fallen trees of different scales.

[0057] In some embodiments, the MSGhost module is applied to the bottleneck layer (Bottleneck layer) of the C2f module to improve it, and a MSGhost Bottleneck structure with at least two levels is designed. This structure further deepens the network depth by repeatedly using the MSGhostBottleneck layer, effectively optimizes the gradient transfer capability, and strengthens the model's ability to extract deep features of fallen trees of different scales, thereby improving the ability to express features of different scales, and thus significantly improving the model's recognition accuracy of fallen trees of different scales.

[0058] In some embodiments, given that the MSGhost Bottleneck structure will lead to an increase in network depth, in order to avoid problems such as gradient disappearance or increased training difficulty caused by the increase in network depth, the design of the MSGhost Bottleneck layer draws on the idea of ​​the basic residual block in ResNet. Specifically, the MSGhost Bottleneck layer includes two branches, one of which includes a multi-scale convolution branch stacked by two MSGhosts, which is mainly used to fully extract image features of different scales; in addition, it also includes a short connection composed of deep convolution and ordinary convolution to effectively solve the problem of gradient loss, making it possible to build a deep network.

[0059] The MSGhost Bottleneck layer introduces short connections into the structure, which not only effectively alleviates the gradient loss problem, but also improves the training efficiency and the expressiveness of the model while maintaining the network capacity, thereby achieving stable training of deeper networks.

[0060] In some embodiments, MSGhost improves the second part of the Ghost module, decomposes its channels according to the number of scales, divides them into at least two scales, and then performs convolution operations on them respectively. For example, three scales of k=3, 5, and 7 can be set to extract multi-scale features respectively.

[0061] Exemplarily, it is assumed that the tree recognition model is improved based on the YOLO series model. Exemplarily, for YOLOv5s / YOLOv8s, the backbone network and / or the C3 module of the neck network of YOLOv5s will be replaced with the C2fMSG module having the structure of any of the above embodiments, or the backbone network and / or the C2f module of the neck network of YOLOv8s will be replaced.

[0062] In addition to the variety of tree scales, there are also problems with complex environmental backgrounds and unclear features in the process of tree fall identification. Complex environmental backgrounds may cause a lot of irrelevant information to interfere with target identification, such as changes in lighting conditions, mutual occlusion between trees, and diverse background noise. The problem of unclear features is reflected in the fact that the color of the tree trunk is similar to the environment, the shape contour is incomplete, and the surface lacks significant texture. These factors together increase the difficulty of image processing and feature extraction, thereby placing higher requirements on the accuracy and robustness of tree fall identification.

[0063] In some embodiments, in order to improve the accuracy of the model in identifying fallen trees of different scales in complex scenarios, a W-SPPELAN module may be introduced into the backbone network.

[0064] The W-SPPELAN module combines the ideas of Spatial Pyramid Pooling (SPP) and Efficient Layer Aggregation (ELAN):

[0065] The SPP module is a technology used for image recognition and target recognition. It extracts and encodes features at different scales, scales input images of any size to a fixed size, and generates a feature vector of fixed length. The SPP module performs maximum pooling (MaxPooling) on ​​features through pooling kernels of different sizes (such as 5×5, 9×9, and 13×13) to extract larger scale (global) information, and concatenates these features to achieve multi-scale feature fusion.

[0066] ELAN optimizes feature representation capabilities and improves feature fusion efficiency through an efficient hierarchical aggregation mechanism, thereby enhancing the network's expressiveness. Therefore, combining the advantages of the SPP module and ELAN, the W-SPPELAN module was designed.

[0067] While integrating multi-scale feature information, the W-SPPELAN module can dynamically adjust channel weights and assign higher weights to key features. This network structure design can not only effectively suppress the interference of background noise on recognition, but also enhance the tree recognition model's ability to recognize the characteristics of fallen trees at different scales, thereby significantly improving the robustness and accuracy of recognition and meeting the recognition needs in complex scenarios.

[0068] Specifically, if the backbone network includes m feature extraction modules of different scales, m is a positive integer, and m≥2; the W-SPPELAN module will be set after the mth feature extraction module.

[0069] For example, if the base network of the tree recognition model is YOLOv5s / YOLOv8s, the improved Spatial Pyramid Pooling –Fast (SPPF) module of the backbone network of YOLOv5s / YOLOv8s will be replaced with the W-SPPELAN module.

[0070] In some embodiments, given that Max Pooling plays a key role in the SPP module, it effectively compresses the feature size and reduces computational complexity by selecting the maximum value in each small area, while retaining key information, thereby improving operating efficiency. It has strong robustness to slight offsets, rotations or deformations of the input features, and gives the network certain translation and rotation invariance. In addition, Max Pooling can extract significant features (such as edges and textures) in each small area, enhance the network's ability to focus on important parts of the image, and improve feature expression and discrimination performance. These characteristics make it a commonly used feature dimensionality reduction and extraction tool in deep learning.

[0071] Therefore, the W-SPPELAN module still retains Max Pooling. After the compression channel module, a serial 3-level Max Pooling structure is set. The pooling kernel of each level of Max Pooling is the same, for example, 5×5. On the one hand, this setting can obtain more feature information and improve the recognition accuracy of the network; on the other hand, under the condition of ensuring similar accuracy and the same receptive field, it reduces the amount of calculation and thus improves the network recognition speed.

[0072] Specifically, the output of the pooling result of each level will be spliced ​​at the channel level with the features after the compressed channel; the W-SPPELAN module is also equipped with a weight module for calculating the weights of different channels of the splicing results, so as to dynamically adjust the channel weights and give higher weights to key features.

[0073] By setting receptive fields of different sizes and combining adaptive channel weight learning, the W-SPPELAN module can fully capture contextual semantic information and make comprehensive judgments, effectively addressing the challenges of complex background environment and unclear recognition features in tree falling identification, thereby significantly improving the recognition accuracy of fallen trees.

[0074] For example, if the YOLOv8s basic network is improved, specifically by using the C2fMSG module and the W-SPPELAN module, the network structure of the tree recognition model can be found in Figure 1 .

[0075] For example, see Figure 2 In order to further enhance the neck network's ability to fuse features of different scales, if the backbone network is divided into four feature extraction stages, the features extracted in the first two stages (the first stage and the second stage) can be scaled down by downsampling, thereby realizing channel-dimensional feature splicing with features of the same scale in the neck network. In this way, it can be ensured that the fused features output by the neck network to the detection heads of different scales can contain the feature information extracted in each stage, thereby improving the model's ability to recognize multi-scale targets.

[0076] Specifically, the features of the first two stages are downsampled to a scale that matches the subsequent stages, and then the features of the channel dimension are spliced ​​in the neck network. In this way, the fused features of different scales can contain both shallow features (such as edge information) and deep features (such as semantic information), thereby achieving a comprehensive capture of target details and global semantics. This structural improvement effectively improves the multi-scale feature expression capability of the neck network, and further enhances the robustness and accuracy of the tree recognition model in complex scenes and multi-scale tree fall recognition.

[0077] In some embodiments, in order to ensure that the trained tree recognition model can meet expectations and accurately understand fallen trees in different scenes and at different scales, a special dataset can be created for training the tree recognition model. Specifically, a dataset called Trees Lodging is created. The dataset is collected by a vehicle-mounted camera under various environmental conditions. It includes 1,000 images with a resolution of 1280x720, and each image is annotated (the annotation method is not limited, and can be manually annotated or automatically annotated by a related algorithm). Figure 3 , Figure 3 An example of an image from the dataset is shown.

[0078] For example, in order to build a high-quality dataset, in addition to ensuring a sufficient number of images, attention should also be paid to the diversity of scenes. The collected images can be mainly urban roads, while also covering other scenes such as parks, suburbs, forests, etc., so as to improve the representativeness and adaptability of the dataset and meet the generalization requirements of the model in different environments.

[0079] For example, each scene can include multiple time periods and weather conditions such as day, night, sunny, rainy and foggy days to increase the complexity and comprehensiveness of the scene and enhance the model's adaptability to diverse environments. At the same time, each image can be equipped with a corresponding txt label file to detail the location, range and related feature information of fallen trees. For scenes in coastal cities, it is possible to focus on collecting and labeling data on fallen trees under extreme weather conditions such as typhoons to enhance the model's recognition ability in special environments and ensure its robustness and accuracy for actual complex application scenarios.

[0080] Such datasets can support research and applications in areas such as intelligent transportation systems, autonomous vehicles, and road safety monitoring to improve road safety and traffic efficiency.

[0081] In order to ensure the effectiveness of model training and the reliability of evaluation results, each image in the dataset of this application is divided into a training set and a validation set in a ratio of 8:2.

[0082] Preferably, in the process of dividing the data set, special attention should be paid to maintaining the balance of sample distribution between the training set and the validation set, ensuring that the two have similar statistical characteristics in terms of tree type, size, background environment, etc. This can avoid evaluation bias caused by uneven sample distribution and make the validation results more representative and credible.

[0083] In some embodiments, the loss of the above-mentioned tree recognition model mainly consists of two parts: regression loss and classification loss. The classification loss adopts binary cross entropy loss (BCE Loss) to determine the specific category to which the anchor box belongs. The regression loss consists of focal extended intersection over union loss (Focal-EIoU Loss) and distribution focal loss (DFLLoss), among which Focal-EIoU Loss is used to measure the overlapping error between the predicted bounding box and the true bounding box, and optimize the target recognition performance through the focal mechanism; DFL Loss further improves the fine regression accuracy of the bounding box. The matching strategy of positive and negative samples adopts task aligned learning dynamic matching (TAL), which can accurately match positive and negative samples, improve the distribution accuracy of positive samples, and thus enhance the robustness and recognition performance of the model.

[0084] Specifically, the formula of BCE Loss is as follows:

[0085]

[0086] in, L BCE is the binary cross entropy loss, n is the number of image samples of all fallen trees, y i For the i The true category of the image samples, is the predicted category of the i-th image sample.

[0087] The regression loss is used to calculate the error between the predicted border and the true border. In related technologies, the regression loss uses the CIOU loss (Complete Intersection over Union loss, L SIoU ), which is an improved version of commonly used losses such as L1, L2, Intersection over Union (IoU) and GIoU. L CIoUBy adding a penalty term for the aspect ratio, when the center points of the predicted bounding box and the true bounding box coincide, the errors in different situations can be better distinguished and scale invariant.

[0088]

[0089] in, b and b gt They are the center points of the predicted bounding box and the true bounding box respectively; ρ is the Euclidean distance between the predicted bounding box and the true bounding box; c The distance between the diagonal of the closed area of ​​the predicted border and the true border; v To ensure the consistency of the relative proportions between the predicted bounding box and the true bounding box, IoU It is the intersection-over-union ratio of the predicted bounding box and the true bounding box; α is the weight coefficient; w and are the width of the predicted border and the true border respectively; h and are the heights of the predicted bounding box and the true bounding box respectively.

[0090] Although CIoU takes into account three important factors of positioning loss: overlapping area, center point distance, and aspect ratio, in the CIoU formula, αv There are still some problems in this design, which slows down the convergence speed. It is reflected in the following three aspects:

[0091] (1) v It only reflects the difference in aspect ratio, but does not reflect the w and The relationship between h and That is to say, CIoU only needs the aspect ratio of the real box to be the same as the aspect ratio of the predicted box. v It is equal to 0, so this design is flawed. =8, =4, / =2, but the predicted w =6, h =3, w / h =2, v =0.

[0092] (2) v right w and h The partial derivatives are as follows:

[0093]

[0094] From the two partial derivatives we can see that: and The signs of are opposite, that is, as long as w and h If one of them increases, the other decreases. w < and h < ,or w > and h > The situation that occurs indicates that it is unreasonable.

[0095] (3) Due to v Only reflecting the difference in aspect ratio, CIoU loss may optimize the similarity (of aspect ratio) in an unreasonable way. For example, the height and width of the real box are: =1, =1, the predicted initial height and width are w =1, h =2.4. After multiple iterations, for example, 50 iterations, the height and width of the predicted box w =1.64, h = 2.84. The CIoU loss does increase the similarity of aspect ratios, but it prevents the model from effectively reducing ( w , h )and( , ) between the two.

[0096] In order to solve the CIoU problem, the EIoU loss splits the aspect ratio penalty into a height-width penalty (height penalty + width penalty) based on the CIoU loss penalty, and adds Focal to focus on high-quality prediction boxes.

[0097] EIoU loss consists of three parts: intersection over union loss, center distance loss, and width and height loss. The first two parts of the regression loss in this application continue the method in CIoU, and the final width and height loss is split into height penalty + width penalty. This not only speeds up the convergence speed, but also makes the predicted positioning results more accurate.

[0098]

[0099] in, To expand the intersection-over-union loss, IoU is the intersection-over-union ratio of the predicted bounding box and the true bounding box. To control the degree of outlier suppression, is the intersection-and-join loss, is the center distance loss; is the width and height loss; b and Represent the center point of the real box and the center point of the predicted box respectively; Respectively represent the Euclidean distance between the center point of the real box and the predicted box; and Respectively represent the width and height of the minimum bounding rectangle.

[0100] Considering that there is also an imbalanced training sample problem in bounding box regression, that is, the number of prediction boxes with small prediction errors and high quality in an image is far less than the number of prediction boxes with large errors and low quality. The prediction boxes with poor quality will produce too large gradients and affect the training process. Based on EIoU and Focal Loss, this paper proposes Focal EIoULoss, the formula is as follows:

[0101]

[0102] in, A parameter that controls the degree of outlier suppression.

[0103] The formula for DFL loss is as follows:

[0104]

[0105] Among them, DFL loss is written as , S i is the cross entropy loss between the real border and the predicted border on the left, S i+1 It is the cross entropy loss between the real box and the predicted box on the right.

[0106] Total loss of the tree recognition model Loss The formula is as follows:

[0107]

[0108] Among them, λ 1 and λ 1 is the balance coefficient.

[0109] Training the tree recognition model described in any of the above embodiments based on the total loss of the tree recognition model can improve the model convergence speed, thereby improving the efficiency of model training. By continuously iterating the tree recognition model, the tree recognition model can better and more stably recognize different tree falling states, so that a robust version can be selected from multiple converged versions as the trained tree recognition model.

[0110] In some embodiments, in order to comprehensively and accurately measure the performance of each version of the tree recognition model, after at least one version of the tree recognition model is converged based on the training set, the converged model can be evaluated through the validation set to avoid model overfitting, verify the generalization of the model, and ensure that the model can run stably after deployment.

[0111] Specifically, the performance of the tree recognition model on the validation set can be evaluated according to preset conditions.

[0112] For example, these preset conditions may include performance indicators such as the intersection-over-union ratio, recognition accuracy and recall rate of the tree recognition device for road diseases.

[0113] IoU is an indicator to evaluate the overlap between the predicted border and the true border. It calculates the ratio of the intersection and union of the predicted border and the true border. It plays a key role in determining whether it is a correct recognition. The calculation formula of IoU can be written as:

[0114]

[0115] Precision (p), also known as the precision rate, refers to the proportion of correct positive predictions to all positive predictions, as shown in the formula:

[0116]

[0117] Recall (R), also known as recall rate, refers to the proportion of correctly predicted positives to all actual positives, as shown in the formula:

[0118]

[0119] The average precision (AP) is calculated from the precision and recall. A line graph of the precision is drawn based on the recall value, and the area under the line is calculated, as shown in the formula:

[0120]

[0121] The mean average precision (mAP) refers to the average value of the average precision AP of C different road disease categories, as shown in the formula:

[0122]

[0123] That is to say, after verifying each version of the tree recognition model through the verification set, the tree recognition model of each version can be comprehensively evaluated based on the above-mentioned indicators, so as to determine the tree recognition model with the best performance from each version as the trained tree recognition model.

[0124] Based on the tree recognition model network structure in the above embodiment, this application proposes a deep learning tree fall recognition method, which aims to identify whether there are fallen trees in the image to be recognized (such as a scene containing street trees). This is a binary classification task. In addition, in addition to the identified tree fall events, this method can also achieve accurate labeling to assist in the implementation of subsequent processing or response measures.

[0125] The deep learning tree fall recognition method provided in the embodiment of the present application can be applied to electronic devices with computing capabilities, such as mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) equipment, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0126] In order to illustrate the technical solution proposed in this application, each embodiment will be described below with an electronic device as the execution subject.

[0127] Figure 4 A schematic flow chart of a deep learning method for identifying fallen trees provided by the present application is shown. The deep learning method for identifying fallen trees includes:

[0128] Step 410: The electronic device extracts features of the image to be identified based on the backbone network of the pre-trained tree identification model to obtain image features.

[0129] Step 420: The electronic device fuses the image features based on the neck network of the tree recognition model to obtain fused features.

[0130] Step 430: The electronic device identifies the fused features based on the recognition network of the tree recognition model to obtain a recognition result of fallen trees in the image to be identified.

[0131] In this embodiment, a multi-scale convolution module (Multi-Scale Ghost Module, MSGhost module) based on the C2f module, namely, a C2fMSG module, is provided in the backbone network and / or the neck network. Among them, MSGhost is a lightweight network structure embedded with multi-scale convolution, which can effectively extract and fuse effective image features of fallen trees of different scales while reducing computational complexity, which can not only greatly improve the recognition speed of fallen trees, but also facilitate the recognition of fallen trees of different scales. By extracting and fusing the image features of fallen trees of each scale through such a backbone network and / or neck network, it is possible to obtain fusion features that fully characterize fallen trees of different scales, and recognition based on the fusion features can improve the recognition efficiency and accuracy of the model for fallen trees of different scales.

[0132] In some embodiments, the C2fMSG module includes at least 2 levels of MSG Bottleneck structures, and the MSGBottleneck structure is obtained by improving the Bottleneck of the C2f module by embedding a lightweight network structure with multi-scale convolutions; for the first feature of the input C2fMSG module:

[0133] Step A1: The electronic device inputs the first feature after the compression channel into the MSGhost Bottleneck structure to obtain a multi-scale deep feature.

[0134] Step A2: The electronic device splices the multi-scale deep feature with the first feature after the compression channel to obtain a first spliced ​​feature.

[0135] Step A3: The electronic device expands the channel of the first splicing feature to obtain a first output feature with the same scale as the first feature.

[0136] The compression channel can be implemented through a set of CBS convolution operations, where CBS refers to the sequential execution of convolution, normalization (Batch Normalization) and activation operations to generate the first feature after the compression channel. Subsequently, the first feature is subjected to multi-level deep feature extraction through the MSGhost Bottleneck structure to further generate multi-scale deep features. The MSGhost Bottleneck structure forms a deep network by stacking MSGhost Bottleneck layers multiple times, which can significantly improve the feature extraction capability; in order to avoid gradient loss, the first feature after the compression channel can be transmitted through a short connection, and it can be fused and spliced ​​with the generated multi-scale deep features in the channel dimension to obtain the first spliced ​​feature. Finally, the channel of the first spliced ​​feature is expanded through another set of CBS convolution operations to generate the first output feature consistent with the scale of the first input feature.

[0137] For example, see Figure 5 , Figure 5 The schematic diagram of the structure of the C2fMSG module is shown. Based on this structure, for the first feature of the input ,in, N is the batch size, C 1 is the number of input channels, H and W are the height and width of the input respectively.

[0138] First, a set of 1x1 CBS convolution operations are performed to convert the input channel C 1 Expanded from 2 C Channel here C= e × C 2 , C 2 is the number of output channels, e is the expansion factor.

[0139]

[0140] Next, output Y Split into two equal parts along the channel dimension.

[0141]

[0142] Then, Y 2 The input is a MSGA Bottleneck structure consisting of n MSGA Bottleneck layers, each of which will output a C Channel mapping back C aisle.

[0143]

[0144] Finally, Y 1 With all Z i The results are concatenated together, and then a set of 1x1 CBS convolution operations are performed to convert the first concatenated feature from (2+n)× C The channel is reduced to C 2 Channel, output first output feature O 1 .

[0145]

[0146] The design of the C2fMSG module aims to solve the challenges of inconsistent scales and insufficient real-time recognition in tree-fallen identification. By embedding multi-scale convolutions in the lightweight network structure Ghost module, the module can efficiently extract and fuse multi-scale features, which not only significantly improves the speed of tree-fallen identification, but also enhances the ability to identify tree-fallen features of different scales, thereby improving the overall performance of identification.

[0147] In some embodiments, the MSGhost Bottleneck structure includes at least 2 sequentially connected MSGhostBottleneck layers; for the second feature of each MSGhost Bottleneck layer:

[0148] Step B1: The electronic device performs two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature.

[0149] Step B2: The electronic device sequentially performs a depth convolution operation and a convolution operation on the second feature to obtain a second convolution feature.

[0150] Step B3: The electronic device concatenates the multi-scale feature with the second convolution feature to obtain a second output feature corresponding to the second feature.

[0151] The MSGhost Bottleneck layer includes two branches. The two MSGhost stacked branches can sequentially perform two multi-scale convolution operations on the second feature to obtain the final multi-scale feature; the branch containing deep convolution and conventional convolution can sequentially perform deep convolution operations and convolution operations on the second feature to obtain the second convolution feature. Finally, the multi-scale feature is concatenated with the second convolution feature to obtain the second output feature corresponding to the second feature.

[0152] It can be understood that the multi-scale deep features are obtained by fusion of the second output features of each MSGhost Bottleneck layer. For example, if there are 20 layers of MSGhost Bottleneck layers, then a total of 20 corresponding second output features can be obtained. After splicing these 20 second output features, the multi-scale deep features are obtained;

[0153] For example, see Figure 6 , Figure 6 The network structure diagram of the MSGhost Bottleneck layer is shown. Based on this structure, for the second feature of the input ,in, N is the batch size, C 1 is the number of input channels, H and W are the height and width of the input, respectively. C2 is the number of output channels, X 2 Enter two MSGhosts in series, and the formula can be written as follows:

[0154]

[0155] in, Z 1 For the first MSGhost X 2 Output after multi-scale convolution; Z 2 For the second MSGhost Z 2 Output after multi-scale convolution.

[0156] Will X 2 Input the depth convolution layer and the convolution layer, the formula can be written as follows:

[0157]

[0158] in, S ( X 2 )for X 2 The output after depth convolution and normal convolution;

[0159] Will Z 2 and S ( X 2 ) to obtain:

[0160]

[0161] O 2 for X 2 The corresponding second output feature.

[0162] In some embodiments, the MSGhost Bottleneck layer includes a multi-scale convolution branch, which is used to perform two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature; the multi-scale convolution branch includes two MSGhosts stacked in sequence; for the third feature of each MSGhost input:

[0163] Step C1: The electronic device performs at least two convolution operations with different convolution kernels on the third feature after the compression channel to obtain corresponding at least two third convolution features.

[0164] Step C2: The electronic device concatenates the third feature after the compression channel with each third convolution feature to obtain a third output feature with the same scale as the third feature.

[0165] The third feature after the compression channel can perform the corresponding number of convolution operations according to the number of divided scales to extract features of different scales; in addition, the third feature after the compression channel also performs identity mapping, and finally merges with features of different scales and expands the channel to obtain the third output feature corresponding to the third feature, and the third feature has the same scale as the third output feature.

[0166] For example, if k=3, 5, and 7 are set to extract multi-scale features respectively, the network structure diagram of MSGhost can be found in Figure 7 Based on this structure, for the third feature of the input ,in, N is the batch size, C 1 is the number of input channels, H and W are the height and width of the input respectively.

[0167] First, a 1x1 standard convolution is used to reduce the number of channels of the input feature from C 1 Reduce to C 2 / 2( ), C 2 is the number of output channels, which can achieve channel dimensionality reduction while retaining important features.

[0168]

[0169] Then Rearrange into multiple groups, each containing / 3 channels, assuming the number of groups is G (Determined by the length of kernels), this example is set to 3.

[0170]

[0171] Convolution operations are performed on each group with different kernel sizes k of 3, 5, and 7, and the results of all groups are stacked and rearranged back to the original shape again.

[0172]

[0173]

[0174] Finally, X3 and Y Splicing is performed on the channel dimension, and then a 1x1 convolution is used to fuse the information and adjust the number of channels to C 2 ,get X 2 The corresponding second output feature O 2 .

[0175]

[0176] The lightweight network structure MSGhost embedded with multi-scale convolution can effectively extract and fuse the effective image features of fallen trees of different scales while reducing the computational complexity. This can not only greatly improve the recognition speed of fallen trees, but also facilitate the recognition of fallen trees of different scales.

[0177] In some embodiments, in order to enhance the tree recognition model's ability to recognize tree fall features of different scales, thereby significantly improving the robustness and accuracy of recognition and meeting recognition requirements in complex scenarios, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; a W-SPPELAN module is provided after the mth feature extraction module, and the W-SPPELAN module is used to fuse multi-scale feature information and dynamically adjust the channel weights of the fused feature information.

[0178] In some embodiments, for the fourth feature of each W-SPPELAN module input:

[0179] Step D1, the electronic device step inputs the fourth feature after the compression channel into the serial 3-level maximum pooling structure to obtain the pooling results corresponding to the maximum pooling structure of each level.

[0180] Step D2: The electronic device concatenates the fourth feature after the compression channel with each pooling result to obtain a second concatenated feature.

[0181] Step D3: The electronic device performs a normalization operation on each channel sub-feature of the second splicing feature to determine a weight value of the sub-feature of each channel.

[0182] Step D4: The electronic device performs weighted splicing on each sub-feature based on each weight value to obtain a third splicing feature.

[0183] Step D5: The electronic device performs channel expansion on the third splicing feature to obtain a fourth output feature having the same scale as the fourth feature.

[0184] The compression channel can be realized through a set of CBS convolution operations. The fourth feature after the compression channel can input 3 serial maximum pooling results, perform multi-level pooling operations, and obtain the pooling results output by each maximum pooling layer; after splicing the fourth feature after the compression channel with each pooling result, a second splicing feature can be obtained; in order to enhance the expression ability of the key features, the electronic device can perform a normalization operation on each channel sub-feature of the second splicing feature, and then determine the weight value corresponding to each channel sub-feature; by weighted splicing of each sub-feature, a third splicing feature that can highlight the key features can be obtained, and finally a set of CBS convolution operations are performed on the third splicing feature again to expand the channel to obtain a fourth output feature corresponding to the fourth feature.

[0185] For example, the network structure diagram of the W-SPPELAN module can be found in Figure 8 Based on this structure, for the fourth feature of the input ,in, N is the batch size, C 1 is the number of input channels, H and W are the height and width of the input, respectively. C 2 is the number of output channels.

[0186] First, a 1x1 standard convolution is used to reduce the number of channels of the input feature from C 1 Reduce to C 2 / 2( ).

[0187]

[0188] Next, three maximum pooling Maxpool2d (two-dimensional maximum pooling) layers of the same size and a pooling kernel of 5 are used in series to perform maximum pooling operations on the reduced dimensionality features.

[0189]

[0190] in, Y 1 , Y 2 and Y 3 They are the outputs corresponding to the three maximum pooling layers respectively; represents maximum pooling with pooling kernel k equal to 5.

[0191] Then, X 4 and Y 1 ,Y 2 and Y 3 Splicing is performed on the channel dimension, and a 3x3 convolutional layer is used to generate a weight map, which has 4 channels, corresponding to the 4 channels of the second splicing feature, that is, corresponding to X 4 , Y 1 , Y 2 and Y 3 , Softmax is applied to normalize the weight map so that the sum of the weights of each channel is 1.

[0192]

[0193]

[0194] Finally, for the four X 4 , Y 1 , Y 2 and Y 3 Perform weighted concatenation and adjust the number of channels through a 1x1 convolution to obtain X 4 The corresponding fourth output feature O 4 .

[0195]

[0196] in, Represents a convolution operation with a kernel size of k=1.

[0197] The introduction of the W-SPPELAN module aims to address the challenges of complex background environments and unclear identification features in the process of tree fall identification. By fusing receptive fields of different sizes, the module can fully extract and integrate contextual semantic information to achieve accurate identification of tree fall features, thereby significantly improving the accuracy and robustness of identification.

[0198] In some embodiments, the hardware environment in which the tree recognition model is run is: the processor is an Intel(R) Xeon(R) Platinum 8255C CPU, 314 G memory, the graphics card is an NVIDIA Tesla V100 32 G, CentOS 8.5.2 64-bit operating system, and the deep neural network is built on the PyTorch deep learning framework. The input image size of the network is [640,640], and a multi-scale training mode is adopted. The data input batch is 64, the training epoch is 300, and the AdamW optimizer with an initial learning rate of 0.01 and a cosine decay strategy are used.

[0199] It has been verified that the recognition method of the present application realizes automatic recognition of fallen trees through a trained tree recognition model. On the one hand, a C2fMSG module is embedded in the backbone network and / or neck network of the tree recognition model. The C2fMSG module introduces multi-scale convolution blocks in the lightweight network structure Ghost to achieve efficient multi-scale feature extraction and fusion, which not only significantly improves the real-time performance of tree fall detection, but also enhances the recognition ability of tree fall features of different scales. The MSG Bottleneck layer built on MSGhost further optimizes the feature expression capability through a structure composed of nested multi-level MSG Bottleneck layers, so that the C2fMSG module can maintain efficient and accurate tree fall detection performance in complex scenes and meet the detection needs of diverse scenes.

[0200] On the other hand, setting the W-SPPELAN module after the last scale of the backbone network can help the model better integrate information from different scales and dynamically adjust the channel weights of different scales through self-learning, which helps to enhance the model's ability to capture effective features in the tree identification process, thereby better coping with the complex background environment and unclear identification features faced in the identification of fallen trees, and improving the identification accuracy of fallen trees.

[0201] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0202] Corresponding to the tree falling identification method based on deep learning in the above embodiment, Fig. 9 A structural block diagram of a deep learning tree fall identification device 9 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0203] Reference Fig. 9 , the deep learning tree fall identification device 9 includes:

[0204] An extraction module 91 is used to extract features of an image to be identified based on a backbone network of a pre-trained tree identification model to obtain image features; the image to be identified includes roadside trees;

[0205] A fusion module 92, used for fusing image features based on the neck network of the tree recognition model to obtain fusion features;

[0206] The recognition module 93 is used to recognize the fused features based on the recognition network of the tree recognition model to obtain the recognition result of the fallen tree in the image to be recognized;

[0207] The backbone network and / or the neck network is provided with a C2fMSG module, which is obtained by improving the C2f module by embedding a lightweight network structure with multi-scale convolutions. The C2fMSG module is used to extract and fuse multi-scale deep features.

[0208] Optionally, the C2fMSG module includes at least 2 levels of MSG Bottleneck structures, where the MSG Bottleneck structure is obtained by improving the Bottleneck of the C2f module by embedding a lightweight network structure with multi-scale convolutions; the extraction module 91 and / or the fusion module 92 includes a first extraction and fusion module for:

[0209] First feature for input C2fMSG module:

[0210] The first feature after the compression channel is input into the MSG Bottleneck structure to obtain multi-scale deep features;

[0211] The multi-scale deep feature is concatenated with the first feature after the channel compression to obtain a first concatenated feature;

[0212] The channels of the first concatenated feature are expanded to obtain a first output feature having the same scale as the first feature.

[0213] Optionally, the MSG Bottleneck structure includes at least two sequentially connected MSG Bottleneck layers; and for the second feature input into each MSG Bottleneck layer, the first extraction and fusion module is specifically used for:

[0214] Perform two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature;

[0215] Performing a depthwise convolution operation and a convolution operation on the second feature in sequence to obtain a second convolution feature;

[0216] The multi-scale feature is concatenated with the second convolutional feature to obtain the second output feature corresponding to the second feature. The multi-scale deep feature is obtained based on the fusion of the second output features of each MSG Bottleneck layer.

[0217] Optionally, the MSG Bottleneck layer includes a multi-scale convolution branch, which is used to perform two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature; the multi-scale convolution branch includes two multi-scale convolution blocks stacked in sequence; for the third feature input into each multi-scale convolution block, the first extraction and fusion module is specifically used to:

[0218] Perform at least two convolution operations with different convolution kernels on the third feature after the channel compression to obtain at least two corresponding third convolution features;

[0219] The third feature after the compression channel is concatenated with each third convolution feature to obtain a third output feature with the same scale as the third feature.

[0220] Optionally, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; a W-SPPELAN module is provided after the mth feature extraction module, and the W-SPPELAN module is used to fuse multi-scale feature information and dynamically adjust the channel weights of the fused feature information.

[0221] Optionally, for the fourth feature input into each W-SPPELAN module, the extraction module 91 is specifically used for:

[0222] Input the fourth feature after the compression channel into the serial 3-level maximum pooling structure to obtain the pooling results corresponding to the maximum pooling structure of each level;

[0223] The fourth feature after the compression channel is concatenated with each pooling result to obtain a second concatenated feature;

[0224] Performing a normalization operation on each channel sub-feature of the second concatenated feature to determine a weight value of the sub-feature of each channel;

[0225] Perform weighted splicing on each sub-feature based on each weight value to obtain a third splicing feature;

[0226] Channel expansion is performed on the third concatenated feature to obtain a fourth output feature having the same scale as the fourth feature.

[0227] Optionally, the tree recognition model is trained based on regression loss and classification loss; the regression loss is:

[0228]

[0229]

[0230] in, is the regression loss; is the extended intersection-over-union loss; IoU is the intersection-over-union ratio of the predicted bounding box and the true bounding box, To control the degree of outlier suppression, is the intersection-and-join loss, is the center distance loss; is the width and height loss; b and Represent the center point of the real box and the center point of the predicted box respectively; Respectively represent the Euclidean distance between the center point of the real box and the predicted box; and Respectively represent the width and height of the minimum bounding rectangle.

[0231] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0232] Fig.10 This is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: at least one processor 100 ( Fig.10 Only one processor is shown in the figure), a memory 101, and a computer program 102 stored in the memory 101 and executable on at least one processor 100. When the processor 100 executes the computer program 102, the steps of any of the above-mentioned deep learning tree falling recognition method embodiments are implemented, such as Figure 4 Steps 410-430 are shown.

[0233] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0234] In some embodiments, the memory 101 may be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10. In other embodiments, the memory 101 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 10.

[0235] Furthermore, the memory 101 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 101 is used to store operating devices, applications, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0236] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0237] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0238] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0239] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0240] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0241] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0242] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0243] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0244] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A deep learning method for identifying fallen trees, characterized in that: include: The backbone network based on the pre-trained tree recognition model extracts features from the image to be recognized to obtain image features; The image to be identified includes roadside trees; fusing the image features based on the neck network of the tree recognition model to obtain fused features; The recognition network based on the tree recognition model recognizes the fusion features to obtain a recognition result of fallen trees in the image to be recognized; Wherein, the backbone network and / or the neck network is provided with a C2fMSG module, the C2fMSG module is obtained by improving the C2f module by embedding a lightweight network structure with multi-scale convolution, and the C2fMSG module is used to extract and fuse multi-scale deep features; The C2fMSG module includes at least 2 levels of MSG Bottleneck structures, and the MSG Bottleneck structure is obtained by improving the Bottleneck of the C2f module through the lightweight network structure embedded with multi-scale convolution; for the first feature of the input to the C2fMSG module: Inputting the first feature after the channel compression into the MSG Bottleneck structure to obtain a multi-scale deep feature; Splicing the multi-scale deep feature with the first feature after the channel compression to obtain a first spliced ​​feature; Expanding the channel of the first concatenated feature to obtain a first output feature having the same scale as the first feature; The MSG Bottleneck structure includes at least 2 MSG Bottleneck layers connected in sequence; for the second feature of each MSG Bottleneck layer input: Performing two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature; Performing a depthwise convolution operation and a convolution operation on the second feature in sequence to obtain a second convolution feature; The multi-scale feature is concatenated with the second convolutional feature to obtain a second output feature corresponding to the second feature, and the multi-scale deep feature is obtained by fusing the second output features of each MSG Bottleneck layer.

2. The method for identifying fallen trees according to claim 1, characterized in that: The MSG Bottleneck layer includes a multi-scale convolution branch, which is used to perform two multi-scale convolution operations on the second feature in sequence to obtain a multi-scale feature; the multi-scale convolution branch includes two MSGhosts stacked in sequence; for the third feature of each of the MSGhosts input: Perform at least two convolution operations with different convolution kernels on the third feature after the compression channel to obtain at least two corresponding third convolution features; The third feature after the compression channel is concatenated with each of the third convolutional features to obtain a third output feature with the same scale as the third feature.

3. The method for identifying fallen trees according to claim 1 or 2, characterized in that: The backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; a W-SPPELAN module is arranged after the mth feature extraction module, and the W-SPPELAN module is used to fuse multi-scale feature information and dynamically adjust the channel weight of the fused feature information.

4. The method for identifying fallen trees according to claim 3, characterized in that: For the fourth feature of each of the W-SPPELAN modules: Input the fourth feature after the compression channel into the serial 3-level maximum pooling structure to obtain the pooling results corresponding to the maximum pooling structure of each level; Splicing the fourth feature after the channel compression with each of the pooling results to obtain a second splicing feature; Performing a normalization operation on each channel sub-feature of the second splicing feature to determine a weight value of the sub-feature of each channel; Perform weighted splicing on each of the sub-features based on each of the weight values ​​to obtain a third splicing feature; Channel expansion is performed on the third concatenated feature to obtain a fourth output feature having the same scale as the fourth feature.

5. The method for identifying fallen trees according to claim 1 or 2, characterized in that: The tree recognition model is trained based on regression loss and classification loss; the regression loss is: Among them, the is the regression loss; is the extended intersection-over-union loss; the IoU is the intersection-over-union ratio of the predicted border and the real border, To control the degree of outlier suppression, is the intersection-and-joint loss, is the center distance loss; is the width and height loss; b and stated Respectively represent the center point of the real box and the center point of the predicted box; Respectively represent the Euclidean distance between the center point of the real box and the predicted box; w and are the widths of the predicted border and the true border respectively; Represents the Euclidean distance between the width of the predicted bounding box and the true bounding box; h and are the heights of the predicted border and the true border respectively; Represents the Euclidean distance between the height of the predicted border and the true border; and stated Respectively represent the width and height of the minimum bounding rectangle.

6. A tree falling identification device, characterized in that: include: An extraction module is used to extract features of an image to be identified based on a backbone network of a pre-trained tree identification model to obtain image features; the image to be identified includes roadside trees; A fusion module, used for fusing the image features based on the neck network of the tree recognition model to obtain fusion features; A recognition module, used to recognize the fused features based on the recognition network of the tree recognition model, and obtain a recognition result of fallen trees in the image to be recognized; Wherein, the backbone network and / or the neck network is provided with a C2fMSG module, the C2fMSG module is obtained by improving the C2f module by embedding a lightweight network structure with multi-scale convolution, and the C2fMSG module is used to extract and fuse multi-scale deep features; The C2fMSG module includes at least two levels of MSG Bottleneck structures, and the MSG Bottleneck structure is obtained by improving the Bottleneck of the C2f module through the lightweight network structure embedded with multi-scale convolution; the extraction module or the fusion module includes a first extraction fusion module, and the first extraction fusion module is used to: Regarding the first feature of the C2fMSG module: Inputting the first feature after the channel compression into the MSG Bottleneck structure to obtain a multi-scale deep feature; Splicing the multi-scale deep feature with the first feature after the channel compression to obtain a first spliced ​​feature; Expanding the channel of the first concatenated feature to obtain a first output feature having the same scale as the first feature; The MSG Bottleneck structure includes at least two sequentially connected MSG Bottleneck layers; for the second feature of each MSG Bottleneck layer, the first extraction and fusion module is specifically used to: Performing a depthwise convolution operation and a convolution operation on the second feature in sequence to obtain a second convolution feature; The multi-scale feature is concatenated with the second convolutional feature to obtain the second output feature corresponding to the second feature. The multi-scale deep feature is obtained based on the fusion of the second output features of each MSG Bottleneck layer.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the deep learning method for identifying fallen trees according to any one of claims 1 to 5 is implemented.

8. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the computer program is executed by a processor, the deep learning method for identifying fallen trees according to any one of claims 1 to 5 is implemented.

Images