Abnormal urban facility detection method, device and electronic equipment based on YOLO

Abstract

The present application discloses a method for detecting abnormal urban facilities based on YOLO, an abnormal urban facilities detection device based on YOLO, an electronic device and a computer-readable storage medium. The detection method is implemented based on a trained facility detection model. The facility detection model sets a C2fMRFF block in the backbone network and / or the neck network. The C2fMRFF block is obtained by replacing the Bottleneck of the C2f block with an MRFF structure composed of deep convolutions with different void rates. It can fuse multi-level spatial information and effectively balance the acquisition of local details and global background information to improve the model's detection ability for abnormal urban facilities of different scales in complex scenes. The neck network performs multi-scale feature splicing through two MSFC blocks to integrate spatial information at different levels, fuse local detail information and global semantic information, and thereby increase the model's ability to understand image details and context. In addition, the loss function is improved to fully consider the direction between the real border and the predicted border to improve the convergence speed of the model.

Description

Technical Field

[0001] The present application belongs to the field of target detection technology, and in particular, relates to a YOLO-based abnormal urban facility detection method, a YOLO-based abnormal urban facility detection device, an electronic device, and a computer-readable storage medium. Background Art

[0002] As the global urbanization process continues to accelerate, the scale and complexity of urban infrastructure are growing, which requires urban management to have a higher level of intelligence to ensure the efficient operation and sustainable development of the city. Various facility anomalies are inevitable in the daily operation of the city, such as damage to various public safety facilities. If the urban facility anomalies caused by these events cannot be quickly detected and effectively responded to, it will not only reduce the efficiency of urban management, but may also directly threaten the safety and quality of life of citizens.

[0003] Abnormal urban facility detection often faces the challenge of diverse target scales, which increases the difficulty of detecting abnormal urban facilities and makes the model's accuracy in detecting abnormal urban facilities at different scales low. Summary of the invention

[0004] The present application provides an abnormal urban facility detection method, an abnormal urban facility detection device, an electronic device and a computer-readable storage medium, which can integrate local features and global features at different levels, improve the feature expression ability of the target, and thus improve the accuracy of abnormal urban facility detection at different scales.

[0005] In a first aspect, the present application provides a method for detecting abnormal urban facilities based on YOLO, comprising:

[0006] The backbone network of the pre-trained facility detection model extracts features from the image to be detected to obtain image features; the image to be detected includes urban facilities;

[0007] The neck network based on the facility detection model fuses the image features to obtain fused features;

[0008] The detection network based on the facility detection model detects the fused features and obtains the detection results of abnormal urban facilities in the image to be detected;

[0009] The backbone network and / or the neck network is provided with a C2fMRFF block, and the C2fMRFF block is used to fuse local features and global features at different levels.

[0010] In the second aspect, the present application provides a device for detecting abnormal urban facilities based on YOLO, comprising:

[0011] An extraction module is used to extract features from an image to be detected based on a backbone network of a pre-trained facility detection model to obtain image features; the image to be detected includes urban facilities;

[0012] A fusion module, used for fusing image features based on the neck network of the facility detection model to obtain fusion features;

[0013] A detection module is used to detect the fused features based on the detection network of the facility detection model to obtain the detection results of abnormal urban facilities in the image to be detected;

[0014] The backbone network and / or the neck network is provided with a C2fMRFF block, and the C2fMRFF block is used to fuse local features and global features at different levels.

[0015] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0017] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0018] Compared with the prior art, the present application has the following beneficial effects: a multi-receptive field fusion module (MRFF) based on a C2f block, i.e., a C2fMRFF block, is set in the backbone network and / or the neck network, which effectively balances the acquisition of local details and global background information while fusing multi-level spatial information, so as to enhance the feature expression capability of abnormal urban facilities of different scales, strengthen the distinction between abnormal urban facilities of different scales and the background, and thus improve the model's detection capability of abnormal urban facilities of different scales in complex scenes. Through such a backbone network and / or the neck network, image features containing more contextual information can be extracted, and then fusion features obtained by image fusion based on image features, the fusion features contain more local semantic information and global background information of abnormal urban facilities of different scales, and thus, by detecting the fusion features, the accuracy of abnormal urban facilities detection of different scales can be improved.

[0019] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is a schematic diagram of classification examples of 11 types of abnormal urban facilities provided in an embodiment of the present application;

[0022] Figure 2 It is a quantitative statistical histogram of 11 types of abnormal urban facilities provided in the embodiment of the present application;

[0023] Figure 3 is a schematic diagram of angle loss provided in an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of distance loss provided in an embodiment of the present application;

[0025] Figure 5 is a schematic flow chart of a method for detecting abnormal urban facilities based on YOLO provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of the network structure of the C2fMRFF block provided in an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of the network structure of the MRFF structure provided in the embodiment of the present application;

[0028] Figure 8 Schematic diagram of the network structure of the facility detection model based on YOLOv8 provided in the embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of the network structure of the MSFC block provided in an embodiment of the present application;

[0030] Fig.10 It is a data comparison diagram of key evaluation indicators of the facility detection model and the YOLO v8s model provided in the embodiment of the present application;

[0031] Fig.11 is a schematic diagram of the structure of an abnormal urban facility detection device based on YOLO provided in an embodiment of the present application;

[0032] Fig.12It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0034] Urban facilities may include one or more of roads, manhole covers, guardrails, walls, telescopic devices, drainage facilities, isolation fences, crash barriers, and vehicle stoppers.

[0035] In order to enable the facility detection model to more accurately understand and detect abnormal urban facilities, this application creates a professional dataset called Abnormal City Facilities (ACF), which focuses on collecting and analyzing abnormal urban facility instances. The uniqueness of this dataset is that all image data comes from on-site collection in a real vehicle environment, which ensures the authenticity and diversity of the data.

[0036] It can be understood that the more images there are in the dataset, the better the performance of the trained facility detection model. Assuming that the entire dataset covers 40,000 high-quality images, in order to ensure the effectiveness and generalization ability of the algorithm model, the dataset is divided into two parts according to the classic 8:2 ratio: a training set containing 32,000 images for training and optimizing the facility detection model; a validation set containing 8,000 images for verifying the detection ability, accuracy and generalization of the facility detection model, and ensuring the performance of the facility detection model on unknown data.

[0037] Optionally, in order to provide a higher quality dataset, in addition to the number of images, the images can also cover as many abnormal urban facilities commonly found in urban environments as possible, so as to comprehensively cover and accurately focus on multiple challenges in urban management.

[0038] For example, this application roughly divides abnormal urban facilities into 11 categories, named from the first to the eleventh category, and the corresponding diagrams of each category can be found in Figure 1 Of course, different classifications can be adjusted according to actual needs, and are not limited in this application.

[0039] Furthermore, in order to clearly show the prevalence and distribution characteristics of various types of abnormal urban facilities, the number of images in each category can be counted separately. Based on the above classification example, in order to intuitively display the statistical results, refer to Figure 2, the statistical results can be presented in the form of charts. This chart can intuitively reflect the number of different abnormal urban equipment, and then provide data support for subsequent priority sorting, resource allocation and even the formulation of prevention strategies. It is an indispensable part of studying the optimization of urban facility management. Through data visualization, we can more systematically understand which problems are high-frequency pain points in urban management, so as to effectively optimize and deploy corresponding resources and countermeasures for high-frequency pain points.

[0040] It is not difficult to see from the two figures that within the same category, such as the first and eighth categories, the scales are large and small; while between different categories, such as the third and fifth categories, the former has a smaller scale and the latter has a larger scale. Obviously, for the detection of abnormal urban facilities, the multi-scale characteristics of the detection target are particularly prominent, which significantly increases the difficulty of detecting abnormal urban facilities.

[0041] In order to solve this problem, this application proposes a facility detection model, which includes a backbone network, a neck network and a detection head. Specifically, the backbone network is used to extract features from the input image to obtain image features; the neck network is used to fuse the image features to obtain fused features; the detection head is used to detect the fused features to obtain the detection results of abnormal urban facilities in the image to be processed.

[0042] Among them, in order to improve the accuracy of target detection at different scales, the present application may set a C2fMRFF block in the backbone network and / or the neck network. While fusing multi-level spatial information, the C2fMRFF block can effectively balance the acquisition of local details and global background information to enhance the feature expression capability of abnormal urban facilities at different scales, strengthen the distinction between abnormal urban facilities at different scales and the background, thereby improving the model's detection capability of abnormal urban facilities at different scales in complex scenes. Through such a backbone network and / or neck network, image features containing more contextual information can be extracted, and then fusion features obtained by image fusion based on image features, the fusion features contain more local semantic information and global background information of abnormal urban facilities at different scales, and thus, by detecting the fusion features, the accuracy of detection of abnormal urban facilities at different scales can be improved.

[0043] In some embodiments, the C2fMRFF block is obtained by replacing the Bottleneck structure in the C2f module with the MRFF structure, and the MRFF structure is used to fuse local features and global features at each level.

[0044] The C2f block can promote the flow and reuse of features by establishing connections between different network layers, which not only increases the network depth, but also improves the diversity of features, enabling the model to more comprehensively capture the multi-level information of the input image. In addition, the C2f block can enhance the nonlinear expression ability of the network, effectively alleviate the gradient vanishing problem during training, and further improve the training efficiency and performance of the model.

[0045] Therefore, in order to make full use of the advantages of the C2f block and effectively balance the acquisition of local details and global background information, so as to enhance the feature expression ability of abnormal urban facilities of different scales and strengthen the distinction between abnormal urban facilities of different scales and the background, the C2f block can be combined with the MRFF structure to obtain the C2fMRFF block.

[0046] It is assumed that the facility detection model is improved based on the model of the YOLO series. For example, for YOLOv5s, the C3 block of the backbone network and / or the neck network of YOLOv5s will be replaced with the C2fMRFF block; for YOLOv8s, the C2fMRFF block will be replaced with the C2f block of the backbone network and / or the neck network of YOLOv5s in YOLOv8s.

[0047] In some embodiments, the MRFF structure includes a sequentially connected convolution layer for compressing channels, a multi-receptive field convolution layer, and a convolution layer for expanding channels, wherein the multi-receptive field convolution layer includes at least two parallel depthwise convolution (DWC) layers with different hole rates and an identity mapping (Identity) layer.

[0048] In an embodiment of the present application, on the one hand, the C2fMRFF block can combine contextual information of different scales and can provide richer feature representation, which helps to solve the problem of missing local semantic information of small targets. At the same time, it effectively balances the capture of local details and non-local background information through dilated convolutions with different voiding rates, thereby improving the model's ability to understand complex scenes. On the other hand, the C2fMRFF block can extract and fuse multiple levels of spatial information to form a more comprehensive and in-depth feature map. This module uses parallel deep convolutions with multiple different voiding rates to absorb richer contextual information while maintaining local features. This balance ensures that the model has strong insight when parsing complex scenes, and has good robustness to scale changes in target objects. Therefore, the C2fMRFF block can well solve the problems of small target detection and target scale diversity in abnormal urban facility detection, thereby greatly improving the robustness of the model to abnormal urban facility detection with multi-scale characteristics.

[0049] In related technologies, abnormal urban facility detection not only has the problem of diverse target scales, but also has the problem of semantic understanding in scene understanding. For some abnormal urban facilities, there are complexity of uncertain context dependence of classification boundaries and ambiguity in semantic understanding, making it difficult for the model to accurately classify them into a specific category. For example Figure 1 and Figure 2 The first, fifth and tenth categories.

[0050] In order to solve this problem, the present application also sets two multi-scale feature concat (MSFC) blocks in the neck network, namely the first MSFC block and the second MSFC block. When the base model of the facility detection model is the YOLO series model, the backbone network includes sequentially connected preset convolutional layers, 4 stage layers, each stage layer includes a feature extraction module composed of a preset convolutional layer and a C2fMRFF block, and the last stage layer is connected with an SPPF module after the feature extraction module.

[0051] Accordingly, the first MSFC block is used to fuse the extracted features of the 2nd to 4th stage layers after convolution; the second MSFC block is used to fuse the extracted features of the first two stage layers after convolution, as well as the fused features of the first MSFC block after preliminary fusion and convolution; the preliminary fusion is achieved through the C2fMRFF block.

[0052] In some embodiments, the MSFC block includes a parallel downsampling layer, a transfer layer, an upsampling layer, and a splicing layer. The downsampling layer is used to perform a downsampling operation on the highest resolution feature to reduce the feature size; the transfer layer is used to transfer the middle resolution feature, and the upsampling layer is used to perform an upsampling operation on the minimum resolution feature to expand the feature size. Specifically, the downsampling layer can be composed of a maximum pooling layer and an average pooling layer. In some embodiments, the loss function of the above-mentioned facility detection model mainly consists of two parts: regression loss and classification loss. Among them, the classification loss is a binary cross entropy loss BCE Loss, which is used to determine the specific category in the anchor frame. The regression loss is a border regression loss (Skew Intersection over Union Loss, L SIoU ) function and dynamic category loss (Distribution Focal Loss, L DFL ) function is used to calculate the error between the predicted bounding box and the ground truth box.

[0053] Specifically, the formula of BCE Loss is as follows:

[0054]

[0055] in, L BCE is the binary cross entropy loss, n is the number of image samples of all abnormal urban facilities, y i For the i The true category of the image samples, is the predicted category of the i-th image sample.

[0056] The regression loss is used to calculate the error between the predicted border and the true border. In related technologies, the regression loss uses the CIOU loss (Complete Intersection over Union loss, L SIoU ), which is an improved version of commonly used loss functions such as L1, L2, Intersection over Union (IoU) and GIoU. L CIoU By adding a penalty term for the aspect ratio, when the center points of the predicted bounding box and the true bounding box coincide, the errors in different situations can be better distinguished and scale invariant.

[0057]

[0058] in, b and b gt They are the center points of the predicted bounding box and the true bounding box respectively; ρ is the Euclidean distance between the predicted bounding box and the true bounding box; c The distance between the diagonal of the enclosed area of ​​the predicted bounding box and the true bounding box; v To ensure the consistency of the relative proportions between the predicted bounding box and the true bounding box, IoU It is the intersection-over-union ratio of the predicted bounding box and the true bounding box; α is the weight coefficient; w and w gt are the width of the predicted border and the true border respectively; h and h gt are the heights of the predicted bounding box and the true bounding box respectively.

[0059] However, L CIoU The direction between the real box and the predicted box is not taken into account, which often causes the network to "wander around" during training, converges slowly and has low efficiency, which easily reduces the robustness of the model.

[0060] To solve this problem, this application considers the vector angle between the real border and the predicted border and redefines the penalty index, which includes four parts: angle cost, distance cost, shape cost, and IoU cost.

[0061] 1. Angle loss

[0062] See also Figure 3 , the formula for angle loss Λ can be written as follows:

[0063]

[0064] in, is the center point coordinate of the predicted bounding box, is the coordinate of the center point of the real border, σ is the distance between the center points of the real border and the predicted border, d h is the height difference between the center point of the real box and the predicted box, d w is the height difference between the center point of the real bounding box and the predicted bounding box.

[0065] 2. Distance loss

[0066] See also Figure 4 , the formula for distance loss Δ can be written as follows:

[0067]

[0068] c h and c w They are the width and height of the minimum enclosing rectangle of the real border and the predicted border, respectively.

[0069] 3. Shape loss

[0070] The shape loss Ω can be written as follows:

[0071]

[0072] w , h , w gt and h gt are the width and height of the predicted border and the true border respectively, and θ is a hyperparameter that can control the degree of attention paid to the shape loss.

[0073] L SIoUCombining the above three aspects of loss can improve the comprehensiveness of each part of the loss, making the convergence speed faster and the accuracy higher, which is better than L CioU . L SIoU It can be written as follows:

[0074]

[0075] The formula for DFL loss is as follows:

[0076]

[0077] in, S i is the cross entropy loss between the real border and the predicted border on the left, S i+1 It is the cross entropy loss between the real box and the predicted box on the right.

[0078] Total loss function of the facility detection model Loss The formula is as follows:

[0079]

[0080] Among them, λ1 and λ1 are balance coefficients.

[0081] Training the facility detection model described in any of the above embodiments based on the total loss function of the facility detection model can improve the model convergence speed, thereby improving the efficiency of model training. By continuously iterating the facility detection model so that the facility detection model has a better and more stable detection effect on each abnormal urban facility, a robust version can be selected from multiple converged versions as the trained facility detection model.

[0082] In some embodiments, in order to comprehensively and accurately measure the performance of each version of the facility detection device, it can be evaluated through a validation set. Specifically, the performance of the facility detection device on the validation set can be evaluated according to preset conditions. For example, these preset conditions may include performance indicators such as the intersection-over-union ratio, detection accuracy, and recall rate of the facility detection device for road defects.

[0083] IoU is an indicator to evaluate the degree of overlap between the predicted border and the true border. It calculates the ratio of the intersection and union of the predicted border and the true border. It plays a key role in determining whether it is a correct detection. The calculation formula of IoU can be written as:

[0084]

[0085] Precision (p), also known as the precision rate, refers to the proportion of correct positive predictions to all positive predictions, as shown in the formula:

[0086]

[0087] Recall (R), also known as recall rate, refers to the proportion of correctly predicted positives to all actual positives, as shown in the formula:

[0088]

[0089] Average Precision (AP) is calculated from precision and recall. A line graph of precision is drawn based on the recall value, and the area under the line is calculated, as shown in the formula:

[0090]

[0091] Mean Average Precision (mAP) refers to the average value of the average precision AP of C different road disease categories, as shown in the formula:

[0092]

[0093] That is to say, after verifying each version of the facility detection model through the verification set, the facility detection model of each version can be comprehensively evaluated based on the above-mentioned indicators, so as to determine the facility detection model with the best performance from each version as the trained facility detection model.

[0094] Based on the facility detection model of the previous embodiment, this application proposes a YOLO-based abnormal urban facility detection method.

[0095] The abnormal urban facility detection method based on YOLO provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc., and the electronic device is equipped with a trained facility detection module. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0096] In order to illustrate the technical solution proposed in this application, each embodiment will be described below with an electronic device as the execution subject.

[0097] Figure 5 A schematic flow chart of the abnormal urban facility detection method based on YOLO provided in the present application is shown, and the detection method includes:

[0098] Step 510: The electronic device extracts features of the image to be detected based on the backbone network to obtain image features.

[0099] The image to be detected includes urban facilities.

[0100] Step 520: The electronic device performs a feature fusion operation on the image features based on the neck network to obtain fused features.

[0101] Step 530: The electronic device detects the fused features based on the detection head to obtain the detection result of the abnormal urban facilities in the image to be processed.

[0102] In this embodiment, the improved backbone network benefits from the introduction of the C2fMRFF block, and the electronic device can use the C2fMRFF block to fuse multi-level spatial information and effectively balance the acquisition of local details and global background information to enhance the feature expression ability of abnormal urban facilities of different scales, strengthen the distinction between abnormal urban facilities of different scales and the background, thereby improving the model's detection ability of abnormal urban facilities of different scales in complex scenes. Through such a backbone network and / or neck network, image features containing more contextual information can be extracted, and then fusion features obtained by image fusion based on image features, the fusion features contain more local semantic information of abnormal urban facilities of different scales and global background information, thereby, by detecting the fusion features, the accuracy of abnormal urban facilities detection of different scales can be improved.

[0103] In some embodiments, see Figure 6 , Figure 6 A schematic diagram of the network structure of the C2fMRFF block is shown. For the second input feature of each C2fMRFF block, the electronic device may perform the following operations:

[0104] Step A1: The electronic device performs channel expansion on the second input feature to obtain an enhanced feature.

[0105] Step A2: The electronic device inputs the enhanced features into the sequentially connected multi-level MRFF structure to obtain the deep features corresponding to each level of the MRFF structure.

[0106] Step A3: The electronic device performs channel compression after splicing the enhanced features and the deep features to obtain a fusion feature corresponding to the second input feature.

[0107] The electronic device can perform a CBS (S=1, K=1) operation on the second input feature to expand the channel to enhance the second input feature and obtain an enhanced feature. The enhanced feature will be input into the first layer MRFF structure and the splicing layer respectively. After the first layer MRFF structure processes the enhanced feature, the first deep feature can be obtained, and the first deep feature is input into the second layer MRFF structure and the splicing layer respectively, and so on, until the n-1 deep feature is processed by the n-th layer MRFF structure to obtain the n-th deep feature, and the n-th deep feature is input into the n-th layer MRFF structure and the splicing layer respectively. Thus, the splicing layer can splice the enhanced feature and the n deep features, and restore the number of channels corresponding to the second input feature by performing a CBS (S=1, K=1) operation to obtain the fusion feature corresponding to the second input feature.

[0108] In some embodiments, for the third input feature of each MRFF structure:

[0109] Step B1: The electronic device performs a deep convolution operation on the third input feature after channel compression based on at least two dilation rates to obtain at least two deep convolution results.

[0110] Step B2: After concatenating at least two deep convolution results with the third input feature, perform channel expansion to obtain a deep feature corresponding to the third input feature.

[0111] For example, the depthwise convolutional layers with dilation rates of 1, 3, and 5 can be set. Figure 7 , Figure 7 A schematic diagram of the network structure of the MRFF structure is shown.

[0112] based on Figure 7 The MRFF structure of the proposed method is shown in Figure 2. The third input feature first passes through a 1×1 convolution layer for preliminary channel dimension reduction and feature compression. Then, the compressed features are sent to a set of parallel deep convolutions with different hole rates for processing. Among them, different hole rates can form receptive fields of different sizes to capture contextual information of different scales. At the same time, the compressed features can also perform identity operations, that is, directly pass feature information to retain the details of the original features. The outputs of each deep convolution and identity are directly spliced ​​together to form a multi-scale feature containing different levels of information from fine to coarse. Finally, the spliced ​​features are again passed through a 1×1 convolution layer for feature fusion, which helps the model learn how to integrate features from different scales to generate more powerful representations. Such a module design effectively combines local and global information and enhances the robustness of the model to scale changes of target objects.

[0113] For example, assuming that the second input feature X ∈ R H×W×C, then the processing process of the MRFF structure can be described as:

[0114]

[0115] in, Indicates that the convolution kernel size is 1x1 and the output channel is C / 2 convolution is used to integrate features and compress channels, corresponding to Y 1.

[0116] represents the depth convolution with the hole rates of 1, 3 and 5 respectively. The convolution kernel size of the three depth convolutions is 3×3, and their output channels are C / 2, through these three depth convolution pairs Y 1 After processing, F 1. F 2 and F 3.

[0117] Y 2 by Y 1. F 1. F 2 and F 3 are concatenated in the channel dimension. O is the output of the MRFF structure, with a convolution kernel size of 1 and an output channel of C The convolution integrates the features and expands the channels to get O.

[0118] In some embodiments, see Figure 8 , Figure 8 The YOLOv8-based facility detection model is shown. The backbone network includes CBS2, 4 stage layers connected sequentially, each stage layer includes a feature extraction module composed of CBS2 and C2fMRFF blocks, and the last stage layer is connected with an efficient Spatial Pyramid Pooling Fast (SPPF) module after the feature extraction module.

[0119] Among them, CBS2 is a set of operations including convolution (Convolution, Conv) with a stride of 2, batch normalization (BatchNormalization, BN), and SiLU activation.

[0120] The neck network is provided with a first MSFC block and a second MSFC block with the same structure; the first MSFC block is used to fuse the extracted features of the 2nd to 4th stage layers after convolution; the second MSFC block is used to fuse the extracted features of the first two stage layers after convolution, as well as the fused features of the first MSFC block after preliminary fusion and convolution; the preliminary fusion is achieved through the C2fMRFF block.

[0121] In some embodiments, the fused features include first-scale fused features, second-scale fused features, and third-scale fused features; step 520 may include:

[0122] Step C1: The electronic device performs a preliminary fusion operation on the fusion features of the second MSFC block based on the C2fMRFF block to obtain a first-scale fusion feature.

[0123] Step C2: The electronic device performs a preliminary fusion operation on the first splicing feature based on the C2fMRFF block to obtain a second-scale fusion feature.

[0124] Step C3: The electronic device performs a preliminary fusion operation on the second splicing feature based on the C2fMRFF block to obtain a third-scale fusion feature.

[0125] In this embodiment, fusion features at three scales are obtained through different fusion operations. Detection is performed through the fusion features of these three scales, which can improve the accuracy of abnormal urban facility detection at each scale.

[0126] In some embodiments, the corresponding detection head is a decoupled detection head, that is, a detection head is set for each scale.

[0127] In some embodiments, it can be understood that the first MSFC block and the second MSFC block have the same structure, but are set at different positions in the neck network, so the specific data processed are different. If the features input to the first MSFC block and the second MSFC block are collectively referred to as 3 first input features, then for the 3 first input features of each MSFC block:

[0128] Step D1: The electronic device performs maximum pooling and average pooling operations on the first input feature corresponding to the highest resolution and then sums the results to obtain a first basic feature.

[0129] Step D2: The electronic device performs an upsampling operation on the first input feature corresponding to the lowest resolution to obtain a second basic feature, wherein the size of the second basic feature is twice that of the first input feature corresponding to the lowest resolution.

[0130] Step D3: The electronic device concatenates the first basic feature, the first input feature corresponding to the center resolution, and the second basic feature to obtain fused features corresponding to the three first input features.

[0131] In order to maintain the size compatibility of features of different scales when splicing, on the one hand, the features of the maximum resolution are downsampled through the maximum pooling layer and the average pooling layer respectively, thereby reducing the size of the features to make them consistent with the size of the features of the middle resolution. The average pooling layer can consider all features in a region, so that the network can focus on the complete semantic information of the target; the maximum pooling layer can obtain local information and retain more texture and edge information that is beneficial to target detection. These two pooling methods help to retain and concentrate important features.

[0132] On the other hand, the smallest resolution features are doubled in size through upsampling layers, such as bilinear interpolation or nearest neighbor interpolation, to match the feature size of the middle resolution. At the same time, upsampling helps to recover some lost details.

[0133] Finally, the resized maximum resolution features (through maximum pooling and average pooling), the original center resolution features, and the enlarged minimum resolution features are concatenated along the channel dimension through the concatenation layer. The essence of channel dimension concatenation is to connect feature information of different scales in series to form a composite feature containing rich multi-level information, which helps the model to understand the image content more comprehensively in subsequent processing and can effectively capture information of multiple scales from small targets to large backgrounds.

[0134] For example, see Fig. 9 , Fig. 9 The network structure of the MSFC block is shown. Based on this network structure, if the three first input features are recorded as the features with the maximum resolution F L ∈ , mid-resolution features F M ∈ and the minimum resolution feature F S ∈ ;The processing of the MSFC block can be described as:

[0135]

[0136] in, for F L Pooling result after the maximum pooling operation; for F L Pooling result after average pooling operation; for F S The sampling result after the upsampling operation.

[0137] The two pooling results, sampling results and the features of the center resolution are spliced ​​in the channel dimension to form a fusion feature F O ∈ :

[0138]

[0139] In some embodiments, if Figure 8 The model structure is used as the facility detection model. The electronic equipment is equipped with an Intel(R) Xeon(R) Platinum 8255C central processor, equipped with 314GB memory, and uses an NVIDIA Tesla V10032GB graphics card as an accelerator, running on a CentOS 8.5.2 64-bit system platform. The facility detection model is constructed using the PyTorch framework, and is programmed using PyCharm as the development environment.

[0140] The facility detection model and YOLOv8s are trained using the ACF-training set customized by this application, and the performance of the two models is evaluated based on the ACF-validation set. Fig.10 , Fig.10 A data comparison of the key evaluation indicators of the two models is shown.

[0141] By comparing the data of evaluation indicators, we can find that:

[0142] 1. Compared with YOLOv8s, the facility detection model has shown certain performance gains in the task of detecting anomalies in urban facilities. Specifically, the facility detection model has achieved a 1.5% increase in precision, a 1.0% increase in recall, and a 1.1% increase in mean average precision (mAP). These improvements demonstrate the superiority of the facility detection model in capturing and distinguishing target anomalies, and further enhance the accuracy and comprehensiveness of abnormal urban facility detection.

[0143] 2. The first and eighth types of abnormal urban facilities are characterized by scale diversity. The facility detection model shows good robustness in scale changes. Its precision increases by -0.6% and 1.3% respectively, the recall increases by 1.2% and 0.9% respectively, and the average precision (mAP) increases by 0.3% and 0.7% respectively.

[0144] 3. The third, fifth, ninth and eleventh categories of abnormal urban facilities have problems such as difficulty in scene understanding and ambiguous classification boundaries during the detection process. The facility detection model has demonstrated stronger detection capabilities in these aspects. The precision of the four categories increased by 3.3%, 1.0%, 3.3% and 2.3% respectively, the recall rate increased by 3.5%, 1.8%, 6.0% and 0% respectively, and the average precision (mAP) increased by 1.3%, 1.4%, 3.3% and 5.7% respectively.

[0145] By comparing the performance of the two models, it is not difficult to see that the improvement of the network structure of the facility detection model can effectively deal with the difficulties of urban facility anomaly detection such as the diversity of target sizes, small target detection, and complexity of scene semantic understanding, thereby achieving better detection results in related difficult categories.

[0146] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0147] Corresponding to the abnormal urban facility detection method based on YOLO in the above embodiment, Fig.11 A structural block diagram of a YOLO-based abnormal urban facility detection device 11 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0148] Reference Fig.11 , the abnormal urban facility detection device 11 based on YOLO includes:

[0149] An extraction module 111 is used to extract features from an image to be detected based on a backbone network of a pre-trained facility detection model to obtain image features; the image to be detected includes urban facilities;

[0150] A fusion module 112, used for fusing image features based on the neck network of the facility detection model to obtain fused features;

[0151] The detection module 113 is used to detect the fused features based on the detection network of the facility detection model to obtain the detection result of the abnormal urban facilities in the image to be detected;

[0152] The backbone network and / or the neck network is provided with a C2fMRFF block, and the C2fMRFF block is used to fuse local features and global features at different levels.

[0153] Optionally, the backbone network includes a preset convolutional layer and four stage layers connected in sequence, each stage layer includes a feature extraction module consisting of a preset convolutional layer and a C2fMRFF block, and the last stage layer is connected with an SPPF module after the feature extraction module;

[0154] The neck network is provided with a first MSFC block and a second MSFC block with the same structure; the first MSFC block is used to fuse the extracted features of the 2nd to 4th stage layers after convolution; the second MSFC block is used to fuse the extracted features of the first two stage layers after convolution, as well as the fused features of the first MSFC block after preliminary fusion and convolution; the preliminary fusion is achieved through the C2fMRFF block.

[0155] Optionally, for the 3 first input features of each MSFC block:

[0156] Performing maximum pooling and average pooling operations on the first input feature corresponding to the highest resolution and summing them up to obtain the first basic feature;

[0157] An upsampling operation is performed on the first input feature corresponding to the lowest resolution to obtain a second basic feature, where the size of the second basic feature is twice that of the first input feature corresponding to the lowest resolution;

[0158] The first basic feature, the first input feature corresponding to the middle resolution, and the second basic feature are concatenated to obtain fused features corresponding to the three first input features.

[0159] Optionally, the fusion feature includes a first scale fusion feature, a second scale fusion feature and a third scale fusion feature; the fusion module 112 is used for:

[0160] After performing a preliminary fusion operation on the fusion features of the second MSFC block based on the C2fMRFF block, the first scale fusion features are obtained;

[0161] After performing a preliminary fusion operation on the first spliced ​​features based on the C2fMRFF block, the second scale fusion features are obtained; the first spliced ​​features are obtained by splicing the fusion features of the first MSFC block after preliminary fusion and the first scale fusion features after convolution;

[0162] After a preliminary fusion operation is performed on the second splicing feature based on the C2fMRFF block, the third-scale fusion feature is obtained; the second splicing feature is obtained by splicing the extracted features of the 4th stage layer and the second-scale fusion feature after convolution.

[0163] Optionally, an ARFA submodule is provided in the feature extraction module corresponding to each scale, and the ARFA submodule is used to fuse different regions of the input features at a single scale based on different weights.

[0164] Optionally, the C2fMRFF block is obtained by replacing the Bottleneck structure in the C2f module with the MRFF structure; the MRFF structure is used to fuse local features and global features at each level; for the second input feature of each C2fMRFF block:

[0165] Perform channel expansion on the second input feature to obtain enhanced features;

[0166] The enhanced features are input into the sequentially connected multi-level MRFF structure to obtain the deep features corresponding to each level of the MRFF structure;

[0167] After concatenating the enhanced features and each deep feature, channel compression is performed to obtain the fusion feature corresponding to the second input feature.

[0168] Optionally, for the third input feature of each MRFF structure:

[0169] Performing a depth convolution operation on the third input feature after channel compression based on at least 2 dilation rates to obtain at least 2 depth convolution results;

[0170] After concatenating at least two deep convolution results with the third input feature, channel expansion is performed to obtain a deep feature corresponding to the third input feature.

[0171] Optionally, the facility detection model is trained based on a regression loss function and a classification loss function; the regression loss function is:

[0172]

[0173] in, L reg is the regression loss function, L SIoU is the bounding box regression loss function, L DFL is the dynamic category loss function; IoU is the intersection-over-union ratio, Δ is the distance loss, and Ω is the shape loss; S i is the cross entropy loss between the real border and the predicted border on the left, S i+1 is the cross entropy loss between the real border and the predicted border on the right; y is the real border, y i is the predicted box to the left of the true box, y i+1 is the predicted bounding box to the right of the true bounding box.

[0174] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0175] Fig.12 This is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. Fig.12 As shown, the electronic device 12 of this embodiment includes: at least one processor 120 ( Fig.12 Only one processor is shown in the figure), a memory 121, and a computer program 122 stored in the memory 121 and executable on at least one processor 120. When the processor 120 executes the computer program 122, the steps in any of the above-mentioned abnormal urban facility detection method embodiments based on YOLO are implemented, for example Figure 5 Steps 510-530 are shown.

[0176] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0177] In some embodiments, the memory 121 may be an internal storage unit of the electronic device 12, such as a hard disk or memory of the electronic device 12. In other embodiments, the memory 121 may also be an external storage device of the electronic device 12, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 12.

[0178] Furthermore, the memory 121 may also include both an internal storage unit of the electronic device 12 and an external storage device. The memory 121 is used to store operating devices, applications, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 121 may also be used to temporarily store data that has been output or is to be output.

[0179] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0180] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0181] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0183] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0184] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0185] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting abnormal urban facilities based on YOLO, characterized in that: include: Based on the backbone network of the pre-trained facility detection model, feature extraction is performed on the image to be detected to obtain image features; The image to be detected includes urban facilities; Based on the neck network of the facility detection model, the image features are fused to obtain fused features; The detection network based on the facility detection model detects the fused features to obtain a detection result of abnormal urban facilities in the image to be detected; Wherein, the backbone network and / or the neck network is provided with a C2fMRFF block, and the C2fMRFF block is used to fuse local features and global features at different levels; The backbone network includes a preset convolutional layer and four stage layers connected in sequence, each stage layer includes a feature extraction module composed of a preset convolutional layer and the C2fMRFF block, and the last stage layer is connected with an SPPF module after the feature extraction module; The neck network is provided with a first MSFC block and a second MSFC block; the first MSFC block is used to fuse the extracted features of the second to fourth stage layers after convolution; the second MSFC block is used to fuse the extracted features of the first two stage layers after convolution, and the fused features of the first MSFC block after preliminary fusion and convolution; the preliminary fusion is realized by the C2fMRFF block; For each of the three first input features of the MSFC block: Performing a maximum pooling operation and an average pooling operation on the first input features corresponding to the highest resolution and summing the results to obtain a first basic feature; Performing an upsampling operation on the first input feature corresponding to the lowest resolution to obtain a second basic feature, wherein a size of the second basic feature is twice that of the first input feature corresponding to the lowest resolution; The first basic feature, the first input feature corresponding to the middle resolution, and the second basic feature are concatenated to obtain three fused features corresponding to the first input feature.

2. The detection method according to claim 1, characterized in that The fusion features include first scale fusion features, second scale fusion features and third scale fusion features; The neck network based on the facility detection model fuses the image features to obtain fused features, including: After performing a preliminary fusion operation on the fusion features of the second MSFC block based on the C2fMRFF block, obtaining the first scale fusion features; After performing a preliminary fusion operation on the first splicing feature based on the C2fMRFF block, the second scale fusion feature is obtained; the first splicing feature is obtained by splicing the fusion feature of the first MSFC block after preliminary fusion and the first scale fusion feature after convolution; After performing a preliminary fusion operation on the second splicing feature based on the C2fMRFF block, the third-scale fusion feature is obtained; the second splicing feature is obtained by splicing the extracted features of the fourth stage layer and the second-scale fusion feature after convolution.

3. The detection method according to claim 1 or 2, characterized in that: The C2fMRFF block is obtained by replacing the Bottleneck structure in the C2f module with the MRFF structure; the MRFF structure is used to fuse the local features and global features at each level; for the second input feature of each C2fMRFF block: Performing channel expansion on the second input feature to obtain an enhanced feature; Inputting the enhanced features into the sequentially connected multi-level MRFF structures to obtain deep features corresponding to the MRFF structures at each level; After concatenating the enhanced features and each of the deep features, channel compression is performed to obtain a fusion feature corresponding to the second input feature.

4. The detection method according to claim 3, characterized in that For each of the third input features of the MRFF structure: Performing a deep convolution operation on the third input feature after channel compression based on at least 2 dilation rates to obtain at least 2 deep convolution results; After concatenating at least two of the deep convolution results with the third input feature, channel expansion is performed to obtain the deep feature corresponding to the third input feature.

5. The detection method according to claim 2, characterized in that: The facility detection model is trained based on a regression loss function and a classification loss function; the regression loss function is: Among them, the L reg is the regression loss function, L SIoU is the border regression loss function, L DFL is a dynamic category loss function; IoU is the intersection-over-union ratio, Δ is the distance loss, and Ω is the shape loss; S i is the cross entropy loss between the real border and the predicted border on the left. S i+1 is the cross entropy loss between the real border and the predicted border on the right; y is the real border, y i is the predicted box to the left of the real box, the y i+1 is the predicted bounding box to the right of the true bounding box.

6. A YOLO-based abnormal urban facility detection device, characterized in that: include: An extraction module is used to extract features from the image to be detected based on the backbone network of the pre-trained facility detection model to obtain image features; The image to be detected includes urban facilities; A fusion module, used for fusing the image features based on the neck network of the facility detection model to obtain fusion features; A detection module, used to detect the fused features based on the detection network of the facility detection model to obtain a detection result of abnormal urban facilities in the image to be detected; Wherein, the backbone network and / or the neck network is provided with a C2fMRFF block, and the C2fMRFF block is used to fuse local features and global features at different levels; The backbone network includes a preset convolutional layer and four stage layers connected in sequence, each stage layer includes a feature extraction module composed of a preset convolutional layer and the C2fMRFF block, and the last stage layer is connected with an SPPF module after the feature extraction module; The neck network is provided with a first MSFC block and a second MSFC block; the first MSFC block is used to fuse the extracted features of the second to fourth stage layers after convolution; the second MSFC block is used to fuse the extracted features of the first two stage layers after convolution, and the fused features of the first MSFC block after preliminary fusion and convolution; the preliminary fusion is realized by the C2fMRFF block; For each of the three first input features of the MSFC block: Performing a maximum pooling operation and an average pooling operation on the first input features corresponding to the highest resolution and summing the results to obtain a first basic feature; Performing an upsampling operation on the first input feature corresponding to the lowest resolution to obtain a second basic feature, wherein a size of the second basic feature is twice that of the first input feature corresponding to the lowest resolution; The first basic feature, the first input feature corresponding to the middle resolution, and the second basic feature are concatenated to obtain three fused features corresponding to the first input feature.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the abnormal urban facility detection method based on YOLO as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the abnormal urban facility detection method based on YOLO as described in any one of claims 1 to 5 is implemented.

Images