Urban facility anomaly identification method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide an urban facility anomaly identification method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring an image waiting for identification, said image comprising: an urban facility; and inputting said image into a pre-established target detection model to determine an anomaly category of the urban facility, wherein the target detection model comprises: a backbone network, a neck network, and a detection head connected in sequence, the backbone network comprises: residual focused convolution modules and multi-scale fusion attention mechanism modules, the residual focused convolution modules are used for extracting and fusing feature information of an image at different scales, and the multi-scale fusion attention mechanism modules are used for performing a multi-scale convolution fusion operation on the feature information of the image, such that an anomaly of an urban facility can be accurately identified.

Description

Urban facility anomaly identification method and device, electronic equipment and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202410788998.X, filed on June 19, 2024 in the China Patent Office, and entitled "Urban facility anomaly identification method and device, electronic equipment and storage medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application belongs to the technical field of urban facility anomaly detection, and particularly relates to an urban facility anomaly identification method and device, an electronic equipment and a storage medium. BACKGROUND

[0003] Urban facility anomaly events, such as traffic accidents and facility damage, can be timely identified and handled, which not only improves the efficiency of urban operation, but also ensures the safety of residents' life and property. Although existing monitoring systems and automation control technologies can detect certain abnormal situations to some extent, in the face of complex and changeable urban environment, traditional methods usually rely on fixed thresholds or rules to identify anomalies, which often lack flexibility and adaptability, and have low identification accuracy for urban facility anomalies. TECHNICAL PROBLEM

[0004] To solve the above problems, the present application provides an urban facility anomaly identification method and device, an electronic equipment and a storage medium, which can accurately identify the anomaly of urban facilities. TECHNICAL SOLUTION

[0005] The present application provides an urban facility anomaly identification method, comprising:

[0006] Obtaining an image to be identified, the image comprising: urban facilities;

[0007] Inputting the image into a pre-established target detection model to determine the anomaly category of the urban facilities, wherein the target detection model comprises a backbone network, a neck network and a detection head connected in sequence, the neck network is used to output different scale feature information output by the backbone network to the corresponding detection head, the backbone network comprises a residual focus convolution module and a multi-scale fusion attention mechanism module, the residual focus convolution module is used to extract and fuse the feature information of different scales of the image, and the multi-scale fusion attention mechanism module is used to perform multi-scale convolution fusion operation on the feature information of the image.

[0008] In some embodiments, the residual focus convolution module comprises a first residual focus convolution module, a second residual focus convolution module, a third residual focus convolution module, and a fourth residual focus convolution module, the multi-scale fusion attention mechanism module comprises a first multi-scale fusion attention mechanism module, a second multi-scale fusion attention mechanism module, and a third multi-scale fusion attention mechanism module, the backbone network further comprises a convolution module, a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, and a spatial pyramid pooling module, the convolution module is used for inputting an image, the convolution module, the first CBS module, the first residual focus convolution module, the second CBS module, the second residual focus convolution module, the first multi-scale fusion attention mechanism module, the third CBS module, the third residual focus convolution module, the second multi-scale fusion attention mechanism module, the fourth CBS module, the fourth residual focus convolution module, the third multi-scale fusion attention mechanism module, and the spatial pyramid pooling module are sequentially connected, the first multi-scale fusion attention mechanism module is used for outputting first scale feature information to a neck network of the target detection model, the second multi-scale fusion attention mechanism module is used for outputting second scale feature information to the neck network, and the spatial pyramid pooling module is used for outputting third scale feature information to the neck network.

[0009] In some embodiments, the residual focus convolution module comprises a residual focus convolution block, a fifth CBS module, and a sixth CBS module, the fifth CBS module is used for providing feature information of different scales to the residual focus convolution block, the residual focus convolution block is used for feature extraction on the feature information of different scales, nonlinear transformation on the extracted feature information, gradient flow maintenance of the target detection model, and input of the processed feature information to the sixth CBS module.

[0010] In some embodiments, the residual focus convolution block comprises a first depth convolution of a first convolution kernel size, a second depth convolution of the first convolution kernel size, a LayerNorm layer, a convolution of a second convolution kernel size, a GELU unit, and a convolution of the second convolution kernel size, which are sequentially connected, wherein an input of the first depth convolution is in residual connection with the first depth convolution, an input of the second depth convolution is in residual connection with the second depth convolution, the GELU unit is used for introducing nonlinearity, and the convolution of the second convolution kernel size is used for integrating feature information and adjusting a number of channels.

[0011] In some embodiments, the multi-scale fusion attention mechanism module comprises: a multi-scale convolution module and an SE module connected in sequence, the multi-scale convolution module is used to obtain feature information of different scales, and the feature information of different scales is spliced and the spliced feature information is output, and the SE module is used to extract attention weight of the spliced feature information, determine a shared channel attention vector based on the attention weight, and correct the shared channel attention vector, and apply the corrected attention vector to the multi-scale feature map, and output the result.

[0012] In some embodiments, the feature information of different scales corresponds to different detection heads, each detection head comprises: a REG module, a first convolution shared BN separation layer, a second convolution shared BN separation layer, and a seventh CBS module, the input of the REG module is the output of the neck network, the output of the REG module is the input of the first convolution shared BN separation layer, the output of the first convolution shared BN separation layer is the input of the bounding box regression loss function, the input of the seventh CBS module is the output of the neck network, the output of the seventh CBS module is the input of the second convolution shared BN separation layer, and the output of the second convolution shared BN separation layer is the input of the classification loss function, and the first convolution shared BN separation layer and the second convolution shared BN separation layer are used to share convolution weights between different layers.

[0013] In some embodiments, the neck network comprises: a first concat module, a second concat module, a third concat module, a fourth concat module, a fifth residual focus convolution module, a sixth residual focus convolution module, a seventh residual focus convolution module, an eighth residual focus convolution module, an eighth CBS module, a ninth CBS module, a first upsampling module, and a second upsampling module, an input of the first concat module is an output of the first multi-scale fusion attention mechanism module and the first upsampling module, an output of the first concat module is an input of the fifth residual focus convolution module, an output of the fifth residual focus convolution module is an input of the eighth CBS module and a detection head, an input of the second concat module is an output of the second upsampling module and the second multi-scale fusion attention mechanism module, an output of the second concat module is an input of the sixth residual focus convolution module, an output of the sixth residual focus convolution module is an input of the first upsampling module and the third concat module, an input of the third concat module is an output of the eighth CBS module, an output of the third concat module is an input of the seventh residual focus convolution module, an output of the seventh residual focus convolution module is an input of the detection head and the ninth CBS module, an input of the fourth concat module is an output of the spatial pyramid pooling module and the ninth CBS module, an input of the second upsampling module is an output of the spatial pyramid pooling module, an output of the fourth concat module is an input of the eighth residual focus convolution module, and an output of the eighth residual focus convolution module is an input of the detection head.

[0014] The embodiment of the present application provides a kind of urban facility anomaly identification device, comprising:

[0015] Acquisition module, for acquiring image to be identified, the image includes: urban facilities;

[0016] Determination module, for inputting the image into the target detection model established in advance, determine the abnormal class of the urban facilities, wherein the target detection model includes: backbone network, neck network and detection head connected in turn, the neck network is used to export the feature information of different scales exported by backbone network to corresponding detection head, the backbone network includes: residual focus convolution module and multi-scale fusion attention mechanism module, the residual focus convolution module is used to extract and fuse the feature information of different scales of image, and the multi-scale fusion attention mechanism module is used to carry out multi-scale convolution fusion operation to the feature information of image.

[0017] An electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of any one of the above when executing the computer program.

[0018] A computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of the above.

[0019] A computer program product is provided, and when the computer program product is executed on a terminal device, the electronic device executes the method of any one of the above.

[0020] The method, device, electronic device and storage medium provided by the embodiments of the present application can realize more accurate identification of the abnormality of the urban facility by acquiring an image to be identified, the image including an urban facility, inputting the image into a pre-established target detection model to determine an abnormality category of the urban facility, wherein the target detection model includes a backbone network, a neck network and a detection head connected in sequence, the neck network is used to output different scale feature information output by the backbone network to the corresponding detection head, the backbone network includes a residual focus convolution module and a multi-scale fusion attention mechanism module, the residual focus convolution module is used to extract and fuse the feature information of different scales of the image, and the multi-scale fusion attention mechanism module is used to perform a multi-scale convolution fusion operation on the feature information of the image. BRIEF DESCRIPTION OF DRAWINGS

[0021] The present application will be described in more detail below based on the embodiments and with reference to the accompanying drawings.

[0022] FIG. 1 is a schematic diagram of an implementation process of a method for identifying an abnormality of an urban facility according to an embodiment of the present application;

[0023] FIG. 2 is a schematic diagram of a structure of a target detection model according to an embodiment of the present application;

[0024] FIG. 3 is a schematic diagram of a structure of a residual focus convolution module according to an embodiment of the present application;

[0025] FIG. 4 is a schematic diagram of a structure of a residual focus convolution block according to an embodiment of the present application;

[0026] FIG. 5 is a schematic diagram of a processing process of a multi-scale fusion attention mechanism module according to an embodiment of the present application;

[0027] FIG. 6 is a schematic diagram of a structure of a device for identifying an abnormality of an urban facility according to an embodiment of the present application;

[0028] FIG. 7 is a schematic diagram of a component structure of an electronic device according to an embodiment of the present application.

[0029] In the drawings, the drawings are not drawn according to the actual proportion. Embodiments of the present application

[0030] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0032] If similar descriptions of "first\second\third" appear in the application file, the following description is added, in the following description, the terms "first\second\third" referred to only distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first\second\third" can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] Based on the problems in the related art, the present application provides a city facility anomaly identification method, the city facility anomaly identification method provided in the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The present application does not make any limitation on the specific type of electronic device.

[0035] The function of the city facility anomaly identification method provided in the embodiments of the present application can be realized by calling program code of a processor of an electronic device, wherein the program code can be stored in a computer storage medium.

[0036] The embodiments of the present application provide a city facility anomaly identification method, and FIG. 1 is a schematic diagram of an implementation process of a city facility anomaly identification method provided in the embodiments of the present application. As shown in FIG. 1, the city facility anomaly identification method comprises the following steps.

[0037] In step S101, an image to be identified is acquired, wherein the image comprises city facilities.

[0038] In the embodiments of the present application, the image to be identified can be acquired by photographing using a camera, a drone or other image acquisition devices. The acquired image to be identified can comprise city facilities, and the city facilities can comprise one or more of roads, manhole covers, guardrails, wall surfaces, expansion devices, drainage facilities, isolation fences, anti-collision columns, and car stop stones.

[0039] In the embodiments of the present application, when the camera is used, the camera can be arranged on a vehicle, and the image to be identified can be acquired by moving the vehicle. The image information can comprise a picture or a video.

[0040] In some embodiments, the electronic device can also acquire the image to be identified through the Internet.

[0041] In step S102, the image is input into a pre-established target detection model to determine an anomaly category of the city facility.

[0042] In the embodiments of the present application, the image information can be input into the pre-established target detection model, and after processing by the model, the anomaly category of the city facility can be determined.

[0043] In the embodiments of the present application, the anomaly category can comprise a plurality of categories, for example, one or more of a first category, a second category, a third category and the like.

[0044] In the embodiments of the present application, the target detection model comprises a backbone network, a neck network and a detection head connected in sequence, the neck network is used to output different scale feature information output by the backbone network to the corresponding detection head, the backbone network comprises a residual focus convolution module and a multi-scale fusion attention mechanism module, the residual focus convolution module is used to extract and fuse the different scale feature information of the image, and the multi-scale fusion attention mechanism module is used to perform a multi-scale convolution fusion operation on the feature information of the image.

[0045] In the embodiments of the present application, the residual focus convolution module can perform feature extraction and fusion of different scales, and can provide more context information for the model to solve the problem of ambiguous class semantics in urban facility anomaly recognition.

[0046] In the embodiments of the present application, the multi-scale fusion attention mechanism module can introduce multi-scale convolution operation under a single scale to achieve the effect of multi-scale fusion, thereby solving the problem of multi-scale difference between intra-class and inter-class in urban facility anomaly recognition, and helping to improve the detection effect of the model.

[0047] In the embodiments of the present application, the target detection model is improved based on the MFSNet network.

[0048] The urban facility anomaly recognition method provided in the embodiments of the present application can accurately identify the anomaly of urban facilities by acquiring an image to be recognized, inputting the image into a pre-established target detection model, and determining the anomaly class of the urban facility, wherein the target detection model comprises a backbone network, a neck network and a detection head connected in sequence, the neck network is used to output different scale feature information output by the backbone network to the corresponding detection head, the backbone network comprises a residual focus convolution module and a multi-scale fusion attention mechanism module, the residual focus convolution module is used to extract and fuse the feature information of the image of different scales, and the multi-scale fusion attention mechanism module is used to perform multi-scale convolution fusion operation on the feature information of the image.

[0049] In some embodiments, the residual focus convolution module includes a first residual focus convolution module, a second residual focus convolution module, a third residual focus convolution module, and a fourth residual focus convolution module, the multi-scale fusion attention mechanism module includes a first multi-scale fusion attention mechanism module, a second multi-scale fusion attention mechanism module, and a third multi-scale fusion attention mechanism module, the backbone network further includes a convolution module, a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, and a spatial pyramid pooling module, the convolution module is used for inputting an image, the convolution module, the first CBS module, the first residual focus convolution module, the second CBS module, the second residual focus convolution module, the first multi-scale fusion attention mechanism module, the third CBS module, the third residual focus convolution module, the second multi-scale fusion attention mechanism module, the fourth CBS module, the fourth residual focus convolution module, the third multi-scale fusion attention mechanism module, and the spatial pyramid pooling module are sequentially connected, the first multi-scale fusion attention mechanism module is used for outputting feature information of a first scale to a neck network of the target detection model, the second multi-scale fusion attention mechanism module is used for outputting feature information of a second scale to the neck network, and the spatial pyramid pooling module is used for outputting feature information of a third scale to the neck network.

[0050] The CBS (Cross stage Partial Network) module is a cross-stage partial network used for extracting features of different scales. The CBS module can capture targets of different sizes at different levels.

[0051] In the embodiments of the present application, the convolution module uses the Conv function to convolve the input image.

[0052] In the embodiments of the present application, the first scale, the second scale, and the third scale are different, for example, the first scale is a feature map of 1 / 8, the second scale is a feature map of 1 / 16, and the third scale is a feature map of 1 / 32.

[0053] In the embodiments of the present application, the spatial pyramid pooling module can be SPPF (Spatial Pyramid Pooling Fusion), which is a spatial pyramid pooling technology used for processing images or feature maps of variable sizes. SPPF can divide an image or a feature map into grids of different sizes and perform pooling operations in each grid, thereby encoding inputs of different sizes on a fixed-size output. SPPF can help neural networks maintain the richness of spatial information when processing inputs of different sizes.

[0054] FIG. 2 is a structural schematic diagram of a target detection model provided by an embodiment of the present application, as shown in FIG. 2, a convolution module is denoted as Conv, a residuals focused convolution module can be denoted as C2RFC, and a multi-scale fusion attention mechanism module can be denoted as MSFA. A spatial pyramid pooling module is denoted as SPPF.

[0055] In some embodiments, the residuals focused convolution module comprises a residuals focused convolution block, a fifth CBS module and a sixth CBS module, the fifth CBS module is configured to provide feature information of different scales to the residuals focused convolution block, the residuals focused convolution block is configured to perform feature extraction on the feature information of different scales, perform nonlinear transformation on the extracted feature information, maintain gradient flow of the target detection model, and input the processed feature information to the sixth CBS module.

[0056] In an embodiment of the present application, the residuals focused convolution block can be an RFC Block (Residuals Focused Convlution Block).

[0057] FIG. 3 is a structural schematic diagram of a residuals focused convolution module provided by an embodiment of the present application, as shown in FIG. 3, the fifth CBS module is an input end of the residuals focused convolution module, and the sixth CBS module is an output end of the residuals focused convolution module.

[0058] In an embodiment of the present application, S=1 and K=1 for the fifth CBS module, and S=1 and K=1 for the sixth CBS module.

[0059] In some embodiments, the residuals focused convolution block comprises a first depth convolution of a first convolution kernel size, a second depth convolution of the first convolution kernel size, a LayerNorm layer, a convolution of a second convolution kernel size, a GELU unit and a convolution of the second convolution kernel size, which are sequentially connected, wherein the input of the first depth convolution is connected to the first depth convolution in a residual manner, the input of the second depth convolution is connected to the second depth convolution in a residual manner, the GELU unit is configured to introduce nonlinearity, and the convolution of the second convolution kernel size is configured to integrate feature information and adjust the number of channels. The GELU unit is a Gaussian Error Linear Unit, which is an activation function and is usually used in the hidden layer of a neural network.

[0060] FIG. 4 is a structural schematic diagram of a residual focus convolution block provided in an embodiment of the present application. As shown in FIG. 4, the residual focus convolution block includes two residual 7x7 deep convolutions, a LayerNorm layer, a 1x1 convolution, a GELU unit, a 1x1 convolution, and a large residual structure with a skip connection. The residual focus convolution block can effectively extract features and perform nonlinear transformation while maintaining gradient flow and avoiding the problem of gradient vanishing.

[0061] In an embodiment of the present application, there is an input feature map X ∈ R H×W×C , and the structure of the RFC Block can be represented as: Y1 = DWConv 7×7 (X) + X; Y2 = DWConv 7×7 (Y1) + Y1; O = Conv 1×1 (GELU(Conv 1×1 (LayerNorm(Y2)))) + X.

[0062] wherein DWConv 7×7 represents a deep convolution with a convolution kernel size of 7x7, which is the same as the residual 7x7 deep convolution described above. Y1 represents the residual connection formed by the original input X and the first 7x7 deep convolution, which helps to optimize training and solve the problem of gradient vanishing. Y2 represents the residual connection formed by Y1 and the second 7x7 deep convolution, which is used to capture more extensive context information in the spatial dimension and can serve as a focus target. LN is LayerNorm, which is used to accelerate the training process and improve the stability of the model. The GELU unit is an activation function to introduce nonlinearity. Conv 1×1 represents a convolution with a convolution kernel size of 1x1, which is used to integrate features and adjust the number of channels.

[0063] In an embodiment of the present application, through the structural design of the RFC Block, a small amount of additional computational cost can be introduced to enhance the learning ability of the model for complex features. The RFC Block can simultaneously combine fine-grained local features and coarse-grained global features for interaction.

[0064] In some embodiments, the multi-scale fusion attention mechanism module includes a multi-scale convolution module and an SE module connected in sequence. The multi-scale convolution module is used to obtain feature information of different scales, and to splice and output the spliced feature information. The SE module is used to extract attention weights from the spliced feature information, determine a shared channel attention vector based on the attention weights, and correct the shared channel attention vector. The corrected attention vector is applied to the multi-scale feature map, and the result is output.

[0065] Figure 5 is a schematic diagram of the processing flow of a multi-scale fusion attention mechanism module provided in an embodiment of this application. As shown in Figure 5, the multi-scale fusion attention mechanism first obtains feature information of different scales through the multi-scale convolution module MSC; then, it obtains shared channel-level attention vectors through the SE module by weight learning; finally, it uses Softmax to recalibrate the channel attention vectors obtained above, and applies the calibrated attention vectors to the multi-scale feature map to output the results.

[0066] In this embodiment, the SE module refers to the Squeeze-and-Excitation module, which is an attention mechanism module used to enhance the performance of convolutional neural networks.

[0067] In Figure 5, Softmax is the Softmax function, which converts the elements in the input vector into probability values.

[0068] In this embodiment, the output of the MSC module has rich multi-scale information. Assume the input is X. H×W×C First, features are extracted from X using grouped convolutions (g = 1, 4, 8, 16) with different kernel sizes (k = 3, 5, 7, 9), resulting in feature maps of different scales. Then, the extracted multi-scale features are concatenated using `concat`. This process can be simply described as follows: F i =Conv(k i ×k i ,g i )(X),i=0,1,…,S-1; F=concat([F0,F1,…,F S-1 ]);

[0069] Based on the above features, attention weights are extracted from different feature subsets and concatenated across different dimensions to achieve attention information interaction and fusion of cross-dimensional information. The formula is as follows: Z i =SEW(F i ),i=0,1,…,S-1; Z=concat([Z0,Z1,…,Z S-1 ]);

[0070] Then the obtained attention weights are normalized, defined as follows: attention = Softmax(Z);

[0071] Finally, the corrected feature is obtained: Y = F·attention;

[0072] In the embodiments of the present application, the MSFA attention mechanism is added behind the feature maps of different scales in the backbone network, which can introduce multi-scale convolution operations in a single scale to achieve multi-scale fusion effect, thereby solving the multi-scale difference problem between intra-class and inter-class in urban facility anomaly identification, and helping to improve the detection effect of the model.

[0073] In some embodiments, different scale feature information corresponds to different detection heads, and each detection head includes: a REG module, a first convolution shared BN separation layer, a second convolution shared BN separation layer, and a seventh CBS module. The input of the REG module is the output of the neck network, the output of the REG module is the input of the first convolution shared BN separation layer, the output of the first convolution shared BN separation layer is the input of the bounding box regression loss function, the input of the seventh CBS module is the output of the neck network, the output of the seventh CBS module is the input of the second convolution shared BN separation layer, and the output of the second convolution shared separation layer is the input of the classification loss function. The first convolution shared BN separation layer and the second convolution shared BN separation layer are used to share convolution weights between different layers.

[0074] Continuing to refer to FIG. 2, in the embodiments of the present application, the BN separation layer is a batch normalization (Batch Normalization, BN) layer, and the first convolution shared BN separation layer and the second convolution shared BN separation layer are used to share convolution weights between different layers. The bounding box regression loss function can be represented by Bbox Loss, and the classification loss function can be represented by Cls Loss. The REG (Region Proposal Network or Regression Network) module generally refers to a region proposal network or a regression network. This module is used to generate candidate object frames or accurately regress the bounding box of the object frame in the target detection algorithm. In the target detection task, a series of candidate regions that may contain target objects are usually generated, and the accurate bounding box position of the target object in these candidate regions is determined in the subsequent processing. The REG module generates candidate frames or regresses the position of the candidate frames by learning feature mapping, thereby helping the model to more accurately locate the target object.

[0075] Based on the above example, the 1 / 8 feature map corresponds to one detection head, the 1 / 16 feature map corresponds to one detection head, and the 1 / 32 feature map corresponds to one detection head.

[0076] In the embodiments of the present application, the detection head can be considered as a convolution shared BN separation detection head SVSBHead (Share Convolution Separate BN Head).

[0077] In the embodiments of the present application, each detection head is realized by independently calculating the statistics of BN by sharing the convolution weights between different layers. The detection head not only reduces the parameter quantity and the calculation quantity, but also improves the flexibility and adaptability of the model when processing different features at different layers, thereby improving the overall performance and generalization ability of the model. At the same time, BN is more effective than other normalization layers (such as Group normalization) because it directly uses the statistics calculated in training in inference.

[0078] In some embodiments, continuing to refer to FIG. 2, the neck network comprises: a first concat module, a second concat module, a third concat module, a fourth concat module, a fifth residual focus convolution module, a sixth residual focus convolution module, a seventh residual focus convolution module, an eighth residual focus convolution module, an eighth CBS module, a ninth CBS module, a first upsampling module, and a second upsampling module, an input of the first concat module is an output of the first multi-scale fusion attention mechanism module and the first upsampling module, an output of the first concat module is an input of the fifth residual focus convolution module, an output of the fifth residual focus convolution module is an input of the eighth CBS module and a detection head, an input of the second concat module is an output of the second upsampling module and the second multi-scale fusion attention mechanism module, an output of the second concat module is an input of the sixth residual focus convolution module, an output of the sixth residual focus convolution module is an input of the first upsampling module and the third concat module, an input of the third concat module is an output of the eighth CBS module, an output of the third concat module is an input of the seventh residual focus convolution module, an output of the seventh residual focus convolution module is an input of the detection head and the ninth CBS module, an input of the fourth concat module is an output of the spatial pyramid pooling module and the ninth CBS module, an input of the second upsampling module is an output of the spatial pyramid pooling module, an output of the fourth concat module is an input of the eighth residual focus convolution module, and an output of the eighth residual focus convolution module is an input of the detection head.

[0079] In the embodiments of the present application, the role of the concat module is to splice or connect the features of multiple inputs along the specified dimension to combine information from different sources. By splicing multiple input features, the neural network can utilize information from different sources at the same time, thereby improving the representation ability and performance of the network.

[0080] In some embodiments, before step S102, the method further comprises:

[0081] Step S1021, obtain a sample dataset.

[0082] In an embodiment of the present application, the sample dataset can be an urban facility anomaly dataset.

[0083] The images in the sample dataset are labeled with anomaly types. In an embodiment of the present application, a camera or other image acquisition device can be used to capture images in the real world. The captured images are labeled to mark the anomaly types. Specialized image labeling tools such as LabelImg, CVAT, etc. can be used to label the anomaly types of each image. Then the labeled images and corresponding label data are organized into a dataset. When the sample dataset is obtained, it can be used to train the target detection model.

[0084] Step S1022, train an initial target detection model based on the sample dataset to obtain the target detection model.

[0085] In an embodiment of the present application, the sample dataset can be preprocessed, which can include image scaling, cropping, rotation, etc. and formatting and processing of the label data to make it suitable for the training requirements of the target detection model. Then an initial target detection model can be selected and the selected initial target detection model can be trained using the sample dataset. During the training process, the model will learn how to accurately detect anomaly types in images through multiple iterations. During the training process, the hyperparameters of the model may need to be tuned to obtain better performance. This can include adjustment of learning rate, batch size, number of iterations, etc. The target detection model obtained by training needs to be evaluated. Then the model that performs best on the validation set evaluation metrics is selected as the final target detection model, which can be deployed to actual applications for urban facility anomalies.

[0086] In some embodiments, step S1022 can be implemented by the following steps:

[0087] Step S1, divide the sample dataset into a training set and a validation set.

[0088] In an embodiment of the present application, the training set accounts for a large portion of the total dataset (e.g. 70-80%), while the validation set accounts for a smaller portion (20-30%).

[0089] Step S2, train the initial target detection model based on the training set and test the trained initial target detection model through the validation set to obtain evaluation metrics.

[0090] In an embodiment of the present application, the training set is used to train the initial target detection model, and the model training and tuning are performed according to the steps mentioned above.

[0091] In the embodiments of the present application, the evaluation index can include one or more of IOU (Intersection over Union), precision, recall, average precision, and mean average precision, wherein the calculation method of the evaluation index is as follows:

[0092] IOU is an index for evaluating the overlapping degree of the true frame and the predicted frame, and is the ratio of the intersection to the union, which plays a key role in determining whether the frame is a correct detection.

[0093] The precision (P) is also called the recall rate, which refers to the proportion of samples that are actually abnormal in all samples predicted to be abnormal, and is a measurement standard, as shown in the formula:

[0094] The recall rate (R) is also called the recall rate, which refers to the proportion of samples that are actually abnormal in all samples predicted to be abnormal, and is a measurement standard, as shown in the formula:

[0095] The average precision (AP) refers to the area under the precision-recall curve of the model output in the task of urban facility anomaly identification, and is usually used to evaluate the performance of the model at a series of different IoU thresholds, as shown in the formula:

[0096] The mean average precision (mAP) refers to the average value of the average precision AP of C different categories, which is set to 11 in this paper, as shown in the formula:

[0097] In step S3, if the evaluation index meets the best requirement, the initial target detection model after training is determined as the target detection model.

[0098] In the embodiments of the present application, if the performance of the model does not meet the requirements, the structure, hyperparameters, etc. of the model may need to be adjusted, and then the model is retrained and evaluated.

[0099] In the embodiments of the present application, the loss function of the target detection model is composed of two parts of regression loss and classification loss. The classification loss is binary cross-entropy loss BCE Loss, which is used to judge the specific category in the anchor frame. The regression loss is CIoU Loss and DFL loss, which is used to calculate the error between the predicted boundary frame (predicted bound box) and the true boundary frame (ground truth box). At the same time, the matching strategy between positive and negative samples adopts the TAL dynamic matching method.

[0100] Binary Cross-Entropy Loss BCE Loss can be expressed as:

[0101] where y i is the true class; y' i is the predicted class.

[0102] Regression Loss CIoU Loss is an improved version based on previous L1, L2, IoU and GIoU loss functions. CIOU adds a penalty term for the aspect ratio, which can better distinguish errors in different situations when the center points of the predicted and real bounding boxes coincide, and has scale invariance.

[0103] Regression Loss can be expressed as:

[0104] where v is used to measure the consistency of the relative proportion of the two rectangular frames, and a is the weight coefficient:

[0105] where b and b gt represent the center points of the two rectangular frames, p represents the Euclidean distance between the two rectangular frames, and c represents the distance of the diagonal line of the closure area of the two rectangular frames.

[0106] Regression Loss DFL Loss can be expressed as:

[0107] where S i is the cross-entropy loss of the left target true value and the predicted value, and S i+1 is the cross-entropy loss of the right target true value and the predicted value.

[0108] Total Loss Loss can be expressed as: Loss=λ1L cls +λ2L reg =λ1·L BCE +λ2(L CIoU +L DFL );

[0109] where λ1, λ2 are balance coefficients.

[0110] Based on the foregoing target detection network, the model is trained on the self-built dataset by the target detection network and the YOLOv8s network, and the performance on the validation set is counted. The evaluation index comparison is shown in the following table. When performing verification, the hardware environment for experimental operation is: the processor is Intel(R) Xeon(R) Platinum 8255C CPU, 314G memory, the graphics card is NVIDIA Tesla V100 32G, CentOS 8.5.2 64-bit operating system, the deep neural network is built on the pytorch deep learning framework, and the compiler is pycharm. Table 1 is an evaluation index comparison table provided by the embodiment of the application, as shown in Table 1,

[0111] Table 1 is an evaluation index comparison of YOLOv8s and the target detection network

[0112] Through analysis of the table, it can be found that compared with the YOLOv8s network, the target detection network decreases by 3% in precision P, increases by 0.7% in recall R, and increases by 1.0% in mAP. It can be seen that the target detection network can significantly improve the recognition accuracy of urban facility anomalies. For the YOLOv8s network, the first category, the third type and the ninth category are three categories with low recognition accuracy, which are difficult categories. The target detection network improves the precision P of these three categories by 2%, 7.3% and 7.2% respectively, and the average precision mAP is improved by 0.5%, 1.9% and 4.0% respectively. It can be seen that the target detection network provided by the embodiment of the application has good recognition effect for small targets, semantically ambiguous and scale-uniform categories.

[0113] Based on the foregoing embodiment, the embodiment of the application provides a city facility anomaly recognition device. Each module included in the device and each unit included in the module can be implemented by a processor in a computer device. Of course, it can also be implemented by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0114] The embodiment of the application provides a city facility anomaly recognition device. Figure 6 is a structural schematic diagram of a city facility anomaly recognition device provided by the embodiment of the application. As shown in Figure 6, the city facility anomaly recognition device 600 includes:

[0115] The acquisition module 601 is configured to acquire an image to be recognized, wherein the image comprises urban facilities.

[0116] The determination module 602 is configured to input the image into a pre-established target detection model to determine an abnormal category of the urban facilities, wherein the target detection model comprises a backbone network, a neck network and a detection head connected in sequence, the neck network is configured to output feature information of different scales output by the backbone network to corresponding detection heads, the backbone network comprises a residual focus convolution module and a multi-scale fusion attention mechanism module, the residual focus convolution module is configured to extract and fuse feature information of different scales of the image, and the multi-scale fusion attention mechanism module is configured to perform a multi-scale convolution fusion operation on the feature information of the image.

[0117] In some embodiments, the residual focus convolution module comprises a first residual focus convolution module, a second residual focus convolution module, a third residual focus convolution module and a fourth residual focus convolution module, the multi-scale fusion attention mechanism module comprises a first multi-scale fusion attention mechanism module, a second multi-scale fusion attention mechanism module and a third multi-scale fusion attention mechanism module, and the backbone network further comprises a convolution module, a first CBS module, a second CBS module, a third CBS module, a fourth CBS module and a spatial pyramid pooling module, the convolution module is configured to input the image, the convolution module, the first CBS module, the first residual focus convolution module, the second CBS module, the second residual focus convolution module, the first multi-scale fusion attention mechanism module, the third CBS module, the third residual focus convolution module, the second multi-scale fusion attention mechanism module, the fourth CBS module, the fourth residual focus convolution module, the third multi-scale fusion attention mechanism module and the spatial pyramid pooling module are connected in sequence, the first multi-scale fusion attention mechanism module is configured to output feature information of a first scale to the neck network of the target detection model, the second multi-scale fusion attention mechanism module is configured to output feature information of a second scale to the neck network, and the spatial pyramid pooling module is configured to output feature information of a third scale to the neck network.

[0118] In some embodiments, the residual focus convolution module comprises a residual focus convolution block, a fifth CBS module and a sixth CBS module, the fifth CBS module is configured to provide feature information of different scales to the residual focus convolution block, the residual focus convolution block is configured to extract features from the feature information of different scales, perform a nonlinear transformation on the extracted feature information, maintain gradient flow of the target detection model, and input the processed feature information to the sixth CBS module.

[0119] In some embodiments, the residual focus convolution block comprises: a first deep convolution of a first convolution kernel size, a second deep convolution of the first convolution kernel size, a LayerNorm layer, a convolution of a second convolution kernel size, a GELU unit and a convolution of the second convolution kernel size connected in sequence, wherein the input of the first deep convolution forms a residual connection with the first deep convolution, the input of the second deep convolution forms a residual connection with the second deep convolution, the GELU unit is used to introduce nonlinearity, and the convolution of the second convolution kernel size is used to integrate feature information and adjust the number of channels.

[0120] In some embodiments, the multi-scale fusion attention mechanism module comprises: a multi-scale convolution module and an SE module connected in sequence, the multi-scale convolution module is used to obtain feature information of different scales, and to splice the feature information of different scales and output the spliced feature information, and the SE module is used to extract attention weights from the spliced feature information, determine a shared channel attention vector based on the attention weights, re-correct the shared channel attention vector, and apply the re-corrected attention vector to multi-scale feature maps and output a result.

[0121] In some embodiments, the target detection model further comprises: a detection head and a neck network, the neck network is used to output different scale feature information input by the backbone network to a corresponding detection head, wherein different scale feature information corresponds to different detection heads, each detection head comprises: a REG module, a first convolution shared BN separation layer, a second convolution shared BN separation layer and a seventh CBS module, the input of the REG module is the output of the neck network, the output of the REG module is the input of the first convolution shared BN separation layer, the output of the first convolution shared BN separation layer is the input of a bounding box regression loss function, the input of the seventh CBS module is the output of the neck network, the output of the seventh CBS module is the input of the second convolution shared BN separation layer, the output of the second convolution shared separation layer is the input of a classification loss function, and the first convolution shared BN separation layer and the second convolution shared BN separation layer are used to share convolution weights between different layers.

[0122] In some embodiments, the neck network comprises: a first concat module, a second concat module, a third concat module, a fourth concat module, a fifth residual focus convolution module, a sixth residual focus convolution module, a seventh residual focus convolution module, an eighth residual focus convolution module, an eighth CBS module, a ninth CBS module, a first up-sampling module, and a second up-sampling module, an input of the first concat module is an output of the first multi-scale fusion attention mechanism module and the first up-sampling module, an output of the first concat module is an input of the fifth residual focus convolution module, an output of the fifth residual focus convolution module is an input of the eighth CBS module and a detection head, an input of the second concat module is an output of the second up-sampling module and the second multi-scale fusion attention mechanism module, an output of the second concat module is an input of the sixth residual focus convolution module, an output of the sixth residual focus convolution module is an input of the first up-sampling module and the third concat module, an input of the third concat module is an output of the eighth CBS module, an output of the third concat module is an input of the seventh residual focus convolution module, an output of the seventh residual focus convolution module is an input of the detection head and the ninth CBS module, an input of the fourth concat module is an output of the spatial pyramid pooling module and the ninth CBS module, an input of the second up-sampling module is an output of the spatial pyramid pooling module, an output of the fourth concat module is an input of the eighth residual focus convolution module, and an output of the eighth residual focus convolution module is an input of the detection head.

[0123] It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0124] FIG. 7 is a structural schematic diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 7, the electronic device 3 according to the embodiment can include at least one processor 30 (only one processor 30 is shown in FIG. 7), a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30. The processor 30 implements the steps in any of the above method embodiments, such as steps S101-S102 in the embodiment shown in FIG. 1, when executing the computer program 32. Alternatively, the processor 30 implements the functions of the modules / units in any of the above device embodiments, such as the functions of the modules 601-602 shown in FIG. 6, when executing the computer program 32.

[0125] For example, the computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31 and executed by the processor 30 to complete the present application. One or more modules / units can be a series of computer program 32 instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 32 in the electronic device 3.

[0126] The embodiments of the present application also provide a computer readable storage medium, which stores the computer program 32. The computer program 32 is executed by the processor 30 to implement the steps in any of the above method embodiments.

[0127] The embodiments of the present application provide a computer program product. When the computer program product is run on an electronic device, the electronic device is caused to implement the steps in any of the above method embodiments.

[0128] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. According to such understanding, the computer program 32 can be used to instruct the related hardware to complete all or part of the processes in the above-mentioned embodiments. The computer program 32 can be stored in a computer readable storage medium, and the computer program 32 can implement the steps of each method embodiment described above when executed by the processor 30. The computer program 32 includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the terminal, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunications signal.

[0129] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0130] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0131] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are only schematic. The division of the modules or units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0133] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for identifying anomalies in urban facilities, characterized in that, include: Acquire an image to be identified, the image including: urban facilities; The image is input into a pre-established target detection model to determine the anomaly category of the urban facilities. The target detection model includes a backbone network, a neck network, and a detection head connected in sequence. The neck network is used to output feature information of different scales from the backbone network to the corresponding detection head. The backbone network includes a residual focusing convolution module and a multi-scale fusion attention mechanism module. The residual focusing convolution module is used to extract and fuse feature information of different scales of the image. The multi-scale fusion attention mechanism module is used to perform multi-scale convolution fusion operation on the feature information of the image.

2. The method according to claim 1, characterized in that, The residual focusing convolution module includes: a first residual focusing convolution module, a second residual focusing convolution module, a third residual focusing convolution module, and a fourth residual focusing convolution module. The multi-scale fusion attention mechanism module includes: a first multi-scale fusion attention mechanism module, a second multi-scale fusion attention mechanism module, and a third multi-scale fusion attention mechanism module. The backbone network further includes: a convolution module, a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, and a spatial pyramid pooling module. The convolution module is used for input images. The differential focusing convolution module, the first multi-scale fusion attention mechanism module, the third CBS module, the third residual focusing convolution module, the second multi-scale fusion attention mechanism module, the fourth CBS module, the fourth residual focusing convolution module, the third multi-scale fusion attention mechanism module, and the spatial pyramid pooling module are connected in sequence. The first multi-scale fusion attention mechanism module is used to output feature information at the first scale to the neck network of the target detection model, the second multi-scale fusion attention mechanism module is used to output feature information at the second scale to the neck network, and the spatial pyramid pooling module is used to output feature information at the third scale to the neck network.

3. The method according to claim 2, characterized in that, The residual focusing convolution module includes: a residual focusing convolution block, a fifth CBS module, and a sixth CBS module. The fifth CBS module is used to provide feature information of different scales to the residual focusing convolution block. The residual focusing convolution block is used to extract features from the feature information of different scales, perform nonlinear transformation on the extracted feature information, maintain the gradient flow of the target detection model, and input the processed feature information into the sixth CBS module.

4. The method according to claim 3, characterized in that, The residual focusing convolutional block includes: a first depthwise convolution of the size of a first convolutional kernel, a second depthwise convolution of the size of the first convolutional kernel, a LayerNorm layer, a convolution of the size of a second convolutional kernel, a GELU unit, and a convolution of the size of a second convolutional kernel, connected in sequence. The input of the first depthwise convolution forms a residual connection with the first depthwise convolution, and the input of the second depthwise convolution forms a residual connection with the second depthwise convolution. The GELU unit is used to introduce nonlinearity, and the convolution of the size of the second convolutional kernel is used to integrate feature information and adjust the number of channels.

5. The method according to claim 3, characterized in that, The multi-scale fusion attention mechanism module includes: a multi-scale convolution module and an SE module connected in sequence. The multi-scale convolution module is used to obtain feature information at different scales, concatenate the feature information at different scales, and output the concatenated feature information. The SE module is used to extract attention weights from the concatenated feature information, determine the shared channel attention vector based on the attention weights, recalibrate the shared channel attention vector, apply the recalibrated attention vector to the multi-scale feature map, and output the result.

6. The method according to claim 2, characterized in that, Different scales of feature information correspond to different detection heads. Each detection head includes: a REG module, a first convolutional shared BN separation layer, a second convolutional shared BN separation layer, and a seventh CBS module. The input of the REG module is the output of the neck network, and the output of the REG module is the input of the first convolutional shared BN separation layer. The output of the first convolutional shared BN separation layer is the input of the bounding box regression loss function. The input of the seventh CBS module is the output of the neck network, and the output of the seventh CBS module is the input of the second convolutional shared BN separation layer. The output of the second convolutional shared BN separation layer is the input of the classification loss function. The first and second convolutional shared BN separation layers are used to share convolutional weights between different layers.

7. The method according to claim 6, characterized in that, The neck network includes: a first concat module, a second concat module, a third concat module, a fourth concat module, a fifth residual focusing convolution module, a sixth residual focusing convolution module, a seventh residual focusing convolution module, an eighth residual focusing convolution module, an eighth CBS module, a ninth CBS module, a first upsampling module, and a second upsampling module. The input of the first concat module is the output of the first multi-scale fusion attention mechanism module and the first upsampling module. The output of the first concat module is the input of the fifth residual focusing convolution module. The output of the fifth residual focusing convolution module is the input of the eighth CBS module and the detection head. The input of the second concat module is the output of the second upsampling module and the second multi-scale fusion attention mechanism module. The output of the second concat module is the input of the sixth residual focusing convolution module. The output of the sixth residual focusing convolution module is the input of the first upsampling module and the third concat module. The input of the third concat module is the output of the eighth CBS module. The output of the third concat module is the input of the seventh residual focusing convolution module. The output of the seventh residual focusing convolution module is the input of the detection head and the ninth CBS module. The input of the fourth concat module is the output of the spatial pyramid pooling module and the ninth CBS module. The input of the second upsampling module is the output of the spatial pyramid pooling module. The output of the fourth concat module is the input of the eighth residual focusing convolution module. The output of the eighth residual focusing convolution module is the input of the detection head.

8. A device for identifying anomalies in urban facilities, characterized in that, include: The acquisition module is used to acquire an image to be identified, the image including: urban facilities; A determination module is used to input the image into a pre-established target detection model to determine the anomaly category of the urban facilities. The target detection model includes a backbone network, a neck network, and a detection head connected in sequence. The neck network is used to output feature information of different scales from the backbone network to the corresponding detection head. The backbone network includes a residual focusing convolution module and a multi-scale fusion attention mechanism module. The residual focusing convolution module is used to extract and fuse feature information of different scales of the image. The multi-scale fusion attention mechanism module is used to perform multi-scale convolution fusion operation on the feature information of the image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images