Urban facility anomaly detection method and device based on multi-modal model, electronic equipment and program product

Abstract

The invention discloses an urban facility anomaly detection method and device based on a multi-modal model, electronic equipment and a computer program product. The detection method is realized based on a trained anomaly detection model. According to the anomaly detection model, a TgVL module is arranged in a neck network, the TgVL module enhances image features through a VFE branch and learns the weight of the image features under text guidance through a TgV branch, attention is given to the enhanced image features through the weight, feature fusion of two modals can be promoted, and the problem that small sample learning is difficult is effectively solved. Based on this, the multi-modal fusion in combination with the TVF detection head, the text semantic information and the spatial features of the image can improve the understanding of a complex scene, and the detection result can be determined more accurately and efficiently. Besides, an AGLPU module is arranged in the backbone network and / or the TgVL module, local features and global features can be extracted by using an LPU branch and a GPU branch, and adaptive weighted fusion is performed on the local features and the global features, so that the detection precision of the urban facility anomaly is further improved.

Description

Technical Field

[0001] The present application belongs to the field of smart city technology, and in particular relates to a method for detecting anomalies of urban facilities based on a multimodal model, an apparatus for detecting anomalies of urban facilities based on a multimodal model, an electronic device and a computer program product. Background Art

[0002] As the global urbanization process continues to accelerate, the scale and complexity of urban infrastructure are growing, which requires urban management to have a higher level of intelligence to ensure the efficient operation and sustainable development of the city. Various abnormal conditions of facilities will inevitably occur in the daily operation of the city, such as damage to various public safety facilities. If these abnormal conditions of urban facilities cannot be quickly detected and effectively handled, it will not only reduce the efficiency of urban management, but may also directly reduce the quality of life of citizens and even threaten their safety.

[0003] Due to the wide variety and complexity of urban facility anomalies, some sample data is difficult to obtain, resulting in a limited number of samples and imbalanced data of various types. Using such samples to train models may cause the model to be biased towards common anomalies, making it difficult to effectively detect rare anomalies, thereby affecting its generalization ability and accuracy. Summary of the invention

[0004] The present application provides a method for detecting anomalies of urban facilities, a device for detecting anomalies of urban facilities, an electronic device and a computer program product, which can improve the generalization ability and accuracy of anomaly detection models.

[0005] In a first aspect, the present application provides a method for detecting anomalies of urban facilities based on a multimodal model, comprising: The backbone network of the pre-trained anomaly detection model extracts features from the image to be detected to obtain image features; the image to be detected includes urban facilities; The neck network based on the abnormality detection model fuses the image features to obtain the first fused feature; The detection head based on the anomaly detection model detects the first fusion feature to obtain a detection result of the anomaly of the urban facilities in the image to be detected; The neck network is provided with a TgVL module, which is used to fuse the input first image feature and the text feature of the preset guide text to obtain the second fusion feature, and the first fusion feature is obtained based on the second fusion feature.

[0006] Furthermore, the TgVL module includes an image feature enhancement branch, a text-guided image branch, and a first fusion layer; for the first image feature and the text feature: Processing the first image feature through the image feature enhancement branch to obtain an enhanced image feature; The text-guided image branch sequentially performs summation and weighted activation operations on the preprocessed first image features and text features to obtain guided learning weights; The enhanced image feature and the guided learning weight are multiplied by the first fusion layer and a convolution operation is performed to obtain a second fusion feature corresponding to the first image feature and the text feature.

[0007] Furthermore, the text-guided image branch includes an image feature preprocessing sub-branch, a text feature preprocessing sub-branch, an Einstein summation layer, and a MaxSigmoid activation function layer; the text-guided image branch sequentially performs an Einstein summation operation and a weighted activation operation on the preprocessed first image feature and the text feature to obtain a guided learning weight, including: Performing convolution operations and reshaping operations on the first image feature in sequence through the image feature preprocessing sub-branch to obtain a preprocessed first image feature; Through the text feature preprocessing sub-branch, linear operations and reshaping operations are performed on the text features in sequence to obtain preprocessed text features; Performing an Einstein sum operation on the preprocessed first image feature and the preprocessed first image feature through an Einstein sum layer to obtain a third fusion feature; The MaxSigmoid activation function layer performs a MaxSigmoid function activation operation on the third fusion feature to obtain the guided learning weight.

[0008] Furthermore, the detection head includes a regression branch, a classification branch and a splicing layer. The detection head based on the anomaly detection model detects the first fusion feature to obtain a detection result of anomalies of urban facilities in the image to be detected, including: Performing a deep convolution operation on the first fusion feature through the regression branch to obtain a regression result; The classification branch sequentially performs summing and scaling operations on the classified first fusion features and the preprocessed text features to obtain a classification result; The regression results and classification results are spliced ​​through the splicing layer and activated based on the preset activation function to obtain a visual detection result.

[0009] Furthermore, an AGLPU module is provided in the backbone network and / or the TgVL module, and the AGLPU module is used to extract and fuse local features and global features of the input second image features to obtain a fourth fused feature.

[0010] Furthermore, the AGLPU module includes a convolution layer, a local perception unit branch, a global perception unit branch, and a second fusion layer. For each second image feature: Performing a convolution operation on the second image feature through a convolution layer to obtain a convolution feature; The convolution feature is extracted through the local perception unit branch to obtain the local feature; The convolution feature is extracted through the global perception unit to obtain the global feature; The local features and the global features are adaptively weighted fused through the second fusion layer to obtain the fourth fusion feature.

[0011] Furthermore, the local perception unit branch is composed of two bottleneck structures with residual connections connected in series; the convolutional features are extracted through the local perception unit branch to obtain local features; Perform convolution operation, depth convolution operation and convolution operation on the convolution feature in sequence to obtain the local first sub-feature; Concatenate the convolution feature with the local first sub-feature to obtain the first concatenated feature; Performing a convolution operation, a depthwise convolution operation, and a convolution operation on the first concatenated feature in sequence to obtain a local second sub-feature; The first splicing feature is spliced ​​with the local second sub-feature to obtain a local feature.

[0012] Furthermore, the global perception unit branch is composed of a LMHSA structure with a residual connection and a FFN structure with a residual connection in series; the convolutional features are extracted through the global perception unit to obtain global features, including: Perform layer normalization, multi-head self-attention and regularization operations on the convolutional features in sequence to obtain the global first sub-feature; Concatenate the convolution feature with the global first sub-feature to obtain the second concatenated feature; Performing a layer normalization operation, a feature transformation operation, and a regularization operation on the second concatenated feature to obtain a global second sub-feature; The second concatenated feature is concatenated with the global second sub-feature to obtain a global feature.

[0013] In a second aspect, the present application provides an urban facility anomaly detection device based on a multimodal model, comprising: A feature extraction module is used to extract features from the image to be detected based on the backbone network of the pre-trained anomaly detection model to obtain image features; the image to be detected includes urban facilities; A feature fusion module, used for fusing image features based on the neck network of the anomaly detection model to obtain a first fused feature; A detection module, used for detecting the first fusion feature based on the detection head of the anomaly detection model to obtain a detection result of the anomaly of the urban facilities in the image to be detected; The neck network is provided with a TgVL module, which is used to fuse the input first image feature and the text feature of the preset guide text to obtain the second fusion feature, and the first fusion feature is obtained based on the second fusion feature.

[0014] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0016] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0017] The first aspect of the present application has the following beneficial effects compared to the prior art: the present application sets a text-guided visual learning module (TgVL) in the neck network, which can enhance the expressiveness of effective image features under the guidance of text by fully integrating the features of the two modalities of image and text. That is, the TgVL module guides image feature learning by using text information, so that the model can extract more expressive and richer effective features, namely the second fusion feature. The first fusion feature determined based on the second fusion feature is used for urban facility anomaly detection, which can enable the anomaly detection model to obtain stronger generalization ability and higher accuracy.

[0018] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a schematic diagram of abnormalities of some urban facilities provided in the embodiment of the present application; Figure 2Schematic diagram of the network structure of the OV-MFS-YOLO network provided in the embodiment of the present application; Figure 3 is a schematic diagram of angle loss provided in an embodiment of the present application; Figure 4 is a schematic diagram of distance loss provided in an embodiment of the present application; Figure 5 is a schematic flow chart of a method for detecting anomalies of urban facilities based on a multimodal model provided in an embodiment of the present application; Figure 6 is a schematic diagram of the network structure of the TgVL module provided in an embodiment of the present application; Figure 7 is a schematic diagram of the network structure of the TVF detection head provided in an embodiment of the present application; Figure 8 is a schematic diagram of the network structure of the AGLPU module provided in an embodiment of the present application; Fig. 9 It is a structural schematic diagram of an urban facility anomaly detection device based on a multimodal model provided in an embodiment of the present application; Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0022] There are many types of urban facilities, including roads, manhole covers, guardrails, walls, telescopic devices, drainage facilities, isolation fences, anti-collision columns, vehicle stoppers, etc. Based on the abnormal conditions that occur in these urban facilities, the types and manifestations of abnormalities are also more diverse and complex.

[0023] This application focuses on identifying and classifying 152 abnormal conditions of facilities in the urban environment, with the aim of comprehensively and accurately responding to the various challenges faced by urban management. These abnormal conditions can be divided into four categories: the first is the cleanliness category, which covers 30 situations such as scattered garbage, overflowing garbage cans, chaotic placement of garbage cans, and blocked drainage facilities; the second is the order category, which includes 14 situations such as random parking of non-motor vehicles, dumping of non-motor vehicles, disorderly stalls (mobile vendors), and road occupation; the third category is the neatness category, which involves 83 situations such as stains on the facades of street buildings, damaged or missing manhole covers, damaged or missing store signs, and damaged anti-collision barrels; the last is the greening and beauty category, which includes 25 situations such as greening garbage, dirty and messy roads caused by construction, fallen trees, and unclean flower pots, flower boxes, and flower beds.

[0024] Figure 1 Some representative pictures of actual cases are shown to more clearly demonstrate the actual conditions of various abnormal situations.

[0025] In order to improve the detection accuracy of the anomaly detection model for abnormal urban facilities, this application constructs a professional dataset called Abnormal City Facilities (ACF), which focuses on collecting and analyzing abnormal urban facilities. The uniqueness of this dataset is that all image data comes from on-site collection in a real vehicle environment, ensuring the authenticity and diversity of the data.

[0026] The number of images in the dataset is closely related to the performance of the trained anomaly detection model. Assuming that the dataset contains 1 million high-quality images, to ensure the effectiveness and generalization of the algorithm model, the dataset will follow the classic 8:2 division ratio: 800,000 images are used as the training set to train and optimize the anomaly detection model; the remaining 200,000 images are used as the validation set to verify the model's detection capability, accuracy, and generalization ability, ensuring the model's performance on unknown data.

[0027] Optionally, in order to comprehensively consider the anomalies of urban facilities in different environments, the collected images cover a variety of lighting conditions (such as strong light, weak light, backlight, etc.), different weather conditions (such as sunny days, rainy days, snowy days, foggy days, etc.), and road and environmental characteristics of most cities in the country. This diversified data collection method ensures the authenticity, richness and comprehensiveness of the data set, providing a more accurate and reliable foundation for related research and applications.

[0028] The urban facility anomaly detection focused on in this application is different from existing research. It does not only focus on frequent or common urban facility anomalies, but comprehensively covers various facility anomalies in the urban environment. Therefore, when performing urban facility anomaly detection, there is an important challenge - small sample learning, which is mainly reflected in the following three aspects: (1) Diversity and complexity: There are many types of urban facility anomalies with different manifestations, ranging from minor garbage spillage to serious road collapse. Each anomaly has its own unique characteristics and manifestations. In addition, the same type of anomaly may show significant differences in different environments and at different times, which increases the difficulty of anomaly detection.

[0029] (2) Data acquisition and labeling costs: Abnormal events are usually sudden and uncertain, which makes it impossible to collect large-scale, systematic data through conventional means. At the same time, accurate labeling of abnormal images requires a lot of manpower and time, which increases the cost and complexity of data labeling.

[0030] (3) Data imbalance: In practical applications, some types of abnormal events may be more common, while other types are relatively rare. Data imbalance can cause the model to be biased during training, thus affecting its generalization ability and causing the model to perform poorly in detecting rare types of abnormalities.

[0031] These factors require that when performing anomaly detection in urban facilities, the model needs to cope with complex and changing data environments while overcoming the challenges brought about by small sample learning.

[0032] In order to solve this problem, in addition to collecting high-quality training data based on the above-mentioned training sample optimization method, this application also improves the network structure of the model and proposes an anomaly detection model, which includes a backbone network, a neck network and a detection head. Specifically, the backbone network is used to extract features from the input image to obtain image features; the neck network is used to fuse image features to obtain fused features; the detection head is used to detect the fused features to obtain the detection results of the anomalies of urban facilities in the image to be processed.

[0033] Specifically, in order to overcome the challenges brought by small sample learning in the process of urban facility anomaly detection, this application sets a text-guided visual learning module (TgVL) in the neck network, which can enhance the expressiveness of effective image features under the guidance of text by fully integrating the features of the two modalities of image and text. That is, the TgVL module uses text information to guide image feature learning, so that the model can extract more expressive and richer effective features, namely the second fusion feature. Using the first fusion feature determined based on the second fusion feature for urban facility anomaly detection can enable the anomaly detection model to obtain stronger generalization ability and higher accuracy.

[0034] In some embodiments, the TgVL module may include a Vision Feature Enhancement (VFE) branch, a Text-guided Vision (TgV) branch, and a first fusion layer. The first fusion layer is a Text-Vision Feature Fusion (TVFF) layer. The VFE branch is used to process the first image feature to enhance the feature expression capability of the first image feature and obtain enhanced image features; the TgV branch guides the extraction of effective features in the first image feature through text features to learn guided learning weights; the TVFF layer performs attention operations on the enhanced image features based on the guided learning weights, which can promote the feature fusion of the two modalities and effectively solve the problem of small sample learning difficulties.

[0035] In some embodiments, preset text features can be extracted based on the CLIP (Contrastive Language-Image Pretraining) model to obtain a semantically rich text representation with contrastive learning capabilities. The text data may include a description of urban facility anomalies, which is used to provide prior knowledge for the anomaly detection model to help it identify unseen categories of urban facility anomalies in the urban facility anomaly detection task, and achieve more accurate cross-modal understanding and generalization capabilities.

[0036] In some embodiments, in order to improve the performance of the anomaly detection model, the TgV branch can be implemented through an image feature preprocessing sub-branch, a text feature preprocessing sub-branch, an Einstein summation layer, and a MaxSigmoid activation function layer. After the first image feature and the text feature are preprocessed, they can be effectively fused through the Einstein summation layer to combine the feature information of the two modalities; then, the fused features are nonlinearly activated with the help of the MaxSigmoid activation function layer, which can further emphasize the key information and generate high-precision guided learning weights. As a result, the TgV branch can enhance the feature expression ability of the anomaly detection model by learning high-precision guided learning weights, thereby improving its accuracy and generalization ability.

[0037] In some embodiments, in order to further enhance the advantages of text-guided image learning, make full use of multimodal information fusion to improve the ability of the anomaly detection model to understand complex scenes, and thus improve the detection accuracy of the model, the detection head may include a regression branch, a classification branch, and a concatenation layer. Among them, the regression branch can enhance and refine the first fusion feature of the input to obtain a regression result with higher accuracy, and the classification branch combines the first fusion feature with the text feature. Regularization technology is used in the combination process to ensure the stability and generalization ability of the model, and the Einstein summation convention is introduced to efficiently perform operations between features, and then the log_scale coefficient is applied to scale the result to achieve more refined feature fusion and obtain a classification result. Finally, the outputs of the regression branch and the classification branch are merged together through a concatenation operation. At this time, the concatenation result obtained is actually still a feature. In order to present a visual detection result, it can be activated by a preset activation function to form a final visual detection result. The detection result is still an image, in which there are annotated boxes of urban facility anomalies and categories of urban facility anomalies.

[0038] The improved detection head can be called the Text-Vision Feature (TVF) detection head, which can combine the semantic information of the text and the spatial features of the image to perform multimodal information fusion to improve the ability to understand complex scenes, thereby improving the detection accuracy of urban facility anomalies. At the same time, by comprehensively analyzing data from different sources, it can more accurately detect the area corresponding to the text in the image and reduce the probability of misjudgment. It can be considered that the anomaly detection model using the TVF detection head has significant advantages in multiple dimensions such as accuracy, efficiency, flexibility and application scope.

[0039] When detecting anomalies in urban facilities, in addition to the problem of small sample learning, there are also semantic challenges in scene understanding, as follows: (1) Fuzziness of classification boundaries: In urban infrastructure anomaly detection, the boundaries between many categories are not clear. For example, whether a slight bend in a guardrail should be considered damage or normal wear and tear requires the anomaly detection model to have a high judgment accuracy.

[0040] (2) Contextual dependency: The occurrence of abnormal events is often closely related to a specific environmental background. When detecting anomalies, the model needs to fully consider the surrounding scene information, such as weather, lighting, traffic conditions, etc., which undoubtedly increases the complexity of detection.

[0041] (3) Ambiguity in semantic understanding: There may be different interpretations and understandings of certain abnormal phenomena. This semantic ambiguity will cause the model to be confused when processing related data, thus affecting the accuracy of the detection results.

[0042] In order to overcome the semantic challenges of scene understanding to a certain extent, the present application also improves the backbone network and / or TgVL module, that is, an Adaptive Global-Local Perception Unit (AGLPU) module is set in the backbone network and / or TgVL module. The AGLPU module achieves more accurate scene understanding by respectively extracting local features and global semantic information of the input second image features and fusing them.

[0043] Specifically, the AGLPU module mainly performs two parts of work: local feature extraction and global semantic understanding. The local feature extraction part focuses on the local area in the image, such as road cracks, damaged manhole covers and other specific anomalies, which can efficiently capture detailed information; while the global semantic understanding part focuses on the contextual information of the entire scene, including weather, lighting changes and surrounding environmental characteristics, in order to more comprehensively understand the background of abnormal events.

[0044] By integrating these two features, the AGLPU module can not only improve the model's sensitivity to detailed anomalies, but also more accurately judge the nature of anomalies in complex backgrounds. The introduction of this module helps to enhance the semantic perception of the anomaly detection model, thereby effectively overcoming problems such as fuzziness of classification boundaries, context dependency, and ambiguity in semantic understanding.

[0045] This combination of global and local perception not only improves the judgment accuracy of the anomaly detection model, but also further optimizes the scene understanding capability, enabling the anomaly detection model to more accurately and stably detect various facility anomalies in complex urban environments.

[0046] Convolutional neural networks (CNNs) effectively extract local features such as edges and textures through convolutional layers, and reduce the number of parameters and the risk of overfitting through parameter sharing mechanisms. Pooling layers help reduce computational complexity and memory usage, while translation invariance enhances its generalization ability. However, the limitations of CNNs are also obvious: their ability to capture global information is limited, they cannot handle long-distance dependencies, and they have a large demand for training data, making them prone to overfitting when data is insufficient. In addition, deep network structures increase computational complexity.

[0047] The advantage of Transformer is that its self-attention mechanism can capture the dependencies between different positions in the sequence, achieve global context awareness, and effectively capture long-distance dependencies, which is particularly suitable for sequence data such as natural language. Transformer improves training and reasoning efficiency by processing sequences in parallel. However, the disadvantage of Transformer is that it has high computational complexity, requires a lot of computing resources and memory, and usually relies on a lot of data and computing resources for pre-training and fine-tuning. The complexity of the model also leads to its poor interpretability, which brings challenges to tasks that require interpretability.

[0048] Therefore, when designing the structure of the AGLPU module, the advantages of CNN and Transformer are combined to give full play to their complementary characteristics. CNN is good at extracting local features, while Transformer captures global information and long-distance dependencies through the self-attention mechanism. Combining the two can achieve better performance in the task of urban facility anomaly detection.

[0049] In some embodiments, the AGLPU module includes a convolution layer, a local perception unit (LPU) branch, a global perception unit (GPU) branch, and a second fusion layer. The convolution layer is responsible for integrating input features and achieving channel dimensionality reduction, providing a more compact feature representation for subsequent processing. The LPU branch focuses on extracting local detail features in the image, such as specific abnormal locations and morphological details, and can capture fine information. The GPU branch is responsible for extracting global semantic information and enhancing the comprehensive understanding of facility anomalies by considering the overall environmental background (such as lighting, weather, surrounding facilities, etc.). Finally, the second fusion layer effectively fuses local features and global features through an adaptive weighting mechanism, thereby improving the detection capability of the model in complex scenes.

[0050] Among them, the LPU branch uses the CNN architecture to extract local detail features, and the GPU branch uses the Transformer structure to extract global semantic information. However, in traditional methods, the two are usually simply connected in series or in parallel and directly summed. This equal-weight feature calculation method is not conducive to the effective fusion of features. The reason is that the detail features are more prominent in the early stage of the network. At this time, the semantic information should be given a higher weight to make up for the lack of detail features; as the network goes deeper, high-level semantic information gradually dominates, and the detail information gradually weakens. At this time, the weight of the detail features needs to be increased. If this difference is ignored, it will be difficult to balance local detail features and global semantic information.

[0051] Therefore, this embodiment introduces adaptive weights in the fusion process. These weight coefficients can be dynamically adjusted according to the feedback during the training process, so as to achieve a more flexible fusion of local detail features and global semantic information, and improve the quality of feature representation and the performance of subsequent tasks.

[0052] In this embodiment, the AGLPU module extracts local features and global semantic information through the LPU branch and the GPU branch respectively, and adaptively weights and fuses the two. The module can flexibly adjust the proportion and structure of CNN and Transformer according to specific task requirements, thereby building a more efficient and powerful image processing model. That is, the AGLPU module can effectively solve the problems of fuzzy classification boundaries, ambiguous semantic understanding, and high context dependence in urban facility anomaly detection, thereby improving the accuracy and robustness of the model.

[0053] In some embodiments, the LPU branch consists of two bottleneck structures with residual connections connected in series to extract local detail features. Each bottleneck structure includes multiple convolutional layers to capture subtle changes in the image, such as local information such as edges, textures, and shapes. The introduction of residual connections ensures the smooth transmission of information flow, avoids the gradient vanishing problem, and also accelerates the training process of the network. In this way, the LPU branch can efficiently extract key local features from the image, providing strong support for subsequent anomaly detection.

[0054] In some embodiments, the GPU branch's local multi-head self-attention (LMHSA) structure with residual connections and the feedforward neural network (FFN) structure with residual connections are connected in series to form an LMHSA structure, which is used to capture the global semantic information of the input features, effectively model the relationship between different positions through the multi-head self-attention mechanism, and enhance the model's perception of long-distance dependencies. The FFN structure further strengthens the nonlinear transformation of global features and improves the expression of semantic information. The introduction of residual connections not only helps alleviate the gradient vanishing problem in deep network training, but also improves the stability and convergence speed of the model. Through this series structure, the GPU branch can efficiently extract global semantic information, and after fusing with the local features of the LPU branch, enhance the overall performance of the model.

[0055] In some embodiments, the AGLPU module can be set in the VEE branch of the TgVL module to optimize and improve the quality of image features extracted from the image, thereby enhancing the expressiveness and detection accuracy of the features.

[0056] In some embodiments, the anomaly detection model sets a TgVL module in the neck network. The TgVL module enhances image features through the VFE branch, and learns the weights of image features under text guidance through the TgV branch. Combined with the TVF detection head, the multimodal fusion of text semantic information and image spatial features can enhance the understanding of complex scenes and determine the detection results more accurately and efficiently. In addition, the AGLPU module set in the backbone network and the TgVL module can use the LPU branch and the GPU branch to extract local features and global features, and perform adaptive weighted fusion on the two to further improve the accuracy of urban facility anomaly detection.

[0057] For example, if the YOLO series model is used as the basic model, the basic model is improved by the TgVL module, the TVF detection head, and the AGLPU module, and the obtained model can be referred to Figure 2 , this model can be called Open-Vocabulary Multimodal Facility Anomaly DetectionAlgorithm based on YOLO Network (OV-MFS-YOLO network).

[0058] In some embodiments, the above-mentioned anomaly detection model may adopt a two-stage training method combining rapid convergence and fine optimization during training. A total of N rounds are set in the training process, of which the first n rounds are used for rapid convergence. The main purpose is to quickly approach the optimal solution area through a larger learning rate to accelerate the initial training of the model. This stage focuses on improving the overall performance of the model, so that the performance of the model on the training set is rapidly improved. The remaining rounds will enter the fine optimization stage, using a smaller learning rate to further adjust the parameters of the model to achieve more accurate parameter adjustment and performance improvement. At this stage, the model can better capture subtle differences in anomaly detection, reduce the risk of overfitting, and improve the generalization ability on the validation set. Through this step-by-step optimization method, the model can take into account both training efficiency and high precision, ensuring the best results in complex urban facility anomaly detection tasks.

[0059] In some embodiments, the loss function of the above anomaly detection model mainly consists of two parts: regression loss and classification loss. When a two-stage training method is adopted, the classification losses of both stages are binary cross entropy loss (BCE Loss), which is used to determine the specific category in the anchor frame.

[0060] Among them, the regression loss of the first stage is the complete intersection over union loss (CIOU Loss) and the dynamic category loss (DFL loss), which are used to calculate the error between the predicted bounding box and the ground truth box. It is used for preliminary prediction, and the positive and negative samples are matched using the Adaptive Training Sample Selection (ATSS) static matching strategy.

[0061] The regression loss in the second stage is the Skew Intersection over Union Loss (SIoU Loss) and DFL loss, which are used to fine-tune the prediction box. The matching strategy between positive and negative samples adopts DFL dynamic matching.

[0062] Specifically, the formula of BCE Loss is as follows:

[0063] in, L BCE is BCE Loss, n is the number of image samples of all city facilities anomalies, y i For the i The true category of the image samples, is the predicted category of the i-th image sample.

[0064] CIOU Loss is an improved version of commonly used loss functions such as L1, L2, Intersection over Union (IoU) and GIoU. It adds a penalty term for aspect ratio. When the center points of the predicted border and the true border coincide, it can better distinguish errors in different situations and has scale invariance. The formula is as follows:

[0065] in, L CIoU For CIOU Loss, b and b gt They are the center points of the predicted bounding box and the true bounding box respectively; ρ is the Euclidean distance between the predicted bounding box and the true bounding box; c The distance between the diagonal of the closed area of ​​the predicted border and the true border; vTo ensure the consistency of the relative proportions between the predicted bounding box and the true bounding box, IoU It is the intersection-over-union ratio of the predicted bounding box and the true bounding box; α is the weight coefficient; w and w gt are the width of the predicted border and the true border respectively; h and h gt are the heights of the predicted bounding box and the true bounding box respectively.

[0066] However, L CIoU The direction between the real box and the predicted box is not taken into account, which often causes the network to "wander around" during training, converges slowly and has low efficiency, which easily reduces the robustness of the model.

[0067] To solve this problem, the SIoU Loss used in the second stage takes into account the vector angle between the real border and the predicted border and redefines the penalty index. It specifically includes four parts: angle loss (Angle cost), distance loss (Distance cost), shape loss (Shape cost), and IoU loss (IoU cost).

[0068] 1. Angle loss See also Figure 3 , the formula for angle loss Λ can be written as follows:

[0069] in, is the center point coordinate of the predicted bounding box, is the center point coordinate of the real border, σ is the center point distance between the real border and the predicted border, d h is the height difference between the center point of the real box and the predicted box, d w is the height difference between the center point of the real bounding box and the predicted bounding box.

[0070] 2. Distance loss See also Figure 4 , the formula for distance loss Δ can be written as follows:

[0071] c h and c w are the width and height of the minimum enclosing rectangle of the real border and the predicted border, respectively.

[0072] 3. Shape loss The shape loss Ω can be written as follows:

[0073] w , h , w gt and h gt are the width and height of the predicted border and the true border respectively, and θ is a hyperparameter that can control the degree of attention paid to the shape loss.

[0074] L SIoU Combining the above three aspects of loss can improve the comprehensiveness of each part of the loss, making the convergence speed faster and the accuracy higher, which is better than L CioU . L SIoU It can be written as follows:

[0075] The formula for DFL loss is as follows:

[0076] in, S i is the cross entropy loss between the real border and the predicted border on the left, S i+1 It is the cross entropy loss between the real box and the predicted box on the right.

[0077] The formula for the total loss function of the first stage of training of the anomaly detection model is as follows:

[0078] Among them, λ1 and λ2 are the classification and regression balance coefficients of the first stage.

[0079] The formula for the total loss function of the second stage of training of the anomaly detection model is as follows:

[0080] λ3 and λ4 are the classification and regression balance coefficients in the second stage.

[0081] Based on the two total loss functions of the anomaly detection model, the anomaly detection model described in any of the above embodiments is trained in two stages, which can improve the model convergence speed and thus improve the efficiency of model training. By continuously iterating the anomaly detection model, the anomaly detection model can detect anomalies of various urban facilities better and more stably, and finally a robust version can be selected from multiple versions of the two-stage convergence as the trained anomaly detection model.

[0082] In some embodiments, in order to comprehensively and accurately measure the performance of each version of the anomaly detection model, it can be evaluated through a validation set. Specifically, the performance of the anomaly detection model on the validation set can be evaluated according to preset conditions. For example, these preset conditions may include performance indicators such as the intersection-over-union ratio, detection accuracy, and recall rate of the anomaly detection model for urban facility anomalies.

[0083] IoU is an indicator to evaluate the degree of overlap between the predicted border and the true border. It calculates the ratio of the intersection and union of the predicted border and the true border. It plays a key role in determining whether it is a correct detection. The calculation formula of IoU can be written as:

[0084] Precision (p), also known as the precision rate, refers to the proportion of correct positive predictions to all positive predictions, as shown in the formula:

[0085] Recall (R), also known as recall rate, refers to the proportion of correctly predicted positives to all actual positives, as shown in the formula:

[0086] Average Precision (AP) is calculated from precision and recall. A line graph of precision is drawn based on the recall value, and the area under the line is calculated, as shown in the formula:

[0087] The mean average precision (mAP) refers to the average value of the average precision AP of C different urban facility anomaly categories, as shown in the formula:

[0088] That is to say, after verifying each version of the anomaly detection model through the validation set, the anomaly detection model of each version can be comprehensively evaluated based on the above indicators, so as to determine the anomaly detection model with the best performance from each version as the trained anomaly detection model.

[0089] Based on the anomaly detection model of the previous embodiment, the present application proposes an urban facility anomaly detection method based on a multimodal model.

[0090] The urban facility anomaly detection method based on a multimodal model provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc., and the electronic device is equipped with a trained facility detection module. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0091] In order to illustrate the technical solution proposed in this application, each embodiment will be described below with an electronic device as the execution subject.

[0092] Figure 5 A schematic flow chart of a method for detecting anomalies of urban facilities based on a multimodal model provided by the present application is shown, and the detection method comprises: Step 510: The electronic device extracts features of the image to be detected based on the backbone network of the pre-trained anomaly detection model to obtain image features. The image to be detected includes urban facilities.

[0093] Step 520: The electronic device fuses the image features based on the neck network of the abnormality detection model to obtain a first fused feature.

[0094] Step 530: The electronic device detects the first fusion feature based on the detection head of the anomaly detection model to obtain a detection result of the anomaly of the urban facilities in the image to be detected.

[0095] In this embodiment, the improved neck network benefits from the introduction of the TgVL module. The electronic device can use the TgVL module to fully fuse the features of the two modalities of image and text, which can enhance the expressiveness of effective image features under the guidance of text. That is, the TgVL module guides image feature learning by using text information, so that the model can extract more expressive and richer effective features, namely the second fusion feature. The first fusion feature determined based on the second fusion feature is used for urban facility anomaly detection, which can enable the anomaly detection model to obtain stronger generalization ability and higher accuracy, and effectively solve the problem of small sample learning in the process of urban facility anomaly detection.

[0096] In some embodiments, for a TgVL module including a VFE branch and a TgV branch, the electronic device may perform the following operations: Step A1: The electronic device processes the first image feature through the VFE branch to obtain an enhanced image feature.

[0097] Step A2: The electronic device sequentially performs a summing operation and a weighted activation operation on the preprocessed first image feature and the text feature through the TgV branch to obtain a guided learning weight.

[0098] Step A3: The electronic device multiplies the enhanced image feature and the guided learning weight through the TVFF layer and performs a convolution operation to obtain a second fusion feature corresponding to the first image feature and the text feature.

[0099] In this embodiment, the electronic device processes the first image feature through the VFE branch to enhance its feature expression capability and generate enhanced image features; uses the text feature to guide the extraction of effective information in the first image feature through the TgV branch, and learns the guided weights; and performs attention operations on the enhanced image features based on the guided learning weights through the TVFF layer to promote the fusion of image and text modal features, so that the introduction of the TgVL module can effectively alleviate the challenges brought by small sample learning.

[0100] In some embodiments, when the TgV branch can be implemented by an image feature preprocessing sub-branch, a text feature preprocessing sub-branch, an Einstein summation layer, and a MaxSigmoid activation function layer, the aforementioned step A2 specifically includes: Step A21: The electronic device sequentially performs convolution operation and reshaping operation on the first image feature through the image feature preprocessing sub-branch to obtain the preprocessed first image feature.

[0101] Step A22: The electronic device sequentially performs linear operations and reshaping operations on the text features through the text feature preprocessing sub-branch to obtain preprocessed text features.

[0102] Step A23: The electronic device performs an Einstein summation operation on the preprocessed first image feature and the preprocessed first image feature through an Einstein summation layer to obtain a third fusion feature.

[0103] Step A24: The electronic device performs a MaxSigmoid function activation operation on the third fusion feature through a MaxSigmoid activation function layer to obtain a guided learning weight.

[0104] The electronic device first performs preliminary dimensionality adjustment on the first image feature and text feature using the preset convolutional layer and linear layer through the TgV branch. Then, the shape of the adjusted first image feature and text feature is changed through the reshaping operation, and then the Einstein summation operation is performed to integrate these features. Finally, this branch outputs MaxSigmoid weights, which are multiplied with the output of the VFE branch to play the role of an attention mechanism, emphasizing key information and improving the performance of the model.

[0105] Among them, the convolution operation can be implemented by a convolution layer with a specified convolution kernel, such as a 3×3 convolution layer. The reshaping operation refers to the Reshape operation, which can adjust the dimension of the data by changing the shape of the tensor to adapt to the input requirements of different network layers. Through the Reshape operation, the model can flexibly convert the data format, thereby improving the adaptability and flexibility of the network structure.

[0106] Performing linear operations on text features means mapping each word or character to a high-dimensional vector space through an embedding layer to obtain a representation of the text unit. Next, these embedded vectors are processed using linear transformations (such as matrix multiplication), and a weight matrix is ​​usually used to adjust the feature dimensions to achieve feature compression or expansion. Finally, activation functions (such as ReLU, Sigmoid, etc.) are often used to perform nonlinear activation on the results of the linear transformation to enhance the expressive power of the model. This linear operation helps to effectively learn text features during training and capture the potential semantic relationships of the text.

[0107] In some embodiments, the VFE branch is mainly composed of two 1x1 convolutional layers and an AGLFU module. The first 1x1 convolution is used to reduce the channel dimension, and the second 1x1 convolution is used to increase the channel dimension. The goal of the AGLFU module is to optimize and improve the quality of image features extracted from the image, thereby enhancing the expressiveness of the features and the detection accuracy.

[0108] In some embodiments, the TVFF layer includes two 3x3 convolutional layers, whose main function is to perform deeper fusion and quality improvement of the input features. Through the processing of these two convolutional layers, the module can optimize the representation of features and enhance their discriminability and effectiveness.

[0109] For example, Figure 6 The specific structural diagram of the TgVL module (including the VFE branch, TgV branch and TVFF layer) is shown. Figure 6 The structure of the TgVL module in , assuming that the first input image feature is and text features , H, W and C represent the height, width and channel of the first image feature respectively; N represents the category of the text content, where N ranges from 1 to 152 (from the above description, it can be seen that there are 152 categories in total); E represents the encoding dimension of the text feature. Specifically, the TgVL module X V and X T The processing can be expressed as follows:

[0110] in, Indicates a convolution with a convolution kernel size of 1x1; Represents a convolution with a kernel size of 3x3; the Reshape operation represents the corresponding reshaping operation; Y V Represents enhanced high-quality image features; Linear represents a linear layer; einsum represents the Einstein summation operation; W TgV represents the weights of text-guided image learning; O Represents the output of the entire TgVL module.

[0111] In some embodiments, the TVF detection head includes a regression branch, a classification branch, and a concatenation layer, and step 530 specifically includes: Step B1: The electronic device performs a deep convolution operation on the first fusion feature through a regression branch to obtain a regression result.

[0112] Step B2: the electronic device sequentially performs a summing operation and a scaling operation on the classified first fusion feature and the preprocessed text feature through the classification branch to obtain a classification result; Step B3: The electronic device splices the regression result and the classification result through the splicing layer and activates them based on a preset activation function to obtain a visualized detection result.

[0113] For example, Figure 7 A schematic diagram of the network structure of the TVF detection head (including regression branch, classification branch and splicing layer) is shown.

[0114] Among them, the electronic device enhances and refines the first fused features of the input through two consecutive 3x3 deep convolutions (DWConv) in the regression branch. The deep convolution operation can effectively extract local features while reducing the amount of calculation and parameters by processing each channel independently in each convolution layer. This process helps to refine image features, enhance the model's sensitivity to details, and provide higher quality feature representation for subsequent anomaly detection.

[0115] The electronic device is also responsible for combining the processed first fusion features with the input text features through the classification branch. In this process, the first fusion feature is firstly further processed by deep convolution, and then regularization techniques (such as Normalize) are used to improve stability and ensure the generalization ability of the model. For text features, regularization techniques are directly used to ensure their consistency and reliability. In order to efficiently perform operations between features, the Einstein summation convention is introduced to optimize the calculation process. Finally, the log_scale coefficient is applied to scale the results to achieve more refined feature fusion, thereby improving the model's ability to process multimodal data and the accuracy of anomaly detection.

[0116] For the features obtained from the two branches, the electronic device first splices them and activates the splicing results through a preset activation function to form the final visual detection result. The detection result is still an image with an annotated box of the abnormal urban facilities and the category of the abnormal urban facilities.

[0117] In some embodiments, an AGLPU module is provided in the backbone network and / or the TgVL module, and the AGLPU module is used to extract and fuse local features and global features of the input second image features to obtain a fourth fused feature.

[0118] In some embodiments, in order to fully and comprehensively extract the local features and global features of the second image features, the AGLPU module may include a convolution layer, an LPU branch, a GPU branch, and a second fusion layer, wherein the electronic device implements the extraction of local detail features based on the CNN architecture through the LPU branch, and implements the extraction of global semantic information based on the Transformer structure through the GPU branch. The specific extraction steps are as follows: Step C1: The electronic device performs a convolution operation on the second image feature through a convolution layer to obtain a convolution feature.

[0119] Step C2: The electronic device extracts the convolution feature through the local perception unit branch to obtain the local feature.

[0120] Step C3: The electronic device extracts the convolution feature through the global perception unit to obtain the global feature.

[0121] Step C4: The electronic device performs adaptive weighted fusion on the local features and the global features through the second fusion layer to obtain a fourth fusion feature.

[0122] The convolution layer is mainly used for feature integration and channel dimension reduction, such as the 1×1 convolution layer. The convolution features output by the convolution layer can be input into the LPU branch and the GPU branch respectively to extract local detail features and global semantic information. In order to fully balance local features and global features, the electronic device introduces adaptive weights in the fusion process through the second fusion layer. These weight coefficients can be dynamically adjusted according to the feedback during the training process, thereby realizing the flexible fusion of local detail features and global semantic information, improving the quality of feature representation and the performance of subsequent tasks.

[0123] In this embodiment, the AGLPU module is based on the LPU branch and the GPU branch, which effectively solves the problems of fuzzy classification boundaries, ambiguous semantic understanding, and high context dependence in urban facility anomaly detection, and improves the accuracy and robustness of the model.

[0124] In some embodiments, the LPU branch is composed of two bottleneck structures with residual connections in series; the aforementioned step C2 specifically includes: Step C21: The electronic device sequentially performs a convolution operation, a depthwise convolution operation, and a convolution operation on the convolution feature to obtain a local first sub-feature.

[0125] Step C22: The electronic device concatenates the convolution feature with the local first sub-feature to obtain a first concatenated feature.

[0126] Step C23: The electronic device sequentially performs a convolution operation, a depthwise convolution operation, and a convolution operation on the first splicing feature to obtain a local second sub-feature.

[0127] Step C24: The electronic device splices the first splicing feature with the local second sub-feature to obtain a local feature.

[0128] The bottleneck structure with residual connection consists of a convolutional layer, a deep convolutional layer, and a convolutional layer connected in series. The electronic device compresses the channel and integrates the features through the first convolutional layer, extracts deep local features through the deep convolutional layer, and restores the channel through the last convolutional layer. The electronic device also alleviates the gradient vanishing problem in deep network training by introducing residual connections, thereby improving the stability and convergence speed of the model. By connecting two bottleneck structures with residual connections in series, the electronic device can fully extract the local detail information in the convolutional features, thereby enhancing the model's ability to express local features.

[0129] In some embodiments, the GPU branch includes a LMHSA structure with a residual connection and a FFN structure with a residual connection connected in series; the aforementioned step C3 specifically includes: Step C31: The electronic device sequentially performs layer normalization operation, multi-head self-attention operation and regularization operation on the convolutional features to obtain the global first sub-feature.

[0130] Step C32: The electronic device concatenates the convolution feature with the global first sub-feature to obtain a second concatenated feature.

[0131] Step C33: The electronic device performs a layer normalization operation, a feature transformation operation, and a regularization operation on the second splicing feature to obtain a global second sub-feature.

[0132] Step C34: The electronic device splices the second splicing feature with the global second sub-feature to obtain a global feature.

[0133] The LMHSA structure with residual connection includes a normalization layer, a Lite Multi-Head Self-Attention module (Lite Multi-Head Self-Attention, LMHSA), a regularization layer, and a residual connection. The FFN structure with residual connection includes a normalization layer, a Feed-Forward Network (FFN), a regularization layer, and a residual connection. The two structures connected in series are used to extract global semantic information. For example, the regularization layer can be Dropout / DropPath.

[0134] The LMHSA module first performs a linear transformation on the convolutional features to generate query (Q), key (K), and value (V), and then performs a self-attention operation. To reduce the computational complexity, we introduce k×k Depthwise convolution to reduce spatial dimensions and add relative position bias in self-attention calculation B To enhance the location perception ability. FFN consists of two fully connected layers (Fully Connected, FC) + GELU (Gaussian Error Linear Unit) activation function and a deep convolution layer, where the first FC layer is used to reduce the feature dimension, the deep convolution layer is used to enhance the local feature modeling ability, and the second FC layer is used to expand the feature dimension, thereby improving the expression ability of the model.

[0135] For example, Figure 8 The schematic diagram of the network structure (including convolutional layer, LPU branch, GPU branch and second fusion layer) is shown. Figure 8 The structure of the AGLPU module in , assuming that the input second image feature is , H, W and C represent the height, width and channel of the second image feature respectively; specifically, the AGLPU module processes the input X The processing can be expressed as follows:

[0136]

[0137] in, represents the convolution feature, Indicates that the convolution kernel size is 1×1 and the output channel is C / 2 convolution is used for channel compression and feature integration; Indicates that the convolution kernel size is 3×3, and its output channel is C / 4 depth convolution; Y C1 Represents the output of the first bottleneck structure with residual connection in the CNN branch, i.e., LPU; similarly, YC2 Represents the output of the second bottleneck structure with residual connection in the CNN branch, i.e., LPU; LN Represents the normalization (LayerNorm) layer; LMHSA Represents a lightweight multi-head self-attention module; FFN Represents a feedforward neural network, which is used to enhance the nonlinear capabilities of the network; Y T1 Represents the output of the first LMHSA structure with residual connection in the Transformer branch, i.e., GPU; similarly, Y T2 Represents the output of the Transformer branch, i.e. the FFN structure with residual connections in the GPU; Y Represents the output of the entire AGLPU module, which is formed by the weighted LPU branch and GPU branch, where the weight α is a network hyperparameter learned by network training.

[0138] In combination with the detection methods shown in the above embodiments, the anomaly detection model of the present application sets a TgVL module in the neck network. The TgVL module enhances image features through the VFE branch, and learns the weights of image features under the guidance of text through the TgV branch. By paying attention to the enhanced image features through the weights, the feature fusion of the two modalities can be promoted, effectively solving the problem of small sample learning difficulties. Combined with the TVF detection head, the multimodal fusion of text semantic information and image spatial features can enhance the understanding of complex scenes and determine the detection results more accurately and efficiently. In addition, the AGLPU module set in the backbone network and / or TgVL module can use the LPU branch and the GPU branch to extract local features and global features, and perform adaptive weighted fusion on the two to further improve the accuracy of urban facility anomaly detection.

[0139] In some embodiments, the urban facility anomaly detection method based on the OV-MFS-YOLO network integrates visual and text information, analyzes urban image data captured by large-scale vehicle-mounted cameras through deep learning, automatically mines complex environmental features and detects anomalies. This method can accurately detect and classify various types of urban facility anomalies without relying on preset rules, showing a high degree of automation and intelligence. Its specific advantages are summarized as follows: (1) Cross-modal fusion and generalization capabilities: The OV-MFS-YOLO network breaks through the limitations of traditional target detection methods and jointly trains text and image modalities. It can not only accurately detect known categories, but also effectively detect and classify new, undefined abnormal categories. Its powerful generalization ability enables it to complete small sample learning and even zero-sample detection tasks even when the sample size is limited.

[0140] (2) Incremental learning and knowledge retention: The algorithm supports continuous incremental learning, which can effectively alleviate the problem of "catastrophic forgetting" and ensure that while absorbing new data, existing knowledge can still be retained and optimized, thereby continuously improving model performance.

[0141] (3) Efficient reasoning and flexible deployment: The OV-MFS-YOLO network supports online prompt vocabulary learning and offline vocabulary prediction, which significantly improves the network's reasoning speed while reducing computing resource requirements. It is suitable for private deployment and efficient application in actual scenarios.

[0142] At present, the OV-MFS-YOLO network has broad application prospects in the fields of smart cities and smart urban management, and can be used for tasks such as anomaly detection of urban facilities, monitoring of illegal behaviors, and environmental governance. Its cross-modal fusion capabilities and incremental learning characteristics enable it to adapt to dynamically changing urban environments and achieve accurate and real-time anomaly detection. The network can also be extended to task scenarios such as autonomous driving, smart security, and road inspections that require continuous incremental learning and small sample learning.

[0143] For example, in autonomous driving, it can be used to detect road anomalies, traffic sign changes and emergencies to improve driving safety.

[0144] For example, in smart security, abnormal events such as illegal intrusion and facility destruction can be monitored to enhance the intelligence level of the security system.

[0145] For example, during road inspections, it can assist traffic management departments in detecting road damage, obstacles, and abnormalities in traffic facilities, thereby improving inspection efficiency and accuracy.

[0146] In summary, with the continuous optimization of the network, its application scope will be further expanded, promoting the in-depth application of multimodal perception technology in the fields of smart city management and smart transportation.

[0147] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0148] Corresponding to the urban facility anomaly detection method based on a multimodal model in the above embodiment, Fig. 9 A structural block diagram of an urban facility anomaly detection device 9 based on a multimodal model provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0149] Reference Fig. 9 , the urban facility anomaly detection device 9 based on the multimodal model includes: A feature extraction module 91 is used to extract features from the image to be detected based on the backbone network of the pre-trained anomaly detection model to obtain image features; the image to be detected includes urban facilities; A feature fusion module 92, used to fuse image features based on the neck network of the abnormality detection model to obtain a first fused feature; A detection module 93, configured to detect the first fusion feature based on a detection network of an anomaly detection model, and obtain a detection result of anomalies of urban facilities in the image to be detected; The neck network is provided with a TgVL module, which is used to fuse the input first image feature and the text feature of the preset guide text to obtain the second fusion feature, and the first fusion feature is obtained based on the second fusion feature.

[0150] Optionally, the TgVL module includes an image feature enhancement branch, a text-guided image branch, and a first fusion layer; for the first image feature and the text feature: Processing the first image feature through the image feature enhancement branch to obtain an enhanced image feature; The text-guided image branch sequentially performs summation and weighted activation operations on the preprocessed first image features and text features to obtain guided learning weights; The enhanced image feature and the guided learning weight are multiplied by the first fusion layer and a convolution operation is performed to obtain a second fusion feature corresponding to the first image feature and the text feature.

[0151] Optionally, the text-guided image branch includes an image feature preprocessing sub-branch, a text feature preprocessing sub-branch, an Einstein summation layer, and a MaxSigmoid activation function layer; the text-guided image branch sequentially performs an Einstein summation operation and a weighted activation operation on the preprocessed first image feature and the text feature to obtain a guided learning weight, including: Performing convolution operations and reshaping operations on the first image feature in sequence through the image feature preprocessing sub-branch to obtain a preprocessed first image feature; Through the text feature preprocessing sub-branch, linear operations and reshaping operations are performed on the text features in sequence to obtain preprocessed text features; Performing an Einstein sum operation on the preprocessed first image feature and the preprocessed first image feature through an Einstein sum layer to obtain a third fusion feature; The MaxSigmoid activation function layer performs a MaxSigmoid function activation operation on the third fusion feature to obtain the guided learning weight.

[0152] Optionally, the detection network includes a regression branch, a classification branch and a concatenation layer, and the detection module 93 is used to: Performing a deep convolution operation on the first fusion feature through the regression branch to obtain a regression result; The classification branch sequentially performs summing and scaling operations on the classified first fusion features and the preprocessed text features to obtain a classification result; The regression result and the classification result are spliced ​​through the splicing layer and activated based on the preset activation function to obtain the visualized detection result.

[0153] Optionally, an AGLPU module is provided in the backbone network and / or the TgVL module, and the AGLPU module is used to extract and fuse local features and global features of the input second image features to obtain a fourth fused feature.

[0154] Optionally, the AGLPU module includes a convolution layer, a local perception unit branch, a global perception unit branch, and a second fusion layer, and for each second image feature: Performing a convolution operation on the second image feature through a convolution layer to obtain a convolution feature; The convolution feature is extracted through the local perception unit branch to obtain the local feature; The convolution feature is extracted through the global perception unit to obtain the global feature; The local features and the global features are adaptively weighted fused through the second fusion layer to obtain the fourth fusion feature.

[0155] Optionally, the local perception unit branch is composed of two bottleneck structures with residual connections connected in series; the convolution feature is extracted through the local perception unit branch to obtain the local feature; Perform convolution operation, depth convolution operation and convolution operation on the convolution feature in sequence to obtain the local first sub-feature; Concatenate the convolution feature with the local first sub-feature to obtain the first concatenated feature; Performing a convolution operation, a depthwise convolution operation, and a convolution operation on the first concatenated feature in sequence to obtain a local second sub-feature; The first splicing feature is spliced ​​with the local second sub-feature to obtain a local feature.

[0156] Optionally, the global perception unit branch is composed of a LMHSA structure with a residual connection and a FFN structure with a residual connection in series; the convolution feature is extracted by the global perception unit to obtain a global feature, including: Perform layer normalization, multi-head self-attention and regularization operations on the convolutional features in sequence to obtain the global first sub-feature; Concatenate the convolution feature with the global first sub-feature to obtain the second concatenated feature; Performing a layer normalization operation, a feature transformation operation, and a regularization operation on the second concatenated feature to obtain a global second sub-feature; The second concatenated feature is concatenated with the global second sub-feature to obtain a global feature.

[0157] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0158] Fig.10 This is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: at least one processor 100 ( Fig.10 Only one processor is shown in the figure), a memory 101, and a computer program 102 stored in the memory 101 and executable on at least one processor 100. When the processor 100 executes the computer program 102, the steps in any of the above-mentioned embodiments of the method for detecting anomalies of urban facilities based on a multimodal model are implemented, for example Figure 5 Steps 510-530 are shown.

[0159] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0160] In some embodiments, the memory 101 may be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10. In other embodiments, the memory 101 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 10.

[0161] Furthermore, the memory 101 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 101 is used to store operating devices, applications, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0162] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0163] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0164] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0166] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0167] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0168] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting anomalies of urban facilities based on a multimodal model, characterized in that: include: The backbone network of the pre-trained anomaly detection model extracts features from the image to be detected to obtain image features; The image to be detected includes urban facilities; Based on the neck network of the abnormality detection model, the image features are fused to obtain a first fused feature; The detection head based on the anomaly detection model detects the first fusion feature to obtain a detection result of the anomaly of the urban facilities in the image to be detected; The neck network is provided with a TgVL module, and the TgVL module is used to fuse the input first image feature and the text feature of the preset guide text to obtain a second fusion feature, and the first fusion feature is obtained based on the second fusion feature.

2. The detection method according to claim 1, characterized in that The TgVL module includes an image feature enhancement branch, a text-guided image branch, and a first fusion layer; for the first image feature and the text feature: Processing the first image feature by the image feature enhancement branch to obtain an enhanced image feature; By using the text-guided image branch, sequentially performing a summation operation and a weighted activation operation on the preprocessed first image feature and the text feature to obtain a guided learning weight; The enhanced image feature and the guided learning weight are multiplied by the first fusion layer and a convolution operation is performed to obtain a second fusion feature corresponding to the first image feature and the text feature.

3. The detection method according to claim 2, characterized in that The text-guided image branch includes an image feature preprocessing sub-branch, a text feature preprocessing sub-branch, an Einstein summation layer, and a MaxSigmoid activation function layer; the text-guided image branch sequentially performs an Einstein summation operation and a weighted activation operation on the preprocessed first image feature and the text feature to obtain a guided learning weight, including: Performing a convolution operation and a reshaping operation on the first image feature in sequence through the image feature preprocessing sub-branch to obtain the preprocessed first image feature; The text feature preprocessing sub-branch sequentially performs a linear operation and a reshaping operation on the text feature to obtain the preprocessed text feature; Performing an Einstein summation operation on the preprocessed first image feature and the preprocessed first image feature through the Einstein summation layer to obtain a third fusion feature; The MaxSigmoid activation function layer performs a MaxSigmoid function activation operation on the third fusion feature to obtain the guided learning weight.

4. The detection method according to claim 1, characterized in that The detection head includes a regression branch, a classification branch and a splicing layer. The detection head based on the anomaly detection model detects the first fusion feature to obtain a detection result of the anomaly of urban facilities in the image to be detected, including: Performing a deep convolution operation on the first fusion feature through the regression branch to obtain a regression result; By means of the classification branch, a summing operation and a scaling operation are sequentially performed on the first fusion feature after classification and the text feature after preprocessing to obtain a classification result; The regression result and the classification result are spliced ​​through the splicing layer and activated based on a preset activation function to obtain a visualized detection result.

5. The detection method according to any one of claims 1 to 4, characterized in that: An AGLPU module is provided in the backbone network and / or the TgVL module, and the AGLPU module is used to extract and fuse local features and global features of the input second image features to obtain a fourth fused feature.

6. The detection method according to claim 5, characterized in that The AGLPU module includes a convolution layer, a local perception unit branch, a global perception unit branch and a second fusion layer. For each of the second image features: Performing a convolution operation on the second image feature through the convolution layer to obtain a convolution feature; Extracting the convolution feature through the local perception unit branch to obtain the local feature; Extracting the convolution feature through the global perception unit to obtain the global feature; The local feature and the global feature are adaptively weighted fused through the second fusion layer to obtain the fourth fused feature.

7. The detection method according to claim 6, characterized in that The local perception unit branch is composed of two bottleneck structures with residual connections connected in series; the convolution feature is extracted through the local perception unit branch to obtain the local feature; Performing a convolution operation, a depthwise convolution operation, and a convolution operation on the convolution feature in sequence to obtain a local first sub-feature; Concatenate the convolution feature with the local first sub-feature to obtain a first concatenated feature; Performing a convolution operation, a depthwise convolution operation, and a convolution operation on the first concatenated feature in sequence to obtain a local second sub-feature; The first splicing feature is spliced ​​with the local second sub-feature to obtain the local feature.

8. The detection method according to claim 6, characterized in that: The global perception unit branch is composed of a LMHSA structure with a residual connection and a FFN structure with a residual connection in series; the global perception unit is used to extract the convolutional features to obtain the global features, including: Performing a layer normalization operation, a multi-head self-attention operation, and a regularization operation on the convolutional features in sequence to obtain a global first sub-feature; Concatenate the convolution feature with the global first sub-feature to obtain a second concatenated feature; Performing a layer normalization operation, a feature transformation operation, and a regularization operation on the second concatenated feature to obtain a global second sub-feature; The second splicing feature is spliced ​​with the global second sub-feature to obtain the global feature.

9. An urban facility anomaly detection device based on a multimodal model, characterized in that: include: A feature extraction module is used to extract features from an image to be detected based on a backbone network of a pre-trained anomaly detection model to obtain image features; the image to be detected includes urban facilities; A feature fusion module, used for fusing the image features based on the neck network of the abnormality detection model to obtain a first fused feature; A detection module, configured to detect the first fusion feature based on the detection head of the anomaly detection model, and obtain a detection result of anomalies of urban facilities in the image to be detected; The neck network is provided with a TgVL module, and the TgVL module is used to fuse the input first image feature and the text feature of the preset guide text to obtain a second fusion feature, and the first fusion feature is obtained based on the second fusion feature.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting anomalies of urban facilities based on a multimodal model is implemented as described in any one of claims 1 to 8.

11. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting anomalies of urban facilities based on a multimodal model according to any one of claims 1 to 8 is implemented.

Images