Abnormal alarm method, device and equipment for traffic infrastructure and storage medium

Abstract

The invention discloses a traffic infrastructure abnormity alarm method, device and equipment and a storage medium, and relates to the technical field of equipment detection, and the method comprises the steps: obtaining a target traffic infrastructure RGB image and a point cloud image collected by data collection equipment, and extracting corresponding two-dimensional features and three-dimensional features corresponding to the point cloud image; then performing alignment operation to obtain a target two-dimensional feature map and a target three-dimensional feature map with pixel levels aligned; performing cross-modal mapping on the target two-dimensional feature map based on a preset feature mapping function to obtain a corresponding first mapped feature map, and performing cross-modal mapping on the target three-dimensional feature map to obtain a corresponding second mapped feature map; and determining an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map, and carrying out fault abnormity alarm. Therefore, equipment abnormity alarm can be carried out in time.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment detection, and in particular to an abnormal alarm method, device, equipment and storage medium for traffic infrastructure. Background Art

[0002] Transportation infrastructure such as bridges, tunnels, and roads are directly related to people's life safety. Once there is a problem with these infrastructures, serious traffic accidents may occur, and even cause casualties. In recent years, extreme weather and natural disasters such as earthquakes, floods, and landslides have occurred frequently, posing an increasing threat to transportation infrastructure. These disasters are often sudden. Traditional transportation infrastructure detection usually relies on manual inspections and regular maintenance, which not only consumes a lot of manpower and resources, but may also miss some minor but important structural problems due to manual negligence; and for traditional natural disasters with sudden occurrences, traditional monitoring systems are difficult to warn and respond in time. Therefore, how to improve the detection efficiency of the safety status of transportation infrastructure, timely discover and deal with potential disaster risks, and ensure safe and smooth traffic is a problem that needs to be solved. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an abnormal alarm method, device, equipment and storage medium for traffic infrastructure, which can improve the accuracy and reliability of abnormal detection of traffic facilities so that staff can respond quickly to abnormal situations in a timely manner. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses an abnormality alarm method for traffic infrastructure, comprising:

[0005] Acquire a facility status image of a target transportation infrastructure acquired by a data acquisition device; the facility status image includes an RGB image and a point cloud image;

[0006] Using a preset feature extraction technology to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image, and performing a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level;

[0007] Based on a preset feature mapping function, the target two-dimensional feature map is cross-modally mapped to obtain a corresponding first mapped feature map, and the target three-dimensional feature map is cross-modally mapped to obtain a corresponding second mapped feature map; the first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map;

[0008] determining an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map;

[0009] Based on the abnormal state image, it is determined that there are abnormal target facilities in the target traffic infrastructure, and a fault abnormality alarm is performed based on a preset abnormality alarm method.

[0010] Optionally, the acquiring of facility status images of target transportation infrastructure collected by a data collection device includes:

[0011] The target traffic infrastructure is monitored and the corresponding RGB images and point cloud images are collected by a multifunctional camera pre-installed at the corresponding position of the target traffic infrastructure; the multifunctional camera is a camera that integrates infrared sensors, multispectral sensors, thermal imaging functions and 3D model generation functions and whose imaging accuracy meets preset high-precision conditions.

[0012] Optionally, the extracting the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image respectively by using a preset feature extraction technology includes:

[0013] Extracting two-dimensional features corresponding to the RGB image by using a 2D feature extractor and a bilinear upsampling technique;

[0014] The initial features corresponding to the point cloud image are obtained by using a 3D feature extractor, and a target calculation feature is determined based on the distance between the initial features and a preset feature. The target calculation feature is then weightedly calculated to obtain the three-dimensional features corresponding to the point cloud image.

[0015] Optionally, performing a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level includes:

[0016] The three-dimensional features are projected onto a 2D image plane based on a preset pixel position setting, and the projected three-dimensional features are processed using a smoothing kernel technique to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level.

[0017] Optionally, performing cross-modal mapping on the target two-dimensional feature map based on a preset feature mapping function to obtain a corresponding first mapped feature map, and simultaneously performing cross-modal mapping on the target three-dimensional feature map to obtain a corresponding second mapped feature map, includes:

[0018] Deploy the corresponding preset feature mapping function through the preset multi-layer perceptron;

[0019] The preset feature mapping function is used to map the feature vector in the target two-dimensional feature map from a two-dimensional state to a three-dimensional state to obtain a first mapped feature map, and the feature vector in the target three-dimensional feature map is mapped from a three-dimensional state to a two-dimensional state to obtain a second mapped feature map.

[0020] Optionally, the determining the abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map includes:

[0021] Compare the target two-dimensional feature map with the second mapped feature map pixel by pixel based on a preset difference function to obtain a first difference in a two-dimensional state, and compare the target three-dimensional feature map with the first mapped feature map pixel by pixel based on the preset difference function to obtain a second difference in a three-dimensional state;

[0022] An abnormality score determined based on the first difference and the second difference determines an abnormal state image among the corresponding facility state images.

[0023] Optionally, the determining the abnormal state image in the corresponding facility state image according to the abnormality score determined based on the first difference and the second difference includes:

[0024] The first difference and the second difference are fused based on an aggregation function to obtain a fused difference, and the fused difference is smoothed based on a preset Gaussian kernel to obtain a processed anomaly map;

[0025] An abnormal state image in the corresponding facility state image is determined based on the abnormal score corresponding to the processed abnormal image.

[0026] In a second aspect, the present application discloses an abnormality alarm device for traffic infrastructure, comprising:

[0027] An image acquisition module, used to acquire a facility status image of a target transportation infrastructure acquired by a data acquisition device; the facility status image includes an RGB image and a point cloud image;

[0028] A feature extraction module, used to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image using a preset feature extraction technology, and perform a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level;

[0029] A feature mapping module, configured to perform cross-modal mapping on the target two-dimensional feature map based on a preset feature mapping function to obtain a corresponding first mapped feature map, and to perform cross-modal mapping on the target three-dimensional feature map to obtain a corresponding second mapped feature map; the first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map;

[0030] an abnormality determination module, configured to determine an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map;

[0031] A fault alarm module is used to determine the target facilities with abnormalities in the target traffic infrastructure based on the abnormal state image, and to perform fault abnormality alarm based on a preset abnormality alarm method.

[0032] In a third aspect, the present application discloses an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the aforementioned abnormality alarm method for traffic infrastructure.

[0035] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned abnormal alarm method for traffic infrastructure.

[0036] It can be seen that in the present application, a facility status image for a target traffic infrastructure is acquired by a data acquisition device; the facility status image includes an RGB image and a point cloud image; a preset feature extraction technology is used to extract two-dimensional features corresponding to the RGB image and three-dimensional features corresponding to the point cloud image, respectively, and pixel alignment operations are performed on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level; based on a preset feature mapping function, the target two-dimensional feature map is cross-modally mapped to obtain a corresponding first mapped feature map, and at the same time, the target three-dimensional feature map is cross-modally mapped to obtain a corresponding second mapped feature map; the first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map; based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map, an abnormal status image in the facility status image is determined; based on the abnormal status image, an abnormal target facility in the target traffic infrastructure is determined, and a fault abnormality alarm is performed based on a preset abnormal alarm method. That is, by using multimodal technology to compare the features extracted from different data types (two-dimensional and three-dimensional), the accuracy and reliability of traffic facility anomaly detection can be improved by comparing the internal consistency between different modalities, so that staff can respond quickly to abnormal situations in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of an abnormality alarm method for traffic infrastructure disclosed in this application;

[0039] Figure 2 A flowchart of a specific abnormality alarm method for traffic infrastructure disclosed in this application;

[0040] Figure 3 A flowchart of a specific abnormality alarm method for traffic infrastructure disclosed in this application;

[0041] Figure 4 A toy example diagram of a specific multimodal mapping disclosed in this application;

[0042] Figure 5 A schematic diagram of the structure of an abnormality alarm device for traffic infrastructure disclosed in this application;

[0043] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The maintenance of transportation infrastructure usually requires a lot of money and resources, especially in widely distributed transportation networks, where frequent manual inspections are costly. This application will introduce an abnormality alarm method for transportation infrastructure, which can detect abnormalities in facilities safely, efficiently and intelligently.

[0046] See also Figure 1 As shown, the embodiment of the present application discloses a method for abnormal alarm of traffic infrastructure, comprising:

[0047] Step S11: Acquire a facility status image of a target transportation infrastructure acquired by a data acquisition device; the facility status image includes an RGB image and a point cloud image.

[0048] In this embodiment, the target traffic infrastructure is monitored and the corresponding RGB (Red Green Blue) images and point cloud images are collected by pre-installing a multifunctional camera at the corresponding position of the target traffic infrastructure; the multifunctional camera is a camera that integrates infrared sensors, multispectral sensors, thermal imaging functions and 3D (Three Dimensions) model generation functions and whose imaging accuracy meets the preset high-precision conditions. That is, the camera that meets the preset high-precision conditions is installed at the key parts of the target traffic infrastructure (such as bridge support points, road slopes, etc.), which can collect images, videos and physical parameters (such as vibration, temperature, humidity, etc.) of the health status of the structure in real time.

[0049] Among them, the composition of the camera with preset high-precision conditions described in this embodiment is specifically introduced as follows. High-definition visible light sensors can capture high-resolution images and videos. This camera has functions such as autofocus, auto exposure and high dynamic range (HDR, i.e. High-Dynamic Range), and can provide clear images under different lighting conditions (such as daytime and nighttime). This part of the camera is mainly used to detect physical changes on the surface of infrastructure, such as cracks, corrosion, deformation, etc.) Infrared sensors can monitor in low light or even no light conditions. Infrared cameras capture thermal radiation imaging of objects, which is suitable for nighttime monitoring or dim environments such as tunnels, and can also detect temperature anomalies, such as overheated areas. Multispectral sensors can capture spectral information in different bands. This camera can not only obtain visible light images, but also capture invisible spectra such as near-infrared and ultraviolet. This has unique advantages in identifying material aging, moisture distribution, chemical changes, etc. For example, the degradation of some transportation facility materials is not obvious under visible light, but may be apparent in infrared or ultraviolet bands. Thermal imaging can directly display the temperature distribution of objects and detect hot spots in infrastructure to identify possible overheating, material aging or stress concentration areas. Depth sensors can generate 3D models of infrastructure through LiDAR (Light Laser Detection and Ranging) or structured light technology. It helps to monitor the three-dimensional morphological changes of infrastructure, such as road depressions and bridge bends. 3D imaging can provide richer information than flat images, especially in the monitoring of complex structures. ) Multifunctional cameras have high protection levels, such as IP67 (Ingress Protection Rating 67) or higher, and can work stably and for a long time in harsh environments. They are dustproof, waterproof, shockproof, and resistant to high and low temperatures, and are suitable for outdoor bridges, roads, tunnels and other scenes. In addition, some cameras are also resistant to strong light and electromagnetic interference, ensuring normal operation under various extreme conditions. In addition, they support remote control functions, and equipment managers can remotely adjust the camera's shooting angle, focal length, aperture and other parameters to meet different monitoring needs. Some high-end cameras also have an adaptive adjustment function, which can automatically adjust the shooting parameters according to changes in the environment to ensure the best monitoring effect under any conditions. In this embodiment, the camera is designed as a low-power device that can be powered by solar energy or a low-voltage power supply, and is suitable for deployment in remote areas or areas where it is difficult to access electricity. At the same time, the camera's energy management system can intelligently adjust power consumption to extend the device's operating time and reduce maintenance requirements.In addition, in order to meet possible technology upgrade needs in the future, the multi-function camera is designed with an expansion interface that can connect more sensors or external devices, such as meteorological sensors, air quality monitoring equipment, etc., to further enhance monitoring capabilities and the diversity of data collection.

[0050] In this embodiment, the transmission methods of data, i.e., facility status images, include wired, such as optical fiber, Ethernet, and wireless, such as Wi-Fi (wireless network communication technology), 4G (The 4th Generation Mobile Communication Technology, the fourth generation mobile communication technology) / 5G (The 5th Generation Mobile Communication Technology, the fifth generation mobile communication technology), LoRa (Long Range Radio), etc., to transmit data from various monitoring points to a central data processing unit. They also usually have built-in memory to store data locally to ensure that data is not lost when the network is interrupted. At the same time, encryption technology is used to ensure the security of data transmission.

[0051] Step S12: using a preset feature extraction technology to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image, and performing a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level.

[0052] In this embodiment, the use of a preset feature extraction technology to extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image respectively includes: extracting the two-dimensional features corresponding to the RGB image by a 2D feature extractor and a bilinear upsampling technology; using a 3D feature extractor to extract the initial features corresponding to the point cloud image, and determining the target calculation features based on the distance between the initial features and the preset features, and then performing weighted calculation processing on the target calculation features to obtain the three-dimensional features corresponding to the point cloud image. Specifically, extracting a 2D image (denoted as I) for each pixel 2D ) features, and 3D point cloud (denoted as P 3D ) extracts features from each point in the image. In the framework of this application, both feature extractors are trained on a large external dataset and remain frozen, that is, their weights are not updated. 2D , that is, RGB image, this embodiment uses 2D feature extractor F 2D Process it and generate a size of H f ×W f ×D 2D The feature map of H fand W f The sizes of f are smaller than the original H and W. Therefore, during the operation, this embodiment adopts a bilinear upsampling operation to obtain a feature map E of size H×W×D 2D 2D , thereby obtaining a feature vector for each pixel position. For a point cloud of size N×3, that is, a point cloud image, in this embodiment, a 3D feature extractor F 3D is used to process it, obtaining N f feature vectors of size D 3D . Each feature vector f c is associated with a specific point c P 3D in the original point cloud. In fact, many 3D feature extractors do not estimate features for each input point, but only for a subset of them, that is, N f < N. Therefore, in order to obtain the feature vector f P 3D for each point p p in the point cloud, f p is calculated by taking the weighted sum of the three feature vectors closest to p among the N 3D feature vectors extracted by F f . In this way, a set of N interpolated feature vectors E' 3D of size D 3D is finally obtained. Considering the standard settings for multimodal anomaly detection, this embodiment sets that the pixels of the 3D data and the image are corresponding. Therefore, the pixel position corresponding to each 3D point is known in this solution. Since E 2D and E' 3D have been interpolated to match the resolution of the original image and the point cloud, this embodiment can project E' 3D onto the 2D image plane to obtain a feature map E 3D of size H×W×D 3D . During this process, the vectors at pixel positions without corresponding 3D features are set to zero. Finally, a 3×3 smoothing kernel is applied to E 3D . After this process, two feature maps E 2D and E 3D aligned at the pixel level are finally obtained. Regarding the feature extraction network mentioned in this embodiment, the same frozen Transformer model as M3DM (Multi-3D-Memory, that is, a multimodal industrial anomaly detection method) is adopted to implement F 2D and F 3D ​The feature extractors are DINO, ViT-B / 8, trained on ImageNet (a large visual database used for visual object recognition software research), and Point-MAE, trained on ShapeNet (a widely used dataset for 3D shape understanding and analysis). 2D Process the 224×224 RGB image and output a 28×28×768 feature map, then upsample it to 224×224×768 through bilinear upsampling and input the feature map into M 2D→3D . F 3D The processing is based on the 1024 sets of 3D data points, generating feature vectors of 1152 dimensions each. These features are interpolated and aligned, adjusted to 224×224×1152, and then input into M 3D→2D . It should be noted that the feature extractor used in this embodiment is based on a Transformer encoder consisting of 64 layers. The difference between features at different levels is that different degrees of self-attention processing are applied to the original input. As the input features pass through the encoder layer, the contextual information of the features gradually increases. By learning the mapping between frozen shallow extractor features, significant improvements can be made in terms of memory requirements and inference speed, with limited impact on the effect. This embodiment selects a frozen feature extractor (F 2D ,F 3D ) and discarding the layers from l+1 to the end to perform layer pruning. As a result, the cross-modal network M l 2D→3D and M l 3D→2D Map the features of the 2D encoder layer l to the features of the 3D encoder layer l, and vice versa.

[0053] Step S13: Based on a preset feature mapping function, the target two-dimensional feature map is cross-modally mapped to obtain a corresponding first mapped feature map, and the target three-dimensional feature map is cross-modally mapped to obtain a corresponding second mapped feature map; the first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map.

[0054] In this embodiment, once E is obtained 2D and E 3D , this application will deploy two feature mapping functions, which are implemented by lightweight multi-layer perceptron (MLP), namely M 2D→3D and M 3D→2D . M 2D→3D A size D 2D The feature vector is mapped to another dimension D 3D The eigenvector of M 3D→2DThe reverse operation is performed. Each network processes each pixel location independently and predicts features from one modality to the other.

[0055] Step S14: determining an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map.

[0056] In this embodiment, the determination of the abnormal state image in the facility state image based on the first difference between the target two-dimensional feature map and the second mapped feature map and the second difference between the target three-dimensional feature map and the first mapped feature map includes: comparing the target two-dimensional feature map with the second mapped feature map pixel by pixel based on a preset difference function to obtain the first difference in the two-dimensional state, and comparing the target three-dimensional feature map with the first mapped feature map pixel by pixel based on the preset difference function to obtain the second difference in the three-dimensional state; determining the abnormal state image in the corresponding facility state image based on the abnormal score determined based on the first difference and the second difference. Wherein, the determination of the abnormal state image in the corresponding facility state image based on the abnormal score determined based on the first difference and the second difference includes: fusing the first difference and the second difference based on an aggregation function to obtain a fused difference, and smoothing the fused difference based on a preset Gaussian kernel to obtain a processed abnormal map; determining the abnormal state image in the corresponding facility state image based on the abnormal score corresponding to the processed abnormal map. That is, the test sample is passed to the feature extraction and mapping network to obtain two pairs of extracted and predicted feature maps ( , )and( , ). For all individual eigenvectors 2 After normalization, the extracted and predicted feature maps are passed through the difference function Pixel-by-pixel comparison to obtain modality-specific anomaly maps 2D and 3D :

[0057] ;

[0058] Here, Euclidean distance is used as the difference function The above anomaly graphs are combined through the aggregation function A to obtain the final anomaly graph =A( 2D , 3D). This step fuses the information from both modalities together, providing a comprehensive understanding of the anomalies present in both the image and the point cloud. Anomalies are located only when the predictions are inconsistent with the observations, which provides high sensitivity for various types of anomalies and good robustness in reducing false detections. Therefore, the pixel-wise product is used as the aggregation function in this application: = 2D · 3D , the anomaly score of a pixel location will be high only when both modality-specific scores are high, i.e., anomaly detection must be confirmed by both modalities. The aggregated anomaly map is obtained by =4 for smoothing, and the global anomaly score used to perform sample-level anomaly detection is the maximum value of the anomaly map Ψ. Among them, the alarm threshold can be set according to the actual situation. When the detection results exceed these thresholds, a warning or alarm is triggered. For example, when a certain vibration amplitude, temperature rise, or structural deformation exceeds the set value, a warning signal is immediately generated.

[0059] Step S15: determining the target facilities with abnormalities in the target traffic infrastructure based on the abnormal state image, and performing fault abnormality alarm based on a preset abnormality alarm method.

[0060] In this embodiment, when an abnormality or potential risk is detected, relevant personnel are notified in a timely manner or automated response measures are triggered. According to the severity of the abnormality, this embodiment sets multiple levels of warnings, such as yellow (minor), orange (medium), and red (serious), and each level corresponds to different response measures. When an accident occurs, an audible and visual alarm device can be set on site, such as flashing light strips on the highway, etc. When an abnormality is detected, an alarm sound or flashing light is immediately issued to alert commuters on site; or relevant managers and technicians are notified by text messages, phone calls, emails, etc., including detailed information such as the location, type, and severity of the abnormality; furthermore, real-time alerts are pushed to administrators or maintenance personnel through dedicated mobile applications, and images, videos, and sensor data related to the abnormality are provided for easy viewing and processing anytime, anywhere.

[0061] In this embodiment, for certain serious or urgent abnormal situations, a series of preset emergency measures can be automatically triggered, such as closing roads, starting backup power supplies, activating fire protection systems, etc. In addition, preset emergency response plans can be stored, and appropriate response plans can be automatically suggested according to the situation when an alarm is issued, for managers to refer to and execute. Moreover, when the system issues an alarm, managers are allowed to confirm the authenticity of the alarm through the remote monitoring platform and decide on the next action, such as dispatching a maintenance team, conducting further inspections, etc.

[0062] In addition, this embodiment records all triggered warning and alarm events, including time, location, triggering reasons, processing results, etc. The log can assist in subsequent event analysis and system optimization. All alarm records and related data need to be stored in a secure database and backed up regularly to ensure data security and integrity. In addition, through visualization tools, alarm history, trend analysis and key indicators are displayed, and regular alarm reports are generated to help management evaluate the operating effect and safety status of the system. Furthermore, big data analysis technology can be used to analyze the changing trends of structural health status and predict possible failures or damages; based on historical data, the overall health status of transportation infrastructure can be evaluated and a health report can be generated; based on data analysis results, maintenance and repair recommendations can be provided to optimize resource usage.

[0063] It should be noted that this embodiment provides a visual interface that can view the status of each monitoring point in real time, monitor early warning and alarm conditions. It supports alarm information management from different channels, and managers can view, filter, and respond to different types of alarm information. In this embodiment, all alarm data transmission and storage use encryption technology to ensure that the information is not illegally intercepted or tampered with. Redundancy technology allows the alarm function to still work normally when some hardware or software fails.

[0064] It can be seen that in this embodiment, Figure 2As shown, a facility status image for a target traffic infrastructure acquired by a data acquisition device is obtained; the facility status image includes an RGB image and a point cloud image; a preset feature extraction technology is used to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image, and pixel alignment operations are performed on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level; based on a preset feature mapping function, the target two-dimensional feature map is cross-modally mapped to obtain a corresponding first mapped feature map, and at the same time, the target three-dimensional feature map is cross-modally mapped to obtain a corresponding second mapped feature map; the first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map; based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map, an abnormal status image in the facility status image is determined; based on the abnormal status image, an abnormal target facility in the target traffic infrastructure is determined, and a fault abnormality alarm is performed based on a preset abnormal alarm method. That is, data from different types of sensors and cameras, including images, videos, infrared, vibration, strain, etc., are integrated, and features extracted from different data types (two-dimensional and three-dimensional) are compared using multimodal technology. By comparing the internal consistency between different modalities, the accuracy and reliability of anomaly detection in traffic facilities can be improved, so that staff can respond quickly to abnormal situations in a timely manner.

[0065] In the previous embodiment, the abnormal alarm process of the entire traffic infrastructure was introduced as a whole. This embodiment will specifically introduce the process and principle of cross-modal mapping.

[0066] See also Figure 3 As shown, the embodiment of the present application discloses a specific abnormal alarm method for traffic infrastructure, including:

[0067] Step S21: Acquire a facility status image of a target transportation infrastructure acquired by a data acquisition device; the facility status image includes an RGB image and a point cloud image.

[0068] Step S22: using a preset feature extraction technology to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image, and performing a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level.

[0069] Step S23: Deploy the corresponding preset feature mapping function through the preset multilayer perceptron, and then use the preset feature mapping function to map the feature vector in the target two-dimensional feature map from a two-dimensional state to a three-dimensional state to obtain a first mapped feature map, and at the same time map the feature vector in the target three-dimensional feature map from a three-dimensional state to a two-dimensional state to obtain a second mapped feature map.

[0070] In this embodiment, given a pixel position i and its corresponding 2D and 3D features E i 2D and E i 3D , we can get the prediction features of another mode:

[0071] ;

[0072] When processing pixel positions that have no 3D point association, the corresponding prediction features are set to zero. By processing all pixels, a H×W×D 2D Prediction feature map of and dimensions are H×W×D 3D of During training, M 2D→3D and M 3D→2D All normal samples in the dataset are jointly optimized by minimizing the cosine distance between the feature maps of the two modalities of the input data and the predicted feature maps. Therefore, the loss for each pixel is:

[0073] ;

[0074] Among them, M 2D→3D and M 3D→2D They all consist of three linear layers, each of which (except the last one) is followed by a GeLU activation function. 2D→3D The number of units in each layer is 768, 960, and 1152 respectively, and M 3D→2D The number of units in are 1152, 960, and 768, respectively. The two networks are trained together for 250 cycles using the Adam optimizer with a learning rate of 0.001. The 3D point cloud is fitted to a plane using a random sample consensus algorithm, and points with a distance less than 0.005 from the plane are considered background points, which are removed from the input to F3D. This process speeds up the processing of 3D features and reduces background noise in abnormal images.

[0075] This embodiment uses Figure 4 During training (top left), red 2D patterns on flat 3D surfaces and blue 2D patterns on curved surfaces: M 2D→3D and M 3D→2DLearn to predict the characteristic relationships between these data. When inferring, if an abnormal yellow 2D pattern appears on the surface (upper right corner), M 3D→2D The predicted 2D feature corresponds to the blue pattern, while the observed 2D feature is the yellow pattern. 2D→3D Receiving an input feature that was not seen during training is unlikely to output a 3D feature on the actual surface. Therefore, the method of this embodiment can perceive the difference between the prediction and observation between 2D and 3D features. Similarly, when a normal 2D pattern appears on an abnormal 3D surface (lower left corner), the two prediction results are inconsistent with the observations. This also applies when both modalities are abnormal: since the input data has not been seen during training, both cross-modal predictions are unlikely to match the observations. Each modality follows a normal distribution individually, but their joint occurrence is abnormal. This can be illustrated with an example of a red pattern appearing on a surface (lower right corner): Again, M 2D→3D The output is the 3D features of the flat area, while M 3D→2D The output is the 2D features of the blue pattern, and both predictions are inconsistent with the observed values.

[0076] It should be pointed out that due to the diversity of normal samples, the mapping between 2D and 3D features may not be unique. Figure 3 In the example, there may be both flat and curved red surfaces. This one-to-many mapping makes M 2D→3D It is difficult to learn to map the features of the 2D red region to the correct 3D features. Therefore, when presented with a red region, M 2D→3D M may predict wrong 3D features or unlikely features, resulting in discrepancies between the predicted and observed 3D features. However, since the 3D→2D mapping is many-to-one, M 3D→2D It is possible to predict the 2D features of the red area. Therefore, it is possible to avoid false positives by locating anomalies only when both predictions are inconsistent with the observed value. Of course, due to the higher variability of normal samples, you may also encounter many-to-many mappings from 3D to 2D during operation, such as in the figure, where blue and red patterns on the surface exist at the same time. In this case, when a red area on a surface is presented, M 2D→3D may incorrectly predict features in flat areas, while M 3D→2D It is possible to predict features in the blue area, which ultimately leads to false positives because both predictions are inconsistent with the observed values.

[0077] In the framework, the potential one-to-many mapping problem in cross-modality can be solved by the highly contextualized 2D and 3D features provided by the Transformer architecture. In fact, a contextualized 2D feature, such as describing a red area surrounded by blue and purple areas, often corresponds to a specific contextualized 3D feature, such as representing a flat area to the right of an undulating surface area. In other words, the highly contextualized 2D and 3D features extracted by the Transformer are unlikely to achieve a one-to-many cross-modal mapping. For the above reasons, this embodiment is based on F 2D and F 3D Transformer is used in both.

[0078] Step S24: determining an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map.

[0079] Step S25: determining the target facilities with abnormalities in the target traffic infrastructure based on the abnormal state image, and performing fault abnormality alarm based on a preset abnormality alarm method.

[0080] Among them, the specific processes of the above steps S21, S22, S24 and S25 can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0081] It can be seen that by comparing the features extracted from different data types (two-dimensional and three-dimensional) using multimodal technology, the accuracy and reliability of traffic facility anomaly detection can be improved by comparing the internal consistency between different modalities.

[0082] refer to Figure 5 The embodiment of the present application also discloses an abnormality alarm device for traffic infrastructure, including:

[0083] An image acquisition module 11 is used to acquire a facility status image of a target transportation infrastructure acquired by a data acquisition device; the facility status image includes an RGB image and a point cloud image;

[0084] A feature extraction module 12 is used to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image using a preset feature extraction technology, and perform a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level;

[0085] A feature mapping module 13 is used to perform cross-modal mapping on the target two-dimensional feature map based on a preset feature mapping function to obtain a corresponding first mapped feature map, and to perform cross-modal mapping on the target three-dimensional feature map to obtain a corresponding second mapped feature map; the first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map;

[0086] an abnormality determination module 14, configured to determine an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map;

[0087] The fault alarm module 15 is used to determine the target facilities with abnormalities in the target traffic infrastructure based on the abnormal state image, and to perform fault abnormality alarm based on a preset abnormality alarm method.

[0088] It can be seen that in this embodiment, multimodal technology is used to compare the features extracted from different data types (two-dimensional and three-dimensional). By comparing the internal consistency between different modes, the accuracy and reliability of traffic facility anomaly detection can be improved, so that staff can respond quickly to abnormal situations in a timely manner.

[0089] In some specific embodiments, the image acquisition module 11 can be specifically used to monitor the target traffic infrastructure and collect corresponding RGB images and point cloud images through a multi-function camera pre-installed at the corresponding position of the target traffic infrastructure; the multi-function camera is a camera that integrates infrared sensors, multi-spectral sensors, thermal imaging functions and 3D model generation functions and whose imaging accuracy meets preset high-precision conditions.

[0090] In some specific embodiments, the feature extraction module 12 may specifically include:

[0091] A two-dimensional feature extraction unit, used to extract two-dimensional features corresponding to the RGB image through a 2D feature extractor and a bilinear upsampling technique;

[0092] The three-dimensional feature extraction unit is used to use the initial features corresponding to the point cloud image of the 3D feature extractor, determine the target calculation features based on the distance between the initial features and the preset features, and then perform weighted calculation processing on the target calculation features to obtain the three-dimensional features corresponding to the point cloud image.

[0093] In some specific embodiments, the feature extraction module 12 may specifically include:

[0094] A pixel alignment unit is used to project the three-dimensional features onto a 2D image plane based on a preset pixel position setting, and process the projected three-dimensional features using a smoothing kernel technique to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level.

[0095] In some specific embodiments, the feature mapping module 13 may specifically include:

[0096] A function building unit, used to deploy a corresponding preset feature mapping function through a preset multi-layer perceptron;

[0097] A cross-modal mapping unit is used to use the preset feature mapping function to map the feature vector in the target two-dimensional feature map from a two-dimensional state to a three-dimensional state to obtain a first mapped feature map, and to map the feature vector in the target three-dimensional feature map from a three-dimensional state to a two-dimensional state to obtain a second mapped feature map.

[0098] In some specific embodiments, the abnormality determination module 14 may specifically include:

[0099] a difference comparison unit, configured to compare the target two-dimensional feature map with the second mapped feature map pixel by pixel based on a preset difference function to obtain a first difference in a two-dimensional state, and to compare the target three-dimensional feature map with the first mapped feature map pixel by pixel based on the preset difference function to obtain a second difference in a three-dimensional state;

[0100] The abnormality determination submodule is used to determine an abnormal state image in the corresponding facility state image based on an abnormality score determined by the first difference and the second difference.

[0101] In some specific embodiments, the abnormality determination submodule may specifically include:

[0102] a difference fusion unit, configured to fuse the first difference and the second difference based on an aggregation function to obtain a fused difference, and to smooth the fused difference based on a preset Gaussian kernel to obtain a processed anomaly map;

[0103] The abnormal state image determining unit is used to determine the abnormal state image in the corresponding facility state image based on the abnormality score corresponding to the processed abnormal image.

[0104] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0105] Figure 6The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the abnormal alarm method for traffic infrastructure disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0106] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0107] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0108] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the abnormal alarm method of the traffic infrastructure executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0109] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the abnormal alarm method for traffic infrastructure disclosed above is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0110] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0111] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0112] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0113] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0114] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for abnormal alarm of traffic infrastructure, characterized in that: include: Acquire facility status images of target transportation infrastructure collected by data collection equipment; The facility status image includes an RGB image and a point cloud image; Using a preset feature extraction technology to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image, and performing a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level; Based on a preset feature mapping function, cross-modal mapping is performed on the target two-dimensional feature map to obtain a corresponding first mapped feature map, and cross-modal mapping is performed on the target three-dimensional feature map to obtain a corresponding second mapped feature map; The first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map; determining an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map; Based on the abnormal state image, it is determined that there are abnormal target facilities in the target traffic infrastructure, and a fault abnormality alarm is performed based on a preset abnormality alarm method.

2. The abnormality alarm method of traffic infrastructure according to claim 1, characterized in that: The step of obtaining a facility status image of a target transportation infrastructure facility collected by a data collection device includes: The target traffic infrastructure is monitored and the corresponding RGB images and point cloud images are collected by a multifunctional camera pre-installed at the corresponding position of the target traffic infrastructure; the multifunctional camera is a camera that integrates infrared sensors, multispectral sensors, thermal imaging functions and 3D model generation functions and whose imaging accuracy meets preset high-precision conditions.

3. The abnormality alarm method of traffic infrastructure according to claim 1, characterized in that: The extracting the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image respectively by using a preset feature extraction technology includes: Extracting two-dimensional features corresponding to the RGB image by using a 2D feature extractor and a bilinear upsampling technique; The initial features corresponding to the point cloud image are obtained by using a 3D feature extractor, and a target calculation feature is determined based on the distance between the initial features and a preset feature. The target calculation feature is then weightedly calculated to obtain the three-dimensional features corresponding to the point cloud image.

4. The abnormality alarm method of traffic infrastructure according to claim 1, characterized in that: The performing pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level includes: The three-dimensional features are projected onto a 2D image plane based on a preset pixel position setting, and the projected three-dimensional features are processed using a smoothing kernel technique to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level.

5. The abnormality alarm method of traffic infrastructure according to claim 1, characterized in that: The cross-modal mapping of the target two-dimensional feature map based on a preset feature mapping function to obtain a corresponding first mapped feature map, and the cross-modal mapping of the target three-dimensional feature map to obtain a corresponding second mapped feature map, includes: Deploy the corresponding preset feature mapping function through the preset multi-layer perceptron; The preset feature mapping function is used to map the feature vector in the target two-dimensional feature map from a two-dimensional state to a three-dimensional state to obtain a first mapped feature map, and the feature vector in the target three-dimensional feature map is mapped from a three-dimensional state to a two-dimensional state to obtain a second mapped feature map.

6. The abnormality alarm method for traffic infrastructure according to any one of claims 1 to 5, characterized in that: The determining of the abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map comprises: Compare the target two-dimensional feature map with the second mapped feature map pixel by pixel based on a preset difference function to obtain a first difference in a two-dimensional state, and compare the target three-dimensional feature map with the first mapped feature map pixel by pixel based on the preset difference function to obtain a second difference in a three-dimensional state; An abnormality score determined based on the first difference and the second difference determines an abnormal state image among the corresponding facility state images.

7. The abnormality alarm method of traffic infrastructure according to claim 6, characterized in that: The determining of the abnormal state image in the corresponding facility state image according to the abnormality score determined based on the first difference and the second difference includes: The first difference and the second difference are fused based on an aggregation function to obtain a fused difference, and the fused difference is smoothed based on a preset Gaussian kernel to obtain a processed anomaly map; An abnormal state image in the corresponding facility state image is determined based on the abnormal score corresponding to the processed abnormal image.

8. An abnormal alarm device for traffic infrastructure, characterized in that: include: An image acquisition module, used to acquire facility status images of target transportation infrastructure acquired by a data acquisition device; The facility status image includes an RGB image and a point cloud image; A feature extraction module, used to respectively extract the two-dimensional features corresponding to the RGB image and the three-dimensional features corresponding to the point cloud image using a preset feature extraction technology, and perform a pixel alignment operation on the two-dimensional features and the three-dimensional features to obtain a target two-dimensional feature map and a target three-dimensional feature map aligned at the pixel level; A feature mapping module, configured to perform cross-modal mapping on the target two-dimensional feature map based on a preset feature mapping function to obtain a corresponding first mapped feature map, and to perform cross-modal mapping on the target three-dimensional feature map to obtain a corresponding second mapped feature map; The first mapped feature map is a three-dimensional feature map, and the second mapped feature map is a two-dimensional feature map; an abnormality determination module, configured to determine an abnormal state image in the facility state image based on a first difference between the target two-dimensional feature map and the second mapped feature map and a second difference between the target three-dimensional feature map and the first mapped feature map; A fault alarm module is used to determine the target facilities with abnormalities in the target traffic infrastructure based on the abnormal state image, and to perform fault abnormality alarm based on a preset abnormality alarm method.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the abnormality alarm method for traffic infrastructure as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the abnormal alarm method for traffic infrastructure as described in any one of claims 1 to 7.

Images