Road anomaly object detection method, device and equipment and storage medium

Abstract

The application relates to the technical field of object detection, and discloses a road abnormal object detection method, device and equipment and a storage medium, the method comprising the following steps: inputting a to-be-detected image into a preset abnormality detection model to obtain a corresponding abnormality prediction image, the preset abnormality detection model is used for image preprocessing on the input image, and abnormal object features are determined according to the preprocessed input image; performing superpixel segmentation on the abnormality prediction image to obtain a superpixel segmentation image; scoring the superpixel segmentation image through a preset scoring formula to obtain an abnormality scoring image; and determining corresponding road abnormal objects in the to-be-detected image according to the abnormality scoring image. The to-be-detected image is preprocessed through the preset abnormality detection model to determine the abnormality prediction image, then superpixel segmentation is performed and an abnormality scoring image is output, so that the road abnormal objects in the to-be-detected image are strengthened, the difference detection is more targeted, and effective detection of the road abnormal objects is realized.

Description

Technical Field

[0001] This invention relates to the field of object detection technology, and in particular to a method, apparatus, equipment and storage medium for detecting abnormal objects on roads. Background Technology

[0002] In modern urban development, road network planning has become increasingly convenient, and car travel has become one of the main modes of transportation. With the increasing number of cars on the road, Advanced Driving Assistance Systems (ADAS) have gradually become standard features in various vehicles. ADAS can analyze and calculate data in real time during vehicle operation through environmental perception, detection, and tracking, thereby providing passive warnings to the driver or proactive intervention to take over the vehicle and prevent accidents. For example, it can accurately identify abnormal objects on the road and provide timely feedback to avoid accidents.

[0003] However, the appearance of abnormal objects on the road can be classified as a sudden abnormal situation. Its biggest feature is that it is not common in the road environment and has a complex shape. The existing methods are not very accurate in detecting abnormal objects that are far away or small.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for detecting abnormal objects on roads, aiming to solve the technical problem that the accuracy of traditional methods for detecting abnormal objects is not high.

[0006] To achieve the above objectives, the present invention provides a method for detecting abnormal objects on roads, the method comprising:

[0007] The image to be detected is input into a preset anomaly detection model to obtain an anomaly prediction image corresponding to the image to be detected. The preset anomaly detection model is used to preprocess the input image and determine the features of the abnormal object based on the preprocessed input image.

[0008] Perform superpixel segmentation on the anomaly prediction image to obtain the superpixel segmentation image corresponding to the anomaly prediction image;

[0009] The superpixel segmentation image is scored using a preset scoring formula to obtain anomaly scoring images;

[0010] The abnormal road objects in the image to be detected are determined based on the abnormal scoring image.

[0011] Optionally, the preset anomaly detection model includes a segmentation module, a synthesis module, a depth module, and a difference module; the step of inputting the image to be detected into the preset anomaly detection model to obtain the anomaly prediction image corresponding to the image to be detected includes:

[0012] The image to be detected is input into the segmentation module for semantic segmentation to obtain the semantic image and uncertainty image corresponding to the image to be detected.

[0013] The image to be detected, the semantic image, and the uncertain image are input into the synthesis module for image processing to obtain a synthesized image and an abnormal image.

[0014] The image to be detected and the corresponding depth disparity map are input into the depth module for feature mapping to obtain the feature fusion image corresponding to the image to be detected.

[0015] The image to be detected, the synthesized image, the feature fusion image, the semantic image, the abnormal image, and the uncertainty image are input into the difference module for anomaly prediction to obtain the anomaly prediction image corresponding to the image to be detected.

[0016] Optionally, the step of inputting the image to be detected, the semantic image, and the uncertain image into the synthesis module for image processing to obtain a synthesized image and an anomaly image includes:

[0017] Obtain the semantic mapping features corresponding to the image to be detected when the segmentation module performs semantic segmentation;

[0018] The semantic mapping features are input into the synthesis module to perform image synthesis, thereby obtaining the synthesized image corresponding to the image to be detected;

[0019] The perceptual loss corresponding to the image to be detected and the synthesized image is determined according to a preset feature extractor;

[0020] The abnormal image corresponding to the image to be detected is obtained based on the perceptual loss and the uncertainty image.

[0021] Optionally, the depth module includes an original image network branch, a deep network branch, and an upsampling module; the image to be detected and its corresponding depth disparity map are input into the depth module for feature mapping to obtain a feature fusion image corresponding to the image to be detected, including:

[0022] Obtain the depth disparity map corresponding to the image to be detected, the depth disparity map reflecting the distance and shape of objects in the image;

[0023] The image to be detected is input into the original image network branch for feature mapping to obtain the original feature mapping map corresponding to the image to be detected.

[0024] The depth disparity map is input into the deep network branch for feature mapping to obtain the depth feature map corresponding to the depth disparity map.

[0025] The original feature map and the deep feature map are upsampled by the upsampling module to obtain the feature fusion image corresponding to the image to be detected.

[0026] Optionally, the difference module includes a convolution module, a fusion module, and a decoder module; the step of inputting the image to be detected, the synthesized image, the feature fusion image, the semantic image, the anomaly image, and the uncertainty image into the difference module for anomaly prediction to obtain the anomaly prediction image corresponding to the image to be detected includes:

[0027] The image to be detected, the synthesized image, the feature fusion image, the semantic image, the abnormal image, and the uncertain image are input into the convolution module for convolution processing to obtain multiple feature mapping images;

[0028] Each feature-mapped image is input into the fusion module for image fusion to obtain a fused image;

[0029] The decoder module performs convolutional filtering on the fused image to obtain the anomaly prediction image corresponding to the image to be detected.

[0030] Optionally, performing superpixel segmentation on the anomaly prediction image to obtain a superpixel segmented image corresponding to the anomaly prediction image includes:

[0031] The anomaly prediction image is converted to a color space, and the five-dimensional vector coordinates corresponding to each pixel are obtained based on the conversion result.

[0032] Cluster each pixel to obtain multiple cluster center groups;

[0033] The average vector value of each cluster center group is obtained based on the five-dimensional vector coordinates, and the clustering convergence of each cluster center group is performed based on the average vector value to obtain the superpixel segmentation image corresponding to the anomaly prediction image.

[0034] Optionally, before inputting the image to be detected into a preset anomaly detection model to obtain the anomaly prediction image corresponding to the image to be detected, the method further includes:

[0035] Anomalies are marked on a pre-defined training dataset to obtain a labeled dataset;

[0036] Image preprocessing is performed on the labeled dataset to obtain the original semantic image set, the synthetic image set, the uncertain image set, and the abnormal image set corresponding to the labeled dataset;

[0037] The original semantic image set, the synthetic image set, the uncertain image set, and the abnormal image set are input into the initial model for training to obtain a preset anomaly detection model.

[0038] Furthermore, to achieve the above objectives, the present invention also proposes a road abnormality detection device, the device comprising:

[0039] An anomaly prediction module is used to input the image to be detected into a preset anomaly detection model to obtain an anomaly prediction image corresponding to the image to be detected. The preset anomaly detection model is used to perform image preprocessing on the input image and determine the features of the abnormal object based on the preprocessed input image.

[0040] A pixel segmentation module is used to perform superpixel segmentation on the anomaly prediction image to obtain a superpixel segmented image corresponding to the anomaly prediction image.

[0041] An anomaly scoring module is used to score the superpixel segmentation image using a preset scoring formula to obtain an anomaly scoring image;

[0042] The foreign object identification module is used to identify the corresponding abnormal road objects in the image to be detected based on the abnormality scoring image.

[0043] Furthermore, to achieve the above objectives, the present invention also proposes a road anomaly detection device, the device comprising: a memory, a processor, and a road anomaly detection program stored in the memory and executable on the processor, the road anomaly detection program being configured to implement the steps of the road anomaly detection method as described above.

[0044] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a road anomaly detection program, which, when executed by a processor, implements the steps of the road anomaly detection method as described above.

[0045] This invention inputs the image to be detected into a preset anomaly detection model to obtain an anomaly prediction image corresponding to the image to be detected. The preset anomaly detection model is used to preprocess the input image and determine the features of abnormal objects based on the preprocessed input image. Then, superpixel segmentation is performed on the anomaly prediction image to obtain a superpixel segmentation image corresponding to the anomaly prediction image. Next, the superpixel segmentation image is scored using a preset scoring formula to obtain an anomaly scoring image. Finally, the corresponding road anomaly in the image to be detected is determined based on the anomaly scoring image. This invention enhances the localization of road anomalies in the image to be detected by using a preset anomaly detection model to preprocess the image to be detected to determine the anomaly prediction image, then performing superpixel segmentation and outputting an anomaly scoring image, making the difference detection more targeted and achieving effective detection of anomalies. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the structure of a road abnormal object detection device in the hardware operating environment involved in the embodiments of the present invention;

[0047] Figure 2 This is a flowchart illustrating the first embodiment of the road anomaly detection method of the present invention;

[0048] Figure 3 This is a flowchart illustrating the second embodiment of the road anomaly detection method of the present invention;

[0049] Figure 4 This is a network structure diagram of the depth module in the second embodiment of the road anomaly detection method of the present invention;

[0050] Figure 5 This is a network structure diagram of the difference module in the second embodiment of the road anomaly detection method of the present invention;

[0051] Figure 6 This is a flowchart illustrating the third embodiment of the road anomaly detection method of the present invention;

[0052] Figure 7 This is a superpixel segmentation result image of the third embodiment of the road anomaly detection method of the present invention;

[0053] Figure 8 This is a structural block diagram of the first embodiment of the road abnormal object detection device of the present invention.

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] Reference Figure 1 , Figure 1 This is a schematic diagram of the road abnormal object detection device structure in the hardware operating environment involved in the embodiments of the present invention.

[0057] like Figure 1 As shown, the road anomaly detection device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the road anomaly detection device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a road anomaly detection program.

[0060] exist Figure 1 In the road anomaly detection device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the road anomaly detection device of the present invention can be set in the road anomaly detection device, and the road anomaly detection device calls the road anomaly detection program stored in the memory 1005 through the processor 1001 and executes the road anomaly detection method provided in the embodiment of the present invention.

[0061] This invention provides a method for detecting abnormal objects on roads, referring to... Figure 2 , Figure 2This is a flowchart illustrating the first embodiment of the road anomaly detection method of the present invention.

[0062] In this embodiment, the method for detecting abnormal objects on the road includes the following steps:

[0063] Step S10: Input the image to be detected into the preset anomaly detection model to obtain the anomaly prediction image corresponding to the image to be detected. The preset anomaly detection model is used to preprocess the input image and determine the features of the abnormal object based on the preprocessed input image.

[0064] It should be noted that the executing entity of the method in this embodiment can be a computing service device with abnormal object detection function, such as an ADAS driver assistance system, or other electronic devices capable of performing the same or similar functions, such as the aforementioned road abnormal object detection device. This embodiment does not limit this. Here, the aforementioned road abnormal object detection device (hereinafter referred to as the detection device) will be used to specifically describe this embodiment and the following embodiments.

[0065] Understandably, the image to be detected is a raw image of a road scene taken while a vehicle is driving on the road. The preset anomaly detection model is a model pre-set in the detection device to predict abnormal objects in the input image. Therefore, the preset anomaly detection model performs image preprocessing on the input image, such as image semantic segmentation, synthesis processing, deep feature extraction, and anomaly feature prediction, making the difference detection more targeted, achieving effective detection of abnormal objects, and thus outputting an anomaly prediction image.

[0066] In practice, the detection device inputs the image to be detected (i.e. the original image) into a preset anomaly detection model. The model performs a series of image preprocessing on the image to be detected, such as image semantic segmentation, depth feature extraction and anomaly feature prediction, to make the difference detection more targeted and achieve effective detection of abnormal objects, thereby outputting an anomaly prediction image.

[0067] Step S20: Perform superpixel segmentation on the anomaly prediction image to obtain the superpixel segmentation image corresponding to the anomaly prediction image.

[0068] It should be noted that superpixel segmentation is an image segmentation technique that divides a digital image into multiple sub-regions (sets of pixels, also known as superpixels). Superpixel segmentation can classify pixels with adjacent locations and similar characteristics such as color, texture, and brightness, thus processing the image in the form of small regions. It can also highlight the boundary information of objects, improving the segmentation accuracy of smaller pixel regions in the image, thereby enhancing the feature identification of abnormal objects in anomaly prediction images.

[0069] Step S30: The superpixel segmentation image is scored using a preset scoring formula to obtain an abnormal scoring image.

[0070] Step S40: Determine the corresponding road anomaly in the image to be detected based on the anomaly scoring image.

[0071] Understandably, the preset scoring formula is a scoring formula preset in the detection device, used to score the anomalies of each superpixel region in the superpixel segmentation image. The final score is compared with a preset threshold, and the regions that exceed the threshold are identified as regions containing abnormal objects, thereby obtaining an anomaly scoring image.

[0072] It should be understood that road anomalies are abnormal objects identified in an image after the image to be detected is examined by detection equipment.

[0073] In its implementation, the detection device performs superpixel segmentation on the anomaly prediction image, dividing it into multiple image sub-regions (superpixels) to obtain a superpixel segmentation image. Then, anomaly scoring is applied to each superpixel region in the superpixel segmentation image using a preset scoring formula, resulting in an anomaly scoring image. Finally, based on the scoring results in the anomaly scoring image, the corresponding road anomaly in the image to be detected is determined.

[0074] This embodiment inputs the image to be detected into a preset anomaly detection model. The model performs a series of image preprocessing steps on the image, such as semantic segmentation, depth feature extraction, and anomaly feature prediction, making the difference detection more targeted and effectively detecting abnormal objects, thus outputting an anomaly prediction image. The detection device then performs superpixel segmentation on the anomaly prediction image, subdividing it into multiple image sub-regions (superpixels) to obtain a superpixel segmentation image. Next, a preset scoring formula is used to score each superpixel region in the superpixel segmentation image, obtaining an anomaly scoring image. Finally, based on the scoring results in the anomaly scoring image, the corresponding road anomaly in the image to be detected is determined. Compared to traditional segmentation methods that cannot detect objects outside the training set distribution, this embodiment uses a preset anomaly detection model to perform a series of image preprocessing steps, making the difference detection more targeted and effectively detecting abnormal objects. Then, by performing superpixel segmentation on the anomaly detection image output by the preset anomaly detection model and outputting an anomaly scoring image, the localization of abnormal objects in the input image is strengthened. The anomaly scoring formula is designed to focus more on anomalies on the road and ignore anomalies in other scenes as much as possible, reducing false positives.

[0075] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the road anomaly detection method of the present invention.

[0076] Based on the first embodiment described above, in this embodiment, considering the accuracy of the preset anomaly detection model, the preset anomaly detection model includes a segmentation module, a synthesis module, a depth module, and a difference module; step S10 includes:

[0077] Step S11: Input the image to be detected into the segmentation module for semantic segmentation to obtain the semantic image and uncertainty image corresponding to the image to be detected.

[0078] It should be noted that the segmentation module, synthesis module, depth module, and difference module are functional modules in the pre-defined anomaly detection model. The output of the segmentation module is connected to the input of the synthesis module, the output of the synthesis module is connected to the input of the difference module, and the output of the depth module is connected to the input of the difference module. Specifically, the segmentation module performs semantic segmentation on the image, obtaining semantic features and generating a semantic map and an uncertainty map (the uncertainty map contains uncertain values ​​such as the discrete metrics softmax entropy U and softmax distance D). The synthesis module further extracts features from the input image, reducing the ambiguity of the semantic mapping and improving the image's significant differences. The depth module performs multi-scale feature mapping on the image to be detected, allowing the basic segmentation method to utilize supplementary information from the depth map, improving segmentation accuracy and facilitating its application in the difference module. The difference module performs anomaly detection on the input and output images of the above modules. Through the processing of these modules, the influence of unknown noise on the difference module's comparison process can be reduced, making difference detection more targeted and achieving effective detection of abnormal objects on roads.

[0079] Understandably, a semantic image is an image obtained by the segmentation module through a segmentation network performing semantic segmentation. Semantic segmentation can perform dense predictions on each pixel in the image to obtain semantic features, thereby achieving pixel-by-pixel category labeling. Uncertainty images utilize discrete metrics to quantify the uncertainty in semantic image prediction; for example, two discrete metrics, softmax entropy U and softmax distance D, can be used.

[0080] It's important to note that semantic segmentation algorithms are commonly used in image detection and classification. They can perform dense predictions for each pixel, achieving pixel-by-pixel category labeling. For example, Fully Convolutional Networks (FCNs) are advantageous because they are unaffected by the size of the input image and replace fully connected layers with skip layers and deconvolutional layers, thus achieving pixel-level image segmentation. However, for semantic segmentation methods to identify all categories in testing, the model must have seen these categories during training. This is clearly unrealistic in some complex scenarios, such as suddenly appearing abnormal objects in road traffic scenes. Since these abnormal objects haven't appeared in the model's training, they are likely to be predicted as known categories or completely unpredictable, which would be fatal for autonomous driving scenarios. Therefore, the probability from the softmax entropy U and softmax distance D distributions can be used to measure uncertainty, and simple statistics can be used to classify samples as out-of-distribution samples, reducing false positive predictions for images.

[0081] For ease of understanding, the functional process of the segmentation module is described, but this does not limit the scope of the proposed solution. The segmentation module acquires the image to be detected and feeds it into the segmentation network to obtain a semantic image. Based on the semantic image, two discrete metrics are calculated to quantify the uncertainty in the prediction of the image to be detected. These two discrete metrics can be the softmax entropy U and the softmax distance D (i.e., the difference between the two largest softmax values), which play a positive role in understanding the error within the segmentation. For a pixel x in the semantic image, these two metrics are calculated as follows:

[0082]

[0083]

[0084] Here, p(c) is the softmax probability of class c, and both quantities can be standardized to [0,1].

[0085] In the specific implementation, the image to be detected is input into the segmentation module for semantic segmentation. Dense prediction is performed on each pixel in the image to obtain semantic features in the image, and pixel-by-pixel category labeling is achieved to obtain semantic features in the image. Then, semantic map and uncertainty map are obtained based on semantic features.

[0086] Step S12: Input the image to be detected, the semantic image, and the uncertain image into the synthesis module for image processing to obtain a synthesized image and an abnormal image.

[0087] It should be noted that the synthesized image is a real image generated by the synthesis module through the semantic mapping given in the semantic image. There is a correspondence between the pixels of the synthesized image and the pixels of the semantic image. However, there will be abnormal appearance differences between the resynthesized image and the input image. Therefore, abnormal objects can be located by using these abnormal appearance differences.

[0088] It should be understood that the abnormal image is an image obtained by the synthesis module through further feature mapping based on the difference between the image to be detected and the synthesized image, in order to improve the significant difference between the synthesized image and the image to be detected.

[0089] Understandably, there will be differences in appearance between the synthesized image generated by the synthesis module and the image to be detected. Therefore, these differences in appearance can be used to locate abnormal objects. An adversarial network can be used in the synthesis module to synthesize a new input image using the semantic feature maps generated by the segmentation module, thus better utilizing the feature differences between the input and synthesized images. The advantage of this method is that it does not rely on the segmentation quality of the segmentation network and does not require retraining the segmentation network, thereby improving the model's detection performance of abnormal objects in the scene.

[0090] In practice, the detection device generates a composite image by using a semantic mapping given in the semantic image, and then further obtains anomaly images by feature mapping based on the image to be detected and the uncertain image, so as to improve the significant difference between the composite image and the image to be detected.

[0091] Step S13: Input the image to be detected and the depth disparity map corresponding to the image to be detected into the depth module for feature mapping to obtain the feature fusion image corresponding to the image to be detected.

[0092] It should be noted that the depth disparity map is the disparity image corresponding to the image to be detected. Compared with the ordinary image to be detected, the depth disparity map has undergone further image processing, which can reflect the distance and shape features of each object in the image.

[0093] Understandably, the depth module utilizes the appearance information of the original RGB image of the image to be detected and the position and contour information of the depth disparity map. By fusing these two as inputs to the depth module, features containing more information in the resulting feature fusion image can be given higher weight values. This allows the basic segmentation method to utilize the supplementary information in the depth disparity map, which helps to improve the segmentation accuracy and makes it more suitable for application in the disparity module.

[0094] In its implementation, the detection device inputs the image to be detected and its corresponding depth disparity map into the depth module for feature mapping, obtaining a feature fusion image. This allows the basic segmentation method to utilize supplementary information from the depth disparity map, improving segmentation accuracy and facilitating its application in the disparity module.

[0095] Step S14: Input the image to be detected, the synthesized image, the feature fusion image, the semantic image, the abnormal image, and the uncertainty image into the difference module to perform anomaly prediction and obtain the anomaly prediction image corresponding to the image to be detected.

[0096] It should be noted that, through the difference module, the features of the above images can be effectively integrated and utilized, thereby improving the prediction performance of the anomaly segmentation map.

[0097] In the specific implementation, the image to be detected, the synthesized image, the feature fusion image, the semantic image, the abnormal image, and the uncertain image are input into the difference module to perform operations such as feature extraction, fusion, and feature mapping to predict abnormal objects in the image and obtain the abnormal prediction image.

[0098] Furthermore, considering the accuracy of the synthesis module, in this embodiment, step S12 includes: acquiring the semantic mapping features corresponding to the image to be detected when the segmentation module performs semantic segmentation; inputting the semantic mapping features into the synthesis module to perform image synthesis to obtain the synthesized image corresponding to the image to be detected; determining the perceptual loss corresponding to the image to be detected and the synthesized image according to a preset feature extractor; and obtaining the abnormal image corresponding to the image to be detected according to the perceptual loss and the uncertainty image.

[0099] It should be noted that the preset feature extractor is a preset extractor in the detection model. It can extract features from the image and find the pixel with the greatest feature difference. By detecting abnormal objects or classification errors, it can represent the erroneous features of the synthesized image.

[0100] Understandably, the perceptual loss is a quantized loss calculated from the difference between the detected image and the synthesized image. The semantic mapping around anomalous objects is generally ambiguous, and it is precisely this ambiguity that causes a significant difference between the synthesized image regenerated from the semantic mapping and the image to be detected. Therefore, the ambiguity can be measured by the softmax entropy U in the uncertain image, and this, combined with the perceptual loss, can be used to measure the appearance difference, thereby obtaining the anomalous image.

[0101] For ease of understanding, the synthesis process of the synthesis module is used as an example, but this does not limit the scope of the proposed solution. The synthesis module can generate realistic synthetic images using given semantic mapping features, where there is a correspondence between pixels. The synthesis module is trained using a conditional generative adversarial network (GAN) to match the semantic distribution of the input image with that of the synthetic image. Since the semantic image itself lacks color and appearance information, the synthetic image cannot perfectly reproduce color and appearance, allowing for pixel-by-pixel comparison. Therefore, objects can be compared by calculating the perceptual difference between the target image and the synthetic image, rather than based on low-level features such as color and texture. ImageNet pre-trained VGG can be used as a feature extractor to extract features and find the pixels with the largest feature difference. Errors in the synthetic image's feature representation can be detected by identifying anomalous objects or classification errors. Therefore, perceptual differences are more sensitive to these differences and can be represented by a perceptual loss. For a pixel x in the target image and its corresponding pixel r in the synthetic image, the perceptual loss is calculated as follows:

[0102]

[0103] Among them, F (i) This is the output of the i-th feature layer of the VGG network. Since the performance of the difference module in detecting anomalous objects is related to the choice of feature layers, deeper feature layers may miss anomalous objects. Therefore, the outputs of the first 4 feature layers can be selected, and this discrete metric can be normalized to [0,1].

[0104] Since the semantic mapping features around anomalous objects are generally ambiguous, it is precisely because of this ambiguity that the synthesized image regenerated from the semantic mapping differs significantly from the input image. Therefore, the ambiguity can be measured by the softmax entropy U in the aforementioned uncertain image, and the appearance difference can be measured by the calculation of the perceptual loss L. By multiplying these two metrics, an anomalous image A with deep features is generated.

[0105]

[0106] Furthermore, considering the accuracy of the depth module, the depth module includes an original image network branch, a depth network branch, and an upsampling module. In this embodiment, step S13 includes: obtaining a depth disparity map corresponding to the image to be detected, the depth disparity map reflecting the distance and shape of objects in the image; inputting the image to be detected into the original image network branch for feature mapping to obtain an original feature map corresponding to the image to be detected; inputting the depth disparity map into the depth network branch for feature mapping to obtain a depth feature map corresponding to the depth disparity map; and upsampling the original feature map and the depth feature map through the upsampling module to obtain a feature fusion image corresponding to the image to be detected.

[0107] It should be noted that the outputs of the original image network branch and the deep network branch are connected to the input of the upsampling module, respectively. The original image network branch is used to extract feature information such as the appearance of the RGB original image of the image to be detected, and the deep network branch is used to extract feature information such as the position and contour of the depth disparity map. The upsampling module is used to upsample the feature maps extracted by the original image network branch and the deep network branch to restore the resolution of these feature maps.

[0108] For ease of understanding, the process of feature mapping using deep modules is explained, but this does not limit the scope of this approach. Figure 4 As shown, Figure 4 This is a network structure diagram of the depth module in the second embodiment of the road anomaly detection method of the present invention. The depth module utilizes the appearance information of the image to be detected (Input) and the position and contour information of the depth disparity map (Depth Input), and uses both as inputs to the depth module. ResNet-18 can be used as the backbone network to extract features, and the network structure is as follows. Figure 4 As shown, the upper part of the network encoder extracts features from the input image using two branches: the original image network branch and the deep network branch. After each layer of the ResNet-18 network, the Attention Feature Complementary (AFC) module fuses the features from these two branches. Finally, the spatial feature pyramid module generates feature maps with multi-scale information (Spatial pyramid pooling), and the resolution of these feature maps is restored through upsampling to obtain the fused feature image.

[0109] In the above structure, two feature maps can be obtained, namely the original feature map O. in =[O in1 L, O inC] and deep feature map D in =[D in1 L, D inC The AFC module introduces two channel attention mechanisms to process the feature maps of the two branches respectively. It uses global average pooling to process the feature maps as channel descriptors, then adds a 1×1 convolutional layer (1×1 Conv layer) with the same number of channels for full connection, and finally activates the convolution result using the sigmoid function, restricting the values ​​of the weight matrix to between 0 and 1. After obtaining the attention weight matrices of the two branches, they are respectively out-productted with the corresponding input feature maps. The results of the two branches are then added together to obtain the final feature fusion image. , means as follows:

[0110]

[0111] in, Let φ represent the outer product, φ represent the calculation of global average pooling and 1×1 convolution, and σ represent the sigmoid function. By introducing a channel attention mechanism into the original image network branch and the deep network branch, features containing more information in the feature map can be given higher weight values, allowing the basic segmentation method to utilize supplementary information in the depth map. This further helps to improve the accuracy of segmentation and is better applied in the difference module.

[0112] Furthermore, considering the accuracy of the difference module, the difference module includes a convolution module, a fusion module, and a decoder module; in this embodiment, step S14 includes: inputting the image to be detected, the synthesized image, the feature fusion image, the semantic image, the abnormal image, and the uncertainty image into the convolution module for convolution processing to obtain multiple feature mapping images; inputting each feature mapping image into the fusion module for image fusion to obtain a fused image; and performing convolution filtering on the fused image according to the decoder module to obtain the abnormal prediction image corresponding to the image to be detected.

[0113] It should be noted that the output of the convolution module is connected to the input of the fusion module, and the output of the fusion module is connected to the input of the decoder module. The convolution module is used to extract feature maps from the image to be detected, the synthesized image, the feature fusion image, the semantic image, the anomaly image, and the uncertain image using a convolutional network. The fusion module is used to connect the feature maps at different resolutions and perform a fully connected operation on the total feature map. The decoder module is used to perform convolutional filtering on the features output by the fusion module.

[0114] For ease of understanding, the process of anomaly prediction using the difference module is explained, but this does not limit the scope of this solution. (Refer to...) Figure 5 , Figure 5 This is a network structure diagram of the difference module in the second embodiment of the road anomaly detection method of the present invention. The difference module utilizes multiple input features to output an anomaly prediction map. These input features include the image to be detected (Input), the synthesized image, the depth image, the semantic image, and the uncertainty image (e.g., softmax distance) and anomaly map calculated by the segmentation network. By using the difference network, these features can be effectively integrated and utilized, thereby improving the prediction performance of the anomaly segmentation map. The network structure is as follows: Figure 5 As shown.

[0115] The difference module comprises two convolutional modules (CNN and VGG16 convolutional modules), two fusion modules, and four decoder modules. The CNN and VGG16 convolutional modules share the same structure, consisting of three max-pooling layers, each outputting a feature map. These feature maps are then combined with the final feature map output by the fusion module, resulting in a total of four feature maps. The VGG16 convolutional module shares weights for encoding the image to be detected and the synthesized image. The CNN convolutional module consists of one 7×7 convolutional layer and three 3×3 convolutional layers, each outputting a feature map. The fusion module encodes the semantic image, the synthesized image, and the uncertain image, using different weights for each type of information. The fusion module concatenates the feature maps of the input image, semantic image, and synthesized image at each different resolution, and then fully correlates the total feature map using a 1×1 convolutional layer. Finally, the result of the 1×1 convolution is correlated point-by-point with the discrete feature maps. The fusion module outputs a feature map at each resolution. The decoder module uses four decoders. The first and second decoders have the same structure, containing two 3×3 convolutional layers with the same number of filters, two SPADE-normalized SELU layers, and a 2×2 transposed convolutional layer. The third decoder contains two 3×3 convolutional layers with different numbers of filters, two SPADE-normalized SELU layers, and a 2×2 transposed convolutional layer. The fourth decoder contains two 3×3 convolutional layers with different numbers of filters, two SPADE-normalized SELU layers, and a 1×1 transposed convolutional layer. The first decoder obtains the lowest resolution feature map, the second decoder obtains the concatenation result of the feature map from the fusion module and the output of the first decoder as its output, and the inputs of subsequent decoders follow the same pattern.

[0116] This embodiment segments the image to be detected and inputs the predicted semantic image into the synthesis module, which then synthesizes a composite image highly similar to the semantic image. Since there are feature differences between the composite image and the input image, the perceptual differences can be calculated using the different features of the two images. To enhance the difference module's ability to extract differential features of abnormal objects, an RGB-D network is used to extract features from two branches: the image to be detected and the depth disparity map. The feature mappings of the two branches are then processed using a channel attention mechanism, fusing them into a feature fusion image. Finally, all processed images and the image to be detected are sent to the difference module for training to obtain an anomaly prediction image. This embodiment utilizes the complementary characteristics of auxiliary and resynthetic methods to reduce the impact of unknown noise on the difference module's comparison process, making difference detection more targeted and achieving effective detection of abnormal objects, thereby improving the module's ability to detect abnormal objects on roads.

[0117] refer to Figure 6 , Figure 6 This is a flowchart illustrating the third embodiment of the road anomaly detection method of the present invention.

[0118] Based on the above embodiments, in this embodiment, considering the accuracy of superpixel segmentation, step S20 includes:

[0119] Step S21: Perform color space conversion on the anomaly prediction image and obtain the five-dimensional vector coordinates corresponding to each pixel based on the conversion result.

[0120] Step S22: Cluster each pixel to obtain multiple cluster center groups.

[0121] Step S23: Obtain the average vector value of each cluster center group based on the five-dimensional vector coordinates, and perform clustering convergence on each cluster center group based on the average vector value to obtain the superpixel segmentation image corresponding to the anomaly prediction image.

[0122] It should be noted that superpixel segmentation technology can classify pixels with adjacent positions and similar characteristics such as color, texture, and brightness, thereby processing the image in the form of small regions. It can also highlight the boundary information of objects and improve the segmentation accuracy of small pixel regions in the image.

[0123] For ease of understanding, the superpixel segmentation process is used as an example, but this method is not limited to it. Superpixel segmentation converts the image to the CIELAB color space and combines the color value and location information of each pixel into a five-dimensional vector. Using a clustering-based approach, a set of seed points is generated, and the nearest pixels to each seed point are searched and grouped together until all pixels are classified. Then, the average vector value of each superpixel (i.e., the cluster center) is calculated to obtain new cluster centers, and the search for surrounding pixels is repeated until convergence. Figure 7 As shown, Figure 7 This is a superpixel segmentation result image of the third embodiment of the road anomaly object detection method of the present invention. The left image shows the anomaly prediction image output by the difference module. The right image shows the corresponding superpixel segmentation image. As can be seen from the right image, after superpixel segmentation, the scene is divided into many grid regions, and the boundaries of objects in the scene are clearly distinguished, reducing the processing range and utilizing more refined features for subsequent processing.

[0124] Furthermore, based on the aforementioned superpixel segmentation, this embodiment proposes a formula for calculating anomaly scores. The input to this formula is the anomaly prediction map from the difference module. Based on this, the score for anomaly objects in the i-th superpixel is defined as follows:

[0125]

[0126] Where, α i It is the average of the anomaly scores in the i-th superpixel, n j p is the number of pixels in the j-th superpixel. j r is the average probability of the road label in the j-th superpixel. i,j ω is the Euclidean distance between the center positions of the i-th and j-th superpixels, and ω is the median of the Euclidean distances between the center positions of each pair of superpixels. The final score is standardized to [0,1]. Then, by setting a threshold, the calculated S... i Compared to a threshold, areas exceeding the threshold are identified as containing anomalous objects. This anomaly scoring method focuses more on anomalous objects in the road, assigning them higher scores and sharpening the anomaly map of each local area, thus obtaining more reasonable anomaly scoring results.

[0127] Furthermore, considering the efficiency of image anomaly prediction, this embodiment further includes the following steps before step S10: anomaly labeling of a preset training dataset to obtain a labeled dataset; image preprocessing of the labeled dataset to obtain the original semantic image set, synthetic image set, uncertain image set, and anomaly image set corresponding to the labeled dataset; and inputting the original semantic image set, the synthetic image set, the uncertain image set, and the anomaly image set into an initial model for training to obtain a preset anomaly detection model.

[0128] It should be noted that the Cityscapes dataset can be used as the preset training dataset. The labeled dataset is obtained by labeling all void classes in the Cityscapes dataset as anomalies. This allows for the training of a more robust anomaly detection model, ensuring that the model still has good generalization and robustness in detecting anomalous objects.

[0129] In the specific implementation, the Cityscapes dataset can be used as the preset training dataset, the FS Lost and Found dataset from the Fisherscapes dataset can be used as the evaluation dataset, and the Lost and Found dataset and the RoadObstacle21 dataset can be used as the test dataset. The training dataset for the difference module is based on the Cityscapes dataset. Before training, the Cityscapes dataset can be preprocessed to obtain the original semantic image set, the synthetic image set, the uncertain image set, and the anomaly image set. Then, the image set to be detected and the feature fusion image set are input together into the initial module for training to obtain the preset anomaly detection model, thereby improving the model's generalization ability.

[0130] This embodiment enhances the localization of abnormal objects in the input image by performing superpixel segmentation on the abnormal detection image output by the preset abnormal detection model and outputting an abnormal scoring map. Through the design of the abnormal scoring formula, it focuses more on abnormalities on the road and ignores abnormalities in other scenes as much as possible, reducing false positives. It has a high detection rate for abnormal objects seen in the training set as well as those outside the training set, greatly enhancing the generalization of abnormal object detection and thus improving the accuracy of abnormal object detection on the road.

[0131] Furthermore, this embodiment of the invention also proposes a storage medium storing a road anomaly detection program, which, when executed by a processor, implements the steps of the road anomaly detection method described above.

[0132] Reference Figure 8 , Figure 8 This is a structural block diagram of the first embodiment of the road abnormal object detection device of the present invention.

[0133] like Figure 8 As shown, the road anomaly detection device proposed in this embodiment of the invention includes:

[0134] An anomaly prediction module 801 is used to input the image to be detected into a preset anomaly detection model to obtain an anomaly prediction image corresponding to the image to be detected. The preset anomaly detection model is used to perform image preprocessing on the input image and determine the features of the abnormal object based on the preprocessed input image.

[0135] The pixel segmentation module 802 is used to perform superpixel segmentation on the anomaly prediction image to obtain a superpixel segmented image corresponding to the anomaly prediction image.

[0136] Anomaly scoring module 803 is used to score the superpixel segmentation image using a preset scoring formula to obtain anomaly scoring image;

[0137] The foreign object determination module 804 is used to determine the corresponding abnormal road objects in the image to be detected based on the abnormality scoring image.

[0138] This embodiment inputs the image to be detected (i.e., the original image) into a preset anomaly detection model. The model performs a series of image preprocessing steps on the image to be detected, such as image semantic segmentation, depth feature extraction, and anomaly feature prediction, making the difference detection more targeted and achieving effective detection of abnormal objects, thereby outputting an anomaly prediction image. Then, the detection device performs superpixel segmentation on the anomaly prediction image, subdividing the anomaly prediction image into multiple image sub-regions (superpixels) to obtain a superpixel segmentation image. Next, anomaly scoring is performed on each superpixel region in the superpixel segmentation image using a preset scoring formula to obtain an anomaly scoring image. Finally, the corresponding road anomaly object in the image to be detected is determined based on the scoring results in the anomaly scoring image. Compared to traditional segmentation methods that cannot detect objects outside the training set distribution, this embodiment uses a series of image preprocessing steps through a preset anomaly detection model to make the difference detection more targeted and achieve effective detection of abnormal objects. Then, by performing superpixel segmentation on the anomaly detection image output by the preset anomaly detection model and outputting an anomaly scoring image, the localization of abnormal objects in the input image is strengthened. Through the design of the anomaly scoring formula, it focuses more on anomalies on the road and tries to ignore anomalies in other scenes, reducing false positives.

[0139] Based on the first embodiment of the road abnormal object detection device of the present invention, a second embodiment of the road abnormal object detection device of the present invention is proposed.

[0140] In this embodiment, the preset anomaly detection model includes a segmentation module, a synthesis module, a depth module, and a difference module. The anomaly prediction module 801 is further configured to: input the image to be detected into the segmentation module for semantic segmentation to obtain a semantic image and an uncertainty image corresponding to the image to be detected; input the image to be detected, the semantic image, and the uncertainty image into the synthesis module for image processing to obtain a synthesized image and an anomaly image; input the image to be detected and the depth disparity map corresponding to the image to be detected into the depth module for feature mapping to obtain a feature fusion image corresponding to the image to be detected; and input the image to be detected, the synthesized image, the feature fusion image, the semantic image, the anomaly image, and the uncertainty image into the difference module for anomaly prediction to obtain an anomaly prediction image corresponding to the image to be detected.

[0141] Furthermore, the anomaly prediction module 801 is also used to obtain the semantic mapping features corresponding to the image to be detected when the segmentation module performs semantic segmentation; input the semantic mapping features into the synthesis module to perform image synthesis to obtain the synthesized image corresponding to the image to be detected; determine the perceptual loss corresponding to the image to be detected and the synthesized image according to a preset feature extractor; and obtain the anomaly image corresponding to the image to be detected according to the perceptual loss and the uncertainty image.

[0142] Furthermore, the depth module includes an original image network branch, a deep network branch, and an upsampling module; the anomaly prediction module 801 is also used to obtain a depth disparity map corresponding to the image to be detected, the depth disparity map reflecting the distance and shape of objects in the image; input the image to be detected into the original image network branch for feature mapping to obtain an original feature map corresponding to the image to be detected; input the depth disparity map into the deep network branch for feature mapping to obtain a depth feature map corresponding to the depth disparity map; and upsample the original feature map and the depth feature map through the upsampling module to obtain a feature fusion image corresponding to the image to be detected.

[0143] Furthermore, the difference module includes a convolution module, a fusion module, and a decoder module; the anomaly prediction module 801 is also used to input the image to be detected, the synthesized image, the feature fusion image, the semantic image, the anomaly image, and the uncertainty image into the convolution module for convolution processing to obtain multiple feature mapping images; input each feature mapping image into the fusion module for image fusion to obtain a fused image; and perform convolution filtering on the fused image according to the decoder module to obtain the anomaly prediction image corresponding to the image to be detected.

[0144] Furthermore, the pixel segmentation module 802 is also used to perform color space conversion on the anomaly prediction image and obtain the five-dimensional vector coordinates corresponding to each pixel according to the conversion result; cluster each pixel to obtain multiple cluster center groups; obtain the average vector value of each cluster center group according to the five-dimensional vector coordinates, and perform cluster convergence on each cluster center group according to the average vector value to obtain the superpixel segmentation image corresponding to the anomaly prediction image.

[0145] Furthermore, the road anomaly detection device also includes a model training module 805, which is used to anomaly label a preset training dataset to obtain a labeled dataset; perform image preprocessing on the labeled dataset to obtain the original semantic image set, the synthetic image set, the uncertain image set, and the anomaly image set corresponding to the labeled dataset; and input the original semantic image set, the synthetic image set, the uncertain image set, and the anomaly image set into an initial model for training to obtain a preset anomaly detection model.

[0146] Other embodiments or specific implementations of the road abnormal object detection device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0148] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0150] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for detecting abnormal objects on a road, characterized in that, The method for detecting abnormal objects on roads includes: The image to be detected is input into a preset anomaly detection model to obtain an anomaly prediction image corresponding to the image to be detected. The preset anomaly detection model is used to preprocess the input image and determine the features of the abnormal object based on the preprocessed input image. Perform superpixel segmentation on the anomaly prediction image to obtain the superpixel segmentation image corresponding to the anomaly prediction image; Anomaly scoring is performed on each superpixel region in the superpixel segmentation image to obtain an anomaly scoring image; Based on the anomaly scoring image, determine the corresponding road anomaly in the image to be detected; The preset anomaly detection model includes a segmentation module, a synthesis module, a depth module, and a difference module; the step of inputting the image to be detected into the preset anomaly detection model to obtain the anomaly prediction image corresponding to the image to be detected includes: The image to be detected is input into the segmentation module for semantic segmentation to obtain the semantic image and uncertainty image corresponding to the image to be detected. The image to be detected, the semantic image, and the uncertain image are input into the synthesis module for image processing to obtain a synthesized image and an abnormal image. The depth module includes the original image network branch, the depth network branch, and the upsampling module; Obtain the depth disparity map corresponding to the image to be detected, the depth disparity map reflecting the distance and shape of objects in the image; The image to be detected is input into the original image network branch for feature mapping to obtain the original feature mapping map corresponding to the image to be detected. The depth disparity map is input into the deep network branch for feature mapping to obtain the depth feature map corresponding to the depth disparity map. The original feature map and the depth feature map are upsampled by the upsampling module to obtain the feature fusion image corresponding to the image to be detected. The image to be detected, the synthesized image, the feature fusion image, the semantic image, the abnormal image, and the uncertainty image are input into the difference module for anomaly prediction to obtain the anomaly prediction image corresponding to the image to be detected.

2. The method for detecting abnormal objects on roads as described in claim 1, characterized in that, The step of inputting the image to be detected, the semantic image, and the uncertain image into the synthesis module for image processing to obtain a synthesized image and an abnormal image includes: Obtain the semantic mapping features corresponding to the image to be detected when the segmentation module performs semantic segmentation; The semantic mapping features are input into the synthesis module to perform image synthesis, thereby obtaining the synthesized image corresponding to the image to be detected; The perceptual loss corresponding to the image to be detected and the synthesized image is determined according to a preset feature extractor; The abnormal image corresponding to the image to be detected is obtained based on the perceptual loss and the uncertainty image.

3. The method for detecting abnormal objects on roads as described in claim 1, characterized in that, The difference module includes a convolution module, a fusion module, and a decoder module; the step of inputting the image to be detected, the synthesized image, the feature fusion image, the semantic image, the anomaly image, and the uncertainty image into the difference module for anomaly prediction to obtain the anomaly prediction image corresponding to the image to be detected includes: The image to be detected, the synthesized image, the feature fusion image, the semantic image, the abnormal image, and the uncertain image are input into the convolution module for convolution processing to obtain multiple feature mapping images; Each feature-mapped image is input into the fusion module for image fusion to obtain a fused image; The decoder module performs convolutional filtering on the fused image to obtain the anomaly prediction image corresponding to the image to be detected.

4. The method for detecting abnormal objects on roads as described in claim 1, characterized in that, The step of performing superpixel segmentation on the anomaly prediction image to obtain a superpixel segmented image corresponding to the anomaly prediction image includes: The anomaly prediction image is converted to a color space, and the five-dimensional vector coordinates corresponding to each pixel are obtained based on the conversion result. Cluster each pixel to obtain multiple cluster center groups; The average vector value of each cluster center group is obtained based on the five-dimensional vector coordinates, and the clustering convergence of each cluster center group is performed based on the average vector value to obtain the superpixel segmentation image corresponding to the anomaly prediction image.

5. The method for detecting abnormal objects on a road as described in claim 1, characterized in that, Before inputting the image to be detected into a preset anomaly detection model to obtain the anomaly prediction image corresponding to the image to be detected, the method further includes: Anomalies are marked on a pre-defined training dataset to obtain a labeled dataset; Image preprocessing is performed on the labeled dataset to obtain the original semantic image set, the synthetic image set, the uncertain image set, and the abnormal image set corresponding to the labeled dataset; The original semantic image set, the synthetic image set, the uncertain image set, and the abnormal image set are input into the initial model for training to obtain a preset anomaly detection model.

6. A road abnormality detection device, characterized in that, The device includes: An anomaly prediction module is used to input the image to be detected into a preset anomaly detection model to obtain an anomaly prediction image corresponding to the image to be detected. The preset anomaly detection model is used to perform image preprocessing on the input image and determine the features of the abnormal object based on the preprocessed input image. A pixel segmentation module is used to perform superpixel segmentation on the anomaly prediction image to obtain a superpixel segmented image corresponding to the anomaly prediction image. An anomaly scoring module is used to score the anomalies of each superpixel region in the superpixel segmentation image and obtain an anomaly scoring image. The foreign object detection module is used to determine the corresponding abnormal road objects in the image to be detected based on the abnormality scoring image. The preset anomaly detection model includes a segmentation module, a synthesis module, a depth module, and a difference module. The anomaly prediction module is further configured to: input the image to be detected into the segmentation module for semantic segmentation to obtain a semantic image and an uncertainty image corresponding to the image to be detected; input the image to be detected, the semantic image, and the uncertainty image into the synthesis module for image processing to obtain a synthesized image and an anomaly image; input the image to be detected and its corresponding depth disparity map into the depth module for feature mapping to obtain a feature fusion image corresponding to the image to be detected; and input the image to be detected, the synthesized image, the feature fusion image, the semantic image, the anomaly image, and the uncertainty image into the difference module for anomaly prediction to obtain an anomaly prediction image corresponding to the image to be detected. The depth module includes an original image network branch, a deep network branch, and an upsampling module. The anomaly prediction module is further configured to acquire a depth disparity map corresponding to the image to be detected, the depth disparity map reflecting the distance and shape of objects in the image; input the image to be detected into the original image network branch for feature mapping to obtain an original feature map corresponding to the image to be detected; input the depth disparity map into the deep network branch for feature mapping to obtain a depth feature map corresponding to the depth disparity map; and upsample the original feature map and the depth feature map through the upsampling module to obtain a feature fusion image corresponding to the image to be detected.

7. A road abnormality detection device, characterized in that, The device includes: a memory, a processor, and a road anomaly detection program stored in the memory and executable on the processor, the road anomaly detection program being configured to implement the steps of the road anomaly detection method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a road anomaly detection program, which, when executed by a processor, implements the steps of the road anomaly detection method as described in any one of claims 1 to 5.

Images