Exceptional environment visual perception method, device and medium considering expected functional safety

Abstract

This application discloses a method, apparatus, and medium for visual perception of abnormal environments considering expected functional safety. The method includes semantic processing of an input image to obtain a semantic map and a semantic prediction uncertainty map. The semantic map includes a semantic category confidence map, a semantic category label map, and a semantic category boundary map. A reconstructed image is generated using the semantic category confidence map, semantic category label map, and semantic category boundary map. The structural similarity between the reconstructed image and the input image is calculated to generate a reconstruction error map. A regional risk score is calculated based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map. This application improves the vehicle's high-precision perception and comprehensive understanding of abnormal driving environments, providing a reliable basis for setting safety boundaries for planning and control, and ensuring vehicle driving safety.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a visual perception method, device and medium for abnormal environments that takes into account the expected functional safety. Background Technology

[0002] With the development of autonomous driving technology, the application of intelligent vehicles on open roads is constantly increasing. In addition to identifying regular traffic participants, visual perception systems also need to handle abnormal environments in the training data, such as insufficient coverage, difficulty in stably representing semantic categories, or significant degradation in imaging quality. Generally, abnormal environments include, but are not limited to: road debris, temporary construction cones or guardrails, fallen rocks, fallen tree branches, irregularly shaped small obstacles, local imaging anomalies caused by strong reflections or shadows, texture degradation caused by adverse weather conditions such as rain, fog, snow, and dust, local target loss due to occlusion, and low-confidence areas caused by sensor noise, glare, and dirt.

[0003] The common characteristic of the above-mentioned abnormal environments is that their pixel appearance, semantic category, or spatial location relationship deviates from the distribution of training samples. This can easily lead to high entropy, low confidence interval, or inconsistent cross-sampling predictions in semantic segmentation results, resulting in poor vehicle perception of abnormal driving environments and thus affecting the safety boundary of subsequent planning and control. Summary of the Invention

[0004] This application provides a visual perception method, device, and medium for abnormal environments that takes into account the expected functional safety, thereby improving the vehicle's high-precision perception and comprehensive understanding of abnormal driving environments.

[0005] In a first aspect, embodiments of this application provide a visual perception method for abnormal environments that takes into account expected functional safety, including: A semantic segmentation network is used to perform deterministic reasoning on the input image to generate a first semantic category probability map. Based on the first semantic category probability map, a semantic category label map, a semantic category boundary map, and a semantic category confidence map are generated. Each pixel in the first semantic category probability map is traversed, and the uncertainty index of the pixel is calculated to obtain an uncertainty index set, which includes prediction distribution uncertainty and confidence interval uncertainty. The uncertainty indices in the uncertainty index set are fused to generate a semantic prediction uncertainty map. A reconstructed image is generated using the semantic category confidence map, semantic category label map, and semantic category boundary map. Calculate the structural similarity between the reconstructed image and the input image to generate a reconstruction error map; The regional risk score is calculated based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map.

[0006] In conjunction with the first aspect, in one implementation, the maximum probability of the semantic category among all semantic categories of each pixel in the first semantic category probability map is extracted; and a semantic category confidence map is generated based on the maximum probability of the semantic category of each pixel. And / or, extract the semantic category with the highest probability among all semantic categories of each pixel in the first semantic category probability map; generate a semantic category label map based on the semantic category with the highest probability of each pixel.

[0007] In conjunction with the first aspect, in one implementation, each pixel in the semantic category label map is traversed, and it is determined whether the label value of the pixel is consistent with that of its neighboring pixels. If they are inconsistent, the corresponding position of the pixel in the semantic category label map is marked as the semantic category boundary. Alternatively, perform morphological gradient calculation or edge detection on the semantic category label map, and mark the pixels whose calculation results exceed the first preset boundary threshold as semantic category boundaries; Alternatively, the gradient magnitude of the first semantic category probability map in the spatial direction is calculated, and each pixel in the first semantic category probability map is traversed. When the spatial gradient magnitude at a pixel exceeds a second preset boundary threshold, the corresponding position of the pixel in the first semantic category probability map is marked as a semantic category boundary.

[0008] In conjunction with the first aspect, in one implementation, the uncertainty index set further includes the probability average variance, and the method further includes: The input image is subjected to N random inferences through a semantic segmentation network, generating N second semantic category probability maps, where N≥2; Traverse each pixel in the second semantic category probability map, and calculate the average variance of the probability corresponding to each semantic category of the pixel based on the probability of the pixel in N second semantic category probability maps, as an uncertainty index.

[0009] In conjunction with the first aspect, in one implementation, the information entropy of the pixel is calculated based on the probability of each semantic category of the pixel, and is used as the uncertainty of the prediction distribution; And / or, calculate the confidence interval uncertainty based on the probability of the largest semantic category and the probability of the second largest semantic category of the pixel.

[0010] In conjunction with the first aspect, in one implementation, the uncertainty indicators in the uncertainty indicator set are fused to generate a semantic prediction uncertainty graph, including: Input the uncertainty indicators from the uncertainty indicator set into the uncertainty semantic channel recalibration module; The dynamic weights of each uncertain index are generated through the uncertain semantic channel recalibration module. The uncertain indicators are weighted and fused according to the dynamic weights to obtain the semantic prediction uncertainty map.

[0011] In conjunction with the first aspect, in one implementation, the uncertain semantic channel recalibration module includes multiple channel description value generation units, channel weight generation units, and uncertainty fusion units; The channel description value generation unit is configured to perform spatial pooling processing on the uncertainty index corresponding to the input to obtain the channel description value of the uncertainty index. The channel weight generation unit is configured to generate dynamic weights corresponding to each uncertainty index by processing the channel description values ​​of each uncertainty index through nonlinear mapping. The uncertainty fusion unit is configured to perform weighted fusion of each uncertainty indicator using the dynamic weights of each uncertainty indicator to obtain the semantic prediction uncertainty map.

[0012] In conjunction with the first aspect, in one implementation, generating a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map includes: The semantic category label map is subjected to feature extraction processing to obtain semantic condition features; The semantic category boundary map and the semantic category confidence map are fused with the semantic condition features to construct a semantic condition tensor; The semantic condition tensor is used as a conditional input to the semantic conditional diffusion model to generate a reconstructed image that is consistent with the semantic layout of the input image.

[0013] In conjunction with the first aspect, in one implementation, calculating the structural similarity between the reconstructed image and the input image to generate a reconstruction error map includes: The structural similarity algorithm is used to calculate the local window differences between the input image and the reconstructed image; The reconstruction error map is generated based on the local window differences.

[0014] In conjunction with the first aspect, in one implementation, calculating a regional risk score based on the reconstruction error map, the semantic category label map, and the semantic prediction uncertainty map includes: An anomaly region prediction map is generated based on the reconstruction error map and the semantic category label map; The regional risk score is calculated based on the anomaly region prediction map and the semantic prediction uncertainty map.

[0015] In conjunction with the first aspect, in one implementation, generating an anomaly region prediction map based on the reconstruction error map and the semantic category label map includes: Extract the semantic features of the semantic category label map and the reconstruction error features of the reconstruction error map; The semantic features and the reconstruction error features are fused to generate the anomaly region prediction map.

[0016] In conjunction with the first aspect, in one implementation, calculating the regional risk score based on the anomaly region prediction map and the semantic prediction uncertainty map includes: Based on the anomaly region prediction map and the semantic prediction uncertainty map, calculate the pixel-level risk value of each pixel. A pixel-level risk map is generated based on the pixel-level risk values; Based on the pixel-level risk map, the connected regions are determined; The region risk score is calculated based on the pixel-level risk values ​​of all pixels contained in the connected region.

[0017] In conjunction with the first aspect, in one implementation, determining connected regions based on the pixel-level risk map includes: Traverse each pixel of the pixel-level risk map and mark pixels with pixel-level risk values ​​greater than a set risk threshold as high-risk candidate points; Perform connectivity analysis on high-risk candidate points to aggregate all interconnected high-risk candidate points into connected regions.

[0018] In conjunction with the first aspect, in one implementation, a region risk score is calculated based on the pixel-level risk values ​​of all pixels contained in the connected region, including: Calculate the arithmetic mean of the pixel-level risk values ​​of each pixel within the connected region; Extract the maximum risk value from the pixel-level risk values ​​of each pixel within the connected region; The regional risk score is obtained based on the arithmetic mean and the maximum risk value.

[0019] In conjunction with the first aspect, in one implementation, it further includes: A safety trigger signal is generated based on the regional risk score and the preset safety threshold.

[0020] Secondly, embodiments of this application provide an abnormal environment visual perception device that takes into account the expected functional safety, including: A semantic processing module is used to perform semantic processing on the input image to obtain a semantic map and a semantic prediction uncertainty map. The semantic map includes a semantic category confidence map, a semantic category label map, and a semantic category boundary map. An image reconstruction module is used to generate a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map. An error generation module is used to calculate the structural similarity between the reconstructed image and the input image to generate a reconstruction error map; The risk scoring module is used to calculate the regional risk score based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map.

[0021] Thirdly, embodiments of this application provide an abnormal environment visual perception device that considers expected functional safety, including a processor, a memory, and an abnormal environment visual perception program that considers expected functional safety stored in the memory and executable by the processor, wherein when the abnormal environment visual perception program that considers expected functional safety is executed by the processor, it implements the steps of the abnormal environment visual perception method that considers expected functional safety as described in any of the preceding claims.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing an abnormal environment visual perception program that considers expected functional safety, wherein when the abnormal environment visual perception program that considers expected functional safety is executed by a processor, it implements the steps of the abnormal environment visual perception method that considers expected functional safety as described in any of the preceding claims.

[0023] The beneficial effects of the technical solution provided in this application include: This application provides a method, apparatus, and medium for visual perception of abnormal environments considering expected functional safety. It obtains a semantic category confidence map, a semantic category label map, and a semantic category boundary map by semantically processing the input image. A reconstructed image is generated using these maps. The structural similarity between the reconstructed image and the input image is calculated to generate a reconstruction error map. A region risk score is calculated based on the reconstruction error map, the semantic category label map, and the semantic prediction uncertainty map. This solves the problem of high entropy or low confidence intervals in semantic segmentation results caused by pixel appearance or spatial relationships deviating from the training distribution in abnormal environments. Semantic information is used to constrain the reconstructed image to maintain a normal semantic layout, making abnormal regions in the input image significantly prominent in the reconstruction error map. Simultaneously, the semantic prediction uncertainty map is integrated to quantify the model confidence, thereby improving the vehicle's high-precision perception and comprehensive understanding of abnormal driving environments. This provides a reliable basis for setting safety boundaries for planning and control, ensuring vehicle driving safety. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating an embodiment of a visual perception method for abnormal environments with intended functional safety in accordance with this application; Figure 2 This is a schematic diagram of the PSPNet semantic segmentation network of this application; Figure 3 This is a schematic diagram illustrating the operation of the uncertainty semantic channel recalibration module in this application; Figure 4 This is a schematic diagram illustrating the image reconstruction work performed using the semantic conditional diffusion model of this application. Figure 1 ; Figure 5 This is a schematic diagram illustrating the image reconstruction work performed using the semantic conditional diffusion model of this application. Figure 2 ; Figure 6 This is a schematic diagram of the anomaly detection process in this application; Figure 7 A schematic diagram of the functional modules of an embodiment of an abnormal environment visual perception device for the purpose of ensuring the intended functional safety of this application; Figure 8 This is a schematic diagram of the hardware structure of an abnormal environment visual perception device that takes into account the expected functional safety in the embodiments of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.

[0028] Safety of the Intended Functionality (SOTIF) aims to address risks arising not from system failure, but from functional deficiencies or foreseeable misuse.

[0029] In this application, the focus of functional safety is to reduce unreasonable risks caused by insufficient functional design, performance limitations, or reasonably foreseeable misuse through the quantification of perceived reliability and the design of safety interfaces during the design phase. Specifically, this application identifies areas with insufficient perceived reliability through anomaly region prediction, semantic prediction uncertainty maps, and regional risk scoring, and provides the risk quantification results to the downstream safety policy module. The downstream safety policy module can then execute preset strategies such as deceleration, increasing the safety distance, requesting takeover, or entering a conservative driving mode. The above content pertains to the design of perception system output and safety interfaces, and does not limit the implementation method of specific vehicle controllers or planning optimizers.

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0031] In a first aspect, embodiments of this application provide a visual perception method for abnormal environments that takes into account the expected functional safety.

[0032] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of a visual perception method for abnormal environments, considering the intended functional safety of this application. Figure 1 As shown, visual perception methods for abnormal environments that consider expected functional safety include: S101: Perform semantic processing on the input image to obtain a semantic map and a semantic prediction uncertainty map, wherein the semantic map includes a semantic category confidence map, a semantic category label map, and a semantic category boundary map.

[0033] The vehicle acquires input images based on its perception system (such as cameras, radar, etc.). For example, a camera can capture RGB images.

[0034] The input image is fed into a semantic segmentation network for semantic segmentation, and high-quality semantic maps are extracted in abnormal scenarios.

[0035] As an example, the PSPNet (Pyramid Scene Parsing Network), a deep learning-based image segmentation network, is used to perform semantic segmentation on the input image, extract semantic information from the image, and generate a semantic category label for each pixel.

[0036] See Figure 2 As shown, PSPNet, as a preferred semantic segmentation network, includes a backbone network, a pyramid pooling module, and a decoding output layer.

[0037] The PSPNet backbone uses a pre-trained DLA102 (an image classification model based on Deep Layer Aggregation (DLA) technology), which extracts image features through convolutional operations. The Pyramid Pooling Module captures contextual information at different scales in the image through pooling operations of varying sizes. For example, the Pyramid Pooling Module performs pooling operations at four scales: 1×1, 2×2, 3×3, and 6×6, and upsamples the pooled feature maps before concatenating them with the original feature maps. This helps extract global contextual information about roads, lane boundaries, traffic participants, and local obstacles, enhancing the network's understanding of details. The pyramid-pooled feature maps are then fused with the original feature maps and decoded through convolutional operations to obtain the final semantic segmentation map.

[0038] In different embodiments, the backbone network may also employ ResNet50, ResNet101, or the lightweight MobileNet series, while the semantic segmentation network may employ BiSeNet, HRNet, DeepLabV3+, SegFormer, or other segmentation networks with multi-scale context modeling capabilities. Compared to networks such as BiSeNet, HRNet, and SegFormer, PSPNet in this embodiment obtains stable global scene priors with lower structural complexity and provides semantic constraints for subsequent semantic conditional diffusion models.

[0039] Semantic processing is performed on the input image to obtain a semantic prediction uncertainty map, which can stably characterize the competition and distribution of semantic categories.

[0040] S102: Generate a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map.

[0041] By using the semantic category confidence map, semantic category label map, and semantic category boundary map as conditional inputs to the semantic conditional diffusion model, the reconstructed image maintains spatial and semantic association with the input image, reduces insufficient regional association caused by unconditional generation, and reduces false anomalies caused by distortion in normal region reconstruction.

[0042] The semantic conditional diffusion model is not used to restore the true appearance of abnormal regions, but to generate normal road reconstruction images that are consistent with the current semantic layout. When there are unknown obstacles, low confidence regions or semantic misclassification regions in the input image, abnormal clues will be formed between the reconstruction results and the original input in the local window SSIM error and deep feature differences.

[0043] S103: Calculate the structural similarity between the reconstructed image and the input image to generate a reconstruction error map.

[0044] S104: Calculate the regional risk score based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map.

[0045] Calculating regional risk scores and using these scores to assess risk levels enhances the application's ability to perceive complex scenarios and unexpected obstacles, providing more reliable information support for decision-makers.

[0046] In this embodiment, semantic processing is performed on the input image to obtain a semantic category confidence map, a semantic category label map, and a semantic category boundary map. A reconstructed image is then generated using these maps. The structural similarity between the reconstructed image and the input image is calculated to generate a reconstruction error map. Furthermore, a region risk score is calculated based on the reconstruction error map, the semantic category label map, and the semantic prediction uncertainty map. This addresses the problem of high entropy or low confidence intervals in semantic segmentation results caused by pixel appearance or spatial relationships deviating from the training distribution in abnormal environments. Semantic information is used to constrain the reconstructed image to maintain a normal semantic layout, making abnormal regions in the input image significantly prominent in the reconstruction error map. Simultaneously, the semantic prediction uncertainty map is integrated to quantify the model confidence, thereby improving the vehicle's high-precision perception and comprehensive understanding of abnormal driving environments. This provides a reliable basis for setting safety boundaries for planning and control, ensuring vehicle driving safety.

[0047] In particular, when an abnormal region causes the semantic category to be incorrectly segmented into a known category, the semantic conditional diffusion model will generate the appearance of the corresponding category object based on the erroneous semantics. However, since the structure and texture of normal objects in the training data are significantly different from those of real abnormal objects, the structural similarity calculation can still capture the low similarity features of the local region. At the same time, the semantic prediction uncertainty map of the region will show a high value. The fusion of the two ensures the accuracy of the risk score of the abnormal region.

[0048] Furthermore, in one embodiment, semantic processing is performed on the input image to generate a semantic map, including: S201: Perform deterministic reasoning on the input image through a semantic segmentation network to generate a first semantic category probability map.

[0049] Perform deterministic reasoning, disable random deactivation Dropout, propagate forward only once, and output a semantic category probability map.

[0050] For example, if the input image is a spatially sized image of size H×W, where H represents the image height and W represents the image width (both in pixels), then the spatially sized first semantic category probability map P is H×W, i.e. The number of channels in the semantic category probability map corresponds to the total number of semantic categories, and its spatial size is consistent with the size of the input image or the upsampled feature map, i.e., C is the total number of semantic categories. Semantic categories include road and infrastructure categories (such as roads, sidewalks, buildings, etc.), traffic facilities categories (such as traffic lights, traffic signs, etc.), and dynamic target categories (such as pedestrians, cars, motorcycles, etc.).

[0051] In the semantic category probability map, each pixel has a probability value for all semantic categories.

[0052] S202: Generate a semantic category label map, a semantic category boundary map, and a semantic category confidence map based on the first semantic category probability map.

[0053] The semantic category boundary map is used to prompt the semantic conditional diffusion model to focus on lane boundaries, obstacle outlines, and semantic category transition regions, reducing structural misalignment in the reconstructed image near the boundaries. This application uses the semantic category label map, semantic category boundary map, and semantic category confidence map as conditional inputs to the semantic conditional diffusion model, subjecting the model to road semantic layout constraints during reconstruction. When unseen categories or abnormal obstacles exist in the input image, this region typically exhibits low confidence or incorrect semantics in the semantic conditions, and the reconstructed image tends to recover the appearance corresponding to normal road semantics, thus forming detectable differences in local structural errors and deep feature errors.

[0054] It is understandable that the spatial dimensions of the semantic category label map S, the semantic category boundary map B, and the semantic category confidence map are all H×W, i.e. .

[0055] In this embodiment, a semantic segmentation network is used to perform deterministic reasoning on the input image to generate a first semantic category probability map. Based on the first semantic category probability map, a semantic category label map, a semantic category boundary map, and a semantic category confidence map are generated. This solves the problem that a single semantic label cannot reflect the prediction confidence and boundary information, and provides accurate semantic condition constraints for subsequent image reconstruction. It also provides data support for uncertainty quantification and effectively ensures the reliability of the semantic processing results.

[0056] Further, in one embodiment, the maximum probability of the semantic category among all semantic categories of each pixel in the first semantic category probability map is extracted; based on the maximum probability of the semantic category of each pixel, a semantic category confidence map is generated; in this embodiment, the maximum probability among all semantic categories of each pixel is used as the confidence of that pixel, thereby forming a semantic category confidence map.

[0057] Further, in one embodiment, the semantic category with the highest probability among all semantic categories of each pixel in the first semantic category probability map is extracted; a semantic category label map is generated based on the semantic category with the highest probability of each pixel; in this embodiment, the semantic category with the highest probability among all semantic categories of each pixel is used as the label of that pixel, thereby forming a semantic category label map. Each semantic category is assigned a label value, for example, the road label value is 1, the car label value is 2, and the sky label value is 19.

[0058] Furthermore, in one embodiment, each pixel in the semantic category label map is traversed, and it is determined whether the label value of the pixel is consistent with that of its neighboring pixels. If they are inconsistent, the corresponding position of the pixel in the semantic category label map is marked as the semantic category boundary.

[0059] Furthermore, in one embodiment, morphological gradient calculation or edge detection is performed on the semantic category label map, and pixels whose calculation results exceed a first preset boundary threshold are marked as semantic category boundaries.

[0060] Further, in one embodiment, the gradient magnitude of the first semantic category probability map in the spatial direction is calculated, and each pixel in the first semantic category probability map is traversed. When the spatial gradient magnitude at the pixel exceeds a second preset boundary threshold, the corresponding position of the pixel in the first semantic category probability map is marked as a semantic category boundary.

[0061] All semantic category boundaries constitute a semantic category boundary map. The first preset boundary threshold and the second preset boundary threshold can be set according to actual needs.

[0062] Further, in one embodiment, semantic processing is performed on the input image to generate a semantic prediction uncertainty map, including: S301: Traverse each pixel in the first semantic category probability map and calculate the uncertainty index of the pixel to obtain an uncertainty index set, which includes prediction distribution uncertainty and confidence interval uncertainty.

[0063] Predicted distribution uncertainty This is used to characterize the degree of disorder in the semantic category distribution of each pixel in the first semantic category probability map, that is, the "uncertainty" of the semantic segmentation network's prediction distribution for a certain pixel. If the semantic category probability value of a certain pixel is relatively uniformly distributed, it indicates that the prediction of that pixel has high uncertainty.

[0064] Confidence interval uncertainty It is used to characterize the distinguishability between the highest probability of the semantic category and the second highest probability of the semantic category for each pixel in the first semantic category probability map, indicating the degree of competition between the first two semantic category predictions.

[0065] In this embodiment, for each pixel in the first semantic category probability map, its prediction distribution uncertainty and confidence interval uncertainty are calculated, thereby obtaining a prediction distribution uncertainty map and a confidence interval uncertainty map. , The two types of uncertainty graphs are used as two input channels, denoted as the semantic uncertainty tensor. .

[0066]

[0067]

[0068] S302: The uncertainty indicators in the uncertainty indicator set are fused to generate a semantic prediction uncertainty map.

[0069] semantic uncertainty tensor Input the uncertain semantic channel recalibration module USCRM to obtain the semantic prediction uncertainty map of a single channel. , Directly input the SOTIF risk trigger interface to calculate the risk area score together with the anomaly area prediction map.

[0070] This embodiment calculates the uncertainty of the prediction distribution and the uncertainty of the confidence interval by traversing the probability graph of the first semantic category to obtain a set of uncertainty indicators. The uncertainty indicators are then fused to generate a semantic prediction uncertainty graph, which solves the problem that a single-dimensional indicator cannot fully represent the confidence of the model prediction. By utilizing the complementary characteristics of multi-dimensional uncertainty, the prediction risk is comprehensively quantified, providing a reliable basis for the detection of abnormal areas and enhancing the ability to identify low-confidence areas.

[0071] Furthermore, in one embodiment, the information entropy of the pixel is calculated based on the probability of each semantic category of the pixel, which is used as the prediction distribution uncertainty; the confidence interval uncertainty is calculated based on the probability of the largest semantic category and the probability of the second largest semantic category of the pixel.

[0072] For example, as an example, the Softmax entropy is calculated based on the probability of each semantic category of the pixel to obtain the uncertainty of the prediction distribution. :

[0073] in, is the probability that pixel z belongs to semantic category c, where C is the total number of semantic categories. A higher entropy value indicates that the semantic segmentation network is uncertain in predicting the semantic category of that pixel.

[0074] The semantic category probabilities at pixel z are sorted in descending order, and the first probability is denoted as z. (z), the second most likely is (z), then:

[0075] The larger the value, the more difficult it is for the semantic segmentation network to distinguish between the two most likely categories. The more likely the pixel is to be located at a semantic boundary, an anomalous region, or an area under-covered by the training distribution.

[0076] When the semantic category with the highest probability and the semantic category with the second highest probability are close, A value close to 1 indicates that the prediction boundary or category assignment is unstable; when the semantic category with the highest probability is significantly higher than other semantic categories, A value close to 0 indicates a relatively certain prediction. This definition focuses on the two classes most likely to be confused, thus avoiding measurement distortion caused by tail class probabilities close to 0.

[0077] During prediction, there may be a consistency problem between multiple predictions made by the semantic segmentation network. Therefore, in order to eliminate this problem, this application introduces the probability average variance as an uncertain probability index to approximate the cognitive uncertainty of the model.

[0078] Furthermore, in one embodiment, to avoid the inability to define variance in a single deterministic inference, this embodiment enables one of the following methods during the inference phase: random deactivation during testing, lightweight multi-head classifier, or small-scale model ensemble, to obtain N semantic category probability maps for the same input image, where N is preferably 4 to 8. Specifically, semantic processing is performed on the input image to generate a semantic prediction uncertainty map, including: S401: Perform N random inferences on the input image through a semantic segmentation network to generate N second semantic category probability maps, where N≥2.

[0079] Stochastic reasoning, by introducing random perturbations, simulates the "possible variations" of semantic segmentation network parameters, thereby exposing the "cognitive instability" of the semantic segmentation network for certain input regions. The aim is to improve the robustness of the system.

[0080] Understandably, the second semantic category probability graph With the first semantic category probability Figure 1 Similarly, the space dimensions are H×W, that is... .

[0081] S402: Traverse each pixel in the second semantic category probability map, and calculate the average variance of the probability corresponding to each semantic category of the pixel based on the probability of the pixel in N second semantic category probability maps, as an uncertainty index.

[0082] Specifically, for each pixel, the probability variances of the C semantic categories in the second semantic category probability map obtained from N predictions are calculated to obtain the C semantic category variance values. Then, the average of these C semantic category variance values ​​is calculated to obtain the average probability variance of that pixel. .

[0083]

[0084]

[0085] in, (z) represents the probability that pixel z belongs to semantic category c in the nth prediction. This represents the average probability that pixel z belongs to semantic category c in N predictions. This refers to variance calculation. The larger the value, the more inconsistent the prediction results are, and the higher the uncertainty in the semantic segmentation network's understanding of that pixel.

[0086] In scenarios with high real-time requirements, it is advisable to first... and Select candidate high-risk regions, and then perform multiple samplings only on these candidate regions. When N is 1, no calculation is performed. Alternatively, it can be set to 0, and its weight can be automatically reduced by the uncertain semantic channel recalibration module.

[0087] S403: Traverse each pixel in the first semantic category probability map, calculate the uncertainty index of the pixel to obtain an uncertainty index set, the uncertainty index set including at least one of prediction distribution uncertainty and confidence interval uncertainty, and the probability average variance.

[0088] S404: The uncertainty indicators in the uncertainty indicator set are fused to generate a semantic prediction uncertainty map.

[0089] In this embodiment, for each pixel in the first semantic category probability map, its prediction distribution uncertainty and confidence interval uncertainty are calculated, thus obtaining a prediction distribution uncertainty map and a confidence interval uncertainty map. For each pixel in the second semantic category probability map, its probability mean variance is calculated, thus obtaining a probability mean variance map. , The three types of uncertainty graphs are used as three input channels, denoted as semantic uncertainty tensors. .

[0090]

[0091]

[0092] semantic uncertainty tensor Input the uncertain semantic channel recalibration module USCRM to obtain the semantic prediction uncertainty map of a single channel. , Directly input the SOTIF risk trigger interface to calculate the risk area score together with the anomaly area prediction map.

[0093] By constructing a semantic prediction uncertainty map using multiple indicators, the expression of abnormal regions and low-reliability perception regions can be enhanced, and false detections caused by a single uncertainty indicator can be reduced.

[0094] Furthermore, in one embodiment, the uncertainty indicators in the uncertainty indicator set are fused to generate a semantic prediction uncertainty graph, including: S501: Input the uncertainty indicators in the uncertainty indicator set into the uncertainty semantic channel recalibration module.

[0095] S502: Generate dynamic weights for each of the uncertain indicators through the uncertainty semantic channel recalibration module.

[0096] S503: The uncertain indicators are weighted and fused according to the dynamic weights to obtain the semantic prediction uncertainty map.

[0097] In this embodiment, an Uncertainty-Semantic Channel Recalibration Module (USCRM) is set up. USCRM is designed for uncertainty metrics, using a semantic uncertainty tensor composed of multiple uncertainty metrics as multi-channel input. USCRM borrows the channel recalibration idea from the standard Squeeze-and-Excitation (SE) module; however, while the input channels of the standard SE module are typically abstract feature channels generated by convolutional networks, the input channels of USCRM are uncertainty metric channels with explicit physical meaning. The standard SE module outputs ordinary feature channel weights, while USCRM outputs fused weights from different uncertainty sources. These output weights can be interpreted as the degree of dependence of the current scene on different uncertainty sources.

[0098] In this embodiment, by inputting the uncertainty indicators from the uncertainty indicator set into the uncertainty semantic channel recalibration module, dynamic weights are generated for each uncertainty indicator. The semantic prediction uncertainty map is then obtained by weighted fusion of these dynamic weights, solving the problem that fixed-weight fusion cannot adapt to the differences in indicator contribution across various scenarios. By using the module to adaptively learn the importance of each indicator and dynamically adjust the weights, the fusion result more closely reflects the actual predicted risk, avoiding the dominance of a single indicator, improving the accuracy and reliability of the semantic prediction uncertainty map, enhancing the precision of uncertainty quantification in abnormal environments, and providing a reliable basis for subsequent regional risk scoring.

[0099] The uncertainty semantic channel recalibration module includes multiple channel description value generation units, channel weight generation units, and uncertainty fusion units.

[0100] See Figure 3 As shown, the channel description value generation unit is configured to perform spatial pooling processing on the uncertainty index corresponding to the input to obtain the channel description value of the uncertainty index; for example, the input is... At that time, the USCRM module contains three channel descriptor value generation units, used to assign dynamic weights to three types of uncertainty indicators. The three channel descriptor value generation units correspond to... Each channel description value generation unit performs global average pooling on the input uncertainty map to generate values ​​corresponding to... The three channel description values.

[0101] The channel weight generation unit is configured to process the channel description values ​​of each uncertainty index through nonlinear mapping to generate dynamic weights corresponding to each uncertainty index; for example, the channel weight generation unit performs a two-layer fully connected mapping and Sigmoid activation on the three channel description values ​​to generate three dynamic weights. , and The following are, in order, the uncertainty of the predicted distribution. Dynamic weights and confidence interval uncertainty Dynamic weights and probability average variance Dynamic weights.

[0102] The uncertainty fusion unit is configured to perform weighted fusion of the uncertainty indicators using dynamic weights to obtain the semantic prediction uncertainty map. For example, the three types of uncertainty maps are weighted channel by channel and mapped to a single-channel semantic prediction uncertainty map through 1×1 convolution or weighted summation. .

[0103] The dynamic weights generated by USCRM can address the uncertainty of the prediction distribution based on the degree of semantic category disorder, the degree of semantic category competition, and the consistency of multiple predictions in the current scenario. Confidence interval uncertainty and probability mean variance Adaptive fusion is performed. Dynamic weights. , and Generated by the USCRM module and can change as the input image changes.

[0104]

[0105] in, This represents a pixel-level single-channel semantic prediction uncertainty map with a spatial size of H×W, i.e. Semantic prediction uncertainty graph after normalization Each pixel in value The value range is [0, 1], indicating that there is little confidence in predicting the value of this pixel.

[0106] Further, in one embodiment, generating a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map includes: S601: Perform feature extraction processing on the semantic category label map to obtain semantic condition features.

[0107] S602: The semantic category boundary map and the semantic category confidence map are fused with the semantic condition features to construct a semantic condition tensor.

[0108] S603: Input the semantic condition tensor as a condition into the semantic conditional diffusion model to generate a reconstructed image that is consistent with the semantic layout of the input image.

[0109] See Figure 4 and Figure 5As shown, in the image reconstruction process, the semantic conditional diffusion model uses a semantic conditional tensor as the conditional input. Specifically, the semantic category label map is mapped to semantic conditional features through an embedding layer or 1×1 convolution. The semantic category boundary map and semantic category confidence map are then concatenated and fused with these semantic conditional features along the channel direction to construct a semantic conditional tensor. This tensor is then input into the encoding, intermediate, and decoding layers of the semantic conditional diffusion model U-Net through a conditional injection layer. The conditional injection method can be one or more of channel concatenation, conditional normalization, cross-attention, or control branching. The forward diffusion process progressively adds noise to the latent variables of the normal image, and the inverse denoising network learns to predict noise or the denoised mean under given semantic conditions. The training loss can include noise prediction loss, semantic consistency loss, and optional perceptual loss. Through the above conditional mechanism, the reconstructed image maintains a comparable spatial correspondence with the input image.

[0110] In this embodiment, semantic condition features are obtained by extracting features from the semantic category label map. The semantic category boundary map and semantic category confidence map are then fused with the semantic condition features to construct a semantic condition tensor, which is input into the semantic condition diffusion model to generate a reconstructed image. On the one hand, this solves the problem of high noise in the error map caused by chaotic semantic layout in traditional reconstruction methods. By using boundary and confidence constraints to constrain the diffusion model generation process, the semantic layout of the reconstructed image is consistent with that of the input image, and abnormal textures are removed, providing a clean benchmark for subsequent calculation of structural similarity and improving the accuracy of abnormal region localization. On the other hand, it can simultaneously solve two types of problems: "unclear" and "unrecognizable". Specifically, 1) For "unclear" caused by insufficient sensor function: Due to the deterioration of imaging quality (such as darkness, glare, rain and fog blurring, texture degradation, etc.), the diffusion model needs to reconstruct a "normal" image under semantic condition constraints, and low-quality areas are highlighted in the reconstruction error map; then, the model confidence is quantified by the semantic prediction uncertainty map. 2) For the "unrecognized" target due to the lack of coverage in the training data: the semantic segmentation network provides low confidence or incorrect labels, and the diffusion model reconstructs the "normal" appearance based on the incorrect semantics, which is significantly different from the input image.

[0111] Further, in one embodiment, calculating the structural similarity between the reconstructed image and the input image to generate a reconstruction error map includes: calculating the local window difference between the input image and the reconstructed image using a structural similarity algorithm; and generating the reconstruction error map based on the local window difference.

[0112] Specifically, the structural similarity index (SSIM) is used to measure the structural similarity between the input image and the reconstructed image within a local window, rather than calculating the SSIM directly for individual pixels.

[0113] set up (i,j) and (i,j) represent the corresponding local windows cropped from the input and reconstructed images, centered on the pixel with pixel coordinates (i,j). SSIM calculates the brightness, contrast, and structure statistics within these local windows.

[0114]

[0115] Where x and y are the input images and reconstructed images , and It is the local mean of the x and y values ​​of the image. and It is the local variance of the image x and y. It is the local covariance between the x and y values ​​of two images. and It is a constant used to avoid the denominator being zero.

[0116] The closer the SSIM value is to 1, the more similar the corresponding local windows are. By sliding the local window at each pixel location and calculating the window-level SSIM, a reconstruction error map is obtained. ; The pixel value reflects the structural difference between the input image and the reconstructed image in the local neighborhood centered on the pixel with pixel coordinates (i,j), i.e., the reconstruction error.

[0117]

[0118] In this embodiment, a reconstruction error map is generated by calculating the local window difference between the input image and the reconstructed image using a structural similarity algorithm. This solves the problem that pixel-level direct subtraction is easily affected by illumination noise, highlights the loss of structural information in abnormal regions, suppresses normal background fluctuations, and improves the robustness of anomaly detection.

[0119] Further, in one embodiment, calculating a regional risk score based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map includes: S701: Generate an anomaly region prediction map based on the reconstruction error map and the semantic category label map.

[0120] Specifically, semantic features of the semantic category label map and reconstruction error features of the reconstruction error map are extracted, and the semantic features and reconstruction error features are fused to generate the anomaly region prediction map. (See [link to relevant documentation]). Figure 6 As shown.

[0121] semantic category label map S and reconstruction error map Feature extraction is performed separately. Considering that semantic category label maps are usually single-channel category index maps, while VGG (Visual Geometry Group) networks pre-trained on ImageNet (a large visualization database used for research on visual object recognition software) typically use three-channel images as input, channel adaptation is performed before inputting the image into the VGG network. Specifically, the semantic category label map can be converted into a one-hot semantic probability representation or category embedding representation, and then mapped to three-channel or specified-channel features compatible with the number of input channels of VGG through 1×1 convolution, lookup table mapping, or channel duplication. For single-channel reconstruction error maps... Alternatively, it can be input into VGG or a lightweight feature extraction module after channel copying or 1×1 convolutional dimensionality upscaling.

[0122] After channel adaptation is completed, the VGG16 network pre-trained on ImageNet is preferred, and Conv3_3 and / or Conv4_3 layers are used as mid-to-high-level features. In other embodiments, VGG11, VGG13, VGG19, MobileNetV3, or lightweight convolutional blocks can also be used instead. VGG11 has lower computational cost but weaker feature representation. VGG13 can be used as an intermediate solution between speed and expressive power. VGG16 is suitable as a compromise between accuracy and speed in this embodiment. VGG19 has deeper feature representation capabilities but higher computational cost. In real-time vehicle scenarios, the choice can be made based on the computing power budget.

[0123]

[0124]

[0125] in, and These represent the semantic category label map and the reconstruction error map, respectively. Channel adaptation mapping before entering the feature extraction network. and These are the semantic feature map formed by semantic features extracted through channel adaptation and VGG network, and the reconstruction error feature map formed by reconstruction error features, respectively.

[0126] Feature maps extracted from VGG network and The connections are made and fused using 1×1 convolution. After normalization, a single-channel anomaly region prediction map corresponding to the spatial size of the input image is obtained. .

[0127]

[0128] Anomaly Area Prediction Map The value of each pixel in The value range is [0, 1], indicating how abnormal the pixel appears. (Abnormal region prediction map) The space dimensions are H×W, that is .

[0129] S702: Calculate the regional risk score based on the anomaly region prediction map and the semantic prediction uncertainty map.

[0130] Specifically, based on the anomaly region prediction map and the semantic prediction uncertainty map, a pixel-level risk value is calculated for each pixel. Based on the pixel-level risk value, a pixel-level risk map is generated. Based on the pixel-level risk map, connected regions are determined. Based on the pixel-level risk values ​​of all pixels contained in the connected regions, a region risk score is calculated.

[0131] Based on the anomaly area prediction map and semantic prediction uncertainty graph The risk value of each pixel is combined with the output of a comprehensive scoring mechanism. A higher risk score indicates that the area is more likely to have potential danger or insufficient perceived reliability.

[0132]

[0133] in, Let (i,j) be the predicted anomaly value at pixel (i,j). Let α be the comprehensive uncertainty value at pixel (i,j), and β be the weighting coefficients of the abnormal prediction value and the comprehensive uncertainty value, respectively. These can be obtained through validation set grid search, Bayesian optimization, or preset according to security requirements.

[0134] From each pixel Pixel-level risk maps can be obtained.

[0135] For connected regions First, calculate the arithmetic mean of the pixel-level risk values ​​of each pixel within the connected region. Then extract the maximum risk value from the pixel-level risk values ​​of each pixel within the connected region. Finally, the regional risk score is obtained based on the arithmetic mean and the maximum risk value. .

[0136] Regional risk scores can be expressed as:

[0137] Here, mean() represents the arithmetic mean function, and max() represents the maximum value function. Connected region Regional risk score, This is the weighting coefficient for the highest risk item.

[0138] Further, in one embodiment, determining the connected region based on the pixel-level risk map includes: traversing each pixel of the pixel-level risk map, marking pixels with pixel-level risk values ​​greater than a set risk threshold as high-risk candidate points and recording them as 1, and recording the remaining pixels as 0; performing connected component analysis on the high-risk candidate points, and aggregating all interconnected high-risk candidate points into a connected region.

[0139] Risk thresholds can be set according to actual needs.

[0140] For example, perform connectivity analysis on the pixels marked "1". Two pixels marked "1" are considered connected if they are adjacent (e.g., their four neighbors: top, bottom, left, and right). All interconnected high-risk candidate pixels are aggregated into a region; this is the connected component. .

[0141] Furthermore, in one embodiment, a safety trigger signal is generated based on the regional risk score and a preset safety threshold.

[0142] Based on the aforementioned uncertainty measurement method, when the regional risk score of a connected region exceeds a safety threshold, it indicates that the perception reliability of that region is insufficient or that there is a potential anomaly, requiring a safety trigger signal to be output to the downstream decision-making module. This mechanism first assesses the impact of uncertain regions in the current driving scenario and dynamically adjusts the current driving decision based on the regional risk score.

[0143] The security mechanism is controlled through the following decision rules:

[0144] in, For indicator functions, Connected region Regional risk score, The preset security threshold can be determined jointly based on the false positive rate, false negative rate on the validation set, and the SOTIF security objective.

[0145] This refers to the safety control adjustment amount executed by the downstream control module. The above decision rule uses the regional risk score as the trigger value, and no other uncertainty symbols are set; if direct triggering at the pixel level is required, this trigger value is uniformly denoted as... The notation is the same as that used in the aforementioned semantic prediction uncertainty graph. The decision space may include maintaining the current strategy, decelerating, increasing following distance, prohibiting lane changes, requesting driver intervention, pulling over, or entering a minimum-risk state; the state space may include vehicle speed, heading, target lane, candidate abnormal area location, area risk value, drivable area boundary, and sensor confidence. The loss function may be a weighted average of collision risk, comfort cost, route deviation cost, and uncertainty penalty term. This application outputs the aforementioned risk quantification results and interface variables; the specific control optimizer can be determined based on the vehicle control system configuration.

[0146] After the safety mechanism is triggered, the system needs to dynamically adjust its decision-making strategy. Depending on the specific location and impact of the uncertainty area, the autonomous driving system switches between multiple strategies. For example, when an obstacle is detected, the system may choose to decelerate or switch to a "conservative" driving mode. This decision-making strategy adjustment can be modeled through the following dynamic optimization process:

[0147] in, Let x0 represent the loss function based on the current state and the measurement of uncertainty, where x0 is the current state variable, z0 is the perceived input or the observation of the abnormal region, and u0 is the current decision policy. It is a regularization term used to prevent over-adjustment. This is the regularization coefficient, which controls the degree of dynamic adjustment.

[0148] This application, based on a semantic segmentation model, utilizes PSPNet to extract semantic category probability maps, semantic category label maps, and semantic category boundary maps for anomalous scenes, and then uses a semantic conditional diffusion model to generate reconstructed images. Uncertainty in the prediction distribution is introduced into the semantic category probability map. Confidence interval uncertainty and probability mean variance By combining USCRM to generate dynamic weights, a semantic prediction uncertainty graph is obtained. By comparing the reconstructed image with the input image, the reconstruction error is calculated using the SSIM metric. The reconstruction error features are then concatenated with the semantic features of the semantic category label map and fused using convolutional blocks to ultimately generate an anomaly region prediction map. Combined with anomaly area prediction map With semantic prediction uncertainty graph Design area risk scoring When a high-risk area is detected in an image, a safety trigger signal can be output to the downstream control module, thereby ensuring the safe operation of the vehicle in complex or sudden environments. This application enhances the robustness and adaptability of the autonomous driving system in complex environments through precise anomaly detection and uncertainty measurement, and provides more reliable safety assurance.

[0149] Specifically, the model training and configuration are as follows. The semantic segmentation network can be pre-trained on Cityscapes, CamVid, BDD100K, or other road scene datasets, and then fine-tuned using data containing abnormal obstacles or pseudo-abnormal regions. The PSPNet training loss can use cross-entropy loss, Dice loss, or a weighted combination of both; when there is class imbalance, the weights of small objects and road boundaries can be increased.

[0150] When PSPNet uses Cityscapes 19-class pre-trained parameters to transfer to BDD100K or a self-built road dataset, if the number of semantic categories or category definitions in the target dataset are not completely consistent with Cityscapes, the segmentation head should be replaced or fine-tuned. To ensure the stability of the conditional input dimension of the semantic conditional diffusion model, a mapping from the target category to a unified set of road semantic categories can be established first, projecting the semantic category probability map, semantic category label map, and semantic category boundary map onto a conditional representation with a fixed number of categories. If the number of categories, semantic probability channels, or conditional encoding channels output by the segmentation head changes, the input dimension of the conditional encoding layer of the diffusion model should also be adjusted synchronously, and retraining or at least joint fine-tuning should be performed based on the new conditional representation. It is not advisable to directly reuse diffusion model parameters with mismatched conditional dimensions.

[0151] During training, the semantic conditional diffusion model takes a normal road image and its semantic conditions as input and learns to reconstruct the appearance of a normal road under a given semantic layout. The diffusion model loss includes noise prediction loss, semantic consistency loss, and optional perceptual loss. After training, the input RGB image is processed by PSPNet to obtain semantic conditions during the inference phase, and then the diffusion model generates the reconstructed image.

[0152] The anomaly detection branch includes USCRM, an SSIM reconstruction error calculation module, a VGG16 or lightweight feature extraction module, a convolutional fusion module, and a risk scoring module. USCRM can be trained end-to-end with the anomaly detection branch, or the parameters of PSPNet and the diffusion model can be fixed first, and only the fusion module can be trained. On vehicle platforms with limited computing power, it is preferable to freeze the PSPNet backbone and the diffusion model, and only fine-tune USCRM, the 1×1 convolutional mapping layer, and the anomaly fusion convolutional block.

[0153] The threshold and weight are determined as follows. , as well as The region can be determined by searching the validation set based on anomaly region recall, false positive rate, and security objectives; when the application scenario places greater emphasis on security, this improves... To enhance risk weighting in high-uncertainty areas; when application scenarios place greater emphasis on comfort and false trigger control, it can improve... Or require an anomaly region prediction map With semantic prediction uncertainty graph Both conditions must be met.

[0154] Secondly, embodiments of this application also provide an abnormal environment visual perception device that takes into account the expected functional safety.

[0155] In one embodiment, reference is made to Figure 7 , Figure 7 This is a functional module diagram of an embodiment of a visual perception device for abnormal environments, designed with the intended functional safety in mind for this application. Figure 7 As shown, the visual perception device for abnormal environments, considering the expected functional safety, includes: A semantic processing module is used to perform semantic processing on the input image to obtain a semantic map and a semantic prediction uncertainty map. The semantic map includes a semantic category confidence map, a semantic category label map, and a semantic category boundary map. An image reconstruction module is used to generate a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map. An error generation module is used to calculate the structural similarity between the reconstructed image and the input image to generate a reconstruction error map; The risk scoring module is used to calculate the regional risk score based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map.

[0156] Furthermore, in one embodiment, the semantic processing module performs semantic processing on the input image to generate a semantic map, including: performing deterministic reasoning on the input image through a semantic segmentation network to generate a first semantic category probability map; and generating a semantic category label map, a semantic category boundary map, and a semantic category confidence map based on the first semantic category probability map.

[0157] Furthermore, in one embodiment, the semantic processing module is also used to extract the maximum probability of the semantic category among all semantic categories of each pixel in the first semantic category probability map; and to generate a semantic category confidence map based on the maximum probability of the semantic category of each pixel.

[0158] Furthermore, in one embodiment, the semantic processing module is also used to extract the semantic category with the highest probability among all semantic categories of each pixel in the first semantic category probability map; and to generate a semantic category label map based on the semantic category with the highest probability of each pixel.

[0159] Furthermore, in one embodiment, the semantic processing module is also used to traverse each pixel in the semantic category label map, determine whether the label value of the pixel is consistent with that of its neighboring pixels, and if they are inconsistent, mark the corresponding position of the pixel in the semantic category label map as the semantic category boundary.

[0160] Furthermore, in one embodiment, the semantic processing module is also used to perform morphological gradient calculation or edge detection on the semantic category label map, and mark the pixels whose calculation results exceed a first preset boundary threshold as semantic category boundaries.

[0161] Furthermore, in one embodiment, the semantic processing module is also used to calculate the gradient magnitude of the first semantic category probability map in the spatial direction, traverse each pixel in the first semantic category probability map, and when the spatial gradient magnitude at the pixel exceeds the second preset boundary threshold, mark the corresponding position of the pixel in the first semantic category probability map as the semantic category boundary.

[0162] Furthermore, in one embodiment, the semantic processing module performs semantic processing on the input image to generate a semantic prediction uncertainty map, including: traversing each pixel in the first semantic category probability map, calculating the uncertainty index of the pixel to obtain an uncertainty index set, the uncertainty index set including prediction distribution uncertainty and confidence interval uncertainty; fusing the uncertainty indices in the uncertainty index set to generate a semantic prediction uncertainty map.

[0163] Further, in one embodiment, the semantic processing module performs semantic processing on the input image to generate a semantic prediction uncertainty map, including: performing N random inferences on the input image through a semantic segmentation network to generate N second semantic category probability maps, where N≥2; traversing each pixel in the second semantic category probability map, and calculating the average variance of the probability corresponding to each semantic category of the pixel as an uncertainty index based on the probability of the pixel in the N second semantic category probability maps; traversing each pixel in the first semantic category probability map, calculating the uncertainty index of the pixel to obtain an uncertainty index set, wherein the uncertainty index set includes at least one of prediction distribution uncertainty and confidence interval uncertainty, and the average variance of the probability; and fusing the uncertainty indices in the uncertainty index set to generate a semantic prediction uncertainty map.

[0164] Furthermore, in one embodiment, the semantic processing module is also used to calculate the information entropy of the pixel based on the probability of each semantic category of the pixel, as the uncertainty of the prediction distribution.

[0165] Furthermore, in one embodiment, the semantic processing module is also used to calculate the confidence interval uncertainty based on the probability of the largest semantic category and the probability of the second largest semantic category of the pixel.

[0166] Furthermore, in one embodiment, the semantic processing module fuses the uncertainty indicators in the uncertainty indicator set to generate a semantic prediction uncertainty map, including: inputting the uncertainty indicators in the uncertainty indicator set into the uncertainty semantic channel recalibration module; generating dynamic weights for each uncertainty indicator through the uncertainty semantic channel recalibration module; and performing weighted fusion of each uncertainty indicator according to the dynamic weights to obtain the semantic prediction uncertainty map.

[0167] Furthermore, in one embodiment, the image reconstruction module generates a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map, including: performing feature extraction processing on the semantic category label map to obtain semantic condition features; fusing the semantic category boundary map and the semantic category confidence map with the semantic condition features to construct a semantic condition tensor; and inputting the semantic condition tensor as a condition into a semantic condition diffusion model to generate a reconstructed image consistent with the semantic layout of the input image.

[0168] Furthermore, in one embodiment, the error generation module calculates the structural similarity between the reconstructed image and the input image to generate a reconstruction error map, including: calculating the local window difference between the input image and the reconstructed image using a structural similarity algorithm; and generating the reconstruction error map based on the local window difference.

[0169] Furthermore, in one embodiment, the risk scoring module calculates a regional risk score based on the reconstruction error map, the semantic category label map, and the semantic prediction uncertainty map, including: generating an abnormal region prediction map based on the reconstruction error map and the semantic category label map; and calculating the regional risk score based on the abnormal region prediction map and the semantic prediction uncertainty map.

[0170] Furthermore, in one embodiment, the risk scoring module generates an anomaly region prediction map based on the reconstruction error map and the semantic category label map, including: extracting semantic features of the semantic category label map and reconstruction error features of the reconstruction error map; and fusing the semantic features and the reconstruction error features to generate the anomaly region prediction map.

[0171] Further, in one embodiment, the risk scoring module calculates the region risk score based on the abnormal region prediction map and the semantic prediction uncertainty map, including: calculating the pixel-level risk value of each pixel based on the abnormal region prediction map and the semantic prediction uncertainty map; generating a pixel-level risk map based on the pixel-level risk values; determining connected regions based on the pixel-level risk map; and calculating the region risk score based on the pixel-level risk values ​​of all pixels contained in the connected regions.

[0172] Furthermore, in one embodiment, the risk scoring module determines connected regions based on the pixel-level risk map, including: traversing each pixel of the pixel-level risk map and marking pixels with pixel-level risk values ​​greater than a set risk threshold as high-risk candidate points; performing connected component analysis on the high-risk candidate points and aggregating all interconnected high-risk candidate points into connected regions.

[0173] Further, in one embodiment, the risk scoring module calculates a region risk score based on the pixel-level risk values ​​of all pixels contained in the connected region, including: calculating the arithmetic mean of the pixel-level risk values ​​of each pixel in the connected region; extracting the maximum risk value among the pixel-level risk values ​​of each pixel in the connected region; and obtaining the region risk score based on the arithmetic mean and the maximum risk value.

[0174] Furthermore, in one embodiment, the risk scoring module is also used to generate a safety trigger signal based on the regional risk score and a preset safety threshold.

[0175] The functional implementation of each module in the above-mentioned abnormal environment visual perception device that considers expected functional safety corresponds to each step in the above-mentioned abnormal environment visual perception method embodiment that considers expected functional safety, and its functions and implementation process will not be described in detail here.

[0176] Thirdly, embodiments of this application provide an abnormal environment visual perception device that takes into account expected functional safety. The abnormal environment visual perception device that takes into account expected functional safety can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0177] Reference Figure 8 , Figure 8 This is a schematic diagram of the hardware structure of an abnormal environment visual perception device considering expected functional safety in an embodiment of this application. In this embodiment, the abnormal environment visual perception device considering expected functional safety may include a processor, a memory, a communication interface, and a communication bus.

[0178] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0179] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of visual sensing devices in abnormal environments while considering expected functional safety, as well as interfaces used for interconnecting these devices with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0180] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0181] The processor can be a general-purpose processor, which can call a visual perception program for abnormal environments that considers expected functional safety stored in memory and execute the visual perception method for abnormal environments that considers expected functional safety provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the visual perception program for abnormal environments that considers expected functional safety is called can refer to the various embodiments of the visual perception method for abnormal environments that considers expected functional safety in this application, and will not be repeated here.

[0182] Those skilled in the art will understand that Figure 8 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0183] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0184] The present application stores an abnormal environment visual perception program that takes into account expected functional safety on a computer-readable storage medium, wherein when the abnormal environment visual perception program that takes into account expected functional safety is executed by a processor, it implements the steps of the abnormal environment visual perception method that takes into account expected functional safety as described above.

[0185] The method implemented when the abnormal environment visual perception program considering expected functional safety is executed can be referred to in various embodiments of the abnormal environment visual perception method considering expected functional safety in this application, and will not be repeated here.

[0186] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A visual perception method for abnormal environments that considers expected functional safety, characterized in that, include: The input image is subjected to deterministic reasoning through a semantic segmentation network to generate a first semantic category probability map. A semantic category label map, a semantic category boundary map, and a semantic category confidence map are generated based on the first semantic category probability map. Each pixel in the first semantic category probability map is traversed, and the uncertainty index of the pixel is calculated to obtain an uncertainty index set. The uncertainty index set includes prediction distribution uncertainty and confidence interval uncertainty. The uncertainty indicators in the uncertainty indicator set are fused to generate a semantic prediction uncertainty map; A reconstructed image is generated using the semantic category confidence map, semantic category label map, and semantic category boundary map. Calculate the structural similarity between the reconstructed image and the input image to generate a reconstruction error map; The regional risk score is calculated based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map.

2. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that: Extract the maximum probability of the semantic category from all semantic categories of each pixel in the first semantic category probability map; generate a semantic category confidence map based on the maximum probability of the semantic category of each pixel; And / or, extract the semantic category with the highest probability among all semantic categories of each pixel in the first semantic category probability map; generate a semantic category label map based on the semantic category with the highest probability of each pixel.

3. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that: Traverse each pixel in the semantic category label graph and determine whether the label value of the pixel is consistent with that of its neighboring pixels. If they are inconsistent, mark the corresponding position of the pixel in the semantic category label graph as the semantic category boundary. Alternatively, perform morphological gradient calculation or edge detection on the semantic category label map, and mark the pixels whose calculation results exceed the first preset boundary threshold as semantic category boundaries; Alternatively, the gradient magnitude of the first semantic category probability map in the spatial direction is calculated, and each pixel in the first semantic category probability map is traversed. When the spatial gradient magnitude at a pixel exceeds a second preset boundary threshold, the corresponding position of the pixel in the first semantic category probability map is marked as a semantic category boundary.

4. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that, The uncertainty index set also includes probability average variance, and the method further includes: The input image is subjected to N random inferences through a semantic segmentation network, generating N second semantic category probability maps, where N≥2; Traverse each pixel in the second semantic category probability map, and calculate the average variance of the probability corresponding to each semantic category of the pixel based on the probability of the pixel in N second semantic category probability maps, as an uncertainty index.

5. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that: Based on the probability of each semantic category of the pixel, the information entropy of the pixel is calculated as the uncertainty of the prediction distribution; And / or, calculate the confidence interval uncertainty based on the probability of the largest semantic category and the probability of the second largest semantic category of the pixel.

6. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that, The uncertainty indicators in the uncertainty indicator set are fused to generate a semantic prediction uncertainty map, including: Input the uncertainty indicators from the uncertainty indicator set into the uncertainty semantic channel recalibration module; The dynamic weights of each uncertain index are generated through the uncertain semantic channel recalibration module. The uncertain indicators are weighted and fused according to the dynamic weights to obtain the semantic prediction uncertainty map.

7. The visual perception method for abnormal environments considering expected functional safety as described in claim 6, characterized in that: The uncertainty semantic channel recalibration module includes multiple channel description value generation units, channel weight generation units, and uncertainty fusion units; The channel description value generation unit is configured to perform spatial pooling processing on the uncertainty index corresponding to the input to obtain the channel description value of the uncertainty index. The channel weight generation unit is configured to generate dynamic weights corresponding to each uncertainty index by processing the channel description values ​​of each uncertainty index through nonlinear mapping. The uncertainty fusion unit is configured to perform weighted fusion of each uncertainty indicator using the dynamic weights of each uncertainty indicator to obtain the semantic prediction uncertainty map.

8. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that, Generating a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map includes: The semantic category label map is subjected to feature extraction processing to obtain semantic condition features; The semantic category boundary map and the semantic category confidence map are fused with the semantic condition features to construct a semantic condition tensor; The semantic condition tensor is used as a conditional input to the semantic conditional diffusion model to generate a reconstructed image that is consistent with the semantic layout of the input image.

9. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that, Calculating the structural similarity between the reconstructed image and the input image to generate a reconstruction error map includes: The structural similarity algorithm is used to calculate the local window differences between the input image and the reconstructed image; The reconstruction error map is generated based on the local window differences.

10. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that, The regional risk score is calculated based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map, including: An anomaly region prediction map is generated based on the reconstruction error map and the semantic category label map; The regional risk score is calculated based on the anomaly region prediction map and the semantic prediction uncertainty map.

11. The visual perception method for abnormal environments considering expected functional safety as described in claim 10, characterized in that, An anomaly region prediction map is generated based on the reconstruction error map and the semantic category label map, including: Extract the semantic features of the semantic category label map and the reconstruction error features of the reconstruction error map; The semantic features and the reconstruction error features are fused to generate the anomaly region prediction map.

12. The visual perception method for abnormal environments considering expected functional safety as described in claim 10, characterized in that, The risk score of the region is calculated based on the anomaly region prediction map and the semantic prediction uncertainty map, including: Based on the anomaly region prediction map and the semantic prediction uncertainty map, calculate the pixel-level risk value of each pixel. A pixel-level risk map is generated based on the pixel-level risk values; Based on the pixel-level risk map, the connected regions are determined; The region risk score is calculated based on the pixel-level risk values ​​of all pixels contained in the connected region.

13. The visual perception method for abnormal environments considering expected functional safety as described in claim 12, characterized in that, Based on the pixel-level risk map, connected regions are determined, including: Traverse each pixel of the pixel-level risk map and mark pixels with pixel-level risk values ​​greater than a set risk threshold as high-risk candidate points; Perform connectivity analysis on high-risk candidate points to aggregate all interconnected high-risk candidate points into connected regions.

14. The visual perception method for abnormal environments considering expected functional safety as described in claim 12, characterized in that, Based on the pixel-level risk values ​​of all pixels contained in the connected region, a region risk score is calculated, including: Calculate the arithmetic mean of the pixel-level risk values ​​of each pixel within the connected region; Extract the maximum risk value from the pixel-level risk values ​​of each pixel within the connected region; The regional risk score is obtained based on the arithmetic mean and the maximum risk value.

15. The visual perception method for abnormal environments considering expected functional safety as described in claim 1, characterized in that, Also includes: A safety trigger signal is generated based on the regional risk score and the preset safety threshold.

16. A visual perception device for abnormal environments that takes into account the expected functional safety, characterized in that, include: The semantic processing module is used to perform deterministic reasoning on the input image through a semantic segmentation network, generate a first semantic category probability map, generate a semantic category label map, a semantic category boundary map, and a semantic category confidence map based on the first semantic category probability map, traverse each pixel in the first semantic category probability map, calculate the uncertainty index of the pixel to obtain an uncertainty index set, the uncertainty index set including prediction distribution uncertainty and confidence interval uncertainty; and fuse the uncertainty indexes in the uncertainty index set to generate a semantic prediction uncertainty map. An image reconstruction module is used to generate a reconstructed image using the semantic category confidence map, semantic category label map, and semantic category boundary map. An error generation module is used to calculate the structural similarity between the reconstructed image and the input image to generate a reconstruction error map; The risk scoring module is used to calculate the regional risk score based on the reconstruction error map, semantic category label map, and semantic prediction uncertainty map.

17. A visual perception device for abnormal environments that takes into account the expected functional safety, characterized in that, The method includes a processor, a memory, and an abnormal environment visual perception program that considers expected functional safety and is stored in the memory and can be executed by the processor, wherein when the abnormal environment visual perception program that considers expected functional safety is executed by the processor, it implements the steps of the abnormal environment visual perception method that considers expected functional safety as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, It stores an abnormal environment visual perception program that takes into account expected functional safety, wherein when the abnormal environment visual perception program that takes into account expected functional safety is executed by a processor, it implements the steps of the abnormal environment visual perception method that takes into account expected functional safety as described in any one of claims 1 to 15.

Images