Self-cognizant perception method, apparatus and system suitable for harsh environments

Abstract

This invention relates to the field of environmental sensing technology, specifically disclosing a self-aware sensing method, apparatus, and system suitable for harsh environments. The method includes: acquiring environmental data from multiple sensors; performing image enhancement processing on environmental images using a task-driven data augmentation network; extracting features from the enhanced environmental images and environmental point clouds; performing cross-modal attention fusion based on the environmental image features and environmental point cloud features; assessing the health status of the multiple sensors and evaluating the confidence level of the sensing results; if both the confidence level evaluation result and the health status evaluation result meet threshold requirements, then performing sensing processing on the cross-modal environmental fusion features; if the confidence level evaluation result and / or the health status evaluation result of the multiple sensors do not meet the threshold requirements, then returning to adjustment. The self-aware sensing method for harsh environments provided by this invention can achieve the assessment of the sensor's self-aware health status and the dynamic optimization of its sensing capabilities under various harsh operating conditions.

Description

Technical Field

[0001] This invention relates to the field of environmental sensing technology, and in particular to a self-aware sensing method, a self-aware sensing device, and a self-aware sensing system suitable for harsh environments. Background Technology

[0002] With the rapid development of artificial intelligence, sensor technology, and autonomous driving technology, environmental perception has become a core component of various automated systems and intelligent agents. Especially in harsh environments (including extreme temperatures, high humidity, high dust, strong corrosion, and electromagnetic interference), traditional perception systems often face problems such as sensor failure, data anomalies, and decreased perception accuracy, leading to an inability to accurately understand the surrounding environment and impacting overall safety and reliability. Existing environmental perception methods mostly rely on single or limited sensor perception modes, lacking the ability to deeply fuse multi-dimensional and multi-modal information. Furthermore, they often cannot perform self-diagnosis and adaptive adjustment in the face of perception failure or degradation, resulting in insufficient overall system robustness. In addition, existing technologies have limited research on the self-health status (self-awareness) of perception systems in harsh environments. These systems cannot promptly detect their own perception degradation or effectively compensate for or reconstruct perception anomalies, easily leading to misjudgments or perception blind spots, thus limiting their application and promotion in extreme environments such as disaster relief, mining operations, and field exploration.

[0003] Therefore, how to provide a method that enables sensors to self-perceive their health status and dynamically optimize their sensing capabilities under various harsh operating conditions has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a self-aware sensing method, a self-aware sensing device, and a self-aware sensing system suitable for harsh environments, solving the problem in related technologies that cannot achieve sensor self-awareness assessment of health status and dynamic optimization of sensing capabilities under various harsh working conditions.

[0005] As a first aspect of the present invention, a self-aware perception method suitable for harsh environments is provided, comprising:

[0006] Acquire environmental data from multiple source sensors, wherein the multiple source sensors include at least an image sensor, a millimeter-wave radar, and a lidar, and the environmental data includes at least environmental image sequences and environmental point cloud data.

[0007] The environmental image sequence is enhanced by a task-driven data augmentation network to obtain an environmental image augmentation sequence. The objective function of the task-driven data augmentation network includes an objective function formed by joint optimization of recovery loss, feature identity loss and high-level task loss.

[0008] Feature extraction is performed on the environmental image enhancement sequence and the environmental point cloud data respectively to obtain environmental image features and environmental point cloud features;

[0009] Cross-modal attention fusion is performed based on the environmental image features and the environmental point cloud features to obtain cross-modal environmental fusion features;

[0010] The health status of the multi-source sensors is assessed based on their operating status, and the confidence level of the sensing results is assessed based on the cross-modal environment fusion characteristics.

[0011] If the confidence assessment result is greater than or equal to the preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to the health threshold, then the cross-modal environment fusion feature is input into the diffusion model-based perception model for perception processing, and the environment perception result is obtained and output.

[0012] If the confidence assessment result is less than the preset confidence threshold and / or the health status assessment result of the multi-source sensor is less than the health threshold, then the weight coefficients of the cross-modal attention fusion of the environmental image features and the environmental point cloud features are adjusted, and / or the sensor redundancy / compensation is adjusted, and the environmental perception result is determined based on the adjusted result.

[0013] Furthermore, based on the adjusted results, a decision is made on whether to output the environmental perception results, including:

[0014] If the number of adjustments is less than the preset adjustment number threshold, and the adjusted confidence assessment result is greater than or equal to the preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to the health threshold, then the adjusted cross-modal environment fusion features are input into the diffusion model-based perception model for perception processing, and the environment perception result is obtained and output.

[0015] If the number of adjustments is greater than or equal to the preset adjustment number threshold and the confidence assessment result after adjustment is less than the preset confidence threshold or the health status assessment result of the multi-source sensor is less than the health threshold, an abnormal prompt message will be issued.

[0016] Furthermore, the task-driven data augmentation network includes: an augmentation network, a high-level perception network, and a feature and identity extraction network. The input of the augmentation network is the input of the task-driven data augmentation network. The output of the augmentation network is connected to the input of the high-level perception network and the input of the feature and identity extraction network, respectively. The output of the high-level perception network is connected to the input of the feature and identity extraction network. The output of the feature and identity extraction network is the output of the task-driven data augmentation network.

[0017] Image enhancement processing of the environmental image sequence is performed according to a task-driven data augmentation network, including:

[0018] The environmental image sequence is input into the enhancement network to obtain the restored image information and the features of the last layer of the enhancement network.

[0019] The restored image information is input into the high-level perception network to obtain the high-level perception result and the features of the last layer of high-level perception.

[0020] Both the features of the last layer of the augmented network and the features of the last layer of the high-level perception are input into the feature identity extraction network to obtain the augmented network identity features and the high-level perception identity features in a unified representation space.

[0021] The recovery loss is calculated based on the restored image information and the preset target image; the feature identity loss is calculated based on the enhanced network identity features and the high-level perception identity features; and the high-level task loss is calculated based on the high-level perception results and the preset task labels.

[0022] The objective function of the task-driven data augmentation network is constructed by jointly optimizing the recovery loss, the feature identity loss, and the high-level task loss.

[0023] The network parameters of the objective function of the task-driven data augmentation network are iteratively updated to achieve image augmentation processing of the environmental image sequence, thereby obtaining an environmental image augmentation sequence.

[0024] Furthermore, the objective function of the task-driven data augmentation network is expressed as follows:

[0025] ,

[0026] in, This represents the objective function of a task-driven data augmentation network. Indicates recovery from losses. This indicates the loss of feature identity. This indicates a loss of high-level mission resources. The weights represent the feature identity loss. The weight representing the loss of high-level tasks;

[0027] in:

[0028] ,

[0029] ,

[0030] ,

[0031] in, This indicates the recovery of image information. Indicates the preset target image. This indicates enhanced online identity features. This indicates the perceived identity characteristics of high-level personnel. Represents the cross-entropy loss function. Indicates category prediction, Indicates the truth value. Represents the bounding box. Represents the smoothing loss function. Represents the weights of the smoothing loss function. This represents the generalized IoU loss. express Weights.

[0032] Further, feature extraction is performed on the environmental image enhancement sequence and the environmental point cloud data respectively to obtain environmental image features and environmental point cloud features, including:

[0033] The environmental image features are obtained by extracting features from the environmental image enhancement sequence based on the feature identity extraction network in the task-driven data augmentation network.

[0034] The environmental point cloud data is used to extract point cloud features based on a point cloud feature extraction network to obtain environmental point cloud features.

[0035] Furthermore, a health status assessment of the multi-source sensors is performed based on their operating states, including:

[0036] The signal quality score of the sensor is calculated based on the sensor's signal-to-noise ratio, echo intensity, and point cloud density.

[0037] The historical error fluctuation of the sensor is estimated based on the trend of the predicted residual and the measurement variance over time.

[0038] The frame rate or missing rate of the sensor is determined based on data integrity and continuity.

[0039] A health index for multi-source sensors is constructed based on the sensor's signal quality score, historical error fluctuation, and frame rate or missing rate.

[0040] The health status of the multi-source sensors is evaluated based on the health indicators of the multi-source sensors to obtain the health status evaluation results.

[0041] Furthermore, the confidence assessment of the perception results based on the cross-modal environment fusion features includes:

[0042] Based on the cross-modal environment fusion features and after adding trainable weight vectors, the data is mapped to the [0, 1] interval;

[0043] The mapped results are output in probability form to obtain the confidence assessment results.

[0044] The closer the confidence assessment result is to 1, the higher the reliability of the perceived result output;

[0045] The closer the confidence assessment result is to 0, the higher the uncertainty of the perceived result.

[0046] Furthermore, the cross-modal environment fusion features are input into a diffusion-based perception model for perception processing, including:

[0047] The cross-modal environment fusion features are denoised using a variational autoencoder to obtain denoised features.

[0048] The environmental perception result is obtained by gradually denoising the noisy features using a neural network.

[0049] As another aspect of the present invention, a self-aware sensing device suitable for harsh environments is provided for implementing the self-aware sensing method for harsh environments described above, wherein the device includes:

[0050] An acquisition module is used to acquire environmental information collected by multiple source sensors, wherein the multiple source sensors include at least an image sensor, a millimeter-wave radar and a lidar, and the environmental information collected includes at least an environmental image sequence and environmental point cloud data.

[0051] The image enhancement processing module is used to perform image enhancement processing on the environmental image sequence according to the task-driven data augmentation network to obtain an environmental image enhancement sequence. The objective function of the task-driven data augmentation network includes an objective function formed by joint optimization based on recovery loss, feature identity loss and high-level task loss.

[0052] The feature extraction module is used to extract features from the environmental image enhancement sequence and the environmental point cloud data respectively, to obtain environmental image features and environmental point cloud features;

[0053] The cross-modal fusion module is used to perform cross-modal attention fusion based on the environmental image features and the environmental point cloud features to obtain cross-modal environmental fusion features;

[0054] The health status and confidence assessment module is used to assess the health status of the multi-source sensors based on their working status, and to assess the confidence of the perception results based on the cross-modal environment fusion features.

[0055] The perception processing module is used to input the cross-modal environment fusion features into the diffusion-based perception model for perception processing if the confidence assessment result is greater than or equal to a preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to a health threshold, so as to obtain and output the environment perception result.

[0056] The feedback adjustment module is used to adjust the weight coefficients of the environmental image features and the environmental point cloud features for cross-modal attention fusion if the confidence assessment result is less than a preset confidence threshold and / or the health status assessment result of the multi-source sensor is less than a health threshold, and / or adjust the sensor redundancy / compensation, and decide whether to output the environmental perception result based on the adjusted result.

[0057] As another aspect of the present invention, a self-aware sensing system suitable for harsh environments is provided, comprising: a multi-source sensor and the aforementioned self-aware sensing device suitable for harsh environments, wherein the multi-source sensor is communicatively connected to the self-aware sensing device suitable for harsh environments.

[0058] This invention provides a self-aware perception method suitable for harsh environments. It acquires environmental data from multiple sensors and performs image enhancement processing on environmental image sequences using a task-driven data augmentation network. Then, it extracts features from both the enhanced environmental image sequences and environmental point cloud data. Finally, it performs cross-modal attention fusion based on these environmental image and point cloud features to obtain cross-modal environmental fusion features. Based on the operating status of the multiple sensors, it assesses their health status and evaluates the confidence level of the perception results based on the cross-modal environmental fusion features. Ultimately, it determines whether to output the environmental perception result based on the evaluation results and can provide feedback adjustments if the evaluation requirements are not met. This self-aware perception method for harsh environments uses multi-source sensor information as input and effectively overcomes the problem of insufficient information from a single sensor in harsh environments through a feature fusion method based on an attention mechanism. Simultaneously, by combining a task-driven data augmentation strategy with sensor health status monitoring and confidence perception mechanisms, it achieves accurate, robust, and adaptive perception output of environmental elements in harsh environments, significantly improving the stability and safety of the system under complex operating conditions. Attached Figure Description

[0059] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.

[0060] Figure 1 A flowchart of the self-cognition perception method for harsh environments provided by the present invention.

[0061] Figure 2 This invention provides a flowchart for determining whether to output environmental perception results based on the adjusted results.

[0062] Figure 3 This is an architecture diagram of the task-driven data augmentation network provided by the present invention.

[0063] Figure 4 This is a flowchart for image enhancement processing of environmental image sequences provided by the present invention.

[0064] Figure 5 The flowcharts for image feature extraction and point cloud feature extraction provided by the present invention are shown.

[0065] Figure 6 A flowchart for assessing the health status of multi-source sensors provided by the present invention.

[0066] Figure 7 A flowchart for the confidence assessment of the perception results provided by the present invention.

[0067] Figure 8 A flowchart of the perception processing provided by the present invention.

[0068] Figure 9 The structural block diagram of the self-recognition sensing device suitable for harsh environments provided by the present invention. Detailed Implementation

[0069] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0070] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0071] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] This embodiment provides a self-aware perception method suitable for harsh environments. Figure 1 This is a flowchart of a self-cognitive perception method suitable for harsh environments provided according to an embodiment of the present invention, such as... Figure 1 As shown, it includes:

[0073] S100. Acquire environmental acquisition information from multiple source sensors, wherein the multiple source sensors include at least an image sensor, a millimeter-wave radar, and a lidar, and the environmental acquisition information includes at least an environmental image sequence and environmental point cloud data.

[0074] In this embodiment of the invention, multi-source sensors can be installed on the autonomous vehicle. These multi-source sensors may include image sensors, millimeter-wave radar, and lidar, etc. Therefore, the environmental sampling information obtained includes at least environmental image sequences and environmental point cloud data.

[0075] S200. The environmental image sequence is enhanced by the task-driven data augmentation network to obtain an environmental image augmentation sequence. The objective function of the task-driven data augmentation network includes an objective function formed by joint optimization of recovery loss, feature identity loss and high-level task loss.

[0076] Specifically, environmental image sequences are enhanced using a task-driven data augmentation network. The objective function of this network is obtained by jointly optimizing the recovery loss, feature identity loss, and high-level task loss. Unlike traditional fixed image augmentation methods, this approach can perform targeted enhancement of input data based on the needs of the perception task and changes in the environment, significantly improving the generalizability and robustness of deep perception networks in extreme environments.

[0077] S300: Extract features from the environmental image enhancement sequence and the environmental point cloud data respectively to obtain environmental image features and environmental point cloud features;

[0078] In this embodiment of the invention, after image enhancement is performed on the environmental image sequence, environmental image features are obtained in advance based on the environmental image enhancement sequence, and environmental point cloud features are obtained by feature extraction based on the environmental point cloud data.

[0079] S400. Perform cross-modal attention fusion based on the environmental image features and the environmental point cloud features to obtain cross-modal environmental fusion features;

[0080] In this embodiment of the invention, cross-modal attention fusion is performed on environmental image features and environmental point cloud features, that is, the features under different modalities are fused based on attention features to obtain cross-modal environmental fusion features.

[0081] S500: Assess the health status of the multi-source sensors based on their operating status, and assess the confidence level of the sensing results based on the cross-modal environment fusion features.

[0082] In this embodiment of the invention, the operating status of multi-source sensors is evaluated, and the confidence level of the sensing results for cross-modal environmental fusion features is also assessed. That is, through a dual evaluation of the self-sensing of multi-source sensors and the confidence level of the sensing results, the operating status of the sensors and the reliability of their sensing results are perceived in real time. The sensing measurements are then dynamically adjusted based on the health status, thereby enhancing overall safety and sustainability.

[0083] S600. If the confidence assessment result is greater than or equal to the preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to the health threshold, then the cross-modal environment fusion feature is input into the diffusion model-based perception model for perception processing, and the environment perception result is obtained and output.

[0084] In this embodiment of the invention, the final sensing result is obtained only when both the confidence assessment result and the health status assessment result of the multi-source sensor meet the assessment threshold. If either one fails to meet the threshold, feedback adjustment is required.

[0085] S700. If the confidence assessment result is less than the preset confidence threshold and / or the health status assessment result of the multi-source sensor is less than the health threshold, then return to adjust the weight coefficients of the environmental image features and the environmental point cloud features for cross-modal attention fusion, and / or adjust the sensor redundancy / compensation, and decide whether to output the environmental perception result based on the adjusted result.

[0086] It should be understood that when the confidence assessment result does not meet the preset confidence threshold and / or the health status assessment result of the multi-source sensors does not meet the health threshold, the corresponding features will be adjusted accordingly. That is, if the confidence assessment result does not meet the preset confidence threshold, the weight coefficients of the cross-modal attention fusion of the environmental image features and the environmental point cloud features can be adjusted; if the health status assessment result of the multi-source sensors does not meet the health threshold, sensor redundancy / compensation can be adjusted. Whether redundancy or compensation is adjusted depends on the specific circumstances.

[0087] In summary, the self-aware perception method for harsh environments provided by this invention acquires environmental information from multiple sensors, performs image enhancement processing on environmental image sequences using a task-driven data augmentation network, extracts features from both the enhanced environmental image sequences and environmental point cloud data, and then performs cross-modal attention fusion based on the environmental image features and environmental point cloud features to obtain cross-modal environmental fusion features. Based on the operating status of the multiple sensors, the health status of the sensors is assessed, and the confidence level of the perception results is evaluated based on the cross-modal environmental fusion features. Finally, the method determines whether to output environmental perception results based on the evaluation results and can provide feedback adjustments if the evaluation requirements are not met. This self-aware perception method for harsh environments uses multi-source sensor information as input and effectively overcomes the problem of insufficient information from a single sensor in harsh environments through a feature fusion method based on an attention mechanism. Simultaneously, by combining a task-driven data augmentation strategy with sensor health status monitoring and confidence perception mechanisms, it achieves accurate, robust, and adaptive perception output of environmental elements in harsh environments, significantly improving the stability and safety of the system under complex operating conditions.

[0088] In embodiments of the present invention, such as Figure 2 As shown, the decision on whether to output the environment awareness result is based on the adjusted result, including:

[0089] S710. If the number of adjustments is less than the preset adjustment number threshold and the adjusted confidence assessment result is greater than or equal to the preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to the health threshold, then the adjusted cross-modal environment fusion feature is input into the diffusion model-based perception model for perception processing, and the environment perception result is obtained and output.

[0090] S720. If the number of adjustments is greater than or equal to the preset adjustment number threshold and the confidence assessment result after adjustment is less than the preset confidence threshold or the health status assessment result of the multi-source sensor is less than the health threshold, then an abnormal prompt message is issued.

[0091] It should be understood that when adjusting the weight coefficients of the environmental image features and the environmental point cloud features for cross-modal attention fusion, and / or adjusting sensor redundancy / compensation, if the requirements are still not met after one adjustment, the adjustment can be repeated. Specifically, a preset threshold for the number of adjustments can be set, for example, 5 times. If the results after 5 adjustments still do not meet the requirements, then there is an anomaly in the task, such as a problem with all sensors that cannot be resolved by feedback adjustments. In this case, an anomaly prompt should be issued. If the requirements are met in less than 5 adjustments, then perception processing can be performed based on the adjusted results, and the environmental perception results can be output.

[0092] Therefore, by setting this preset number of adjustments, when all problems with sensors and other components cannot be resolved through feedback adjustments, the system will not be stuck in a feedback adjustment loop. Instead, it can issue an error message, thus enabling the self-aware perception method to operate safely and reliably.

[0093] In embodiments of the present invention, such as Figure 3 As shown, the task-driven data augmentation network includes: an augmentation network, a high-level perception network, and a feature and identity extraction network. The input of the augmentation network is the input of the task-driven data augmentation network. The output of the augmentation network is connected to the input of the high-level perception network and the input of the feature and identity extraction network, respectively. The output of the high-level perception network is connected to the input of the feature and identity extraction network, and the output of the feature and identity extraction network is the output of the task-driven data augmentation network.

[0094] The environmental image sequence is enhanced using a task-driven data augmentation network, such as... Figure 4 As shown, it includes:

[0095] S210. Input the environmental image sequence into the enhancement network to obtain the restored image information and the features of the last layer of the enhancement network;

[0096] In this invention, harsh environments are captured in environmental image sequences. Input to the enhanced network The enhanced network outputs restored image information. and enhance the features of the last layer of the network .

[0097] S220. Input the restored image information into the high-level perception network to obtain the high-level perception result and the features of the last layer of high-level perception.

[0098] In this embodiment of the invention, the restored image information Input to high-level sensing network To obtain high-level perception results and the last layer of features of high-level perception It should be noted that the parameters of the high-level perceptual network remain frozen during training, providing stable semantic constraints only for the task teacher network (which can be understood as a network whose parameters are fixed during training and do not participate in updates; here, it can be understood as the high-level perceptual network).

[0099] S230. Input both the last layer features of the augmented network and the last layer features of the high-level perception into the feature identity extraction network to obtain the augmented network identity features and the high-level perception identity features in a unified representation space.

[0100] In this embodiment of the invention, to further ensure that the features output by the augmentation network have the same discriminative ability as the high-level task, a feature identity extraction network is introduced between the final-layer features of the augmentation network and the high-level perception network. , respectively and Inputting this feature-based identity extraction network yields identity features from the unified representation space for the identity feature enhancement network. High-level perception of identity characteristics Furthermore, the feature identity loss function is used to constrain both, thereby maintaining the consistency between different network features.

[0101] S240. Calculate the recovery loss based on the restored image information and the preset target image, calculate the feature identity loss based on the enhanced network identity features and the high-level perception identity features, and calculate the high-level task loss based on the high-level perception results and the preset task labels.

[0102] Specifically, recovering image information With target image The recovery loss is calculated between layers to ensure that the augmentation network has good image reconstruction capabilities and structural fidelity; high-level perception results With task tags The high-level task loss is calculated between these components to maintain the usability and accuracy of the enhanced image in high-level perception tasks.

[0103] S250. Jointly optimize the recovery loss, the feature identity loss, and the high-level task loss to construct the objective function of the task-driven data augmentation network.

[0104] It should be understood that by jointly optimizing the three objective functions—recovery loss, feature identity loss, and high-level task loss—the network parameters can be enhanced. The network parameters ϕ for feature extraction are iteratively updated to enhance and optimize the features of input images in harsh environments.

[0105] In this embodiment of the invention, the objective function of the task-driven data augmentation network is expressed as:

[0106] ,

[0107] in, This represents the objective function of a task-driven data augmentation network. Indicates recovery from losses. This indicates the loss of feature identity. This indicates a loss of high-level mission resources. The weights represent the feature identity loss. The weight representing the loss of high-level tasks;

[0108] in:

[0109] ,

[0110] ,

[0111] ,

[0112] in, This indicates the recovery of image information. Indicates the preset target image. This indicates enhanced online identity features. This indicates the perceived identity characteristics of high-level personnel. Represents the cross-entropy loss function. Indicates category prediction, Indicates the truth value. Represents the bounding box. Represents the smoothing loss function. Represents the weights of the smoothing loss function. This represents the generalized IoU loss. express Weights.

[0113] In this embodiment of the invention, by minimizing the total loss To optimize the parameters, the restoration loss directly constrains the consistency between the augmentation network output and the real image in the grayscale / color space by comparing the differences between the restored image and the target image pixel by pixel. The feature identity loss uses cosine similarity to constrain the features of the final augmentation layer. Features of the last layer of high-level networks Consistency within the identity space; finally, object detection loss is used as the loss function for high-level tasks, where... and For category prediction and true value, and For bounding box, For smoothing , This refers to the loss in the generalized IoU.

[0114] S260. Iteratively update the network parameters of the objective function of the task-driven data augmentation network to achieve image augmentation processing of the environmental image sequence and obtain an environmental image augmentation sequence.

[0115] It should be understood that this implementation ensures that the enhancement network can not only improve the visual quality of degraded images, but also output feature representations that are more friendly to downstream perception tasks, thereby enhancing the stability and robustness of the overall perception system in complex environments.

[0116] In this embodiment of the invention, feature extraction is performed on the environmental image enhancement sequence and the environmental point cloud data respectively to obtain environmental image features and environmental point cloud features, such as... Figure 5 As shown, it includes:

[0117] S310. After extracting features from the environmental image enhancement sequence based on the feature identity extraction network in the task-driven data augmentation network, environmental image features are obtained.

[0118] S320. Extract point cloud features from the environmental point cloud data using a point cloud feature extraction network to obtain environmental point cloud features.

[0119] In this embodiment of the invention, the PointNet++ point cloud feature extraction network is used to extract point cloud features, which are then fused with the output of the feature identity network across modalities through attention, and used as the input of the perception model.

[0120] It should be noted that, in this embodiment of the invention, the cross-modal attention fusion can specifically employ a Cross-Attention mechanism: by calculating the cross-attention weights between visual features and point cloud / radar features, information interaction and dynamic weighting between different modalities are achieved. The specific implementation of this cross-attention method is well known to those skilled in the art and will not be elaborated upon here.

[0121] In this embodiment of the invention, the health status of the multi-source sensors is assessed based on their operating states, such as... Figure 6 As shown, it includes:

[0122] S510a. Calculate the sensor's signal quality score based on the sensor's signal-to-noise ratio, echo intensity, and point cloud density.

[0123] S520a. Estimate the historical error fluctuation of the sensor based on the trend of the predicted residual and the measurement variance over time.

[0124] S530a. Determine the sensor's frame rate or missing rate based on data integrity and continuity;

[0125] S540a: Construct a health index for multi-source sensors based on the sensor's signal quality score, historical error fluctuation, and frame rate or missing rate.

[0126] S550a. The health status of the multi-source sensor is evaluated based on the health index of the multi-source sensor to obtain the health status evaluation result.

[0127] Specifically, for each type of sensor Define health indicators The formula used to quantify its working status is as follows:

[0128] ,

[0129] in, The signal quality score can be calculated from indicators such as signal-to-noise ratio (SNR), echo intensity, and point cloud density. Historical error fluctuations are typically estimated by the changing trends of prediction residuals and measurement variance over time. This represents the frame rate or missing rate, used to characterize data integrity and continuity. (Function) Weighted linear combinations or normalized nonlinear mappings can be used to ensure the comparability of indicators across different dimensions. Then, based on the sensor's health status, confidence levels are estimated for the sensing results. Using the above formula, the sensor's health score at a given time can be obtained. And compare it with the set health threshold n.

[0130] In this embodiment of the invention, the confidence level of the perception result is evaluated based on the cross-modal environment fusion features, such as... Figure 7 As shown, it includes:

[0131] S510b: Based on the cross-modal environment fusion features and after adding trainable weight vectors, map them to the [0, 1] interval;

[0132] S520b: Output the mapped result in probability form to obtain the confidence assessment result;

[0133] S530b: The closer the confidence assessment result is to 1, the higher the reliability of the perceived result output.

[0134] S540b. The closer the confidence assessment result is to 0, the higher the uncertainty of the perceived result.

[0135] Specifically, the perception results after multi-source feature fusion The confidence level is assessed using the following formula:

[0136] ,

[0137] in, This represents the fused feature representation obtained through a multimodal attention mechanism or feature concatenation. Given a trainable weight vector, sigmod() is used to map the linear score to the [0,1] interval, outputting the confidence level of the perceived result in probabilistic form. .

[0138] It should be understood that when The closer the value is to 1, the higher the reliability of the perceived output; conversely, when... A lower value indicates significant uncertainty in the current sensing result. An anomaly handling mechanism is triggered by jointly assessing the sensor's health status and the confidence level of the sensing result. (Confidence threshold) or When the (health threshold) is reached, an anomaly detection alert is triggered and redundant sensor compensation is performed. Among these, Indicates the confidence threshold. This indicates the health threshold. At this point, redundant sensors or historical information will be automatically invoked for compensation, and different modal inputs will be re-weighted and fused to maintain the continuity and reliability of the overall sensing link.

[0139] Through the above mechanism, the present invention can dynamically sense the working status of sensors and the reliability of sensing results in harsh environments, realize self-diagnosis, adaptive adjustment and fault-tolerant operation, and provide more stable and reliable input support for downstream decision-making modules.

[0140] In this embodiment of the invention, the cross-modal environment fusion features are input into a diffusion-based perception model for perception processing, such as... Figure 8 As shown, it includes:

[0141] S610. Noise-adding processing is performed on the cross-modal environment fusion features according to the variational autoencoder to obtain the noise-added features;

[0142] First, a VAE (Variational Autoencoder) is used to encode multi-source features into a latent space representation. And gradually add noise to generate noisy features. As shown in the formula:

[0143] ,

[0144] in, This represents the noise scheduling parameter, specifically the proportion of the original signal retained at step t. The standard Gaussian noise term, ϵ∼𝒩(0,1), is used to gradually introduce random noise into the latent variables;

[0145] S620. The noisy features are gradually denoised using a neural network to obtain the environmental perception result.

[0146] During the diffusion modeling phase, simulated Gaussian noise is gradually incorporated into this latent representation, resulting in multiple noisy versions. In the reverse process phase, the perceptual model uses fused features as conditional input and gradually denoises through a neural network, as shown in the formula:

[0147] ,

[0148] This method reconstructs clearer and more structurally consistent features or original inputs (such as images or point clouds) to achieve the recovery and enhanced perception of disturbed information. To further improve the model's adaptability to different input conditions, this embodiment of the invention employs a "classifier-free guidance" mechanism to guide and control perception tasks such as image restoration and target recognition under unsupervised conditions. Finally, the decoder restores the recovered latent representation into semantic segmentation maps, detection boxes, or reconstructed images and point cloud data for downstream environmental mapping and decision-making. Simultaneously, the sampling steps and guidance intensity of the diffusion process can be dynamically adjusted based on sensor health status and confidence assessment results, thereby maintaining the stability and output reliability of the perception system even under resource-constrained or sensor malfunction conditions. Compared to traditional perception models, the overall method exhibits stronger robustness and recovery capabilities when dealing with complex interferences such as occlusion, degradation, and illumination changes, significantly improving the level of intelligent perception in harsh environments.

[0149] In summary, the self-aware perception method for harsh environments provided by this invention, by introducing task-driven data augmentation strategies and a multimodal attention fusion mechanism, can dynamically adjust the perception data and fusion method according to actual environmental changes, significantly improving the perception accuracy and robustness in complex scenarios such as rain, snow, fog, and strong light at night. Simultaneously, based on a stepwise backward reasoning mechanism using a diffusion model, this invention can gradually reconstruct environmental information from interfered or low-quality inputs, effectively improving the recognition accuracy of occluded objects, weak targets, and degraded features, outperforming traditional CNN or Transformer perception models. By designing a sensor health status monitoring module and a self-aware perception confidence mechanism, it can identify sensor performance anomalies and data degradation in real time and quantify the credibility of perception results, providing reliable input for downstream decision-making modules and effectively improving the system's security and interpretability in critical scenarios. Therefore, this invention constructs a closed-loop perception framework integrating data augmentation, feature fusion, perception recovery, and credible perception, achieving self-optimization and self-adjustment throughout the entire process from perception input to perception output, significantly outperforming traditional perception systems that operate in isolation within the "perception-decision-execution" chain. It has good scalability and practicality, and can be widely applied to intelligent systems in various complex environments such as autonomous driving, unmanned inspection, smart mines, and high-risk industrial scenarios.

[0150] As another embodiment of the present invention, a self-aware perception device 100 suitable for harsh environments is provided to implement the self-aware perception method for harsh environments described above, wherein, as Figure 9 As shown, it includes:

[0151] The acquisition module 110 is used to acquire environmental acquisition information from multiple source sensors, wherein the multiple source sensors include at least an image sensor, a millimeter-wave radar, and a lidar, and the environmental acquisition information includes at least environmental image sequences and environmental point cloud data.

[0152] Image enhancement processing module 120 is used to perform image enhancement processing on the environmental image sequence according to the task-driven data augmentation network to obtain an environmental image enhancement sequence. The objective function of the task-driven data augmentation network includes an objective function formed by joint optimization based on recovery loss, feature identity loss and high-level task loss.

[0153] Feature extraction module 130 is used to extract features from the environmental image enhancement sequence and the environmental point cloud data respectively to obtain environmental image features and environmental point cloud features;

[0154] The cross-modal fusion module 140 is used to perform cross-modal attention fusion based on the environmental image features and the environmental point cloud features to obtain cross-modal environmental fusion features;

[0155] The health status and confidence assessment module 150 is used to assess the health status of the multi-source sensors based on their working status, and to assess the confidence of the perception results based on the cross-modal environment fusion features.

[0156] The perception processing module 160 is used to input the cross-modal environment fusion features into the diffusion model-based perception model for perception processing if the confidence assessment result is greater than or equal to a preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to a health threshold, so as to obtain and output the environment perception result.

[0157] The feedback adjustment module 170 is used to adjust the weight coefficients of the environmental image features and the environmental point cloud features for cross-modal attention fusion if the confidence assessment result is less than a preset confidence threshold and / or the health status assessment result of the multi-source sensor is less than a health threshold, and / or adjust the sensor redundancy / compensation, and decide whether to output the environmental perception result based on the adjusted result.

[0158] This invention provides a self-aware perception device suitable for harsh environments. It acquires environmental information from multiple sensors and performs image enhancement processing on environmental image sequences using a task-driven data augmentation network. Then, it extracts features from both the enhanced environmental image sequences and environmental point cloud data. Finally, it performs cross-modal attention fusion based on the environmental image and point cloud features to obtain cross-modal environmental fusion features. Based on the operating status of the multiple sensors, it assesses their health status and evaluates the confidence level of the perception results based on the cross-modal environmental fusion features. Ultimately, it determines whether to output environmental perception results based on the evaluation results and can provide feedback adjustments if the evaluation requirements are not met. This self-aware perception device for harsh environments uses multi-source sensor information as input and effectively overcomes the problem of insufficient information from a single sensor in harsh environments through a feature fusion method based on an attention mechanism. Simultaneously, by combining a task-driven data augmentation strategy with sensor health status monitoring and confidence perception mechanisms, it achieves accurate, robust, and adaptive perception output of environmental elements in harsh environments, significantly improving the stability and safety of the system under complex operating conditions.

[0159] The specific working principle of the self-cognition sensing device for harsh environments provided by this invention can be referred to the description of the self-cognition sensing method for harsh environments above, and will not be repeated here.

[0160] As another embodiment of the present invention, a self-aware perception system suitable for harsh environments is provided, comprising: a multi-source sensor and the self-aware perception device suitable for harsh environments described above, wherein the multi-source sensor is communicatively connected to the self-aware perception device suitable for harsh environments.

[0161] This invention provides a self-aware perception system suitable for harsh environments. Employing task-driven data augmentation, unlike traditional fixed-rule image augmentation methods, this invention uses a dynamically adjusted data augmentation strategy based on environmental conditions and the perception task, achieving more targeted image and point cloud quality improvement. It can generate specific augmented samples by combining contextual conditions such as weather and lighting, improving the model's generalization ability in harsh environments. For multi-source sensor inputs (such as cameras, LiDAR, and millimeter-wave radar), it proposes an attention-weight adaptive adjustment mechanism to achieve dynamic fusion driven by perception data quality. Perception weights can be adjusted according to different sensor states and input characteristics to address information loss caused by single sensor failure or degradation. A diffusion model is introduced into the perception scene, utilizing its progressive denoising and generation capabilities to restore environmental details under low-quality, occluded, and harsh weather conditions, achieving high-fidelity environmental reconstruction and target recognition. Combining a conditional guidance mechanism and multimodal input makes the perception results more controllable and robust. A real-time health assessment method is introduced, analyzing sensor operating states (such as data rate, noise level, and occlusion rate) to achieve sensor fault early warning. Simultaneously, by combining the confidence estimate output by the perception model, the system can self-judge the reliability of the environmental understanding results, providing credible support for system decision-making.

[0162] Therefore, the self-aware perception system suitable for harsh environments provided by this invention features a closed-loop self-aware perception system architecture. It organically integrates data augmentation, feature fusion, perception recovery, and health monitoring modules to construct a closed-loop optimized perception system. This system can dynamically adjust its operating strategy under different tasks, environments, and fault conditions, maintaining long-term stable operation. In summary, this invention not only achieves breakthroughs in perception accuracy and robustness but also realizes self-awareness, self-adaptation, and self-adjustment capabilities at the system level, providing key technical support for the stable operation of intelligent systems in extreme environments.

[0163] The specific working principle of the self-cognition sensing system for harsh environments provided by this invention can be found in the description of the self-cognition sensing method for harsh environments above, and will not be repeated here.

[0164] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A self-aware sensing method suitable for harsh environments, characterized in that, include: Acquire environmental data from multiple source sensors, wherein the multiple source sensors include at least an image sensor, a millimeter-wave radar, and a lidar, and the environmental data includes at least environmental image sequences and environmental point cloud data. The environmental image sequence is enhanced by a task-driven data augmentation network to obtain an environmental image augmentation sequence. The objective function of the task-driven data augmentation network includes an objective function formed by joint optimization of recovery loss, feature identity loss and high-level task loss. Feature extraction is performed on the environmental image enhancement sequence and the environmental point cloud data respectively to obtain environmental image features and environmental point cloud features; Cross-modal attention fusion is performed based on the environmental image features and the environmental point cloud features to obtain cross-modal environmental fusion features; The health status of the multi-source sensors is assessed based on their operating status, and the confidence level of the sensing results is assessed based on the cross-modal environment fusion characteristics. If the confidence assessment result is greater than or equal to the preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to the health threshold, then the cross-modal environment fusion feature is input into the diffusion model-based perception model for perception processing, and the environment perception result is obtained and output. If the confidence assessment result is less than the preset confidence threshold and / or the health status assessment result of the multi-source sensor is less than the health threshold, then the weight coefficients of the cross-modal attention fusion of the environmental image features and the environmental point cloud features are adjusted, and / or the sensor redundancy / compensation is adjusted, and the environmental perception result is determined based on the adjusted result. The task-driven data augmentation network includes an augmentation network, a high-level perception network, and a feature and identity extraction network. The input of the augmentation network is the input of the task-driven data augmentation network. The output of the augmentation network is connected to the input of the high-level perception network and the input of the feature and identity extraction network, respectively. The output of the high-level perception network is connected to the input of the feature and identity extraction network. The output of the feature and identity extraction network is the output of the task-driven data augmentation network. Image enhancement processing of the environmental image sequence is performed according to a task-driven data augmentation network, including: The environmental image sequence is input into the enhancement network to obtain the restored image information and the features of the last layer of the enhancement network. The restored image information is input into the high-level perception network to obtain the high-level perception result and the features of the last layer of high-level perception. Both the features of the last layer of the augmented network and the features of the last layer of the high-level perception are input into the feature identity extraction network to obtain the augmented network identity features and the high-level perception identity features in a unified representation space. The recovery loss is calculated based on the restored image information and the preset target image; the feature identity loss is calculated based on the enhanced network identity features and the high-level perception identity features; and the high-level task loss is calculated based on the high-level perception results and the preset task labels. The objective function of the task-driven data augmentation network is constructed by jointly optimizing the recovery loss, the feature identity loss, and the high-level task loss. The network parameters of the objective function of the task-driven data augmentation network are iteratively updated to achieve image augmentation processing of the environmental image sequence, thereby obtaining an environmental image augmentation sequence.

2. The self-cognition perception method suitable for harsh environments according to claim 1, characterized in that, Whether to output the environment awareness results based on the adjusted results includes: If the number of adjustments is less than the preset adjustment number threshold, and the adjusted confidence assessment result is greater than or equal to the preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to the health threshold, then the adjusted cross-modal environment fusion features are input into the diffusion model-based perception model for perception processing, and the environment perception result is obtained and output. If the number of adjustments is greater than or equal to the preset adjustment number threshold and the confidence assessment result after adjustment is less than the preset confidence threshold or the health status assessment result of the multi-source sensor is less than the health threshold, an abnormal prompt message will be issued.

3. The self-cognition perception method suitable for harsh environments according to claim 1, characterized in that, The objective function of the task-driven data augmentation network is expressed as follows: ， in, This represents the objective function of a task-driven data augmentation network. Indicates recovery from losses. Indicates feature identity loss, This indicates a loss of high-level mission resources. The weights represent the feature identity loss. Indicates the weight of high-level task losses; in: ， ， ， in, This indicates the recovery of image information. Indicates the preset target image. This indicates enhanced online identity features. This indicates the perceived identity characteristics of high-level personnel. Represents the cross-entropy loss function. Indicates category prediction, Indicates the truth value. Represents the bounding box. Represents the smoothing loss function. Represents the weights of the smoothing loss function. This represents the generalized IoU loss. express Weights.

4. The self-cognitive perception method suitable for harsh environments according to any one of claims 1 to 3, characterized in that, Feature extraction is performed on the environmental image enhancement sequence and the environmental point cloud data respectively to obtain environmental image features and environmental point cloud features, including: The environmental image features are obtained by extracting features from the environmental image enhancement sequence based on the feature identity extraction network in the task-driven data augmentation network. The environmental point cloud data is used to extract point cloud features based on a point cloud feature extraction network to obtain environmental point cloud features.

5. The self-cognitive perception method suitable for harsh environments according to any one of claims 1 to 3, characterized in that, Health status assessment of multi-source sensors is performed based on their operating status, including: The signal quality score of the sensor is calculated based on the sensor's signal-to-noise ratio, echo intensity, and point cloud density. The historical error fluctuation of the sensor is estimated based on the trend of the predicted residual and the measurement variance over time. The frame rate or missing rate of the sensor is determined based on data integrity and continuity. A health index for multi-source sensors is constructed based on the sensor's signal quality score, historical error fluctuation, and frame rate or missing rate. The health status of the multi-source sensors is evaluated based on the health indicators of the multi-source sensors to obtain the health status evaluation results.

6. The self-cognitive perception method suitable for harsh environments according to any one of claims 1 to 3, characterized in that, The confidence assessment of the perception results based on the cross-modal environment fusion features includes: Based on the cross-modal environment fusion features and after adding trainable weight vectors, the data is mapped to the [0, 1] interval; The mapped results are output in probability form to obtain the confidence assessment results. The closer the confidence assessment result is to 1, the higher the reliability of the perceived result output; The closer the confidence assessment result is to 0, the higher the uncertainty of the perceived result.

7. The self-cognitive perception method suitable for harsh environments according to any one of claims 1 to 3, characterized in that, The cross-modal environment fusion features are input into a diffusion-based perception model for perception processing, including: The cross-modal environment fusion features are denoised using a variational autoencoder to obtain denoised features. The environmental perception result is obtained by gradually denoising the noisy features using a neural network.

8. A self-aware sensing device suitable for harsh environments, used to implement the self-aware sensing method suitable for harsh environments as described in any one of claims 1 to 7, characterized in that, include: An acquisition module is used to acquire environmental information collected by multiple source sensors, wherein the multiple source sensors include at least an image sensor, a millimeter-wave radar and a lidar, and the environmental information collected includes at least an environmental image sequence and environmental point cloud data. The image enhancement processing module is used to perform image enhancement processing on the environmental image sequence according to the task-driven data augmentation network to obtain an environmental image enhancement sequence. The objective function of the task-driven data augmentation network includes an objective function formed by joint optimization based on recovery loss, feature identity loss and high-level task loss. The feature extraction module is used to extract features from the environmental image enhancement sequence and the environmental point cloud data respectively, to obtain environmental image features and environmental point cloud features; The cross-modal fusion module is used to perform cross-modal attention fusion based on the environmental image features and the environmental point cloud features to obtain cross-modal environmental fusion features; The health status and confidence assessment module is used to assess the health status of the multi-source sensors based on their working status, and to assess the confidence of the perception results based on the cross-modal environment fusion features. The perception processing module is used to input the cross-modal environment fusion features into the diffusion-based perception model for perception processing if the confidence assessment result is greater than or equal to a preset confidence threshold and the health status assessment result of the multi-source sensor is greater than or equal to a health threshold, so as to obtain and output the environment perception result. The feedback adjustment module is used to adjust the weight coefficients of the environmental image features and the environmental point cloud features for cross-modal attention fusion if the confidence assessment result is less than a preset confidence threshold and / or the health status assessment result of the multi-source sensor is less than a health threshold, and / or adjust the sensor redundancy / compensation, and decide whether to output the environmental perception result based on the adjusted result.

9. A self-cognitive sensing system suitable for harsh environments, characterized in that, include: The multi-source sensor and the self-aware sensing device suitable for harsh environments as described in claim 8, wherein the multi-source sensor is communicatively connected to the self-aware sensing device suitable for harsh environments.

Images