Failure detection method, system and equipment for parallax prediction and medium

Abstract

The invention provides a parallax prediction failure detection method, system and device and a medium, and the method comprises the steps: obtaining a left image and a right image of a target scene, inputting the left image and the right image into a matching cost calculation network, and outputting a corresponding matching cost result; respectively inputting the matching cost result into an uncertainty estimation network and a parallax estimation network, and outputting a corresponding uncertainty prediction result and a parallax prediction result; inputting the uncertainty prediction result and the parallax prediction result into a fusion network, and outputting a corresponding failure detection result; the failure detection result comprises non-failure, scene environment failure, camera calibration parameter failure and lens physical failure. According to the invention, through fusion of multi-network cooperative work, high precision and high reliability of parallax prediction are realized, failure detection and processing are carried out when necessary, and the security and robustness of unmanned aerial vehicle application are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) application technology, and in particular to a method, system, device and medium for failure detection of parallax prediction. Background Technology

[0002] In 3D scene reconstruction methods for UAV scenarios, binocular vision is a key branch of stereo matching technology. Traditional stereo matching methods mainly include four stages: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization. In recent years, deep learning-based stereo matching algorithms have demonstrated excellent performance, but errors still exist in their predicted disparity maps. To evaluate prediction quality, the uncertainty of the network output is estimated by modeling the distribution of disparity errors, thereby scoring the confidence or uncertainty of each disparity value. However, the domain migration problem between the real scene and the training scene, as well as external environmental factors (such as large-area sky interference), can affect image quality and may cause the disparity prediction of the entire image to completely fail. In this case, the disparity prediction of the entire image may be completely wrong, making it impossible to filter out reliable prediction results through a preset confidence threshold. Therefore, an effective method for detecting disparity prediction failure is urgently needed to solve the above problems. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide a method, system, device and medium for failure detection of parallax prediction that overcomes or at least partially solves the above problems.

[0004] To achieve the above and other related objectives, this invention provides a failure detection method for disparity prediction, applied to a convolutional neural network. The convolutional neural network includes a matching cost calculation network, an uncertainty estimation network, a disparity estimation network, and a fusion network. The method includes:

[0005] Obtain the left and right images of the target scene, and input the left and right images into the matching cost calculation network to output the corresponding matching cost results;

[0006] The matching cost results are input into the uncertainty estimation network and the disparity estimation network respectively, and the corresponding uncertainty prediction results and disparity prediction results are output.

[0007] The uncertainty prediction result and the disparity prediction result are input into the fusion network, and the corresponding failure detection result is output; the failure detection result includes no failure, scene environment failure, camera calibration parameter failure and lens physical failure.

[0008] Optionally, before the step of obtaining the left and right images of the target scene, inputting the left and right images into the matching cost calculation network, and outputting the corresponding matching cost results, the method further includes:

[0009] Freeze the parameters of the fusion network, train the uncertainty estimation network and the disparity estimation network, and enable the uncertainty estimation network and the disparity estimation network to learn independently and converge to a stable state;

[0010] Unfreeze the parameters of the fusion network, perform end-to-end joint training of the fusion network with the converged uncertainty estimation network and disparity estimation network, and use the converged uncertainty estimation network and disparity estimation network to guide the learning of the fusion network.

[0011] Optionally, the step of jointly training the fusion network end-to-end with the converged uncertainty estimation network and disparity estimation network, and using the converged uncertainty estimation network and disparity estimation network to guide the learning of the fusion network, includes:

[0012] A sample set containing normal sample images and various failure sample images is constructed, and a classification label is assigned to each sample image in the sample set to form a label set containing four categories: normal, scene environment failure, camera calibration parameter failure, and lens physical failure; wherein, the failure sample images include scene environment failure images generated by simulation or actually acquired, camera calibration parameter failure images, and lens physical failure images.

[0013] The sample set is divided into a training set, a validation set, and a test set according to a predetermined ratio. A neural network architecture is selected as the basis for the fusion network, and its network parameters are initialized.

[0014] The training set and its corresponding labels are input into the fusion network. The cross-entropy loss function is used as the optimization objective. The network parameters are iteratively updated through the backpropagation algorithm and the optimizer to minimize the difference between the predicted labels and the true labels.

[0015] After each training cycle, the performance metrics of the fusion network are evaluated using the validation set. When the validation set loss value converges stably or the accuracy reaches a preset threshold, the early stopping mechanism is triggered to terminate the training, and the model weights with the best performance on the validation set are saved.

[0016] The trained fusion network is evaluated using the test set to comprehensively test its generalization ability and robustness on unknown data. Once the fusion network meets the preset deployment criteria and passes the evaluation, a deployable fusion network is finally obtained.

[0017] Optionally, after the step of inputting the uncertainty prediction result and the disparity prediction result into the fusion network and outputting the corresponding failure detection result, the method further includes:

[0018] When the failure detection result is one or more of the following: scene environment failure, camera calibration parameter failure, lens physical failure, the corresponding response mechanism will be automatically triggered to execute targeted correction measures or compensation strategies.

[0019] Optionally, when the failure detection result is one or more combinations of scene environment failure, camera calibration parameter failure, and lens physical failure, a corresponding response mechanism will be automatically triggered to execute targeted correction measures or compensation strategies, including:

[0020] When the failure detection result indicates that the scene environment is in failure, an environment optimization strategy is triggered or a data re-acquisition process is started to obtain valid images from a new perspective. The user is then prompted through voice broadcast and visual interface text that the current scene environment is limited and that lighting conditions need to be adjusted or the shooting position changed to ensure data validity.

[0021] Optionally, when the failure detection result is one or more combinations of scene environment failure, camera calibration parameter failure, and lens physical failure, a corresponding response mechanism will be automatically triggered to execute targeted correction measures or compensation strategies, including:

[0022] When the failure detection result indicates that the camera calibration parameters have failed, the online automatic camera parameter recalibration program is immediately initiated.

[0023] If the error still exceeds the preset threshold after recalibration or the calibration process fails, the optical distortion correction program will be started for software compensation.

[0024] If software calibration fails, the system will automatically switch to the backup camera and trigger a hardware status check task.

[0025] Optionally, when the failure detection result is one or more combinations of scene environment failure, camera calibration parameter failure, and lens physical failure, a corresponding response mechanism will be automatically triggered to execute targeted correction measures or compensation strategies, including:

[0026] When the failure detection result indicates a physical failure of the lens, the hardware maintenance process is triggered, and the user is informed of the need for hardware repair or replacement of the lens through both voice and text prompts.

[0027] Secondly, the present invention also provides a failure detection system for disparity prediction, applied to a convolutional neural network, wherein the convolutional neural network includes a matching cost calculation network, an uncertainty estimation network, a disparity estimation network, and a fusion network, and the system includes:

[0028] The calculation module is used to acquire the left and right images of the target scene, input the left and right images into the matching cost calculation network, and output the corresponding matching cost results;

[0029] The prediction module is used to input the matching cost result into the uncertainty estimation network and the disparity estimation network respectively, and output the corresponding uncertainty prediction result and disparity prediction result;

[0030] The detection module is used to input the uncertainty prediction result and the disparity prediction result into the fusion network and output the corresponding failure detection result; the failure detection result includes no failure, scene environment failure, camera calibration parameter failure and lens physical failure.

[0031] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to cause the electronic device to perform the steps of the disparity prediction failure detection method as described above.

[0032] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the steps of the failure detection method for disparity prediction as described above.

[0033] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects:

[0034] This invention achieves high accuracy and high reliability in parallax prediction by integrating multiple networks working together, and performs failure detection and handling when necessary, effectively improving the safety and robustness of UAV applications. Attached Figure Description

[0035] Figure 1 The diagram shows a flowchart of a failure detection method for disparity prediction in one embodiment of the present invention.

[0036] Figure 2 The diagram shown is a schematic diagram of a convolutional neural network structure in one embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the functional modules of a failure detection system for disparity prediction in one embodiment of the present invention;

[0038] Figure 4 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0039] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0040] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0041] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0042] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0043] Unless otherwise stated, the term "multiple" means two or more.

[0044] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0045] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0046] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] Please see Figure 1An embodiment of the present invention provides a failure detection method for disparity prediction, applied to a convolutional neural network. The convolutional neural network includes a matching cost calculation network, an uncertainty estimation network, a disparity estimation network, and a fusion network. The method may include the following steps S10 to S30:

[0048] Step S10: Obtain the left and right images of the target scene, input the left and right images into the matching cost calculation network, and output the corresponding matching cost results.

[0049] The convolutional neural network (CNN) includes a matching cost calculation network, an uncertainty estimation network, a disparity estimation network, and a fusion network. For the structure of a CNN, please refer to [link to CNN documentation]. Figure 2 ,in:

[0050] The matching cost calculation network is used to calculate the similarity of pixel pairs between left and right images.

[0051] Uncertainty estimation network is used to analyze matching cost results and predict the uncertainty of each pixel or disparity hypothesis.

[0052] The disparity estimation network is used to calculate the disparity prediction result, i.e., the disparity value of each pixel, based on the matching cost result.

[0053] The fusion network is used to fuse disparity prediction results with uncertainty results. This fusion process detects whether the disparity prediction results are invalid, thereby ensuring the reliability and accuracy of 3D reconstruction.

[0054] The matching cost result, used to represent the similarity between the left and right images by the dot product of feature vectors, can be represented as a vector of width × height × depth.

[0055] In practical implementation, a binocular camera mounted on a drone can capture left and right images of the target scene. These images are then input into a convolutional neural network encoder with shared weights, consisting of three cascaded residual blocks. Through progressive pass-through, each residual block extracts features from the input left and right images, outputting multi-scale left and right feature maps at different depth levels. Subsequently, a matching cost calculation network is used to calculate the similarity of all candidate feature vectors within a preset disparity range for each pixel in the left feature map on the same horizontal line of its corresponding right feature map. Finally, a matching cost volume is generated as the matching cost result. This process can more accurately reflect the matching relationship between pixels, thereby improving the accuracy of disparity estimation.

[0056] Step S20: Input the matching cost result into the uncertainty estimation network and the disparity estimation network respectively, and output the corresponding uncertainty prediction result and disparity prediction result.

[0057] The uncertainty prediction result is an indicator used to quantify the reliability of disparity prediction based on pixels. Its value is negatively correlated with the reliability of the disparity prediction: a high uncertainty value indicates that the current disparity prediction result is significantly affected by environmental factors (such as strong light / backlight, lens dirt, weak texture areas, repetitive textures, occluded boundaries, etc.), resulting in low reliability and potentially large estimation bias. Conversely, a low uncertainty value indicates high reliability and better reflects the true geometric relationships of the scene. This indicator effectively reflects the robustness of the prediction process to environmental noise and imaging defects, providing a reliability assessment basis for subsequent depth estimation or 3D reconstruction tasks.

[0058] The disparity prediction result represents the pixel-level disparity value calculated by the stereo matching algorithm, characterizing the difference in horizontal pixel coordinates of the same 3D scene point in the left and right views of the binocular camera. This difference is converted to obtain the 3D spatial coordinates (i.e., depth information) of objects in the target scene, providing crucial geometric priors and metric basis for subsequent 3D reconstruction, depth perception, and scene understanding.

[0059] In the specific implementation, the matching cost results can be input into the uncertainty estimation network and disparity estimation network deployed in parallel, respectively. Through their respective forward propagation (including deep learning mechanisms such as convolution operations and nonlinear activation), a mapping from the cost space to the information output is achieved. Specifically, the uncertainty estimation network generates pixel-level uncertainty prediction results (i.e., uncertainty maps), where the value of each pixel represents the confidence of the disparity prediction at that location (higher values ​​indicate lower confidence; lower values ​​indicate higher confidence); the disparity estimation network outputs pixel-level visual prediction results (i.e., disparity maps), where the value of each pixel represents the disparity value corresponding to that location (used for subsequent calculation of depth information). The two networks complement each other to provide dual support for subsequent 3D reconstruction or failure detection tasks: the disparity map ensures the accuracy of geometric structure restoration, while the uncertainty map provides confidence guidance for processing complex scenes (such as occlusion and weakly textured regions), thereby enhancing robustness and optimizing the overall reliability and geometric accuracy of the output.

[0060] Step S30: Input the uncertainty prediction result and the disparity prediction result into the fusion network and output the corresponding failure detection result; the failure detection result includes no failure, scene environment failure, camera calibration parameter failure and lens physical failure.

[0061] The failure detection result refers to the result obtained after using the fusion network to complete disparity prediction failure detection, and specifically includes four categories:

[0062] Not invalid (or "normal") indicates that the disparity prediction results are valid and reliable, and can be directly used for subsequent task execution without additional correction or compensation processing;

[0063] Scene environment failure is used to indicate situations where the shooting environment is outside the coverage of the algorithm's robustness, causing the prediction to fail. Specifically, it includes lighting failure (overexposure / underexposure / strong light), weather failure (rain, fog, snow, backlight), texture failure (low texture area, highly reflective surface), and motion failure (high-speed motion blur, scene shaking).

[0064] Camera calibration parameter failure indicates a situation where parallax prediction fails due to drift or errors in camera intrinsic parameters (focal length, distortion coefficients, etc.) or extrinsic parameters (rotation matrix, translation matrix). Specifically, it includes distortion failure (the problem of lens distortion not being effectively corrected due to abnormal intrinsic parameters) and pose failure (the problem of camera position / angle offset due to abnormal extrinsic parameters).

[0065] Lens physical failure is caused by physical contamination, obstruction, or damage to the lens. Specifically, it includes dirt failure (dust, fingerprints, oil, or other contaminants adhering to the lens surface) and obstruction failure (the effective imaging area is limited because the lens is obstructed by foreign objects).

[0066] Accurate identification of failure types provides a basis for system maintenance and optimization decisions: for scene environment issues, shooting conditions can be adjusted or environmental configuration optimized; for camera calibration parameter failures, camera parameters need to be recalibrated; for lens physical failures, hardware needs to be cleaned or replaced. Through detailed diagnosis of failure types, it is ensured that the visual perception system can eliminate potential problems in a targeted manner, guaranteeing its stable operation and reliability in various scenarios.

[0067] In practical implementation, the uncertainty prediction results and disparity prediction results can be used as joint inputs and simultaneously fed into the fusion network. The fusion network relies on deep learning mechanisms (such as feature fusion and attention weighting) to effectively integrate the uncertainty information, thereby performing a global evaluation of the reliability of the disparity prediction. Finally, it outputs a four-class detection result, which directly indicates whether the disparity prediction of the left or right image has failed, and clearly outputs the specific failure type (including no failure, scene environment failure, camera calibration parameter failure, and lens physical failure). Thus, by implementing the above-mentioned failure detection through the fusion network, the long-term stability and security of the UAV visual perception system can be guaranteed.

[0068] In this embodiment, by integrating the collaborative work of multiple networks, high accuracy and high reliability of parallax prediction are achieved, and failure detection and handling are performed when necessary, effectively improving the safety and robustness of UAV applications.

[0069] Based on the foregoing embodiments, a second embodiment of the failure detection method for disparity prediction of the present invention is proposed. In this embodiment, before step S10, the method may further include the following steps S40 to S50:

[0070] Step S40: Freeze the parameters of the fusion network, train the uncertainty estimation network and the disparity estimation network, and enable the uncertainty estimation network and the disparity estimation network to learn independently and converge to a stable state.

[0071] In the specific implementation, the training process is divided into two core stages, and the collaborative optimization of the multi-task network is achieved through staged parameter control. First, the parameters of the fusion network can be frozen, allowing only the uncertainty estimation network and the disparity estimation network to perform gradient updates. This allows the two networks to learn their respective tasks independently without interference from the fusion network and converge to a stable state, outputting reliable disparity maps and corresponding uncertainty metrics. This stage isolates the random initial parameters of the fusion network, avoiding their negative impact on the learning of the sub-networks and ensuring that the sub-networks effectively learn features based on their own data, laying the foundation for subsequent joint optimization.

[0072] Step S50: Unfreeze the parameters of the fusion network, perform end-to-end joint training of the fusion network with the converged uncertainty estimation network and disparity estimation network, and use the converged uncertainty estimation network and disparity estimation network to guide the learning of the fusion network.

[0073] In practical implementation, the parameters of the fusion network can be unfrozen, and the fusion network can be integrated with the converged uncertainty estimation network and disparity estimation network into a unified system through an end-to-end joint training mechanism. The stable output of the sub-network can be used as a guiding signal to optimize the integration capability of the fusion network for multi-task information, realize cross-network collaborative optimization, and thus improve the robustness and accuracy of the overall system output (such as disparity map accuracy and uncertainty calibration).

[0074] Furthermore, in one embodiment, step S50 may further include the following sub-steps A10 to A50:

[0075] Sub-step A10: Construct a sample set containing normal sample images and various failure sample images, and label each sample image in the sample set with a classification label to form a label set containing four categories: normal, scene environment failure, camera calibration parameter failure, and lens physical failure; wherein, the failure sample images include scene environment failure images generated by simulation or actually collected, camera calibration parameter failure images, and lens physical failure images.

[0076] The normal sample images represent baseline images acquired under ideal working conditions (including image pairs of "left" and "right"), free from any failure factors that would degrade the performance of visual algorithms (such as disparity calculation and object detection). These factors include sufficient and uniform lighting, clear and rich scene textures, accurate camera calibration, clean and undamaged lenses, and favorable environmental conditions free from interference, including the absence of severe weather such as rain, fog, or snow, or significant obstructions. These images serve as baseline data for the training set, constructing a "normal" state reference model for the fusion network. By fully learning the stable visual characteristics of these samples, the network can develop a sensitive ability to identify "failure" states, accurately detecting and classifying failure types when the input image deviates from the aforementioned ideal conditions.

[0077] Failure sample images are used to represent images containing one or more failure factors that cause a significant decrease or complete failure in the performance of visual algorithms (such as disparity calculation and object detection); these include: scene environment failure images, camera calibration parameter failure images, and lens physical failure images.

[0078] The sample set is used to represent a large number of normal sample images and various types of failed sample images. It is divided into three mutually exclusive subsets according to a predetermined ratio (such as 70% training set, 15% validation set, 15% test set, or adjusted according to actual needs): training set, validation set, and test set.

[0079] The label set defines the category to which a sample belongs, and contains four mutually exclusive labels: normal, scene environment failure, camera calibration failure, and lens physical failure. The labels must strictly correspond to the failure types in the sample set to ensure classification consistency.

[0080] In this embodiment, a sample set containing normal sample images and various failed sample images can be created, and a label set containing four types of modes can be defined: normal, scene environment failure, camera calibration failure, and lens physical failure; then, a corresponding label can be assigned to each pair of images in the sample set.

[0081] Sub-step A20: Divide the sample set into a training set, a validation set, and a test set according to a predetermined ratio, select a neural network architecture as the basis for the fusion network, and initialize its network parameters.

[0082] In this embodiment, the labeled sample set can be divided according to a predetermined ratio (e.g., 70% training set, 15% validation set, and 15% test set), and the samples of each category can be balanced in the three subsets respectively. Then, according to the application scenario (e.g., computing resources, real-time requirements), a suitable neural network architecture (e.g., ResNet) can be selected as the basis of the fusion network and its network parameters can be initialized (e.g., random initialization or loading pre-trained weights).

[0083] Sub-step A30: Input the training set and its corresponding labels into the fusion network, use the cross-entropy loss function as the optimization objective, and iteratively update the network parameters through the backpropagation algorithm and optimizer to minimize the difference between the predicted labels and the true labels.

[0084] In practice, the training set (containing normal samples and various types of failed sample images) and their corresponding labels can be input into the fusion network. The prediction results are output through forward propagation, and the difference between the predicted labels and the true labels is calculated using the cross-entropy loss function. Subsequently, the gradient is calculated using the backpropagation algorithm, and the network parameters are iteratively updated based on the gradient using an optimizer (such as Adam or SGD) to gradually minimize the loss function and complete the model training.

[0085] Sub-step A40: After each training cycle, the performance metrics of the fusion network are evaluated using the validation set. When the validation set loss value converges stably or the accuracy reaches a preset threshold, the early stopping mechanism is triggered to terminate the training, and the model weights with the best performance on the validation set are saved.

[0086] In practice, after each training cycle, a comprehensive evaluation of the model performance can be performed using an independent validation set. The main monitoring metrics include the validation set loss value (such as cross-entropy loss) and classification accuracy. The trend of the validation set loss value is continuously tracked. If the metric does not show a significant decrease (i.e., enters a stable convergence state) or begins to rise within N consecutive training cycles, an early stopping mechanism is automatically triggered to terminate training, thus avoiding model overfitting at its source. During training, a "checkpoint" saving mechanism is enabled. Whenever the model achieves a new optimal performance on the validation set (e.g., the validation loss reaches a historical low, or the classification accuracy breaks through the historical best record), the current model parameter configuration and running status are immediately saved. After training terminates, the model version with the best validation set performance (i.e., the parameter configuration corresponding to the lowest validation loss or the highest classification accuracy) is selected from all saved checkpoints and used as the final deployed fusion network. By implementing full-process control of "cycle evaluation - early stop to prevent overfitting - checkpoint saving - optimal screening", we can effectively curb the model overfitting problem and ensure that the final deployed model has the optimal classification performance verified by the validation set, providing a solid guarantee for the reliability of subsequent disparity prediction failure detection.

[0087] Sub-step A50: Use the test set to perform a final performance evaluation on the trained fusion network, comprehensively test its generalization ability and robustness on unknown data. Once the fusion network meets the preset deployment criteria and passes the evaluation, a deployable fusion network is finally obtained.

[0088] In practical implementation, an independent test set can be used to conduct the final performance evaluation of the trained fusion network, comprehensively verifying the model's generalization ability and robustness on unseen real-world scene data. The evaluation focuses on the fusion network's classification performance for four types of disparity prediction states (no failure / normal, scene environment failure, camera calibration failure, and lens physical failure), ensuring the accuracy of the classification results. Multi-dimensional metrics are used for quantitative evaluation, including core evaluation indicators such as classification accuracy, recall, and F1 score. The confusion matrix is ​​combined to analyze the misclassification of each category, providing a comprehensive understanding of the model's classification performance. Specialized tests are conducted for complex scenarios in real-world applications to verify the model's output stability under conditions such as dynamic changes in lighting, image noise interference, and edge scenes (e.g., extreme textures, fast motion), ensuring the model's adaptability to real-world application environments. The model must meet the following two deployment criteria to pass the evaluation and be confirmed for deployment: the core metrics on the test set meet preset thresholds (e.g., overall classification accuracy ≥ 95%, recognition rate of each specific failure type ≥ 95%, specific thresholds can be adjusted according to business needs); and it maintains stable output without significant performance degradation in the aforementioned specialized tests for complex scenarios. Once the converged network passes the above evaluation and meets the preset deployment criteria, it is officially confirmed as a deployable converged network version. This version is capable of stably performing disparity prediction failure detection tasks in real-world application scenarios.

[0089] In this embodiment, a phased parameter control strategy is employed. First, the parameters of the fusion network are frozen, allowing the uncertainty estimation network and the disparity estimation network to converge stably to their optimal states during independent learning, thus avoiding interference from the random initialization of parameters in the fusion network on the training of sub-networks. Then, the fusion network parameters are unfrozen, and the outputs of the converged sub-networks are used as guiding signals to construct an end-to-end joint training framework. Multi-task collaborative optimization is achieved through backpropagation. This strategy effectively avoids feature interference and gradient conflicts by decoupling sub-network learning from fusion network optimization, gradually establishing cross-task management, resolving convergence instability issues in multi-task learning, and ultimately improving the overall model performance and generalization ability.

[0090] Based on the foregoing embodiments, a third embodiment of the failure detection method for disparity prediction of the present invention is proposed. In this embodiment, after step S30, the following step S60 may also be included:

[0091] In step S60, when the failure detection result is one or more of the following: scene environment failure, camera calibration parameter failure, and lens physical failure, the corresponding response mechanism will be automatically triggered to execute a targeted correction or compensation strategy.

[0092] Among them, the correction strategy is used to represent the preset operation that, for an eradicateable cause of failure, directly intervenes to eliminate the source of the fault and restores the system to a normal state.

[0093] As an example, if a sudden change in lighting, occlusion, or background distortion is detected, an environmental adjustment device (such as turning the fill light on / off, adjusting the light shield, or removing the obstruction by a robotic arm) is automatically triggered, or compensation is made through software algorithms (such as dynamic balance adjustment or image dehazing) to eliminate image distortion caused by environmental factors.

[0094] As another example, when focal length shift, lens rotation, or abnormal distortion parameters are detected, an automatic calibration program is initiated to reconfigure the camera by calling a pre-stored calibration parameter file, or to correct geometric distortion in real time through an online calibration algorithm to ensure the restoration of imaging geometric accuracy.

[0095] As another example, when lens stains or foreign objects are detected, a cleaning module (such as an automatic wiping robotic arm or an ultrasonic cleaning device) is triggered to perform physical cleaning, or an image restoration algorithm (such as deep learning-based defect filling) is used to temporarily eliminate contaminated areas in the image and restore image clarity.

[0096] Compensation strategies are preset actions taken to maintain core functionality and reduce the impact of persistent failures that cannot be eliminated immediately (such as hardware malfunctions before they are repaired), by dynamically adjusting system parameters or operating modes. It is important to note that correction strategies take precedence over compensation strategies.

[0097] As an example, if the environment is abnormal and cannot be quickly restored (such as continuous severe weather), switch to the backup imaging mode (such as infrared thermal imaging instead of visible light), or lower the detection accuracy threshold and retain only the key target recognition function to avoid system downtime.

[0098] As another example, when calibration parameters cannot be calibrated immediately, reverse distortion simulation compensation is performed on the image through software algorithms, and fusion correction is performed by combining redundant sensor data (such as laser ranging) to ensure that the target positioning accuracy meets the basic operational requirements.

[0099] As another example, if the cleaning mechanism fails or the stains are stubborn, a multi-frame fusion algorithm (such as continuous frame averaging filtering) is enabled to suppress the impact of noise, while triggering a maintenance warning to prompt manual intervention and restricting the execution of related vision-dependent tasks (such as high-precision measurement) from being suspended or downgraded.

[0100] In practical implementation, when the failure detection result is one or more combinations of scene environment failure, camera calibration parameter failure, and lens physical failure, the corresponding response mechanism will be automatically triggered according to the failure type. Following the principle of "correction over compensation," targeted correction or compensation measures will be matched and executed from the preset strategy library. The execution effect is verified in real time through a closed-loop feedback mechanism: if the correction strategy is successful (e.g., verifying that the image quality assessment meets the standards), the system returns to normal operation; if the correction fails or the failure persists, the system automatically switches to the compensation strategy and starts continuous monitoring until the fault is eradicated. This fully automated closed-loop response mechanism, through layered handling and dynamic feedback, ensures the system's adaptive adjustment capability under various failure scenarios, reduces the need for manual intervention, and effectively improves system robustness and operational reliability.

[0101] Furthermore, in one embodiment, step S60 may further include the following sub-step S601:

[0102] In sub-step S601, when the failure detection result is that the scene environment is in failure, an environment optimization strategy is triggered or a data re-acquisition process is started to obtain a valid image from a new perspective. The user is prompted by voice broadcast and visual interface text that the current scene environment is limited and that the lighting conditions need to be adjusted or the shooting position needs to be changed to ensure the validity of the data.

[0103] Among them, the environmental optimization strategy refers to a series of software and hardware operations that are automatically taken to improve imaging conditions when an image quality degradation caused by environmental factors (such as lighting or occlusion) is detected.

[0104] A valid image is an image that contains visual information of sufficient quality, clarity, and completeness to allow subsequent algorithms (such as feature extraction, matching, and fusion networks) to run stably and produce reliable results.

[0105] In this embodiment, when the failure detection result indicates a scene environment failure, the system will automatically trigger corresponding environmental optimization strategies (such as automatically adjusting the light source or angle) or initiate a data re-acquisition process to obtain valid images from a new perspective, based on the failure type and severity. Simultaneously, the system will issue warnings to the user through voice announcements and visual interfaces (such as real-time prompts and indicator light status changes), clearly indicating that the current scene environment is limited and guiding the user to adjust lighting or change the shooting position. This mechanism, through human-machine collaborative intervention, ensures the effectiveness of data acquisition and enhances the system's adaptability.

[0106] Furthermore, in one embodiment, step S60 may further include the following sub-steps B10 to B30:

[0107] Sub-step B10: When the failure detection result indicates that the camera calibration parameters have failed, the online automatic camera parameter recalibration program is started immediately.

[0108] In sub-step B20, if the error still exceeds the preset threshold or the calibration process fails after recalibration, the optical distortion correction program is started for software compensation.

[0109] In sub-step B30, if the software calibration is ineffective, the system will automatically switch to the backup camera and trigger a hardware status check task.

[0110] In this embodiment, when the failure detection result indicates a camera calibration parameter failure, a layered response process is initiated: first, an online automatic camera parameter recalibration program is triggered; if the recalibration residual error exceeds a threshold or the calibration process terminates abnormally, an optical distortion correction program (such as one based on the Brown-Conrady model or a radial-tangential distortion correction algorithm) is automatically activated for software compensation; if the image quality still fails to meet the standards after software correction, a seamless switch is made to a pre-calibrated and normal standby camera, and a hardware status verification task (including lens displacement detection, sensor integrity verification, etc.) is triggered simultaneously. The entire process uses a closed-loop verification mechanism to evaluate the effectiveness of each step in real time, and generates voice alarms and log records synchronously at key nodes to guide maintenance personnel to intervene precisely. This mechanism ensures high availability of the system in calibration parameter failure scenarios through software self-healing and hardware redundancy, while providing data support for root cause analysis of failures through full-process traceability.

[0111] Furthermore, in one embodiment, step S60 may further include sub-step S602:

[0112] In sub-step S602, when the failure detection result is a physical failure of the lens, the hardware maintenance process is triggered, and the user is informed by both voice and text prompts that the lens needs to be repaired or replaced.

[0113] In its implementation, when the failure detection result indicates a physical lens failure (such as severe obstruction, lens damage, or stubborn stains), the hardware maintenance response process is immediately triggered. First, the faulty camera is automatically isolated (and image acquisition is disabled or a backup device is switched on) to prevent erroneous data input. Then, precise maintenance instructions are issued to the user via voice prompts and a visual interface (such as pop-up notifications and indicator lights). Based on the cause of the failure, targeted operation suggestions such as "clean the lens" and "replace the optical module" are pushed, and a traceable maintenance log is generated simultaneously. This mechanism, through fault location and isolation, intelligent prompts, and log recording, ensures rapid guidance for manual intervention, enabling precise handling and rapid recovery of system failures, and reducing the risk of business interruption due to physical lens failure.

[0114] In this embodiment, when one or more of the following combinations are detected: scene environment failure, camera calibration parameter failure, or lens physical failure, the system will automatically trigger a response mechanism that matches the failure type based on the accurate failure diagnosis results output by the fusion network, and execute targeted correction or compensation strategies to quickly restore system functions or ensure basic operational reliability, thereby achieving adaptive adjustment and stable operation under fault scenarios.

[0115] Based on the same inventive concept, the fourth embodiment of the present invention also provides a disparity prediction failure detection system corresponding to the disparity prediction failure detection method of the foregoing embodiments. Since the principle of the system in the fourth embodiment of the present invention for solving the problem is similar to the disparity prediction failure detection method of the foregoing embodiments of the present invention, the implementation of the system can refer to the implementation of the method, and repeated details will not be elaborated further. Please refer to... Figure 3 The present invention provides a failure detection system for parallax prediction, the system comprising:

[0116] The calculation module 10 is used to acquire the left and right images of the target scene, input the left and right images into the matching cost calculation network, and output the corresponding matching cost results.

[0117] Prediction module 20 is used to input the matching cost result into the uncertainty estimation network and the disparity estimation network respectively, and output the corresponding uncertainty prediction result and disparity prediction result;

[0118] The detection module 30 is used to input the uncertainty prediction result and the disparity prediction result into the fusion network and output the corresponding failure detection result; the failure detection result includes no failure, scene environment failure, camera calibration parameter failure and lens physical failure.

[0119] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for failure detection of disparity prediction.

[0120] Figure 4 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general will label all buses as bus systems.

[0121] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0122] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0123] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The failure detection method for parallax prediction provided in this embodiment of the invention can be included in the application program 4022.

[0124] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the parallax prediction failure detection method provided in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0125] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0126] In summary, this invention achieves high accuracy and high reliability in parallax prediction by integrating the collaborative work of multiple networks, and performs failure detection and handling when necessary, effectively improving the safety and robustness of UAV applications.

[0127] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A failure detection method for parallax prediction, characterized in that, Applied to a convolutional neural network, wherein the convolutional neural network includes a matching cost calculation network, an uncertainty estimation network, a disparity estimation network, and a fusion network, the method includes: Obtain the left and right images of the target scene, and input the left and right images into the matching cost calculation network to output the corresponding matching cost results; The matching cost results are input into the uncertainty estimation network and the disparity estimation network respectively, and the corresponding uncertainty prediction results and disparity prediction results are output. The uncertainty prediction result and the disparity prediction result are input into the fusion network, and the corresponding failure detection result is output; the failure detection result includes no failure, scene environment failure, camera calibration parameter failure and lens physical failure.

2. The method according to claim 1, characterized in that, Before the step of obtaining the left and right images of the target scene, inputting the left and right images into the matching cost calculation network, and outputting the corresponding matching cost results, the method further includes: Freeze the parameters of the fusion network, train the uncertainty estimation network and the disparity estimation network, and enable the uncertainty estimation network and the disparity estimation network to learn independently and converge to a stable state; Unfreeze the parameters of the fusion network, perform end-to-end joint training of the fusion network with the converged uncertainty estimation network and disparity estimation network, and use the converged uncertainty estimation network and disparity estimation network to guide the learning of the fusion network.

3. The method according to claim 2, characterized in that, The step of jointly training the fusion network end-to-end with the converged uncertainty estimation network and disparity estimation network, and using the converged uncertainty estimation network and disparity estimation network to guide the learning of the fusion network, includes: A sample set containing normal sample images and various failure sample images is constructed, and each sample image in the sample set is labeled with a classification label to form a label set containing four categories: normal, scene environment failure, camera calibration parameter failure, and lens physical failure; wherein, the failure sample images include scene environment failure images generated by simulation or actually collected, camera calibration parameter failure images, and lens physical failure images. The sample set is divided into a training set, a validation set, and a test set according to a predetermined ratio. A neural network architecture is selected as the basis for the fusion network, and its network parameters are initialized. The training set and its corresponding labels are input into the fusion network. The cross-entropy loss function is used as the optimization objective. The network parameters are iteratively updated through the backpropagation algorithm and the optimizer to minimize the difference between the predicted labels and the true labels. After each training cycle, the performance metrics of the fusion network are evaluated using the validation set. When the validation set loss value converges stably or the accuracy reaches a preset threshold, the early stopping mechanism is triggered to terminate the training, and the model weights with the best performance on the validation set are saved. The trained fusion network is evaluated using the test set to comprehensively test its generalization ability and robustness on unknown data. Once the fusion network meets the preset deployment criteria and passes the evaluation, a deployable fusion network is finally obtained.

4. The method according to claim 1, characterized in that, After the step of inputting the uncertainty prediction result and the disparity prediction result into the fusion network and outputting the corresponding failure detection result, the method further includes: When the failure detection result is one or more of the following: scene environment failure, camera calibration parameter failure, lens physical failure, the corresponding response mechanism will be automatically triggered to execute targeted correction or compensation strategies.

5. The method according to claim 4, characterized in that, When the failure detection result is one or more combinations of scene environment failure, camera calibration parameter failure, and lens physical failure, a corresponding response mechanism will be automatically triggered to execute targeted correction measures or compensation strategies, including: When the failure detection result indicates that the scene environment is in failure, an environment optimization strategy is triggered or a data re-acquisition process is started to obtain valid images from a new perspective. The user is then prompted through voice broadcast and visual interface text that the current scene environment is limited and that lighting conditions need to be adjusted or the shooting position changed to ensure data validity.

6. The method according to claim 4, characterized in that, When the failure detection result is one or more combinations of scene environment failure, camera calibration parameter failure, and lens physical failure, a corresponding response mechanism will be automatically triggered to execute targeted correction measures or compensation strategies, including: When the failure detection result indicates that the camera calibration parameters have failed, the online automatic camera parameter recalibration program is immediately initiated. If the error still exceeds the preset threshold after recalibration or the calibration process fails, the optical distortion correction program will be started for software compensation. If software calibration fails, the system will automatically switch to the backup camera and trigger a hardware status check task.

7. The method according to claim 4, characterized in that, When the failure detection result is one or more combinations of scene environment failure, camera calibration parameter failure, and lens physical failure, a corresponding response mechanism will be automatically triggered to execute targeted correction measures or compensation strategies, including: When the failure detection result indicates a physical failure of the lens, the hardware maintenance process is triggered, and the user is informed of the need for hardware repair or replacement of the lens through both voice and text prompts.

8. A failure detection system for parallax prediction, characterized in that, The system is applied to a convolutional neural network, which includes a matching cost calculation network, an uncertainty estimation network, a disparity estimation network, and a fusion network. The calculation module is used to acquire the left and right images of the target scene, input the left and right images into the matching cost calculation network, and output the corresponding matching cost results; The prediction module is used to input the matching cost result into the uncertainty estimation network and the disparity estimation network respectively, and output the corresponding uncertainty prediction result and disparity prediction result; The detection module is used to input the uncertainty prediction result and the disparity prediction result into the fusion network and output the corresponding failure detection result; the failure detection result includes no failure, scene environment failure, camera calibration parameter failure and lens physical failure.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed, performs the steps of the method according to any one of claims 1 to 7.

Images