Crawler robot intelligent reconnaissance system based on panoramic vision fusion multispectral analysis

Abstract

The application relates to the technical field of robot reconnaissance, and discloses a tracked robot intelligent reconnaissance system based on panoramic vision fusion multispectral analysis, which comprises the following steps: based on the environmental spectrum characteristics, dynamically adjusting sensor parameters to adapt to different light conditions and collecting high-quality panoramic vision and multispectral data; performing panoramic vision fusion processing on the collected data to compensate for light changes in complex environments and enhance image quality; performing multispectral data analysis according to the fusion data, classifying target materials based on the analysis results to improve the reconnaissance accuracy; and combining the correlation optimization algorithm of the visual data and the spectral information to filter the interference in multi-target reconnaissance and feed back and correct the path planning of the robot in real time. According to the analysis results of the multispectral data, the target materials are classified to solve the problem of insufficient target recognition accuracy in the reconnaissance process, and machine learning and a prediction algorithm are combined, so that the success rate and efficiency of the reconnaissance task are ensured.

Description

Technical Field

[0001] This invention relates to the field of robot reconnaissance technology, specifically to an intelligent reconnaissance system for tracked robots based on panoramic visual fusion and multispectral analysis. Background Technology

[0002] The intelligent reconnaissance system for tracked robots based on panoramic vision fusion and multispectral analysis is a technical means to achieve intelligent reconnaissance of complex environments by utilizing panoramic vision and multispectral analysis technologies, combined with the mobility of tracked robots.

[0003] However, this method still faces some challenges. First, how to accurately classify target materials through multispectral data analysis to improve the accuracy of target identification during reconnaissance. Second, the ability to compensate for changes in light in complex environments is insufficient, which affects the stability of image quality. Third, under certain conditions, the reconnaissance accuracy decreases because the sensor parameters cannot be dynamically adjusted to match the spectral characteristics of the environment.

[0004] Furthermore, when facing simultaneous reconnaissance of multiple targets, the lack of efficient algorithm optimization makes it difficult to filter interference due to insufficient correlation between visual data and spectral information. Finally, the coordination between real-time feedback of reconnaissance data and robot path planning is not tight enough, which may lead to the risk of target loss. Solving these problems will be the key to improving this intelligent reconnaissance method. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A tracked robot intelligent reconnaissance system based on panoramic vision fusion and multispectral analysis includes:

[0007] The sensor parameters are dynamically adjusted based on environmental spectral characteristics to adapt to different lighting conditions and acquire high-quality panoramic vision and multispectral data; the acquired data is fused using panoramic vision to compensate for lighting changes in complex environments and enhance image quality; multispectral data analysis is performed based on the fused data, and the target material is classified based on the analysis results to improve reconnaissance accuracy; an optimization algorithm combining the correlation between visual data and spectral information is used to filter interference in multi-target reconnaissance and provide real-time feedback to correct the robot's path planning;

[0008] The dynamic adjustment of sensor parameters based on environmental spectral characteristics further includes: determining the dominant frequency component f_main of the current environmental spectrum; calculating the spectral distribution variance σ2=1 / N∑(f_i f_mean)2, where N is the number of frequency points, f_i represents the intensity of the i-th frequency point, and f_mean is the average frequency intensity; when σ2>threshold1, adjusting the sensor sensitivity to k×σ2, where k is the sensitivity coefficient; and calibrating the sensor to ensure data consistency and accuracy.

[0009] Preferably, the panoramic visual fusion processing of the collected data includes:

[0010] Based on the image pixel distribution, calculate the scene illumination deviation ΔL = |I_max I_min| / I_ave, where I_max and I_min are the maximum and minimum brightness values, respectively, and I_ave is the average brightness value;

[0011] If ΔL exceeds the threshold2, then the global brightness compensation formula C_compensate=C_original+λ×ΔL is used for image enhancement, where λ is the compensation factor;

[0012] The panoramic stitching algorithm is used to seamlessly stitch images into a 360° view;

[0013] Edge-preserving filtering is used to eliminate artifacts and enhance detail quality in complex environments.

[0014] Preferably, the multispectral data analysis based on the fused data includes:

[0015] The energy values ​​E_i of different bands are processed using a normalization algorithm to obtain E_norm(i)=(E_i E_min) / (E_max E_min);

[0016] Define the spectral matching degree of the target material as D = Σ|S_ref(i)E_norm(i)|, where S_ref(i) is the spectral data of the reference material;

[0017] Select the category of S_ref(i) that corresponds to the smallest value of D as the material type;

[0018] A probability classification module was added to the material type to accommodate uncertainty.

[0019] Preferably, the correlation optimization algorithm combining visual data and spectral information includes:

[0020] Construct a joint feature space J = [V1, V2, ..., VS], where Vi represents the position of each target in the feature vector space;

[0021] The weights w_j = Σcov(F_vis(j),F_spec(j)) / Σvar(F_vis(j)) are obtained through covariance analysis, where F_vis and F_spec represent the variances of the visual and spectral feature dimensions, respectively.

[0022] Set the interference removal threshold to threshold 4 and remove interference w_j according to the following conditions. <threshold4;

[0023] The classifier is retrained based on the high-confidence features remaining to accurately identify targets.

[0024] Preferably, the filtering of interference in multi-target reconnaissance further includes:

[0025] The real-time collected interference intensity signal is mapped to the interference level index M, M = log(∑|I_noise I_baseline|^2) / log(N_noise), where N_noise represents the total number of interference frames;

[0026] When M>threshold5, the redundant data verification logic is activated to remove invalid frames.

[0027] Extract the relative motion features H_diff(j) between multiple targets within the effective frame to distinguish between stationary and moving entities;

[0028] Based on the classification results, priority should be given to high-value moving targets to optimize reconnaissance efficiency.

[0029] Preferably, the optimized interference condition is:

[0030] The evaluation formula for updating the interference level by introducing a self-learning adjustment module is A=αM+β∑H_rel, where α and β are weight parameters, and H_rel is the sum of relative relation features;

[0031] If A>threshold6, activate the backup sensing system to ensure reliability;

[0032] Integrate redundant sensor data into the existing reconnaissance model to form a closed-loop correction structure;

[0033] Increase the target localization resolution R_loc ≥ resolution_goal during the keyframe detection stage.

[0034] Preferably, the real-time feedback correction path planning includes:

[0035] Define the proportional deviation of the current position from the baseline trajectory as P_deviation = |pos_current pos_benchmark| / path_len_total;

[0036] If P_deviation > threshold7 or obstacle distance D_obst ≤ threshold8, the replanning procedure is initiated.

[0037] Generate multiple candidate routes r_candidate and filter them according to the estimated cost function COST(r_candidate,t_now);

[0038] The system optimizes the path with the lowest cost while updating historical data to support improved predictive capabilities.

[0039] Preferably, the enhanced path intelligent correction step is as follows:

[0040] Establish a prediction window T_future = time_remain * γ (the time window multiplier γ is set according to the task priority) to predict the set of potential future problems P_future;

[0041] If any p∈P_future satisfies the conflict judgment formula C(p_i,t_estimtated,v_target)>critical_conflict_value, then avoid it in advance;

[0042] A two-tiered obstacle avoidance rule is introduced, first adjusting the angle locally and then adjusting the overall direction to avoid excessive detours;

[0043] After the adjustment is completed, perform the security check again until the standard condition exit_flag=true is met.

[0044] Preferably, improving multi-view synchronization efficiency includes:

[0045] Set the camera synchronization signal trigger delay τ_synchronization and control the actual delay range to be stable by the formula τ_sync_actual=ceil[Δτ / step_unit)] (step_unit is the unit step distance);

[0046] When the frame rate change exceeds the allowable range δfps = |fps_current fps_designed| / fps_designed and is greater than threshold_fps, an early warning mechanism is triggered and the light source intensity is adjusted.

[0047] Ensure that the image quality uniformity error err_quality of all channels does not exceed the set tolerance interval_tolerance;

[0048] Achieve rapid response while reducing energy consumption to meet the needs of long-term operating environments.

[0049] Preferably, the steps for improving image quality include:

[0050] Record the global exposure parameter exp_global and the local brightness ratio factor_bright_local = avg_intensity_high / avg_intensity_low;

[0051] For high-contrast regions, the histogram correction formula hist_new(z) = exp_global × factor_correction × factor_bright_local is applied.

[0052] Integrate deep learning models to fine-tune exposure and sharpening effects, and output optimized images: Q_optimize=net(Q_initial,fine_tuning_params);

[0053] Regularly perform loop-based verification of the repair strategy's effectiveness to ensure consistent and stable performance over the long term.

[0054] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0055] 1. By dynamically adjusting sensor parameters based on environmental spectral characteristics to adapt to different lighting conditions and acquire high-quality panoramic vision and multispectral data; performing panoramic vision fusion processing on the acquired data to compensate for lighting changes in complex environments and enhance image quality; performing multispectral data analysis based on the fused data, and classifying target materials based on the analysis results to improve reconnaissance accuracy; combining visual data and spectral information correlation optimization algorithms to filter interference in multi-target reconnaissance and provide real-time feedback to correct the robot's path planning.

[0056] 2. By combining machine learning and predictive algorithms, the robot's path is dynamically corrected by analyzing current target reconnaissance data and historical path records. This allows the tracked robot to track and avoid obstacles or misjudgments even when the target's location changes, reducing the risk of target loss and ensuring the success rate and efficiency of reconnaissance missions. In summary, this reconnaissance method comprehensively improves the adaptability and reconnaissance capabilities of tracked robots from multiple dimensions. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the reconnaissance system of the present invention;

[0058] Figure 2 This is a schematic diagram of the target identification process of the present invention;

[0059] Figure 3 This is a schematic diagram of the path data flow of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Please see Figure 1 - Figure 3 The intelligent reconnaissance system for tracked robots based on panoramic visual fusion and multispectral analysis of this invention first requires dynamically adjusting sensor parameters to collect high-quality data. Next, it performs fusion processing on images under complex lighting conditions and analyzes the target material based on multispectral data. Simultaneously, it incorporates a correlation optimization algorithm to filter interference, and finally, it corrects path planning based on real-time feedback to improve reliability.

[0062] First, dynamically adjusting sensor parameters based on environmental spectral characteristics is one of the key steps to ensure data accuracy and reliability. In the specific operation, a multispectral sensor installed at the front of the robot collects spectral characteristic information of the environment in a specific area and analyzes the wavelength and intensity distribution of the light source. This step can sense changes in ambient brightness and the presence of special light sources (such as ultraviolet or infrared radiation). Then, the system automatically corrects the working parameters of the camera, lidar, or other sensing modules, such as exposure time, gain value, and filter settings, based on spectral analysis. This method enables the tracked robot to cope with diverse and complex external environmental conditions while maintaining a high level of data collection quality.

[0063] Secondly, the collected data undergoes panoramic visual fusion processing to compensate for image degradation caused by uneven lighting. Multiple sets of raw data from different perspectives or different wavelengths (such as visible light and infrared) are matched under a unified coordinate system. Algorithms are used to reduce distortion errors or noise accumulation caused by overlapping areas. At the same time, advanced contrast stretching technology and local mean balancing technology are used to further enhance the brightness and darkness of the image, achieving an ideal detail presentation effect, thereby effectively improving the visibility and distinguishability in the overall reconnaissance mission.

[0064] To improve reconnaissance accuracy by classifying target materials, meticulous multispectral data analysis is required after acquiring the fused images. This mainly involves using machine learning-trained models to identify the differences in reflectance characteristics of each potential object across various spectral bands. Then, statistical regularities are used to infer the possible material types, such as metal products or natural rocks. In one embodiment, if an unidentified object is found in the reconnaissance area, the corresponding spectral scan results can be retrieved and compared with a pre-stored database standard template to assess and confirm the attribute category, significantly reducing the false alarm rate and improving the accuracy of judgment.

[0065] Combining the correlation between visual data and spectral information is equally essential for optimizing algorithm design, especially in challenging situations where multiple targets coexist. To this end, a specially customized weighting factor formula is introduced, comprehensively considering multiple indicators such as spatial proximity, morphological similarity level, and reflectivity matching probability to construct a joint evaluation criterion. This criterion is used to filter out false information, retaining only the most relevant parts as the basis for further processing. Specifically, when faced with a chaotic battlefield situation with various shell fragments scattered haphazardly, relying solely on ordinary shape identification inevitably leads to omissions. However, by additionally considering approximate absorption and release energy characteristics, truly interesting key elements can be successfully identified, forming a more precise focus and guiding the next action decision, thus moving towards a mature, stable, and accurate development path.

[0066] The final step, and one of the indispensable core elements of the entire reconnaissance process, is to adjust the current trajectory and route in a timely manner based on continuously generated real-time reconnaissance data to avoid falling into a passive predicament again and losing important clues outside the locked area, thus mitigating the risk of such a situation. This flexible response mechanism requires the equipment of an efficient communication unit to ensure unimpeded two-way communication between internal and external parties, coupled with a pre-built reserve of multiple emergency plans to support the switching of optimal action strategies according to actual conditions, ensuring stable and smooth operation of the switching mode. For example, when the reconnaissance is suddenly blocked by dense obstacles, reducing visibility and seriously threatening the success rate of the expected mission, the built-in obstacle avoidance planning subsystem is quickly activated to regenerate alternative detour routes, ensuring that the mission framework is still met to approach the destination, complete all predetermined inspection content, meet the requirements, and successfully conclude the entire process. The entire process is closed-loop managed in a standardized, scientific, advanced, reliable, efficient, and highly effective manner, yielding significant results.

[0067] The sensor parameter adjustment system based on environmental spectral characteristics of the present invention specifically includes four steps, and the role of each step is explained from the aspects of formula calculation and practical application.

[0068] The first step is to determine the dominant frequency component of the current environmental spectrum, analyze the frequency points where the main light source intensities exist in the current environment, and focus on processing these frequency points. In multi-source scenarios, such as urban areas or complex terrain with sunlight reflection, there may be interference from multiple light wavelengths. Therefore, identifying the dominant frequency component helps to optimize the subsequent adjustment algorithm. The dominant frequency component is usually obtained by Fourier transform or other decomposition methods, and ideally, it can represent the maximum power distribution.

[0069] The second step involves calculating the variance of the spectral distribution using the formula: σ² = 1 / N∑(f_i - f_mean)², where N represents the number of different frequency points sampled, the range of which can be large depending on the resolution and detection distance, while the optimal value depends on the maximum bandwidth supported by the specific hardware; f_i refers to the absolute energy value or signal strength value at the i-th frequency, and these measurement data are input into the system after preprocessing such as filtering and noise removal; f_mean is the average of all f_i values ​​to reflect the overall homogenization level. This formula is designed to understand the changes in ambient lighting conditions through variance measurement. If the variance is too large, it indicates that there are obvious inconsistencies or anomalies that need to be identified. Variance is chosen as a key statistical indicator because it is sensitive to fluctuations and can well express heterogeneous information.

[0070] Next, when the calculated variance σ² exceeds a given threshold 1 (this value needs to be set based on experimental results), the third step is performed: resetting the sensor sensitivity. The new sensitivity value is defined as k×σ², where k is the sensitivity adjustment scaling factor. The default value is generally set within a certain range to ensure reasonable gain and avoid saturation. This setting allows the system to automatically amplify weak signals while suppressing excessively strong changes under high-variety environments, ensuring stable operation and good performance of the entire system.

[0071] The fourth and final step focuses on the sensor calibration process, which involves periodic adjustments to maintain consistent response characteristics and accuracy across different devices over long-term use. In one embodiment, if a tracked robot is traversing a complex mix of urban streets and forests filled with vegetation reflecting light, it first captures a full-spectrum image containing diverse characteristics such as sky background and ground material reflections, and extracts dominant frequency information, such as the strong absorption characteristics corresponding to the green band. Simultaneously, the system detects a high σ² indicator indicating significant spatial diversity, so it increases the parameter weights for the camera module and other sensing units regarding the discrimination of objects of approximately a specific color, and then verifies whether the output conforms to a pre-stored standard model curve before making fine adjustments. In this way, reliable monitoring support can be continuously provided regardless of external challenges.

[0072] The present invention also includes panoramic visual fusion processing of the acquired data, which includes four main steps: calculating scene illumination deviation, global brightness compensation, panoramic stitching, and edge-preserving filtering.

[0073] The first step is to calculate the scene illumination deviation based on the image pixel distribution using the formula ΔL = |I_max - I_min| / I_ave. Here, I_max and I_min represent the maximum and minimum brightness values ​​in the image, respectively, while I_ave represents the average brightness. ΔL reflects the degree of brightness variation in the image. Typically, I_max ranges from [0, 255] (taking an 8-bit image as an example), and the value of I_ave depends on the average brightness of the specific image. For example, in bright light environments or under shadows, I_max is significantly higher than I_min, which leads to an increase in the ΔL value, thus reflecting an imbalance in the scene illumination distribution.

[0074] If ΔL exceeds the preset threshold 2, the second step will be initiated: the image will be enhanced using the global brightness compensation formula C_compensate = C_original + λ × ΔL, where C_original is the color value of the original image, and λ is the compensation factor used to adjust the degree of image enhancement, typically selected within the range of [0.5, 1.5]. When ΔL is too high, this formula can balance the image brightness, enhancing shadows while preventing overexposure. The optimal value of λ is usually set to 1, as it achieves the desired enhancement effect under different lighting conditions.

[0075] The third step employs a panoramic stitching algorithm to seamlessly integrate the images into a 360° view. Specifically, by matching and optimizing image feature points, the boundary effects of overlapping areas are eliminated, forming a complete three-dimensional panoramic view. This process is suitable for data fusion in complex environments and ensures the consistency of the generated views.

[0076] The fourth step uses edge-preserving filtering to eliminate artifacts and enhance detail quality in complex environments. This method effectively suppresses noise interference and preserves the texture and structural information of the original image.

[0077] In one embodiment, suppose a tracked robot is performing an intelligent reconnaissance mission outdoors in a complex environment with multiple light sources, including areas of direct sunlight, shaded areas, and grassy backgrounds. Initially, the robot acquires a series of images with significant contrast between light and dark areas using a multispectral camera, and the calculated ΔL value significantly exceeds the threshold2. Therefore, the system automatically performs a global brightness compensation step to adjust the visibility in shaded areas and optimize the display effect in bright areas. Subsequent panoramic stitching connects the views acquired from different directions, forming a continuous and complete display of the surrounding environment. Finally, an edge-preserving filter further enhances the realism and usability of the images, making the reconnaissance results more reliable and clear, meeting the needs of practical application scenarios.

[0078] This invention also includes multispectral data analysis based on fused data. The analysis process mainly includes the following steps: normalizing the energy values ​​of different bands, defining the spectral matching degree of the target material, selecting the reference material with the best matching degree as the material type, and adding a probability classification module to adapt to uncertainty.

[0079] The first step is to process the energy values ​​of different bands using a normalization algorithm. The formula is E_norm(i) = (E_i - E_min) / (E_max - E_min), where E_i represents the original energy value of the i-th band, and E_min and E_max represent the minimum and maximum energy values ​​among all bands, respectively, typically within the set of non-negative real numbers. This formula compresses the energy values ​​to the range of 0 to 1 using a linear stretching method. The purpose is to remove the influence of dimensions, make the data comparable, and improve computational efficiency. For example, when a tracked robot is conducting reconnaissance, if the energy values ​​collected in the infrared and ultraviolet bands are 52 units and 300 units respectively, normalization will unify these two values ​​to the same scale for comparison.

[0080] The second step involves defining the spectral matching degree D of the target material, which is accomplished using the formula D = Σ|S_ref(i) - E_norm(i)|, where S_ref(i) is the spectral data of the reference material, and i is the index of the corresponding band. The logic of this formula is to sum the differences between the reference value and the actual measured value for each band. The greater the difference, the higher the matching degree, and the less ideal the D value. Through this operation, the most similar reference sample can be found. For example, if the reference spectral values ​​for each band of a material sample are [0.7, 0.3], and the measured normalized results E_norm are [0.68, 0.28], then the small error between them indicates that the material may be close to a known sample type.

[0081] Building upon the above, the third step involves selecting the category with the smallest matching degree D as the identification target. The core of this step lies in comparing multiple possible S_ref objects and selecting the best approximation. If, among a set of candidate reference material types, three are labeled A, B, and C respectively, and their respective D values ​​are calculated to be 0.2, 0.15, and 0.13, then category C is ultimately confirmed as the target material. This is because, under assumption, C has the lowest overall total bias.

[0082] The fourth step introduces a probabilistic classification mechanism to handle potential data errors or environmental interference. Even after identifying a primary material, it's necessary to quantify the likelihood of other potential candidate materials. In one embodiment, when a tracked robot is conducting intelligent reconnaissance in conditions of uneven lighting or significant occlusion, even if the initial determination indicates that the metal material is primarily composed of aluminum alloys, a probabilistic label must be added to indicate the probability of it containing an extremely low proportion of ferromagnetic elements or other unknown dopants. Specifically, this additional step ensures greater robustness of the analysis, thereby meeting the complex requirements of real-world applications.

[0083] The present invention also includes a correlation optimization algorithm that combines visual data and spectral information, comprising the following steps: first, constructing a joint feature space; second, calculating weight parameters through covariance analysis; third, setting an interference removal threshold to filter invalid data; and finally, retraining the classifier based on the retained high-confidence features to improve target recognition accuracy.

[0084] The first step is to construct a joint feature space J = [V1, V2, ..., VS], where Vi represents the position of each target in this feature vector space. This operation aims to integrate panoramic vision and multispectral dimensional information into a unified mathematical structure, facilitating subsequent comprehensive analysis. Each dimension Si is defined as the mapped coordinate value of the target in its respective observation dimension, with the range determined by the specific sensor characteristics. For example, in some applications, Vi can be a normalized numerical vector from 0 to 1, ensuring consistency across all inputs for unified processing. This constructed feature matrix allows for efficient utilization of information from multiple data sources and provides a clear and quantifiable way to represent the target.

[0085] The second step is to use covariance analysis to determine the importance of each feature component to the final classification result. The formula is w_j = Σcov(F_vis(j), F_spec(j)) / Σvar(F_vis(j)), where F_vis and F_spec refer to the visual and spectral feature vector sequences extracted from the same scene, respectively; cov measures the common trend of change between the two different sources, i.e., the degree of linear correlation; and var represents the internal fluctuation level or discrete intensity of a single source. The weight parameter w_j represents the evaluation index of the contribution of visual fusion data to the overall system decision support in dimension j. Typically, its optimal range should be between -1 and +1; the closer to a positive number, the more significant the consistency and effectiveness of the information provided by the corresponding channel; if the absolute value is low or negative, it suggests that the two may have low synergistic potential or a risk of misleading.

[0086] Based on the above, a criterion for removing interference items is set, and a fixed constant threshold4 is introduced to compare with the actual weight value. When the inequality condition w_j < threshold4 is met, it indicates that the element may be a non-critical or false signal and needs to be excluded. Here, it is recommended that the threshold4 value is generally selected as a boundary close to zero small positive, such as 0.2, to ensure sufficient sensitivity without being overly conservative. In one embodiment, it is assumed that a large set of noise-mixed samples {w} is obtained in the initial stage. After calculation and verification, it is found that the parts below the threshold generally come from errors caused by factors such as equipment hardware aging and external strong light pollution. By strictly restricting these interferences, a higher-purity data group can be obtained for the next step.

[0087] The final step is to use the high-quality attributes after screening to perform the iterative training process of the classification model again to enhance the accuracy output level of the detection efficiency. Specifically, in a panoramic reconnaissance case, assume that the task of a certain tracked robot is to distinguish normal vegetation from camouflaged objects in the ground environment. The key indicators (such as the matching frequency of color-uniform regions and the performance of short-wave infrared absorption incident differences) retained after the screening in the first three stages will become the key input materials for the new algorithm to recalibrate the prediction logic framework of the SVM support vector machine, and finally, in the actual combat test in complex terrain, it shows a significant improvement in the occurrence probability of misreporting that was prone to occur before, and significantly enhances the system adaptability and practicality.

[0088] The interference filtering in multi-target reconnaissance of the present invention is further defined as: this process includes multiple steps to achieve the goal of optimizing the reconnaissance efficiency. In the first step, the interference intensity signal collected in real time is mapped to the interference level index \(M), and the specific formula is \(M = \log(\sum|I_{text{noise}} - I_{text{baseline}}|^2) / \log(N_{text{noise}}))). In this formula, \(I_{text{noise}}) represents the interference data value monitored in real time, \(I_{text{baseline}}) is the reference value of the baseline interference, which represents the expected environmental interference standard of the system, and generally ranges within the normalized interval of -1 to 1, and its optimal setting depends on the adjustment of experimental data; \(N_{text{noise}}) represents the total number of interference frames, that is, the number of frames of the images detected as being interfered within a unit time, and the range is a positive integer. The design logic of the formula is to measure the energy size of the interference by summing the squares of the signal differences, and to smooth the output of the interference level index \(M) by taking the logarithmic ratio relationship with the total number of interference frames. The main function of this step is to extract a comparable interference magnitude evaluation value from the original data.

[0089] The second step, after obtaining the interference level index (M), is to check whether the threshold condition (M>threshold5) is met. If so, the redundant data verification logic is activated to remove invalid frames. Invalid frames refer to image information fragments that deviate significantly from the baseline. For example, when the robot is in a dusty environment, strong dust reflections may cause some images to be unrecognizable and need to be marked for deletion or ignored in the calculation.

[0090] The third step involves analyzing the relative motion features (H_{text{diff}}(j)) between multiple targets based on the extracted valid intra-frame data. Each (j) represents a set of target position change parameters at different time points. This step is used to distinguish between stationary and moving entities. If a target has a large displacement relative to the background pixels, it may be identified as a moving target rather than a noise source, thus allowing for further in-depth analysis.

[0091] The final step prioritizes high-value moving targets based on the previous classification results, thereby further improving reconnaissance efficiency. In one embodiment, if an enemy tank is detected moving slowly (judged as moving) based on panoramic visual fusion multispectral technology, while roadside objects or swaying grass caused by wind are not considered potential threats, resources will be concentrated on capturing the dynamic trajectories of critical military equipment. Specifically, tanks have significant strategic importance scores, therefore more processor power is allocated to trajectory prediction and warning generation.

[0092] The optimized interference conditions of this invention include four steps: updating the evaluation formula for the interference level, activating the backup sensing system, integrating redundant sensor data to form a closed-loop correction structure, and improving the target localization resolution in the keyframe detection stage.

[0093] Step one involves introducing a self-learning adjustment module to update the interference level assessment formula as \(A=\alphaM+\beta\sum H_{text{rel}}), where the parameters \(\alpha) and \(\beta) represent weights, ranging from [0,1], \(M) represents the baseline interference level measured in the current multispectral analysis, and \(H_{text{rel}}) is the sum of relative relationship characteristics, typically reflecting the correlation strength between multiple sensor data. The optimal values ​​of \(\alpha) and \(\beta) need to be experimentally verified; for example, when environmental complexity increases, \(\alpha=0.6, \beta=0.4) can be set. This formula means that the interference level \(A) is calculated by combining the baseline interference quantity and relative relationship characteristics to achieve a more accurate assessment of the actual interference level. The main purpose of setting the formula is to enhance the comprehensive understanding of interference conditions by combining basic information and the relationships between data.

[0094] The second step is to activate the backup sensing system if the calculated interference level \(A>\text{threshold}_6). Here, \(\text{threshold}_6) is a pre-set threshold value. Its purpose is to promptly activate redundant sensing hardware when interference conditions exceed a specific limit, ensuring that the robot can maintain stable detection performance in complex environments. For example, in one embodiment, the threshold \(\text{threshold}_6) can be set to an empirical value of 35 (the unit varies with the definition). When the result of the evaluation formula exceeds this value, the robot switches to a higher-priority sensor group to avoid data errors leading to functional failure.

[0095] The third step involves combining redundant sensor data with existing reconnaissance models to construct a closed-loop correction mechanism. This step aims to effectively utilize the different redundant signals collected by sensors, thereby adjusting the model output in real time and improving the overall accuracy and reliability of intelligent reconnaissance. Specifically, in scenarios involving panoramic vision and multispectral analysis, by integrating thermal imaging and ordinary visible light images to generate a more robust dataset, the model's ability to recognize environmental features can be dynamically optimized.

[0096] Finally, the target localization resolution in the keyframe detection stage is increased to \(R_{text{loc}}\geq\text{resolution}_{text{goal}}). This aims to enhance the robot's accurate localization of specific objects when handling important reconnaissance tasks, reducing the adverse impact of errors on task completion. In one example, the resolution \(R_{text{loc}}) is required to reach at least 0.2° (angular resolution reaching a specified accuracy target). This design ensures reliable tracking even against distant or fast-moving targets, meeting the needs of high-precision map construction and tactical deployment.

[0097] The real-time feedback correction path planning of this invention first clarifies the definition and implementation process of each step: defining the deviation ratio, setting trigger conditions, generating candidate paths, selecting the preferred path, and updating the prediction capability. These steps are then described in detail below.

[0098] First, the formula for the proportional deviation of the current position from the baseline trajectory is defined as \(P_{text{deviation}}=|pos_{text{current}}-pos_{text{benchmark}}| / path_{text{len\_total}}). The parameters in this formula are explained as follows: \(pos_{text{current}}) represents the robot's current actual position, \(pos_{text{benchmark}}) is the expected position on the preset baseline path, and \(path_{text{len\_total}}) is the total path length. The formula quantifies the degree of trajectory deviation of the robot, and its value typically ranges from [0,1]. The optimal value is dynamically adjusted according to task requirements. This metric is used to assess whether the deviation from the baseline trajectory is excessive.

[0099] Second, if the deviation ratio (P_{text{deviation}}>threshold7), or the distance to an obstacle (D_{text{obst}}\leq threshold8), a replanning procedure is initiated. Here, (threshold7) is the manually set maximum permissible deviation threshold, typically around 0.2; (threshold8) is the minimum safe obstacle distance for risk avoidance, generally set to twice the robot arm's outstretched radius or the outer width of the tracks, for example, around 1 meter. Setting such trigger conditions ensures that path deviations or potential collisions are corrected promptly.

[0100] Third, multiple candidate routes are generated. This process requires combining environmental information (panoramic vision, multispectral analysis data) and kinematic constraints to create a set of executable paths. Each candidate route has different cost estimates for time, space, and energy consumption.

[0101] Fourth, the path is selected based on the estimated cost function \(COST(r_{text{candidate}},t_{text{now}})). The \(COST) function comprehensively considers factors such as distance, estimated arrival time, difficulty of avoiding obstacles, and energy consumption, and selects the candidate path with the lowest cost to perform the operation, while retaining relevant historical data for subsequent iterations to improve model performance.

[0102] In one embodiment, suppose a tracked robot is traveling along a predetermined route in a mountain reconnaissance mission. Due to the irregular terrain, the panoramic camera detects that the robot has deviated to the right by approximately 0.25 times the total path length (P_{text{deviation}})), exceeding a set threshold ((threshold7=0.2)). Additionally, the front sensor indicates that the robot is less than 0.9 meters from an obstacle ((threshold8=1)). Once either of these two conditions is met, the robot automatically enters a path replanning process. At this point, multispectral image processing is used to extract potential surrounding paths and estimate their costs. Ultimately, a left-side detour is selected as the new instruction output to the control unit to continue the mission. This specifically achieves the goal of intelligent feedback correction.

[0103] The further enhanced path intelligent correction steps of this invention are as follows: First, a prediction window is established, and a set of conflict points is selected for avoidance using a judgment formula. Second, adjustments are made by applying a two-level obstacle avoidance rule. Finally, after the adjustment is completed, a second check is performed to ensure that the standard condition exit_flag = True is met.

[0104] The steps are as follows: The first step is to define the time prediction window T_future as the product of the remaining time and a multiplier γ (T_future = time_remain * γ), used to assess potential problem points P_future that the robot may encounter before the task ends. The parameter time_remain represents the remaining time required to complete the current reconnaissance task; the parameter γ represents a time scaling factor based on task urgency, typically set between 0.8 and 2, with an optimal value of 1.5. Higher γ values ​​are set for high-priority tasks, extending the future prediction time to cover more scene complexity. This formula generates a reasonable future analysis period to make the robot's planning more predictive. In one embodiment, if the task requires rapid acquisition of panoramic intelligence data, γ is set to a larger value, such as 1.8, which extends the robot's ability to predict potential situations in advance.

[0105] The second step involves applying the conflict assessment formula C(p_i,t_estimated,v_target) > critical_conflict_value to all points within the problem point P_future to identify points that require special avoidance. In this formula, p_i represents each individual problem location point in the set, t_estimated is the estimated time to reach the target location p_i, v_target is the target's forward speed, and critical_conflict_value is a predefined threshold. This step aims to determine if there are obstacles or task constraints that make the route unsafe. For example, when a tracked robot is traversing an area filled with weeds and boulders, if certain coordinates p are calculated to have a risk score exceeding the critical conflict value, these areas are considered to pose an unacceptable hazard and should be avoided.

[0106] Then, a two-tiered obstacle avoidance strategy is adopted. This involves fine-tuning the robot's head azimuth angle locally to avoid obstacles, followed by readjusting the overall course to prevent excessive detours and unnecessary distance loss. For example, when faced with a small hill blocking the right side ahead, the robot can first slightly change its direction and sway slightly to the left to move away from the edge of the obstacle before resuming the normal path. This approach effectively avoids danger while minimizing the total travel distance, achieving high efficiency.

[0107] The final step is to review the entire process output after all the above corrective actions have been performed to ensure that the `exit_flag` flag is correctly set to the ideal state. Specifically, during a patrol mission, if a path segment is found to be inconsistent with the final safety assessment criteria, the aforementioned corrective and optimization steps are repeated until the entire path is in an ideal state. This iterative verification process ensures the practical usability and robustness of the final solution.

[0108] The present invention improves the efficiency of multi-camera synchronization. The process mainly includes the following steps: setting the camera synchronization signal trigger delay, monitoring frame rate changes and making early warnings and adjustments, ensuring that the imaging quality uniformity error meets the tolerance requirements, and achieving high-efficiency response and energy consumption optimization.

[0109] First, the camera synchronization signal trigger delay is set using the formula \(\tau_{sync\_actual}=\lceil\Delta\tau / step\_unit\rceil) to control the stability of the actual delay range. In the formula, \(\Delta\tau) is the camera desynchronization time error in the current system; \(step\_unit) is the unit step size parameter, which is used to quantify and classify the error, typically ranging from 1 to 5 ms. The optimal value is set according to the application scenario; for example, a smaller step size is preferred during rapid reconnaissance to reduce jitter error. This formula defines the actual delay \(\tau_{sync\_actual}) of synchronization triggering as the result rounded up, aiming to ensure that the triggering timing falls within a stable time window. For example, in a multi-camera system, if the maximum desynchronization error \(\Delta\tau) is detected to be 7 ms, when the unit step size is set to 2 ms, the trigger delay calculated by the formula is 4 ms (quantized to the nearest integer multiple), which can effectively stabilize the synchronization relationship between multiple cameras.

[0110] Secondly, when the frame rate change exceeds the allowable range (delta fps = |fps_current} - fps_designed}| / fps_designed), exceeding the preset threshold threshold_fps, an alarm is triggered, and the light source intensity is adjusted. In this formula, fps_current and fps_designed are the real-time measured and pre-designed frame rates, respectively, and the difference between them is standardized as a relative error to define whether action is needed. When the error exceeds the threshold, it indicates that there may be illumination instability or other environmental disturbances during multispectral imaging. At this time, the light source intensity is adjusted to restore frame rate stability, thereby ensuring the validity of subsequent data processing. For example, in complex outdoor environments, if the current frame rate is 25fps (the designed target frame rate is 30fps) due to unstable light source, i.e., (delta fps = 0.167 > threshold_fps (set to 0.1)), the aperture will be adjusted or an external supplementary light source will be added to improve the situation.

[0111] Additionally, it's crucial to ensure that the image quality uniformity error (err_quality) across all channels does not exceed the set tolerance (interval_tolerance). Here, err_quality includes evaluations of color saturation, resolution, and contrast, requiring standardized comparisons of images from different spectra. Imaging imbalances in some channels can negatively impact the overall panoramic image quality after multispectral fusion. Specifically, in one embodiment, assuming the red visible light sensor experiences a 10% decrease in sharpness under low-light conditions, while the infrared sensor only experiences a slight 3% decrease, the former should be optimized separately to reduce the quality difference to a tolerable level. This could involve using image post-processing techniques to improve the signal-to-noise ratio.

[0112] Finally, the above strategies should minimize energy waste while ensuring high system performance. For example, dynamic voltage-frequency scaling technology can be used to reduce power consumption, or intelligent power scheduling can be used to distribute the load to key functional modules. These methods help tracked robots maintain their operational capability for longer periods during reconnaissance, making them suitable for long-term operations in complex terrain and harsh weather conditions.

[0113] The image quality improvement steps of this invention consist of four parts: recording parameters, applying histogram correction formulas, integrating deep learning models, and loop closure verification.

[0114] First, record the global exposure parameter `exp_global` and the local brightness ratio `factor_bright_localavg_intensity_high / avg_intensity_low`. This step captures information about high-contrast areas by quantifying global and local brightness features. Here, `exp_global` represents the overall exposure compensation coefficient for the current scene (typically between 0.1 and 2), and `factor_bright_local` is calculated based on the ratio of the average intensity of the local bright area `avg_intensity_high` to the average intensity of the dark area `avg_intensity_low`, reflecting differences in local illumination distribution. This process provides data for subsequent processing, such as determining the compensation benchmark when illumination conditions change abruptly.

[0115] Second, a histogram correction formula, `hist_new(z) = exp_global × factor_correction × factor_bright_local`, is used to optimize brightness distribution in high-contrast areas. Here, `z` is the location variable of the image pixel intensity value, and `factor_correction` is the correction factor (a range of 0.5 to 1.5 is recommended, with an optimal value of 1 to suit general scenarios). By adjusting the histogram, this formula aims to map extreme brightness ranges to a more suitable range. Since high contrast may originate from complex natural lighting or artificial interference, in one embodiment, the formula ensures that image details are not lost in the case of overexposure when a tracked robot enters a semi-dark tunnel environment.

[0116] Third, the trained deep learning model is integrated to optimize the exposure and sharpening of the initial image `Q_initial`, generating the final optimized image `Q_optimize = net(Q_initial, fine_tuning_params)`. At this stage, the parameters `fine_tuning_params` are fine-tuned, defining the model's behavior. Their values ​​can be customized for specific tasks, such as increasing the weighting of the infrared spectral band. Specifically, combined with multispectral fusion characteristics, the model can significantly improve edge sharpness and compensate for information distortion. For example, during the reconnaissance process of a tracked robot, if image quality is degraded due to smoke and dust, this step can achieve detail restoration through intelligent analysis.

[0117] Fourth, periodically perform loopback validation to check the performance consistency of the above methods over long-term use. The core objective of this step is to ensure that the algorithm's performance does not degrade with changes in scene conditions or device status. Through multiple iterations, evaluate the output results under different lighting conditions, such as comparing whether the robot's recognition accuracy meets the standard in day-night transition scenes, to ensure the method's stability and reliability.

[0118] The intelligent reconnaissance system for tracked robots based on panoramic visual fusion and multispectral analysis of the present invention includes:

[0119] First, to address the issue of decreased reconnaissance accuracy under specific conditions, this method dynamically adjusts various sensor parameters—such as exposure time, gain, and acquisition frequency—by analyzing environmental spectral characteristics (e.g., different lighting conditions, weather, or obstruction). This dynamic adjustment mechanism ensures that the sensors maintain sensitivity and accuracy in complex environments, thereby providing high-quality data input for subsequent data analysis.

[0120] Secondly, to address image quality issues caused by changes in lighting conditions, this method employs advanced panoramic visual fusion processing technology. By performing fusion operations such as registration, denoising, and enhancement on data from multiple viewpoints and spectral bands, interference from factors such as uneven lighting and shadows is effectively reduced. This method can compensate for the negative impact of lighting changes in complex environments, improving image quality and information integrity.

[0121] Furthermore, during the target material classification process, the unique characteristics of the object's reflectance spectrum are extracted through multispectral data analysis. This is then combined with a pre-established material library model for precise comparison, thus completing the target material classification. This method based on multispectral data analysis overcomes the problem of easily confused material identification in traditional single-image reconnaissance methods, greatly improving the accuracy and robustness of reconnaissance.

[0122] Furthermore, to address the interference problem in multi-target reconnaissance, this invention constructs an optimized algorithm framework based on the strong correlation between visual data and spectral information to extract and enhance target feature information, while filtering background interference signals, enabling the system to focus on key mission targets in real time. By reducing or filtering out the weight of irrelevant information, the performance stability in multi-target scenarios is significantly improved.

[0123] Finally, this method also features path planning based on real-time reconnaissance data feedback. Specifically, it combines machine learning and prediction algorithms to analyze current target reconnaissance data and historical path records, dynamically correcting the robot's path. This allows the tracked robot to track and avoid obstacles or misjudgments in a timely manner even when the target's location changes, reducing the risk of target loss and ensuring the success rate and efficiency of reconnaissance missions. In summary, this reconnaissance method comprehensively improves the adaptability and reconnaissance capabilities of tracked robots from multiple dimensions.

[0124] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A tracked robot intelligent reconnaissance system based on panoramic vision fusion multispectral analysis, characterized in that, Comprise: Dynamic adjustment of sensor parameters based on environmental spectral characteristics to adapt to different light conditions and collect high-quality panoramic vision and multispectral data; Panoramic vision fusion processing of collected data to compensate for light changes in complex environments and enhance image quality; Multispectral data analysis based on fusion data, classification of target materials based on analysis results to improve reconnaissance accuracy, and optimization algorithm combining visual data and spectral information relevance to filter interference in multi-target reconnaissance and real-time feedback correction of robot path planning; The dynamic adjustment of sensor parameters based on environmental spectral characteristics further comprises determining the main frequency component f_main of the current environmental spectrum; Computing the variance of the spectral distribution σ 2 = 1 / N ∑(f_i - f_mean) 2 where N is the number of frequency points, f_i is the intensity of the i-th frequency point, and f_mean is the average frequency intensity; when σ 2 > threshold1, adjusting the sensor sensitivity to k x σ 2 where k is the sensitivity coefficient; calibrating the sensor to ensure consistency and accuracy of the data.

2. The intelligent reconnaissance system for tracked robots based on panoramic vision fusion multispectral analysis according to claim 1, characterized in that, The panoramic vision fusion processing of collected data comprises: According to the image pixel distribution, calculate the scene light deviation ΔL = |I_max-I_min| / I_ave, where I_max and I_min are the maximum and minimum brightness values, and I_ave is the brightness average; If ΔL exceeds threshold2, use the global brightness compensation formula C_compensate = C_original + λ × ΔL for image enhancement, and λ is the compensation factor; Apply panoramic stitching algorithm to seamlessly stitch images into 360° view; Use edge-preserving filtering to eliminate artifacts and enhance detail quality in complex environments.

3. The intelligent reconnaissance system for tracked robots based on panoramic vision fusion multispectral analysis according to claim 1, characterized in that, The multispectral data analysis based on fusion data comprises: Use normalization algorithm to process energy values E_i of different wavebands to get E_norm(i) = (E_i-E_min) / (E_max-E_min); Define target material spectral matching degree D = Σ|S_ref(i)E_norm(i)|, S_ref(i) is the reference material spectral data; Select the category of S_ref(i) corresponding to the minimum D as the material type; Add a probability classification module to the material type to adapt to uncertainty.

4. The intelligent reconnaissance system for tracked robots based on panoramic vision fusion multispectral analysis according to claim 1, characterized in that, The relevance optimization algorithm combining visual data and spectral information comprises: Construct a joint feature space J = [V1, V2,..., VS], where Vi represents the position of each target in the feature vector space; Obtain weight w_j = Σcov(F_vis(j), F_spec(j)) / Σvar(F_vis(j)) through covariance analysis, F_vis and F_spec represent the variance of visual and spectral feature dimensions respectively; Set the interference removal threshold threshold4 and remove interference w_j < threshold4 through the following conditions; Retrain the classifier based on the remaining high-confidence features for accurate target identification.

5. The intelligent reconnaissance system for tracked robots based on panoramic vision fusion multispectral analysis according to claim 1, characterized in that, The interference filtering in multi-target reconnaissance further comprises: Map the real-time collected interference intensity signal to the interference level indicator M, M = log(∑|I_noise I_baseline|^2) / log(N_noise), N_noise represents the total number of interference frames; When M > threshold5, start the redundant data verification logic to remove invalid frames; Extracting relative motion features H_diff(j) among multiple targets in the effective frame to distinguish between static and moving entities; Prioritize high-value moving targets based on classification results to optimize reconnaissance efficiency.

6. The intelligent reconnaissance system of tracked robots based on panoramic vision fusion multispectral analysis according to claim 5, characterized in that, The optimization condition for filtering interference in multi-target reconnaissance is: Introducing a self-learning adjustment module to update the interference level evaluation formula A = αM + β∑H_rel, where α and β are weight parameters, and H_rel is the sum of relative relationship features; If A > threshold6, enable the backup sensing system to ensure reliability; Integrate redundant sensor data into the existing reconnaissance model to form a closed-loop correction structure; Improve target positioning resolution R_loc ≥ resolution_goal in the key frame detection stage.

7. The intelligent reconnaissance system for tracked robots based on panoramic vision fusion multispectral analysis according to claim 1, characterized in that, The real-time feedback correction of the robot's path planning includes: Define the proportional deviation of the current position from the benchmark trajectory as P_deviation = |pos_current pos_benchmark| / path_len_total; When P_deviation > threshold7 or the obstacle distance D_obst ≤ threshold8, start the re-planning program; Generate multiple candidate routes r_candidate and filter them according to the estimated cost function COST(r_candidate, t_now); Opt for the path with the lowest cost while updating historical data to support improved prediction capabilities.

8. The intelligent reconnaissance system of tracked robots based on panoramic vision fusion multispectral analysis according to claim 7, characterized in that, The reinforcement path intelligent correction step in the real-time feedback correction of the robot's path planning is: Establish a prediction window T_future = time_remain * γ, where the time window multiplier γ is set according to task priority, and predict the set of problem points P_future that may be encountered in the future; If any p ∈ P_future satisfies the conflict judgment formula C(p_i, t_estimtated, v_target) > critical_conflict_value, avoid it in advance; Introduce a double-level obstacle avoidance rule to adjust the angle locally and then adjust the direction as a whole to avoid excessive detours; After completing the adjustment, perform safety detection again until the standard condition exit_flag = true is met.

9. The intelligent reconnaissance system of tracked robots based on panoramic vision fusion multispectral analysis according to claim 1, characterized in that, Improving multi-objective synchronization efficiency includes: Set the camera synchronization signal trigger delay τ_synchronization and control the actual delay range to be stable by the formula τ_sync_actual = ceil[Δτ / step_unit)], where step_unit is the unit step distance; When the frame rate changes exceed the allowed range δfps = |fps_current fps_designed| / fps_designed greater than threshold_fps, trigger the early warning mechanism and adjust the light source intensity; Ensure that the imaging quality balance error err_quality of all channels does not exceed the specified tolerance interval_tolerance; Realize fast response while reducing energy consumption to adapt to long-term operating environment requirements.

10. The intelligent reconnaissance system for tracked robots based on panoramic vision fusion multispectral analysis according to claim 1, characterized in that, The steps to enhance image quality problem repair include: Record global exposure parameter exp_global and local brightness ratio factor_bright_local = avg_intensity_high / avg_intensity_low; For high-contrast regions, use the histogram correction formula hist_new(z) = exp_global × factor_correction × factor_bright_local; Integrate deep learning model fine-tuning exposure and sharpening effect output optimized image Q_optimize = net(Q_initial, fine_tuning_params); Periodically loop back to verify the effectiveness of the repair strategy to ensure long-term consistent stability.

Images