A visual recognition method and system

Abstract

This invention discloses a visual recognition method and system. The method includes: acquiring visible light images, infrared thermal images, and depth point cloud data of a target scene to generate a spatiotemporally consistent multimodal heterogeneous feature tensor; inputting the multimodal heterogeneous feature tensor into a spatial frequency-aware optimizer to generate a detail-enhanced optimized feature matrix; based on the optimized feature matrix, using an adversarial generative network to synthesize multi-scale occlusion samples to generate an adversarially robust recognition feature vector; inputting the recognition feature vector into an adaptive decision engine to generate environment-adaptive dynamic decision parameters; constructing a multi-scale verification pyramid based on the dynamic decision parameters; fusing the confidence scores of each level through a self-calibration module to output the final recognition result. Using this invention, highly robust and accurate visual recognition can be achieved.

Description

Technical Field

[0001] This invention belongs to the field of visual recognition technology, specifically a visual recognition method and system. Background Technology

[0002] With the rapid development of applications such as industrial automation, intelligent monitoring, and autonomous driving, visual recognition technology plays an increasingly important role in target detection, behavior analysis, and environmental perception. Traditional visual recognition methods mainly rely on single-spectral images, which struggle to cope with complex environmental interference and occlusion issues, leading to decreased recognition accuracy. Furthermore, factors such as lighting variations, occlusion, and environmental noise severely impact recognition performance, making the achievement of highly robust and accurate target recognition in real-world scenarios a significant challenge. Summary of the Invention

[0003] The purpose of this invention is to provide a visual recognition method and system to overcome the shortcomings of the prior art and achieve highly robust and accurate visual recognition.

[0004] One embodiment of this application provides a visual recognition method, the method comprising:

[0005] The system acquires visible light images, infrared thermal images, and depth point cloud data of the target scene. It then performs cross-domain alignment of the multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network uses a self-attention mechanism to fuse edge response features from different spectral frequency bands.

[0006] The multimodal heterogeneous feature tensor is input into a spatial frequency-aware optimizer, and the feature map is reconstructed through an adaptive high-frequency enhancement filter to generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity.

[0007] Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples. Combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraints are used to force the preservation of temporal feature continuity through an optical flow field loss function.

[0008] The identified feature vector is input into the adaptive decision engine, and the classification boundary is dynamically adjusted through a reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene.

[0009] A multi-scale verification pyramid is constructed based on the dynamic decision parameters. The confidence scores of each level are fused through a self-correction module to output the final recognition result. The self-correction module uses an asymmetric loss function to eliminate cross-level feature conflicts.

[0010] Optionally, the acquisition of visible light images, infrared thermal imaging, and depth point cloud data of the target scene, and the cross-domain alignment of the multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor, wherein the heterogeneous feature fusion network employs a self-attention mechanism to fuse edge response features from different spectral frequency bands, including:

[0011] Based on the frame synchronization signal of the visible light image and the timestamp of the infrared thermal image, a dynamic interpolation compensation algorithm is used to align the sampling interval of the depth point cloud data to generate a three-modal data stream with strict time synchronization.

[0012] Multispectral edge detection is performed on visible light images, and the temperature gradient of infrared thermal imaging is used to correct the visible light edge fracture region. At the same time, the curvature change feature of depth point cloud is extracted to generate a cross-modal edge response map.

[0013] The cross-modal edge response map is input into a dual-path self-attention network, and weights are dynamically allocated through a visible-infrared feature cross-calibration module to generate a spectrally consistent edge feature matrix.

[0014] The fused edge feature matrix is ​​subjected to 3D spatiotemporal convolution, and combined with the spatial topological constraints of the depth point cloud, the spatiotemporally consistent multimodal heterogeneous feature tensor is output.

[0015] Optionally, the step of inputting the multimodal heterogeneous feature tensor into a spatial frequency-aware optimizer and reconstructing the feature map through an adaptive high-frequency enhancement filter to generate a detail-enhanced optimized feature matrix includes:

[0016] Based on the brightness histogram distribution of the visible light image, a dynamic threshold segmentation algorithm is used to extract the scene illumination intensity level and generate an illumination intensity quantization index.

[0017] Two-dimensional wavelet decomposition is performed on the multimodal heterogeneous feature tensor to separate high-frequency detail components from low-frequency contour components. The convolution kernel weight ratio of the high-frequency detail components is dynamically adjusted based on the illumination intensity quantization index.

[0018] The weighted high-frequency detail components and low-frequency contour components are fused using adaptive inverse wavelet transform, and the reconstruction error is compensated by residual connection to generate an optimized feature matrix with enhanced details.

[0019] Local contrast analysis is performed on the optimized feature matrix, and a nonlinear filtering algorithm is used to suppress artifact noise caused by sudden changes in illumination. The final optimized feature matrix after noise suppression is then output.

[0020] Optionally, based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples, and combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraint, through an optical flow field loss function, forces the preservation of temporal feature continuity, including:

[0021] The optimized feature matrix is ​​input into the morphological erosion generator, and the edge regions of the feature map are eroded by random structuring elements to synthesize a multi-scale occlusion sample set.

[0022] Based on the feature matrix optimized by adjacent frames, a dense optical flow estimation algorithm is used to extract the motion trajectory of the sample occlusion area and generate a spatiotemporal continuity constraint vector.

[0023] By combining the spatiotemporal continuity constraint vector of optical flow trajectory with a multi-scale occlusion sample set, a direction consistency loss function is designed to force the adversarial generative network to retain temporal motion features when synthesizing samples.

[0024] The generator trained adversarially enhances the optimized feature matrix that preserves temporal motion features by perturbating the features. The recognition feature vector with enhanced adversarial robustness is then extracted through Siamese network contrastive learning.

[0025] Optionally, the step of inputting the identified feature vector into an adaptive decision engine, dynamically adjusting the classification boundary through a reinforcement learning-driven graph neural network, and generating environment-adaptive dynamic decision parameters, wherein the graph neural network automatically activates subgraph structures of different depths according to the real-time scene complexity, including:

[0026] Based on the spatial distribution density of the identified feature vectors, a node density clustering algorithm is used to calculate the real-time scene complexity index.

[0027] A deep routing strategy based on scene complexity index-triggered graph neural network automatically selects shallow local subgraphs or deep global subgraphs for feature aggregation.

[0028] The classification decision boundary of nodes in shallow local subgraphs or deep global subgraphs is dynamically adjusted through a reinforcement learning reward mechanism. The reward function is determined by both classification confidence and feature distribution density.

[0029] The optimized classification decision boundary is Gaussian smoothed, and the dynamic decision parameter matrix that adapts to the environment is output by combining the scene complexity index.

[0030] Optionally, the step of constructing a multi-scale verification pyramid based on the dynamic decision parameters, fusing the confidence scores of each level through a self-correction module, and outputting the final recognition result, wherein the self-correction module uses an asymmetric loss function to eliminate cross-level feature conflicts, including:

[0031] Based on the scale sensitivity of the dynamic decision parameter matrix, a hierarchical random sampling strategy is adopted to construct a multi-scale validation pyramid and generate hierarchical confidence benchmarks.

[0032] The confidence scores of each level of the pyramid are aligned by bidirectional attention, and the score bias between scales is eliminated by feature similarity measurement to generate a consistent and calibrated confidence distribution.

[0033] The design incorporates an asymmetric loss function with directional constraints, and performs differentiated weighted fusion of high-level semantic confidence and low-level detail confidence to suppress cross-layer feature conflicts.

[0034] The fused confidence distribution is input into a multi-level weighted voting machine, which dynamically selects the optimal level result based on a preset threshold, and outputs the final recognition result and a traceable decision path.

[0035] Optionally, the step of constructing a multi-scale validation pyramid and generating hierarchical confidence benchmarks based on the scale sensitivity of the dynamic decision parameter matrix using a hierarchical random sampling strategy includes:

[0036] Based on the gradient distribution characteristics of the dynamic decision parameter matrix, the ripple diffusion algorithm is used to analyze the sensitivity of parameter changes at different scales, and a three-dimensional distribution map containing the scale-sensitivity mapping relationship is generated.

[0037] Based on the 3D distribution map, feature basis vectors at each scale are extracted through a dynamic registration algorithm, redundant basis vectors are filtered out using feature similarity measurement, and a set of scale orthogonal basis vectors is generated.

[0038] The set of scale-orthogonal basis vectors is input into the adaptive sliding window generator. A pyramid hierarchical structure is constructed through the nonlinear relationship between window size and scale sensitivity. At the same time, a neighborhood smoothing algorithm is used to eliminate noise interference.

[0039] Cross-scale confidence propagation is performed for each pyramid level, and the statistical distribution characteristics of historical identification results are combined to generate a level confidence benchmark and error tolerance threshold.

[0040] Another embodiment of this application provides a visual recognition system, the system comprising:

[0041] The acquisition module is used to acquire visible light images, infrared thermal images and depth point cloud data of the target scene. It performs cross-domain alignment of multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network adopts a self-attention mechanism to fuse edge response features of different spectral frequency bands.

[0042] The enhancement module is used to input the multimodal heterogeneous feature tensor into the spatial frequency-aware optimizer, reconstruct the feature map through an adaptive high-frequency enhancement filter, and generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity.

[0043] The generation module is used to synthesize multi-scale occlusion samples based on the optimized feature matrix using an adversarial generative network, and generate an adversarial robustness-enhanced recognition feature vector by combining a temporal consistency constraint. The temporal consistency constraint forces the preservation of temporal feature continuity through an optical flow field loss function.

[0044] The adjustment module is used to input the recognition feature vector into the adaptive decision engine, and dynamically adjust the classification boundary through the reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene.

[0045] The identification module is used to construct a multi-scale verification pyramid based on the dynamic decision parameters, and to fuse the confidence scores of each level through the self-correction module to output the final identification result. The self-correction module uses an asymmetric loss function to eliminate cross-level feature conflicts.

[0046] Another embodiment of this application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any of the preceding claims when running.

[0047] Another embodiment of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method described in any of the preceding claims.

[0048] Compared with existing technologies, the present invention provides a visual recognition method that acquires visible light images, infrared thermal images, and depth point cloud data of a target scene to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The multimodal heterogeneous feature tensor is then input into a spatial frequency-aware optimizer to generate a detail-enhanced optimized feature matrix. Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples, generating an adversarial robustness-enhanced recognition feature vector. This recognition feature vector is then input into an adaptive decision engine to generate environment-adaptive dynamic decision parameters. A multi-scale verification pyramid is constructed based on these dynamic decision parameters, and a self-calibration module fuses the confidence scores from each level to output the final recognition result, thereby achieving highly robust and accurate visual recognition. Attached Figure Description

[0049] Figure 1A hardware structure block diagram of a computer terminal for a visual recognition method provided in an embodiment of the present invention;

[0050] Figure 2 This is a flowchart illustrating a visual recognition method provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of a visual recognition system provided in an embodiment of the present invention. Detailed Implementation

[0052] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0053] This invention first provides a visual recognition method that can be applied to electronic devices, such as computer terminals, specifically ordinary computers.

[0054] The following detailed explanation uses a computer terminal as an example. Figure 1 This is a hardware structure block diagram of a computer terminal for a visual recognition method provided in an embodiment of the present invention. (See diagram below.) Figure 1 As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0055] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any visual recognition method.

[0056] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0057] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any visual recognition method.

[0058] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0059] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0060] See Figure 2 The present invention provides a visual recognition method, which may include the following steps:

[0061] S201, acquire visible light images, infrared thermal imaging and depth point cloud data of the target scene, and perform cross-domain alignment of multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network adopts a self-attention mechanism to fuse edge response features of different spectral frequency bands.

[0062] This step constructs a 3D environment perception system through multi-sensor collaborative acquisition, employing a self-attention mechanism to address the inherent differences in spectral response and spatial resolution among different modal data. A heterogeneous feature fusion network maps visible light texture, infrared thermal radiation, and depth geometric features to a unified high-dimensional tensor space through frequency domain transformation, overcoming the limitations of single-modal data in traditional visual recognition. Cross-domain feature alignment significantly enhances the environment perception capability under complex lighting conditions, providing standardized feature inputs with spatiotemporal consistency for subsequent processing.

[0063] S202, the multimodal heterogeneous feature tensor is input into the spatial frequency-aware optimizer, and the feature map is reconstructed through an adaptive high-frequency enhancement filter to generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity.

[0064] This innovative step combines image frequency domain analysis with deep learning, achieving feature enhancement through a dynamic filtering mechanism based on illumination. The adaptive high-frequency enhancement filter employs a differentiable wavelet transform kernel, adjusting the frequency band weights in real time according to local illumination conditions. While preserving low-frequency contours, it enhances high-frequency details, solving the performance degradation problem of traditional feature extraction methods under sudden illumination changes. Through physically prior-guided frequency domain optimization, it significantly improves the expressive power of key features such as edges and textures.

[0065] S203, Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples. Combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraints are used to force the preservation of temporal feature continuity through an optical flow field loss function.

[0066] This step constructs a dynamic defense system through generative adversarial learning, utilizing optical flow field constraints to ensure the spatiotemporal rationality of the synthesized samples. A morphological erosion generator simulates realistic occlusion scenarios, while a temporal consistency loss function forces the model to learn motion continuity features, significantly improving the system's robustness to dynamic occlusion and adversarial attacks. Through spatiotemporal joint optimization, the model acquires continuous learning capabilities, adapting to complex scene changes.

[0067] S204, the identification feature vector is input into the adaptive decision engine, and the classification boundary is dynamically adjusted through the reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene.

[0068] This step constructs a dynamic decision-making architecture based on reinforcement learning. The graph neural network automatically switches between shallow local reasoning and deep global reasoning modes through scene complexity awareness. The reward mechanism combines classification confidence with feature distribution density to achieve optimal adjustment of the decision boundary, overcoming the limitations of fixed-structure neural networks and enabling the system to adaptively balance computational efficiency and recognition accuracy according to environmental complexity.

[0069] S205, construct a multi-scale verification pyramid based on the dynamic decision parameters, fuse the confidence scores of each level through a self-correction module, and output the final recognition result. The self-correction module uses an asymmetric loss function to eliminate cross-layer feature conflicts.

[0070] This step designs a hierarchical verification mechanism to address cross-scale feature conflicts, and uses an asymmetric loss function to differentiate between high-level semantics and low-level detailed features. A bidirectional attention mechanism enables cross-layer confidence calibration, and a multi-level voting system provides an interpretable decision path, significantly reducing the risk of mismatches in multi-scale analysis. The traceable hierarchical verification process enhances the system's decision reliability in open environments.

[0071] Specifically, the acquisition of visible light images, infrared thermal imaging, and depth point cloud data of the target scene, followed by cross-domain alignment of the multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor, wherein the heterogeneous feature fusion network employs a self-attention mechanism to fuse edge response features from different spectral frequency bands, including:

[0072] Based on the frame synchronization signal of the visible light image and the timestamp of the infrared thermal image, a dynamic interpolation compensation algorithm is used to align the sampling interval of the depth point cloud data to generate a three-modal data stream with strict time synchronization.

[0073] At the hardware level, the raw data from visible light cameras (such as the Sony IMX477, 30fps), infrared thermal imagers (FLIR A35, 15fps), and depth sensors (Intel RealSense D455, 10Hz point cloud sampling rate) are misaligned on the timeline due to differences in sampling rates. The core of the dynamic interpolation compensation algorithm is to achieve multimodal synchronization through timestamp alignment and data reconstruction.

[0074] Timestamp alignment:

[0075] The visible light and infrared frame synchronization signals are synchronized at the microsecond level through hardware trigger lines (such as GPIO), with the deviation controlled within ±0.5ms.

[0076] The timestamps of the depth point cloud data are aligned with the main control computer clock via NTP (Network Time Protocol) with an accuracy of ±2ms. For 10Hz depth data, a linear interpolation method is used to insert 9 virtual point cloud frames every 100ms interval, increasing its equivalent sampling rate to 100Hz, matching the visible light / infrared frame rate. For example, if real point cloud frames are located at t=100ms and 200ms, interpolated frames are generated at t=110ms, 120ms, etc.

[0077] Data Restructuring:

[0078] Visible light image: original resolution 1920×1080, bicubic interpolation is used to maintain edge sharpness;

[0079] Infrared thermal imaging: 640×480 resolution, with temperature difference contrast enhanced by adaptive histogram equalization;

[0080] Deep point cloud: The interpolated point cloud uses KD-Tree spatial indexing to accelerate nearest neighbor search and fill in missing regions (such as holes caused by occlusion).

[0081] The final output trimodal data stream is strictly aligned based on timestamps, and each frame of data contains:

[0082] Visible light RGB image (30fps);

[0083] Infrared temperature matrix (interpolated from 15fps to 30fps);

[0084] Deep point cloud (10Hz interpolated to 30fps, approximately 300,000 points per frame).

[0085] Multispectral edge detection is performed on visible light images, and the temperature gradient of infrared thermal imaging is used to correct the visible light edge fracture region. At the same time, the curvature change feature of depth point cloud is extracted to generate a cross-modal edge response map.

[0086] Cross-modal edge detection improves the robustness of edge detection by fusing complementary information from visible light, infrared, and depth data.

[0087] Multispectral edge detection:

[0088] Visible light edges: An improved Canny algorithm (Gaussian kernel σ = 1.2, high-low threshold ratio 1:3) is used to extract edges with gradient intensity > 50;

[0089] Infrared correction: Calculate the temperature difference gradient between adjacent pixels. When the temperature difference is >3℃, it is marked as a potential edge. If the visible light edge breaks at the same location (e.g., due to reflection), it is filled with an infrared edge.

[0090] Depth curvature: Local surface fitting (within a 5cm radius neighborhood) is performed on the point cloud, and the average curvature is calculated. Regions with curvature abrupt changes > 0.05mm⁻¹ are marked as geometric edges.

[0091] Edge fusion logic:

[0092] Logic and Fusion: When both visible light and infrared light detect edges at the same location, preserve the edge details of the visible light source;

[0093] Logical OR fusion: In low-light areas (visible light gradient < 20), infrared or depth edge is preferred;

[0094] Conflict resolution: If the spatial positional deviation of the three edge types is greater than 5 pixels, a voting mechanism is initiated (if at least two types are consistent, the edge type is retained).

[0095] The cross-modal edge response map is stored in probabilistic form, with each pixel value representing a confidence level (0–1) that the edge exists. For example, in a metallic reflective region, the visible light edge is broken but the infrared temperature difference is significant, and the final confidence level is adjusted from 0.3 to 0.85.

[0096] The cross-modal edge response map is input into a dual-path self-attention network, and weights are dynamically allocated through a visible-infrared feature cross-calibration module to generate a spectrally consistent edge feature matrix.

[0097] The Dual-path Self-Attention Network (DSAN) consists of a visible light branch, an infrared branch, and a cross-calibration module, as shown in the following structure:

[0098] Visible light branch:

[0099] Input: Cross-modal edge response map (512×512×1);

[0100] Processing: 4 convolutional layers (kernel size 3×3, number of channels 16→32→64→128), ReLU activation, output high-level semantic feature map (64×64×128).

[0101] Infrared branch:

[0102] Input: Infrared temperature matrix (upsampled to 512×512);

[0103] Processing: Four convolutional layers with the same structure output feature maps of the same size.

[0104] Cross-calibration module:

[0105] Self-attention mechanism: Query, key, and value matrices are calculated for the visible light and infrared feature maps, respectively, with dimensions 64×64×128. Attention weights are calculated by scaling the dot product attention (scaling factor 1 / √128).

[0106] Dynamic weight allocation: Softmax is used to generate visible light weight α and infrared weight β (α+β=1), for example, β can reach 0.8 in low light regions;

[0107] Feature fusion: Fusion feature = α × visible light feature + β × infrared feature.

[0108] Output processing:

[0109] The fused features are deconvolved (kernel size 4×4, stride 2) to gradually restore them to the original resolution;

[0110] The final output is a spectrum-consistent edge feature matrix (512×512×1), with an edge continuity error of <1 pixel.

[0111] The fused edge feature matrix is ​​subjected to 3D spatiotemporal convolution, and combined with the spatial topological constraints of the depth point cloud, the spatiotemporally consistent multimodal heterogeneous feature tensor is output.

[0112] 3D spatiotemporal convolution is used to fuse temporal and spatial information, and enhance feature consistency by combining the spatial constraints of deep point clouds.

[0113] Spatiotemporal convolutional structure:

[0114] Input: Edge feature matrix sequence (30 frames / second, 512×512×1 per frame);

[0115] Convolutional layers: 3D convolutional kernels (3×3×3, 3 frames in time dimension, 3×3 in spatial dimension), with the number of channels expanded from 1 to 32;

[0116] Activation function: LeakyReLU (negative slope 0.1);

[0117] Pooling layer: max pooling (2×2×1), downsampling to 256×256×32.

[0118] Depth space constraints:

[0119] Point cloud projection: Project the depth point cloud onto a 2D image plane to generate a depth map (512×512);

[0120] Spatial masking: Apply weight attenuation (weight × 0.2) to regions with a depth > 5m (such as the background) to focus on nearby targets;

[0121] Topology optimization: A point cloud topology network is constructed based on Delaunay triangulation, and the features of the convolution kernel are constrained to propagate between neighboring nodes.

[0122] Output tensor:

[0123] Dimensions: 256×256×32×30 (width×height×channel×time);

[0124] Spatiotemporal consistency index: feature difference between adjacent frames <0.05 (cosine similarity).

[0125] Example: In pedestrian detection scenarios, this tensor can simultaneously represent pedestrian contours (visible light edges), body temperature features (infrared), and three-dimensional position (depth), with spatiotemporal offset error controlled within 3 pixels.

[0126] Specifically, the step of inputting the multimodal heterogeneous feature tensor into a spatial frequency-aware optimizer, reconstructing the feature map through an adaptive high-frequency enhancement filter, and generating an optimized feature matrix with enhanced details, wherein the filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity, including:

[0127] Based on the brightness histogram distribution of the visible light image, a dynamic threshold segmentation algorithm is used to extract the scene illumination intensity level and generate an illumination intensity quantization index.

[0128] The luminance histogram of a visible light image reflects the statistical characteristics of the scene's illumination distribution. First, the input image is converted to grayscale, and its luminance histogram (levels 0-255) is calculated. The dynamic thresholding segmentation algorithm employs a modified Otsu method, determining the optimal segmentation threshold by maximizing the inter-class variance. For example, in a uniformly lit indoor scene, the histogram exhibits a unimodal distribution, and the segmentation threshold might be around 128; while in a high-contrast outdoor scene, the histogram may exhibit a bimodal distribution, and the segmentation threshold might drop to 64.

[0129] Generation of the light intensity quantization index:

[0130] Dynamic interval division: The histogram is divided into 3 intervals (low light: 0-85, medium light: 86-170, high light: 171-255);

[0131] Weighting calculation: Calculate the percentage of pixels in each range, for example, 15% in low light, 60% in medium light, and 25% in high light.

[0132] Exponential mapping: The weights are mapped to a quantization index of 0 to 1 through a piecewise linear function (e.g., a weight of 0.15 for low light corresponds to an index of 0.3, 0.6 for medium light corresponds to an index of 0.7, and 0.25 for high light corresponds to an index of 0.9).

[0133] Example: In an indoor monitoring scenario, the quantization index is 0.7, triggering the conservative mode of subsequent high-frequency enhancement; while in a backlight scenario, the index may drop to 0.4, triggering the strong high-frequency compensation mode.

[0134] Two-dimensional wavelet decomposition is performed on the multimodal heterogeneous feature tensor to separate high-frequency detail components from low-frequency contour components. The convolution kernel weight ratio of the high-frequency detail components is dynamically adjusted based on the illumination intensity quantization index.

[0135] The multimodal heterogeneous feature tensor contains the fusion results of visible light, infrared, and depth features (e.g., size 256×256×64). Two-dimensional wavelet decomposition uses the Daubechies 4 wavelet basis (DB4) and is decomposed into 4 subbands:

[0136] Low-frequency subband (LL): Preserves profile information (size 128×128×64);

[0137] High-frequency subbands (LH, HL, HH): contain horizontal, vertical and diagonal details (128×128×64 each).

[0138] Dynamic adjustment of convolution kernel weights:

[0139] Weighting function: Define the weighting factor α = 1.5 × illumination quantization index (α = 1.05 when the index is 0.7);

[0140] High-frequency enhancement strategy: Apply 3×3 convolution kernels to the LH, HL, and HH subbands respectively, and amplify the kernel weights proportionally. For example, the original kernel weight matrix is ​​[[0.1,0.2,0.1],[0.2,0.3,0.2],[0.1,0.2,0.1]], which becomes [[0.105,0.21,0.105],[0.21,0.315,0.21],[0.105,0.21,0.105]] after adjustment;

[0141] Low-frequency suppression: Apply a weight attenuation factor β = 0.8 to the LL subband to suppress redundant contour information.

[0142] Example: In low-light scenes (index 0.3), α = 0.45, high-frequency components are only slightly enhanced to avoid noise amplification; while in high-light scenes (index 0.9), α = 1.35, detail features are significantly enhanced.

[0143] The weighted high-frequency detail components and low-frequency contour components are fused using adaptive inverse wavelet transform, and the reconstruction error is compensated by residual connection to generate an optimized feature matrix with enhanced details.

[0144] The adaptive inverse wavelet transform employs a dual-channel reconstruction network:

[0145] Low-frequency channel: The attenuated LL subband is upsampled by bilinear interpolation (2 times) to restore the resolution to 256×256;

[0146] High-frequency channels: Deconvolution layers (3×3 kernel, stride 2) are applied to the enhanced LH / HL / HH subbands respectively to restore the detailed structure;

[0147] Feature fusion: The outputs of the low-frequency and high-frequency channels are spliced ​​together (256×256×128) and compressed to 64 channels through 1×1 convolution.

[0148] Residual connection design:

[0149] Skip connection: Add the original feature tensor to the reconstruction result to compensate for information loss in wavelet transform;

[0150] Residual learning: Add residual blocks (containing three 3×3 convolutional layers, each followed by ReLU activation) to learn the amount of compensation for reconstruction error.

[0151] Example: In the reconstruction of complex industrial parts surfaces, residual connections improve edge sharpness by 15% and reduce artifacts by 30%.

[0152] Local contrast analysis is performed on the optimized feature matrix, and a nonlinear filtering algorithm is used to suppress artifact noise caused by sudden changes in illumination. The final optimized feature matrix after noise suppression is then output.

[0153] Local contrast analysis uses a sliding window (16×16 pixels) to traverse the feature matrix:

[0154] Contrast calculation: The standard deviation of pixels within the window is used as the contrast index (e.g., a standard deviation > 25 is considered a high contrast area);

[0155] Noise detection: Combining the thermal distribution stability of infrared features (standard deviation <5℃ is considered a stable region) to distinguish between real edges and noise.

[0156] Nonlinear filtering algorithm:

[0157] Bilateral filtering: The spatial domain kernel (σ_d = 3 pixels) and the gray-level kernel (σ_r = 10 gray levels) work together to preserve edges while smoothing homogeneous areas;

[0158] Adaptive median filtering: dynamically adjusts the window size (from 3×3 to 7×7) to suppress salt-and-pepper noise;

[0159] Illumination equalization: When local overexposed areas are detected (e.g., quantization index > 0.8), the CLAHE algorithm (limited contrast adaptive histogram equalization, tile size 8×8, clip limit = 2.0) is applied to balance the brightness.

[0160] Example: In a backlit face image, the original feature matrix has overexposure artifacts in the cheekbone area. After filtering, the artifact intensity is reduced by 60% while preserving pupil details.

[0161] Specifically, based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples. Combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraints, through an optical flow field loss function, force the preservation of temporal feature continuity, including:

[0162] The optimized feature matrix is ​​input into the morphological erosion generator, and the edge regions of the feature map are eroded by random structuring elements to synthesize a multi-scale occlusion sample set.

[0163] The core objective of the Morphological Erosion Generator (MEG) is to enhance the model's robustness by simulating physical occlusion (such as raindrops, dust, and partial object occlusion). Its implementation consists of three stages:

[0164] Random Structuring Element Generation: The structuring element (SE) is the core of the erosion operation, and MEG employs a dynamic random generation strategy.

[0165] Shape: circle (radius 3-15 pixels), rectangle (side length 5-20 pixels), or irregular polygon (number of vertices 3-6), randomly selected in a 1:1:1 ratio;

[0166] Scale: Dynamically adjusted according to the resolution of the input feature map. For example, for a 512×512 feature map, the maximum SE size is 32×32.

[0167] Density: The coverage of the shaded area is controlled by Poisson distribution (5%–30%).

[0168] For example, in an autonomous driving scenario, the generator might create a circular SE with a radius of 8 pixels that covers 20% of the edge of the vehicle detection box to simulate raindrop occlusion.

[0169] Multi-scale erosion operation: Perform channel-by-channel erosion on the optimized feature matrix (assuming dimensions are H×W×C, such as 256×256×64):

[0170] Erosion kernel sliding: Traverse the feature map with a step size of 4 pixels, and apply SE to each position for minimum filtering to weaken the edge response;

[0171] Multi-scale stacking: Apply different scales of SE (such as 5×5, 9×9, 13×13) to the same feature map in sequence to generate multi-level occlusion samples;

[0172] Channel weighting: Attention weights are applied to the erosion results of different channels (e.g., channel 1 weight 0.8, channel 2 weight 0.5) to preserve key features.

[0173] Enhanced sample diversity:

[0174] Dynamic occlusion: In a video sequence, position interpolation is performed on the occlusion area of ​​consecutive frames to simulate motion occlusion (such as a bird flying past the camera);

[0175] Texture blending: The eroded area is blended with random noise (Gaussian noise σ=0.1) or real occluded textures (such as leaf patterns) to improve the realism of the sample.

[0176] Example: In the pedestrian re-identification task, when MEG generates occlusion samples, it applies a rectangular SE (15×10 pixels) to the upper body area of ​​the pedestrian to cover the backpack area, and at the same time overlays an irregular polygon SE on the lower body to simulate shadow occlusion, and finally generates a sample set containing 10 occlusion modes.

[0177] Based on the feature matrix optimized by adjacent frames, a dense optical flow estimation algorithm is used to extract the motion trajectory of the sample occlusion area and generate a spatiotemporal continuity constraint vector.

[0178] Dense optical flow estimation algorithms are used to capture the temporal consistency of motion in occluded regions, ensuring that the generated occluded samples conform to the laws of physical motion.

[0179] Optical flow estimation model selection: PWC-Net (Pyramidal Warping and Cost Volume Network) is adopted, whose lightweight architecture (9.4M parameters) is suitable for real-time processing. The input is the optimized feature matrix of two adjacent frames (frame t and frame t+1), and the output is the H×W×2 optical flow field (horizontal and vertical displacement).

[0180] Extraction of motion trajectory in occluded areas:

[0181] Forward-backward optical flow consistency check: Calculate the forward optical flow F_forward from t to t+1 and the backward optical flow F_backward from t+1 to t. If ||F_forward+F_backward||>1 pixel, it is determined to be an occlusion area.

[0182] Trajectory chain construction: For each pixel in the occluded area, a motion trajectory chain is recursively generated through optical flow. For example, the trajectory of a raindrop within 5 frames is {(x1,y1),(x2,y2),...,(x5,y5)}, with displacement Δx = 3 pixels / frame and Δy = 0.5 pixels / frame.

[0183] Spatiotemporal continuity constraint vector generation:

[0184] Motion consistency coding: This involves encoding the displacement (Δx, Δy) and acceleration (Δy) of the trajectory chain. 2 x,Δ 2 y) is encoded as a 4-dimensional vector;

[0185] Temporal smoothing filtering: Kalman filtering is applied to the trajectory chain (process noise Q = 0.1, observation noise R = 1.0) to eliminate estimation jitter;

[0186] Region aggregation: DBSCAN clustering is performed on adjacent trajectories (distance < 5 pixels) (eps = 5, min_samples = 3) to generate region-level constraint vectors.

[0187] Example: In surveillance video, a car is partially occluded. Optical flow estimation shows that the occluder (such as a bird) in the hood area is moving upward and to the right at a speed of 15 pixels per second. Constraint vectors encode this motion pattern to ensure that subsequently generated occlusion samples are temporally continuous.

[0188] By combining the spatiotemporal continuity constraint vector of optical flow trajectory with a multi-scale occlusion sample set, a direction consistency loss function is designed to force the adversarial generative network to retain temporal motion features when synthesizing samples.

[0189] The Direction Consistency Loss (DCL) function improves the temporal robustness of adversarial training by constraining the consistency between the motion direction of generated samples and the real optical flow.

[0190] Loss function design:

[0191] Cosine similarity constraint: Calculate the cosine similarity between the optical flow direction (θ_gen) and the real optical flow direction (θ_real) of the occluded region of the generated sample, with the loss term being 1-Similarity, to force orientation alignment;

[0192] Amplitude matching constraint: Apply L1 loss L_magnitude to the optical flow amplitude (||F||);

[0193] Total loss: DCL = 0.7 × (1 - Similarity) + 0.3 × L_magnitude.

[0194] Combat training process:

[0195] Generator (G): Input is the optimized feature matrix, output is the occluded samples;

[0196] Discriminator (D): The input is a real sample or a generated sample, and the output is the authenticity probability + optical flow consistency score;

[0197] Training strategy: Alternately optimize G and D. During each round of G training, minimize DCL + adversarial loss (Wasserstein distance), and during D training, maximize the difference between the real sample score and the generated sample score.

[0198] Dynamic weight adjustment:

[0199] Initial phase (first 1000 rounds): DCL weight is 0.5, focusing on directional consistency;

[0200] Convergence phase (after 1000 rounds): DCL weights are reduced to 0.3, focusing on generation quality.

[0201] Example: In a drone tracking task, the foliage occlusion sample synthesized by the generator may initially deviate from the target's trajectory (error of 10 pixels). After DCL constraint, the error is reduced to within 2 pixels, and the occluder's movement direction is consistent with the target.

[0202] The generator trained adversarially enhances the optimized feature matrix that preserves temporal motion features by perturbating the features. The recognition feature vector with enhanced adversarial robustness is then extracted through Siamese network contrastive learning.

[0203] Feature perturbation enhancement and Siamese network contrastive learning work together to improve the model's robustness to occlusion and dynamic disturbances:

[0204] Enhanced characteristic perturbation:

[0205] Occlusion injection: Multi-scale occlusions are inserted into the optimized feature matrix using the trained generator, with the perturbation ratio controlled between 15% and 40%.

[0206] Random channel blocking: Some channels are randomly blocked with a probability of 0.2 (e.g., 13 out of 64 channels are blocked) to simulate sensor failure;

[0207] Spatiotemporal jitter: Randomly translate (±5 pixels) and rotate (±3°) the feature map to enhance spatial invariance.

[0208] Twin network architecture:

[0209] Backbone network: ResNet-50 (with the last two fully connected layers removed), outputting 1024-dimensional features;

[0210] Contrast loss: InfoNCE (noise contrast estimation) loss is used, with temperature parameter τ = 0.07;

[0211] Construction of positive and negative sample pairs:

[0212] Positive samples: Occluded and unoccluded samples of the same target;

[0213] Negative samples: feature vectors of different targets, or different occlusion patterns of the same target.

[0214] Training strategy:

[0215] Difficult sample mining: 256 samples are sampled in each batch, including 30% of high-difficulty samples (occlusion area > 25%);

[0216] Progressive training: In the initial stage, use light occlusion (10% coverage) and gradually increase it to 40%;

[0217] Feature distillation: The feature distribution of the teacher network (ResNet-50) is mimicked by constraining the student network (lightweight MobileNetV3) with KL divergence.

[0218] Example: In a face recognition task, the generator, after adversarial training, synthesizes occlusions such as glasses and masks. The Siamese network maps different occlusion states of the same person to the nearest neighbor region in the feature space (cosine similarity > 0.85) through contrastive learning, and the feature distance between different people is > 0.5.

[0219] Specifically, the identification feature vector is input into the adaptive decision engine, and the classification boundary is dynamically adjusted through a reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths based on the real-time scene complexity, including:

[0220] Based on the spatial distribution density of the identified feature vectors, a node density clustering algorithm is used to calculate the real-time scene complexity index.

[0221] The identification feature vector is a high-dimensional vector (e.g., 512-dimensional) output by the feature extraction network enhanced with adversarial robustness. Its spatial distribution density reflects the density and diversity of target objects in the scene. To quantify the complexity of real-time scenes, a node density clustering algorithm (such as DBSCAN) is used to perform cluster analysis on the feature vector.

[0222] Parameter settings: Neighborhood radius eps = 0.5 (Euclidean distance threshold), minimum number of samples min_samples = 5.

[0223] Density calculation: For each feature vector, calculate the number of samples in its neighborhood. For example, if there are 8 samples within 0.5 of a vector, it is marked as a core point; if there are only 3 samples, it is marked as a noise point.

[0224] Complexity index generation:

[0225] Number of clusters: If DBSCAN detects more than 5 clusters, it is marked as a high-complexity scenario (index > 0.7);

[0226] Noise percentage: When the noise percentage exceeds 20% (e.g., 200 out of 1000 samples are noise), the index increases by an additional 0.2.

[0227] Example: In an indoor monitoring scenario, if 8 clusters are detected and the noise accounts for 15%, the complexity index is 0.76. Wherein:

[0228] Basic rule settings

[0229] Cluster number threshold: 5 clusters; Noise ratio threshold: 20%; Exponential interval division: Low complexity: 0-0.5 (≤3 clusters); Medium complexity: 0.5-0.7 (4-5 clusters); High complexity: >0.7 (≥6 clusters);

[0230] Example Scenario Calculation Process

[0231] Detecting 8 clusters → exceeding the threshold of 5 clusters → basic index = 0.7 (starting value for high complexity);

[0232] Noise percentage 15% → below the 20% threshold → no additional triggering;

[0233] However, there are 3 more clusters than the threshold of 5 → each additional cluster increases the value by 0.02 → 0.7 + (3 × 0.02) = 0.76;

[0234] Complete calculation formula

[0235] Complexity index = base value (determined by the number of clusters) + cluster compensation (+0.02 for each cluster exceeding the threshold) + noise compensation (+0.05 for each 5% exceeding the threshold).

[0236] Design Principles

[0237] Each new cluster represents one more independent moving target / region in the scene;

[0238] Noise reflects the intensity of environmental disturbance;

[0239] A segmented accumulation system is used to ensure that the index ranges between 0 and 1.

[0240] This design considers both the number of targets (clusters) and environmental interference (noise), achieving refined quantification of complex metrics through a linear compensation mechanism. In practical applications, these parameters can be adjusted and optimized using a validation set based on specific scenarios.

[0241] Dynamic calibration mechanism:

[0242] Illumination Adaptive: Under low light conditions (such as nighttime infrared mode), the neighborhood range is expanded by adjusting eps to 0.8 to avoid cluster splitting caused by feature blurring;

[0243] Motion compensation: Apply time decay weights to the feature vectors of dynamic targets (such as pedestrians), with the weight of recent frames being 1.0 and the weight of frames 10 or older being reduced to 0.3, to reflect the instantaneous complexity of the scene.

[0244] A deep routing strategy based on scene complexity index-triggered graph neural network automatically selects shallow local subgraphs or deep global subgraphs for feature aggregation.

[0245] Graph Neural Networks (GNNs) employ a dynamic routing strategy, selecting subgraph structures based on the scene complexity index.

[0246] Shallow local subgraphs: suitable for low-complexity scenarios (exponent ≤ 0.5), aggregating only the target node and its 1-hop neighbors. For example, in simple face recognition scenarios, only local features of facial features (such as eye and nose nodes) are aggregated.

[0247] Example structure: 2-layer GAT (Graph Attention Network), with 4 attention heads per layer and an output dimension of 256.

[0248] Deep global subgraph: Used for highly complex scenarios (exponent > 0.5), aggregating multi-hop neighbors (e.g., 3-hop) and cross-regional association features. For example, in traffic monitoring, it is necessary to associate the interactions of multiple targets such as vehicles, pedestrians, and traffic lights.

[0249] Example structure: 5-layer GraphSAGE (number of sampled neighbors [10,5,3]), combined with gated recurrent units (GRU) to model temporal dependencies.

[0250] Routing trigger logic:

[0251] Threshold judgment: If the complexity index is >0.5, activate the deep subgraph; otherwise, use the shallow subgraph.

[0252] Hybrid mode: In the transition region (index 0.4 to 0.6), an adaptive hybrid routing is adopted, for example, 70% of traffic goes through the deep layer and 30% goes through the shallow layer.

[0253] Feature aggregation example:

[0254] Local aggregation: For a given vehicle node, aggregate its direct component features such as tires and headlights;

[0255] Global aggregation: Further correlates the movement trends of other vehicles and pedestrians in the same lane.

[0256] The classification decision boundary of nodes in shallow local subgraphs or deep global subgraphs is dynamically adjusted through a reinforcement learning reward mechanism. The reward function is determined by both classification confidence and feature distribution density.

[0257] The reinforcement learning (RL) framework uses GNN nodes as agents and decision boundary adjustments (such as hyperplane shifts in SVM classifiers) as the action space.

[0258] State space: node feature vector (512 dimensions), mean features of neighboring nodes, and scene complexity index.

[0259] Action space: Classification boundary offset (-0.1 to +0.1, step size 0.02).

[0260] Reward function design: R = 0.6 × classification confidence + 0.4 × (1 - feature distribution density).

[0261] Classification confidence: The maximum probability value output by Softmax (e.g., the probability that a node is classified as "pedestrian" is 0.92);

[0262] Feature distribution density: the Euclidean distance variance of features of the same type of nodes (e.g., the variance of 10 "vehicle" node features is 0.3, density = 0.7).

[0263] Training process:

[0264] Policy network: PPO (Proximal Policy Optimization) algorithm is used, with learning rate lr = 3e-4 and batch size = 64;

[0265] Exploration mechanism: ε-greedy policy (ε = 0.1), with a 10% probability of randomly selecting an action;

[0266] Example decision: In a congested scenario, a "vehicle" node can have its classification boundary adjusted from 0.5 to 0.6 using Regression Analysis (RL) to reduce the probability of it being falsely identified as an "obstacle".

[0267] Dynamic effect verification:

[0268] In low-complexity scenarios: the decision boundary adjustment range is small (±0.02), maintaining stability;

[0269] High-complexity scenarios: Allows for larger adjustments (±0.1) to cope with target occlusion and abrupt changes in appearance.

[0270] The optimized classification decision boundary is Gaussian smoothed, and the dynamic decision parameter matrix that adapts to the environment is output by combining the scene complexity index.

[0271] Gaussian smoothing is used to eliminate abrupt noise at decision boundaries and improve model robustness.

[0272] Gaussian kernel parameters: kernel size 5×5, standard deviation σ=1.5, weight matrix generated according to two-dimensional Gaussian distribution;

[0273] Smoothing process: Perform convolution filtering on the decision boundary parameter matrix. For example, if the original value of a boundary is [0.5, 0.6, 0.55], the smoothed value is [0.52, 0.58, 0.56].

[0274] Complexity-weighted: Dynamically adjust the smoothing strength based on the scene complexity exponent.

[0275] High complexity (exponent > 0.7): Enhances smoothing (σ = 2.0) and suppresses overfitting;

[0276] Low complexity (exponent ≤ 0.3): weak smoothness (σ = 0.5), preserving details.

[0277] Dynamic decision parameter matrix generation:

[0278] Matrix structure: rows represent target categories (e.g., 10 categories), columns represent feature dimensions (512 dimensions), and element values ​​are classification thresholds;

[0279] Example: In complex traffic scenarios, the threshold for the "pedestrian" category was reduced from 0.6 to 0.55 to improve detection recall.

[0280] Real-time deployment optimization:

[0281] Edge computing: The parameter matrix is ​​quantized and compressed using TensorRT, enabling real-time inference on NVIDIA Jetson AGX (latency <15ms);

[0282] Feedback mechanism: The classification accuracy is calculated every 30 seconds. If the accuracy drops by more than 5%, the model is fine-tuned (e.g., the Gaussian kernel parameters are updated).

[0283] Real-time decision-making example:

[0284] Input: Pedestrians, vehicles, and bicycles appear simultaneously in the surveillance footage;

[0285] Processing: Complexity index 0.75 → Activate deep subgraph → Adjust decision boundary using RL → Gaussian smoothing parameter σ = 1.8;

[0286] Output: All target classification accuracy > 90%, frame processing time 22ms.

[0287] This method achieves high-precision, low-latency visual recognition in complex scenarios by tightly coupling density clustering, dynamic routing, reinforcement learning, and adaptive smoothing.

[0288] Specifically, the process involves constructing a multi-scale verification pyramid based on the dynamic decision parameters, fusing the confidence scores of each level through a self-correction module, and outputting the final recognition result. The self-correction module employs an asymmetric loss function to eliminate cross-level feature conflicts, including:

[0289] Based on the scale sensitivity of the dynamic decision parameter matrix, a hierarchical random sampling strategy is adopted to construct a multi-scale validation pyramid and generate hierarchical confidence benchmarks.

[0290] The scale sensitivity of the dynamic decision parameter matrix reflects the stability of the classification boundary at different resolution levels. To construct a multi-scale validation pyramid, a ripple diffusion algorithm is first used to simulate the gradient distribution characteristics of the parameter matrix. This algorithm marks sensitive regions at different scales in the parameter matrix by simulating the water wave diffusion process.

[0291] Key parameters: diffusion radius (initial value 5 pixels, step size increases by a factor of 100) and attenuation coefficient (0.8~0.95);

[0292] Sensitivity calculation: Record the rate of change of parameters during the diffusion process. For example, in the target detection scene, the higher level (low resolution) is sensitive to the object category, while the lower level (high resolution) is sensitive to edge details.

[0293] Dynamic registration algorithms are used to extract feature basis vectors:

[0294] Basis vector extraction: Perform principal component analysis (PCA) on the parameter matrix at each scale level, retain the first K principal components (e.g., K=3), and generate orthogonal basis vectors;

[0295] Redundancy filtering: Basis vectors with high similarity are removed by using a cosine similarity threshold (e.g., 0.7) to ensure the independence of features at each scale. For example, in a traffic monitoring scenario, high-level basis vectors may correspond to the overall shape of a vehicle, while low-level ones may correspond to license plate textures.

[0296] An adaptive sliding window generator dynamically adjusts the window size based on scale sensitivity:

[0297] Window size rule: The higher the sensitivity, the smaller the window size (e.g., a sensitivity index of 0.9 corresponds to a 3×3 window, and 0.6 corresponds to a 7×7 window);

[0298] Neighborhood smoothing: A bilateral filtering algorithm (spatial standard deviation σ_s = 1.5, grayscale standard deviation σ_r = 0.1) is used to suppress noise while preserving edges.

[0299] Cross-scale confidence propagation combined with historical data optimizes the baseline value:

[0300] Propagation mechanism: High-level confidence levels serve as prior guidance for low-level calibration. For example, if a high-level system identifies a region as a "vehicle" (confidence level 0.85), the low-level system's confidence level for wheel details in the same region increases from 0.7 to 0.78.

[0301] Error tolerance threshold: dynamically adjusted based on historical recognition accuracy (e.g., when historical accuracy is 90%, the threshold is set to 0.15).

[0302] The final generated pyramid contains four levels (e.g., 32×32, 64×64, 128×128, 256×256), and the confidence benchmark for each level is stored in the form of a probability distribution, for example:

[0303] Level 1 (32×32): Vehicle category confidence level 0.82, pedestrian 0.15;

[0304] Level 4 (256×256): License plate character confidence level 0.91, vehicle body color 0.75.

[0305] The confidence scores of each level of the pyramid are aligned by bidirectional attention, and the score bias between scales is eliminated by feature similarity measurement to generate a consistent and calibrated confidence distribution.

[0306] The bidirectional attention alignment module includes two paths: top-down and bottom-up. It fuses multi-scale information through a cross-attention mechanism.

[0307] Top-down approach: High-level semantic information guides low-level detail calibration. For example, a region identified as a "truck" at a high level forces a higher confidence level in low-level wheel detection.

[0308] Bottom-up approach: Lower-level details correct higher-level misjudgments. For example, a lower-level system detects "broken car window" but a higher-level system fails to recognize it, triggering a reassessment of the confidence level in the higher-level system.

[0309] Feature similarity is calculated jointly using a modified Kullback-Leibler Divergence and cosine similarity:

[0310] KL divergence: measures the difference in confidence distributions across different levels. For example, the KL value between the "vehicles" distribution at level 1 and the "trucks" distribution at level 4 is 0.3; exceeding the threshold of 0.2 triggers calibration.

[0311] Cosine similarity: assesses the directional consistency of feature vectors. A threshold of 0.8 is set; values ​​below this are considered to indicate a conflict between scales.

[0312] Example of calibration process (medical imaging scenario):

[0313] High-level (low-resolution): The overall lung CT slices were judged as "normal" with a confidence level of 0.9;

[0314] Low-level (high-resolution): Confidence level of 0.65 for local nodule regions;

[0315] Two-way attention alignment: The confidence level of the high-level level drops to 0.7, while that of the low-level level rises to 0.8, triggering further checks.

[0316] The consistency calibration algorithm is implemented through iterative optimization:

[0317] Initialization: Confidence levels are loaded with their original values;

[0318] Attention weighting: Top-Down weight 0.6, Bottom-Up weight 0.4;

[0319] Residual compensation: Gaussian blur smoothing (σ=1.0) is applied to areas with calibration deviation >0.1.

[0320] The design incorporates an asymmetric loss function with directional constraints, and performs differentiated weighted fusion of high-level semantic confidence and low-level detail confidence to suppress cross-layer feature conflicts.

[0321] The fused confidence distribution is input into a multi-level weighted voting machine, which dynamically selects the optimal level result based on a preset threshold, and outputs the final recognition result and a traceable decision path.

[0322] The asymmetric loss function designs differentiated penalties for different level characteristics:

[0323] High-level semantic loss: Focusing on category accuracy, Focal Loss (γ = 2.0) is used to alleviate class imbalance;

[0324] Low-level detail loss: Focusing on positioning accuracy, IoU Loss (Intersection over Union) optimization is adopted with a weighting factor of 0.7;

[0325] Cross-layer conflict penalty: When the confidence directions of the higher and lower layers are opposite (e.g., the higher layer judges "true" and the lower layer judges "false"), an additional loss (coefficient 0.5) is applied.

[0326] Weighted fusion example (autonomous driving scenario):

[0327] High-level (road type): Confidence level 0.9 (highway);

[0328] Low level (lane lines): Confidence level 0.6 (dashed line);

[0329] Asymmetric weighting: high-level weight 0.7, low-level weight 0.3, and overall confidence level after fusion 0.81.

[0330] The multi-level weighted voting system employs a dynamic threshold mechanism:

[0331] Threshold preset: Set according to task type, such as 0.85 for safety-critical scenarios (medical, transportation) and 0.7 for normal scenarios;

[0332] Hierarchical voting: Each level is sorted by confidence level, and the top K (e.g., K=2) participate in the voting;

[0333] Decision path tracing: Recording the contribution of each level, for example:

[0334] Level 3 (128×128): Contribution 45% (wheel detected);

[0335] Level 4 (256×256): Contribution 35% (License plate detected);

[0336] Tier 1 (32×32): Contribution 20% (overall vehicle model matching).

[0337] Final output logic:

[0338] Confidence level met: Output the highest score result directly;

[0339] If the confidence levels are close (e.g., difference < 0.05): Initiate multimodal verification (e.g., call infrared feature secondary verification);

[0340] Conflicts cannot be resolved: Mark as "Pending manual review" and save evidence at all levels.

[0341] Application Case (Industrial Quality Inspection):

[0342] Input: Multi-scale images of the surface of a metal part;

[0343] Pyramid hierarchy:

[0344] Level 1: Overall shape is acceptable (0.88);

[0345] Level 4: Localized microcracks (0.65);

[0346] Asymmetric fusion: Crack confidence increased to 0.72 (weighted coefficient 0.6);

[0347] Voting results: Overall confidence level 0.78 < threshold 0.8 → marked as "suspicious", triggering high-precision 3D scan review.

[0348] Specifically, the step of constructing a multi-scale validation pyramid and generating hierarchical confidence benchmarks based on the scale sensitivity of the dynamic decision parameter matrix using a hierarchical random sampling strategy includes:

[0349] Based on the gradient distribution characteristics of the dynamic decision parameter matrix, the ripple diffusion algorithm is used to analyze the sensitivity of parameter changes at different scales, and a three-dimensional distribution map containing the scale-sensitivity mapping relationship is generated.

[0350] The gradient distribution characteristics of the dynamic decision parameter matrix reflect the degree of influence of features at different scales on classification decisions. The ripple diffusion algorithm quantifies the propagation range and intensity of parameter changes by simulating the physical process of water wave diffusion. The specific steps are as follows:

[0351] Gradient field construction: Calculate the Sobel gradient for the dynamic decision parameter matrix (e.g., a 512×512 matrix, where each element represents the decision weight of a pixel), generating the horizontal gradient Gx and the vertical gradient Gy. The gradient magnitude is calculated using the formula √(Gx / Gy). 2 +Gy 2 The threshold is set to 0.1 (gradients below this value are considered noise and do not participate in diffusion).

[0352] Ripple initialization: Initial ripple source points are generated at locations where the gradient magnitude is greater than a threshold. For example, if the gradient magnitude of a pixel is 0.3, then a source point with a ripple energy value of 0.3 is initialized at that coordinate.

[0353] Diffusion Process: An anisotropic diffusion model is used, with ripple energy propagating along the gradient direction. The diffusion step size is set to 3 pixels, and the attenuation coefficient is 0.8 (i.e., energy decreases by 20% with each diffusion step). For example, a ripple with initial energy of 0.3 will have its energy attenuated to 0.3 × 0.8 after 3 diffusion steps. 3 ≈0.153.

[0354] Scale sensitivity calculation: Statistically calculate the cumulative ripple energy value at different scales (e.g., 1×1, 3×3, 5×5 windows). For example, within a 5×5 window, if the cumulative energy value is 2.5, then the sensitivity is 2.5 / 25 = 0.1.

[0355] 3D distribution map generation: Scale (X-axis), spatial location (Y-axis, Z-axis), and sensitivity (color depth) are mapped to 3D volumetric data, and a visualized sensitivity distribution map is generated using the Marching Cubes algorithm. For example, sensitivity can reach 0.8 in the target edge region, while sensitivity is only 0.05 in flat regions.

[0356] Technical Example: In an industrial parts inspection scenario, the gradient amplitude of the dynamic decision parameter of a bolt edge is 0.4. The high-sensitivity region (>0.6) generated after the ripple diffusion accurately covers the thread structure, while the sensitivity of the background region is <0.1, effectively distinguishing key features from noise.

[0357] Based on the 3D distribution map, feature basis vectors at each scale are extracted through a dynamic registration algorithm, redundant basis vectors are filtered out using feature similarity measurement, and a set of scale orthogonal basis vectors is generated.

[0358] The dynamic registration algorithm aims to extract the most representative feature basis vectors from the 3D sensitivity distribution map and eliminate redundant information.

[0359] Multi-scale sampling: In the three-dimensional distribution map, cubic grids (voxel size 0.1×0.1×0.1) are divided according to scale levels (1×1, 3×3, 5×5, etc.), and sensitivity statistical features (mean, variance, peak value) are extracted within each grid.

[0360] Basis vector extraction: Principal component analysis (PCA) is performed on the grid features at each scale, and principal components with a cumulative contribution rate >85% are retained as basis vectors. For example, the feature dimension of a 5×5 scale is reduced from 100 dimensions to 15 dimensions.

[0361] Similarity measurement: Cosine similarity is used to calculate the correlation between basis vectors. If the similarity is >0.9, it is considered a redundant basis vector. For example, the similarity between a 3×3 basis vector A and a 5×5 basis vector B is 0.92, so B is discarded.

[0362] Orthogonalization: The retained basis vectors are orthogonally projected using the Gram-Schmidt orthogonalization algorithm to generate a set of scaled orthogonal basis vectors. For example, the original 20 basis vectors are filtered and orthogonalized to obtain 12 orthogonal bases.

[0363] Technical Example: In pedestrian detection tasks, 3×3 scale basis vectors mainly capture limb contours, while 5×5 scale basis vectors describe the overall posture. Similarity metrics reveal that the basis vectors of the two scales have a similarity of 0.88 in the torso region. Therefore, the redundant 5×5 scale basis vectors are removed, and the 3×3 scale basis vectors are retained to reduce computational complexity.

[0364] The set of scale-orthogonal basis vectors is input into the adaptive sliding window generator. A pyramid hierarchical structure is constructed through the nonlinear relationship between window size and scale sensitivity. At the same time, a neighborhood smoothing algorithm is used to eliminate noise interference.

[0365] An adaptive sliding window generator dynamically adjusts the window size based on scale sensitivity, constructing a multi-level pyramid structure:

[0366] Window size mapping: Defines the non-linear relationship between window size W and sensitivity S.

[0367] When S < 0.3, W = 3;

[0368] When 0.3 ≤ S < 0.6, W = 5;

[0369] When S≥0.6, W=7.

[0370] This mapping is implemented using a lookup table method to ensure real-time performance.

[0371] Pyramid hierarchy generation:

[0372] Low layer (L0): Original resolution (e.g., 1024×1024), window size 3×3;

[0373] Intermediate layer (L1): downsampled to 512×512, window size 5×5;

[0374] High-rise (L2): Downsampled to 256×256, window size 7×7.

[0375] Neighborhood smoothing algorithm: A bilateral filter (spatial domain σ = 1.5, grayscale domain σ = 0.1) is used to smooth the windows at each level, preserving edges while suppressing noise. For example, the sensitivity jump (0.2→0.8→0.3) caused by uneven lighting in a certain window becomes (0.3→0.7→0.4) after smoothing.

[0376] Technical Example:

[0377] In traffic monitoring scenarios, the taillight area has a sensitivity of up to 0.9, triggering a 7x7 window to capture the overall shape; while the license plate character area has a sensitivity of 0.5, using a 5x5 window to balance detail and efficiency. After bilateral filtering, the jagged noise at the license plate edges is effectively smoothed.

[0378] Cross-scale confidence propagation is performed for each pyramid level, and the statistical distribution characteristics of historical identification results are combined to generate a level confidence benchmark and error tolerance threshold.

[0379] Cross-scale confidence propagation improves decision reliability by integrating multi-level information:

[0380] Confidence propagation mechanism:

[0381] Bottom-up propagation: The local detail confidence (such as edge sharpness) of the bottom layer (L0) is weighted and passed to the top layer (L2) with a weight of 0.6.

[0382] Top-down feedback: The semantic confidence of the higher level (L2) (such as vehicle category) corrects the decision of the lower level (L0) in reverse, with a weight of 0.4.

[0383] Historical statistics fusion:

[0384] Establish a distribution histogram of historical recognition results (e.g., the frequency of the "vehicle" category is 30% and "pedestrian" is 15% in the past 100 frames);

[0385] Adjusting the current confidence level using the Bayesian update formula:

[0386]

[0387] The following is the meaning and technical function of each parameter in the formula:

[0388] P current Current confidence level

[0389] Definition: The initial confidence score obtained by the multi-scale validation pyramid for the current frame (or the current time), ranging from [0,1].

[0390] Technical function: It reflects the model's immediate confidence in the current recognition result. For example, in object detection, if the initial confidence level of classifying a certain region as "vehicle" is 0.85, it means that the model has an 85% certainty in determining that the target is a vehicle based on the current image features.

[0391] Calculation method: It is usually generated by the softmax output of the neural network or by post-processing (such as nonmaximum suppression).

[0392] P history Historical statistical probability

[0393] Definition: Prior probability obtained statistically based on historical data (such as the recognition results of the past N frames), ranging from [0,1].

[0394] Technical function: It introduces contextual information in the time dimension to correct misjudgments caused by instantaneous noise (such as sudden changes in lighting or motion blur). For example, if the average probability of the "vehicle" category in historical data is 30%, the current high confidence result (such as 0.9) may be lowered due to scene inconsistency (such as detecting a vehicle in the sky).

[0395] Calculation method: A sliding window is used to statistically analyze the frequency distribution of historical recognition results. For example, if the "vehicle" category appears 30 times in the past 100 frames, then P... history =0.3.

[0396] P new : Updated confidence level

[0397] Definition: Posterior probability after fusing the current confidence level and historical probability, ranging from [0,1].

[0398] Technical application: Using Bayes' theorem, instantaneous observations (P...) current ) and long-term statistics (P) history Combining these parameters, a more robust confidence level is output. For example:

[0399] If the current confidence level is high (0.9) but the historical probability is low (0.1), the updated confidence level will be significantly reduced (e.g., 0.47) to avoid false positives; if the two are consistent (e.g., both are 0.8), the updated confidence level will remain high (0.8).

[0400] Property: When P history When P = 0.5 (historical unbiased), new =P current (History does not affect the current outcome); when P current When the value is 1 or 0, it also approaches 1 or 0 (extreme confidence levels are not affected by history).

[0401] Technical Examples

[0402] In an industrial quality inspection scenario:

[0403] Current frame: Initial confidence level P for classifying a part as a "defect". current =0.8;

[0404] Historical statistics: Frequency P of "defects" in the past 1000 frames history =0.05 (The defect rate is low on a normal production line).

[0405] The updated confidence level is: P new =0.74.

[0406] Results: Although the current model gives a high confidence level (0.8), the final confidence level was significantly reduced to 17.4% due to the extremely low historical defect rate (5%), which effectively suppressed false alarms.

[0407] Error tolerance threshold calculation:

[0408] Based on the root mean square error (σ) of the confidence distribution, a threshold of μ ± 2σ is set (covering the 95% confidence interval);

[0409] Dynamic adjustment mechanism: If the threshold trigger rate is >90% for 10 consecutive frames, then σ is multiplied by 0.8 to tighten the threshold.

[0410] Technical Example:

[0411] In a drone inspection scenario, a task involving the detection of cracks in a photovoltaic panel is as follows:

[0412] Lowest layer (L0): Edge confidence 0.85 (high detail confidence);

[0413] High level (L2): Semantic confidence score 0.75 (slightly lower due to changes in illumination);

[0414] After cross-layer propagation, the overall confidence level = 0.85 × 0.6 + 0.75 × 0.4 = 0.81;

[0415] Historical statistics show that the average crack detection rate in sunny weather is μ = 0.82 and σ = 0.05, so the threshold is set to [0.72, 0.92]. The current result of 0.81 is within the threshold and is therefore considered valid.

[0416] As can be seen, by acquiring visible light images, infrared thermal imaging, and depth point cloud data of the target scene, a spatiotemporally consistent multimodal heterogeneous feature tensor is generated. This multimodal heterogeneous feature tensor is then input into a spatial frequency-aware optimizer to generate a detail-enhanced optimized feature matrix. Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples, generating adversarially robust recognition feature vectors. These feature vectors are then input into an adaptive decision engine to generate environment-adaptive dynamic decision parameters. A multi-scale verification pyramid is constructed based on these dynamic decision parameters, and the confidence scores from each level are fused through a self-correction module to output the final recognition result, thereby achieving highly robust and accurate visual recognition.

[0417] Another embodiment of the present invention provides a visual recognition system, see [link to relevant documentation]. Figure 3 The system may include:

[0418] The acquisition module 301 is used to acquire visible light images, infrared thermal images and depth point cloud data of the target scene. It performs cross-domain alignment of multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network adopts a self-attention mechanism to fuse edge response features of different spectral frequency bands.

[0419] Enhancement module 302 is used to input the multimodal heterogeneous feature tensor into a spatial frequency-aware optimizer, reconstruct the feature map through an adaptive high-frequency enhancement filter, and generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity.

[0420] The generation module 303 is used to synthesize multi-scale occlusion samples based on the optimized feature matrix using an adversarial generative network, and generate an adversarial robustness-enhanced recognition feature vector by combining a temporal consistency constraint. The temporal consistency constraint forces the preservation of temporal feature continuity through an optical flow field loss function.

[0421] The adjustment module 304 is used to input the recognition feature vector into the adaptive decision engine, and dynamically adjust the classification boundary through the reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene.

[0422] The identification module 305 is used to construct a multi-scale verification pyramid based on the dynamic decision parameters, fuse the confidence scores of each level through the self-correction module, and output the final identification result. The self-correction module uses an asymmetric loss function to eliminate cross-level feature conflicts.

[0423] As can be seen, by acquiring visible light images, infrared thermal imaging, and depth point cloud data of the target scene, a spatiotemporally consistent multimodal heterogeneous feature tensor is generated. This multimodal heterogeneous feature tensor is then input into a spatial frequency-aware optimizer to generate a detail-enhanced optimized feature matrix. Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples, generating adversarially robust recognition feature vectors. These feature vectors are then input into an adaptive decision engine to generate environment-adaptive dynamic decision parameters. A multi-scale verification pyramid is constructed based on these dynamic decision parameters, and the confidence scores from each level are fused through a self-correction module to output the final recognition result, thereby achieving highly robust and accurate visual recognition.

[0424] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0425] Specifically, in this embodiment, the storage medium can be configured to store a computer program for performing the following steps:

[0426] S201, acquire visible light images, infrared thermal imaging and depth point cloud data of the target scene, and perform cross-domain alignment of multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network adopts a self-attention mechanism to fuse edge response features of different spectral frequency bands.

[0427] S202, the multimodal heterogeneous feature tensor is input into the spatial frequency-aware optimizer, and the feature map is reconstructed through an adaptive high-frequency enhancement filter to generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity.

[0428] S203, Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples. Combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraints are used to force the preservation of temporal feature continuity through an optical flow field loss function.

[0429] S204, the identification feature vector is input into the adaptive decision engine, and the classification boundary is dynamically adjusted through the reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene.

[0430] S205, construct a multi-scale verification pyramid based on the dynamic decision parameters, fuse the confidence scores of each level through a self-correction module, and output the final recognition result. The self-correction module uses an asymmetric loss function to eliminate cross-layer feature conflicts.

[0431] As can be seen, by acquiring visible light images, infrared thermal imaging, and depth point cloud data of the target scene, a spatiotemporally consistent multimodal heterogeneous feature tensor is generated. This multimodal heterogeneous feature tensor is then input into a spatial frequency-aware optimizer to generate a detail-enhanced optimized feature matrix. Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples, generating adversarially robust recognition feature vectors. These feature vectors are then input into an adaptive decision engine to generate environment-adaptive dynamic decision parameters. A multi-scale verification pyramid is constructed based on these dynamic decision parameters, and the confidence scores from each level are fused through a self-correction module to output the final recognition result, thereby achieving highly robust and accurate visual recognition.

[0432] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0433] Specifically, the aforementioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the aforementioned processor, and the input / output device is connected to the aforementioned processor.

[0434] Specifically, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0435] S201, acquire visible light images, infrared thermal imaging and depth point cloud data of the target scene, and perform cross-domain alignment of multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network adopts a self-attention mechanism to fuse edge response features of different spectral frequency bands.

[0436] S202, the multimodal heterogeneous feature tensor is input into the spatial frequency-aware optimizer, and the feature map is reconstructed through an adaptive high-frequency enhancement filter to generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity.

[0437] S203, Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples. Combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraints are used to force the preservation of temporal feature continuity through an optical flow field loss function.

[0438] S204, the identification feature vector is input into the adaptive decision engine, and the classification boundary is dynamically adjusted through the reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene.

[0439] S205, construct a multi-scale verification pyramid based on the dynamic decision parameters, fuse the confidence scores of each level through a self-correction module, and output the final recognition result. The self-correction module uses an asymmetric loss function to eliminate cross-layer feature conflicts.

[0440] As can be seen, by acquiring visible light images, infrared thermal imaging, and depth point cloud data of the target scene, a spatiotemporally consistent multimodal heterogeneous feature tensor is generated. This multimodal heterogeneous feature tensor is then input into a spatial frequency-aware optimizer to generate a detail-enhanced optimized feature matrix. Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples, generating adversarially robust recognition feature vectors. These feature vectors are then input into an adaptive decision engine to generate environment-adaptive dynamic decision parameters. A multi-scale verification pyramid is constructed based on these dynamic decision parameters, and the confidence scores from each level are fused through a self-correction module to output the final recognition result, thereby achieving highly robust and accurate visual recognition.

[0441] The above description, based on the embodiments shown in the figures, details the structure, features, and effects of the present invention. The above description is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or equivalent embodiments modified to have equivalent changes, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A visual recognition method, characterized in that, The method includes: The system acquires visible light images, infrared thermal images, and depth point cloud data of the target scene. It then performs cross-domain alignment of the multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network uses a self-attention mechanism to fuse edge response features from different spectral frequency bands. The multimodal heterogeneous feature tensor is input into a spatial frequency-aware optimizer, and the feature map is reconstructed through an adaptive high-frequency enhancement filter to generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity. Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples. Combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraints are used to force the preservation of temporal feature continuity through an optical flow field loss function. The identified feature vector is input into the adaptive decision engine, and the classification boundary is dynamically adjusted through a reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene. A multi-scale verification pyramid is constructed based on the dynamic decision parameters. The confidence scores of each level are fused through a self-correction module to output the final recognition result. The self-correction module uses an asymmetric loss function to eliminate cross-level feature conflicts.

2. The method according to claim 1, characterized in that, The process involves acquiring visible light images, infrared thermal images, and depth point cloud data of the target scene. A heterogeneous feature fusion network is then used to perform cross-domain alignment of the multimodal data, generating a spatiotemporally consistent multimodal heterogeneous feature tensor. This heterogeneous feature fusion network employs a self-attention mechanism to fuse edge response features from different spectral bands, including: Based on the frame synchronization signal of the visible light image and the timestamp of the infrared thermal image, a dynamic interpolation compensation algorithm is used to align the sampling interval of the depth point cloud data to generate a three-modal data stream with strict time synchronization. Multispectral edge detection is performed on visible light images, and the temperature gradient of infrared thermal imaging is used to correct the visible light edge fracture region. At the same time, the curvature change feature of depth point cloud is extracted to generate a cross-modal edge response map. The cross-modal edge response map is input into a dual-path self-attention network, and weights are dynamically allocated through a visible-infrared feature cross-calibration module to generate a spectrally consistent edge feature matrix. The fused edge feature matrix is ​​subjected to 3D spatiotemporal convolution, and combined with the spatial topological constraints of the depth point cloud, the spatiotemporally consistent multimodal heterogeneous feature tensor is output.

3. The method according to claim 2, characterized in that, The step of inputting the multimodal heterogeneous feature tensor into a spatial frequency-aware optimizer and reconstructing the feature map through an adaptive high-frequency enhancement filter to generate an optimized feature matrix with enhanced details includes: The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity. Based on the brightness histogram distribution of the visible light image, a dynamic threshold segmentation algorithm is used to extract the scene illumination intensity level and generate an illumination intensity quantization index. Two-dimensional wavelet decomposition is performed on the multimodal heterogeneous feature tensor to separate high-frequency detail components from low-frequency contour components. The convolution kernel weight ratio of the high-frequency detail components is dynamically adjusted based on the illumination intensity quantization index. The weighted high-frequency detail components and low-frequency contour components are fused using adaptive inverse wavelet transform, and the reconstruction error is compensated by residual connection to generate an optimized feature matrix with enhanced details. Local contrast analysis is performed on the optimized feature matrix, and a nonlinear filtering algorithm is used to suppress artifact noise caused by sudden changes in illumination. The final optimized feature matrix after noise suppression is then output.

4. The method according to claim 3, characterized in that, Based on the optimized feature matrix, an adversarial generative network is used to synthesize multi-scale occlusion samples. Combined with temporal consistency constraints, an adversarial robustness-enhanced recognition feature vector is generated. The temporal consistency constraints are enforced by preserving the continuity of temporal features through an optical flow field loss function, including: The optimized feature matrix is ​​input into the morphological erosion generator, and the edge regions of the feature map are eroded by random structuring elements to synthesize a multi-scale occlusion sample set. Based on the feature matrix optimized by adjacent frames, a dense optical flow estimation algorithm is used to extract the motion trajectory of the sample occlusion area and generate a spatiotemporal continuity constraint vector. By combining the spatiotemporal continuity constraint vector of optical flow trajectory with a multi-scale occlusion sample set, a direction consistency loss function is designed to force the adversarial generative network to retain temporal motion features when synthesizing samples. The generator trained adversarially enhances the optimized feature matrix that preserves temporal motion features by perturbating the features. The recognition feature vector with enhanced adversarial robustness is then extracted through Siamese network contrastive learning.

5. The method according to claim 4, characterized in that, The process involves inputting the identified feature vector into an adaptive decision engine, dynamically adjusting the classification boundary through a reinforcement learning-driven graph neural network, and generating dynamic decision parameters that adapt to the environment. The graph neural network automatically activates subgraph structures of different depths based on the real-time scene complexity, including: Based on the spatial distribution density of the identified feature vectors, a node density clustering algorithm is used to calculate the real-time scene complexity index. A deep routing strategy based on scene complexity index-triggered graph neural network automatically selects shallow local subgraphs or deep global subgraphs for feature aggregation. The classification decision boundary of nodes in shallow local subgraphs or deep global subgraphs is dynamically adjusted through a reinforcement learning reward mechanism. The reward function is determined by both classification confidence and feature distribution density. The optimized classification decision boundary is Gaussian smoothed, and the dynamic decision parameter matrix that adapts to the environment is output by combining the scene complexity index.

6. The method according to claim 5, characterized in that, The process involves constructing a multi-scale verification pyramid based on the dynamic decision parameters, fusing the confidence scores of each level through a self-correction module, and outputting the final recognition result. The self-correction module employs an asymmetric loss function to eliminate cross-level feature conflicts, including: Based on the scale sensitivity of the dynamic decision parameter matrix, a hierarchical random sampling strategy is adopted to construct a multi-scale validation pyramid and generate hierarchical confidence benchmarks. The confidence scores of each level of the pyramid are aligned by bidirectional attention, and the score bias between scales is eliminated by feature similarity measurement to generate a consistent and calibrated confidence distribution. The design incorporates an asymmetric loss function with directional constraints, and performs differentiated weighted fusion of high-level semantic confidence and low-level detail confidence to suppress cross-layer feature conflicts. The fused confidence distribution is input into a multi-level weighted voting machine, which dynamically selects the optimal level result based on a preset threshold, and outputs the final recognition result and a traceable decision path.

7. The method according to claim 6, characterized in that, The process of constructing a multi-scale validation pyramid based on the scale sensitivity of the dynamic decision parameter matrix using a hierarchical random sampling strategy to generate hierarchical confidence benchmarks includes: Based on the gradient distribution characteristics of the dynamic decision parameter matrix, the ripple diffusion algorithm is used to analyze the sensitivity of parameter changes at different scales, and a three-dimensional distribution map containing the scale-sensitivity mapping relationship is generated. Based on the 3D distribution map, feature basis vectors at each scale are extracted through a dynamic registration algorithm, redundant basis vectors are filtered out using feature similarity measurement, and a set of scale orthogonal basis vectors is generated. The set of scale-orthogonal basis vectors is input into the adaptive sliding window generator. A pyramid hierarchical structure is constructed through the nonlinear relationship between window size and scale sensitivity. At the same time, a neighborhood smoothing algorithm is used to eliminate noise interference. Cross-scale confidence propagation is performed for each pyramid level, and the statistical distribution characteristics of historical identification results are combined to generate a level confidence benchmark and error tolerance threshold.

8. A visual recognition system, characterized in that, The system includes: The acquisition module is used to acquire visible light images, infrared thermal images and depth point cloud data of the target scene. It performs cross-domain alignment of multimodal data through a heterogeneous feature fusion network to generate a spatiotemporally consistent multimodal heterogeneous feature tensor. The heterogeneous feature fusion network adopts a self-attention mechanism to fuse edge response features of different spectral frequency bands. The enhancement module is used to input the multimodal heterogeneous feature tensor into the spatial frequency-aware optimizer, reconstruct the feature map through an adaptive high-frequency enhancement filter, and generate an optimized feature matrix with enhanced details. The filter dynamically adjusts the frequency band weight distribution of the convolution kernel according to the illumination intensity. The generation module is used to synthesize multi-scale occlusion samples based on the optimized feature matrix using an adversarial generative network, and generate an adversarial robustness-enhanced recognition feature vector by combining a temporal consistency constraint. The temporal consistency constraint forces the preservation of temporal feature continuity through an optical flow field loss function. The adjustment module is used to input the recognition feature vector into the adaptive decision engine, and dynamically adjust the classification boundary through the reinforcement learning-driven graph neural network to generate dynamic decision parameters that are adaptive to the environment. The graph neural network automatically activates subgraph structures of different depths according to the complexity of the real-time scene. The identification module is used to construct a multi-scale verification pyramid based on the dynamic decision parameters, and to fuse the confidence scores of each level through the self-correction module to output the final identification result. The self-correction module uses an asymmetric loss function to eliminate cross-level feature conflicts.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1-7 when it is run.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1-7.

Images