A target recognition method and system based on deep learning

Abstract

The application relates to the field of artificial intelligence and discloses a target recognition method and system based on deep learning, which comprises the following steps: acquiring visible light and infrared images of the same scene, generating illumination invariance features through an adversarial training network; modeling the feature sequence by using a space-time joint network to generate an initial recognition result; performing space-time verification on the initial recognition results of continuous multiple frames by using a Markov random field model to generate a calibrated recognition result; and performing online model calibration on the space-time joint network based on the correction deviation between the initial recognition result and the calibrated recognition result. The application also provides a recognition system for executing the method. Through the synergistic mechanism of multi-modal feature decoupling, space-time joint modeling and verification, and online feedback calibration, the application effectively improves the all-weather adaptability, recognition accuracy and trajectory continuity of the system in complex scenes such as illumination changes and target occlusions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a target recognition method and system based on deep learning. Background Technology

[0002] In recent years, deep learning-based target recognition technology has been widely used in fields such as security monitoring, autonomous driving, and robot vision, and its recognition accuracy under ideal conditions has reached or even surpassed human levels. However, when these advanced algorithm models move from the laboratory to complex and ever-changing real-world applications, their performance and reliability still face severe challenges.

[0003] Existing recognition systems, especially those relying on a single visible light sensor, are highly dependent on ambient lighting conditions. Image quality deteriorates significantly during day-night cycles, in strong light, in shadow, or in adverse weather conditions such as rain, snow, or fog. This leads to unstable features extracted by the model, resulting in a sharp decline in recognition accuracy and making it difficult to meet the requirements for reliable operation in all weather conditions. Although some solutions attempt to fuse data from other modalities, such as infrared, this often involves simple feature stitching and fails to fundamentally decouple stable and reliable shared features independent of lighting changes, thus limiting improvements in environmental adaptability.

[0004] Furthermore, for object recognition tasks in video streams, it is not only necessary to accurately identify objects in each frame, but also to understand their continuity and dynamic relationships in the temporal dimension. Existing methods often struggle to effectively capture complex interactions, occlusions, and long-term motion patterns between objects when processing temporal information, resulting in a lack of temporal consistency in the output. This leads to problems such as object flickering, frequent ID switching, or loss of objects after brief occlusion, affecting the reliability of downstream tasks (such as tracking and behavior analysis).

[0005] More importantly, deep learning models are typically trained offline on large-scale, static datasets. Once deployed to edge devices, these models face the problem of "domain drift," where the data distribution in the actual application scenario differs from the training data. This difference evolves continuously with changes in time, location, and perspective, leading to a gradual decline in model performance. Traditional solutions involve periodically transmitting the data collected by the device back to the cloud for costly and complete retraining. This is not only time-consuming and expensive but also fails to meet the system's real-time adaptive requirements for environmental changes. Therefore, enabling edge-deployed recognition models to possess low-overhead, high-efficiency online adaptive and continuous learning capabilities, while avoiding the forgetting of old knowledge when learning new knowledge (i.e., "catastrophic forgetting"), is a pressing challenge in current technological development. Summary of the Invention

[0006] The technical problem to be solved by this invention is that existing deep learning-based target recognition methods have defects such as decreased recognition performance due to changes in ambient lighting when applied to complex industrial scenarios, catastrophic forgetting of models when learning new categories of knowledge, low computational efficiency of high-precision models on edge devices, and drift in the positioning results of moving targets.

[0007] To address the aforementioned technical problems, this invention provides a target recognition method and system based on deep learning.

[0008] The first aspect of this invention provides a target recognition method based on deep learning, comprising the following steps:

[0009] Step 1: Acquire visible light and long-wave infrared images of the same scene.

[0010] Step 2: Based on the visible light image and the long-wave infrared image, generate an illumination invariant feature tensor.

[0011] In one specific implementation, this step is achieved through a feature extractor and a modality discriminator.

[0012] The feature extractor processes the visible light image and the long-wave infrared image respectively and maps them to a shared feature space to generate shared features;

[0013] The modality discriminator is used to determine the image source of the shared features.

[0014] By using adversarial training, minimize a task-related loss. and the loss of combat Composite loss function ,in:

[0015] ;

[0016] The resistance loss The feature extractor generates shared features that the modality discriminator cannot distinguish from their source; these shared features are the illumination invariant feature tensors.

[0017] Step 3: Input the illumination invariant feature tensor into the spatiotemporal joint network to generate an initial recognition result containing the target location and category.

[0018] In one specific implementation, the spatiotemporal joint network includes a three-dimensional convolutional branch and a graph attention network branch.

[0019] The three-dimensional convolutional branch is used to extract the multi-scale dynamic geometric features of the target in the illumination-invariant feature tensor;

[0020] The graph attention network branch is used to model the spatial topological relationships between targets in the illumination invariant feature tensor.

[0021] The spatiotemporal joint network may also include a deformable interactive module, which adaptively adjusts the fusion receptive field according to the output features of the three-dimensional convolutional branch and the output features of the graph attention network branch, dynamically fuses the output features of the two branches to form a joint feature expression, and generates the initial recognition result based on the joint feature expression.

[0022] Step 4: Perform spatiotemporal verification on the initial recognition results of multiple consecutive frames to generate calibrated recognition results.

[0023] In one specific implementation, this step is achieved by constructing a spatiotemporal graphical model and utilizing Markov random fields.

[0024] The initial recognition results from multiple consecutive frames are used as nodes in the spatiotemporal graph model, and a global energy function is minimized. To find the optimal location sequence, where:

[0025] ;

[0026] In the formula, For the first The first frame The initial identification results of the targets, univariate potential function The binary potential function characterizes the confidence level of the initial recognition result of a single frame. This characterizes the motion continuity between the initial recognition results of adjacent frames. The result after minimizing energy is the calibrated recognition result.

[0027] Step 5: Based on the correction deviation between the initial identification result and the calibrated identification result, perform online model calibration on the spatiotemporal joint network.

[0028] In one specific implementation, this step treats the target position of the calibrated identification result as a supervision signal, calculates the deviation between it and the target position of the initial identification result, and uses this deviation as a calibration loss. .

[0029] This only applies to the top-level parameters responsible for localization regression in the spatiotemporal joint network. A single-step gradient descent update is performed using the calibration loss:

[0030] ;

[0031] In the formula, The preset learning rate is used. This step is performed in real time at the edge to compensate for the systematic positioning bias of the spatiotemporal joint network.

[0032] In an optional implementation, the method further includes the dynamic evolution of the spatiotemporal joint network, which includes:

[0033] The quality of the initial identification result is continuously evaluated by a real-time confidence entropy monitor; when the quality is lower than a preset threshold, an incremental learning process is triggered.

[0034] The incremental learning process is as follows:

[0035] The feature reconstruction module is used to generate feature proxies that represent the distribution of historical category data.

[0036] The update gradient of new knowledge is calculated by combining the feature agent with the new category data. ;

[0037] The update gradient Projected onto the parameter subspace used to store historical knowledge. On the orthogonal complement space, obtain the orthogonal update gradient. :

[0038] ;

[0039] And update the gradient using the orthogonal method. Update the parameters of the spatiotemporal joint network.

[0040] After the incremental learning process is completed, a hardware-aware neural architecture search can be used to automatically generate a simplified computation graph for the updated spatiotemporal joint network based on the computing power and memory constraints of the target edge device, and then optimized through operator fusion and sparse compilation.

[0041] A second aspect of the present invention provides a target recognition system based on deep learning, comprising:

[0042] An environment perception module is used to acquire visible light images and long-wave infrared images of the same scene, and generate an illumination invariant feature tensor based on the images;

[0043] The spatiotemporal modeling module, connected to the environment perception module, is used to receive the illumination invariant feature tensor and generate an initial recognition result containing the target location and category through the spatiotemporal joint network.

[0044] A spatiotemporal verification module, connected to the spatiotemporal modeling module, is used to perform spatiotemporal verification on the initial recognition results of multiple consecutive frames to generate calibrated recognition results;

[0045] An online calibration module, connected to the spatiotemporal modeling module and the spatiotemporal verification module, is used to perform online model calibration of the spatiotemporal joint network based on the correction deviation between the initial identification result and the calibrated identification result.

[0046] This invention provides a target recognition method and system based on deep learning. It has the following beneficial effects:

[0047] 1. This invention exhibits excellent all-weather environmental adaptability and robust recognition performance, primarily due to its environmental perception module employing an adversarial training network. By jointly processing and decoupling features from visible light and infrared images, it can extract illumination-invariant features. This design effectively overcomes the image quality degradation caused by drastic changes in illumination and severe weather, ensuring stable recognition performance of the system in complex scenarios.

[0048] 2. This invention utilizes parallel 3D convolutional and graph attention networks in the spatiotemporal modeling module, enabling the system to not only capture local motion information of targets but also model global relationships between multiple targets. Furthermore, the spatiotemporal verification module employs Markov random fields to globally optimize the results across multiple consecutive frames, effectively smoothing and correcting instantaneous errors, especially when targets are occluded or interact with each other, ensuring the continuity and stability of target trajectories.

[0049] 3. The online calibration module of this invention uses the high-confidence results output by the spatiotemporal verification module as a guiding signal to perform real-time model fine-tuning of the spatiotemporal modeling network. This design can quickly compensate for performance drift caused by scene changes without interrupting system operation or performing expensive offline retraining, enabling the system to continuously maintain optimal working state. Attached Figure Description

[0050] Figure 1 This is a structural block diagram of a deep learning-based target recognition system according to an embodiment of the present invention;

[0051] Figure 2 This is a flowchart of a deep learning-based target recognition method according to an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the structure of an environmental sensing module according to an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the spatiotemporal modeling module according to an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the model dynamic evolution and edge adaptation process in an optional embodiment of the present invention.

[0055] Among them, 10 is the edge device; 11 is the data acquisition unit; 12 is the edge processing unit; 13 is the communication interface; 20 is the cloud server; 21 is the cloud processing unit; 22 is the model storage unit; 110 is the environmental perception module; 120 is the spatiotemporal modeling module; 130 is the spatiotemporal verification module; and 140 is the online calibration module. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way.

[0057] See attached document Figure 1 , Figure 1 This is a structural block diagram of a deep learning-based target recognition system according to an embodiment of the present invention. The present invention provides a deep learning-based target recognition system.

[0058] The system includes: an environmental perception module 110, a spatiotemporal modeling module 120, a spatiotemporal verification module 130, and an online calibration module 140.

[0059] The environment perception module 110 is used to acquire visible light images and long-wave infrared images of the same scene, and generate an illumination invariant feature tensor based on the images.

[0060] The spatiotemporal modeling module 120 is connected to the environment perception module 110 and is used to receive the illumination invariant feature tensor and generate an initial recognition result containing the target location and category through the spatiotemporal joint network.

[0061] The spatiotemporal verification module 130 is connected to the spatiotemporal modeling module 120 and is used to perform spatiotemporal verification on the initial recognition results of multiple consecutive frames in order to generate calibrated recognition results.

[0062] The online calibration module 140 is connected to the spatiotemporal modeling module 120 and the spatiotemporal verification module 130, and is used to perform online model calibration of the spatiotemporal joint network based on the correction deviation between the initial identification results and the calibrated identification results.

[0063] In one specific embodiment, the system can be implemented using the hardware environment of an edge device 10 and a cloud server 20. The edge device 10 and the cloud server 20 are connected via a communication network.

[0064] The edge device 10 may include a data acquisition unit 11, an edge processing unit 12, and a communication interface 13.

[0065] The environmental perception module 110, the spatiotemporal modeling module 120, the spatiotemporal verification module 130, and the online calibration module 140 can be software modules or functional units implemented by the edge processing unit 12 by executing program code embedded in its internal memory.

[0066] Specifically, the data acquisition unit 11 can integrate a visible light image sensor and a long-wave infrared image sensor. The environmental perception module 110 controls the data acquisition unit 11 to acquire the required visible light image and long-wave infrared image, and transmits the image data to the edge processing unit 12 for subsequent processing.

[0067] The edge processing unit 12 can be a central processing unit (CPU), a graphics processing unit (GPU), or an application-specific integrated circuit (ASIC), which executes corresponding program instructions to sequentially implement the specific functions defined by the environment perception module 110, the spatiotemporal modeling module 120, the spatiotemporal verification module 130, and the online calibration module 140.

[0068] In another optional implementation, the system may further include a model evolution module. This model evolution module may be implemented by a cloud server 20. The cloud server 20 includes a cloud processing unit 21 and a model repository 22. When the monitoring function on the edge device 10 triggers the model evolution process, relevant data can be uploaded to the cloud server 20 through the communication interface 13 and the communication network. The cloud processing unit 21 performs incremental learning and model optimization, stores the optimized new model in the model repository 22, and finally sends it back to the edge device 10.

[0069] See attached document Figure 2 , Figure 2 This is a flowchart of a deep learning-based target recognition method according to an embodiment of the present invention. In a specific embodiment, the target recognition method provided by the present invention can be executed in a system including an edge device 10 and a cloud server 20. The method may include the following steps:

[0070] Step S100 involves performing cross-modal environment perception and feature decoupling. This step is executed on the edge device 10, which acquires visible light and long-wave infrared images of the same scene and processes the images using a specific adversarial training network model to generate an illumination-invariant feature tensor. This feature tensor eliminates modal differences introduced by environmental factors such as illumination and smoke.

[0071] Step S200: Perform spatiotemporal joint modeling and initial identification. This step is executed on the edge device 10, which inputs the illumination-invariant feature tensor generated in step S100 into the spatiotemporal joint network. The spatiotemporal joint network performs joint analysis of the feature tensor in both temporal and spatial dimensions to generate initial identification results containing the target location and category.

[0072] Step S300: Perform spatiotemporal verification. This step is performed on the edge device 10, which receives the initial recognition results of multiple consecutive frames from step S200, establishes the spatiotemporal correlation between these results through a Markov random field model, and performs global optimization to correct the positioning error of a single frame, thereby generating a calibrated recognition result.

[0073] Step S400 involves performing verification-guided online model calibration. This step is performed on the edge device 10, which calculates the correction deviation between the initial identification result generated in step S200 and the calibrated identification result generated in step S300. This correction deviation is then used as a monitoring signal to fine-tune some parameters of the spatiotemporal joint network in step S200 online.

[0074] Steps S100 to S400 constitute a closed-loop processing flow that runs in real time on the edge device 10. The online model calibration result of step S400 will directly affect the processing of subsequent frames in step S200, thereby forming a fast feedback and correction loop.

[0075] In an optional implementation, the method may further include step S500, performing dynamic model evolution and edge adaptation. This step is completed collaboratively by the edge device 10 and the cloud server 20. The edge device 10 continuously evaluates the confidence level of the initial recognition result generated in step S200 using a confidence entropy monitor. When the confidence level falls below a preset threshold, an incremental learning process is triggered on the cloud server 20. After this process is completed, an updated model file is generated and distributed to the edge device 10 to update the spatiotemporal joint network used in step S200.

[0076] See attached document Figure 3 , Figure 3 This is a schematic diagram of the structure of an environmental sensing module according to an embodiment of the present invention. The execution process of step S100 is described in detail below.

[0077] In step S100, firstly, the environment perception module 110 controls the data acquisition unit 11 to synchronously acquire visible light images of the same scene. and long-wave infrared images To ensure the effectiveness of subsequent feature-level fusion, the module performs spatial registration preprocessing on the acquired image pairs. This process calculates an affine transformation matrix based on pre-calibrated camera intrinsic and extrinsic parameters to correct for rotation, scaling, and translation differences between the two modal images, aligning the same physical point in the pixel coordinate system.

[0078] After preprocessing, the environment perception module 110 processes the aligned image pairs using an adversarial network model integrated into the edge processing unit 12. This network model includes a feature extractor. and a modal discriminator .

[0079] Feature extractor It contains two parallel input branches, one for receiving visible light images and the other for receiving visible light images. and long-wave infrared images Each branch consists of a series of convolutional layers, for example, using partial residual blocks from a ResNet network. The output feature maps of the two branches are concatenated and then fed into a shared group of convolutional layers to generate shared features. This shared feature This is the target illumination invariance feature tensor, which is then transmitted to the subsequent spatiotemporal modeling module 120.

[0080] Modal discriminator It is a classification network, for example, consisting of several fully connected layers and activation functions.

[0081] In a specific implementation, the modality discriminator It can be composed of three fully connected layers. The first two fully connected layers are followed by a LeakyReLU activation function, and the last fully connected layer outputs a scalar value, which is normalized to the (0,1) interval by the Sigmoid function to represent the probability.

[0082] Its receiving feature extractor Generated shared features As input, it outputs a scalar value representing the shared features of the input. The probability of originating from a visible light image branch.

[0083] During the training phase, the feature extractor and modal discriminator By optimizing a composite loss function Perform adversarial training. The composite loss function includes task loss. and combat losses :

[0084] ;

[0085] In the formula, These are the preset weighting coefficients used to balance the two losses.

[0086] Mission loss Used to ensure the generation of shared features It is effective for subsequent target recognition tasks. It consists of a classification loss. and regression loss The weighted composition, specifically, is as follows:

[0087] ;

[0088] In the formula, and These are preset weighting coefficients. The classification error of the target category can be calculated using the cross-entropy loss function. The localization error of the target bounding box can be calculated using the smoothed L1 loss function.

[0089] Combating losses Used to drive the feature extractor It generates features independent of the input modality. Specifically, it takes the form of a standard binary cross-entropy loss:

[0090] ;

[0091] In the formula, Represents the mathematical expectation; and These represent the actual data distribution of visible light images and long-wave infrared images, respectively; and These represent shared features generated from visible light images and long-wave infrared images, respectively. This represents the output probability of the modality discriminator.

[0092] The training process is carried out alternately, during the training of the modality discriminator. At that time, fixed feature extractor The parameters are maximized. To improve its accuracy in distinguishing feature sources. This is done during the training of the feature extractor. At that time, fixed mode discriminator The parameters are minimized. This allows the generated features to confuse the modality discriminator.

[0093] In a concrete implementation, this minimization process can be achieved by setting a gradient reversal layer between the feature extractor and the modality discriminator. This layer will transfer gradients from the feature extractor to the modality discriminator during backpropagation. The gradient value is multiplied by a negative constant. Through this adversarial process, the final trained feature extractor is obtained. It can generate illumination-invariant feature tensors.

[0094] See attached document Figure 4 , Figure 4 This is a schematic diagram of the spatiotemporal modeling module according to an embodiment of the present invention. The execution process of step S200 is described in detail below.

[0095] In step S200, the spatiotemporal modeling module 120 receives the illumination invariance feature tensor sequence of T consecutive frames output by the environment perception module 110. This module processes the sequence through a spatiotemporal joint network to generate the initial recognition result for the current frame (frame t). The spatiotemporal joint network includes a 3D convolutional CNN branch, a graph attention network (GAT) branch, and a deformable interactive fusion module.

[0096] The 3D convolution branch is used to extract the dynamic geometric features of the target in the temporal dimension. This branch treats the feature tensor sequence as a 3D data volume (dimensions T×H×W×C, where T is the length of the time series, H and W are the height and width of the feature map, and C is the number of channels) and applies a series of 3D convolution kernels to it. Each 3D convolution kernel performs sliding convolution operations simultaneously in both the spatial (H, W) and temporal (T) dimensions, thereby directly capturing the target's motion, deformation, and other local spatiotemporal patterns from the feature changes in consecutive frames. The output of this branch is a spatiotemporal feature map containing dynamic information. .

[0097] The graph attention network branch is used to model the spatial topological relationships between multiple targets or target parts within the current frame. First, in the feature tensor of the current frame... A Region Proposal Network (RPN) is applied to generate a set of candidate regions. Each candidate region is treated as a node in a graph, and the initial features of the node are derived from the feature tensor corresponding to that region. It is obtained after performing Region of Interest Pooling (RoI Pooling) operation.

[0098] The edge between any two nodes in the graph is established based on the spatial proximity of their corresponding candidate regions. Specifically, if the intersection-over-union (IoU) ratio of two candidate regions is greater than a preset threshold (e.g., 0.5), an edge is established between the two nodes to form the adjacency relationship of the graph.

[0099] Then, a graph attention mechanism is applied to the constructed graph. For any node in the graph... Its neighboring nodes Attention coefficient between Calculated using the following formula:

[0100] ;

[0101] In the formula, and They are nodes and nodes eigenvectors; For shared linear transformation weight matrix; This represents a vector concatenation operation; This represents the weight vector of a single-layer feedforward network. This is a linear unit activation function with leakage correction.

[0102] Attention coefficient After normalization using the Softmax function, the final attention weights are obtained. .node Updated features This is obtained by weighted summation of the features of all neighboring nodes:

[0103] ;

[0104] In the formula, For nodes The set of neighboring nodes, This is a non-linear activation function. After processing by the graph attention network, the feature vector of each node incorporates information about its context-related targets. These updated node features are then remapped back to their original spatial coordinates, forming a feature map that contains global spatial relationship information. .

[0105] The deformable interactive fusion module is used to dynamically fuse spatiotemporal feature maps. Relationship Feature Map For each location on the output fused feature map This module first uses a small convolutional layer, based on and In position Based on the splicing features, predict a set of two-dimensional offsets. and a set of modulation scalars . This is the preset number of sampling points.

[0106] This module utilizes the predicted offset. Adjust the position of the standard grid sampling points for the input. and In non-fixed and adaptive positions respectively Sampling was performed, among which This is a preset offset of the standard grid. The sampled values ​​are calculated using bilinear interpolation. The features obtained after sampling are then compared with the predicted modulation scalar. Multiply. Finally, the fused feature maps are located... value It is obtained by weighted summation of all K modulated sample values.

[0107] Finally, the fused feature map The data is fed into two parallel prediction heads: a classification head and a regression head. The classification head outputs the class probability of each target, and the regression head outputs the bounding box coordinates and instance segmentation mask of each target. Together, they form the initial recognition result and are transmitted to the subsequent spatiotemporal verification module 130.

[0108] In step S300, the spatiotemporal verification module 130 receives and caches the initial identification result sequence within a time window (e.g., N frames) output by the spatiotemporal modeling module 120. Each initial identification result contains the bounding box coordinates and classification confidence of the target. Based on this result sequence, the spatiotemporal verification module 130 constructs a Markov random field (MRF) model to smooth and correct the target localization results.

[0109] Specifically, a spatiotemporal graph is constructed, where the set of nodes consists of all target bounding boxes detected in all frames within the time window. Edges are established only between nodes in adjacent frames, representing potential associations. The aim of this method is to find the most reliable detection path across multiple frames for each real target by solving for the global optimum of this graph model, thereby filtering out isolated, unstable false detections and smoothly retaining the positions of correct detections.

[0110] The solution process is formalized as minimizing a global energy function. The problem is that the energy function is determined by a univariate potential function. and a binary potential function constitute:

[0111] ;

[0112] In the formula, This represents a set of optimal detection results to be solved. For the first The set of all targets detected in a frame; For the first The first frame The bounding box of each target; These are the preset weighting coefficients used to balance the two potential functions.

[0113] Univariate potential function This value is used to quantify the unreliability of a single test result and is negatively correlated with the classification confidence level of that result. A specific calculation method is as follows:

[0114] ;

[0115] In the formula, It is the output of the spatiotemporal modeling module 120 regarding the bounding box. The classification confidence level. A detection result with low confidence level will correspond to a high univariate potential energy value, thus tending to be suppressed in the process of minimizing global energy.

[0116] Binary potential function This is used to quantify the discontinuity between two detection results in adjacent frames, and its value is calculated based on their geometric and motional consistency. A specific calculation method is as follows:

[0117] ;

[0118] In the formula, The intersection-over-union ratio of the two bounding boxes. These are the coordinates of the center point of the bounding box; Let be the square of the L2 norm, and represent the square of the Euclidean distance. and These are preset weighting coefficients. This function applies a higher energy penalty to detectors with low spatial overlap or excessive inter-frame displacement.

[0119] An optimization algorithm, such as Loopy Belief Propagation or GraphCuts, is used to find a global energy function that satisfies this requirement. The optimal set of detection results that reaches the minimum value This set The detection result sequence, by simultaneously considering both single-frame detection confidence and motion continuity across multiple frames, exhibits higher localization stability and a lower error rate compared to the original initial recognition results. This optimal detection result set... This is the result of calibration and is transmitted to the subsequent online calibration module 140.

[0120] See attached document Figure 1 and attached Figure 2 The execution process of step S400 is described in detail below.

[0121] In step S400, the online calibration module 140 performs a real-time model calibration process guided by the verification results on the edge device 10. This process constitutes an internal feedback loop of the method of the present invention, used to continuously correct the systematic positioning deviation of the spatiotemporal modeling module 120.

[0122] The input to this module is two results for the same target in the same time frame: the bounding box in the initial recognition result output by the spatiotemporal modeling module 120. And the bounding boxes in the globally optimized calibrated recognition results output by the spatiotemporal verification module 130. Due to the identification results after calibration It incorporates information from multiple frames and satisfies motion continuity constraints, thus it can be considered a high-confidence monitoring signal or pseudo-true value.

[0123] The online calibration module 140 first calculates the correction deviation between the initial identification result and the calibrated identification result, and defines a calibration loss based on this deviation. .

[0124] Before calculating the loss, this module needs to extract the bounding boxes from the initial recognition results. Bounding boxes in the calibrated recognition results Perform matching. The matching process can be based on the intersection-over-union (IoU) criterion: for each calibrated bounding box... In the initial recognition results, find the initial bounding box with the largest IoU value that is greater than the matching threshold (e.g., 0.7). These constitute a matching pair. For all M successfully matched target pairs within a time window, the specific calculation method for this loss is as follows:

[0125] ;

[0126] In the formula, This indicates the extraction of the four-dimensional coordinate vector of the bounding box (e.g., center point x, center point y, width, height); and These represent the initial bounding box and the calibrated bounding box of the k-th matching pair, respectively; The smoothing L1 loss function is defined as follows:

[0127] ;

[0128] Using a smooth L1 loss function can provide a smooth gradient of a quadratic function when the bias is small, and a gradient of a linear function when the bias is large, thereby enhancing robustness to outliers and avoiding gradient explosion.

[0129] After calculating the calibration loss Subsequently, the module does not update all parameters of the spatiotemporal joint network. To ensure the stability of the model's low-level feature extraction capabilities and reduce the computational overhead of real-time backpropagation at the edge, the calibration update target is limited to a predefined set of top-level parameters. Above. This set of parameters. Specifically refers to the weights and biases of the last few layers of the regression head in a spatiotemporal joint network, which is responsible for outputting the bounding box coordinates.

[0130] The update process is performed using a single-step gradient descent method:

[0131] ;

[0132] In the formula, A preset, small learning rate is used to control the update step size and ensure a smooth calibration process; It is the calibration loss against the target parameter set The gradient.

[0133] This calibration process continues as the system runs in real time. After each frame or every N frames is processed, a new calibration loss is generated and a small parameter update is performed. Through this continuous, incremental adjustment, the spatiotemporal joint network can adaptively compensate for its own systematic positioning biases, such as persistent prediction offsets caused by specific deployment angles or repetitive motion patterns, thereby gradually improving the positioning accuracy of its initial recognition results without relying on offline retraining.

[0134] See attached document Figure 5 , Figure 5 This is a schematic diagram of the model dynamic evolution and edge adaptation process according to an optional embodiment of the present invention. This section describes in detail the execution process of optional step S500, which is completed collaboratively by the edge device 10 and the cloud server 20.

[0135] On the edge device 10, a real-time confidence entropy monitor is deployed. This monitor continuously analyzes the probability distribution vector of the classification head in the initial recognition results output by the spatiotemporal modeling module 120. For each detection result, its classification entropy... Calculated using the following formula:

[0136] ;

[0137] In the formula, The total number of categories, The model predicts that the target belongs to the first... The probability of a class. A high entropy value indicates low certainty in the model's classification result. When the average entropy value of all detection results within a time window consistently exceeds a preset trigger threshold... When the system determines that the current model has encountered a new scene or new category of target that cannot be reliably identified, the edge device 10 packages the raw image data collected within the time window and the corresponding low-confidence recognition results, and uploads them to the cloud server 20 via the communication network to trigger the model evolution process.

[0138] On cloud server 20, cloud processing unit 21 executes a conflict-free incremental learning process to incorporate new knowledge into the model without forgetting historical knowledge. This process first retrieves feature proxies representing the distribution of historical category data from model repository 22. This feature proxy can be generated by a pre-trained generative model or by directly storing a small number of representative historical feature vectors.

[0139] Subsequently, the cloud processing unit 21 uses the uploaded new data and retrieved feature proxies to calculate the new knowledge gradient for updating the model. To avoid this update interfering with historical knowledge, the gradient needs to be orthogonalized. The cloud processing unit 21 loads a set of orthogonal basis vectors from the model repository 22. The basis vectors span a historical knowledge parameter subspace. .

[0140] In a specific implementation, the set of orthogonal basis vectors B can be obtained by performing singular value decomposition (SVD) or principal component analysis (PCA) on a set of gradient vector matrices recorded by the model when learning historical tasks, and then selecting the most important principal component vectors as basis vectors.

[0141] This subspace represents the main directions of parameter updates in past task learning. New knowledge gradient The projection onto this subspace represents the part that conflicts with historical knowledge:

[0142] ;

[0143] Subtracting the projected portion from the original gradient yields the orthogonal update gradient. :

[0144] ;

[0145] Cloud processing unit 21 uses the orthogonal gradient update method. To update the parameters of the spatiotemporal joint network. Because With historical knowledge subspace Since the directions are orthogonal, this update is only performed in directions that do not affect historical knowledge, thus enabling incremental learning of new knowledge.

[0146] After the model parameters are updated, to adapt them to the computing power and memory constraints of the edge device 10, the cloud processing unit 21 further performs a hardware-aware neural architecture search (NAS). This process is conducted within a predefined search space (e.g., number of network layers, convolutional kernel size, number of channels). A search algorithm (e.g., an evolutionary algorithm) continuously generates candidate network architectures. The performance of each candidate architecture is evaluated by a comprehensive evaluator that considers not only the model's recognition accuracy on the validation set but also precisely measures its inference latency and peak memory usage on the target hardware using an edge device hardware simulator deployed in the cloud.

[0147] The goal of the search process is to find a sub-network architecture that achieves Pareto optimality in terms of accuracy, latency, and memory usage. Once the optimal architecture is determined, techniques such as knowledge distillation are used to transfer knowledge from the updated large model to this compact sub-network. Finally, this compact sub-network is fed into a dedicated compiler for optimizations such as operator fusion (e.g., merging convolutional, batch normalization, and activation function layers into a single computational kernel) and sparse compilation. Ultimately, a highly optimized model deployment package is generated and distributed to edge device 10 via a communication network to replace the original spatiotemporal joint network model.

[0148] In summary, this invention first extracts illumination-invariant features from multimodal data through adversarial training to improve adaptability to environmental changes. Subsequently, the system combines a 3D convolutional network and a graph attention network for spatiotemporal joint modeling, and utilizes Markov random fields to validate and smooth multi-frame results, thereby improving the accuracy and continuity of recognition. Furthermore, the system features an online calibration mechanism that allows for real-time fine-tuning of the model based on validation results to compensate for biases. An optional model evolution process utilizes conflict-free incremental learning and hardware-aware optimization in the cloud, supporting long-term model updates and iterations. Through the collaborative work of these modules, this invention aims to construct a target recognition system with robustness, accuracy, and adaptability in complex dynamic scenes.

[0149] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A target recognition method based on deep learning, characterized in that, Includes the following steps: Acquire visible light and long-wave infrared images of the same scene; Based on the visible light image and the long-wave infrared image, an illumination invariance feature tensor is generated; The illumination-invariant feature tensor is input into the spatiotemporal joint network to generate an initial recognition result containing the target location and category; Spatiotemporal verification is performed on the initial recognition results of multiple consecutive frames to generate calibrated recognition results; Based on the correction deviation between the initial identification result and the calibrated identification result, the spatiotemporal joint network is calibrated online.

2. The target recognition method based on deep learning according to claim 1, characterized in that, The step of generating the illumination-invariant feature tensor includes: The visible light image and the long-wave infrared image are mapped to a shared feature space using a feature extractor to generate shared features; The image source of the shared features is determined using a modality discriminator; By training the feature extractor and the modality discriminator adversarially, the shared features become modality-independent, thus serving as the illumination-invariant feature tensor.

3. The target recognition method based on deep learning according to claim 1, characterized in that, The spatiotemporal joint network includes: The three-dimensional convolution branch is used to extract the multi-scale dynamic geometric features of the target in the illumination-invariant feature tensor; A graph attention network branch is used to model the spatial topological relationships between targets in the illumination invariant feature tensor.

4. The target recognition method based on deep learning according to claim 3, characterized in that, The spatiotemporal joint network also includes: The deformable interactive module is used to adaptively adjust the fusion receptive field according to the content of the output features of the three-dimensional convolutional branch and the output features of the graph attention network branch, so as to dynamically fuse the output features, thereby forming a joint feature expression with enhanced discriminative power for occluded or blurred targets, and generating the initial recognition result based on the joint feature expression.

5. The target recognition method based on deep learning according to claim 1, characterized in that, The step of performing spatiotemporal verification on the initial recognition results of multiple consecutive frames includes: The initial recognition results from multiple consecutive frames are used to construct a spatiotemporal graph model; The spacetime graph model is solved by minimizing the energy function, which includes a univariate potential function and a binary potential function. The univariate potential function characterizes the confidence level of the initial recognition result of a single frame, and the binary potential function characterizes the motion continuity between the initial recognition results of adjacent frames, thereby generating the calibrated recognition result.

6. The target recognition method based on deep learning according to claim 1, characterized in that, The method also includes: The quality of the initial identification results is continuously evaluated through real-time confidence entropy monitoring; When the quality is lower than a preset threshold, an incremental learning process for the spatiotemporal joint network is triggered.

7. The target recognition method based on deep learning according to claim 6, characterized in that, The incremental learning process specifically includes: The feature reconstruction module is used to generate feature proxies that represent the distribution of historical category data. The update gradient of new knowledge is calculated by combining the aforementioned feature agent with the new category data; The update gradient of the new knowledge is projected onto the orthogonal complement space of the parameter subspace used to store historical knowledge to obtain the orthogonal update gradient; The parameters of the spatiotemporal joint network are updated using the orthogonal update gradient.

8. The target recognition method based on deep learning according to claim 6, characterized in that, After the incremental learning process is completed, the method further includes: Based on the computing power and memory constraints of the target edge device, a hardware-aware neural architecture search is used to automatically generate a simplified computation graph for the updated spatiotemporal joint network. The simplified computation graph is compiled into an efficient edge deployment package through operator fusion and sparse compilation.

9. The target recognition method based on deep learning according to claim 1, characterized in that, The steps for performing online model calibration include: The target position of the calibrated identification result is taken as a supervision signal, and the deviation between it and the target position of the initial identification result is calculated to obtain the calibration loss; Single-step gradient descent updates are performed using the calibration loss only on the top-level parameters responsible for localization regression in the spatiotemporal joint network, in order to compensate for the systematic localization bias of the network in real time at the edge.

10. A deep learning-based target recognition system for performing the method as described in any one of claims 1-9, characterized in that, include: An environment perception module is used to acquire visible light images and long-wave infrared images of the same scene, and generate an illumination invariant feature tensor based on the images; The spatiotemporal modeling module, connected to the environment perception module, is used to receive the illumination invariant feature tensor and generate an initial recognition result containing the target location and category through the spatiotemporal joint network. A spatiotemporal verification module, connected to the spatiotemporal modeling module, is used to perform spatiotemporal verification on the initial recognition results of multiple consecutive frames to generate calibrated recognition results; An online calibration module, connected to the spatiotemporal modeling module and the spatiotemporal verification module, is used to perform online model calibration of the spatiotemporal joint network based on the correction deviation between the initial identification result and the calibrated identification result.

Images