A system and method for identifying an object in a scene using a multi-modal and pulse neural network

Abstract

This invention discloses a system and method for incremental identification of projectiles integrating multiple modules and a spiking neural network. The system includes a pulse feature extraction module, an orthogonal projection module, an attention mechanism module, a task memory matrix, a memory-gated network, a feature fusion module, and a classifier module. The method includes: pulse encoding the input projectile image to extract pulse features; processing the pulse features through the orthogonal projection module to preserve learned task knowledge; sequentially inputting the pulse features into the attention mechanism module and the memory-gated network, and performing knowledge retrieval in conjunction with the task memory matrix; fusing the output features from the attention mechanism module, the memory-gated network, and the task memory matrix in the feature fusion module and inputting them into the classifier module to obtain the projectile category identification result; when learning a new task, updating the lateral connection weights of the orthogonal projection module, the parameters of the attention mechanism module, the task memory matrix, and the memory-gated network, while freezing relevant parameters from the old task.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, intelligent transportation and neuromorphic computing, and specifically relates to a system and method for identifying incremental spillage by integrating multiple modules and a spiking neural network. Background Technology

[0002] In recent years, deep learning-based debris recognition models have continuously improved the accuracy and speed of debris recognition, which is of great significance for promoting the deep integration of new-generation information technology and transportation, and assisting in the digital management of highways. However, current debris recognition algorithms cannot be directly applied to open traffic scenarios. There is an inherent contradiction between deep learning-based debris recognition methods and open application scenarios: on the one hand, the openness and uncertainty of traffic scenarios determine that the debris test data is an open set, and the number of debris categories is inexhaustible; on the other hand, mainstream target recognition algorithms can only recognize the categories covered by the training data. When a category that does not exist in the training data appears in the scenario, the method cannot recognize the "new category". Therefore, ordinary target recognition models are difficult to adapt to complex open environments and require frequent retraining of the entire model with all datasets, which leads to extremely high training costs. At the same time, artificial neural networks have high energy consumption, affecting the practical application of the model.

[0003] Continuous learning refers to the ability of a machine learning system to continuously learn new knowledge without forgetting old knowledge while continuously receiving new task data. Traditional neural networks suffer from the "catastrophic forgetting" problem, meaning that the performance of old tasks deteriorates significantly when learning new tasks. In existing technologies, the Hebbian learning-based orthogonal projection method (HLOP) effectively alleviates the forgetting problem by introducing lateral connections and the Hebbian learning mechanism into spiking neural networks to extract and orthogonally project the principal component subset space of neural activity. However, this method still suffers from insufficient feature representation, inadequate knowledge fusion between tasks, and limited generalization ability of the classifier in complex dynamic object recognition applications.

[0004] I. A search revealed Chinese invention patent CN121121507A, which discloses a deep learning-based method, device, electronic device, and program product for detecting road spillage. This method utilizes a trained detection model employing the LMSADet detection head. Multi-scale feature extraction and spatial attention mechanisms are introduced into the task branch, decoupling multi-scale modeling from the backbone and neck networks and integrating it into the detection head to directly optimize local details and scale differences of the spilled target. To suppress background interference and improve the recognition of blurred boundaries, an MSHA module is added to the neck network to efficiently capture the semantic relationship between the spilled material and the background, enhancing region understanding. The backbone network incorporates a C3ESP module, which extracts deep features through stacked depthwise separable convolutions and combines residual optimization fusion to avoid information loss. Simultaneously, a PEMA attention mechanism is introduced to dynamically adjust the importance of features in different receptive fields, enabling the model to focus on key features and capture data information more comprehensively, thereby significantly improving detection performance.

[0005] The technical comparison between this application and the aforementioned patent is as follows:

[0006] 1. The patent "A Deep Learning-Based Method, Device, Electronic Equipment, and Program Product for Detecting Road Spillage" focuses on building a traditional convolutional neural network detection model to perform end-to-end detection of road images, achieving bounding box localization and category recognition of road spills. Its core lies in the collaboration of the backbone network, neck network, and LMSADet detection head, introducing multi-scale fusion and spatial attention mechanisms to optimize the recognition of local details and scale differences of spills. In contrast, this patent adopts a Hebbian-based spiking neural network continuous learning framework, integrating four modules including attention mechanisms and memory gating networks to build a continuous learning system. Its core is to solve the "catastrophic forgetting" problem of traditional neural networks while achieving spill recognition, supporting incremental learning of spill categories. The two differ fundamentally in their core working mechanisms: the former is a single-task detection framework under static datasets, while this patent is a continuous learning framework under dynamic incremental scenarios, possessing resistance to catastrophic forgetting.

[0007] 2. The patent "A Deep Learning-Based Method, Device, Electronic Equipment, and Program Product for Road Sprinkler Detection" focuses on improving the accuracy and anti-interference capability of single detections by optimizing network modules (C3ESP, MSHA, PEMA). Model training is based on a fixed sample set, and the parameters are fixed after training, lacking incremental learning capability. The output is the predicted bounding box and category of the spilled material. This patent, however, constructs a principal component subset space through Hebbian learning and performs orthogonal projection. Combined with a task memory matrix, it explicitly stores the feature prototypes of each task, effectively preserving knowledge from older tasks while continuously learning new tasks, and can be extended to new spilled material category recognition. The two patents differ significantly in their model update strategies and task adaptability.

[0008] II. A search revealed Chinese invention patent CN114782891A, which discloses a road litter detection method based on contrastive clustering self-learning. The method constructs a road litter detection network model, which includes a feature extraction module, a candidate region module, an unknown label generation module, and a litter recognition module based on the maximum logical value. The road litter detection network model is trained using training samples, and then used to detect litter in the image to be detected, outputting the detection results. This invention can identify litter stationary on highways and has strong anti-interference capabilities. It can detect litter targets not included in the litter categories of the dataset, thus possessing greater practical value.

[0009] The technical comparison between this application and the aforementioned patent is as follows:

[0010] 1. The patent "A Road Spill Detection Method Based on Contrastive Clustering Self-Learning" focuses on building a traditional deep learning self-learning detection model. It achieves self-learning detection of road spills through the synergy of feature extraction, candidate region screening, unknown label generation, and maximum logistic value identification. Its core lies in improving the ability to identify unknown categories of spills not included in the dataset through contrastive clustering and logistic value analysis. In contrast, this patent adopts a spiking neural network architecture, constructing a principal component subspace through Hebbian learning for orthogonal projection. Combined with a task memory matrix, it explicitly stores the feature prototypes of each task, achieving cross-task knowledge protection and transfer. Furthermore, it adapts to the feature differences of various forms of spills through an attention mechanism. The two patents differ fundamentally in their core working mechanisms: the former belongs to a traditional deep learning framework for unknown categories, while the latter is a continuous learning framework driven by spiking neural networks, combining subspace projection and memory retrieval, and places greater emphasis on knowledge protection in multi-task continuous learning scenarios.

[0011] 2. The patent "A Road Spill Detection Method Based on Contrastive Clustering Self-Learning" focuses on achieving open-world detection capabilities through unknown label generation and feature storage sets, with the model identifying unknown categories during single-task training. This patent, however, protects learned task knowledge through an orthogonal projection mechanism and achieves cross-task knowledge reuse through a task memory matrix, maintaining high recognition accuracy for each task while continuously learning multiple tasks. The two patents differ fundamentally in their knowledge utilization and forgetting suppression strategies.

[0012] III. A search revealed Chinese invention patent CN119418265A, which discloses a method for analyzing the impact of spillage on highways based on a multimodal large model, belonging to the field of intelligent transportation technology. To improve the real-time performance and accuracy of spillage monitoring, the instance segmentation module of this invention performs fine-grained segmentation on the image data output by the video acquisition and management module, segmenting and initially identifying spillage in the images; the instruction generation module integrates data from the instance segmentation module, the memory module, and the spillage detection task description, generating output instructions that are sent to the large model inference module to execute the spillage analysis task; the input to the large model inference module includes the output instructions from the instruction generation module and the preprocessed image data from the video acquisition and management module, and the large model inference module performs in-depth analysis and inference on the input data to generate the impact analysis results of highway spillage; the evaluation module is used to comprehensively evaluate the highway spillage impact analysis results output by the large model inference module.

[0013] The technical comparison between this application and the aforementioned patent is as follows:

[0014] 1. The patent "A Method for Analyzing the Impact of Spilled Material Based on a Multimodal Large Model in Highway Scenarios" focuses on building a multimodal large model collaborative system. Through the collaborative process of video acquisition, instance segmentation, large model inference, and evaluation, it achieves the analysis of the impact of spilled material on highways. Its core lies in leveraging the general inference capabilities and historical experience of the large model to improve the accuracy of the analysis. The large model inference outputs the impact results of spilled material on traffic; spilled material identification is merely a preliminary step in the subsequent analysis. In contrast, this patent adopts a spiking neural network architecture, based on improved Hebbian learning and multi-module integration, to build a continuous learning system. Its core is to achieve incremental identification of spilled material, solving the catastrophic forgetting problem of traditional models. It focuses on the accurate identification and continuous learning of spilled material categories, representing a cross-application of neuromorphic computing and computer vision. The two patents differ fundamentally in their core working mechanisms: the former belongs to a multimodal large model impact analysis framework, while the latter belongs to a spiking neural network-driven continuous identification framework; their task objectives and technical paths are completely different.

[0015] 2. The patent "A Method for Effort Impact Analysis Based on a Multimodal Large Model in Highway Scenarios" focuses on multi-module collaborative data integration and inference evaluation, outputting effort impact analysis results and evaluation reports. It relies on the strong inference capabilities of the large model but lacks incremental learning capabilities, and effort impact identification only meets basic analysis needs. This patent, however, emphasizes the accuracy and continuous learning capabilities of effort impact identification. It mitigates forgetting through multi-module collaboration and Hebbian learning's local update rules, supports incremental identification of multi-form and multi-category effort impact, outputs accurate category prediction results, and its pulsed design is suitable for low-power neuromorphic hardware deployment. It places greater emphasis on the continuity of the identification task and hardware adaptability. The two patents differ significantly in task objectives, output format, and hardware adaptability.

[0016] IV. A search revealed that Chinese invention patent CN117079035A discloses a method for detecting debris on highways that integrates scene semantic information. The method constructs a highway debris detection network model, which includes a potential candidate box extraction module, a non-debris suppression module integrating scene semantic information, and an adaptive candidate box selection module. The highway debris detection network model is trained using training samples, and then used to detect images to be detected, outputting the detection results. This invention can identify debris stationary on highways and has strong anti-interference capabilities. It can detect debris targets not included in the debris categories of the dataset, thus possessing greater practical value.

[0017] The technical comparison between this application and the aforementioned patent is as follows:

[0018] 1. The patent "A Method for Detecting Spilled Objects on Highways by Integrating Scene Semantic Information" focuses on constructing a detection network that includes a potential candidate box extraction module, a non-spilled object suppression module that integrates scene semantic information, and an adaptive candidate box selection module. It suppresses non-spilled object candidate boxes by matching scene semantic information, and its core lies in improving detection accuracy using semantic prior knowledge. In contrast, this patent stores the feature prototypes of each task through a task memory matrix and uses a memory retrieval mechanism to achieve cross-task knowledge reuse, rather than relying on external scene semantic priors. The two differ in their knowledge utilization methods: the former relies on external semantic prior knowledge; this patent relies on the task's inherent feature memory and retrieval mechanism, making it more adaptable to dynamically changing continuous learning scenarios.

[0019] 2. The patent "A Method for Detecting Highway Spilled Objects by Integrating Scene Semantic Information" focuses on suppressing non-spilled object candidate boxes through scene semantic information to improve the accuracy of single-task detection. This patent, on the other hand, uses Hebbian learning to extract principal component subsets online and combines them with a task memory matrix to achieve cross-task knowledge protection, effectively suppressing interference with old task knowledge when continuously learning new tasks. The two are fundamentally different in terms of anti-interference mechanisms and adaptability to application scenarios.

[0020] V. A search revealed Chinese invention patent CN115311850A, which discloses a crowdsourcing-based method and system for identifying and warning of spilled materials. On the vehicle side, data is collected from moving vehicles. Based on this data, the system identifies the presence of spilled materials, locates the target material, assesses the risk, and then uploads the information to the cloud. In the cloud, upon receiving the spilled material information, it first checks if there is already recorded spilled material information at the same location. If not, the information is recorded as newly discovered. If so, it is merged with existing information and updated. Finally, a spilled material warning is pushed to the terminal. This invention employs a crowdsourcing model, leveraging the crowdsourced data and capabilities of numerous vehicles. It utilizes multi-sensor data from a large number of vehicles for spilled material identification and, through collaboration with cloud information, achieves iterative updates and dissemination of precise spilled material information on the road. This improves the accuracy of road spilled material identification and warning, reduces costs, and enhances road safety.

[0021] The technical comparison between this application and the aforementioned patent is as follows:

[0022] 1. The patent "A Crowdsourcing-Based Method and System for Identifying and Warning of Spilled Objects" focuses on building a crowdsourcing-based system for identifying and warning of spilled objects. It achieves identification, location, and warning of spilled objects on roads through the collaboration of vehicle-side data collection, cloud-based information fusion, and warning push notifications. Its core lies in utilizing crowdsourced data to improve identification coverage and accuracy, emphasizing multi-device collaboration and information iteration at the system level. In contrast, this patent focuses on enhancing the continuous learning capabilities within a single intelligent agent (device), achieving knowledge protection and incremental learning through the structure and learning rules of the neural network itself, rather than relying on multi-agent collaboration and data fusion. The two patents differ fundamentally in their system architecture: the former belongs to a distributed collaborative perception framework, while this patent belongs to a single-node intelligent enhancement framework.

[0023] 2. The patent "A Crowdsourcing-Based Method and System for Spray Recognition and Early Warning" focuses on vehicle-to-cloud collaboration and iterative updates and early warning pushes for spill information. Its spill recognition relies on conventional models and lacks incremental learning capabilities, only achieving the fusion and update of spill information at the same location. This patent, however, emphasizes the model's own continuous learning capabilities. Through modules such as orthogonal projection and task memory matrices, it achieves incremental learning of new spill categories while retaining old knowledge. It does not rely on crowdsourced data and improves the recognition accuracy of multi-form spills through attention and feature fusion modules, outputting category prediction results. It is suitable for neuromorphic hardware deployment and places greater emphasis on the model's independence and continuous learning capabilities. The two patents differ significantly in their technological focus, deployment flexibility, and dependencies.

[0024] VI. A search revealed that Chinese invention patent CN118799623A discloses a system and method for tracking road debris and detecting road defects. The system includes: an image acquisition device, a terminal device, and a display device. The image acquisition device acquires road video images and transmits them to the terminal device. The terminal device has a built-in road debris tracking module and a road defect detection module. The road debris tracking module uses a target tracking algorithm to filter suspected debris after target localization to obtain the target debris. The road defect detection module uses an ACD-YOLOv3 model to detect road defects in the road video images transmitted by the image acquisition device and generate road defect detection results. The display device displays the detection results of the target debris and the road defects. This system ensures both the accuracy of detecting small target debris or small target road defects and the real-time performance of the detection.

[0025] The technical comparison between this application and the aforementioned patent is as follows:

[0026] 1. The patent "A System and Method for Tracking Road Spillage and Detecting Road Defects" focuses on building an integrated system for tracking road spillage and detecting road defects. It uses a Gaussian mixture model for foreground / background segmentation, a target tracking algorithm for spillage screening, and an optimized YOLOv3 model for defect detection. Its core lies in the collaborative execution of both spillage tracking and road defect detection, improving the accuracy and real-time performance of detecting small-target spillage and road defects. In contrast, this patent employs a spiking neural network architecture, based on improved Hebbian learning's orthogonal projection and multi-module integration to construct a continuous learning system. Its core is a single incremental spillage recognition task, addressing the problem of catastrophic forgetting. Through multi-module collaboration, it enhances the representation of spillage features and knowledge retention, adapting to the recognition of various spillage forms. The two patents differ fundamentally in their core working mechanisms: the former belongs to a traditional deep learning dual-task detection and tracking framework, while the latter belongs to a spiking neural network-driven single-task continuous recognition framework.

[0027] 2. The patent "A Road Spill Tracking and Road Defect Detection System and Method" focuses on the collaborative task of spill tracking and road defect detection, optimizing the model to improve detection real-time performance and small target recognition accuracy. However, its model training relies on a fixed sample set, lacks incremental learning capability, and outputs spill tracking and road defect detection results. This patent, on the other hand, focuses on incremental recognition of a single spill category. It utilizes Hebbian learning's local update rules and a task memory matrix to retain historical knowledge, supporting incremental learning of new spill categories. Furthermore, its pulsed design is suitable for neuromorphic hardware deployment, outputting spill category prediction results and emphasizing the continuity of the recognition task and hardware adaptability. The two patents differ significantly in terms of the number of tasks, model update strategies, and hardware adaptability.

[0028] VII. A search revealed Chinese invention patent CN120877151A, which discloses an automatic identification system for highway spills based on unmanned aerial vehicles (UAVs). This system relates to the fields of intelligent transportation and UAV technology, and includes: a UAV swarm deployment module employing a mesh self-organizing network to cover 150-250m wide lanes; a multimodal data acquisition module collecting infrared, visible light, depth map, and point cloud data; an image preprocessing module for downsampling, filtering and denoising, and enhancing contrast; a spill identification module fusing multimodal features with a confidence threshold of 0.6-0.7, verifying suspected targets through point cloud clustering; a 3D modeling and localization module fusing binocular and lidar data; an intelligent decision-making and scheduling module that generates priorities based on traffic flow and plans inspection routes; and a coordinated response module that pushes information via 4G / 5G to coordinate with information boards and road administration vehicles for response. This invention improves monitoring coverage and identification accuracy, shortens response time, reduces positioning errors, optimizes response efficiency, and ensures highway traffic safety.

[0029] The technical comparison between this application and the aforementioned patent is as follows:

[0030] 1. The patent "An Automatic Identification System for Highway Spilled Objects Based on Unmanned Aerial Vehicles" focuses on building an automatic identification system for spilled objects through the collaborative efforts of unmanned aerial vehicle (UAV) swarms. It achieves high-precision identification, location, and coordinated handling of spilled objects on highways through the synergy of multimodal data acquisition from UAVs, improved YOLOv8 recognition, and 3D modeling and positioning. Its core lies in the fusion of UAV hardware swarms and multimodal data to improve monitoring coverage and identification accuracy. In contrast, this patent focuses on enhancing the continuous learning capabilities of ground-based fixed or vehicle-mounted edge devices. It achieves efficient incremental learning through spiking neural networks and Hebbian learning, rather than relying on UAV swarms and multimodal hardware collaboration. It enhances feature representation and knowledge memorization through multi-module collaboration. The two patents differ in application scenarios and hardware dependencies: the former is suitable for large-scale inspections; this patent is suitable for continuous, fixed monitoring scenarios.

[0031] 2. The patent "An Automatic Recognition System for Highway Spilled Objects Based on Unmanned Aerial Vehicles" focuses on UAV swarm deployment, multimodal data fusion, and 3D localization, relying on hardware collaboration to improve recognition and positioning accuracy. The model uses a modified YOLOv8, lacks incremental learning capability, and outputs spilled object recognition results, location information, and disposal instructions. This patent, however, emphasizes the model's continuous learning capability. It enhances the continuous learning ability of a single node device through spiking neural networks and Hebbian learning, adapts to multi-morphological spilled object recognition through attention and feature fusion modules, and its spiking design is suitable for low-power neuromorphic hardware deployment. It outputs spilled object category prediction results, requires no multimodal sensors or UAV swarms, and places greater emphasis on model flexibility and continuous learning capability. The two patents differ significantly in hardware dependence, technology dependence, and model update strategies. Summary of the Invention

[0032] To address the limitations of existing technologies in rapidly identifying new types of spilled materials to adapt to dynamic application scenarios, and the high energy consumption of artificial neural networks, this invention proposes an incremental spilled material identification system and method integrating multiple modules and a spiking neural network. By integrating an attention mechanism, a memory gating network, a task memory matrix, and a feature fusion module, the system enhances the model's ability to extract, memorize, and fuse features from multiple types of spilled materials, significantly improving classification accuracy and anti-forgetting capabilities during continuous learning, and achieving stable identification of both new and old spilled material types.

[0033] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0034] A system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network, comprising:

[0035] The pulse feature extraction module is used to receive the input image and extract pulse features;

[0036] The orthogonal projection module, based on lateral connectivity and the Hebbian learning mechanism, performs orthogonal projection processing on the weight update vector to preserve the knowledge representation of the old task when learning a new task.

[0037] The attention mechanism module dynamically weights the pulse features to enhance attention to key regions in the image.

[0038] The task memory matrix stores the feature representations unique to each task and supports cross-task knowledge retrieval.

[0039] Memory gating networks are used to selectively remember and forget features of historical tasks.

[0040] The feature fusion module is used to fuse output features from the attention mechanism module, the memory gating network, and the task memory matrix;

[0041] The classifier module outputs a category prediction result based on the fused features.

[0042] As a preferred technical solution of the present invention: the system is suitable for incremental recognition of images of projectiles, including but not limited to five categories: cloud-like objects, particulate matter, block-like objects, strip-like objects, and sheet-like objects.

[0043] As a preferred embodiment of the present invention: the pulse feature extraction module employs leaky integrated firing neurons, combined with convolutional layers, batch normalization layers, fully connected layers, and pooling layers, with the time step set to... Its specific implementation includes:

[0044] (1);

[0045] in, Indicates the first Time step One input sample data, It is the first A module containing convolutional layers, batch normalization layers, fully connected layers, pooling layers, and LIF neurons. It is network output.

[0046] The specific implementation of LIF neurons includes:

[0047] (2);

[0048] (3);

[0049] in, In time step Time Neuronal membrane potential of the layer In time step Time The first layer The membrane potential of a neuron It is the membrane potential attenuation coefficient. It is the first Layer weights In time step Time The first layer The pulse fired by each neuron It is the membrane potential threshold for the pulse being emitted.

[0050] As a preferred embodiment of the present invention: the orthogonal projection module extracts principal component subsets based on Hebbian learning and implements orthogonal projection, specifically including:

[0051] Principal component subspaces of learned task features are extracted based on lateral connectivity networks, and projection gradients orthogonal to these principal component subspaces are constructed. As shown in the formula below:

[0052] (4);

[0053] in, For the weights of the lateral connection network, It is the identity matrix;

[0054] Based on the above projection gradient Construct the weight update gradients orthogonal to the principal component subset space of the learned task features. As shown in the formula below:

[0055] (5);

[0056] in, It's an error. It is the trajectory of neuron activity;

[0057] Weights of lateral connection networks Based on Oja rule updates, the gradient of weight updates is also updated. As shown in the formula below:

[0058] (6);

[0059] in, For lateral connection, positive output. This refers to the learning rate.

[0060] As a preferred embodiment of the present invention: the attention mechanism module obtains a scaled attention score through a fully connected layer-ReLU activation function-fully connected layer-Sigmoid activation function, thereby guiding the attention allocation of the original input features. The attention score is obtained through the attention subnetwork. Next, input With attention mask Multiplication is performed to achieve weighted summation of specific features.

[0061] Output features Represented as:

[0062] (7).

[0063] As a preferred embodiment of the present invention: the task memory matrix is ​​an updatable matrix structure, and its retrieval and update mechanism includes:

[0064] The memory matrix is ​​represented as Each row corresponds to a task feature prototype. ;

[0065] For input features Retrieve the memory row with the highest similarity:

[0066] (8);

[0067] The task memory matrix uses a momentum update rule:

[0068] (9);

[0069] in, This is the momentum coefficient.

[0070] As a preferred technical solution of the present invention: the memory gating network adopts a gated cyclic unit structure, and the specific implementation and update process includes:

[0071] Update Gate The update process controls the proportion of retained memory features and the proportion of new features incorporated, and is as follows:

[0072] (10);

[0073] Reset door The system is responsible for filtering memory features and determining which memory features can participate in the generation of the current candidate state. The update process is as follows:

[0074] (11);

[0075] By fusing the filtered memory features with the current input features, candidate states for the current time step are generated. The update process is as follows:

[0076] (12);

[0077] in, This indicates element-wise multiplication, which allows for the filtering of memory features;

[0078] By updating the door The final output of the GRU is obtained by dynamically balancing historical memory features and candidate hidden states. The update process is as follows:

[0079] (13);

[0080] in, It is the proportion of retained memory features. It is the proportion of the new features that are incorporated. The two are added together to complete the fusion of the memory features and the new features.

[0081] As a preferred embodiment of the present invention: the feature fusion module is implemented using a weighted sum method, and the enhanced feature after fusion is:

[0082] (14).

[0083] As a preferred embodiment of the present invention: the classifier module is implemented using a fully connected layer and a Softmax activation function, and outputs the probability of each category:

[0084] (15);

[0085] in, For classifier weights, For classifier bias, This is a feature of fusion.

[0086] A method for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network includes the following steps:

[0087] S1. Perform pulse coding on the input projectile image and extract pulse features;

[0088] S2. Process the pulse characteristics through the orthogonal projection module to protect the learned task knowledge;

[0089] S3. Input the pulse features sequentially into the attention mechanism module and the memory gating network, and perform knowledge retrieval in conjunction with the task memory matrix. The task memory matrix is ​​updated after each task is completed, adding the feature prototype of the task, and supporting cosine similarity-based retrieval in subsequent tasks.

[0090] S4. In the feature fusion module, the output features from the attention mechanism module, the memory gating network and the task memory matrix are fused and input into the classifier module to obtain the object category recognition result.

[0091] S5. When learning a new task, update the lateral connection weights of the orthogonal projection module, the attention mechanism module, the task memory matrix, and the memory gating network parameters, while freezing the relevant parameters of the old task.

[0092] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0093] This invention constructs a hierarchical, memory-enhanced continuous learning system based on HLOP-SNN by introducing an attention mechanism module, a memory gating network, a task memory matrix, and a feature fusion module. It is particularly suitable for incremental recognition scenarios involving multi-form, highly dynamic objects like spilled materials. While improving recognition accuracy, it effectively protects historical knowledge, demonstrating good practicality and promotional value. The average accuracy of spilled material recognition increased from 82.7% to 86.3%, an absolute improvement of 3.6%; it effectively overcomes catastrophic forgetting, improving the backward transfer (BWT) index from -1.8% to +0.4%, achieving positive knowledge transfer; it meets the real-time requirements of monitoring scenarios such as urban roads and highways and supports online learning. Attached Figure Description

[0094] Figure 1 This is a flowchart of the method of the present invention;

[0095] Figure 2 This is the original HLOP-SNN network structure diagram;

[0096] Figure 3 This is a diagram of the overall structure of the Attention-SNN network in this invention;

[0097] Figure 4 This is a structural diagram of the attention mechanism module in this invention;

[0098] Figure 5 This is a diagram of the memory gating network structure in this invention. Detailed Implementation

[0099] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0100] This invention proposes a system for identifying incremental spillage by integrating multiple modules and a spiking neural network, comprising:

[0101] The pulse feature extraction module is used to receive the input image and extract pulse features;

[0102] The pulse feature extraction module employs leaky integrated firing neurons, combined with convolutional layers, batch normalization layers, fully connected layers, and pooling layers, with the time step set to... Its specific implementation includes:

[0103] (1);

[0104] in, Indicates the first Time step One input sample data, It is the first A module containing convolutional layers, batch normalization layers, fully connected layers, pooling layers, and LIF neurons. It is network output.

[0105] The specific implementation of LIF neurons includes:

[0106] (2);

[0107] (3);

[0108] in, In time step Time Neuronal membrane potential of the layer In time step Time The first layer The membrane potential of a neuron It is the membrane potential attenuation coefficient. It is the first Layer weights In time step Time The first layer The pulse fired by each neuron It is the membrane potential threshold for the pulse being emitted.

[0109] The orthogonal projection module, based on lateral connectivity and the Hebbian learning mechanism, performs orthogonal projection processing on the weight update vector to preserve the knowledge representation of the old task when learning a new task.

[0110] The orthogonal projection module extracts principal component subsets based on Hebbian learning and implements orthogonal projection. Specifically, the implementation includes:

[0111] Principal component subspaces of learned task features are extracted based on lateral connectivity networks, and projection gradients orthogonal to these principal component subspaces are constructed. As shown in the formula below:

[0112] (4);

[0113] in, For the weights of the lateral connection network, It is the identity matrix;

[0114] Based on the above projection gradient Construct the weight update gradients orthogonal to the principal component subset space of the learned task features. As shown in the formula below:

[0115] (5);

[0116] in, It's an error. It is the trajectory of neuron activity;

[0117] Weights of lateral connection networks Based on Oja rule updates, the gradient of weight updates is also updated. As shown in the formula below:

[0118] (6);

[0119] in, For lateral connection, positive output. This refers to the learning rate.

[0120] The attention mechanism module dynamically weights the pulse features to enhance attention to key regions in the image.

[0121] The attention mechanism module obtains a scaled attention score through a fully connected layer-ReLU activation function-fully connected layer-Sigmoid activation function, which guides the attention allocation of the original input features. The attention score is obtained through the attention subnetwork. Next, input With attention mask Multiplication is performed to achieve weighted summation of specific features.

[0122] Output features Represented as:

[0123] (7).

[0124] The task memory matrix stores the feature representations unique to each task and supports cross-task knowledge retrieval.

[0125] The task memory matrix is ​​an updatable matrix structure, and its retrieval and update mechanism includes:

[0126] The memory matrix is ​​represented as Each row corresponds to a task feature prototype. ;

[0127] For input features Retrieve the memory row with the highest similarity:

[0128] (8);

[0129] The task memory matrix uses a momentum update rule:

[0130] (9);

[0131] in, This is the momentum coefficient.

[0132] Memory gating networks are used to selectively remember and forget features of historical tasks.

[0133] The memory-gated network adopts a gated cyclic unit structure, and its implementation and update process includes:

[0134] Update Gate The update process controls the proportion of retained memory features and the proportion of new features incorporated, and is as follows:

[0135] (10);

[0136] Reset door The system is responsible for filtering memory features and determining which memory features can participate in the generation of the current candidate state. The update process is as follows:

[0137] (11);

[0138] By fusing the filtered memory features with the current input features, candidate states for the current time step are generated. The update process is as follows:

[0139] (12);

[0140] in, This indicates element-wise multiplication, which allows for the filtering of memory features;

[0141] By updating the door The final output of the GRU is obtained by dynamically balancing historical memory features and candidate hidden states. The update process is as follows:

[0142] (13);

[0143] in, It is the proportion of retained memory features. It is the proportion of the new features that are incorporated. The two are added together to complete the fusion of the memory features and the new features.

[0144] The feature fusion module is used to fuse output features from the attention mechanism module, the memory gating network, and the task memory matrix;

[0145] The feature fusion module is implemented using a weighted sum method, and the enhanced features after fusion are:

[0146] (14).

[0147] The classifier module outputs a category prediction result based on the fused features;

[0148] The classifier module is implemented using a fully connected layer and a Softmax activation function, and outputs the probability of each class:

[0149] (15);

[0150] in, For classifier weights, For classifier bias, This is a feature of fusion.

[0151] The present invention proposes an incremental spray identification system integrating multiple modules and spiking neural networks. The deep learning model used is based on the Hebbian learning-based spiking neural network continuous learning model (HLOP-SNN), and is obtained by introducing an attention mechanism module, a memory gating network, a task memory matrix and a feature fusion module before the classifier, which is called the Attention-SNN model.

[0152] The Attention-SNN model includes: a spike feature extraction module, an attention mechanism module, a memory gating network, a task memory matrix, a feature fusion module, and a classifier module. The specific process of the Attention-SNN model is as follows:

[0153] The projectile image data is input to the pulse feature extraction module to extract projectile features from the input image, obtaining a basic feature map. This basic feature map is then input to the attention mechanism module to calculate the corresponding attention weights. Based on the basic feature map extracted for the current task, corresponding memory features are retrieved from the learnable task memory matrix to obtain a memory feature map. Simultaneously, the basic feature map and memory features are input to a memory gating network to generate a fused feature map of the memory and basic features. Finally, the basic feature map, attention weights, memory features, and the fused feature map of the basic features are input together to the feature fusion module to obtain an enhanced feature map. The specific fusion method is as follows: The enhanced feature map is input into the classifier, and the output is the type of spill.

[0154] The attention mechanism module is a three-layer feedforward neural network, consisting of: a feature transformation layer, which comprises a 128-dimensional fully connected layer, a batch normalization layer, and a ReLU activation function layer, used to perform a 128-dimensional nonlinear mapping on the input projectile features; an attention calculation layer, which comprises a 64-dimensional fully connected layer, a batch normalization layer, and a ReLU activation function layer, used to perform a 128-dimensional to 64-dimensional calculation on the transformed features to evaluate the importance of different feature dimensions; and an attention weight output layer, which comprises a 32-dimensional fully connected layer, a batch normalization layer, a ReLU activation function layer, a 1-dimensional fully connected layer, and a Sigmoid activation function layer, used to compress the 64-dimensional features to 1 dimension and output a normalized attention score through the Sigmoid activation function.

[0155] Among them, the task memory matrix is ​​a one-dimensional matrix. The learnable parameter matrix, where The total number of types of spilled material. The dimension of the feature vector is denoted as ; the corresponding row vector is queried and extracted from the task memory matrix based on feature similarity, and used as the memory feature map of the current recognition task; the task memory matrix is ​​updated and optimized through gradient backpropagation during model training.

[0156] Among them, the memory-gated network uses a gated recurrent unit to achieve the fusion of the basic features of the current task and the memory features.

[0157] The feature fusion module is implemented using the following formula: Wherein, the basic feature map is the projectile feature extracted by the baseline SNN network, and the attention weight is the output of the attention mechanism module. This is the output of the memory-gated network.

[0158] like Figure 1 As shown, the present invention proposes a method for identifying incremental spillage by integrating multiple modules and a spiking neural network, comprising the following steps:

[0159] S1. Perform pulse coding on the input projectile image and extract pulse features;

[0160] S2. Process the pulse characteristics through the orthogonal projection module to protect the learned task knowledge;

[0161] S3. Input the pulse features sequentially into the attention mechanism module and the memory gating network, and perform knowledge retrieval in conjunction with the task memory matrix. The task memory matrix is ​​updated after each task is completed, adding the feature prototype of the task, and supporting cosine similarity-based retrieval in subsequent tasks.

[0162] S4. In the feature fusion module, the output features from the attention mechanism module, the memory gating network and the task memory matrix are fused and input into the classifier module to obtain the object category recognition result.

[0163] S5. When learning a new task, update the lateral connection weights, task memory matrix, and memory gating network parameters of the orthogonal projection module, while freezing the relevant parameters of the old task.

[0164] The identification method of this invention collects data on different types of projectiles. Based on the shape of the projectiles, the data for each type of projectile is grouped into a subset. Identification of each subset is considered a sub-identification task. Each subset is further divided into a training set, a validation set, and a test set. The parameters of each layer of the Attention-SNN model are trained using the training set of the first subset. The performance of the Attention-SNN model is validated using its validation set, and the optimal parameters are stored. Based on the first identification task, the parameters of each layer of the Attention-SNN model are trained using the training set of the second subset based on the orthogonal projection method. This achieves incremental identification of the second sub-task without reducing the identification accuracy of the first sub-identification task. The performance of the Attention-SNN model is validated using its validation set, and the optimal parameters are stored. This process is repeated to achieve incremental identification of different types of projectiles. The trained Attention-SNN model is tested using the test set of each subset. If the identification accuracy meets the standard, a deployable Attention-SNN model is obtained; otherwise, the Attention-SNN model is retrained until it meets the requirements for deployment. If a new type of projectile appears during model usage, data on the new type of projectile can be collected to incrementally train the Attention-SNN model. The Attention-SNN model can achieve accurate identification of the new type of projectile without reducing the accuracy of identifying existing types of projectiles.

[0165] The present invention will be described in detail below with reference to specific embodiments:

[0166] First: Collect data on different types of projectiles. Based on the shape of the projectiles, the data of each type of projectile is divided into a subset. In this embodiment, there are 5 types of projectiles, including cloud-like objects, particulate matter, block-like objects, strip-like objects, and sheet-like objects, with a total of 2854 images. The identification of each subset is regarded as a sub-identification task.

[0167] Then: each subset of the dataset is further divided into a training set, a validation set, and a test set in an 8:1:1 ratio;

[0168] Secondly, based on the orthogonal projection method, the Attention-SNN model is incrementally trained using the training set of each subset of the dataset. After each training iteration, the model performance is tested using the validation set of that subset of the dataset, and the optimal model is saved. The specific network training process is as follows: the spiking neural network is trained using the DSR (Differential Training Based on Impulse Representation) method; the HLOP lateral connection weights are updated online through Hebbian learning; the attention weights, gating network parameters, and task memory matrix are jointly trained through error backpropagation and gradient descent; after each task is trained, the HLOP lateral weights and task memory matrix are frozen, and new subspace neurons and memory units are only expanded in new tasks.

[0169] Secondly: Test the saved optimal Attention-SNN model using the test set of each subset of the dataset. If the recognition accuracy meets the standard, deploy and use it; otherwise, retrain the Attention-SNN model until it meets the requirements for deployment and use.

[0170] Finally: During deployment and use, if a new type of debris appears, data on the new type of debris will be collected, and the Attention-SNN model will be incrementally trained to enable the Attention-SNN model to recognize the new type of debris.

[0171] The structure of the original HLOP-SNN model is as follows: Figure 2 As shown, LIF neurons and HLOP modules are combined on the basis of ResNet-18 network to overcome the catastrophic forgetting problem in incremental learning.

[0172] The structure of the Attention-SNN model is as follows: Figure 3 As shown, an attention mechanism module, a memory gating network, a task memory matrix, and a feature fusion module were added to the original HLOP-SNN model.

[0173] The image data of the spilled material is input into the feature extraction module (i.e. Figure 3 The original HLOP-SNN model extracts projectile features from the input image to obtain a basic feature map. This basic feature map is then input into the attention mechanism module to calculate the corresponding attention weights. Based on the basic feature map extracted for the current task, corresponding memory features are retrieved from the learnable task memory matrix to obtain a memory feature map. Simultaneously, the basic feature map and memory features are input into a memory gating network to generate a fused feature map of the memory and basic features. The basic feature map, attention weights, memory features, and the fused feature map of the basic features are then input into the feature fusion module to obtain an enhanced feature map. The specific fusion method is: basic feature map + basic feature map × attention weights + fused feature map of the memory and basic features. The enhanced feature map is then input into a classifier to output the projectile type.

[0174] The pulse feature extraction module is sequentially connected in series with a convolutional layer, eight BasicBlocks modules, a batch normalization layer, a LIF neuron layer, a global average pooling layer, and a fully connected layer. The BasicBlocks modules are sequentially connected in series with two modules consisting of a convolutional layer, a batch normalization layer, and a LIF neuron layer, with residual connections between the output and input. Furthermore, each trainable convolutional layer and fully connected layer is laterally connected to an HLOP module to achieve orthogonal projection of network parameters, overcoming the catastrophic forgetting problem in incremental learning.

[0175] The input spill image first passes through one convolutional layer to output a 20×42×42 feature map. Then, it passes through eight BasicBlocks modules to extract the feature information of the highway spill image. After passing through batch normalization layers and global average pooling, it obtains 160-dimensional feature information. Finally, it passes through a fully connected layer to obtain a 128-dimensional feature map, which is the output of the feature extraction module.

[0176] Attention mechanism module such as Figure 4 As shown, it includes the following components: a fully connected layer (Linear) for feature dimensionality reduction, a batch normalization layer for data distribution normalization, an activation function layer (ReLU, Sigmoid) for nonlinear transformation, and a Dropout layer for parameter sparsity.

[0177] The feature map is further processed by a 128-dimensional fully connected layer to extract features, followed by batch normalization and ReLU activation, and then network lightweighting using Dropout. The output 128-dimensional feature map is further compressed by a 64-dimensional fully connected layer, followed by batch normalization and ReLU activation, and then network lightweighting using Dropout. The output 64-dimensional feature map then passes through a 32-dimensional fully connected layer, a ReLU activation function layer, and a 1-dimensional fully connected layer to obtain compressed 1-dimensional features, and finally passes through a sigmoid function to obtain attention weights.

[0178] The structure of a memory-gated network is as follows Figure 5 As shown, based on the gated loop unit structure, selective memorization and forgetting of historical task features are performed; by resetting the gate... Filter memory features to determine which memory features can participate in the generation of the current candidate state; update the gate. Control the proportion of retained memory features and the proportion of new features incorporated; fuse the filtered memory features with the current input features to generate the candidate state for the current time step. By updating the door The system dynamically balances historical memory features and candidate hidden states, and outputs the gated feature representation.

[0179] The feature fusion module performs multimodal fusion of attention-weighted features, gating filtering features, and task memory retrieval features, using weighted concatenation or feature addition to generate the final fused feature vector fed into the classifier; the specific fusion method is as follows: This enhances the robustness and discriminative power of feature representation; the enhanced feature map is input into the classifier, and the output is the type of spill.

[0180] The Attention-SNN model is trained sequentially using the training sets of each subset of datasets, training the parameters of each layer of the network. The model is then fine-tuned using the validation set to obtain the trained model. The trained model is then tested using the test sets of all subsets of datasets to obtain the output. The model that passes the test is used for object recognition.

[0181] For each sub-recognition task, the training set is input into the Attention-SNN model described above. After setting the hyperparameters, training is started for 50 rounds. The trained network model is then validated using a validation set. Finally, the weight file with the best detection performance is selected as the weight file of the Attention-SNN model, and the detection weight file is saved.

[0182] During training, the training set is used as the images for each round of training, primarily to determine the parameter variables, such as weights, that the model learns through each round of training. and bias The validation set is used later in the training process to determine the network structure and tune the network model parameters.

[0183] In this embodiment, the hyperparameters are specifically set as follows: image size / pixels is 84×84, batch processing number is 64, the number of iterations per sub-recognition task is 50 rounds, the optimizer is AdamW, the initial learning rate is 0.001, and the weight decay is 0.0001. Data augmentation includes random color distortion, unfolding, cropping, flipping, and resizing operations.

[0184] The hardware system used in this experiment was Ubuntu, the GPU was an NVIDIA RTX 3080 (16G), and the environment used was Python==3.8, torch==1.12.1, torchvision==0.13.1, CUDA==11.6.

[0185] Input the test set into the trained Attention-SNN model to obtain the prediction results after object recognition.

[0186] Based on the above technical solution, this invention can effectively alleviate catastrophic forgetting in the incremental identification scenario of spilled objects, improve the identification accuracy and cross-task knowledge transfer capability of multiple types of spilled objects such as cloud-like objects, particulate matter, block-like objects, strip-like objects, and sheet-like objects, and is suitable for continuous visual perception tasks in dynamic environments and is compatible with neuromorphic hardware deployment.

[0187] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network, characterized in that, include: The pulse feature extraction module is used to receive the input image and extract pulse features; The orthogonal projection module, based on lateral connectivity and the Hebbian learning mechanism, performs orthogonal projection processing on the weight update vector to preserve the knowledge representation of the old task when learning a new task. The attention mechanism module dynamically weights the pulse features to enhance attention to key regions in the image. The task memory matrix stores the feature representations unique to each task and supports cross-task knowledge retrieval. Memory gating networks are used to selectively remember and forget features of historical tasks. The feature fusion module is used to fuse output features from the attention mechanism module, the memory gating network, and the task memory matrix; The classifier module outputs a category prediction result based on the fused features.

2. The system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The system is suitable for incremental recognition of images of spilled objects, including but not limited to five categories: cloud-like objects, particulate matter, block-like objects, strip-like objects, and sheet-like objects.

3. The system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The pulse feature extraction module employs leaky integrated firing neurons, combined with convolutional layers, batch normalization layers, fully connected layers, and pooling layers, with the time step set to... Its specific implementation includes: （1）； in, Indicates the first Time step One input sample data, It is the first A module containing convolutional layers, batch normalization layers, fully connected layers, pooling layers, and LIF neurons. It is network output. The specific implementation of LIF neurons includes: （2）； （3）； in, In time step Time Neuronal membrane potential of the layer In time step Time The first layer The membrane potential of a neuron It is the membrane potential attenuation coefficient. It is the first Layer weights In time step Time The first layer The pulse fired by each neuron It is the membrane potential threshold for the pulse being emitted.

4. The system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The orthogonal projection module extracts principal component subsets based on Hebbian learning and implements orthogonal projection. Specifically, the implementation includes: Principal component subspaces of learned task features are extracted based on lateral connectivity networks, and projection gradients orthogonal to these principal component subspaces are constructed. As shown in the formula below: （4）； in, For the weights of the lateral connection network, It is the identity matrix; Based on the above projection gradient Construct the weight update gradients orthogonal to the principal component subset space of the learned task features. As shown in the formula below: （5）； in, It's an error. It is the trajectory of neuron activity; Weights of lateral connection networks Based on Oja rule updates, the gradient of weight updates is also updated. As shown in the formula below: （6）； in, For lateral connection, positive output. This refers to the learning rate.

5. The system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The attention mechanism module obtains a scaled attention score through a fully connected layer-ReLU activation function-fully connected layer-Sigmoid activation function, which guides the attention allocation of the original input features. The attention score is obtained through the attention subnetwork. Next, input With attention mask Multiplication is performed to achieve weighted summation of specific features. Output features Represented as: （7）。 6. The system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The task memory matrix is ​​an updatable matrix structure, and its retrieval and update mechanism includes: The memory matrix is ​​represented as Each row corresponds to a task feature prototype. ; For input features Retrieve the memory row with the highest similarity: （8）； The task memory matrix uses a momentum update rule: （9）； in, This is the momentum coefficient.

7. The system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The memory-gated network adopts a gated cyclic unit structure, and its implementation and update process includes: Update Gate The update process controls the proportion of retained memory features and the proportion of new features incorporated, and is as follows: （10）； Reset door The system is responsible for filtering memory features and determining which memory features can participate in the generation of the current candidate state. The update process is as follows: （11）； By fusing the filtered memory features with the current input features, candidate states for the current time step are generated. The update process is as follows: （12）； in, This indicates element-wise multiplication, which allows for the filtering of memory features; By updating the door The final output of the GRU is obtained by dynamically balancing historical memory features and candidate hidden states. The update process is as follows: （13）； in, It is the proportion of retained memory features. It is the proportion of the new features that are incorporated. The two are added together to complete the fusion of the memory features and the new features.

8. The system for identifying the incremental amount of spilled material integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The feature fusion module is implemented using a weighted sum method, and the enhanced features after fusion are: （14）。 9. The system for identifying the incremental amount of projectiles integrating multiple modules and a spiking neural network according to claim 1, characterized in that, The classifier module is implemented using a fully connected layer and a Softmax activation function, and outputs the probability of each class: （15）； in, For classifier weights, For classifier bias, This is a feature of fusion.

10. A method for identifying the incremental amount of projectiles integrating multiple modules and a spiking neural network, based on any one of claims 1-9, characterized in that, Includes the following steps: S1. Perform pulse coding on the input projectile image and extract pulse features; S2. Process the pulse characteristics through the orthogonal projection module to protect the learned task knowledge; S3. Input the pulse features sequentially into the attention mechanism module and the memory gating network, and perform knowledge retrieval in conjunction with the task memory matrix. The task memory matrix is ​​updated after each task is completed, adding the feature prototype of the task, and supporting cosine similarity-based retrieval in subsequent tasks. S4. In the feature fusion module, the output features from the attention mechanism module, the memory gating network and the task memory matrix are fused and input into the classifier module to obtain the object category recognition result. S5. When learning a new task, update the lateral connection weights of the orthogonal projection module, the attention mechanism module, the task memory matrix, and the memory gating network parameters, while freezing the relevant parameters of the old task.

Images