Traffic sign recognition method and related apparatus

Abstract

This invention provides a traffic sign recognition method and related apparatus. The method includes: acquiring real-time image data and environmental data collected during vehicle movement; concatenating the preprocessed environmental data into an environmental vector, and generating a modulation vector based on the environmental vector; applying channel attention modulation to a feature map using the modulation vector to obtain a modulated multi-scale image feature map; performing feature recalibration processing on the multi-scale image feature map to obtain a recalibrated feature map; performing traffic sign detection based on the recalibrated feature map and outputting an interpretable detection result for the traffic sign; and performing online incremental learning on the target detection model based on the interpretable detection result according to preset triggering conditions, and collaborating with a cloud server through federated aggregation. Based on this, this invention can intelligently recognize traffic signs, obtain interpretable detection results, and support online incremental learning.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a traffic sign recognition method and related apparatus. Background Technology

[0002] Current traffic sign recognition primarily relies on deep learning object detection models (such as YOLO, SSD, Faster R-CNN, etc.) for identification. However, existing traffic sign recognition methods face multiple challenges in practical deployment: 1. Poor environmental adaptability: The recognition rate drops sharply in complex scenes such as changes in lighting, rain and fog, and low contrast. 2. Missed detection of small targets and long distances: Existing multi-scale fusion methods have insufficient recall rate for extremely small targets (such as long-distance speed limit signs); 3. Fixed model lacks adaptability: The trained model cannot adapt to differences in traffic sign styles in different regions or long-term drift; 4. The recognition results are uninterpretable: they cannot provide a basis for decision-making, making it difficult to meet the safety and reliability requirements of autonomous driving systems. Summary of the Invention

[0003] This invention provides a traffic sign recognition method and related apparatus. It adopts a multimodal environment adaptive modulation mechanism and enhances small target detection through cross-scale feature recalibration. It can output interpretable detection results and supports online incremental learning triggered by multimodality. It forms a closed-loop mechanism for intelligent traffic sign recognition that integrates multimodal perception, dynamic feature modulation, feature recalibration enhancement, interpretable detection, and online incremental learning. This enables the target detection model to adapt to regional differences and long-term data drift without forgetting.

[0004] In a first aspect, embodiments of the present invention provide a traffic sign recognition method, comprising: The system acquires real-time image data and environmental data during vehicle operation, including vehicle speed data, light intensity data, GPS location data, and weather data. The preprocessed environmental data is concatenated into an environmental vector, and a modulation vector is generated based on the environmental vector. The modulation vector is used to perform channel attention modulation on the feature map to obtain a modulated multi-scale image feature map, wherein the feature map is extracted from preprocessed image data; The multi-scale image feature map is subjected to feature recalibration processing to obtain a recalibrated feature map. The feature recalibration processing includes global context aggregation of each feature map layer and cross-scale interaction with the global context information of adjacent feature maps to generate recalibration weights for each feature map layer. The feature maps are then weighted and residual connected according to the recalibration weights. Traffic sign detection is performed based on the recalibrated feature map, and an interpretable detection result of the traffic sign is output. The interpretable detection result includes a semantic heatmap, decision path backtracking information, and calibrated confidence. Based on preset triggering conditions, the target detection model is subjected to online incremental learning based on the interpretable detection results, and is coordinated with the cloud server through federated aggregation.

[0005] In some embodiments, the calculation process of using the modulation vector to perform channel attention modulation on the feature map to obtain the modulated multi-scale image feature map is as follows: F'= F⊙σ(W m ·m + b m ) Wherein, F' is the multi-scale image feature map, F is the feature map, m is the modulation vector, and W... m and b m Here, σ is a learnable parameter, σ is the Sigmoid function, and ⊙ represents channel-by-channel multiplication.

[0006] In some embodiments, the step of performing feature recalibration processing on the multi-scale image feature map to obtain a recalibrated feature map includes: Global average pooling is performed on the i-th layer feature map in the multi-scale image feature map to obtain the global descriptor corresponding to the i-th layer feature map; The global descriptor of the i-th layer feature map is concatenated with the global descriptors of the adjacent layer feature maps to obtain a concatenated vector; The concatenated vector is input into a shared multilayer perceptron (MLP) to generate a recalibrated weight vector corresponding to the i-th layer feature map; The recalibrated weight vector is multiplied channel by channel with the i-th layer feature map to obtain the enhanced feature map; The i-th layer feature map and the enhanced feature map are connected by a weighted residual to obtain the recalibrated feature map.

[0007] In some embodiments, the target detection model includes an operator fusion acceleration unit and a detection head, the detection head including a convolutional layer (Conv), a batch normalization layer (BN), and an activation function layer (ReLU), and the method further includes: The operator fusion acceleration unit fuses the convolutional layer Conv, the batch normalization layer BN, and the activation function layer ReLU into a single operator. Channel pruning is performed on the depthwise separable convolutional layers in the target detection model based on the scaling factor of the batch normalization layer (BN). Different numerical precisions are used to quantize different layers in the target detection model.

[0008] In some embodiments, the computation process of fusing the convolutional layer Conv, the batch normalization layer BN, and the activation function layer ReLU into a single operator through the operator fusion acceleration unit is as follows:

[0009] Where γ, β, μ, and σ are the scaling factor, translation factor, mean, and variance of the batch normalization layer BN, respectively, ε is a minimum constant, and W conv b represents the weights of the convolutional layer Conv. conv The bias of the convolutional layer Conv; The merged single operator is:

[0010] Where y is the fused single operator, W fused b represents the weights after fusion. fused This is the bias after fusion.

[0011] In some embodiments, the preset triggering condition includes at least one of the following: The confidence level of detection is below the threshold for multiple consecutive frames; The sudden change in light intensity exceeds the threshold; The vehicle speed change exceeded the threshold. The GPS area code has changed; The weather type has changed; Vehicle driver markings were misidentified or missed.

[0012] In some embodiments, the online incremental learning employs the Elastic Weight Consolidation (EWC) algorithm, whose loss function is:

[0013] Among them, L new (θ) represents the loss from the new task, F i Let θ be the diagonal element of the Fisher information matrix. old,i λ is an important parameter for the old task, and λ is a tradeoff coefficient.

[0014] Secondly, embodiments of the present invention also provide a traffic sign recognition device, the device comprising: The multimodal perception module is used to acquire real-time image data and environmental data during vehicle operation, wherein the environmental data includes vehicle speed data, light intensity data, GPS location data and weather data; An environment-aware coding module is used to concatenate the preprocessed environmental data into an environment vector and generate a modulation vector based on the environment vector. The dynamic feature modulation module is used to perform channel attention modulation on the feature map using the modulation vector to obtain a modulated multi-scale image feature map, wherein the feature map is extracted from preprocessed image data; The feature recalibration module is used to perform feature recalibration processing on the multi-scale image feature map to obtain a recalibrated feature map. The feature recalibration processing includes performing global context aggregation on each layer of feature map and cross-scale interaction with the global context information of adjacent layer feature maps to generate recalibration weights for each layer of feature map, and performing feature weighting and residual connection on the feature map according to the recalibration weights. The detection and interpretability module is used to detect traffic signs based on the recalibrated feature map and output interpretable detection results of traffic signs, wherein the interpretable detection results include semantic heatmap, decision path backtracking information and calibrated confidence. The online incremental learning module is used to perform online incremental learning on the target detection model based on the interpretable detection results according to preset trigger conditions, and to collaborate with the cloud server through federated aggregation.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the traffic sign recognition method as described in the first aspect.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the traffic sign recognition method as described in the first aspect.

[0017] According to embodiments of the present invention, a traffic sign recognition method and related apparatus are provided. The traffic sign recognition method includes: acquiring real-time image data and environmental data collected during vehicle operation, wherein the environmental data includes vehicle speed data, light intensity data, GPS location data, and weather data; concatenating preprocessed environmental data into an environmental vector, and generating a modulation vector based on the environmental vector; applying channel attention modulation to a feature map using the modulation vector to obtain a modulated multi-scale image feature map, wherein the feature map is extracted from the preprocessed image data; and performing feature recalibration processing on the multi-scale image feature map to obtain a recalibrated feature map. The feature map generation process includes feature recalibration, which involves global context aggregation of each feature map layer and cross-scale interaction with the global context information of adjacent feature maps to generate recalibration weights for each feature map layer. Based on these recalibrated weights, the feature maps are weighted and residual connections are performed. Traffic sign detection is then performed based on the recalibrated feature maps, outputting interpretable detection results. These interpretable results include semantic heatmaps, decision path backtracking information, and calibrated confidence scores. According to preset triggering conditions, the target detection model undergoes online incremental learning based on the interpretable detection results, and collaborates with a cloud server through federated aggregation. Based on this, this embodiment of the invention employs a multimodal environment adaptive modulation mechanism and enhances small target detection through cross-scale feature recalibration, outputting interpretable detection results. It also supports multimodal triggered online incremental learning, forming a closed-loop mechanism for intelligent traffic sign recognition that integrates multimodal perception, dynamic feature modulation, feature recalibration enhancement, interpretable detection, and online incremental learning. This enables the target detection model to adapt to regional differences and long-term data drift without forgetting. Attached Figure Description

[0018] Figure 1 This is a main flowchart of a traffic sign recognition method provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a dynamic feature modulation module provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of the feature recalibration module (FRM) provided in one embodiment of the present invention; Figure 4 This is an overall architecture diagram of a target detection model provided in one embodiment of the present invention; Figure 5 This is a schematic diagram of the mathematical derivation of operator fusion provided in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a traffic sign recognition device provided in one embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0021] In this embodiment of the invention, the terms "furthermore," "exemplarily," or "optionally" are used as examples, illustrations, or descriptions and should not be construed as being more preferred or advantageous than other embodiments or designs. The use of the terms "furthermore," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.

[0022] To facilitate a more convenient description of the working principle of the embodiments of the present invention, the following introduction of relevant technical scenarios is given first.

[0023] Current traffic sign recognition primarily relies on deep learning object detection models (such as YOLO, SSD, Faster R-CNN, etc.) for identification. However, existing traffic sign recognition methods face multiple challenges in practical deployment: 1. Poor environmental adaptability: The recognition rate drops sharply in complex scenes such as changes in lighting, rain and fog, and low contrast. 2. Missed detection of small targets and long distances: Existing multi-scale fusion methods have insufficient recall rate for extremely small targets (such as long-distance speed limit signs); 3. Fixed model lacks adaptability: The trained model cannot adapt to differences in traffic sign styles in different regions or long-term drift; 4. The recognition results are uninterpretable: they cannot provide a basis for decision-making, making it difficult to meet the safety and reliability requirements of autonomous driving systems.

[0024] Based on this, the present invention provides a traffic sign recognition method and related apparatus. The traffic sign recognition method includes: acquiring real-time image data and environmental data collected during vehicle movement, wherein the environmental data includes vehicle speed data, light intensity data, GPS location data, and weather data; concatenating the preprocessed environmental data into an environmental vector, and generating a modulation vector based on the environmental vector; applying channel attention modulation to the feature map using the modulation vector to obtain a modulated multi-scale image feature map, wherein the feature map is extracted from the preprocessed image data; and performing feature recalibration processing on the multi-scale image feature map to obtain a recalibrated feature map, wherein the feature recalibration processing... This involves performing global context aggregation on each feature map layer and cross-scale interaction with the global context information of adjacent feature maps to generate recalibrated weights for each feature map layer. Based on these recalibrated weights, the feature maps are then weighted and residual connections are made. Traffic sign detection is performed based on the recalibrated feature maps, outputting interpretable detection results. These interpretable results include semantic heatmaps, decision path backtracking information, and calibrated confidence scores. According to preset triggering conditions, the target detection model undergoes online incremental learning based on the interpretable detection results, and collaborates with a cloud server through federated aggregation. Based on this, this embodiment of the invention employs a multimodal environment adaptive modulation mechanism and enhances small target detection through cross-scale feature recalibration, outputting interpretable detection results. It also supports multimodal triggered online incremental learning, forming a closed-loop mechanism for intelligent traffic sign recognition encompassing multimodal perception, dynamic feature modulation, feature recalibration enhancement, interpretable detection, and online incremental learning. This enables the target detection model to adapt to regional differences and long-term data drift without forgetting.

[0025] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0026] like Figure 1 As shown, Figure 1 This is a flowchart of a traffic sign recognition method provided in an embodiment of the present invention. The traffic sign recognition method may include, but is not limited to, steps S101 to S106.

[0027] Step S101: Acquire real-time image data and environmental data collected during vehicle operation, including vehicle speed data, light intensity data, GPS location data, and weather data. Step S102: The preprocessed environmental data is concatenated into an environmental vector, and a modulation vector is generated based on the environmental vector. Step S103: Channel attention modulation is performed on the feature map using the modulation vector to obtain the modulated multi-scale image feature map, wherein the feature map is extracted from the preprocessed image data; Step S104: Perform feature recalibration processing on the multi-scale image feature map to obtain the recalibrated feature map. The feature recalibration processing includes global context aggregation of each layer feature map and cross-scale interaction with the global context information of adjacent layer feature maps to generate recalibration weights for each layer feature map. Then, feature weighting and residual connection are performed on the feature map according to the recalibration weights. Step S105: Traffic sign detection is performed based on the recalibrated feature map, and the interpretable detection results of the traffic signs are output. The interpretable detection results include semantic heatmap, decision path backtracking information and calibrated confidence. Step S106: Based on the preset triggering conditions, the target detection model is subjected to online incremental learning based on the interpretable detection results, and is coordinated with the cloud server through federated aggregation.

[0028] It is understood that the traffic sign recognition method of the present invention can be applied to a target detection model, which may include a multimodal perception module, an environment perception coding module, a dynamic feature modulation module, a feature recalibration module, a detection and interpretability module, and an online incremental learning module.

[0029] Understandably, multimodal data acquisition and preprocessing can be achieved through a multimodal sensing module. Multimodal data includes image data and environmental data acquired in real time during vehicle operation. Environmental data includes vehicle speed data v, illumination intensity data l, GPS location data g, and weather data w. Data preprocessing includes, but is not limited to, image denoising (bilateral filtering), illumination correction (histogram equalization), distortion correction (camera calibration parameters), and multi-source data temporal alignment. The multi-source data temporal alignment uses a sliding window synchronization mechanism, taking the nearest neighbor vehicle speed data / illumination intensity data / GPS location data as the reference image frame.

[0030] By fusing multi-source data such as images, vehicle speed, light intensity, GPS, and weather type, and dynamically adjusting model parameters, the target detection model can improve its recognition rate by more than 25% under adverse weather conditions.

[0031] Understandably, the perception and processing of environmental data can be achieved through an environmental perception coding module. For example, the environmental perception coding module concatenates vehicle speed data, light intensity data, GPS location data, and weather data to obtain an environmental vector e=[v, l, g, w], where v is vehicle speed (km / h), l is light intensity (lux), g is the GPS area code (integer), and w is the weather type (one-hot encoding: sunny / rainy / foggy / snow). A modulation vector m is then generated based on the environmental vector, containing a compressed representation of the current environment.

[0032] It is understandable that, such as Figure 2 As shown, dynamic feature modulation can be implemented using a dynamic feature modulation module. Channel attention modulation is applied to the feature map using the modulation vector to obtain a modulated multi-scale image feature map, where the feature map is extracted from preprocessed image data. The specific calculation process of dynamic feature modulation is as follows: F'= F⊙σ(W m ·m + b m ) Where F' is the multi-scale image feature map, F is the feature map, m is the modulation vector, and W is the modulation vector. m and b m Here, σ is a learnable parameter, σ is the Sigmoid function, and ⊙ represents channel-by-channel multiplication.

[0033] Based on this, the target detection model can dynamically adjust the response of each channel according to the current environmental data. For example, it enhances the edge-sensitive channel under low light conditions and enhances the low-frequency channel in rainy or foggy weather, thereby achieving environmental adaptation.

[0034] Understandably, a Feature Recalibration Module (FRM) can be used to recalibrate multi-scale image feature maps to obtain recalibrated feature maps. The feature recalibration process includes global context aggregation of each feature map layer and cross-scale interaction with the global context information of adjacent feature maps to generate recalibration weights for each feature map layer. Finally, feature weighting and residual connections are performed on the feature maps based on these recalibration weights.

[0035] like Figure 3 As shown, the input to the feature recalibration module can be three-layer feature maps P3 (80×80×64), P4 (40×40×128), and P5 (20×20×256). Global average pooling is performed on each feature map layer to obtain the global descriptor for the corresponding feature map. P4 / P5 are then dimensionality-reduced by 1×1 convolution and concatenated with P3 to obtain a concatenated vector. This concatenated vector is input into a shared multilayer perceptron (MLP) to generate the recalibration weight vector for the corresponding feature map. The recalibration weight vector is multiplied channel-by-channel with feature map P3 to obtain the enhanced feature map. A weighted residual concatenation is then performed between feature map P3 and the enhanced feature map to obtain the recalibrated feature map.

[0036] Based on this, through cross-scale interaction, small-scale feature maps can perceive large-scale global information, thereby significantly enhancing the performance of small target detection and greatly improving the target recall rate of small target detection.

[0037] Understandably, traffic sign detection based on recalibrated feature maps outputs interpretable detection results, which can be achieved through a detection and interpretability module. Interpretable detection results can include, but are not limited to, semantic heatmaps, decision path backtracking information, and calibrated confidence scores. The semantic heatmap is generated using Grad-CAM++ to generate class activation maps, overlaid with OCR recognition labels. Grad-CAM++ is a visualization technique for interpreting the decision-making process of deep convolutional neural networks (CNNs). It is an improved version of Grad-CAM, designed to more accurately locate regions in images that play a crucial role in model prediction, especially performing better in complex scenes involving multiple targets, small targets, or repeated occurrences of similar targets. Decision path backtracking information is obtained by constructing a decision tree diagram by recording the neurons with the strongest responses to the feature maps corresponding to each candidate box in the detection head. Confidence score calibration is achieved through temperature scaling techniques in model calibration, resulting in calibrated confidence scores. The generated semantic heatmap and decision path backtracking information can intuitively display the model's areas of interest and judgment logic, supporting fault tracing. It should be noted that interpretable detection results can be integrated into a structured JSON format output.

[0038] Based on this, the traffic sign recognition method of the present invention can make the recognition results transparent and traceable, output interpretable decision-making basis, and meet the interpretability requirements.

[0039] Understandably, based on interpretable detection results and preset trigger conditions, the target detection model undergoes online incremental learning, and collaborates with the cloud server through federated aggregation. This can be achieved through an online incremental learning module. The trigger conditions can be automatic or manual. Automatic trigger conditions include multiple consecutive frames with detection confidence below a threshold, sudden changes in light intensity exceeding a threshold, vehicle speed changes exceeding a threshold, changes in GPS area coding, and changes in weather type. Examples include 10 consecutive frames with confidence <0.6; sudden changes in light intensity >2000 lux; drastic changes in vehicle speed (Δv > 30 km / h); changes in GPS area coding; and changes in weather type. Manual trigger conditions include driver marker misidentification or missed identification.

[0040] Online incremental learning can employ the Elastic Weight Consolidation (EWC) algorithm, whose loss function is:

[0041] Among them, L new (θ) represents the loss from the new task, F i Let θ be the diagonal element of the Fisher information matrix. old,i λ is an important parameter for the old task, and λ is a tradeoff coefficient.

[0042] It should be noted that the object detection model uses an improved EfficientNet-Lite as its backbone network, and is fine-tuned on the traffic sign dataset based on ImageNet pre-training. The object detection model outputs multi-scale feature maps, which are input into the decoupled detection head for classification and regression.

[0043] For the local update strategy of the object detection model, the update scope is limited to the detection head and decoupled branches, while the backbone network remains frozen. It also collaborates with the cloud server through federated aggregation. By combining local incremental learning with the cloud server, the model adapts to new scenarios, significantly reducing the long-term recognition drift rate.

[0044] Based on this, such as Figure 4 As shown, the multimodal perception module collects and aligns multimodal data; the environment vector is converted into a modulation vector by the environment perception encoding module, and the dynamic feature modulation module performs channel attention modulation on the feature map according to the modulation vector to obtain the modulated multi-scale image feature map. The feature recalibration module performs feature recalibration processing on the multi-scale image feature map to obtain the recalibrated feature map; the detection and interpretability module performs traffic sign detection based on the recalibrated feature map and outputs the interpretable detection results of traffic signs; the online incremental learning module updates locally according to the trigger conditions, uploads the gradient to the cloud server for aggregation, and updates the target detection model to form a closed loop.

[0045] Understandably, to reduce the number of parameters in the high-precision model, the traffic sign recognition method of this invention also includes model acceleration to achieve real-time inference (<30ms / frame) of the target detection model on resource-constrained embedded platforms (such as TDA4 and Jetson Orin), thereby meeting the requirements for real-time operation on automotive-grade embedded platforms. This invention employs a three-level optimization strategy of operator fusion, structured pruning, and hybrid precision quantization, enabling an inference speed of 23ms / frame on the TDA4 platform, which is 34.3% faster than the default optimization of TensorRT.

[0046] The object detection model includes an operator fusion acceleration unit and a detection head. The detection head includes a convolutional layer (Conv), a batch normalization layer (BN), and an activation function layer (ReLU). The fusion acceleration specifically includes the following steps: Operator fusion: The operator fusion acceleration unit fuses the Convolutional layer (Conv), the Batch Normalization layer (BN), and the ReLU activation function layer into a single operator; Structured pruning: Channel pruning of depthwise separable convolutional layers in the object detection model based on the scaling factor of batch normalization layer (BN). Hybrid precision quantization: Different numerical precisions are used to quantize different layers in the target detection model.

[0047] Specifically, for operator fusion, the computation process of fusing the convolutional layer (Conv), the batch normalization layer (BN), and the activation function layer (ReLU) into a single operator through the operator fusion acceleration unit is as follows:

[0048] Where γ, β, μ, and σ are the scaling factor, translation factor, mean, and variance of the batch normalization layer (BN), respectively, ε is the minimum constant, and W conv b represents the weights of the convolutional layer Conv. conv This is the bias of the convolutional layer Conv; The merged single operator is:

[0049] Where y is the fused single operator, W fused b represents the weights after fusion. fused This is the bias after fusion.

[0050] The operator fusion acceleration unit proposed in this invention achieves parameter-level fusion based on the mathematical equivalence derivation of Conv+BN+ReLU. Specifically, the original serial Conv+BN+ReLU computation is mathematically derived to fuse the BN parameters into the convolution weights and biases, resulting in a fused single operator, thus achieving equivalent accelerated computation. The derivation process is as follows: Figure 5 As shown.

[0051] Hardware-specific optimizations can also be performed on the TDA4 NPU, including data arrangement, memory access, and operator scheduling. Specifically, the data arrangement optimization strategy is to convert NHWC (last channel) to NCHW (channel first) to adapt to the internal computing units of the NPU. The memory access optimization strategy is to align the convolution kernel weights to 16 bytes and prefetch them via direct memory access (DMA). The operator scheduling optimization strategy is to merge consecutive small convolution kernels into a single task.

[0052] For structured pruning, the pruning targets are the depthwise convolutional layers of each MBConv module in EfficientNet-Lite. Pruning is based on the γ coefficient (scaling factor) of the BN layer. Performance tests show that pruning based on the γ coefficient of the BN layer, through iterative search to determine the pruning threshold for each layer, can reduce the target FLOPs by 40%, decrease the number of parameters from 4.2M to 2.8M, and reduce mAP by <0.5%.

[0053] For mixed-precision quantization, the configuration table is as follows:

[0054] By quantizing different layers in the target detection model with different numerical precisions, high accuracy and low latency of the model can be achieved.

[0055] To verify the effectiveness of the traffic sign recognition method proposed in this invention, performance was tested through experiments.

[0056] Ablation experiments were performed on the Feature Recalibration Module (FRM), and the results are shown in the table below:

[0057] The experimental results in the table above show that FRM further improves mAP and small target AP compared to existing models, but the number of parameters is not significantly different.

[0058] The modulation effects of dynamic feature modulation in different environmental scenarios are shown in the table below:

[0059] As can be seen from the table above, the dynamic feature modulation used in this invention can significantly improve the recognition rate in harsh environments.

[0060] The test results for operator fusion performance (TDA4) are shown in the table below:

[0061] As can be seen from the table above, the operator fusion method used in this invention can further reduce latency.

[0062] The long-term incremental learning effect of the online incremental learning and EWC anti-forgetting mechanism based on multimodal data triggered by this invention is shown in the table below:

[0063] As can be seen from the comparison in the table above, the EWC anti-forgetting mechanism adopted in this invention can effectively prevent forgetting and maintain long-term performance.

[0064] Based on this, the embodiments of the present invention adopt a multimodal environment adaptive modulation mechanism and enhance small target detection through cross-scale feature recalibration, which can output interpretable detection results. At the same time, it supports online incremental learning triggered by multimodality, forming a closed-loop mechanism for intelligent recognition of traffic signs that includes multimodal perception, dynamic feature modulation, feature recalibration enhancement, interpretable detection, and online incremental learning. This enables the target detection model to adapt to regional differences and long-term data drift without forgetting.

[0065] In addition, such as Figure 6 As shown, one embodiment of the present invention also discloses a traffic sign recognition device, which includes: The multimodal perception module 110 is used to acquire real-time image data and environmental data during vehicle operation. The environmental data includes vehicle speed data, light intensity data, GPS location data, and weather data. The environment perception coding module 120 is used to concatenate preprocessed environmental data into an environment vector and generate a modulation vector based on the environment vector. The dynamic feature modulation module 130 is used to perform channel attention modulation on the feature map using a modulation vector to obtain a modulated multi-scale image feature map, wherein the feature map is extracted from preprocessed image data; The feature recalibration module 140 is used to perform feature recalibration processing on multi-scale image feature maps to obtain recalibrated feature maps. The feature recalibration processing includes global context aggregation of each layer feature map and cross-scale interaction with the global context information of adjacent layer feature maps to generate recalibration weights for each layer feature map, and feature weighting and residual connection of the feature maps according to the recalibration weights. The detection and interpretability module 150 is used to detect traffic signs based on the recalibrated feature map and output interpretable detection results of traffic signs. The interpretable detection results include semantic heatmap, decision path backtracking information and calibrated confidence. The online incremental learning module 160 is used to perform online incremental learning of the target detection model based on interpretable detection results according to preset trigger conditions, and to collaborate with the cloud server through federated aggregation.

[0066] The traffic sign recognition device of this invention is used to execute the traffic sign recognition method in the above embodiments. Its specific processing procedure is the same as that of the traffic sign recognition method in the above embodiments, and will not be described in detail here.

[0067] In addition, such as Figure 7 As shown, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; and when the at least one program is executed by the at least one processor 210, it implements the traffic sign recognition method as in any of the preceding embodiments.

[0068] In addition, one embodiment of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing the traffic sign recognition method as described in any of the preceding embodiments.

[0069] The system architecture and application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0070] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0071] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0072] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).

Claims

1. A traffic sign recognition method, applied to a target detection model, the method comprising: The system acquires real-time image data and environmental data during vehicle operation, including vehicle speed data, light intensity data, GPS location data, and weather data. The preprocessed environmental data is concatenated into an environmental vector, and a modulation vector is generated based on the environmental vector. The modulation vector is used to perform channel attention modulation on the feature map to obtain a modulated multi-scale image feature map, wherein the feature map is extracted from preprocessed image data; The multi-scale image feature map is subjected to feature recalibration processing to obtain a recalibrated feature map. The feature recalibration processing includes global context aggregation of each feature map layer and cross-scale interaction with the global context information of adjacent feature maps to generate recalibration weights for each feature map layer. The feature maps are then weighted and residual connected according to the recalibration weights. Traffic sign detection is performed based on the recalibrated feature map, and an interpretable detection result of the traffic sign is output. The interpretable detection result includes a semantic heatmap, decision path backtracking information, and calibrated confidence. Based on preset triggering conditions, the target detection model is subjected to online incremental learning based on the interpretable detection results, and is coordinated with the cloud server through federated aggregation.

2. The method according to claim 1, characterized in that, The calculation process for obtaining the modulated multi-scale image feature map by applying the modulation vector to the feature map using channel attention modulation is as follows: F'= F⊙σ(W m ·m + b m ) Wherein, F' is the multi-scale image feature map, F is the feature map, m is the modulation vector, and W... m and b m Here, σ is a learnable parameter, σ is the Sigmoid function, and ⊙ represents channel-by-channel multiplication.

3. The method according to claim 1, characterized in that, The step of performing feature recalibration processing on the multi-scale image feature map to obtain the recalibrated feature map includes: Global average pooling is performed on the i-th layer feature map in the multi-scale image feature map to obtain the global descriptor corresponding to the i-th layer feature map; The global descriptor of the i-th layer feature map is concatenated with the global descriptors of the adjacent layer feature maps to obtain a concatenated vector; The concatenated vector is input into a shared multilayer perceptron (MLP) to generate a recalibrated weight vector corresponding to the i-th layer feature map; The recalibrated weight vector is multiplied channel by channel with the i-th layer feature map to obtain the enhanced feature map; The i-th layer feature map and the enhanced feature map are connected by a weighted residual to obtain the recalibrated feature map.

4. The method according to claim 1, characterized in that, The target detection model includes an operator fusion acceleration unit and a detection head. The detection head includes a convolutional layer (Conv), a batch normalization layer (BN), and an activation function layer (ReLU). The method further includes: The operator fusion acceleration unit fuses the convolutional layer Conv, the batch normalization layer BN, and the activation function layer ReLU into a single operator. Channel pruning is performed on the depthwise separable convolutional layers in the target detection model based on the scaling factor of the batch normalization layer (BN). Different numerical precisions are used to quantize different layers in the target detection model.

5. The method according to claim 4, characterized in that, The computation process of fusing the convolutional layer (Conv), the batch normalization layer (BN), and the activation function layer (ReLU) into a single operator through the operator fusion acceleration unit is as follows: Where γ, β, μ, and σ are the scaling factor, translation factor, mean, and variance of the batch normalization layer BN, respectively, ε is a minimum constant, and W conv b represents the weights of the convolutional layer Conv. conv The bias of the convolutional layer Conv; The merged single operator is: Where y is the fused single operator, W fused b represents the weights after fusion. fused This is the bias after fusion.

6. The method according to claim 1, characterized in that, The preset triggering conditions include at least one of the following: The confidence level of detection is below the threshold for multiple consecutive frames; The sudden change in light intensity exceeds the threshold. The vehicle speed change exceeded the threshold. The GPS area code has changed; The weather type has changed; Vehicle driver markings were misidentified or missed.

7. The method according to claim 6, characterized in that, The online incremental learning employs the Elastic Weight Consolidation (EWC) algorithm, whose loss function is: Among them, L new (θ) represents the loss from the new task, F i Let θ be the diagonal element of the Fisher information matrix. old,i λ is an important parameter for the old task, and λ is a tradeoff coefficient.

8. A traffic sign recognition device, characterized in that, The device includes: The multimodal perception module is used to acquire real-time image data and environmental data during vehicle operation, wherein the environmental data includes vehicle speed data, light intensity data, GPS location data and weather data; An environment-aware coding module is used to concatenate the preprocessed environmental data into an environment vector and generate a modulation vector based on the environment vector. The dynamic feature modulation module is used to perform channel attention modulation on the feature map using the modulation vector to obtain a modulated multi-scale image feature map, wherein the feature map is extracted from preprocessed image data; The feature recalibration module is used to perform feature recalibration processing on the multi-scale image feature map to obtain a recalibrated feature map. The feature recalibration processing includes performing global context aggregation on each layer of feature map and cross-scale interaction with the global context information of adjacent layer feature maps to generate recalibration weights for each layer of feature map, and performing feature weighting and residual connection on the feature map according to the recalibration weights. The detection and interpretability module is used to detect traffic signs based on the recalibrated feature map and output interpretable detection results of traffic signs, wherein the interpretable detection results include semantic heatmap, decision path backtracking information and calibrated confidence. The online incremental learning module is used to perform online incremental learning on the target detection model based on the interpretable detection results according to preset trigger conditions, and to collaborate with the cloud server through federated aggregation.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the traffic sign recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions for performing the traffic sign recognition method as described in any one of claims 1 to 7.

Images