A machine vision-based traffic sign automatic recognition method and system

Abstract

The present application relates to the technical field of machine vision, and particularly relates to a traffic sign automatic recognition method and system based on machine vision. After detecting that a target traffic sign is blocked or damaged, multi-source image data of different perspectives and different times are loaded, preset category labels are obtained, sign region images are extracted and preliminary feature vectors are obtained through a feature extraction network; combined with spatial position parameter coding as a position vector and splicing, a complete feature representation is output through an attention fusion module; a pre-trained conditional generative reconstruction model is input to generate an unblocked complete sign image; a reconstruction confidence is obtained through quality evaluation, and if the standard is met, the reconstructed image and the category label are output for traffic decision. The integrity, accuracy and robustness of traffic sign recognition in complex road environments are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method and system for automatic recognition of traffic signs based on machine vision. Background Technology

[0002] Traffic signs, as an important component of road infrastructure, play a crucial role in regulating traffic behavior and ensuring driving safety. However, in real-world road environments, traffic signs often face problems such as being obscured by vegetation, aging naturally, and vandalism, resulting in signs being partially or completely invisible and severely impacting the reliability of the identification system.

[0003] Existing traffic sign recognition methods are mainly divided into two categories: one is recognition methods based on improved detection networks (such as CN116665187A), which improves the detection accuracy of small targets and blurred signs by optimizing the structural design of target detection models such as YOLO, but this method cannot recover the semantic information of the occluded parts; the other is defect detection methods based on contour comparison (such as CN119515836A), which uses the SAM model to extract the sign contour and compare it with the standard contour, and can identify defect types such as deformation and dirt, but this method can only output the defect judgment result and cannot accurately reconstruct and restore the content of occluded or damaged signs.

[0004] Therefore, how to overcome the limitations of existing technologies that can only detect but not accurately reconstruct traffic signs, and achieve complete information recovery of obscured or damaged traffic signs, has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention addresses the technical problem of existing technologies being unable to accurately reconstruct and restore obscured or damaged sign content by providing a machine vision-based automatic traffic sign recognition method and system.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0007] In a first aspect, the present invention provides a method for automatic recognition of traffic signs based on machine vision, comprising:

[0008] When a detection result indicates that the target traffic sign is obscured or damaged, the multi-source image data of the target traffic sign is loaded. The multi-source image data includes multiple images containing the target traffic sign collected from different perspectives and at different times.

[0009] Based on the multi-source image data or the detection results, obtain the preset category label of the target traffic sign;

[0010] Extract the region of interest from each image in the multi-source image data to obtain multiple labeled region images;

[0011] The images of the multiple identified regions are input into a feature extraction network to obtain multiple preliminary feature vectors;

[0012] The spatial location parameters of each image are acquired, the spatial location parameters are encoded into a location vector, and concatenated with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information.

[0013] The multiple feature vectors with spatial information are input into the attention fusion module, and through adaptive weighted fusion, the complete feature representation of the target traffic sign is output.

[0014] The complete feature representation and the preset category label are input into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which is then used as the reconstructed sign image.

[0015] The reconstructed sign image is quality evaluated to obtain reconstruction confidence. When the reconstruction confidence is greater than or equal to a preset confidence threshold, the reconstructed sign image and its corresponding preset category label are output for traffic decision-making.

[0016] Optionally, the detection results of whether the target traffic sign is obscured or damaged are obtained through the following methods:

[0017] Obtain the output confidence score of the real-time recognition model for the target traffic sign;

[0018] The original image containing the target traffic sign is segmented to extract the outline of the target traffic sign;

[0019] The outline is compared with a standard identification template to calculate the outline completeness.

[0020] The detection result is generated when the output confidence is lower than a preset recognition threshold and the contour integrity is lower than a preset integrity threshold.

[0021] Optionally, loading the multi-source image data of the target traffic sign includes:

[0022] Based on the geographical location of the target traffic sign, a data request is sent to one or more sensing devices covering the geographical location;

[0023] Receive image data returned by the one or more sensing devices based on the data request, as the multi-source image data;

[0024] The sensing devices include fixed roadside cameras deployed around the geographical location, vehicle-mounted cameras of passing vehicles, adjacent edge computing nodes, and cloud servers storing historical image data.

[0025] Optionally, the multiple identified region images are input into a feature extraction network to obtain multiple preliminary feature vectors, including:

[0026] After adjusting each labeled region image to a uniform size, it is input into a lightweight convolutional neural network for multi-level feature extraction, obtaining feature maps of multiple scales from the intermediate layers of the network.

[0027] Global average pooling is performed on the feature maps at multiple scales to obtain multiple scale-related feature vectors.

[0028] A channel attention mechanism associated with the preset category label of the target traffic sign is introduced to perform channel-dimensional weighted filtering on the feature vectors related to multiple scales, thereby enhancing the semantic features of the sign.

[0029] The multiple scale-related feature vectors, after weighted filtering, are concatenated to obtain the preliminary feature vector corresponding to each identified region image.

[0030] Optionally, the attention fusion module employs a multi-head attention mechanism to perform a weighted summation of the multiple feature vectors with spatial information. The weights are determined by the similarity between each feature vector and the query vector, and the query vector is generated based on the initial estimate of the complete feature representation.

[0031] Optionally, the complete feature representation and the preset category label are input into a pre-trained conditional generative reconstruction model to generate an unoccluded complete sign image corresponding to the target traffic sign, which serves as the reconstructed sign image, including:

[0032] Retrieve the identifier standard template corresponding to the preset category label from the prior knowledge base, which is stored locally on the edge computing node;

[0033] The complete feature representation, the preset category label, and the identification standard template are concatenated into a joint condition vector;

[0034] The randomly sampled latent space noise vector and the joint conditional vector are input together into the generator of the conditional generative reconstruction model. The generator iteratively optimizes the latent space noise vector based on the diffusion mechanism, so that the latent space noise vector gradually approaches the target distribution under the constraint of the joint conditional vector.

[0035] The optimized latent space noise vector is decoded into pixel space to generate an unoccluded complete label image that conforms to the specification requirements of the label standard template and is semantically coherent with the visible part of the complete feature representation.

[0036] Optionally, the reconstructed identifier image is subjected to quality assessment to obtain reconstruction confidence, including:

[0037] The reconstructed image is input into a quality assessment network to obtain an image quality score;

[0038] The reconstructed image is input into multiple independent recognition models with different architectures to obtain multiple recognition results and multiple recognition confidence levels;

[0039] The image quality score is normalized and used as the first component; the proportion of recognition models that are consistent with the majority of the recognition results is used as the second component; and the average or weighted average of the confidence scores of the multiple recognition results is used as the third component.

[0040] The weighted sum of the first, second, and third components is used as the reconstruction confidence level.

[0041] The process involves inputting the reconstructed image into a quality assessment network to obtain an image quality score, including:

[0042] The reconstructed identifier image is aligned pixel-level with the image acquired latest from the multi-source image data.

[0043] Calculate the structural similarity index of the two aligned images in the non-occluded region to obtain the structural similarity components;

[0044] The reconstructed label image is compared with the standard label template corresponding to the preset category label using a color histogram to obtain the color matching component.

[0045] The image quality score is obtained by weighted summation of the structural similarity component and the color matching component.

[0046] Optional, also includes:

[0047] When the reconstruction confidence level is less than the preset confidence threshold, the multi-source image data and the reconstructed identifier image are pushed to the cloud platform to request manual confirmation.

[0048] Receive confirmation information from human feedback and use the confirmation information to optimize the conditional generative reconstruction model.

[0049] Secondly, the present invention provides a machine vision-based automatic traffic sign recognition system, comprising:

[0050] The multi-source image data acquisition module is used to load the multi-source image data of the target traffic sign when a detection result shows that the target traffic sign is obscured or damaged. The multi-source image data includes multiple images containing the target traffic sign collected from different perspectives and at different time points.

[0051] The preset category label acquisition module is used to acquire the preset category label of the target traffic sign based on the multi-source image data or the detection result;

[0052] The region-of-interest (ROI) image acquisition module is used to extract the region of interest from each image in the multi-source image data to obtain multiple region-of-interest (ROI) images.

[0053] The preliminary feature vector acquisition module is used to input the multiple identified region images into the feature extraction network to obtain multiple preliminary feature vectors;

[0054] The feature vector acquisition module is used to acquire the spatial position parameters when acquiring each image, encode the spatial position parameters into a position vector, and concatenate them with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information.

[0055] The complete feature representation acquisition module is used to input the multiple feature vectors with spatial information into the attention fusion module, and output the complete feature representation of the target traffic sign through adaptive weighted fusion;

[0056] The reconstructed sign image acquisition module is used to input the complete feature representation and the preset category label into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which is used as the reconstructed sign image.

[0057] The reconstruction confidence acquisition module is used to perform quality assessment on the reconstructed sign image and obtain reconstruction confidence; when the reconstruction confidence is greater than or equal to a preset confidence threshold, the reconstructed sign image and its corresponding preset category label are output for traffic decision-making.

[0058] By implementing this invention, when a detection result indicates that a target traffic sign is obscured or damaged, multi-source image data of the target traffic sign can be loaded. The multi-source image data includes multiple images containing the target traffic sign collected from different perspectives and at different time points. The redundant information from multiple perspectives and time dimensions is used to compensate for the occlusion defects of a single image, providing sufficient and complementary raw data for subsequent feature fusion and image reconstruction, thus solving the problem of insufficient information from a single image.

[0059] By implementing this invention, it is possible to obtain a preset category label for the target traffic sign based on the multi-source image data or the detection results; to provide clear semantic constraints for the reconstruction model, limit the generation range, avoid meaningless or incorrect category reconstruction, ensure that the reconstructed content conforms to traffic sign specifications, and greatly improve the accuracy and rationality of reconstruction.

[0060] By implementing this invention, it is possible to extract the region of interest from each image in the multi-source image data to obtain multiple labeled region images; eliminate redundant interference information, reduce the amount of subsequent feature extraction and model calculation, allow the algorithm to focus on the labeled core region, improve processing efficiency and reduce the negative impact of background noise on feature expression.

[0061] By implementing this invention, it is possible to input the multiple identified region images into a feature extraction network to obtain multiple preliminary feature vectors; to transform the images into high-dimensional semantic features, retain key information such as shape, texture, and color, while strengthening effective features related to the category and weakening irrelevant features, thus providing high-quality basic features for subsequent fusion.

[0062] By implementing this invention, it is possible to acquire spatial location parameters when acquiring each image, encode the spatial location parameters into a location vector, and concatenate them with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information; to add accurate spatial perspective information to the features, enabling the reconstruction model to understand the geometric relationship between each image, improve the rationality of cross-perspective feature matching and fusion, and enhance the robustness of feature expression in complex scenes.

[0063] By implementing this invention, the multiple feature vectors with spatial information can be input into the attention fusion module, and through adaptive weighted fusion, the complete feature representation of the target traffic sign can be output; clear and effective features are automatically assigned higher weights, while occlusion, blurring, and noise features are weakened, and multi-source information is fully aggregated to obtain a complete, stable, and highly recognizable overall feature expression of the sign.

[0064] By implementing this invention, the complete feature representation and the preset category label can be input into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which serves as the reconstructed sign image. This breaks through the limitation of traditional methods that only detect but do not reconstruct, accurately restores the semantics and appearance of the occluded / damaged parts, generates standardized, complete, and semantically coherent standard signs, and achieves the complete recovery of incomplete information.

[0065] By implementing this invention, the quality of the reconstructed sign image can be evaluated to obtain a reconstruction confidence level. When the reconstruction confidence level is greater than or equal to a preset confidence threshold, the reconstructed sign image and its corresponding preset category label can be output for traffic decision-making. Multi-dimensional quality verification ensures that the output results are reliable and trustworthy, avoiding low-quality reconstruction from misleading decisions such as autonomous driving and traffic control, and improving the overall safety and practical reliability of the system.

[0066] In summary, by implementing this invention, the limitations of existing technologies, which can only detect defects but cannot recover content, can be overcome, significantly improving the integrity, accuracy, and robustness of traffic sign recognition in complex road environments, and stably outputting high-quality traffic sign recognition results that can be directly used for traffic decision-making. Attached Figure Description

[0067] Figure 1 A flowchart illustrating an automatic traffic sign recognition method based on machine vision provided by the present invention;

[0068] Figure 2 This is a schematic diagram of the structure of an automatic traffic sign recognition system based on machine vision provided by the present invention.

[0069] In the attached diagram, the components represented by each number are as follows:

[0070] The module includes a multi-source image data acquisition module 11, a preset category label acquisition module 12, a labeled region image acquisition module 13, a preliminary feature vector acquisition module 14, a feature vector acquisition module 15, a complete feature representation acquisition module 16, a reconstructed labeled image acquisition module 17, and a reconstruction confidence acquisition module 18. Detailed Implementation

[0071] Example 1, as Figure 1 As shown, this embodiment of the invention provides a method and system for automatic traffic sign recognition based on machine vision, including:

[0072] S100 When a detection result indicates that the target traffic sign is obscured or damaged, load the multi-source image data of the target traffic sign, wherein the multi-source image data includes multiple images containing the target traffic sign collected from different perspectives and at different time points;

[0073] S200 obtains the preset category label of the target traffic sign based on the multi-source image data or the detection result;

[0074] S300 extracts the region of interest from each image in the multi-source image data to obtain multiple labeled region images;

[0075] S400 inputs the multiple identified region images into a feature extraction network to obtain multiple preliminary feature vectors;

[0076] S500 acquires the spatial position parameters when acquiring each image, encodes the spatial position parameters into a position vector, and concatenates them with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information.

[0077] S600 inputs the multiple feature vectors with spatial information into the attention fusion module, and outputs the complete feature representation of the target traffic sign through adaptive weighted fusion;

[0078] S700 inputs the complete feature representation and the preset category label into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which is then used as the reconstructed sign image.

[0079] S800 performs a quality assessment on the reconstructed sign image to obtain a reconstruction confidence score. When the reconstruction confidence score is greater than or equal to a preset confidence threshold, the reconstructed sign image and its corresponding preset category label are output for traffic decision-making.

[0080] In step S100 of this application embodiment, the detection result of the target traffic sign being obscured or damaged is obtained in the following way:

[0081] Obtain the output confidence score of the real-time recognition model for the target traffic sign;

[0082] The original image containing the target traffic sign is segmented to extract the outline of the target traffic sign;

[0083] The outline is compared with a standard identification template to calculate the outline completeness.

[0084] The detection result is generated when the output confidence is lower than a preset recognition threshold and the contour integrity is lower than a preset integrity threshold.

[0085] In step S100 of this application embodiment, the purpose of the above step is to accurately determine whether the target traffic sign is occluded or damaged by the dual judgment of confidence and contour integrity output by the real-time recognition model, so as to provide a trigger basis for the subsequent multi-source image reconstruction process.

[0086] To achieve the above objectives, it is first necessary to obtain the output confidence of the real-time recognition model for the target traffic sign;

[0087] In other words, after the real-time recognition model completes the recognition of the target traffic sign, it outputs a numerical value that represents the reliability of the recognition; this value is the output confidence score. For example, after the real-time recognition model recognizes a target traffic sign with a speed limit of 60 km / h, it outputs a confidence score of 0.3.

[0088] The real-time recognition model is a deep learning model used to automatically detect and recognize target traffic signs in the original image. This model takes the original image containing the target traffic sign as input, extracts features, and classifies the image to output the recognition result and corresponding confidence level. This confidence level characterizes the reliability of the recognition result and provides a basis for subsequent judgments on whether the target traffic sign is obscured or damaged.

[0089] The method for acquiring the real-time recognition model can be to pre-train the model based on a large number of traffic sign sample images, and then deploy the trained real-time recognition model in the system. During the detection of occlusion or damage to the target traffic sign, the deployed real-time recognition model is directly invoked to perform forward inference on the input original image containing the target traffic sign, thereby obtaining the confidence level of the model's output for that target traffic sign.

[0090] The specific method for obtaining the real-time recognition model is existing technology and will not be described in detail here.

[0091] The original image containing the target traffic sign is segmented to extract the outline of the target traffic sign;

[0092] This involves using image segmentation techniques to process the original image containing the target traffic sign, separating the edge lines of the target traffic sign from the image background to obtain the outline of the target traffic sign. For example, image segmentation can be performed on an original image containing a speed limit sign of 60km / h that is obscured by tree branches to extract the incomplete outer and inner contour lines of the sign.

[0093] The image segmentation technique described herein is an image processing technique that distinguishes and divides the area containing the target traffic sign from the background area in the original image. Its function is to isolate the pixel region corresponding to the target traffic sign from a complex image, and then extract the outline of the target traffic sign. This technique classifies the features of pixels, locates the boundary of the target traffic sign, retains only the pixels belonging to the target traffic sign, and removes irrelevant background pixels such as road surfaces, trees, and vehicles, providing accurate contour data for subsequent contour integrity calculations. This type of technique is existing technology and will not be described in detail here.

[0094] The outline is compared with a standard identification template to calculate the outline completeness.

[0095] This involves matching the extracted outline of the target traffic sign with the outline of the corresponding standard sign template, and calculating the outline completeness based on the overlap ratio. For example, comparing the outline of an incomplete 60km / h speed limit sign with the outline of a standard 60km / h speed limit sign template yields an outline completeness of 0.4.

[0096] The outline completeness is calculated by matching the extracted target traffic sign outline with the standard sign template outline. First, the outline point sets of the two are aligned and normalized. Then, the number of overlapping outline points is counted. The ratio obtained by dividing the number of overlapping outline points by the total number of outline points in the standard sign template is the outline completeness.

[0097] The detection result is generated when the output confidence is lower than a preset recognition threshold and the contour integrity is lower than a preset integrity threshold.

[0098] If both the output confidence score and the contour integrity score are less than a preset recognition threshold and a preset integrity threshold, respectively, a detection result indicating that the target traffic sign is occluded or damaged can be generated. For example, if the preset recognition threshold is 0.6 and the preset integrity threshold is 0.7, and the current output confidence score of 0.3 is lower than 0.6 and the contour integrity score of 0.4 is lower than 0.7, the conditions are met, and a detection result indicating that the target traffic sign is occluded or damaged can be generated.

[0099] The preset recognition threshold and preset integrity threshold are obtained as follows: First, statistical analysis and experiments are conducted based on a large amount of sample data containing normal, occluded, and damaged target traffic signs to determine the distinction thresholds for the confidence level and contour integrity of the real-time recognition model output. Then, according to the recognition accuracy and false negative rate requirements of the actual application scenario, the above thresholds are adjusted and verified to finally obtain stable and reliable preset recognition thresholds and preset integrity thresholds.

[0100] In step S100 of this application embodiment, loading the multi-source image data of the target traffic sign includes:

[0101] Based on the geographical location of the target traffic sign, a data request is sent to one or more sensing devices covering the geographical location;

[0102] Receive image data returned by the one or more sensing devices based on the data request, as the multi-source image data;

[0103] The sensing devices include fixed roadside cameras deployed around the geographical location, vehicle-mounted cameras of passing vehicles, adjacent edge computing nodes, and cloud servers storing historical image data.

[0104] In step S100 of this application embodiment, the purpose of the above step is to obtain corresponding image data from various types of sensing devices covering the geographical location of the target traffic sign, thereby completing the multi-source image data loading of the target traffic sign and providing a multi-dimensional, multi-source image data foundation for subsequent image reconstruction or status judgment of the target traffic sign.

[0105] To achieve the above objective, it is first necessary to send a data request to one or more sensing devices covering the geographical location of the target traffic sign.

[0106] This involves determining the range of sensing devices capable of capturing or storing images of the target traffic sign based on its geographical location, and then sending a data request, or instruction to acquire image data, to one or more of these sensing devices. For example, if the target traffic sign is located at an intersection on a main urban road, a data request would be sent to multiple sensing devices covering that intersection.

[0107] Next, image data returned by the one or more sensing devices based on the data request is received as the multi-source image data; wherein, the sensing devices include fixed roadside cameras deployed around the geographical location, vehicle-mounted cameras of passing vehicles, adjacent edge computing nodes, and cloud servers storing historical image data.

[0108] This involves receiving image data from sensing devices after receiving a data request, and unifying the image data from different sensing devices into multi-source image data for the target traffic sign. For example, it involves receiving corresponding image data from fixed roadside cameras, onboard cameras of passing vehicles, adjacent edge computing nodes, and cloud servers, and combining these image data into multi-source image data.

[0109] The sensing devices include fixed roadside cameras deployed around the geographical location, vehicle-mounted cameras of passing vehicles, adjacent edge computing nodes, and cloud servers storing historical image data. These devices are distributed around the target traffic sign's geographical location and can collect or store image information of the target traffic sign from different angles and time dimensions, collectively forming a source of multi-source image data. For example, fixed roadside cameras are used to collect real-time frontal images of the target traffic sign from the roadside, vehicle-mounted cameras of passing vehicles are used to collect side images of the target traffic sign while it is in motion, adjacent edge computing nodes are used to cache and return recent images, and the cloud server is used to retrieve and return historical image data.

[0110] Before accessing multi-source image data using the aforementioned sensing devices, device authorization verification and privacy compliance review must be completed. The privacy protection standards for the entire process of data collection, transmission, and storage must be strictly followed. Irrelevant personal information and sensitive data in the images must be desensitized, and only image data related to the target traffic signs must be extracted to ensure the legality and compliance of data use and the security of user privacy.

[0111] S200 obtains the preset category label of the target traffic sign based on the multi-source image data or the detection result;

[0112] In step S200 of this application embodiment, the purpose of the above steps is to determine and obtain the preset category label corresponding to the target traffic sign based on the obtained multi-source image data or the detection results of the target traffic sign being occluded or damaged, so as to provide a category basis for subsequent sign restoration, sign recognition or sign processing of the target traffic sign.

[0113] To achieve the above objectives, it is necessary to use the target traffic sign information carried in the multi-source image data, or the detection results of the target traffic sign being occluded or damaged, as the basis for judgment, and to match and extract the preset category label corresponding to the target traffic sign from the pre-defined category labels. For example, based on the speed limit pattern features presented in the multi-source image data, or based on the detection results of the target traffic sign being occluded or damaged, the preset category label corresponding to the target traffic sign can be obtained as speed limit sign.

[0114] The multi-source image data includes feature information such as the appearance, pattern, color, and shape of the target traffic sign. Based on these features, the corresponding preset category label can be directly matched. The detection result can reflect whether the target traffic sign is obscured or damaged. Combining the known sign attributes, historical data, or location information of the target traffic sign before it was obscured or damaged, the corresponding preset category label can also be determined. Therefore, the preset category label of the target traffic sign can be obtained through the above two channels.

[0115] For example, the target traffic sign information carried by multi-source image data, or the detection results of the target traffic sign being occluded or damaged, needs to be used as the basis for judgment. A preset category label corresponding to the target traffic sign is then matched and extracted from pre-defined category labels. Specifically, the multi-source image data is first used for sign region detection and segmentation to determine the location of the target traffic sign. Then, features are extracted from the visible parts of the sign region. The extracted shape features, pattern features, and color features are compared one by one with various standard sign templates stored in the prior knowledge base. Simultaneously, combined with the detection results of the target traffic sign being occluded or damaged, the interference of occluded areas on feature matching is eliminated. The category corresponding to the standard template with the highest similarity is selected, and the preset category label corresponding to the current target traffic sign is determined and extracted from the pre-defined category labels. For example, based on the speed limit pattern features, circular border features, and color features of white background with red circle and black text presented in multi-source image data, or based on the detection results of the target traffic sign being occluded or damaged, feature matching is performed with the standard speed limit sign template in the prior knowledge base to obtain the preset category label corresponding to the target traffic sign as speed limit sign.

[0116] S300 extracts the region of interest from each image in the multi-source image data to obtain multiple labeled region images;

[0117] In step S300 of this application embodiment, the purpose of the above steps is to extract the region of interest for each image in the multi-source image data, filter out the effective regions containing the target traffic sign, and obtain multiple sign region images, so as to provide an accurate and unified regional image basis for subsequent processing, recognition or reconstruction of the target traffic sign.

[0118] To achieve the above objectives, for each image in the multi-source image data, the region of interest (ROI) related to the target traffic sign is located and extracted. These regions extracted from different images are then used as separate signage region images, resulting in multiple signage region images. For example, if the multi-source image data contains five target traffic sign images from different sensing devices, the ROI containing the target traffic sign is extracted from each image, ultimately resulting in five signage region images.

[0119] The region of interest (ROI) refers to the area in multi-source image data that is related to the target traffic sign and contains its main features. Extracting the ROI for each image involves locating and cropping the valid image portion containing only the target traffic sign from the entire image, while removing irrelevant background and redundant information.

[0120] In step S400 of this embodiment, the multiple identified region images are input into a feature extraction network to obtain multiple preliminary feature vectors, including:

[0121] After adjusting each labeled region image to a uniform size, it is input into a lightweight convolutional neural network for multi-level feature extraction, obtaining feature maps of multiple scales from the intermediate layers of the network.

[0122] Global average pooling is performed on the feature maps at multiple scales to obtain multiple scale-related feature vectors.

[0123] A channel attention mechanism associated with the preset category label of the target traffic sign is introduced to perform channel-dimensional weighted filtering on the feature vectors related to multiple scales, thereby enhancing the semantic features of the sign.

[0124] The multiple scale-related feature vectors, after weighted filtering, are concatenated to obtain the preliminary feature vector corresponding to each identified region image.

[0125] In step S400 of this application embodiment, the purpose of the above step is to input multiple identification area images into the feature extraction network to obtain multiple corresponding preliminary feature vectors, so as to provide a feature foundation with high recognition, multi-scale fusion and semantic information enhancement for subsequent image restoration or recognition of target traffic signs.

[0126] To achieve the above objectives, each labeled region image first needs to be adjusted to a uniform size and then input into a lightweight convolutional neural network for multi-level feature extraction to obtain feature maps of multiple scales from the intermediate layers of the network.

[0127] After adjusting each signage area image to a uniform size, it is input into a lightweight convolutional neural network for multi-level feature extraction, obtaining feature maps at multiple scales from the network's intermediate layers. This step first normalizes the size of all signage area images, then extracts features from different network layers using a lightweight convolutional neural network, resulting in feature maps at multiple scales with different resolutions and information granularities. The uniform size standard is set based on traffic sign standards and the input requirements of the lightweight convolutional neural network, using a fixed pixel size as the unified adjustment standard. This uniform size balances feature extraction accuracy and model computational efficiency, adapting to the multi-scale feature extraction needs of the network's intermediate layers. The uniform size parameter is pre-embedded in the system configuration module, and all signage area images are scaled proportionally and have their boundaries padded according to this fixed standard, ensuring that the image size input to the network is completely consistent, thus guaranteeing the stability and effectiveness of multi-level feature extraction.

[0128] The lightweight convolutional neural network (CNN) is a type of CNN with a small parameter size, low computational cost, and simplified structure. While ensuring effective feature extraction, it reduces computational resource consumption, improves feature extraction speed, and is suitable for traffic sign processing scenarios with high real-time requirements. This network is used to perform multi-level feature extraction on uniformly sized sign area images, outputting feature maps of multiple scales from the network's intermediate layers. For example, in target traffic sign recognition scenarios, using a lightweight CNN allows for rapid feature extraction of sign area images on edge devices while meeting real-time processing requirements.

[0129] For example, the lightweight convolutional neural network can be MobileNetV2, whose core hyperparameters include: a kernel size of 3×3, a dilation rate of 1, a stride of 1, a width multiplier of 0.75, 17 inverted residual blocks, and a ReLU6 activation function. A uniformly sized 256×256 image of the identified region is input into this lightweight convolutional neural network. Multi-level feature extraction is performed through the 17 inverted residual blocks, outputting feature maps at three scales: 128×128, 64×64, and 32×32 from the intermediate layers of layers 8, 12, and 16.

[0130] Next, global average pooling is performed on the multiple scale feature maps respectively to obtain multiple scale-related feature vectors;

[0131] This involves performing global average pooling on the feature map at each scale, compressing the two-dimensional feature map into a one-dimensional vector, and obtaining the feature vector corresponding to each scale. For example, performing global average pooling on feature maps at three different scales will yield three feature vectors, each related to its corresponding scale.

[0132] Then, a channel attention mechanism associated with the preset category label of the target traffic sign is introduced to perform channel-dimensional weighted filtering on the feature vectors related to multiple scales, thereby enhancing the semantic features of the sign.

[0133] Based on preset category labels, a channel attention mechanism is used to assign weights to different channels of the feature vector, thereby filtering and highlighting semantic features related to the target traffic sign. For example, if the preset category label for the target traffic sign is speed limit sign, introducing the corresponding channel attention mechanism will assign higher weights to the channels in the feature vector related to the speed limit sign, thus strengthening the semantic features of the speed limit sign.

[0134] Finally, the multiple scale-related feature vectors after weighted filtering are concatenated to obtain the preliminary feature vector corresponding to each labeled region image.

[0135] This involves concatenating and fusing multiple scale-related feature vectors that have undergone channel-weighted filtering to form a single, more informative vector, which serves as the initial feature vector for the identified region image. For example, concatenating three weighted scale-related feature vectors sequentially yields a preliminary feature vector with higher dimensions and more complete information.

[0136] S500 acquires the spatial position parameters when acquiring each image, encodes the spatial position parameters into a position vector, and concatenates them with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information.

[0137] In step S500 of this application embodiment, the purpose of the above steps is to obtain the spatial location parameters corresponding to each marked area image, convert them into a location vector and fuse them with the preliminary feature vector to obtain a feature vector with spatial information, so as to provide a location constraint basis for subsequent multi-source image feature fusion and traffic sign reconstruction.

[0138] To achieve the above objectives, the spatial location parameters corresponding to each image at the time of acquisition are first obtained. These location-related parameters are then encoded into location vectors that can be fused with image features. These location vectors are then concatenated with the preliminary feature vectors corresponding to the image, ultimately resulting in a feature vector that integrates image features and spatial location information. For example, spatial location parameters such as longitude, latitude, altitude, and shooting angle corresponding to the acquisition of an image are obtained, encoded into location vectors, and then concatenated with the preliminary feature vectors corresponding to the image to obtain a feature vector with spatial information.

[0139] S600 inputs the multiple feature vectors with spatial information into the attention fusion module, and outputs the complete feature representation of the target traffic sign through adaptive weighted fusion;

[0140] In step S600 of this application embodiment, the attention fusion module adopts a multi-head attention mechanism to perform a weighted summation of the multiple feature vectors with spatial information. The weights are determined by the similarity between each feature vector and the query vector. The query vector is generated based on the initial estimate of the complete feature representation.

[0141] In step S600 of this application embodiment, the purpose of the above step is to input multiple feature vectors with spatial information into the attention fusion module, use the multi-head attention mechanism to perform adaptive weighted fusion, and output a complete feature representation that can comprehensively characterize the target traffic sign information, so as to provide a unified and accurate feature basis for the subsequent restoration and reconstruction of the target traffic sign.

[0142] To achieve the above objectives, multiple feature vectors with spatial information need to be input into the attention fusion module first. Through adaptive weighted fusion, the complete feature representation of the target traffic sign is output.

[0143] The attention fusion module adopts a multi-head attention mechanism, which calculates the corresponding weights based on the similarity between each feature vector with spatial information and the query vector, and then performs a weighted summation of multiple feature vectors with spatial information based on the calculated weights to achieve adaptive feature fusion.

[0144] For example, six feature vectors with spatial information from different devices are input into the attention fusion module, and different weights are assigned to them according to their similarity with the query vector. The feature vectors with high similarity are given greater weights, and the complete feature representation of the target traffic sign is output after weighted summation.

[0145] The query vector is a reference vector set up to guide the multi-head attention mechanism in identifying which feature vectors to focus on. It is not randomly generated, but rather it is generated by first performing a preliminary fusion of multiple feature vectors with spatial information to obtain an initial estimate of the complete feature representation of the target traffic sign. The query vector is then generated based on this initial estimate. This allows the attention mechanism to calculate similarity around a reference direction that better aligns with the target traffic sign, making the weighted fusion more reasonable and accurate. For example, an initial estimate can be obtained by averaging multiple feature vectors with spatial information, and then the corresponding query vector can be generated based on this estimate.

[0146] In step S700 of this embodiment, the complete feature representation and the preset category label are input into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which serves as the reconstructed sign image. This includes:

[0147] Retrieve the identifier standard template corresponding to the preset category label from the prior knowledge base, which is stored locally on the edge computing node;

[0148] The complete feature representation, the preset category label, and the identification standard template are concatenated into a joint condition vector;

[0149] The randomly sampled latent space noise vector and the joint conditional vector are input together into the generator of the conditional generative reconstruction model. The generator iteratively optimizes the latent space noise vector based on the diffusion mechanism, so that the latent space noise vector gradually approaches the target distribution under the constraint of the joint conditional vector.

[0150] The optimized latent space noise vector is decoded into pixel space to generate an unoccluded complete label image that conforms to the specification requirements of the label standard template and is semantically coherent with the visible part of the complete feature representation.

[0151] In step S700 of this application embodiment, the purpose of the above steps is to input the complete feature representation and preset category label into the pre-trained conditional generative reconstruction model, and combine it with the locally stored standard identification template to generate a complete, unobstructed identification image that conforms to the specifications and is semantically coherent, so as to provide a usable reconstructed identification image for the target traffic sign that is obstructed or damaged.

[0152] To achieve the above objectives, it is first necessary to retrieve the identification standard template corresponding to the preset category label from the prior knowledge base, which is stored locally on the edge computing node.

[0153] This involves searching for a standard template of an identifier that matches a preset category label from the prior knowledge base stored locally on the edge computing node. For example, if the preset category label is a rate limit identifier, a standard rate limit identifier template would be retrieved from the prior knowledge base on the edge computing node.

[0154] The prior knowledge base is a database stored locally on the edge computing node, used to store standard templates for various target traffic signs. It pre-stores standard traffic sign styles, sizes, colors, patterns, and other specification information corresponding to different preset category labels. When reconstructing the sign image, the corresponding standard template can be retrieved from this database based on the preset category labels, providing specification constraints for the conditional generative reconstruction model and ensuring that the generated unobstructed and complete sign image conforms to the unified standards and style requirements of traffic signs.

[0155] Next, the complete feature representation, the preset category label, and the identification standard template are concatenated into a joint condition vector;

[0156] The complete feature representation, preset category labels, and standard label template are concatenated along the feature dimension to form a unified joint condition vector. For example, concatenating the complete feature representation, speed limit label category labels, and standard speed limit label template yields a joint condition vector used to constrain the generation process.

[0157] Then, the randomly sampled latent space noise vector and the joint condition vector are input into the generator of the conditional generative reconstruction model. The generator iteratively optimizes the latent space noise vector based on the diffusion mechanism, so that the latent space noise vector gradually approaches the target distribution under the constraint of the joint condition vector.

[0158] The generator in the conditional generative reconstruction model is the core component responsible for image generation and reconstruction. It receives randomly sampled latent space noise vectors and a joint conditional vector, and iteratively optimizes the latent space noise vectors using a diffusion mechanism. Under the constraint of the joint conditional vector, the latent space noise vectors gradually approximate the target distribution corresponding to the target traffic sign. The optimized latent space noise vectors are then decoded into pixel space, ultimately outputting an unobstructed, complete sign image that conforms to the standard sign template specification and has semantic coherence with the visible parts.

[0159] The conditional generative reconstruction model is a pre-trained, condition-constraint-oriented traffic sign generation model. This model uses a joint conditional vector composed of complete feature representations, preset category labels, and a standard sign template as the generation basis. Simultaneously, it incorporates randomly sampled latent space noise vectors. An internal generator iteratively optimizes the noise vectors using a diffusion mechanism, making them approximate the target distribution under conditional constraints. This noise vector is then decoded into a pixel-space image, ultimately outputting an unobstructed, complete sign image that conforms to the standard template specifications and has semantic coherence with the visible parts.

[0160] For example, the conditional generative reconstruction model is obtained through pre-training. A training dataset containing various traffic sign samples is constructed, using preset category labels, standard sign templates, traffic sign images with occlusion or damage, and corresponding unoccluded complete sign images as training data to supervise the training of the diffusion-based generative model.

[0161] The parameters of the conditional generative reconstruction model can be set as follows: the generator contains 4 convolutional layers, 2 pooling layers, and 1 decoding layer, with convolutional kernel sizes of 3×3, 3×3, 5×5, and 5×5 respectively, and a stride of 1 for each layer. The decoding layer uses deconvolution and has 3 output channels. The learning rate is 0.001, the batch size is 16, the latent space noise vector dimension is 128, and the diffusion iteration step is 100. The sample size is 5000 images. The training epochs are 200. The convergence criterion is that the similarity between the unoccluded complete labeled image generated by the model and the standard template reaches more than 95%, and the loss value tends to stabilize without significant fluctuations after 10 consecutive training epochs.

[0162] Finally, the optimized latent space noise vector is decoded into pixel space to generate an unoccluded complete label image that conforms to the specification requirements of the label standard template and is semantically coherent with the visible part of the complete feature representation.

[0163] The complete feature representation, preset category labels, and standard identification template are concatenated into a joint conditional vector, which is then input into the generator of the conditional generative reconstruction model along with a randomly sampled latent space noise vector. The generator iteratively optimizes and decodes the data into pixel space based on a diffusion mechanism, ultimately resulting in an unoccluded complete identification image that conforms to the specification requirements of the standard identification template and is semantically coherent with the visible part of the complete feature representation.

[0164] In step S800 of this embodiment, the quality assessment of the reconstructed identifier image is performed to obtain the reconstruction confidence level, including:

[0165] The reconstructed image is input into a quality assessment network to obtain an image quality score;

[0166] The reconstructed image is input into multiple independent recognition models with different architectures to obtain multiple recognition results and multiple recognition confidence levels;

[0167] The image quality score is normalized and used as the first component; the proportion of recognition models that are consistent with the majority of the recognition results is used as the second component; and the average or weighted average of the confidence scores of the multiple recognition results is used as the third component.

[0168] The weighted sum of the first, second, and third components is used as the reconstruction confidence level.

[0169] In step S800 of this application embodiment, the purpose of the above step is to perform multi-dimensional quality assessment on the reconstructed sign image, and to calculate the reconstruction confidence by comprehensively considering the image quality, the recognition results of multiple independent recognition models and the recognition confidence, so as to determine whether the reconstructed sign image meets the usage requirements and to provide a quantitative basis for whether to adopt the reconstructed sign image in the future.

[0170] To achieve the above objectives, the reconstructed image needs to be input into a quality assessment network to obtain an image quality score.

[0171] In step S800 of this embodiment, the reconstructed identifier image is input into a quality evaluation network to obtain an image quality score, including:

[0172] The reconstructed identifier image is aligned pixel-level with the image acquired latest from the multi-source image data.

[0173] Calculate the structural similarity index of the two aligned images in the non-occluded region to obtain the structural similarity components;

[0174] The reconstructed label image is compared with the standard label template corresponding to the preset category label using a color histogram to obtain the color matching component.

[0175] The image quality score is obtained by weighted summation of the structural similarity component and the color matching component.

[0176] In step S800 of this application embodiment, the purpose of obtaining the image quality score is to conduct a multi-dimensional quality evaluation of the reconstructed label image. By comparing it with the multi-source image with the latest acquisition time and the standard label template, an image quality score that can objectively reflect the true quality of the reconstructed label image is obtained, providing a reliable basis for subsequent calculation of reconstruction confidence.

[0177] The first step is to align the reconstructed identifier image with the image acquired most recently in the multi-source image data at the pixel level.

[0178] The reconstructed sign image is matched and calibrated at the pixel level with the original image acquired most recently to ensure that subsequent comparisons are performed in the same spatial location. For example, the reconstructed sign image is pixel-level aligned with the real-time road image acquired most recently, so that the target traffic sign in the two images is at the same pixel position.

[0179] The image with the latest acquisition time refers to the real-time road image whose acquisition time is closest to the current time among all the multi-source image data used in this processing.

[0180] The second step is to calculate the structural similarity index of the two aligned images in the non-occluded region to obtain the structural similarity components.

[0181] This involves calculating a structural similarity index within the unoccluded region after alignment of two images, and using this index as a structural similarity component characterizing structural consistency. For example, a structural similarity index of 0.92 is calculated within the aligned unoccluded region, and this value is used as the structural similarity component.

[0182] The structural similarity index is an evaluation metric used to measure the degree of structural similarity between two images. It is obtained by first aligning the reconstructed image with the image acquired most recently at the pixel level. Then, selecting non-occluded regions from both images, and comparing the brightness, contrast, and structural information within these regions, the structural similarity index is calculated accordingly. A higher index value indicates a closer structural similarity between the reconstructed image and the actual acquired image, resulting in higher reconstruction quality. For example, after comparing brightness, contrast, and structural information, the aligned non-occluded region yielded a structural similarity index of 0.92.

[0183] For example, the corresponding calculation can calculate the mean brightness, variance brightness, and covariance brightness of the two images respectively, and then perform numerical calculations on the three dimensions of brightness, contrast, and structure using similar methods. The numerical calculation results of the three dimensions are weighted and summed to directly obtain the structural similarity index.

[0184] The third step is to perform a color histogram comparison between the reconstructed label image and the standard label template corresponding to the preset category label to obtain the color matching component.

[0185] The color histogram comparison is a method used to measure the consistency of color distribution between two images. First, color histograms are extracted from the reconstructed label image and the standard label template corresponding to the preset category labels, respectively. The percentage of pixels of each color in the images is then calculated, and the distance or similarity between the two histograms is calculated. The closer the values ​​are, the more consistent the color distribution between the reconstructed label image and the standard label template, and the higher the color matching degree. For example, by comparing the color histograms of the red, green, and blue channels of the reconstructed label image and the standard label template, a color matching degree component, such as 0.95, can be obtained.

[0186] The fourth step is to perform a weighted summation of the structural similarity component and the color conformity component to obtain the image quality score.

[0187] For example, to ensure that structural and color information contribute equally to the evaluation and to avoid a single factor from dominating the scoring results, the structural similarity component and the color conformity component are assigned weights of 0.5 and 0.5 respectively. After weighted summation, the image quality score is 0.935.

[0188] Furthermore, the reconstructed image is input into multiple independent recognition models with different architectures to obtain multiple recognition results and multiple recognition confidence levels;

[0189] Reconstruction is imminent

[0190] The labeled image is input into independent recognition models with different structures to obtain the recognition results output by each model and the recognition confidence score, which represents the reliability of the results. For example, inputting the reconstructed labeled image into three independent recognition models with different architectures will yield three recognition results and three recognition confidence scores.

[0191] The multiple independent recognition models with different architectures are obtained by pre-training and then deploying them independently. Three different architectures suitable for traffic sign recognition are selected. The method is simple and meets the requirements of the task, as detailed below:

[0192] Three different architectures were selected: lightweight convolutional neural network, residual network, and convolutional attention network, all of which were pre-trained for target traffic sign recognition scenarios.

[0193] The training samples are consistent with the conditional generative reconstruction model, consisting of unobstructed and damaged / obstructed images of various target traffic signs and standard sign templates, ensuring recognition adaptability.

[0194] After pre-training, the three models with different architectures are deployed independently without interfering with each other. They can then be used as independent recognition models with different architectures to input reconstructed label images to obtain recognition results and recognition confidence.

[0195] The methods for training the lightweight convolutional neural network, residual network, and convolutional attention network based on the training samples are all existing mature technologies and will not be described in detail here.

[0196] Then, the image quality score is normalized and used as the first component, the proportion of the number of recognition models that are consistent with the majority of the recognition results is used as the second component, and the average or weighted average of the confidence scores of the multiple recognitions is used as the third component.

[0197] This involves processing image quality, consistency of recognition results, and recognition confidence separately to obtain three evaluation components. For example, if the normalized image quality score is 0.9, the consistency rate between the two results in the three models is 2 / 3, and the average recognition confidence score is 0.85, these are respectively used as the first, second, and third components.

[0198] Finally, the weighted sum of the first, second, and third components is used as the reconstruction confidence level.

[0199] For example, after assigning appropriate weights to the three components and calculating the weighted sum, the reconstruction confidence level is 0.88. The weights of the three components are set according to the degree of influence of each component on the reconstruction quality, prioritizing a comprehensive and balanced assessment.

[0200] In step S800 of the embodiments of this application, the following is also included:

[0201] When the reconstruction confidence level is greater than or equal to a preset confidence threshold, the reconstructed identification image and its corresponding preset category label are output for traffic decision-making.

[0202] When the reconstruction confidence level is less than the preset confidence threshold, the multi-source image data and the reconstructed identifier image are pushed to the cloud platform to request manual confirmation.

[0203] Receive confirmation information from human feedback and use the confirmation information to optimize the conditional generative reconstruction model.

[0204] In step S800 of this application embodiment, the purpose of the above steps is to perform hierarchical processing on the reconstructed sign image based on the comparison result between the reconstruction confidence level and the preset confidence threshold. If the confidence level requirement is met, it is directly used for traffic decision-making; if not, it is pushed to the cloud platform for manual confirmation. The manual confirmation information is then used to optimize the conditional generative reconstruction model and improve the subsequent reconstruction accuracy.

[0205] To achieve the above objective, when the reconstruction confidence level is greater than or equal to a preset confidence threshold, the reconstructed identification image and its corresponding preset category label should be output for traffic decision-making.

[0206] The reconstructed identification images that are about to reach the reliability standard are directly output along with their corresponding preset category labels, providing a reliable basis for traffic scenarios. The output is displayed on the local interactive interface of the edge computing node, the display interface of the vehicle terminal, and the visualization interface of the roadside traffic management platform. At the same time, the data is synchronously uploaded to the cloud-based traffic decision-making platform for direct access by the autonomous driving system and the traffic signal control system for traffic decision-making.

[0207] For example, if the preset confidence threshold is 0.85 and the reconstruction confidence is 0.9, which is greater than the preset confidence threshold, the reconstructed identification image and preset category label are directly output for traffic decision-making.

[0208] The preset confidence threshold is a critical value used to determine whether the reconstructed signage image is reliable and whether it can be directly used for traffic decision-making. It is obtained through experimental verification and statistical analysis.

[0209] During the validation phase of the generative reconstruction model, a large number of samples with known correct results were used for testing. The preset reliability threshold was gradually adjusted, and the accuracy, false rejection rate, and false acceptance rate of the reconstructed sign image were statistically analyzed under different preset reliability thresholds. The value that makes the three indicators reach the optimal level at the same time and meets the safety requirements of the traffic scenario was selected as the preset reliability threshold.

[0210] When the reconstruction confidence level is less than the preset confidence threshold, the multi-source image data and the reconstructed identifier image are pushed to the cloud platform to request manual confirmation.

[0211] Receive confirmation information from human feedback and use the confirmation information to optimize the conditional generative reconstruction model.

[0212] This involves receiving confirmation information from human feedback and using this information to optimize the conditional generative reconstruction model. The confirmation information includes the correct annotation of the reconstructed signage image, correction results, and a judgment on its reliability. This confirmation information is used as supervision samples to supplement the model's training dataset, iteratively fine-tuning the conditional generative reconstruction model and updating the generator's relevant parameters. By continuously introducing human confirmation information, the accuracy and reliability of the model's reconstructed signage images in occluded and damaged scenarios are continuously improved, making the subsequently output reconstructed signage images more closely resemble the characteristics and specifications of real traffic signs.

[0213] Example 2, as Figure 2 As shown, based on the same inventive concept as the machine vision-based automatic traffic sign recognition method provided in Embodiment 1, this embodiment of the invention also provides a machine vision-based automatic traffic sign recognition system, including:

[0214] The multi-source image data acquisition module 11 is used to load the multi-source image data of the target traffic sign when a detection result of the target traffic sign being obscured or damaged occurs. The multi-source image data includes multiple images containing the target traffic sign collected from different perspectives and at different time points.

[0215] The preset category label acquisition module 12 is used to acquire the preset category label of the target traffic sign based on the multi-source image data or the detection result;

[0216] The region identification image acquisition module 13 is used to extract the region of interest from each image in the multi-source image data to obtain multiple region identification images;

[0217] The preliminary feature vector acquisition module 14 is used to input the multiple identified region images into the feature extraction network to obtain multiple preliminary feature vectors;

[0218] The feature vector acquisition module 15 is used to acquire the spatial position parameters when acquiring each image, encode the spatial position parameters into a position vector, and concatenate them with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information.

[0219] The complete feature representation acquisition module 16 is used to input the multiple feature vectors with spatial information into the attention fusion module, and output the complete feature representation of the target traffic sign through adaptive weighted fusion;

[0220] The reconstructed sign image acquisition module 17 is used to input the complete feature representation and the preset category label into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which is used as the reconstructed sign image.

[0221] The reconstruction confidence acquisition module 18 is used to perform quality assessment on the reconstructed sign image and obtain reconstruction confidence; when the reconstruction confidence is greater than or equal to a preset confidence threshold, the reconstructed sign image and its corresponding preset category label are output for traffic decision-making.

[0222] Furthermore, the multi-source image data acquisition module 11 includes the following execution steps:

[0223] Obtain the output confidence score of the real-time recognition model for the target traffic sign;

[0224] The original image containing the target traffic sign is segmented to extract the outline of the target traffic sign;

[0225] The outline is compared with a standard identification template to calculate the outline completeness.

[0226] The detection result is generated when the output confidence is lower than a preset recognition threshold and the contour integrity is lower than a preset integrity threshold.

[0227] Based on the geographical location of the target traffic sign, a data request is sent to one or more sensing devices covering the geographical location;

[0228] Receive image data returned by the one or more sensing devices based on the data request, as the multi-source image data;

[0229] The sensing devices include fixed roadside cameras deployed around the geographical location, vehicle-mounted cameras of passing vehicles, adjacent edge computing nodes, and cloud servers storing historical image data.

[0230] Furthermore, the preliminary feature vector acquisition module 14 includes the following execution steps:

[0231] After adjusting each labeled region image to a uniform size, it is input into a lightweight convolutional neural network for multi-level feature extraction, obtaining feature maps of multiple scales from the intermediate layers of the network.

[0232] Global average pooling is performed on the feature maps at multiple scales to obtain multiple scale-related feature vectors.

[0233] A channel attention mechanism associated with the preset category label of the target traffic sign is introduced to perform channel-dimensional weighted filtering on the feature vectors related to multiple scales, thereby enhancing the semantic features of the sign.

[0234] The multiple scale-related feature vectors, after weighted filtering, are concatenated to obtain the preliminary feature vector corresponding to each identified region image.

[0235] Furthermore, the reconstructed identifier image acquisition module 17 includes the following execution steps:

[0236] Retrieve the identifier standard template corresponding to the preset category label from the prior knowledge base, which is stored locally on the edge computing node;

[0237] The complete feature representation, the preset category label, and the identification standard template are concatenated into a joint condition vector;

[0238] The randomly sampled latent space noise vector and the joint conditional vector are input together into the generator of the conditional generative reconstruction model. The generator iteratively optimizes the latent space noise vector based on the diffusion mechanism, so that the latent space noise vector gradually approaches the target distribution under the constraint of the joint conditional vector.

[0239] The optimized latent space noise vector is decoded into pixel space to generate an unoccluded complete label image that conforms to the specification requirements of the label standard template and is semantically coherent with the visible part of the complete feature representation.

[0240] Furthermore, the confidence acquisition module 18 for reconstruction includes the following execution steps:

[0241] The reconstructed image is input into a quality assessment network to obtain an image quality score;

[0242] The reconstructed image is input into multiple independent recognition models with different architectures to obtain multiple recognition results and multiple recognition confidence levels;

[0243] The image quality score is normalized and used as the first component; the proportion of recognition models that are consistent with the majority of the recognition results is used as the second component; and the average or weighted average of the confidence scores of the multiple recognition results is used as the third component.

[0244] The weighted sum of the first, second, and third components is used as the reconstruction confidence level.

[0245] The process involves inputting the reconstructed image into a quality assessment network to obtain an image quality score, including:

[0246] The reconstructed identifier image is aligned pixel-level with the image acquired latest from the multi-source image data.

[0247] Calculate the structural similarity index of the two aligned images in the non-occluded region to obtain the structural similarity components;

[0248] The reconstructed label image is compared with the standard label template corresponding to the preset category label using a color histogram to obtain the color matching component.

[0249] The image quality score is obtained by weighted summation of the structural similarity component and the color matching component.

[0250] This also includes:

[0251] When the reconstruction confidence level is greater than or equal to a preset confidence threshold, the reconstructed identification image and its corresponding preset category label are output for traffic decision-making.

[0252] When the reconstruction confidence level is less than the preset confidence threshold, the multi-source image data and the reconstructed identifier image are pushed to the cloud platform to request manual confirmation.

[0253] Receive confirmation information from human feedback and use the confirmation information to optimize the conditional generative reconstruction model.

Claims

1. A method for automatic recognition of traffic signs based on machine vision, characterized in that, include: When a detection result indicates that the target traffic sign is obscured or damaged, the multi-source image data of the target traffic sign is loaded. The multi-source image data includes multiple images containing the target traffic sign collected from different perspectives and at different times. Based on the multi-source image data or the detection results, obtain the preset category label of the target traffic sign; Extract the region of interest from each image in the multi-source image data to obtain multiple labeled region images; The images of the multiple identified regions are input into a feature extraction network to obtain multiple preliminary feature vectors; The spatial location parameters of each image are acquired, the spatial location parameters are encoded into a location vector, and concatenated with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information. The multiple feature vectors with spatial information are input into the attention fusion module, and through adaptive weighted fusion, the complete feature representation of the target traffic sign is output. The complete feature representation and the preset category label are input into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which is then used as the reconstructed sign image. The reconstructed signage image is subjected to quality assessment to obtain reconstruction confidence. When the reconstruction confidence is greater than or equal to a preset confidence threshold, the reconstructed signage image and its corresponding preset category label are output for traffic decision-making. Specifically, the complete feature representation and the preset category label are input into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which serves as the reconstructed sign image, including: Retrieve the identifier standard template corresponding to the preset category label from the prior knowledge base, which is stored locally on the edge computing node; The complete feature representation, the preset category label, and the identification standard template are concatenated into a joint condition vector; The randomly sampled latent space noise vector and the joint conditional vector are input together into the generator of the conditional generative reconstruction model. The generator iteratively optimizes the latent space noise vector based on the diffusion mechanism, so that the latent space noise vector gradually approaches the target distribution under the constraint of the joint conditional vector. The optimized latent space noise vector is decoded into pixel space to generate an unoccluded complete label image that conforms to the specification requirements of the label standard template and is semantically coherent with the visible part of the complete feature representation.

2. The automatic traffic sign recognition method based on machine vision as described in claim 1, characterized in that, The detection results of the target traffic sign being obscured or damaged are obtained through the following methods: Obtain the output confidence score of the real-time recognition model for the target traffic sign; The original image containing the target traffic sign is segmented to extract the outline of the target traffic sign; The outline is compared with a standard identification template to calculate the outline completeness. The detection result is generated when the output confidence is lower than a preset recognition threshold and the contour integrity is lower than a preset integrity threshold.

3. The automatic traffic sign recognition method based on machine vision as described in claim 1, characterized in that, Loading the multi-source image data of the target traffic sign includes: Based on the geographical location of the target traffic sign, a data request is sent to one or more sensing devices covering the geographical location; Receive image data returned by the one or more sensing devices based on the data request, as the multi-source image data; The sensing devices include fixed roadside cameras deployed around the geographical location, vehicle-mounted cameras of passing vehicles, adjacent edge computing nodes, and cloud servers storing historical image data.

4. The automatic traffic sign recognition method based on machine vision as described in claim 1, characterized in that, The images of the multiple identified regions are input into a feature extraction network to obtain multiple preliminary feature vectors, including: After adjusting each labeled region image to a uniform size, it is input into a lightweight convolutional neural network for multi-level feature extraction, obtaining feature maps of multiple scales from the intermediate layers of the network. Global average pooling is performed on the feature maps at multiple scales to obtain multiple scale-related feature vectors. A channel attention mechanism associated with the preset category label of the target traffic sign is introduced to perform channel-dimensional weighted filtering on the feature vectors related to multiple scales, thereby enhancing the semantic features of the sign. The multiple scale-related feature vectors, after weighted filtering, are concatenated to obtain the preliminary feature vector corresponding to each identified region image.

5. The automatic traffic sign recognition method based on machine vision as described in claim 1, characterized in that, The attention fusion module employs a multi-head attention mechanism to perform a weighted summation of the multiple feature vectors with spatial information. The weights are determined by the similarity between each feature vector and the query vector, and the query vector is generated based on the initial estimate of the complete feature representation.

6. The automatic traffic sign recognition method based on machine vision as described in claim 1, characterized in that, The quality of the reconstructed image is evaluated to obtain reconstruction confidence, including: The reconstructed image is input into a quality assessment network to obtain an image quality score; The reconstructed image is input into multiple independent recognition models with different architectures to obtain multiple recognition results and multiple recognition confidence levels; The image quality score is normalized and used as the first component; the proportion of recognition models that are consistent with the majority of the recognition results is used as the second component; and the average or weighted average of the confidence scores of the multiple recognition results is used as the third component. The weighted sum of the first, second, and third components is used as the reconstruction confidence level.

7. The automatic traffic sign recognition method based on machine vision as described in claim 6, characterized in that, The reconstructed image is input into a quality assessment network to obtain an image quality score, including: The reconstructed identifier image is aligned pixel-level with the image acquired latest from the multi-source image data. Calculate the structural similarity index of the two aligned images in the non-occluded region to obtain the structural similarity components; The reconstructed label image is compared with the standard label template corresponding to the preset category label using a color histogram to obtain the color matching component. The image quality score is obtained by weighted summation of the structural similarity component and the color matching component.

8. The automatic traffic sign recognition method based on machine vision as described in claim 1, characterized in that, Also includes: When the reconstruction confidence level is less than the preset confidence threshold, the multi-source image data and the reconstructed identifier image are pushed to the cloud platform to request manual confirmation. Receive confirmation information from human feedback and use the confirmation information to optimize the conditional generative reconstruction model.

9. A machine vision-based automatic traffic sign recognition system, characterized in that, The system is used to implement the automatic traffic sign recognition method based on machine vision as described in any one of claims 1-8, and the system includes: The multi-source image data acquisition module is used to load the multi-source image data of the target traffic sign when a detection result shows that the target traffic sign is obscured or damaged. The multi-source image data includes multiple images containing the target traffic sign collected from different perspectives and at different time points. The preset category label acquisition module is used to acquire the preset category label of the target traffic sign based on the multi-source image data or the detection result; The region-of-interest (ROI) image acquisition module is used to extract the region of interest from each image in the multi-source image data to obtain multiple region-of-interest (ROI) images. The preliminary feature vector acquisition module is used to input the multiple identified region images into the feature extraction network to obtain multiple preliminary feature vectors; The feature vector acquisition module is used to acquire the spatial position parameters when acquiring each image, encode the spatial position parameters into a position vector, and concatenate them with the corresponding preliminary feature vector to obtain multiple feature vectors with spatial information. The complete feature representation acquisition module is used to input the multiple feature vectors with spatial information into the attention fusion module, and output the complete feature representation of the target traffic sign through adaptive weighted fusion; The reconstructed sign image acquisition module is used to input the complete feature representation and the preset category label into a pre-trained conditional generative reconstruction model to generate an unobstructed complete sign image corresponding to the target traffic sign, which is used as the reconstructed sign image. The reconstruction confidence acquisition module is used to perform quality assessment on the reconstructed sign image and obtain reconstruction confidence; when the reconstruction confidence is greater than or equal to a preset confidence threshold, the reconstructed sign image and its corresponding preset category label are output for traffic decision-making.

Images