Method, system and storage medium for extracting speed, height and weight limit information from traffic signs

Abstract

The present invention discloses a method for extracting speed, height, and weight limit information from traffic signs, comprising the following steps: 1. obtaining road network data and sampling at equal intervals to obtain a panoramic view of the road, and cropping the panoramic view; 2. extracting and fusing existing traffic sign data using a public traffic dataset to construct a training set; 3. locating traffic signs from street scenes; and 4. extracting speed, height, and weight limit information from traffic signs. The present invention also discloses a system and storage medium for extracting speed, height, and weight limit information from traffic signs. The above method, system, and storage medium can accurately extract information from traffic signs that appear less frequently, especially from various types of weight limit traffic signs.

Description

Technical Field

[0001] The present invention relates to an image information extraction method, system and storage medium, and in particular to a traffic sign speed limit, height limit and weight limit information extraction method, system and storage medium. Background Art

[0002] With the continuous development and application of automatic traffic sign recognition, the update and verification of traffic sign attribute information has become an inevitable trend of future development. One of the key technologies that needs to be solved to achieve the update and verification of traffic sign attribute information is the problem of traffic sign information extraction.

[0003] Traffic sign information extraction is widely used in the transportation sector. As a crucial component of road safety, it serves to remind and warn drivers to drive safely. However, current traffic sign information extraction methods are not accurate enough for complex road signs. Weight restriction signs, in particular, are often of various types, making information extraction more challenging than speed limit signs, which tend to be more monotonous.

[0004] In addition, existing research on traffic sign information extraction faces other challenges. For example, some traffic signs appear less frequently, making it difficult to accurately segment and extract information using recognition algorithms based on limited samples. Accurate information extraction from traffic signs is also difficult to guarantee in scenarios with varying lighting conditions, obstructions, or sign fading. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method, system and storage medium for extracting speed, height and weight limit information from traffic signs, which can improve the accuracy of information extraction.

[0006] Technical solution: The method for extracting speed limit, height limit and weight limit information from traffic signs of the present invention comprises the following steps:

[0007] Step 1: Obtain road network data and sample at equal intervals to obtain a panoramic image of the road and crop the panoramic image;

[0008] Step 2: Use the public traffic dataset to extract and integrate existing traffic sign data to construct a training set;

[0009] Step 3: Locate traffic signs from street view;

[0010] Step 4: Extract the speed limit, height limit and weight limit information from traffic signs.

[0011] Wherein, the step 1 includes the following sub-steps:

[0012] Step 11: Use the tool to obtain the road network data that meets the query conditions in the OSM database and save it;

[0013] Step 12: Using the tool, generate equally spaced sampling points along the geometry of the road network layer;

[0014] Step 13: Get the panoramic ID of the coordinate point corresponding to the sampling point, and download the panoramic image of the corresponding coordinate point;

[0015] Step 14: Crop the panorama.

[0016] The step 14 specifically includes: retaining the middle 1 / 3 of the panoramic image in the height direction, and cropping the retained portion in sequence along the horizontal direction with a set width value and an overlap rate of 0.15, wherein the last image is cropped to the left with the set width value based on the right edge of the panoramic image.

[0017] The step 2 includes the following sub-steps:

[0018] Step 21: Extract samples from the TT100K and MTSD datasets and fuse them with the existing traffic sign data to fuse each sub-category with detailed values;

[0019] Step 22: According to the category table provided by Mapillary, traverse the training set and validation set ID files in the annotation folder, find their annotation files based on the ID index, and retain the traffic sign samples in the annotation files that meet the Chinese traffic sign specifications;

[0020] Step 23: Merge the individual sample annotation files into the COCOjson annotation format, and then fuse them with the samples extracted from TT100K to form a training set;

[0021] The step 3 includes the following sub-steps:

[0022] Step 31: Evaluate the recognition performance of various object recognition algorithms on the fused training set;

[0023] Step 32: Based on the recognition performance of the target recognition algorithm, the Faster R-CNN algorithm based on Swin-Transformer is selected to extract and locate traffic signs;

[0024] Step 33: Based on the evaluation index mAP (mean average precision), perform a final performance evaluation on each recognition algorithm.

[0025] The step 32 further includes: selecting the YOLOV5 algorithm to locate the traffic sign containing both speed limit information and weight limit information through the online sample enhancement function.

[0026] The extraction of the weight limit information of traffic signs includes the following steps:

[0027] Step 41: Extract the attribute information of the weight limit sign, crop the located traffic sign, and save the relevant metadata and detection output information in the form of a file name;

[0028] Step 42: For circular weight limit traffic signs that only contain numerical and alphabetical text, use an optical character recognition model to obtain the coordinates (x1, y1) of the center point of the text box, and at the same time use an image segmentation method to obtain the coordinates (x2, y2) of the center of the sign. If the difference in the x-coordinate or y-coordinate of the two center points is greater than 2 pixels, output it as a type-one weight limit sign, otherwise discard it;

[0029] Step 43: For weight limit traffic signs that contain both numerical text and literal text, classify them according to their attribute information characteristics. One category is a combination of a prohibition sign and a literal sign, and the other category is a combination of a speed limit sign and a literal sign. After classification, use an optical character recognition model to perform optical character recognition on the information indicating the weight limit in the literal sign;

[0030] Step 431: If the optical character recognition result contains the character "以" and does not contain any of the numbers "90", "60", or "100", extract the result and save it as a type-two weight limit sign;

[0031] Step 432: If the optical character recognition result contains both the number "90" and any Chinese character, extract the result and save it as a type-three weight limit sign.

[0032] Step 41 specifically includes: Cropping the traffic signs located by the Faster R-CNN algorithm based on Swin-Transformer, only retaining the image within the target detection rectangle detection frame, and saving the relevant metadata and detection output information in the form of a file name; Cropping the traffic signs located by the YOLO V5 algorithm, retaining the image that is expanded to 6 times the size centered on the target detection rectangle detection frame, and saving the relevant metadata and detection output information in the form of a file name.

[0033] The traffic sign speed limit, height limit, and weight limit information extraction system described in the present invention is characterized by including:

[0034] An acquisition and processing module, used to acquire OSM road network data and the corresponding panoramic images, and crop the panoramic images in terms of width and height;

[0035] A production module, used to extract samples from the public dataset and fuse the existing traffic sign data, and fuse each detailed classification category with detailed numerical values to form a training set;

[0036] Positioning module, used to detect speed limit, height limit and weight limit traffic signs and locate their location;

[0037] Extraction module, used to extract attribute information such as text on speed limit, height limit and weight limit traffic signs;

[0038] The computer-readable storage medium of the present invention stores instructions, which, when executed, enable the computer to execute the above-mentioned method for extracting speed, height and weight limit information from traffic signs.

[0039] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: 1. It can accurately extract information from traffic signs that appear less frequently, especially various types of weight-limit traffic signs; 2. Traffic sign information is extracted and investigated based on road network images, which can monitor speeding and overloaded vehicles on the road, issue early warnings and take measures in time to reduce the occurrence of traffic accidents and improve road traffic safety. 3. The present invention adopts the idea of first detecting traffic signs and then extracting numerical values to extract road height, speed and weight limit attribute information, which supports the update of traffic sign data asset management data, and also provides a new data source for multi-source verification of high-precision map related attributes in response to the current situation that map service providers do not disclose traffic sign attribute information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the method for extracting speed, height and weight limit information from traffic signs according to the present invention;

[0041] Figure 2 This is a schematic diagram of the OSM road network selection according to the present invention;

[0042] Figure 3 This is a schematic diagram of the process of extracting height and speed limit information according to the present invention;

[0043] Figure 4 This is a schematic diagram of the weight limit information extraction process of the present invention;

[0044] Figure 5 This is a schematic diagram of the functional hierarchy of the traffic sign speed limit, height limit and weight limit information extraction system described in the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0046] like Figure 1As shown, the process of the method for extracting speed, height and weight limit information on traffic signs described in the present invention includes step 1: obtaining road network data and sampling at equal intervals, obtaining a panoramic view of the road, and cropping the panoramic view; step 2: extracting and fusing existing traffic sign data using a public traffic data set to construct a training set; step 3: locating traffic signs from street scenes; step 4: extracting speed, height and weight limit information on traffic signs. Specifically, tools can be used to obtain OSM road network data and sample at equal intervals, crawl the panoramic ID to download the panoramic view, and crop the height and length of the panoramic view. Extract and fuse existing traffic sign data using a public traffic data set to construct a training sample set. Use the Faster R-CNN algorithm based on Swin-Transformer and the YOLO V5 algorithm to locate signs from street scenes. Use the PP-OCRv3 text recognition model and image segmentation technology to extract speed, height and weight limit attribute information;

[0047] like Figure 2 As shown, step 1 can refer to the map feature classification table in OSM, select the highway network field (key = highway) as four types of data: main road (value = primary), expressway (value = motorway), urban expressway (value = trunk) and secondary road (value = secondary), quickly obtain it through the QuickOSM plug-in of QGIS, and query the road network data that meets the query conditions in the OSM database through the "https: / / lz4.overpass-api.de / api / " UR, and save the query results as a file in geojson format to facilitate subsequent equal-interval sampling.

[0048] After obtaining OSM road network data, the process of obtaining street view images is described as follows:

[0049] The "Points along geometry" function of the software QGIS is used to generate sampling points with equal intervals of 10 meters along the geometric shape of the road network layer; the Google developer key that supports the Street View API is obtained, a GET request is sent to the Google Street View API, the sampling point data is traversed through a Python program, and a URL for requesting the Google API is constructed based on the latitude and longitude of the sampling point and the obtained Google developer key value. A field with the same name is added to the geojson file, and then associated with the geographic information of each sampling point, and the panoramic ID of the corresponding coordinate point is obtained. The panoramic image of the corresponding point location is downloaded using the Street View Download360 software.

[0050] The specific process of cropping a panorama is as follows:

[0051] The middle 1 / 3 of the panoramic image in the height direction is retained; the retained portion is cropped in sequence along the horizontal direction with a set width value and an overlap rate of 0.15; the last image is based on the right edge of the panoramic image and cropped to the left with a set width value to ensure that a rectangular street scene to be identified is formed.

[0052] Traffic signs include speed limit, height limit, and three types of weight limit signs. For step 2, we use the public traffic dataset to extract and fuse the existing speed limit, height limit, and weight limit data. The steps to construct the training set are as follows:

[0053] We extracted and fused existing speed and height limit data from the TT100K and MTSD datasets, and fused each sub-category with detailed values, namely, speed limit, height limit, and weight limit.

[0054] According to the category table provided by Mapillary, we selected speed limit samples, height limit samples, and weight limit samples that meet the Chinese traffic sign specifications for fusion. The specific method of sample fusion is to traverse the training set (train.txt) and validation set (val.txt) ID files in the annotation folder, find their annotation files based on the ID index, and retain the samples in the annotation files that meet the speed limit or height limit category specifications;

[0055] Merge the individual sample annotation files into the COCOjson annotation format and then fuse them with the training set samples extracted from TT100K.

[0056] Step 3 involves evaluating the recognition performance of four typical object recognition algorithms on the fused test set. Specifically, they are ResNet101-FPN-based Faster R-CNN, Swin-Transformer-based Faster R-CNN, YOLOv5, and Deformable DETR. These object recognition algorithms comprehensively cover various recognition methods, including two-stage, single-stage, and Transformer-based methods.

[0057] All of the above models were trained and tested in a Linux environment with two NVIDIA 2080ti GPUs. Faster R-CNN based on ResNet101-FPN, Faster R-CNN based on Swin-Transformer, and Deformable DETR were implemented on the MMdetection software framework, and YOLOv5 was implemented based on the MMYOLO framework. All models used the Adam algorithm as the optimizer and were trained for 50 epochs.

[0058] Based on comprehensive performance, the Faster R-CNN algorithm based on Swin-Transformer was selected for traffic sign extraction and location. Specifically, for traffic signs containing both speed and weight limit information, the YOLO V5 algorithm was selected, leveraging its online sample enhancement feature to locate complex, low-frequency, and small sample sizes of traffic signs.

[0059] Finally, the performance of each recognition algorithm was evaluated based on the evaluation indicator mAP (mean average precision) in the COCO object detection dataset.

[0060] For step 3, this paper designs a complete processing flow that can call deep learning frameworks on demand, such as the YOLO v5 algorithm and the Faster R-CNN algorithm based on the Swin-Transformer. The following describes and implements these two methods in detail.

[0061] First, let's introduce the YOLO v5 algorithm. YOLO v5 is an object detection algorithm whose model structure mainly includes the following parts: Input: The Head network consists of three different output layers, responsible for detecting large, medium, and small-scale objects respectively. Backbone network: Using CSPDarknet53 as the backbone network, it has powerful feature extraction capabilities and efficient computing performance. Neck network: Using the FPN (Feature Pyramid Network) structure, it can detect at different feature map levels and fuse information from each layer to improve object detection performance. Output: The Focal Loss loss function is used to alleviate the problem of class imbalance in object detection, thereby improving model performance. At the same time, overlapping object boxes are processed through non-maximum suppression (NMS) to obtain the final detection result. In terms of activation function, YOLOv5 uses the Mish activation function, which is an alternative to ReLU and can further improve model performance.

[0062] First, the input side mainly consists of mosaic image enhancement, adaptive anchor box calculation, and adaptive image scaling. Mosaic image enhancement is a commonly used data enhancement method for target detection. New training images are generated by combining multiple different images, with the position and size of each small image being random. For each target in the large image, its position and size in the large image need to be recalculated, and the new position and size of each target in the large image need to be annotated, and new annotation information is generated. This method increases the diversity and complexity of the training set, helping to improve the robustness and generalization ability of the target detection model. In addition, mosaic image enhancement can effectively reduce the risk of overfitting and improve the training effect of the model.

[0063] Adaptive anchor box calculation aims to automatically calculate the anchor box parameters that best suit the input image through learning without manual setting. YOLO v5 uses an adaptive anchor box calculation method called ATSS (Adaptive Training Sample Selection). The core of this method is to automatically select positive and negative samples to improve the performance of the detector. The main goal of ATSS is to select appropriate positive and negative samples in an adaptive manner instead of relying on fixed manual thresholds. This method helps improve detection accuracy when dealing with objects of different scales. The specific steps are as follows:

[0064] (1) Calculate candidate anchor boxes: For each target box, calculate the IoU values of all anchor boxes and select the top k anchor boxes with the highest IoU as candidate anchor boxes;

[0065] (2) Calculate the center metric: Calculate the Euclidean distance between the center point of each candidate anchor box and the center point of the target box;

[0066] (3) Calculate the adaptive threshold: Calculate the mean and standard deviation of the distances between the center points of all candidate anchor boxes, and use these values to calculate the adaptive threshold;

[0067] thureshold=μ+σ

[0068] Where μ is the mean of the distances and σ is the standard deviation of the distances.

[0069] (4) Select positive and negative samples: Use an adaptive threshold to select positive samples, that is, those anchor boxes whose distance is less than or equal to the threshold are considered positive samples, and the rest are considered negative samples;

[0070] Adaptive Image Scaling is an image scaling method based on the target scale. It can adaptively adjust the size of the input image to better detect objects of different scales. Its specific operation is as follows:

[0071] (1) Preprocessing: The input image is scaled to a reference size. If the image size is smaller than the reference size, it is enlarged to the reference size through an interpolation algorithm; if the image size is larger than the reference size, it is reduced to a smaller size and appropriately adjusted in the subsequent scaling process;

[0072] (2) Network input stage: Before inputting the image into the network, the size of the largest object in the image is calculated, and the image is adjusted to an appropriate size according to certain scaling rules to ensure that the largest object can be detected;

[0073] (3) Network output stage: At the network output, the detection results are mapped back to the original image coordinate system based on the position and size of the predicted box. Since the input image may have been scaled, a corresponding descaling operation is required to accurately map the detection results back to the original image coordinate system;

[0074] The second is the Backbone layer, which is mainly composed of the Focus structure and the CSP structure. The Focus structure is a convolutional neural network layer for feature extraction, which is used to compress and combine the information in the input feature map to extract higher-level feature representations. The Focus structure is a special convolution operation in YOLOv5. It is used as the first convolution layer in the network to downsample the input feature map to reduce the amount of calculation and parameters. Specifically, the Focus structure divides the input feature map into four sub-maps and splices the four sub-maps channels to obtain a smaller feature map. Assuming that the size of the input feature map is N×N×C, where N is the size of the feature map and C is the number of channels, the calculation process of the Focus structure can be divided into the following steps:

[0075] (1) Separate the input feature map into two feature maps of size N×N×C / 4, denoted as x and y;

[0076] (2) Perform convolution operations on x and y with a stride of 2 in the horizontal and vertical directions, respectively, to obtain two feature maps of size N / 2×N / 2×C / 4, denoted as x' and y';

[0077] (3) Concatenate x' and y' channels to obtain a feature map of size N / 2 × N / 2 × C / 2, denoted as z;

[0078] (4) Perform convolution operations on z with a step size of 2 in the horizontal and vertical directions to obtain a feature map of size N / 4×N / 4×C / 2, which is the output of the Focus structure;

[0079] Because the Focus structure downsamples the input feature map, it can effectively reduce the amount of computation and parameters while retaining important information in the input feature map, helping to improve the model's feature extraction capabilities. In YOLOv5, the Focus structure is widely used in the first convolutional layer of each convolutional block to achieve downsampling and feature compression of the input feature map.

[0080] The CSP (Cross Stage Partial) architecture is a key component of YOLO v5 and is used to build the backbone network. Its core idea is to split the input feature map into two parts. One part is processed by a small convolutional network (called a subnetwork), while the other part is directly processed by the next layer. The two feature maps are then concatenated and used as input for the next layer. This design effectively reduces network parameters and computational complexity while improving feature extraction efficiency. The specific steps are as follows:

[0081] (1) The input feature map is divided into two parts, one part enters the sub-network for processing, and the other part is directly passed to the next layer;

[0082] (2) In the sub-network, a convolutional layer is first used to compress the input feature map, followed by a series of convolution operations, and finally a convolutional layer is used to expand it. This method can extract relatively fewer but higher-level features;

[0083] (3) In the next layer, the feature map processed by the sub-network is concatenated with the directly transmitted feature map, and then a series of convolution operations are performed. This combination can fuse low-level detail features with high-level abstract features, improving the efficiency of feature extraction;

[0084] The CSP architecture is widely used in YOLOv5, including multiple stages in the backbone network and some modules in the head network. It not only significantly reduces network parameters and computational complexity, but also improves feature extraction efficiency, thereby accelerating model training and inference.

[0085] A Neck network is an intermediate feature extraction network added to the backbone network. It is primarily used to enhance the model's feature representation and receptive field, thereby further improving its detection performance. YOLO v5 utilizes two different Neck network architectures: SPP (Spatial Pyramid Pooling) and PAN (Path Aggregation Network). SPP is a pyramid pooling architecture that pools feature maps of varying sizes, enhancing the model's ability to perceive objects of varying scales. Specifically, the SPP architecture performs 1x1, 2x2, and 3x3 pooling operations on the input feature maps and concatenates the pooling results at different scales to form the SPP output. This approach helps the model better understand and process objects of varying scales. PAN is a multi-scale feature fusion architecture designed to effectively fuse features at different levels while minimizing information loss. PAN consists of two main modules: the Feature Pyramid Module and the Feature Fusion Module. The Feature Pyramid Module typically consists of multiple convolutional and pooling layers of varying sizes to generate feature maps of varying scales. These feature maps of different scales provide target information of different granularities and can overcome positional deviations on the feature maps. Feature fusion module: used to fuse feature maps of different scales to improve the model's feature expression and perception capabilities. The specific steps are as follows:

[0086] (1) Align the feature map scale of the higher level with the feature map scale of the lower level through upsampling;

[0087] (2) These feature maps are fused through the feature fusion module, and the fused feature map is finally output;

[0088] Output: YOLO v5's output primarily consists of predicted boxes. Each predicted box contains the following information: Confidence score: Indicates the probability of an object existing within the box, ranging from 0 to 1. Class probabilities: Indicates the probability that the object within the box belongs to each class, with a predefined number of classes. Bounding box coordinates: Indicates the location and size of the object, typically represented by a rectangular box.

[0089] The output layer typically includes three feature maps of different scales, each corresponding to a prediction box of a different scale. Specifically, the output layer uses anchor boxes to predict the position and size of the object's bounding box. A softmax function is then applied to the prediction results for each anchor box to calculate the class probability. This structure effectively handles objects of different scales, improving both the accuracy and efficiency of object detection.

[0090] The bounding box loss function uses the IoU loss function, which is mainly used to measure the difference between the predicted bounding box and the true bounding box. IoU loss is a variation of Intersection over Union (IoU), which is an indicator used to measure the degree of overlap between the predicted bounding box and the true bounding box. In target detection, IoU is usually used to evaluate the overlap between the predicted box and the true box to determine whether the predicted box is correct. Specifically, for each predicted bounding box, its IoU value with all true bounding boxes is calculated, and then the true bounding box with the largest IoU is selected as its corresponding matching target, thereby calculating its IoU loss. It specifically uses the DIOU loss function, and the formula is as follows:

[0091]

[0092] Where p is the Euclidean distance between the center point of the predicted bounding box and the center point of the true bounding box. c is the diagonal length of the minimum enclosing box that encloses the predicted bounding box and the true bounding box bgt. v is a measure of the difference in aspect ratio between the predicted box and the true box, and a is a weight that controls the contribution to the total loss. w is the width of the predicted bounding box, h is the height of the predicted bounding box, and w is the width of the predicted bounding box. gt The width of the ground-truth bounding box, h gt is the height of the ground-truth bounding box.

[0093] In the object detection task, in order to avoid multiple detections of the same object, it is necessary to filter out duplicate prediction boxes. This process is called non-maximum suppression (NMS). The implementation steps of NMS are as follows:

[0094] (1) Sorting: Sort all prediction boxes from high to low according to confidence;

[0095] (2) Traverse the prediction box: Starting from the prediction box with the highest confidence, traverse each prediction box in turn to determine whether the IoU value between the prediction box and all subsequent prediction boxes is greater than the set threshold;

[0096] (3) Eliminate overlapping boxes: If the IoU value is greater than the threshold, the predicted box is removed from the candidate box list; otherwise, the predicted box is retained;

[0097] (4) Continue traversal: Continue traversing the next prediction box and repeat the above steps until all prediction boxes have been traversed;

[0098] After NMS processing, the remaining prediction boxes are the best detection results for each object. The core of this method is to compare the IoU values between repeated prediction boxes, remove redundant prediction boxes, and retain only the best results, thereby improving detection accuracy and efficiency.

[0099] Next, we will introduce the Faster R-CNN algorithm based on Swin-Transformer. Faster R-CNN is a classic object detection model. Its structure can be divided into four main parts: Backbone: used for feature extraction. Region Proposal Network (RPN): used to generate candidate regions (region proposals). ROIPooling: used to map candidate regions to feature maps. Detection Head: used for classification and regression bounding box. Swin-Transformer is a model based on Vision Transformer (ViT), which introduces a sliding window mechanism to better capture local and global information of the image. The main parts include:

[0100] Patch Partition: Divides the image into several small patches (patches). Linear Embedding: Linearly maps these patches to feature vectors. Swin-Transformer Block: The core part, including Multi-Head Self-Attention (MHSA) and Multi-Layer Perceptron (MLP). The algorithm flow is as follows:

[0101] (1) Feature extraction: Use convolutional neural networks to extract features from the input image. Faster R-CNN uses Swin-Transformer as the feature extraction network to extract features. The specific feature extraction steps are as follows:

[0102] (1.1) Patch Partition: The input image is divided into several fixed-size, non-overlapping patches (patches), and then these patches are expanded into vectors to facilitate subsequent linear embedding and self-attention processing. Specifically, given an input image of size 224×224, for example, it is first divided into N×N patches (patches), with each patch of size P×P (for example, 4×4). This reduces computational complexity, allowing the Transformer to process high-resolution images.

[0103] (1.2) Linear Embedding: The pixel vector of each patch is mapped to a high-dimensional feature space through a linear transformation to generate a feature vector. These feature vectors will serve as the input of the Transformer. Given a patch of length P×P×C (for example, 4×4×3), linear embedding maps it to a feature vector of length D (for example, 768). This step is achieved through a linear projection layer (i.e., fully connected layer). The low-dimensional image patch vector is mapped to a high-dimensional space to capture richer features;

[0104] (1.3) Swin Transformer Block: Swin Transformer Block is the core component of Swin-Transformer. It includes the Multi-Head Self-Attention mechanism (MHSA) and the Multi-Layer Perceptron (MLP), and introduces a shifted window mechanism to better capture local and global information of the image.

[0105] MHSA achieves richer feature representation by focusing on features at different positions simultaneously through multiple attention heads. The self-attention mechanism calculates the influence of each element in the input sequence on all other elements, thereby capturing global dependencies. The input feature vector is divided into multiple subspaces (each subspace corresponds to a head). Each head independently calculates self-attention to obtain a weight matrix. The outputs of all heads are spliced to form the final feature representation. The Shifted Window mechanism divides the image into non-overlapping windows (with a fixed window size) and applies MHSA within each window. Then, the window is shifted and MHSA is applied again to capture cross-window dependencies. This mechanism can effectively balance the extraction of local and global features; MLP consists of two fully connected layers, and an activation function (such as GELU) is used in the middle for nonlinear transformation. MLP is used in the SwinTransformer Block to perform further linear transformation and nonlinear activation on features, thereby improving feature expression capabilities;

[0106] (2) RPN network: The RPN network is the core component of Faster R-CNN. Its main function is to generate region proposals, which can obtain regions containing objects. Subsequent parts of the network can perform further detection based on these regions. Its implementation steps are as follows:

[0107] (2.1) Use anchors to exhaustively enumerate the area where the target object is located. Anchors are rectangular boxes that may contain target objects. In target detection tasks, a common method is to predefine a set of boxes called anchors for each pixel in the image. These boxes have different area sizes and aspect ratios. By densely placing anchors on the image, all positions and areas containing objects are covered. Faster R-CNN scales and deforms the anchor corresponding to each point. The former corresponds to the rectangular area, and the latter corresponds to the aspect ratio of the rectangle. Each size corresponds to three aspect ratios. There are three sizes of 128x128, 256x256, and 512x512, and the three aspect ratios are 1:1, 1:2, and 2:1, respectively. One point corresponds to 9 anchors, and the shape and size of each anchor are not exactly the same.

[0108] (2.2) Calculate the classification and regression results at each feature point. The classification here is binary, distinguishing only between foreground and background. Specifically, first perform a 3x3 convolution on the fm to further extract features, then use a 1x1 convolution to map the number of channels to 18 = 9x2, corresponding to the two categories of the 9 anchors. Then, the channel dimension is split into another dimension, so that the last dimension is 2, representing the foreground and background. Finally, use softmax to calculate the probability. The regression method uses a 1x1 convolution to map the number of channels to 36 = 9x4, corresponding to the position and size of the 9 anchors.

[0109] (2.3) Post-process the classification and regression results to generate RoI (Region of Interest). This is the last step in the One-Stage algorithm implementation, also known as the Proposal Creator (Proposal Layer). It post-processes the classification and regression results output by the RPN and outputs the region that the network believes contains the object.

[0110] (3) ROI pooling: For a given feature map area (of size h×w), it is necessary to adapt it to a fixed-size output (assuming it is H×W); the h×w feature map area is divided into H×W sub-windows. The size of each sub-window is (h / H)×(w / W); in each sub-window, a maximum pooling operation is performed, that is, the maximum value from the current sub-window is extracted as the output value of the sub-window; the previous step is repeated until all sub-windows are traversed. The final output ROI pooling result is a feature map of size H×W, where the value of each position represents the maximum value in the corresponding sub-window. For the case where the input size of the classification network in Faster R-CNN is fixed to 7×7, regardless of the size of the candidate box or candidate area, a 7×7 grid needs to be used to cover the original area. Then, in each 7×7 grid, a maximum pooling operation is performed, and the maximum value in the area covered by the current grid is taken as the output value. Through such an operation, candidate boxes of any size are pooled into a fixed-size 7×7 feature map;

[0111] (4) Classification and regression: The output of the ROI pooling layer is input into the classifier and regressor for object classification and position regression. The specific process is as follows: in the regressor, through the 1*1 convolution layer, the output scores = 36 = 4*9, 4 is the parameters tx, ty, tw, th that determine the position of the border; in the classifier, through the 1*1 convolution layer, the output scores = 18 = 2*9, 2 represents the foreground and background probability (i.e., whether it is the target or the background), where the number 9 represents the 9 anchors with different scales and aspect ratios;

[0112] (5) Post-processing: Use the non-maximum suppression (NMS) algorithm to remove duplicate candidate boxes that overlap. Scale the network output to the original image size, then denormalize the regression result (multiply by the standard deviation and add the mean), calculate the position of the bbox (upper left corner coordinates and lower right corner coordinates) based on the position and size of the ROIS, and crop it to the original image size. Then select a rectangular box with a confidence score larger than the threshold, and finally use non-maximum suppression (NMS) to remove the bbox with high overlap, and finally output the category and position information of each candidate box to obtain the target recognition result;

[0113] (6) Loss function: Faster R-CNN uses a multi-task loss function for end-to-end training, including classification and regression tasks. The softmax loss function is used for classification tasks, and the smooth L1 loss function is used for regression tasks.

[0114] For step 4, the present invention designs a complete processing flow that can call deep learning frameworks, such as the PP-OCRv3 deep learning algorithm, on demand. The following is a detailed introduction and implementation of this method.

[0115] The PP-OCRv3 deep learning algorithm process consists of three parts: locating the text position based on DBNet, rotating the positioning box to be horizontal with the text, and performing character recognition based on the SVTR algorithm. The process is described as follows:

[0116] (1) DBNet locates scene text based on the idea of segmentation, which is suitable for text with different directions or even circular distribution;

[0117] (2) When the text localization algorithm is segmented, it is necessary to predict the possible location of the text based on the probability map. This method uses a differentiable binarization process to input the threshold into the segmentation network for optimization, generating a threshold map that adapts to different probability map regions.

[0118] (3) Turn off the rotated text detection classifier to improve the detection model's ability to accurately recognize high-frequency numbers such as 6 and 9.

[0119] (4) Character recognition methods based on recurrent neural networks (RNNs) typically include two steps: feature extraction and text sequence translation. SVTR fully utilizes contextual information by introducing the Transformers structure and directly outputs text content based on a single visual model, thereby optimizing the problem of low sequence modeling efficiency in the character recognition process. When extracting features, the algorithm first performs local feature extraction (Local Mixing) to extract character features, and then performs global feature extraction (Global Mixing) to extract the contextual dependencies between different characters. By stacking local and global mixing blocks, a multi-level feature description is formed.

[0120] like Figure 3 As shown, the process of extracting height and speed limit information is as follows:

[0121] (1) The traffic sign located by the Faster R-CNN algorithm based on Swin-Transformer is cropped, and only the image within the target detection rectangular detection box is retained. The relevant metadata and detection output information are saved in the form of a file name;

[0122] (2) Constrain the area of the located traffic signs and only retain detection targets with an area larger than 300 pixels;

[0123] (3) Recognize the cropped traffic sign target based on the PP-OCRv3 text detection model, and discard it if no result is output;

[0124] (4) If there is character output in the said result and the output result contains numbers, record it to optimize the misdetection of the target recognition model in some scenarios;

[0125] (5) According to the prior information of the numerical value ranges of speed limit, height limit and weight limit traffic signs, perform semantic constraint and text cleaning; for speed limit data, retain the values within the range of (0 - 150), and at the same time, if the output speed limit value is not a multiple of 5, discard it;

[0126] (6) For the said height limit data, first extract the numbers and decimal points as the preliminary screening detection results, and the retained record numerical range is (1 - 100). If the extracted value is greater than 10, divide it by 10 to convert it into a decimal, and retain one significant digit after the decimal point (without rounding) as the final screening result.

[0127] As Figure 4 shown, the process of extracting weight limit information is as follows:

[0128] (1) Crop the traffic signs located by the Faster R-CNN algorithm based on Swin-Transformer, and only retain the image within the target detection rectangular detection frame; crop the traffic signs located by the YOLO V5 algorithm, and retain the image that is amplified to 6 times in size centered on the target detection rectangular detection frame, and save the relevant metadata and detection output information in the form of file names;

[0129] (2) For the category one weight limit sign of the weight limit sign, use the text recognition model to obtain the center point coordinates (x1, y1) of the text box, and at the same time use the image segmentation method to obtain the center coordinates (x2, y2) of the sign board. If the difference in the x coordinates or y coordinates of the two center points is greater than 2 pixels, output it as the category one weight limit sign, otherwise discard it;

[0130] (3) For the category two and category three weight limit signs, extract them according to their attribute information characteristics. Category two and category three signs are mostly combinations of prohibition signs and text signs, and extract the information indicating the weight limit from their text signs;

[0131] (4) If the said text recognition result contains the character "以" and does not contain any of the numbers "ninety", "sixty", "one hundred", extract the result and retain it as the category two weight limit sign;

[0132] (5) If the said text recognition result contains both the number "ninety" and any Chinese characters, extract the result and retain it as the category three weight limit sign.

[0133] The above-mentioned Category 1 weight limit sign refers to a circular weight limit sign with only numbers and letters, Category 2 weight limit sign refers to a combination of a prohibition sign and a text sign with weight limit information, and Category 3 weight limit sign is a combination of a speed limit sign and a text sign with weight limit information.

[0134] like Figure 5 As shown, the traffic sign speed limit, height limit and weight limit information extraction system described in the present invention includes: an acquisition and processing module, which is used to obtain OSM road network data and corresponding panoramic pictures, and crop the panoramic pictures in width and height; a production module, which is used to extract samples from public data sets and fuse existing traffic sign data, and fuse each sub-category with detailed numerical values to form a training set; a positioning module, which is used to detect speed limit, height limit and weight limit traffic signs and locate their positions; an extraction module, which is used to extract attribute information such as text on speed limit, height limit and weight limit traffic signs.

[0135] The computer-readable storage medium of the present invention stores instructions, which, when executed, enable the computer to execute the above-mentioned method for extracting speed, height and weight limit information from traffic signs.

[0136] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A method for extracting speed limit, height limit and weight limit information on traffic signs, characterized in that: It includes the following steps: Step 1: Obtain road network data and sample at equal intervals, obtain the panoramic view of the road, and crop the panoramic view; Step 2: Extract and fuse existing traffic sign data using publicly available traffic datasets to construct a training set; including: Step 21: Extract samples from the TT100K and MTSD datasets and fuse existing traffic sign data, and fuse each sub-category with detailed numerical values; Step 22: According to the category table provided by Mapillary, traverse the ID files of the divided training set and validation set in the annotation folder, find their annotation files according to the ID index, and retain the traffic sign samples that conform to the Chinese traffic sign specification in the annotation files; Step 23: Merge individual sample annotation files into the COCOjson annotation format, and then fuse them with the samples extracted from TT100K to form a training set; Step 3: Locate traffic signs in the street view; Step 4: Extract speed limit, height limit, and weight limit information of traffic signs; Crop the traffic signs located by the FasterR-CNN algorithm based on Swin-Transformer, only retain the image within the target detection rectangle detection frame, and save the relevant metadata and detection output information in the form of file names; Constrain the area of the cropped traffic signs, and only retain the detection targets with an area greater than 300 pixels; Based on the PP-OCRv3 text detection model, identify the constrained traffic sign targets to obtain the recognition results, and discard them if there is no result output; If the result has character output and the output result contains numbers, retain the record and optimize the misdetection of the target recognition model in some scenarios; According to the prior information of the numerical ranges of speed limit, height limit, and weight limit traffic signs, perform semantic constraint and text cleaning; for speed limit data, retain the values within the range of 0-150, and at the same time, if the output speed limit value is not a multiple of 5, discard it; For height limit data, first extract the numbers and decimal points as the preliminary screening detection results, and the retained record numerical range is 1-100; if the extracted value is greater than 10, divide it by 10 to convert it into a decimal, and retain one significant digit after the decimal point as the final screening result; For circular weight limit traffic signs that only contain numerical and alphabetical text, use the text recognition model to obtain the center coordinates (x1, y1) of the text box, and at the same time use the image segmentation method to obtain the center coordinates (x2, y2) of the sign circle. If the difference in the x coordinates or y coordinates of the two centers is greater than 2 pixels, output it as a category one weight limit sign, otherwise discard it; For weight limit traffic signs that contain both numerical text and literal text, classify them according to their attribute information characteristics. One category is the combination of prohibition signs and literal signs, and the other category is the combination of speed limit signs and literal signs. After classification, use the text recognition model to recognize the information indicating the weight limit in the literal sign; If the text recognition result contains the character "以" and does not contain any of the numbers "90", "60", "100", extract the result and retain it as a category two weight limit sign; If the text recognition result contains the number "90" and any Chinese characters at the same time, the result will be extracted and retained as a Category 3 Limited Repeat Card.

2. The method for extracting speed limit, height limit and weight limit information from traffic signs according to claim 1, characterized in that: The step 1 includes the following sub-steps: Step 11: Use the tool to obtain the road network data that meets the query conditions in the OSM database and save it; Step 12: Using the tool, generate equally spaced sampling points along the geometry of the road network layer; Step 13: Get the panoramic ID of the coordinate point corresponding to the sampling point, and download the panoramic image of the corresponding coordinate point; Step 14: Crop the panorama.

3. The method for extracting speed limit, height limit and weight limit information from traffic signs according to claim 2, characterized in that: Step 14 specifically includes: retaining the middle 1 / 3 of the panoramic image in the height direction, and cropping the above-mentioned retained parts in sequence along the horizontal direction with a set width value and an overlap rate of 0.15, wherein the last picture is cropped to the left with the set width value based on the right edge of the panoramic image.

4. The method for extracting speed limit, height limit and weight limit information from traffic signs according to claim 1, characterized in that: The step 3 includes the following sub-steps: Step 31: Evaluate the recognition performance of various object recognition algorithms on the fused training set; Step 32: Based on the recognition performance of the target recognition algorithm, the Faster R-CNN algorithm based on Swin-Transformer is selected to extract and locate traffic signs; Step 33: Based on the evaluation index mAP (mean average precision), perform a final performance evaluation on each recognition algorithm.

5. The method for extracting speed limit, height limit and weight limit information from traffic signs according to claim 4, characterized in that: Step 32 also includes: for traffic signs containing both speed limit information and weight limit information, YOLO V5 algorithm is selected to locate them through online sample enhancement function.

6. The method for extracting speed limit, height limit and weight limit information from traffic signs according to claim 1, characterized in that: Step 4 specifically includes: cropping the traffic sign located by the Faster R-CNN algorithm based on Swin-Transformer, retaining only the image within the target detection rectangular detection frame, and saving the relevant metadata and detection output information in the form of a file name; cropping the traffic sign located by the YOLO V5 algorithm, retaining the image centered on the target detection rectangular detection frame and enlarged to 6 times the size, and saving the relevant metadata and detection output information in the form of a file name.

7. A traffic sign speed limit, height limit and weight limit information extraction system, characterized in that include: The acquisition and processing module is used to obtain OSM road network data and corresponding panoramic images, and to crop the panoramic images in width and height; The production module is used to extract samples from the public dataset and integrate existing traffic sign data to form a training set for each sub-category with detailed values; it includes: Extract samples from the TT100K and MTSD datasets and fuse them with existing traffic sign data to fuse each subdivision category with detailed values; According to the category table provided by Mapillary, traverse the training set and validation set ID files divided in the annotation folder, find their annotation files based on the ID index, and retain the traffic sign samples in the annotation files that meet the Chinese traffic sign specifications; Merge the individual sample annotation files into the COCOjson annotation format, and then fuse them with the samples extracted from TT100K to form a training set; Positioning module, used to detect speed limit, height limit and weight limit traffic signs and locate their location; An extraction module for extracting the text attribute information of speed limit, height limit, and weight limit traffic signs; Crop the traffic signs located by the FasterR-CNN algorithm based on Swin-Transformer, only retain the image within the target detection rectangle detection frame, and save the relevant metadata and detection output information in the form of a file name; Constrain the area of the cropped traffic signs, only retain the detection targets with an area greater than 300 pixels; Identify the constrained traffic sign targets based on the PP-OCRv3 text detection model to obtain the recognition results, and discard them if no results are output; If the result has character output and the output result contains numbers, retain the record to optimize the misdetection of the target recognition model in some scenarios; According to the prior information of the numerical ranges of speed limit, height limit, and weight limit traffic signs, perform semantic constraint and text cleaning; for speed limit data, retain the values within the range of 0-150, and at the same time, if the output speed limit value is not a multiple of 5, discard it; For height limit data, first extract the numbers and decimal points as the initial screening detection results, and the retained record value range is 1-100; if the extracted value is greater than 10, divide it by 10 to convert it into a decimal, and retain one significant digit after the decimal point as the final screening result; For circular weight limit traffic signs that only contain text of numbers and letters, use the text recognition model to obtain the center coordinates (x1, y1) of the text box, and at the same time use the image segmentation method to obtain the center coordinates (x2, y2) of the signboard center. If the difference in the x coordinates or y coordinates of the two centers is more than 2 pixels, output it as a category one weight limit sign, otherwise discard it; For weight limit traffic signs that contain both numerical text and literal text, classify them according to their attribute information characteristics. One category is the combination of a prohibition sign and a text sign, and the other category is the combination of a speed limit sign and a text sign. After classification, use the text recognition model to perform text recognition on the information indicating the weight limit in the text sign; further include: If the text recognition result contains the character "以" and does not contain any of the numbers "90", "60", "100", then extract the result and retain it as a category two weight limit sign; If the text recognition result contains both the number "90" and any Chinese character, then extract the result and retain it as a category three weight limit sign.

8. A computer-readable storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed, it can enable a computer to execute the traffic sign speed limit, height limit, and weight limit information extraction method according to any one of claims 1 to 6.

Images