A method for extracting store sign information based on deep learning
By improving the YOLOv5 object detection model and combining multi-scale feature constraints and feature enhancement techniques, the problems of occlusion and small object detection in store sign information extraction are solved, achieving efficient and fast information extraction, which is suitable for edge devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LIAONING TECHNICAL UNIVERSITY
- Filing Date
- 2022-10-14
- Publication Date
- 2026-05-26
AI Technical Summary
Traditional methods are time-consuming, labor-intensive, and expensive in extracting store sign information. Furthermore, existing deep learning models cannot achieve the same level of visual interpretation as human visual interpretation, especially in the detection of occlusion and small objects.
An improved YOLOv5 object detection model is adopted, which combines SP module, PSA module, Coordconv layer, CoT module and CBAM for feature enhancement, constructs multi-scale rectangular feature constraints, solves the problems of occlusion and small object detection, and generates a model suitable for edge devices.
It improved the work efficiency of inspection personnel, reduced costs, and enhanced the prediction inference speed and detection accuracy through a single-stage target detection model.
Smart Images
Figure CN115661703B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of industrial vision and smart cities, specifically relating to a method for extracting store sign information based on deep learning. Background Technology
[0002] Early urban infrastructure information was primarily compiled and summarized through on-site surveys. While the data was relatively accurate, this method was extremely costly in terms of time and resources for large-scale urban infrastructure surveys, and the results were often outdated, significantly reducing their usability. Information on store signage construction is one of the most important types of urban infrastructure information, possessing significant value for urban appearance management, urban economic development analysis, and three-dimensional urban reconstruction.
[0003] To efficiently obtain information about ground features, scholars both domestically and internationally primarily employ two approaches. The first is the object-oriented method. This method uses image segmentation algorithms to cluster pixels with similar features, forming segmented region units, and then utilizes image classification algorithms. However, traditional methods rely on the spectral and texture information of images during segmentation and classification, i.e., based on pixel homogeneity, a bottom-up aggregation to form image objects. Store signs, as identification targets, vary greatly in size and color rendering, undoubtedly leading to significant spatial heterogeneity and increasing the difficulty of extracting store sign information. At the technical level, segmentation and classification methods for obtaining image objects essentially do not consider the high-level features of images, such as morphological and contextual semantic information; they are based on pixel aggregation of spectral features, failing to fully utilize other features of street view images. Therefore, the object units obtained by traditional methods often do not match people's morphological perception of actual ground features, resulting in object-level classification results that cannot be converted into interpretation results with actual entity meaning. The second approach is deep learning methods. This method primarily relies on data-driven decision-making and self-iterative optimization. Through representation learning in deep networks, it enables multi-level abstraction in semantic image analysis, resulting in significantly superior performance compared to traditional methods in remote sensing applications. For example, to address occlusion and small target problems, Xu et al. used the repulsive loss function Rep Loss to solve the target occlusion issue and employed a scale-aware dual-path sub-network for detection at two different target scales, finally performing perceptual fusion at the inference end. For the localization of targets occluded by perspective panels under multivariate conditions, Morera et al. used a self-labeled dataset, employing various data augmentation techniques and combining them with loss functions such as localization loss. For the detection of partially occluded targets under low-resolution conditions, Li Linsheng designed candidate box sizes, generating nine different candidate regions, increasing the model's ability to recognize different features. Two sliding windows were added to the original RPN, further expanding the model's information intake range. The aim was to use an inherent deep learning model to classify and extract different types of targets in images, but this approach consistently failed to achieve the same effect as manual visual interpretation; that is, each type of target required a separate deep learning extraction algorithm designed according to its specific visual characteristics. Leveraging the advantages of YOLOv5's high detection accuracy and fast inference speed, this invention proposes a store sign information extraction network that integrates positional attention and topology reconstruction under rectangular feature constraints. Summary of the Invention
[0004] To address the problems of time-consuming, labor-intensive, and costly traditional manual inspection of store sign information, this invention provides a method for extracting store sign information based on deep learning, thereby overcoming existing technical deficiencies.
[0005] This invention proposes a method for extracting store sign information based on deep learning, which specifically includes the following steps:
[0006] S1: Obtain data for building deep learning models and divide the dataset;
[0007] S2: Improve the feature extraction structure and enhance the pyramid features to further build a model for extracting store sign information;
[0008] S3: Train and validate the object detection model to obtain the optimal detection model weight file;
[0009] S4: Import the target image to be detected, detect its location, and output the target detection result according to the preset threshold. The deep learning target detection method generates a smaller model, which is easier to deploy and port to edge devices for result calculation, thereby improving the work efficiency of inspection personnel and reducing costs.
[0010] Furthermore, step S1 mainly includes the following steps:
[0011] S11. Organize the data, considering only the target imaged from the frontal viewpoint as the initial data;
[0012] S12. Import initial data, manually draw frames using the LabelImg annotation software to mark the location information of store signs, perform target detection information annotation, and follow the format. <object-class-id><X center><Y center><Frame width><Frame height> represent the annotated text file;
[0013] S13. Convert the obtained VOC format dataset into TXT format, and further divide it into training set, validation set and test set according to the ratio of 8:1:1 to obtain the dataset required for final training.
[0014] Furthermore, the deep learning model is constructed based on the YOLOv5 object detection model.
[0015] Furthermore, step S2 mainly includes the following steps:
[0016] S21. For the multi-scale rectangular features of the store signboard, integrate the SP module and the PSA module to form a multi-scale rectangular feature constraint. On the basis of ensuring the multi-scale perception performance, further build the MFSPP module to improve the generalization ability of the model;
[0017] S22. For the spatial distribution characteristics of the store signboard, introduce the Coordconv layer to improve the attention to the middle position of the image and reduce the interference of similar ground objects in other positions; reconstruct the topological relationship by integrating the CoT module to solve the problem of target fragmentation caused by occlusion;
[0018] S23. For the weakening problem that features are propagated from the high level to the low level of the network, introduce CBAM for feature enhancement to ensure the sufficiency and accuracy of feature utilization.
[0019] Furthermore, step S3 mainly includes the following steps:
[0020] S31. Set the path of the pre-trained model, the path of dataset reading, further set the number of training iterations and the size of the input images, and set other parameter options to default, then start to train the target model;
[0021] S32. Visualize the training process and view the change curves of the metrics related to the model;
[0022] S33. After training, compare, view and save the best target model and output it to the specified path;
[0023] S34. Perform prediction inference on the test samples on the trained target model to verify the target model.
[0024] Beneficial effects: A method for extracting store signboard information based on deep learning has the following advantages compared with the existing technology:
[0025] (1) Through the deep learning object detection method, the generated model is smaller, which is beneficial to be deployed and transplanted to edge devices for result calculation, thus improving the work efficiency of patrol personnel and reducing costs;
[0026] (2) The single-stage target detection model effectively improves the speed of prediction inference. Attached Figure Description
[0027] The description of the invention will become apparent and readily understood in conjunction with the following drawings, wherein:
[0028] Figure 1 This is a flowchart illustrating the method for extracting store sign information based on deep learning in a specific embodiment of the present invention.
[0029] Figure 2 This is a schematic diagram of the overall process of the method for extracting store sign information based on deep learning in a specific embodiment of the present invention;
[0030] Figure 3-1 , 3-2 These are schematic diagrams illustrating the method for extracting store sign information based on deep learning in specific embodiments of the present invention, namely, filtering valid data and storing it in a dataset, and labeling the dataset with tags;
[0031] Figures 4-1 to 4-9 The diagrams show the overall architecture of the improved YOLOv5 object detection model, the CoordConv module, the SP module, the CoT module, the PSA module, the BottleNeck module in C3_P, the MFSPP module, the BottleNeck module in C3_SC, and the attention module in the method for extracting store sign information based on deep learning, as described in the specific embodiments of the present invention.
[0032] Figures 5-1 to 5-8 These are schematic diagrams illustrating the visualization prediction process, labels_correlogram, performance of the training and validation sets on the model, P_curve, PR_curve, R_curve, positive sample prediction results, and a preview of the sampling results of the positive sample prediction set in the method for extracting store sign information based on deep learning according to specific embodiments of the present invention.
[0033] Figure 6 The visualization results of the deep learning-based store sign information extraction method and other methods in the CTW dataset are shown below, representing a specific implementation of the present invention. Detailed Implementation
[0034] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced.
[0035] Figure 1 An embodiment of the present invention discloses a method for extracting store sign information based on deep learning, such as... Figure 1 and Figure 2 As shown, the method specifically includes the following steps:
[0036] S1: Obtain data for building deep learning models and divide the dataset;
[0037] Furthermore, S1 specifically includes:
[0038] S11. Organize the data, considering only the target imaged from the frontal viewpoint as the initial data;
[0039] S12. Import initial data, manually draw frames using the LabelImg annotation software to mark the location information of store signs, perform target detection information annotation, and follow the format. <object-class-id><X center><Y center><box width><box height> represent the annotated text file;
[0040] S13. Convert the obtained VOC - formatted dataset into TXT format, and further divide it into a training set, a validation set, and a test set according to the ratio of 8:1:1 to obtain the dataset required for final training.
[0041] The datasets for deep - learning models are divided into several categories according to the tasks performed, such as object detection, image segmentation, image classification, and instance segmentation. In this embodiment, a dataset with an annotation - format file for image segmentation is used, and the obtained annotated dataset is divided into a training set, a validation set, and a test set according to the ratio of 8:1:1.
[0042] Specifically, the VOC - formatted dataset used in this embodiment mainly screens the street - view data obtained by a mobile acquisition vehicle, selects the images covering the store - sign targets as the initial data, performs information annotation for object detection through the annotation software LabelImg, that is, manually draws bounding boxes to annotate the position information of the target objects. Then, the VOC - formatted dataset needs to be converted into the txt format dedicated to the YOLO model. Finally, according to the dataset division ratio, the dataset required for final training is obtained.
[0043] There are usually multiple datasets for constructing the final model. In different stages of model construction, there are usually three types of datasets: the training set, the validation set, and the test set.
[0044] First, the model is fitted on the training dataset. For supervised learning, the training dataset is a set of samples used to fit parameters (such as the weights of the connections between neurons in an artificial neural network). In practice, the training dataset usually consists of data pairs of input vectors (scalars) and output vectors (scalars). The output vector (scalar) is called the target or label. During the training process, the current model makes predictions for each sample in the training dataset and compares the prediction results with the targets. According to the comparison results, the learning algorithm updates the parameters of the model. The process of model fitting may include both feature selection and parameter estimation.
[0045] Next, the fitted model is used to make predictions on a second dataset—the validation dataset. The validation dataset provides an unbiased evaluation of the model fitted on the training set when tuning hyperparameters (such as the number of neurons in the hidden layers of a neural network). The validation set can be used for early stopping in regularization: training is stopped when the validation set error rises (a signal of overfitting on the training set). However, in practice, this approach sometimes fails because the validation set error fluctuates during training. Therefore, rules have been developed to determine signals of better overfitting.
[0046] Finally, the test dataset can be used to provide an unbiased evaluation of the final model. If the test dataset is never used during training (e.g., not used in cross-validation), it is also called the reserved set.
[0047] Reference Figure 3-1 and Figure 3-2 In this embodiment, the specific processing flow for the dataset is as follows:
[0048] 1. Organize the street view images acquired by the mobile data collection vehicle;
[0049] 2. Images of store signs that are collected and organized and can be viewed from the front are stored in the dataset;
[0050] 3. Import the prepared images into the labeling software labelImg for data cleaning and labeling;
[0051] 4. Convert the labeled data file into YOLO format and then train the dataset.
[0052] S2: Improve the feature extraction structure and enhance the pyramid features to further build a model for extracting store sign information;
[0053] Reference Figures 4-1 to 4-8 Specifically, S2 also includes:
[0054] S21. For the multi-scale rectangular features of store signs, integrate the SP module and PSA module to form a multi-scale rectangular feature constraint. On the basis of ensuring multi-scale perception performance, further build the MFSPP module to improve the model's generalization ability.
[0055] S22. To address the spatial distribution characteristics of store signs, a Coordconv layer is introduced to increase the focus on the central position of the image and reduce interference from similar features in other positions; the topological relationship is reconstructed by integrating the CoT module to solve the problem of target fragmentation caused by occlusion.
[0056] S23. To address the weakening problem of features propagating from higher to lower levels of the network, CBAM is introduced to enhance features, ensuring the sufficiency and accuracy of feature utilization.
[0057] S3: Train and validate the object detection model to obtain the optimal detection model weight file;
[0058] The training and validation phases primarily involve setting the hyperparameters of the network model. Several key parameters are involved during training: `data` is the dataset read path, `epochs` is the number of training iterations, `img-size` is the size of the input image, and other parameters are default. Since the dataset is already divided into training, validation, and test sets during the initial setup, the validation set is essentially validated concurrently with the training process.
[0059] Reference Figures 5-1 to 5-8 Specifically, S3 also includes:
[0060] S31. Set the pre-trained model path, dataset reading path, further set the number of training iterations and the size of the input image, set other parameter options to default, and start training the target model.
[0061] S32. Visualize the training process and view the change curves of model-related indicators;
[0062] S33. After training is complete, compare and save the best target model and output it to the specified path;
[0063] S34. Perform prediction and inference on the test samples on the trained target model to verify the target model.
[0064] S4: Import the target image to be detected, detect its position, and output the target detection results according to the preset threshold.
[0065] Specifically, in this embodiment, the preset threshold is set to 0.45. The main function of this threshold is to limit the confidence level. In other words, this threshold will filter out the extra detection results that do not meet the condition of >0.45.
[0066] Figure 6 The visualization results of the deep learning-based store sign information extraction method of this invention and other methods on the CTW dataset are shown below:
[0067] The following experiments verify the proposed method and compare it with existing algorithms. The experiments are the extraction results on the CTW dataset. Figure 6 The visualization results on the CTW dataset include the extraction results of this invention and the results of the comparison method.
[0068] from Figure 6 As can be seen from the results, the biggest challenge for automatic detection on the CTW dataset is that store signs in the images are obscured by trees, resulting in blurred or even invisible boundaries. Compared to other tested methods, our algorithm is less affected by occlusion and can accurately extract store signs. Figure 6 As can be seen, the first column of images shows large-scale store signs in unobstructed conditions, while the third to fifth columns show store signs with increasingly higher levels of occlusion, making automatic extraction increasingly difficult. For the unobstructed large-scale store signs shown in the first column, our method, along with four other methods besides YOLOv5, can obtain complete extraction results. However, under occlusion, these methods perform differently. From the second column, we can see that only our method, Efficientdet, and YOLOX can extract small-scale partially occluded store signs, but the boundaries extracted by Efficientdet and YOLOX are not very accurate. For the severely occluded store signs shown in the third and fourth columns, the results extracted by our method are more consistent with the actual store sign areas in the image compared to other test algorithms, indicating that the OSO-YOLOv5 store sign extraction network proposed in this paper can effectively improve detection accuracy and eliminate the impact of occlusion on road extraction. In summary, our method can perceive multi-scale information and infer occluded parts through contextual information, achieving high detection accuracy.
[0069] Table 1 shows that our method achieved the highest scores across all five metrics on the self-labeled CTW dataset, demonstrating that our algorithm maintains a balance between precision and recall. The corresponding precision and recall are 82.7% and 87.6%, respectively, indicating that 82.7% of the detected features are store signs in the test image, and 87.6% of all store sign features in the image were correctly detected. Our method's F1 score is 82.4%, which is 6.0%–33.8% higher than other methods, indicating a significant advantage. Our method's IoU is 78.1%, 16.3% higher than the second-ranked YOLOv5_l. However, compared to the original YOLOv5_l, our method improves recall by 7.5% and IoU by 3.5%, indicating that our method successfully detected more store signs than the original YOLOv5_l.
[0070] Table 1 Quantitative Evaluation of the CTW Dataset
[0071]
[0072] The technical solution of this invention combines deep learning object detection with store sign detection, generating a smaller model that is easier to deploy and port to edge devices for result calculation, thereby improving the work efficiency of inspection personnel and reducing costs; through a single-stage object detection model, the speed of prediction and inference is effectively improved.
[0073] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.
Claims
1.A method for shop sign information extraction based on deep learning, characterized in that, The deep learning model is constructed based on the YOLOv5 object detection model, including the following steps: S1: Obtain the data for constructing the deep learning object detection model and divide the data set; Specifically, S1 also includes: S11: Organize the data, and only consider the objects imaged from the frontal view as the initial data; S12, import initial data, manually frame through the labeling software LabelImg to label the position information of the store sign, perform information labeling of target detection, and label according to the format <object-class-id><X center><Y center><box width><box height> represents the annotated text file;< / object-class-id> S13: Convert the obtained VOC format data set into TXT format, and further divide it into training set, validation set and test set according to the ratio of 8:1:1 to obtain the data set required for final training; S2: Improve the feature extraction structure, enhance the pyramid features, and further construct an object detection model for store sign information extraction; Specifically, S2 also includes: S21: For the multi-scale rectangular features of store signs, integrate the SP module and the PSA module to form a multi-scale rectangular feature constraint. On the basis of ensuring the multi-scale perception performance, further build the MFSPP module to improve the generalization ability of the object detection model; S22: For the spatial distribution characteristics of store signs, introduce the Coordconv layer to improve the attention to the middle position of the image and reduce the interference of similar objects in other positions; by integrating the CoT module to reconstruct the topological relationship, solve the problem of object fragmentation caused by occlusion; S23: For the weakening problem that features are propagated from the high level to the low level of the network, introduce CBAM for feature enhancement to ensure the sufficiency and accuracy of feature utilization; S3: Train and validate the object detection model to obtain the best object detection model weight file; Specifically, S3 also includes: S31: Set the path of the pre-trained object detection model, the path of data set reading, further set the number of training iterations and the size of the input images, and set other parameter options to the default, and start training the object detection model; S32: Visualize the training process and view the change curves of the indicators related to the object detection model; S33: After training, compare, view and save the best object detection model and output it to the specified path; S34: Perform prediction inference on the test samples on the trained object detection model to verify the object detection model; S4: Import the target image to be detected, detect its position and output the object detection result according to the preset threshold.