Method and device for identifying storefront signs based on image recognition technology

Abstract

The present invention discloses a method and device for identifying store front signs based on image recognition technology, belonging to the field of computer vision technology. The method includes: obtaining an image of a store front sign offline, and performing identification through a pre-constructed store front classification model and a sign element recognition model; combining the recognition results of the store front classification model and the sign element recognition model, extracting the image information on the image of the store front sign offline, and determining whether the image of the store front sign offline meets the requirements of the dealer. The method and device for identifying store front signs based on image recognition technology provided by the present invention can efficiently determine whether the content and placement method of the store front sign are qualified, improve the efficiency of an enterprise in managing offline stores, and reduce the store management cost.

Description

Technical Field

[0001] The present invention relates to a method and device for identifying store front signs based on image recognition technology, belonging to the field of computer vision technology. Background Art

[0002] SFA (Sales Force Automation) is an important business component of the CRM customer relationship management system. Through a series of functions such as visit target setting, route planning, setting execution specifications, task execution, and execution result analysis, SFA standardizes and guides the on-the-go behavior of business personnel, helping business personnel correctly and efficiently complete the specified steps of visits.

[0003] The customer visit system in SFA is an important function used to help dealers complete the visit and inspection tasks of offline signed stores. Salesmen take pictures of the visited offline stores through the customer visit process to check whether the company's products are specifically displayed, whether the display is neat, and whether the store signs are placed according to the company's requirements.

[0004] Dealers have unified requirements for the sign forms of offline signed stores. Dealers not only have requirements for the content of the store signs (such as dealer logos, dealer names, etc.), but also require the placement methods of the signs (such as the sign is placed above the storefront, the sign color, etc.). Qualified store signs can not only improve the advertising effectiveness of dealers, increase product sales, but also enhance the business image of dealers. Therefore, dealers need a method to efficiently determine whether the store sign images submitted by business personnel meet the requirements of dealers, quickly discover unqualified store signs, and reduce the management costs of offline stores.

[0005] When only using a classification model for recognition, although it can identify whether the placement method of the sign is qualified, the recognition of sign elements is poor, resulting in a high misjudgment rate; only using a sign element recognition model can accurately identify the elements associated with the dealer on the sign, but it cannot identify whether the placement method of the sign is qualified. Therefore, the recognition effect of using a single model is not good and cannot meet the actual application requirements. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method and device for identifying store front signs based on image recognition technology, which is used to efficiently determine whether the store images submitted by salesmen meet the requirements and quickly discover offline stores with unqualified signs.

[0007] To achieve the above purpose, the present invention is implemented by the following technical solutions:

[0008] In the first aspect, the present invention provides a method for identifying store front signs based on image recognition technology, including:

[0009] Obtain the images of the storefront signs of offline stores and identify them through a pre-constructed storefront classification model and a sign element recognition model;

[0010] Merge the recognition results of the storefront classification model and the sign element recognition model, extract the image information on the images of the storefront signs of offline stores, and determine whether the images of the storefront signs of offline stores meet the requirements of the dealer.

[0011] Further, the construction of the storefront classification model and the sign element recognition model includes:

[0012] Obtain the images of the storefront signs in the actual scene collected in advance, and perform classification annotation and sign element annotation on the images of the storefront signs;

[0013] Perform classification training on the images of the storefront signs to obtain a storefront classification model, and deploy the model;

[0014] Perform recognition training on the store signs of the storefront signs to obtain a sign element recognition model, and deploy the model.

[0015] Further, the obtaining of the images of the storefront signs in the actual scene collected in advance, and the performing of classification annotation and sign element annotation on the images of the storefront signs, includes:

[0016] Obtain the images of the storefront signs in the actual scene collected in advance as the training data set. Among them, the collected images include qualified storefront sign images, unqualified storefront sign images, and images of other scenes;

[0017] Perform classification annotation on the collected images to obtain a classification data set. Among them, the annotation categories include qualified storefront sign images (the image contains a storefront and there is a store sign above the storefront), unqualified storefront sign images (the image only contains a storefront or only contains a store sign), and other types of images (images that do not belong to the above two categories);

[0018] Perform element annotation on the collected images to obtain an element recognition data set. The annotated elements are the dealer logo, dealer name, and other elements associated with the dealer in the store sign. The annotation method is not limited to the rectangular box annotation method in object detection;

[0019] Among them, the classification data set and the sign element recognition data set are not limited to the same data set.

[0020] Further, the performing of classification training on the images of the storefront signs to obtain a storefront classification model, and the deployment of the model, includes:

[0021] Train a storefront classification model using the labeled data. The storefront classification model includes, but is not limited to, ResNet, VGG, MobileNet, or ShuffleNet;

[0022] Quantize and accelerate the trained storefront classification model. The model deployment methods are not limited to cloud deployment and mobile deployment.

[0023] Further, the training of the store sign element recognition for the storefront sign image to obtain a sign element recognition model and the deployment of the model include:

[0024] Train a sign element recognition model using the labeled data. The sign element recognition model includes, but is not limited to, SSD, YOLO, or FasterRCNN;

[0025] Quantize and accelerate the trained sign element recognition model. The model deployment methods are not limited to cloud deployment and mobile deployment.

[0026] Further, the merging of the recognition results of the storefront classification model and the sign element recognition model, the extraction of the image information on the offline storefront sign image, and the judgment of whether the offline storefront sign image meets the requirements of the dealer include:

[0027] Perform a first judgment on the recognition results of the merged storefront classification model and the sign element recognition model, including: merging the inference outputs of the two models. If the classification result of the storefront classification model is a qualified storefront and the sign element recognition model detects the dealer's logo, the dealer's name, and other dealer-related elements on the store sign, the image meets the requirements;

[0028] Perform a second judgment on the image information extracted around the store sign elements, including: for the images that meet the requirements in the first judgment, extract the image information around the store sign elements, and judge whether the color of the sign is the color specified by the dealer. If it is the color specified by the dealer, the image meets the requirements and the storefront sign is qualified;

[0029] Calculate the recognition accuracy, including: for a batch of randomly collected validation data, according to the classification results of the storefront classification model, the recognition results of the sign element recognition model, and the color recognition results, the accuracy calculation formula is as follows:

[0030]

[0031] D: The data set used for verification; The set that is actually a positive sample and is predicted as a positive sample by the classification model;

[0032] It is actually a set of positive samples where the prediction of the positive sample sign element recognition model is also positive;

[0033] It is actually a set of negative samples where the prediction of the negative sample classification model is also negative;

[0034] It is actually a set of negative samples where the prediction of the negative sample sign element recognition model is also negative;

[0035] p colour : The accuracy rate of color recognition.

[0036] Furthermore, the recognition order of the storefront classification model and the sign element recognition model is not sequential, and they can also be recognized simultaneously.

[0037] Furthermore, the first judgment of the recognition results of the storefront classification model and the sign element recognition model is not limited to whether the inference results of the two models are merged. It is also possible to separately judge the inference results of each model and then merge the judgment results.

[0038] Furthermore, in the second judgment of the image information around the extracted store sign elements, the store sign color recognition method includes:

[0039] Obtain multiple image patches on the store sign;

[0040] Convert the color space of each image patch from RGB to HSV;

[0041] Judge the color of the image patch using the H value range of each image patch in the HSV color space;

[0042] Vote to judge the sign color based on the colors of all image patches.

[0043] In a second aspect, the present invention provides a device for recognizing storefront signs based on image recognition technology, including:

[0044] An acquisition and recognition unit for acquiring an image of an offline storefront sign and performing recognition through a pre-constructed storefront classification model and sign element recognition model;

[0045] A merging, extraction, and judgment unit for merging the recognition results of the storefront classification model and the sign element recognition model, extracting the image information on the offline storefront sign image, and judging whether the offline storefront sign image meets the requirements of the dealer.

[0046] Compared with the prior art, the beneficial effects achieved by the present invention:

[0047] The present invention provides a method and device for identifying store front signs based on image recognition technology, which can efficiently determine whether the content and placement method of store front signs are qualified, improve the efficiency of enterprises in managing offline stores, and reduce the management cost of stores. Description of the Drawings

[0048] Figure 1 is a schematic flowchart of a method for identifying store front signs based on image recognition technology provided by an embodiment of the present invention;

[0049] Figure 2 is a schematic flowchart of color recognition provided by an embodiment of the present invention. Detailed Embodiments

[0050] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention.

[0051] Embodiment 1

[0052] This embodiment introduces a method for identifying store front signs based on image recognition technology, including:

[0053] Obtain an image of a store front sign offline, and perform recognition through a pre-constructed storefront classification model and a sign element recognition model;

[0054] Merge the recognition results of the storefront classification model and the sign element recognition model, extract the image information on the store front sign image offline, and determine whether the store front sign image offline meets the requirements of the dealer.

[0055] As Figures 1 to 2 shown, the method for identifying store front signs based on image recognition technology provided by this embodiment specifically involves the following steps in its application process:

[0056] (1) According to actual needs, collect an image of a store front sign in an actual scene, and perform classification annotation and sign element annotation on the collected image;

[0057] (2) Perform classification training on the collected images to obtain a storefront classification model, and deploy the model;

[0058] (3) Perform training on the identification of store sign elements for the collected images to obtain a sign element recognition model, and deploy the model;

[0059] (4) Merge the recognition results of the storefront classification model and the sign element recognition model, extract the image information on the store sign, and determine whether the store front sign meets the requirements of the dealer.

[0060] Specifically, the following steps are adopted in step (1) to construct a training set, and the specific steps are as follows:

[0061] (1-1) Collect images of storefront signs in the actual scene: Collect images of storefront signs in the actual scene as the training data set. The collected images should include qualified storefront sign images, unqualified storefront sign images, and images of other scenes;

[0062] (1-2) Image classification annotation: Perform classification annotation on the collected images. The annotation categories include qualified storefront sign images (the image contains a storefront and there is a store sign above the storefront), unqualified storefront sign images (the image only contains a storefront or only contains a store sign), and other types of images (images that do not belong to the above two categories).

[0063] (1-3) Sign element annotation: Perform element annotation on the collected images. The annotated elements are the dealer logo, dealer name, and other elements related to the dealer in the store sign. The annotation method is not limited to the rectangular box annotation method in object detection.

[0064] (1-4) The classification data set and the sign element recognition data set are not limited to the same data set.

[0065] Specifically, the following steps are adopted in step (2) for the training and deployment of the classification model:

[0066] (2-1) Train the storefront classification model: Use the annotated data to train the classification model. The classification model includes but is not limited to Resent, VGG, MobileNet, or ShuffleNet, etc.;

[0067] (2-2) Inference and deployment of the storefront classification model: Quantize and accelerate the trained classification model. The model deployment method is not limited to cloud deployment and mobile deployment.

[0068] Specifically, the following steps are adopted in step (3) for the training and deployment of the store sign element recognition model:

[0069] (3-1) Train the classification model: Use the annotated data to train the store sign element recognition model. The store sign element recognition model includes but is not limited to SSD, YOLO, or FasterRCNN, etc.;

[0070] (3-2) Inference and deployment of the recognition model: Quantize and accelerate the trained store sign element recognition model. The model deployment method is not limited to cloud deployment and mobile deployment.

[0071] Specifically, the following steps are adopted in step (4) to judge the storefront sign:

[0072] (4-1) Combine the inference results of the storefront classification model and the recognition model for the first judgment: Combine the inference outputs of the two models. If the classification result of the storefront classification model is a storefront, and the sign element recognition model detects the dealer's logo, the dealer's name, and other dealer-related elements on the store sign, then the image meets the requirements and the storefront store sign is qualified;

[0073] (4-2) Extract the image information around the store sign elements for the second judgment: For the images that meet the requirements in the first judgment, extract the image information around the store sign elements, and judge whether the color of the sign is the color specified by the dealer. If it is the color specified by the dealer, then the image meets the requirements and the storefront store sign is qualified;

[0074] (4-3) Recognition accuracy calculation: For a batch of randomly collected verification data, according to the classification result of the storefront classification model, the recognition result of the sign element recognition model, and the color recognition result, the accuracy calculation formula is as follows:

[0075]

[0076] D: The data set used for verification; Actually, it is the set where the positive sample classification model predicts a positive sample;

[0077] Actually, it is the set where the positive sample sign element recognition model predicts a positive sample;

[0078] Actually, it is the set where the negative sample classification model predicts a negative sample;

[0079] Actually, it is the set where the negative sample sign element recognition model predicts a negative sample;

[0080] p colour : The accuracy of color recognition.

[0081] Specifically, the inference order of the storefront classification model and the sign element recognition model is not sequential, and they can also perform inference simultaneously.

[0082] Specifically, the first judgment on the inference results of the (4-1) storefront classification model and the sign element recognition model is not limited to whether the inference results of the two models are combined. It can also separately judge the inference results of each model and then combine the judgment results.

[0083] Specifically, the (4-2) method for recognizing the color of the store sign is as follows:

[0084] (4-2-1) Obtain multiple image patches on the store sign;

[0085] (4-2-2)Convert the color space of each image block from RGB to HSV;

[0086] (4-2-3)Judge the color of the image block by using the H value range of each image block in the HSV color space;

[0087] (4-2-4)Vote to judge the signboard color according to the colors of all image blocks.

[0088] Embodiment 2

[0089] This embodiment provides a device for identifying the storefront signboard based on image recognition technology, including:

[0090] An acquisition and recognition unit, configured to acquire an image of the offline storefront signboard and perform recognition through a pre-constructed storefront classification model and signboard element recognition model;

[0091] A merging, extracting and judging unit, configured to merge the recognition results of the storefront classification model and the signboard element recognition model, extract the image information on the offline storefront signboard image, and judge whether the offline storefront signboard image meets the requirements of the dealer.

[0092] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for identifying the storefront signs based on image recognition technology, characterized in that, it includes: Obtain the image of the storefront sign of the offline store, and perform identification through a pre-constructed storefront classification model and a sign element recognition model; Merge the recognition results of the storefront classification model and the sign element recognition model, extract the image information on the image of the storefront sign of the offline store, and determine whether the image of the storefront sign of the offline store meets the requirements of the dealer. Specifically, it includes: Perform the first judgment on the recognition results of the merged storefront classification model and sign element recognition model, including: merge the inference outputs of the two models. If the classification result of the storefront classification model is a qualified storefront, and the sign element recognition model detects the dealer's logo, the dealer's name and other dealer-related elements on the store sign, then the image meets the requirements; Perform the second judgment on the image information extracted around the store sign elements, including: for the image that meets the requirements in the first judgment, extract the image information around the store sign elements, and judge whether the color of the sign is the color specified by the dealer. If it is the color specified by the dealer, then the image meets the requirements and the storefront sign is qualified; Calculate the recognition accuracy, including: for a batch of randomly collected verification data, according to the classification result of the storefront classification model, the recognition result of the sign element recognition model and the color recognition result; The accuracy calculation formula is as follows: ； D: The data set for verification; : The set that is actually a positive sample and is also predicted as a positive sample by the positive sample classification model; : It is actually the set of positive samples where the prediction of the positive sample sign element recognition model is also positive; : The set where the actual negative sample classification model prediction is also a negative sample; : It is actually the set of negative samples where the prediction of the negative sample sign element recognition model is also negative; : Accuracy of color recognition.

2. The method for identifying the storefront signs based on image recognition technology according to claim 1, characterized in that, The construction of the storefront classification model and the sign element recognition model includes: Obtain the images of the storefront signs in the actual scene collected in advance, and perform classification annotation and sign element annotation on the images of the storefront signs; Perform classification training on the images of the storefront signs to obtain a storefront classification model, and deploy the model; Perform recognition training on the store sign elements of the images of the storefront signs to obtain a sign element recognition model, and deploy the model.

3. The method for identifying the storefront signs based on image recognition technology according to claim 2, characterized in that, The obtaining of the images of the storefront signs in the actual scene collected in advance and the performing of classification annotation and sign element annotation on the images of the storefront signs includes: Obtain the images of the storefront signs in the actual scene collected in advance as the training data set. Among them, the collected images include qualified storefront sign images, unqualified storefront sign images and images of other scenes; Perform classification annotation on the collected images to obtain a classification data set. Among them, the annotation categories include qualified storefront sign images (the image contains a storefront and there is a store sign above the storefront), unqualified storefront sign images (the image only contains a storefront or only contains a store sign), and other types of images (images that do not belong to the above two categories); Perform element annotation on the collected images to obtain an element recognition data set. The annotated elements are the dealer's logo, the dealer's name and other dealer-related elements in the store sign. The annotation method is not limited to the rectangular box annotation method in object detection; Among them, the classification dataset and the signboard element recognition dataset are not limited to the same dataset.

4. The method for identifying a storefront signboard based on image recognition technology according to claim 2, characterized in that the classification training of the storefront signboard image to obtain a signboard classification model and the deployment of the model include: training a signboard classification model using the annotated data, and the signboard classification model includes but is not limited to ResNet, VGG, MobileNet or ShuffleNet; quantizing and accelerating the trained signboard classification model, and the model deployment method is not limited to cloud deployment and mobile deployment.

5. The method for identifying a storefront signboard based on image recognition technology according to claim 2, characterized in that the recognition training of the storefront signboard elements in the storefront signboard image to obtain a signboard element recognition model and the deployment of the model include: training a signboard element recognition model using the annotated data, and the signboard element recognition model includes but is not limited to SSD, YOLO or FasterRCNN; quantizing and accelerating the trained signboard element recognition model, and the model deployment method is not limited to cloud deployment and mobile deployment.

6. The method for identifying a storefront signboard based on image recognition technology according to claim 1, characterized in that the recognition order of the signboard classification model and the signboard element recognition model is not in sequence, and they can also be recognized simultaneously.

7. The method for identifying a storefront signboard based on image recognition technology according to claim 1, characterized in that the first judgment of the recognition results of the signboard classification model and the signboard element recognition model is not limited to whether the inference results of the two models are merged, and the judgment results can also be merged after separately judging the inference results of each model.

8. The method for identifying a storefront signboard based on image recognition technology according to claim 1, characterized in that in the second judgment of the image information around the extracted storefront signboard elements, the storefront signboard color recognition method includes: obtaining a plurality of image blocks on the storefront signboard; converting the color space of each image block from RGB to HSV; judging the color of the image block using the H value range of each image block in the HSV color space; voting to judge the signboard color according to the colors of all the image blocks.

9. An apparatus for identifying a storefront signboard based on image recognition technology, characterized in that it includes: an acquisition and recognition unit for acquiring an offline storefront signboard image and performing recognition through a pre-constructed signboard classification model and signboard element recognition model; a merging, extraction and judgment unit for merging the recognition results of the signboard classification model and the signboard element recognition model, extracting the image information on the offline storefront signboard image, and judging whether the offline storefront signboard image meets the requirements of the dealer, specifically including: Make the first judgment on the recognition results of the merged storefront classification model and the sign element recognition model, including: merging the inference outputs of the two models. If the classification result of the storefront classification model is a qualified storefront, and the sign element recognition model detects the dealer's logo, the dealer's name and other dealer-related elements on the store sign, then the image meets the requirements; Make the second judgment on the image information extracted around the store sign elements, including: for the images that meet the requirements in the first judgment, extract the image information around the store sign elements, and judge whether the color of the sign is the color specified by the dealer. If it is the color specified by the dealer, then the image meets the requirements and the storefront store sign is qualified; Calculate the recognition accuracy rate, including: for a batch of randomly collected verification data, according to the classification results of the storefront classification model, the recognition results of the sign element recognition model and the color recognition results; The accuracy rate calculation formula is as follows: ； D: The data set used for verification; : The set that is actually a positive sample and is also predicted as a positive sample by the positive sample classification model; : It is actually the set of positive samples predicted by the positive sample sign element recognition model; : The set where the actual negative sample classification model predicts negative samples as well; : The set where the actual negative sample sign element recognition model also predicts negative samples; : Accuracy of color recognition.

Images