Goods shelf display detection method, system and equipment and computer storage medium

Abstract

The invention discloses a shelf display detection method, system and device and a computer storage medium, and relates to the technical field of image detection and recognition, and the method comprises the steps: obtaining a to-be-recognized target shelf map; inputting the target shelf map into a trained shelf object detection model to obtain identification position information of each to-be-identified object in the target shelf map; inputting the target shelf map into a trained shelf object recognition model to obtain recognition feature information of each to-be-recognized object in the target shelf map; and determining the identification position information and the identification feature information as a detection result of the to-be-identified object. According to the method, the target shelf map can be detected by means of the shelf object detection model and the shelf object recognition model, the detection result of the to-be-recognized object in the target shelf map can be obtained without manual participation, and both timeliness and accuracy are high; and the goods shelf object detection model and the goods shelf object identification model are two mutually independent models and can be adjusted according to requirements, so that the flexibility and the applicability are relatively good.

Description

Technical Field

[0001] The present application relates to the technical field of image detection and recognition, and more specifically, to a shelf display detection method, system, device and computer storage medium. Background Art

[0002] At present, all kinds of retail industries are pursuing scientific and intelligent store management, and the display management of store shelf goods is a concrete manifestation of this pursuit. Shelf display refers to placing and displaying products on the shelves in a regular and purposeful manner according to the pre-made sales plan with certain methods and techniques, taking the shelf objects as the main body.

[0003] However, shelf object display detection relies on manual work, such as manually checking the acquired store site images. This is a time-consuming and laborious process, and it is difficult to guarantee the timeliness and accuracy of the display detection work.

[0004] In summary, how to improve the timeliness and accuracy of shelf display detection is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] The purpose of this application is to provide a shelf display detection method, which can solve the technical problem of how to improve the timeliness and accuracy of shelf display detection to a certain extent. This application also provides a shelf display detection system, an electronic device and a computer-readable storage medium.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A shelf display detection method, comprising:

[0008] Obtain a target planogram to be identified;

[0009] Inputting the target planogram into a trained shelf object detection model to obtain identification position information of each object to be identified in the target planogram;

[0010] Inputting the target planogram into a trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram;

[0011] The identification position information and the identification feature information are determined as the detection result of the object to be identified.

[0012] Preferably, the identification position information includes detection frame coordinate information, and before inputting the target planogram into a trained shelf object detection model, the method further includes:

[0013] Obtaining a target object detection model trained with a public data set, wherein the target object detection model is built based on a two-stage target detection model or a one-stage target detection model;

[0014] Acquire a pre-labeled first shelf dataset, where the first shelf dataset includes a first planogram and detection box coordinate information of each object in the first planogram;

[0015] The target object detection model is trained using the first shelf data set to obtain a trained shelf object detection model.

[0016] Preferably, before inputting the target planogram into the trained shelf object recognition model, the method further comprises:

[0017] Obtaining a target recognition model trained with a public data set, wherein the target recognition model is built based on a person re-identification model;

[0018] Acquire a pre-labeled second shelf data set, where the second shelf data set includes a second planogram and feature information of each object in the second planogram;

[0019] The second shelf data set is used to train the object recognition model to obtain a trained shelf object recognition model.

[0020] Preferably, the target recognition model is built based on a RegNet classification model and three fully connected layers, and the three fully connected layers are connected to the last layer of the RegNet classification model.

[0021] Preferably, the second planogram includes a randomly selected anchor planogram, a positive sample planogram of the same category as the anchor planogram, and a negative sample planogram of a different category from the anchor planogram;

[0022] The process of using the second shelf data set to train the object recognition model includes:

[0023] Generate a first feature vector distance between the anchor planogram and the positive sample planogram after forward propagation of the target recognition model;

[0024] Generate a second feature vector distance between the anchor planogram and the negative sample planogram after forward propagation of the target recognition model;

[0025] generating a triplet loss function value according to the first feature vector distance, the second feature vector distance and a training threshold;

[0026] Adding a fully connected layer at the end of the target recognition model to obtain a category recognition model;

[0027] generating a cross entropy loss function value according to a result of the category recognition model on the category of the object in the second planogram;

[0028] A target loss function value of the target recognition model is generated according to the triplet loss function value and the cross entropy loss function value, so as to adjust the target recognition model based on the target loss function value.

[0029] Preferably, after determining the identification position information and the identification feature information as the detection result of the object to be identified, the method further includes:

[0030] Determining the order of the objects to be identified on the shelf according to the identification position information;

[0031] Acquire set shelf display standard information, wherein the shelf display standard information includes target feature information of each order display object;

[0032] Comparing the identification feature information and the target feature information of the same order for consistency, if they are consistent, determining that the display of the object to be identified is qualified, if they are inconsistent, determining that the display of the object to be identified is unqualified;

[0033] Mark the objects to be identified that are qualified and unqualified in the target shelf map respectively, and obtain a comparison result map for display;

[0034] The display qualification rate of the target planogram is generated and displayed.

[0035] Preferably, the step of inputting the target planogram into a trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram includes:

[0036] The identification position information and the target planogram are input into a trained shelf object recognition model to obtain identification feature information of each object to be identified in the target planogram.

[0037] A shelf display detection system, comprising:

[0038] A first acquisition module is used to acquire a target planogram to be identified;

[0039] A first detection module is used to input the target planogram into a trained shelf object detection model to obtain identification position information of each object to be identified in the target planogram;

[0040] A first recognition module, configured to input the target planogram into a trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram;

[0041] The first processing module is used to determine the identification position information and the identification feature information as the detection result of the object to be identified.

[0042] An electronic device, comprising:

[0043] Memory for storing computer programs;

[0044] A processor is used to implement the steps of any of the above shelf display detection methods when executing the computer program.

[0045] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above shelf display detection methods are implemented.

[0046] The present application provides a shelf display detection method, which includes obtaining a target shelf map to be identified; inputting the target shelf map into a trained shelf object detection model to obtain the identification position information of each object to be identified in the target shelf map; inputting the target shelf map into a trained shelf object recognition model to obtain the identification feature information of each object to be identified in the target shelf map; and determining the identification position information and the identification feature information as the detection result of the object to be identified. After obtaining the target shelf map to be identified, the present application can use the shelf object detection model to detect the identification position information of each object to be identified in the target shelf map, and can use the shelf object recognition model to detect the identification feature information of each object to be identified in the target shelf map, thereby obtaining the detection result of the object to be identified in the target shelf map without human intervention, with high timeliness and accuracy; and the shelf object detection model and the shelf object recognition model are two independent models, which can be adjusted according to actual use requirements, and have good flexibility and applicability. The shelf display detection system, electronic device and computer-readable storage medium provided by the present application also solve the corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0048] Figure 1 A flowchart of a shelf display detection method provided in an embodiment of the present application;

[0049] Figure 2 Schematic diagram of model training and application;

[0050] Figure 3Schematic diagram of shelf display detection for cloud servers;

[0051] Figure 4 A schematic diagram of the structure of a shelf display detection system provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0053] Figure 6 Another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] See also Figure 1 , Figure 1 A flowchart of a shelf display detection method provided in an embodiment of the present application.

[0056] A shelf display detection method provided in an embodiment of the present application may include the following steps:

[0057] Step S101: obtaining a target planogram to be identified.

[0058] In practical applications, a target shelf map to be identified may be obtained first. The target shelf map to be identified may be an image obtained by a user after photographing the shelf, etc. The target shelf map may include the shelf and objects to be identified on the shelf, etc. The type of the object to be identified may be flexibly determined according to actual needs.

[0059] Step S102: input the target planogram into the trained shelf object detection model to obtain the identification position information of each object to be identified in the target planogram.

[0060] In practical applications, considering that the positions of various objects to be identified on the shelf are different, it is necessary to identify the category of the object to be identified and the position of the object to be identified on the shelf in order to identify the shelf map. Therefore, after obtaining the target shelf map, the position of the object to be identified on the shelf can be identified first, that is, the target shelf map can be input into the trained shelf object detection model to obtain the identification position information of each object to be identified in the target shelf map, which can be specifically the identification position information of the detection frame of the object to be identified on the shelf, so as to know the position of each object to be identified on the shelf through this identification position information, and the identification position information can also be used as reference information for class identification of the object to be identified, etc., which is not specifically limited in this application. In addition, in order to expand the cognition of the object to be identified, the shelf object detection model can also output the appearance type and confidence level of the object to be identified, such as the confidence level of the appearance of the object to be identified, such as the appearance of the boxed, bagged, bottled, etc.

[0061] In specific application scenarios, the shelf object recognition model needs to be obtained through training, that is, before the recognition position information includes the detection box coordinate information and the target shelf map is input into the trained shelf object detection model, the target object detection model trained with a public data set can also be obtained. The target object detection model is built based on a two-stage target detection model or a one-stage target detection model. The public data set can be a large-scale public data set such as ImageNet and / or MSCOCO. Although the target object detection model at this time has the target object detection capability, it may not be suitable for shelf object detection. Therefore, it is necessary to obtain a pre-labeled first shelf data set, which includes a first shelf map and the detection box coordinate information of each object in the first shelf map. The target object detection model is trained using the first shelf data set to obtain a trained shelf object detection model.

[0062] It should be noted that the two-stage target detection model refers to a target detection model that includes two stages, region proposal and detection, in the algorithm model. When performing object detection, such models can first use modules such as Selective Search (SS) or Region Proposal Network (RPN) to obtain candidate target regions, and then obtain the final detection results through modules such as classifiers and bounding box regressors. Such models can include Faster R-CNN, SPPNet models, etc. The one-stage target detection model refers to a model that omits the region candidate stage compared to the two-stage target detection model. The output layer of its network model directly generates the category probability and position coordinates of the target object, such as YOLOv4 / v5, SSD and RetinaNet models.

[0063] Step S103: input the target planogram into the trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram.

[0064] In practical applications, after determining the location information of the object to be identified on the shelf, the category of the object to be identified can be identified, that is, the target shelf map can be input into the trained shelf object recognition model to obtain the identification feature information of each object to be identified in the target shelf map, so as to record the features of the object to be identified through this identification feature information, thereby reflecting the category of the object to be identified through this feature. The type of identification feature information can be determined according to application requirements, such as the feature vector of the object to be identified, etc. In addition, display detection can be performed on a single shelf image, or display detection can be performed on shelf maps in batches, etc.

[0065] It should be noted that the recognition method of the shelf object recognition model for the object feature information in the planogram can be determined according to the training method of the shelf object recognition model. For example, when only the planogram is used as a training sample to train the shelf object recognition model, the target planogram can be input into the trained shelf object recognition model to obtain the recognition feature information of each object to be recognized in the target planogram; and when the planogram and the location information of the objects in the planogram are used as training samples to train the shelf object recognition model, the recognition location information and the target shelf position information can be input into the trained shelf object recognition model. Figure 1 The trained shelf object recognition model is input to obtain the recognition feature information of each object to be recognized in the target shelf image.

[0066] In a specific application scenario, before inputting the target shelf map into the trained shelf object recognition model, the shelf object recognition model can also be trained, that is, to obtain a target recognition model trained with a public data set, and the target recognition model is built based on a pedestrian re-identification model; obtain a pre-labeled second shelf data set, the second shelf data set includes a second shelf map and feature information of each object in the second shelf map; use the second shelf data set to train the target recognition model to obtain a trained shelf object recognition model. The entire model training and application process is as follows: Figure 2 As shown in the figure, the entire model training uses the pre-trained model as the implementation basis of the detection model and the recognition model, from general multi-category target detection to target detection of specific categories in specific scenarios (store shelf goods), from general category image feature extraction to specific category image feature extraction, which fully taps the potential for reuse of deep learning network models and can achieve better detection and recognition effects at a lower training cost.

[0067] It should be noted that person re-identification (ReID) aims to identify and match images of the same pedestrian across different cameras or time points. The key to ReID technology is to extract recognizable features from pedestrian images captured from different perspectives and conditions in order to accurately match them in large databases. ReID systems usually include the following key steps. Feature extraction: Use a deep learning model (such as a convolutional neural network CNN) to extract feature vectors that can characterize the identity of pedestrians from pedestrian images. These features need to be robust to posture changes, lighting changes, occlusion, etc. Therefore, a public dataset of pedestrian images can be used to train the pedestrian re-identification model to obtain a target recognition model. And considering that pedestrians and shelves are quite different, in order to make the target recognition model suitable for shelves, this application needs to use shelf-related data to continue training the target recognition model to obtain a shelf object recognition model that can perform feature recognition on objects to be identified in the shelf map.

[0068] In specific application scenarios, the structure of the target recognition model can be flexibly adjusted according to actual needs. For example, it can be built based on the RegNet classification model and three layers of fully connected layers, and the three layers of fully connected layers are sequentially connected to the last layer of the RegNet classification model, and the target recognition model can output a 256-dimensional feature vector as the feature vector of the object to be recognized. Among them, the RegNet classification model can be composed of stem, body and head. The stem can be a convolution layer (including bn and relu by default), the convolution kernel size can be 3x3, the stride can be 2, and the number of convolution kernels can be 32; the body can be composed of 4 stage stacks, and each stage will reduce the height and width of the input feature matrix to half of the original, and each stage is composed of a series of block stacks. In the first block of each stage, there are group convolutions (on the main branch) and ordinary convolutions (on the shortcut branch) with a stride of 2, and the convolution strides in the remaining blocks are all 1, similar to ResNet; the head can be a common classifier in the classification network, consisting of a global average pooling layer and a fully connected layer. The specific structure of the RegNet classification model can be flexibly adjusted according to the application scenario.

[0069] In a specific application scenario, the second planogram may include a randomly selected anchor planogram, a positive planogram of the same category as the anchor planogram, and a negative planogram of a different category from the anchor planogram; accordingly, in the process of using the second shelf data set to train the target recognition model, a first feature vector distance between the anchor planogram and the positive planogram after forward propagation of the target recognition model can be generated, that is, the distance between the positive sample pair in the feature space can be generated; a second feature vector distance between the anchor planogram and the negative sample planogram after forward propagation of the target recognition model can be generated, that is, the distance between the negative sample pair in the feature space can be generated; according to the first feature vector distance between the anchor planogram and the negative sample planogram after forward propagation of the target recognition model, The eigenvector distance, the second eigenvector distance and the training threshold are used to generate a triplet loss function value; a fully connected layer is added at the end of the target recognition model to obtain a category recognition model, and the category recognition model is used to output the number of categories of the object to be recognized; according to the category recognition result of the object in the second shelf map by the category recognition model, a cross entropy loss function value is generated; according to the triplet loss function value and the cross entropy loss function value, a target loss function value of the target recognition model is generated. For example, the triplet loss function value and the cross entropy loss function value can be added to generate the target loss function value, etc., so as to adjust the target recognition model based on the target loss function value.

[0070] It should be noted that the feature vector distance can be Euclidean distance or cosine distance, etc. Assume that the normalized feature vectors obtained after the two shelf images are forward propagated through the target recognition model are and , then the Euclidean distance can be defined as , the cosine distance can be defined as . And the formula for generating the triplet loss function value can be ,in, represents the triple loss function value, represents the first eigenvector distance, represents the second eigenvector distance, represents the training threshold, a represents the anchor planogram, p represents the positive sample planogram, n represents the negative sample planogram, () + It means taking the larger of the number in the brackets and 0. And the formula for generating the cross entropy loss function value can be ,in, Represents the cross entropy loss function value, N represents the total number of samples, and M represents the number of categories; represents a symbolic function, and the true category of sample i is equal to c Take 1, otherwise Take 0; represents the predicted probability that sample i belongs to category c.

[0071] It should also be noted that the present application divides the shelf object recognition process into two subtasks, target detection and target recognition, and completes them in a collaborative manner using a shelf object detection model and a shelf object recognition model. On this basis, the shelf object detection model and the shelf object recognition model are two independent models. Different detection or recognition model frameworks can be replaced according to actual usage requirements without affecting the final realization of the entire task. Moreover, the shelf object detection model can regard all shelf objects as one category or divide them into only a few simple categories based on their appearance and shape. Therefore, changes in commodity categories do not affect the architecture of the shelf object detection model; the shelf object recognition model adopts metric learning rather than a classification network. It directly learns the similarities between images by constructing a network, and does not require a classification layer to assist feature learning. Therefore, it is also robust to changes in shelf object categories. In summary, the technical solution of the present application has strong flexibility, and the collaborative operation of the two models can effectively ensure the reliability and accuracy of the final display detection results.

[0072] Step S104: determining the recognition position information and the recognition feature information as the detection result of the object to be recognized.

[0073] In practical applications, after obtaining the identification position information and the identification feature information, the identification position information and the identification feature information can be determined as the detection result of the object to be identified, so that the user can know the position of each object to be identified on the shelf through this detection result.

[0074] In practical applications, after the identification position information and the identification feature information are determined as the detection results of the object to be identified, it is also possible to detect whether the display method of the object to be identified in the target shelf map meets the user's needs, that is, the order of the object to be identified on the shelf can be determined according to the identification position information. This order can be the number of rows and columns of the object to be identified on the shelf. For example, the ordinate of the lower right corner of the detection frame of the detection result can be used as the dividing line of the shelf layer number to determine the layer where the commodity is located, and the detection frame of each layer is sorted according to the size of the horizontal coordinate of the point in the middle, so as to obtain the left and right order of the commodity in the detection frame in the layer; obtain the set shelf display standard information, and the shelf display standard information includes the target feature information of the objects displayed in each order set by the user; compare the identification feature information and the target feature information of the same order for consistency, if they are consistent, it is determined that the display of the object to be identified is qualified, if they are inconsistent, it is determined that the display of the object to be identified is unqualified; mark the objects to be identified that are displayed qualifiedly and unqualifiedly in the target shelf map respectively, and obtain a comparison result map for display; generate a display pass rate of the target shelf map for display. In addition, after determining that the display of the objects to be identified is qualified and determining that the display of the objects to be identified is unqualified, the display qualification rate of the shelf can also be calculated for user reference, etc. The display qualification rate can be generated by a formula of (100*s / v)%, where s represents the number of objects to be identified that are displayed qualifiedly, and v represents the total number of objects to be identified on the shelf.

[0075] In a specific application scenario, in the process of obtaining the set shelf display standard information, the set shelf display standard map can be obtained; the shelf display standard map is input into the shelf object detection model to obtain the target position information of each object in the shelf display standard map, and the order of each object in the shelf display standard map is determined according to the target position information; the shelf display standard map is input into the shelf object recognition model to obtain the target feature information of each object in the shelf display standard map, or the shelf display standard map and the target position information can be input into the shelf object recognition model together to obtain the target feature information of each object in the shelf display standard map.

[0076] The present application provides a shelf display detection method, which obtains a target shelf map to be identified; inputs the target shelf map into a trained shelf object detection model to obtain identification position information of each object to be identified in the target shelf map; inputs the target shelf map into a trained shelf object recognition model to obtain identification feature information of each object to be identified in the target shelf map; and determines the identification position information and the identification feature information as the detection result of the object to be identified. After obtaining the target shelf map to be identified, the present application can use the shelf object detection model to detect the identification position information of each object to be identified in the target shelf map, and can use the shelf object recognition model to detect the identification feature information of each object to be identified in the target shelf map, thereby obtaining the detection result of the object to be identified in the target shelf map without human intervention, with high timeliness and accuracy; and the shelf object detection model and the shelf object recognition model are two independent models, which can be adjusted according to actual use needs, and have good flexibility and applicability.

[0077] In order to facilitate understanding of the shelf display detection method provided in this application, it is assumed that the shelf display detection is performed through a cloud server, and the whole process can be as follows: Figure 3 As shown, the following steps are included:

[0078] Users use mobile devices such as smartphones, tablets, cameras and other devices to take photos of shelves at the store. They interactively frame the images of shelves taken, select the shelf areas that need to be inspected, and upload them to the cloud server.

[0079] The cloud server receives information from the mobile terminal and performs data preprocessing based on the shelf display information. For example, it obtains a cropped target shelf map based on the uploaded data information.

[0080] The cloud server loads the shelf object detection model and the shelf object recognition model, inputs the target shelf map into the trained shelf object detection model, and obtains the recognition position information of each object to be recognized in the target shelf map; inputs the target shelf map and the recognition position information into the trained shelf object recognition model, and obtains the recognition feature information of each object to be recognized in the target shelf map, and then obtains the detection result;

[0081] The cloud server uses the shelf object detection model and the shelf object recognition model to detect the standard shelf real-shot image, obtain the shelf display standard information, and generate the shelf display standard image according to the position information and feature information in the shelf display standard information. The shelf display standard image includes the feature information of the objects at each order of the shelf.

[0082] The cloud server combines the standard shelf display diagram and compares the feature information in the above detection results with the feature information of the object in the standard shelf display diagram to determine whether they are the same object; after identifying and determining the object at each detection position, according to the comparison results of each position, different color-marked boxes are drawn for different results on the real shelf picture, such as marking the correctly compared position with a green box and the wrong one with a red box, etc. At the same time, the display detection results are counted in the form of a table or text, and the qualified rate of the shelf display is calculated;

[0083] The cloud server sends the comparison result graph, comparison result table and display qualification rate as the inspection data of this shelf display to the mobile terminal;

[0084] The mobile terminal receives the shelf display detection data for the user to view.

[0085] It should be noted that when the shelf display detection of this application is implemented by combining the mobile terminal with the cloud, the mobile terminal, with its flexible and convenient operation mode, is mainly responsible for the collection and upload of store site data and the display of results; the cloud server relies on its powerful computing power to quickly respond and execute complex intelligent algorithms such as detection and recognition based on deep learning, and promptly feed back the display detection results to the mobile terminal. This processing method in which the mobile terminal and the cloud perform their respective duties and cooperate with each other can quickly and accurately reflect the shelf display status, thereby helping users improve the efficiency of shelf object management.

[0086] See also Figure 4 , Figure 4 A schematic diagram of the structure of a shelf display detection system provided in an embodiment of the present application.

[0087] A shelf display detection system provided in an embodiment of the present application may include:

[0088] A first acquisition module 101 is used to acquire a target planogram to be identified;

[0089] The first detection module 102 is used to input the target planogram into the trained shelf object detection model to obtain the identification position information of each object to be identified in the target planogram;

[0090] The first recognition module 103 is used to input the target planogram into the trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram;

[0091] The first processing module 104 is used to determine the recognition position information and the recognition feature information as the detection result of the object to be recognized.

[0092] The embodiment of the present application provides a shelf display detection system, wherein the recognition position information includes the detection frame coordinate information, and may also include:

[0093] A second acquisition module is used to acquire a target object detection model trained with a public data set before the first detection module inputs the target shelf map into the trained shelf object detection model, where the target object detection model is built based on a two-stage target detection model or a one-stage target detection model;

[0094] A third acquisition module is used to acquire a pre-labeled first shelf data set, where the first shelf data set includes a first shelf map and detection box coordinate information of each object in the first shelf map;

[0095] The first training module is used to train the target object detection model using the first shelf data set to obtain a trained shelf object detection model.

[0096] A shelf display detection system provided in an embodiment of the present application may further include:

[0097] A fourth acquisition module is used to acquire a target recognition model trained with a public data set before the first recognition module inputs the target shelf map into the trained shelf object recognition model, where the target recognition model is built based on the pedestrian re-recognition model;

[0098] A fifth acquisition module is used to acquire a pre-labeled second shelf data set, where the second shelf data set includes a second planogram and feature information of each object in the second planogram;

[0099] The second training module is used to train the target recognition model using the second shelf data set to obtain a trained shelf object recognition model.

[0100] An embodiment of the present application provides a shelf display detection system, in which a target recognition model is built based on a RegNet classification model and three fully connected layers, and the three fully connected layers are connected to the last layer of the RegNet classification model.

[0101] In a shelf display detection system provided by an embodiment of the present application, the second planogram includes a randomly selected anchor planogram, a positive sample planogram of the same category as the anchor planogram, and a negative sample planogram of a different category from the anchor planogram;

[0102] The second training module can be used to: generate a first feature vector distance between the anchored shelf map and the positive sample shelf map after forward propagation of the target recognition model; generate a second feature vector distance between the anchored shelf map and the negative sample shelf map after forward propagation of the target recognition model; generate a triplet loss function value according to the first feature vector distance, the second feature vector distance and the training threshold; add a fully connected layer at the end of the target recognition model to obtain a category recognition model; generate a cross entropy loss function value according to the category recognition result of the object in the second shelf map by the category recognition model; generate a target loss function value of the target recognition model according to the triplet loss function value and the cross entropy loss function value, so as to adjust the target recognition model based on the target loss function value.

[0103] A shelf display detection system provided in an embodiment of the present application may further include:

[0104] A first determination module, configured to determine the order of the objects to be identified on the shelf according to the identification position information after the first processing module determines the identification position information and the identification feature information as the detection result of the objects to be identified;

[0105] A sixth acquisition module is used to acquire set shelf display standard information, where the shelf display standard information includes target feature information of objects displayed in each order;

[0106] The first comparison module is used to compare the identification feature information and the target feature information of the same order for consistency. If they are consistent, it is determined that the display of the object to be identified is qualified; if they are inconsistent, it is determined that the display of the object to be identified is unqualified;

[0107] The first marking module is used to mark the objects to be identified that are qualified and unqualified in the target shelf map, respectively, and obtain a comparison result map for display;

[0108] The first generating module is used to generate the display qualification rate of the target planogram for display.

[0109] In a shelf display detection system provided in an embodiment of the present application, the first recognition module may include:

[0110] The feature recognition unit is used to input the recognition position information and the target shelf map into the trained shelf object recognition model to obtain the recognition feature information of each object to be recognized in the target shelf map.

[0111] In a shelf display detection system provided in an embodiment of the present application, the sixth acquisition module may include:

[0112] A first acquisition unit, used to acquire a set shelf display standard diagram;

[0113] The first detection unit is used to input the shelf display standard diagram into the shelf object detection model, obtain the target position information of each object in the shelf display standard diagram, and determine the order of each object in the shelf display standard diagram according to the target position information;

[0114] The first recognition unit is used to input the shelf display standard diagram into the shelf object recognition model to obtain target feature information of each object in the shelf display standard diagram.

[0115] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the shelf display detection method provided in the embodiment of the present application. Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0116] An electronic device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the steps of the shelf display detection method described in any of the above embodiments are implemented.

[0117] See also Figure 6 , another electronic device provided in the embodiment of the present application may also include: an input port 203 connected to the processor 202, for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202, for displaying the processing results of the processor 202 to the outside; a communication module 205 connected to the processor 202, for realizing communication between the electronic device and the outside. The display unit 204 can be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes but is not limited to mobile high-definition link technology (Mobile High-Definition Link, MHL), Universal Serial Bus (Universal Serial Bus, USB), High-Definition Multimedia Interface (High-DefinitionMultimedia Interface, HDMI), wireless connection: wireless fidelity technology (WIreless Fidelity, WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.

[0118] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the shelf display detection method described in any of the above embodiments are implemented.

[0119] The computer-readable storage medium involved in this application includes random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0120] For the description of the relevant parts of the shelf display detection system, electronic device and computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding parts in the shelf display detection method provided in the embodiments of the present application, which will not be repeated here. In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0121] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0122] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A shelf display detection method, characterized in that: include: Obtain a target planogram to be identified; Inputting the target planogram into a trained shelf object detection model to obtain identification position information of each object to be identified in the target planogram; Inputting the target planogram into a trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram; The identification position information and the identification feature information are determined as the detection result of the object to be identified.

2. The method according to claim 1, characterized in that The recognition position information includes detection frame coordinate information, and before inputting the target planogram into a trained shelf object detection model, the method further includes: Obtaining a target object detection model trained with a public data set, wherein the target object detection model is built based on a two-stage target detection model or a one-stage target detection model; Acquire a pre-labeled first shelf dataset, where the first shelf dataset includes a first planogram and detection box coordinate information of each object in the first planogram; The target object detection model is trained using the first shelf data set to obtain a trained shelf object detection model.

3. The method according to claim 1, characterized in that Before inputting the target planogram into the trained shelf object recognition model, the method further includes: Obtaining a target recognition model trained with a public data set, wherein the target recognition model is built based on a person re-identification model; Acquire a pre-labeled second shelf data set, where the second shelf data set includes a second planogram and feature information of each object in the second planogram; The second shelf data set is used to train the object recognition model to obtain a trained shelf object recognition model.

4. The method according to claim 3, characterized in that The target recognition model is built based on the RegNet classification model and three fully connected layers, and the three fully connected layers are connected to the last layer of the RegNet classification model.

5. The method according to claim 4, characterized in that The second planogram includes a randomly selected anchor planogram, a positive sample planogram of the same category as the anchor planogram, and a negative sample planogram of a different category from the anchor planogram; The process of using the second shelf data set to train the object recognition model includes: Generate a first feature vector distance between the anchor planogram and the positive sample planogram after forward propagation of the target recognition model; Generate a second feature vector distance between the anchor planogram and the negative sample planogram after forward propagation of the target recognition model; generating a triplet loss function value according to the first feature vector distance, the second feature vector distance and a training threshold; Adding a fully connected layer at the end of the target recognition model to obtain a category recognition model; generating a cross entropy loss function value according to a result of the category recognition model on the category of the object in the second planogram; A target loss function value of the target recognition model is generated according to the triplet loss function value and the cross entropy loss function value, so as to adjust the target recognition model based on the target loss function value.

6. The method according to claim 1, characterized in that After determining the identification position information and the identification feature information as the detection result of the object to be identified, the method further includes: Determining the order of the objects to be identified on the shelf according to the identification position information; Acquire set shelf display standard information, wherein the shelf display standard information includes target feature information of each order display object; Comparing the identification feature information and the target feature information of the same order for consistency, if they are consistent, determining that the display of the object to be identified is qualified, if they are inconsistent, determining that the display of the object to be identified is unqualified; Mark the objects to be identified that are qualified and unqualified in the target shelf map respectively, and obtain a comparison result map for display; The display qualification rate of the target planogram is generated and displayed.

7. The method according to any one of claims 1 to 6, characterized in that: The step of inputting the target planogram into a trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram includes: The identification position information and the target planogram are input into a trained shelf object recognition model to obtain identification feature information of each object to be identified in the target planogram.

8. A shelf display detection system, characterized in that: include: A first acquisition module is used to acquire a target planogram to be identified; A first detection module is used to input the target planogram into a trained shelf object detection model to obtain identification position information of each object to be identified in the target planogram; A first recognition module, configured to input the target planogram into a trained shelf object recognition model to obtain recognition feature information of each object to be recognized in the target planogram; The first processing module is used to determine the identification position information and the identification feature information as the detection result of the object to be identified.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the shelf display detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the shelf display detection method according to any one of claims 1 to 7 are implemented.

Images