Goods shelf management method and system based on target detection, terminal and medium

Abstract

The invention discloses a goods shelf management method and system based on target detection, a terminal and a medium, and the method comprises the steps: carrying out the target detection of a goods shelf image, and obtaining a plurality of first commodity frames and a plurality of first vacancy frames in the image, and the position information of each first commodity frame and each first vacancy frame; taking the first commodity frame of which the intersection-to-union ratio with any first vacancy frame is smaller than or equal to a first threshold value as a second commodity frame, layering all the first commodity frames and all the first vacancy frames according to all the position information, and analyzing the width reference quantity of each layer; according to the width reference quantity of the layer where each first vacancy frame is located, the total SKU placement quantity of the shelf vacancy is calculated, and when the total SKU placement quantity is larger than a second threshold value, a reminding signal and the total SKU placement quantity are fed back to the monitoring platform. By adopting the embodiment of the invention, the relationship between the first commodity frames with different overlapping degrees and the shelf vacancy is analyzed respectively, so that the efficient evaluation of the shelf vacancy quantity is realized.

Description

Technical Field

[0001] The present invention relates to the field of shelf management, and in particular to a shelf management method, system, terminal and medium based on target detection. Background Art

[0002] As the main means of displaying goods, the rationality of shelf placement and the filling rate have a great impact on the sales of goods. Especially in the field of fast-moving consumer goods, it is easy for some positions on the shelves to be empty for a long time, or there are many empty spaces. This not only reduces the filling rate of goods, but also leads to inventory backlogs, which ultimately affects the sales of goods. In the past, whether the placement of shelves is reasonable or not mainly depends on the salesperson's on-site shelf inspection or remote camera inspection. This process is not only time-consuming and labor-intensive, but also inefficient, and it is impossible to track the actual situation of the shelves in a timely manner, dynamically adjust the replenishment strategy, and maximize the effectiveness of shelf display.

[0003] In the existing technical solutions, only the empty spaces on the shelf are counted, and it is difficult to obtain the SKU placement quantity in the empty spaces. Fast-moving consumer goods companies are unable to understand the accurate shelf placement status in a timely manner. Summary of the invention

[0004] The embodiments of the present invention provide a shelf management method, system, terminal and medium based on target detection, which considers the influence of different degrees of overlap between product frames and empty space frames on the empty space on the shelf, and analyzes the relationship between the first product frames with different degrees of overlap and the empty space release on the shelf, so as to improve the accuracy of the assessment of the empty space release on the shelf.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a shelf management method based on target detection, comprising:

[0006] Performing target detection on the shelf image acquired in real time, obtaining a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first empty space frames; wherein the first commodity frame is the minimum circumscribed rectangle of the SKU on the shelf;

[0007] The first commodity box whose intersection-and-union ratio with any first empty space box is less than or equal to a first threshold is used as a second commodity box, and all the first commodity boxes and all the first empty space boxes are layered according to all the first position information and all the second position information, and a width reference amount of each layer is analyzed; wherein the width reference amount of each layer is the median of the widths of all the second commodity boxes of each layer on the shelf;

[0008] The total number of SKUs placed in the empty shelf spaces is calculated based on the width reference of the layer where each of the first empty space frames is located. When the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

[0009] In the implementation of the embodiment of the present invention, target detection is performed on the shelf image acquired in real time to preliminarily identify the SKUs and vacant spaces on the shelf, and multiple first product frames and multiple first vacant space frames in the shelf image, as well as first position information of each first product frame and second position information of each first vacant space frame are obtained. Then, since the farther the object is from the camera lens, the smaller the image of the object is in the photo, and some SKUs in the shelf may be placed in the inner position, in the shelf image obtained by photographing the shelf, the first product frames corresponding to these part of SKUs have less overlapping parts with the first vacant space frames. Therefore, the first product frame whose intersection-over-union ratio with any first vacant space frame is less than or equal to the first threshold is used as the second product frame, and all the first product frames and all the first vacant space frames are layered according to all the first position information and all the second position information, and The median of the width of all the second commodity frames on each layer of the shelf is analyzed as the width reference of each layer, and then the total SKU placement of the shelf vacancy is calculated according to the width reference of the layer where each first vacancy frame is located. Through the above screening means, the overlap degree of the first vacancy frame and the first commodity frame can be accurately analyzed based on the interaction ratio of the first vacancy frame and the first commodity frame, and then the first commodity frame with more overlap with the first vacancy frame is filtered out, and only the first commodity frame with less overlap with the first vacancy frame is selected as the second commodity frame, and the total SKU placement of the shelf vacancy is calculated based on the median of the width of all the second commodity frames on each layer of the shelf. The relationship between the first commodity frame with different overlapping degrees and the empty space of the shelf can be analyzed respectively, and then a more accurate total SKU placement of the shelf vacancy is obtained, so as to dynamically track the vacancy status of the shelf. In addition, when the total SKU placement is greater than the second threshold, the reminder signal and the total SKU placement are fed back to the monitoring platform, which can timely remind the management personnel of the monitoring platform or the staff of the fast-moving consumer goods enterprise using the monitoring platform to quickly understand the current shelf placement status and actual sales status of the shelf and make correct decisions.

[0010] As a preferred solution, the first commodity frame whose intersection-over-combination ratio with any first empty space frame is less than or equal to a first threshold is used as a second commodity frame, and all the first commodity frames and all the first empty space frames are layered according to all the first position information and all the second position information, and a width reference amount of each layer is analyzed, specifically:

[0011] Obtaining an intersection-and-union ratio of the first product frame and the first empty space frame, then selecting the first product frame whose intersection-and-union ratio is greater than a first threshold as the second product frame, selecting the first product frame whose intersection-and-union ratio is less than or equal to the first threshold as the third product frame, and using the first empty space frame whose intersection-and-union ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame;

[0012] Acquire the maximum value of the vertical coordinate of each of the third commodity frames from all the first position information, and acquire the maximum value of the vertical coordinate of each of the first empty space frames from all the second position information, and then sort all of the first empty space frames and all of the third commodity frames according to the maximum value of the vertical coordinate of the first empty space frames and the maximum value of the vertical coordinate of the third commodity frames from large to small, thereby obtaining a first sorting result;

[0013] According to the first sorting result, each first empty space frame and each third commodity frame are sequentially traversed, and the maximum value of the vertical coordinate of the current traversed frame is compared with the preset vertical coordinate range of each layer on the shelf. If the maximum value of the vertical coordinate of the current traversed frame is within the preset vertical coordinate range of any layer on the shelf, the layer where the current traversed frame is located is determined to be the layer corresponding to the current preset vertical coordinate range, so as to complete the stratification of all the first empty space frames and all the third commodity frames;

[0014] Obtaining the hierarchical results of all the second empty space frames from the hierarchical results of all the first empty space frames, and assigning the hierarchical results of each of the second empty space frames to the second product frames corresponding to each of the second empty space frames, so as to determine the layer where each of the second product frames is located;

[0015] According to all the first position information and all the second position information, the width reference of each layer on the shelf is analyzed.

[0016] A preferred solution of the embodiment of the present invention is implemented, the intersection and union ratio of the first product frame and the first empty space frame is obtained, and then the first product frame whose intersection and union ratio is greater than the first threshold is selected as the second product frame, and the first product frame whose intersection and union ratio is less than or equal to the first threshold is selected as the third product frame, and the first empty space frame whose intersection and union ratio with each second product frame is greater than the first threshold is used as the second empty space frame corresponding to each second product frame, and then all the first empty space frames and all the third product frames are sorted in descending order according to the maximum value of the vertical coordinate of the first empty space frame and the maximum value of the vertical coordinate of the third product frame to obtain a first sorting result, and each first empty space frame and each third product frame are traversed in turn according to the first sorting result. , compare the maximum value of the vertical coordinate of the current traversal frame with the preset vertical coordinate range of each layer on the shelf. If the maximum value of the vertical coordinate of the current traversal frame is within the preset vertical coordinate range of any layer on the shelf, determine that the layer where the current traversal frame is located is the layer corresponding to the current preset vertical coordinate range, so that the empty space frames and the third commodity frames that meet the preset vertical coordinate range of each layer can be accurately allocated to the corresponding layer of the shelf, and obtain the stratification results of all second empty space frames from the stratification results of all first empty space frames, and assign the stratification results of each second empty space frame to the second commodity frame corresponding to each second empty space frame, so as to complete the stratification of all first empty space frames, all second commodity frames and all third commodity frames in the shelf image.

[0017] As a preferred solution, the width reference of each layer on the shelf is analyzed according to all the first position information and all the second position information, specifically:

[0018] Acquire the maximum and minimum values ​​of the horizontal coordinates of each of the second commodity frames, and the maximum and minimum values ​​of the horizontal coordinates of each of the third commodity frames from all the first position information, and then subtract the maximum value of the horizontal coordinates of each of the second commodity frames from the minimum value of the horizontal coordinates of each of the second commodity frames to obtain the width of each of the second commodity frames, and subtract the maximum value of the horizontal coordinates of each of the third commodity frames from the minimum value of the horizontal coordinates of each of the third commodity frames to obtain the width of each of the third commodity frames;

[0019] Traverse each layer on the shelf, sort the widths of all the second commodity boxes and the widths of all the third commodity boxes in the current traversal layer in ascending or descending order, and select the median in the sorting result as a reference for the width of the current traversal layer.

[0020] A preferred scheme for implementing the embodiment of the present invention is to analyze and calculate the width of each second product box and each third product box according to the position information of each second product box and each third product box, and then traverse each layer on the shelf, and sort the widths of all second product boxes in the current traversal layer and the widths of all third product boxes in the current traversal layer in ascending or descending order, and select the median in the sorting result as a width reference for the current traversal layer. The median in the sorting result can describe the concentration trend of the widths of the product boxes in the current traversal layer. By using the median as a width reference, the influence of extreme data can be avoided, thereby further improving the accuracy of the total SKU placement quantity.

[0021] As a preferred solution, the total quantity of SKUs placed in the empty shelves is calculated according to the width reference of the layer where each of the first empty shelf frames is located, and when the total quantity of SKUs placed is greater than a second threshold, a reminder signal and the total quantity of SKUs placed are fed back to the monitoring platform, specifically:

[0022] Acquire the maximum value of the vertical coordinate of each second commodity frame from all the first position information, and acquire the maximum value of the vertical coordinate of each second empty frame, as well as the maximum value and the minimum value of the horizontal coordinate of each first empty frame from all the second position information, and then respectively subtract the maximum value of the vertical coordinate of each second empty frame from the maximum value of the vertical coordinate of the second commodity frame corresponding to each second empty frame to obtain the bottom deviation value corresponding to each second empty frame;

[0023] Based on the bottom deviation value corresponding to each of the second empty space frames, determine the space type of each of the first empty space frames, and then calculate the SKU placement quantity of each of the first empty space frames according to the space type of each of the first empty space frames, the maximum value and the minimum value of the horizontal coordinate of each of the first empty space frames, and the width reference amount of the layer where each of the first empty space frames is located;

[0024] Adding up the SKU placement quantities of all the first empty slots to obtain the total SKU placement quantity of the empty slots on the shelf;

[0025] When the total placement quantity of the SKU is greater than the second threshold, a reminder signal and the total placement quantity of the SKU are fed back to the monitoring platform.

[0026] According to a preferred solution of the embodiment of the present invention, the maximum value of the ordinate of each second vacant space frame is subtracted from the maximum value of the ordinate of the second commodity frame corresponding to each second vacant space frame to obtain the bottom deviation value corresponding to each second vacant space frame, and the vacant space type of each first vacant space frame is determined based on the bottom deviation value corresponding to each second vacant space frame, and then the SKU placement quantity of each first vacant space frame is calculated according to the vacant space type of each first vacant space frame, the maximum value and the minimum value of the abscissa of each first vacant space frame, and the width reference of the layer where each first vacant space frame is located, so that the bottom deviation values ​​of the second vacant space frame and the second commodity frame with less overlap can be analyzed, and the vacant space type of each first vacant space frame can be determined based on the bottom deviation value corresponding to each second vacant space frame, so as to facilitate the subsequent targeted analysis of the SKU placement quantity of the first vacant space frames belonging to different vacant space types, and further improve the accuracy of the total SKU placement quantity of the vacant space on the shelf.

[0027] As a preferred solution, the target detection is performed on the shelf image acquired in real time to obtain a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as the first position information of each of the first commodity frames and the second position information of each of the first empty space frames, specifically:

[0028] Take real-time photos of the shelves to obtain shelf images;

[0029] Using an object detection model, performing object detection on the shelf image, obtaining a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first empty space frames;

[0030] Among them, the target detection model is obtained by training the target detector using a training data set, and the training data set includes a number of training images, multiple product frames and multiple empty space frames in each of the training images, and position information of each of the product frames and each of the empty space frames, and the training images are obtained by photographing the shelves.

[0031] A preferred scheme for implementing the embodiments of the present invention uses a training data set including several training images, multiple product frames and multiple empty space frames in each training image, and position information of each product frame and each empty space frame to train a target detector to obtain a target detection model, and uses the trained target detection model to perform target detection on the shelf image, which can improve the detection accuracy and efficiency of the first product frame, the first empty space frame, the first position information of the first product frame, and the second position information of the first empty space frame in the shelf image.

[0032] In order to solve the same technical problem, an embodiment of the present invention further provides a shelf management system based on target detection, comprising:

[0033] The target detection module is used to perform target detection on the shelf image acquired in real time, and obtain a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first empty space frames; wherein the first commodity frame is the minimum circumscribed rectangle of the SKU on the shelf;

[0034] a layered analysis module, configured to take the first commodity boxes whose intersection-over-union ratio with any of the first empty spaces is less than or equal to a first threshold as second commodity boxes, and to layer all the first commodity boxes and all the first empty spaces according to all the first position information and all the second position information, and analyze a width reference of each layer; wherein the width reference of each layer is the median of the widths of all the second commodity boxes of each layer on the shelf;

[0035] A calculation feedback module is used to calculate the total number of SKUs placed in the shelf spaces according to the width reference of the layer where each of the first space frames is located, and when the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

[0036] As a preferred solution, the hierarchical analysis module specifically includes:

[0037] a screening unit, configured to obtain an intersection-and-union ratio between the first product frame and the first empty space frame, and then select the first product frame whose intersection-and-union ratio is greater than a first threshold as the second product frame, select the first product frame whose intersection-and-union ratio is less than or equal to the first threshold as the third product frame, and use the first empty space frame whose intersection-and-union ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame;

[0038] a stratification unit, configured to obtain the maximum value of the vertical coordinate of each of the third commodity frames from all of the first position information, and obtain the maximum value of the vertical coordinate of each of the first empty space frames from all of the second position information, and then sort all of the first empty space frames and all of the third commodity frames in descending order according to the maximum value of the vertical coordinate of the first empty space frames and the maximum value of the vertical coordinate of the third commodity frames to obtain a first sorting result; traverse each of the first empty space frames and each of the third commodity frames in turn according to the first sorting result, and compare the maximum value of the vertical coordinate of the current traversal frame with the preset vertical coordinate range of each layer on the shelf; if the maximum value of the vertical coordinate of the current traversal frame is within the preset vertical coordinate range of any layer on the shelf, determine that the layer where the current traversal frame is located is the layer corresponding to the current preset vertical coordinate range, so as to complete the stratification of all of the first empty space frames and all of the third commodity frames; obtain the stratification results of all of the second empty space frames from the stratification results of all of the first empty space frames, and assign the stratification results of each of the second empty space frames to the second commodity frames corresponding to each of the second empty space frames, so as to determine the layer where each of the second commodity frames is located;

[0039] The analyzing unit is used to analyze the width reference of each layer on the shelf according to all the first position information and all the second position information.

[0040] As a preferred solution, the calculation feedback module specifically includes:

[0041] a position analysis unit, configured to obtain a maximum value of a vertical coordinate of each second commodity frame from all the first position information, and obtain a maximum value of a vertical coordinate of each second empty frame, and a maximum value and a minimum value of a horizontal coordinate of each first empty frame from all the second position information, and then respectively subtract the maximum value of the vertical coordinate of each second empty frame from the maximum value of the vertical coordinate of the second commodity frame corresponding to each second empty frame, to obtain a bottom deviation value corresponding to each second empty frame;

[0042] an empty position quantity calculation unit, for determining the empty position type of each first empty position frame based on the bottom deviation value corresponding to each second empty position frame, and then respectively calculating the SKU placement quantity of each first empty position frame according to the empty position type of each first empty position frame, the maximum value of the horizontal coordinate and the minimum value of the horizontal coordinate of each first empty position frame, and the width reference of the layer where each first empty position frame is located; and adding up the SKU placement quantities of all the first empty position frames to obtain the total SKU placement quantity of the empty positions on the shelf;

[0043] The feedback warning unit is used to feedback a reminder signal and the total amount of SKUs placed to the monitoring platform when the total amount of SKUs placed is greater than a second threshold.

[0044] In order to solve the same technical problem, the present invention also provides a terminal, including a processor, a memory and a computer program stored in the memory; wherein the computer program can be executed by the processor to implement the shelf management method based on target detection.

[0045] In order to solve the same technical problem, the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the shelf management method based on target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 : A flowchart of a shelf management method based on target detection provided in Embodiment 1 of the present invention;

[0047] Figure 2 : A structural diagram of a shelf management system based on target detection provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Embodiment one:

[0050] Please refer to Figure 1 , a shelf management method based on target detection is provided in an embodiment of the present invention, the method comprises steps S1 to S3, each step is specifically as follows:

[0051] Step S1, performing target detection on the shelf image acquired in real time, obtaining multiple first commodity boxes (sku_box_res) and multiple first empty boxes (empty_box_res) in the shelf image, as well as first position information of each first commodity box and second position information of each first empty box.

[0052] The first commodity frame is the minimum circumscribed rectangle of the SKU on the shelf, and the SKU refers to the shelf commodity with a specific business barcode.

[0053] In this embodiment, the position information of the box is (x_min, y_min, x_max, y_max, label); wherein x_min represents the minimum value of the horizontal coordinate of the box, y_min represents the minimum value of the vertical coordinate, x_max represents the maximum value of the horizontal coordinate, y_max represents the maximum value of the vertical coordinate, and label represents the label of the box, and label is SKU or empty space.

[0054] As a preferred solution, step S1 includes step S11 to step S12, and each step is specifically as follows:

[0055] Step S11, photographing the shelf in real time to obtain a shelf image.

[0056] Step S12, using the target detection model, performs target detection on the shelf image to obtain multiple first product boxes (sku_box_res) and multiple first empty boxes (empty_box_res) in the shelf image, as well as first position information of each first product box and second position information of each first empty box.

[0057] Among them, the target detection model is obtained by training the target detector using a training data set, the training data set includes several training images, multiple product frames and multiple empty space frames in each training image, and position information of each product frame and each empty space frame, and the training images are obtained by shooting the shelves.

[0058] Step S2, taking a first product frame whose intersection-over-union ratio with any first empty space frame is less than or equal to a first threshold as a second product frame, and layering all first product frames and all first empty space frames according to all first position information and all second position information, and analyzing a width reference of each layer.

[0059] The width reference of each layer is the median of the widths of all second commodity boxes on each layer of the shelf.

[0060] It should be noted that in actual application scenarios, the SKUs on the shelves may be placed in the inner position, while the positions on the shelves are empty. At this time, when the object detection is performed on the shelf image obtained, the intersection and union ratio of some product frames and the empty positions obtained is low. These product frames are not normal shelf frames, and all empty positions are normal shelf frames. Therefore, based on the comparison of the intersection and union ratio with the first threshold, it is convenient to distinguish and handle different overlapping degrees in the subsequent process.

[0061] As a preferred solution, step S2 includes step S21 to step S25, and each step is specifically as follows:

[0062] Step S21, obtaining the intersection-and-union ratio of the first product frame and the first empty space frame, then selecting the first product frame whose intersection-and-union ratio is greater than a first threshold as the second product frame, selecting the first product frame whose intersection-and-union ratio is less than or equal to the first threshold as the third product frame, and using the first empty space frame whose intersection-and-union ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame.

[0063] In this embodiment, all third product boxes and all first empty space boxes are boxes in normal arrangement. The set composed of all boxes in normal arrangement is recorded as sku_empty_box_res, and the set composed of all second product boxes and all second empty space boxes corresponding to the second product boxes is recorded as sku_empty_box_filter_res, and the information format of the set is (second empty space box, second SKU box, IOU), where IOU represents the intersection and union ratio of the second empty space box and its corresponding second product box.

[0064] Step S22, obtaining the maximum value of the vertical coordinate of each third product frame from all the first position information, and obtaining the maximum value of the vertical coordinate of each first empty frame from all the second position information, and then sorting all the first empty frames and all the third product frames at the same time according to the order of the maximum value of the vertical coordinate y_max of the first empty frame and the maximum value of the vertical coordinate y_max of the third product frame from large to small, to obtain a first sorting result.

[0065] Step S23, traverse each first empty space box and each third product box in turn according to the first sorting result, that is, traverse all boxes belonging to the set sku_empty_box_res in turn according to the first sorting result, and compare the maximum vertical coordinate y_max of the current traversal box with the preset vertical coordinate range of each layer on the shelf. If the maximum vertical coordinate y_max of the current traversal box is within the preset vertical coordinate range of any layer on the shelf, determine that the layer where the current traversal box is located is the layer corresponding to the current preset vertical coordinate range, so as to complete the stratification of all first empty space boxes and all third product boxes.

[0066] As an example, the shelf has three layers, the preset vertical coordinate range of the first layer on the shelf is less than the threshold value Q1, the preset vertical coordinate range of the second layer on the shelf is greater than or equal to the threshold value Q1 and less than the threshold value Q2, and the preset vertical coordinate range of the third layer on the shelf is greater than or equal to the threshold value Q2 and less than the threshold value Q3. If the maximum vertical coordinate value y_max of the first empty space box currently traversed is less than the threshold value Q1, then the layer where the third commodity box currently traversed is located is the first layer on the shelf; if the maximum vertical coordinate value y_max of the third commodity box currently traversed is greater than or equal to the threshold value Q2 and less than the threshold value Q3, then the layer where the third commodity box currently traversed is located is the third layer on the shelf.

[0067] It should be noted that after the layering of all the first empty boxes and all the third product boxes is completed, the information format of the boxes belonging to the set sku_empty_box_res should be based on (x_min, y_min, x_max, y_max, label) and add the layer where the box is located, that is, adjusted to (x_min, y_min, x_max, y_max, label, layer).

[0068] Step S24, obtaining the stratification results of all second empty space frames from the stratification results of all first empty space frames, and assigning the stratification results of each second empty space frame to the second commodity frame corresponding to each second empty space frame to determine the layer where each second commodity frame is located.

[0069] Step S25, analyzing the width reference of each layer on the shelf according to all the first position information and all the second position information.

[0070] As a preferred solution, step S25 includes step S251 to step S252, and each step is specifically as follows:

[0071] Step S251, obtain the maximum horizontal coordinate x_max and the minimum horizontal coordinate x_min of each second product frame, as well as the maximum horizontal coordinate x_max and the minimum horizontal coordinate x_min of each third product frame from all the first position information, then subtract the maximum horizontal coordinate x_max of each second product frame from the minimum horizontal coordinate x_min of each second product frame to obtain the width of each second product frame, and subtract the maximum horizontal coordinate x_max of each third product frame from the minimum horizontal coordinate x_min of each third product frame to obtain the width of each third product frame.

[0072] In this embodiment, after executing step S251, a vacancy type threshold acquisition process is also included, and the process includes steps S01 to S03, and each step is specifically as follows:

[0073] Step S01, traverse each layer on the shelf, sort the widths of all second commodity boxes of the current traversal layer in ascending or descending order, then select the median in the sorting result as the first width statistic width_sku_empty_box of the current traversal layer, and count the number number1 of the widths of all second commodity boxes of the current traversal layer, sort the widths of all third commodity boxes of the current traversal layer in ascending or descending order, then select the median in the sorting result as the second width statistic width_sku_box_filter of the current traversal layer, and count the number number2 of the widths of all third commodity boxes of the current traversal layer.

[0074] Step S03, please refer to formula (1), according to the first width statistic width_sku_empty_box of each layer obtained in step S01, the number of widths of all second product boxes number1, the second width statistic width_sku_box_filter and the number of widths of all third product boxes number2, calculate the first empty space type threshold empty_1_thresh and the second empty space type threshold empty_2_thresh corresponding to the empty boxes of each layer.

[0075]

[0076] Where a1 and a2 are empirically set values.

[0077] It should be noted that the number and definition of the empty space types can be adaptively adjusted according to the actual application scenario. In this embodiment, the empty space types of the empty space frame are empty_1, empty_2 and empty_3. Among them, empty_1 means that the empty space frame can hold one SKU in the depth direction, empty_2 means that the empty space frame can hold two SKUs in the depth direction, and empty_3 means that the empty space frame can hold two or more SKUs in the depth direction.

[0078] Step S252, traverse each layer on the shelf, sort the widths of all second commodity boxes and all third commodity boxes of the current traversal layer in ascending or descending order, and select the median in the sorting result as the width reference width_ref of the current traversal layer.

[0079] Step S3, calculating the total quantity of SKUs placed in the shelf spaces according to the width reference of the layer where each first space frame is located, and when the total quantity of SKUs placed is greater than the second threshold, feeding back a reminder signal and the total quantity of SKUs placed to the monitoring platform.

[0080] As a preferred solution, step S3 includes step S31 to step S25, and each step is specifically as follows:

[0081] Step S31, obtaining the maximum vertical coordinate value y_max of each second product frame from all the first position information, and obtaining the maximum vertical coordinate value y_max of each second empty frame, as well as the maximum horizontal coordinate value x_max and the minimum horizontal coordinate value x_min of each first empty frame from all the second position information, and then subtracting the maximum vertical coordinate value y_max of each second empty frame from the maximum vertical coordinate value y_max of the second product frame corresponding to each second empty frame, to obtain the bottom deviation value delta_depth corresponding to each second empty frame.

[0082] Step S32, based on the bottom deviation value delta_depth corresponding to each second vacant space frame, determine the vacancy type of each first vacant space frame, and then calculate the SKU placement sku_occupancy of each first vacant space frame according to the vacancy type of each first vacant space frame, the maximum horizontal coordinate x_max and the minimum horizontal coordinate x_min of each first vacant space frame, and the width reference width_ref of the layer where each first vacant space frame is located.

[0083] In this embodiment, the execution process of step S32 may specifically include step S321 to step S322, and each step is specifically as follows:

[0084] Step S321, first, set the slot type of all first slot boxes to empty_3. The second slot box is a special first slot box, which corresponds to the second commodity box. For each second empty space frame, if the bottom deviation value width_ref corresponding to the current second empty space frame is less than the first empty space type threshold corresponding to the layer where the current second empty space frame is located, the empty space type of the current second empty space frame is determined to be empty_1, and thus the empty space type of the current second empty space frame, which is the first empty space frame, is modified from empty_3 to empty_1; if the bottom deviation value width_ref corresponding to the current second empty space frame is greater than or equal to the first empty space type threshold corresponding to the layer where the current second empty space frame is located and less than the second empty space type threshold corresponding to the layer where the current second empty space frame is located, the empty space type of the current second empty space frame, which is the first empty space frame, is determined to be empty_2, and thus the empty space type of the current second empty space frame, which is the first empty space frame, is modified from empty_3 to empty_2; if the bottom deviation value width_ref corresponding to the current second empty space frame is greater than the second empty space type threshold corresponding to the layer where the current second empty space frame is located, the empty space type of the current second empty space frame is determined to be empty_3, and thus the empty space type of the current second empty space frame, which is the first empty space frame, does not need to be modified.

[0085] Step S322, for the first empty place box of type empty_1, set the type code to depth_number=1, for the first empty place box of type empty_2, set the type code to depth_number=2, for the first empty place box of type empty_3, set the type code to depth_number=3, and then refer to formula (2) to calculate the SKU placement amount sku_occupancy of the current first empty place box according to the width reference amount corresponding to the layer where the current first empty place box is located, the minimum horizontal coordinate value, the maximum horizontal coordinate value and the type code of the current first empty place box.

[0086]

[0087] Step S33, adding up the SKU placement quantities sku_occupancy of all first vacant slots to obtain the total SKU placement quantity of the shelf slots.

[0088] Step S34, when the total SKU placement quantity is greater than the second threshold, a reminder signal and the total SKU placement quantity are fed back to the monitoring platform to remind the management personnel of the monitoring platform or the personnel of the enterprise or other terminals using the monitoring platform to track the total SKU placement quantity of the shelf in real time.

[0089] Please refer to Figure 2 , is a schematic diagram of a shelf management system based on target detection provided by an embodiment of the present invention. The system includes a target detection module M1, a hierarchical analysis module M2 and a calculation feedback module M3. The specific details of each module are as follows:

[0090] The target detection module M1 is used to perform target detection on the shelf image acquired in real time, and obtain a plurality of first commodity frames and a plurality of first space frames in the shelf image, as well as first position information of each first commodity frame and second position information of each first space frame; wherein the first commodity frame is the minimum circumscribed rectangle of the SKU on the shelf;

[0091] The layered analysis module M2 is used to take the first commodity frame whose intersection-over-union ratio with any first empty space frame is less than or equal to the first threshold as the second commodity frame, and to layer all the first commodity frames and all the first empty space frames according to all the first position information and all the second position information, and analyze the width reference of each layer; wherein the width reference of each layer is the median of the width of all the second commodity frames of each layer on the shelf;

[0092] The calculation feedback module M3 is used to calculate the total SKU placement quantity of the shelf vacant space according to the width reference quantity of the layer where each first vacant space frame is located, and when the total SKU placement quantity is greater than the second threshold, a reminder signal and the total SKU placement quantity are fed back to the monitoring platform.

[0093] As a preferred solution, the stratified analysis module M2 specifically includes a screening unit 21, a stratification unit 22 and an analysis unit 23, and the details of each unit are as follows:

[0094] The screening unit 21 is used to obtain the intersection and union ratio of the first product frame and the first empty space frame, and then select the first product frame whose intersection and union ratio is greater than the first threshold as the second product frame, select the first product frame whose intersection and union ratio is less than or equal to the first threshold as the third product frame, and use the first empty space frame whose intersection and union ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame.

[0095] The stratification unit 22 is used to obtain the maximum value of the vertical coordinate of each third commodity frame from all the first position information, and obtain the maximum value of the vertical coordinate of each first empty space frame from all the second position information, and then sort all the first empty space frames and all the third commodity frames in descending order according to the maximum value of the vertical coordinate of the first empty space frame and the maximum value of the vertical coordinate of the third commodity frame to obtain a first sorting result; traverse each first empty space frame and each third commodity frame in turn according to the first sorting result, compare the maximum value of the vertical coordinate of the current traversal frame with the preset vertical coordinate range of each layer on the shelf, if the maximum value of the vertical coordinate of the current traversal frame is within the preset vertical coordinate range of any layer on the shelf, determine that the layer where the current traversal frame is located is the layer corresponding to the current preset vertical coordinate range, so as to complete the stratification of all the first empty space frames and all the third commodity frames; obtain the stratification results of all the second empty space frames from the stratification results of all the first empty space frames, and assign the stratification results of each second empty space frame to the second commodity frame corresponding to each second empty space frame, so as to determine the layer where each second commodity frame is located.

[0096] The analyzing unit 23 is used to analyze the width reference of each layer on the shelf according to all the first position information and all the second position information.

[0097] As a preferred solution, the calculation feedback module M3 specifically includes a position analysis unit 31, an empty position release calculation unit 32 and a feedback warning unit 33, and the specific units are as follows:

[0098] The position analysis unit 31 is used to obtain the maximum value of the vertical coordinate of each second product frame from all the first position information, and obtain the maximum value of the vertical coordinate of each second empty frame from all the second position information, as well as the maximum value and minimum value of the horizontal coordinate of each first empty frame, and then subtract the maximum value of the vertical coordinate of each second empty frame from the maximum value of the vertical coordinate of the second product frame corresponding to each second empty frame, to obtain the bottom deviation value corresponding to each second empty frame.

[0099] The empty space quantity calculation unit 32 is used to determine the empty space type of each first empty space frame based on the bottom deviation value corresponding to each second empty space frame, and then calculate the SKU placement quantity of each first empty space frame according to the empty space type of each first empty space frame, the maximum value and the minimum value of the horizontal coordinate of each first empty space frame, and the width reference of the layer where each first empty space frame is located; the SKU placement quantities of all the first empty space frames are added together to obtain the total SKU placement quantity of the empty space on the shelf.

[0100] The feedback warning unit 33 is used to feed back a reminder signal and the total amount of SKUs placed to the monitoring platform when the total amount of SKUs placed is greater than the second threshold.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0102] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a shelf management method based on target detection in embodiment one.

[0103] In addition, an embodiment of the present invention further provides a terminal, including a processor, a memory, and a computer program stored in the memory; wherein the computer program can be executed by the processor to implement a shelf management method based on target detection in embodiment one.

[0104] Preferably, the computer program can be divided into one or more modules / units (such as computer program, computer program), one or more modules / units are stored in the memory and executed by the processor to complete the present invention. One or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal.

[0105] The processor can be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor. The processor is the control center of the terminal, and various parts of the terminal are connected using various interfaces and lines.

[0106] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory can also be other volatile solid-state storage devices.

[0107] It should be noted that the above-mentioned terminal may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above-mentioned terminal is merely an example and does not constitute a limitation on the terminal. It may include more or fewer components, or a combination of certain components, or different components.

[0108] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0109] The present invention provides a shelf management method, system, terminal and medium based on target detection, which performs target detection on shelf images acquired in real time to preliminarily identify SKUs and vacant spaces on the shelf, and obtain multiple first product frames and multiple first vacant space frames in the shelf image, as well as first position information of each first product frame and second position information of each first vacant space frame. Then, since the farther the object is from the camera lens, the smaller the image of the object is in the photo, and some SKUs in the shelf may be placed in the inner position, in the shelf image obtained by photographing the shelf, the first product frames corresponding to these part of SKUs have less overlapping parts with the first vacant space frames, so the first product frames whose intersection-over-union ratio with any first vacant space frame is less than or equal to the first threshold value are used as the second product frames, and all the first product frames and all the second position information are compared according to all the first position information and all the second position information. The first empty space frames are layered, and the median of the widths of all the second product frames on each layer of the shelf is analyzed as a reference for the width of each layer; then the total SKU placement quantity of the empty shelf spaces is calculated based on the width reference of the layer where each first empty space frame is located; through the above-mentioned screening means, the degree of overlap between the first empty space frames and the first product frames can be accurately analyzed based on the interaction ratio of the first empty space frames and the first product frames, and then the first product frames with more overlap with the first empty space frames are filtered out, and only the first product frames with less overlap with the first empty space frames are selected as the second product frames, and the total SKU placement quantity of the empty shelf spaces is calculated based on the median of the widths of all the second product frames on each layer of the shelf; the relationship between the first product frames with different degrees of overlap and the empty shelf spaces can be analyzed respectively, and then a more accurate total SKU placement quantity of the empty shelf spaces can be obtained, so as to dynamically track the empty space status of the shelf. In addition, when the total SKU placement quantity is greater than the second threshold, a reminder signal and the total SKU placement quantity are fed back to the monitoring platform, which can promptly remind the managers of the monitoring platform or the staff of fast-moving consumer goods companies using the monitoring platform to quickly understand the actual sales status of the shelves and make correct decisions.

[0110] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A shelf management method based on target detection, characterized in that: include: Performing target detection on the shelf image acquired in real time, obtaining a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first empty space frames; wherein the first commodity frame is the minimum circumscribed rectangle of the SKU on the shelf; The first commodity box whose intersection-and-union ratio with any first empty space box is less than or equal to a first threshold is used as a second commodity box, and all the first commodity boxes and all the first empty space boxes are layered according to all the first position information and all the second position information, and a width reference amount of each layer is analyzed; wherein the width reference amount of each layer is the median of the widths of all the second commodity boxes of each layer on the shelf; The total number of SKUs placed in the empty shelf spaces is calculated based on the width reference of the layer where each of the first empty space frames is located. When the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

2. A shelf management method based on target detection as claimed in claim 1, characterized in that: The first commodity frame whose intersection-over-combination ratio with any first empty space frame is less than or equal to a first threshold is used as a second commodity frame, and all the first commodity frames and all the first empty space frames are layered according to all the first position information and all the second position information, and a width reference amount of each layer is analyzed, specifically: Obtaining an intersection-and-union ratio of the first product frame and the first empty space frame, then selecting the first product frame whose intersection-and-union ratio is greater than a first threshold as the second product frame, selecting the first product frame whose intersection-and-union ratio is less than or equal to the first threshold as the third product frame, and using the first empty space frame whose intersection-and-union ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame; Acquire the maximum value of the vertical coordinate of each of the third commodity frames from all the first position information, and acquire the maximum value of the vertical coordinate of each of the first empty space frames from all the second position information, and then sort all of the first empty space frames and all of the third commodity frames according to the maximum value of the vertical coordinate of the first empty space frames and the maximum value of the vertical coordinate of the third commodity frames from large to small, thereby obtaining a first sorting result; According to the first sorting result, each first empty space frame and each third commodity frame are sequentially traversed, and the maximum value of the vertical coordinate of the current traversed frame is compared with the preset vertical coordinate range of each layer on the shelf. If the maximum value of the vertical coordinate of the current traversed frame is within the preset vertical coordinate range of any layer on the shelf, the layer where the current traversed frame is located is determined to be the layer corresponding to the current preset vertical coordinate range, so as to complete the stratification of all the first empty space frames and all the third commodity frames; Obtaining the hierarchical results of all the second empty space frames from the hierarchical results of all the first empty space frames, and assigning the hierarchical results of each of the second empty space frames to the second product frames corresponding to each of the second empty space frames, so as to determine the layer where each of the second product frames is located; According to all the first position information and all the second position information, the width reference of each layer on the shelf is analyzed.

3. A shelf management method based on target detection as claimed in claim 2, characterized in that: The analyzing the width reference of each layer on the shelf according to all the first position information and all the second position information is specifically as follows: Acquire the maximum and minimum values ​​of the horizontal coordinates of each of the second commodity frames, and the maximum and minimum values ​​of the horizontal coordinates of each of the third commodity frames from all the first position information, and then subtract the maximum value of the horizontal coordinates of each of the second commodity frames from the minimum value of the horizontal coordinates of each of the second commodity frames to obtain the width of each of the second commodity frames, and subtract the maximum value of the horizontal coordinates of each of the third commodity frames from the minimum value of the horizontal coordinates of each of the third commodity frames to obtain the width of each of the third commodity frames; Traverse each layer on the shelf, sort the widths of all the second commodity boxes and the widths of all the third commodity boxes in the current traversal layer in ascending or descending order, and select the median in the sorting result as a reference for the width of the current traversal layer.

4. A shelf management method based on target detection as claimed in claim 2, characterized in that: The total number of SKUs placed in the empty shelves is calculated according to the width reference of the layer where each of the first empty shelf frames is located, and when the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform, specifically: Acquire the maximum value of the vertical coordinate of each second commodity frame from all the first position information, and acquire the maximum value of the vertical coordinate of each second empty frame, as well as the maximum value and the minimum value of the horizontal coordinate of each first empty frame from all the second position information, and then respectively subtract the maximum value of the vertical coordinate of each second empty frame from the maximum value of the vertical coordinate of the second commodity frame corresponding to each second empty frame to obtain the bottom deviation value corresponding to each second empty frame; Based on the bottom deviation value corresponding to each of the second empty space frames, determine the space type of each of the first empty space frames, and then calculate the SKU placement quantity of each of the first empty space frames according to the space type of each of the first empty space frames, the maximum value and the minimum value of the horizontal coordinate of each of the first empty space frames, and the width reference amount of the layer where each of the first empty space frames is located; Adding up the SKU placement quantities of all the first empty slots to obtain the total SKU placement quantity of the empty slots on the shelf; When the total placement quantity of the SKU is greater than the second threshold, a reminder signal and the total placement quantity of the SKU are fed back to the monitoring platform.

5. The shelf management method based on target detection according to claim 1, characterized in that: The target detection is performed on the shelf image acquired in real time to obtain a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first empty space frames, specifically: Take real-time photos of the shelves to obtain shelf images; Using an object detection model, performing object detection on the shelf image, obtaining a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first empty space frames; Among them, the target detection model is obtained by training the target detector using a training data set, and the training data set includes a number of training images, multiple product frames and multiple empty space frames in each of the training images, and position information of each of the product frames and each of the empty space frames, and the training images are obtained by photographing the shelves.

6. A shelf management system based on target detection, characterized in that: include: The target detection module is used to perform target detection on the shelf image acquired in real time, and obtain a plurality of first commodity frames and a plurality of first empty space frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first empty space frames; wherein the first commodity frame is the minimum circumscribed rectangle of the SKU on the shelf; a layered analysis module, configured to take the first commodity boxes whose intersection-over-union ratio with any of the first empty spaces is less than or equal to a first threshold as second commodity boxes, and to layer all the first commodity boxes and all the first empty spaces according to all the first position information and all the second position information, and analyze a width reference of each layer; wherein the width reference of each layer is the median of the widths of all the second commodity boxes of each layer on the shelf; A calculation feedback module is used to calculate the total number of SKUs placed in the shelf spaces according to the width reference of the layer where each of the first space frames is located, and when the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

7. A shelf management system based on target detection as claimed in claim 6, characterized in that: The hierarchical analysis module specifically includes: a screening unit, configured to obtain an intersection-and-union ratio between the first product frame and the first empty space frame, and then select the first product frame whose intersection-and-union ratio is greater than a first threshold as the second product frame, select the first product frame whose intersection-and-union ratio is less than or equal to the first threshold as the third product frame, and use the first empty space frame whose intersection-and-union ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame; a stratification unit, configured to obtain the maximum value of the vertical coordinate of each of the third commodity frames from all of the first position information, and obtain the maximum value of the vertical coordinate of each of the first empty space frames from all of the second position information, and then sort all of the first empty space frames and all of the third commodity frames in descending order according to the maximum value of the vertical coordinate of the first empty space frames and the maximum value of the vertical coordinate of the third commodity frames to obtain a first sorting result; traverse each of the first empty space frames and each of the third commodity frames in turn according to the first sorting result, and compare the maximum value of the vertical coordinate of the current traversal frame with the preset vertical coordinate range of each layer on the shelf; if the maximum value of the vertical coordinate of the current traversal frame is within the preset vertical coordinate range of any layer on the shelf, determine that the layer where the current traversal frame is located is the layer corresponding to the current preset vertical coordinate range, so as to complete the stratification of all of the first empty space frames and all of the third commodity frames; obtain the stratification results of all of the second empty space frames from the stratification results of all of the first empty space frames, and assign the stratification results of each of the second empty space frames to the second commodity frames corresponding to each of the second empty space frames, so as to determine the layer where each of the second commodity frames is located; The analyzing unit is used to analyze the width reference of each layer on the shelf according to all the first position information and all the second position information.

8. A shelf management system based on target detection as claimed in claim 7, characterized in that: The calculation feedback module specifically includes: a position analysis unit, configured to obtain a maximum value of a vertical coordinate of each second commodity frame from all the first position information, and obtain a maximum value of a vertical coordinate of each second empty frame, and a maximum value and a minimum value of a horizontal coordinate of each first empty frame from all the second position information, and then respectively subtract the maximum value of the vertical coordinate of each second empty frame from the maximum value of the vertical coordinate of the second commodity frame corresponding to each second empty frame, to obtain a bottom deviation value corresponding to each second empty frame; an empty position quantity calculation unit, for determining the empty position type of each first empty position frame based on the bottom deviation value corresponding to each second empty position frame, and then respectively calculating the SKU placement quantity of each first empty position frame according to the empty position type of each first empty position frame, the maximum value of the horizontal coordinate and the minimum value of the horizontal coordinate of each first empty position frame, and the width reference of the layer where each first empty position frame is located; and adding up the SKU placement quantities of all the first empty position frames to obtain the total SKU placement quantity of the empty positions on the shelf; The feedback warning unit is used to feedback a reminder signal and the total amount of SKUs placed to the monitoring platform when the total amount of SKUs placed is greater than a second threshold.

9. A terminal, characterized in that: It comprises a processor, a memory and a computer program stored in the memory; wherein the computer program can be executed by the processor to implement a shelf management method based on target detection as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a shelf management method based on target detection as described in any one of claims 1 to 5.

Images