A shelf management method, system, terminal and medium based on target detection

Abstract

The present invention discloses a shelf management method, system, terminal and medium based on target detection, including: performing target detection on a shelf image to obtain multiple first commodity frames and multiple first empty space frames in the image, as well as the position information of each first commodity frame and each first empty space frame; taking a first commodity frame whose intersection-over-union ratio with any first empty space frame is less than or equal to a first threshold as a second commodity frame, and layering all first commodity frames and all first empty space frames according to all position information, and analyzing the width reference of each layer; calculating the total SKU placement quantity of the shelf vacancy according to the width reference of the layer where each first empty space frame is located, and when the total SKU placement quantity is greater than the second threshold, feeding back a reminder signal and the total SKU placement quantity to a monitoring platform. Using the embodiment of the present invention, the relationship between the first commodity frames with different degrees of overlap and the shelf vacancy is analyzed respectively to achieve efficient evaluation of the placement quantity of the shelf vacancy.

Description

Technical Field

[0001] The present invention relates to the field of shelf management, and in particular to a shelf management method, system, terminal and medium based on target detection. Background Art

[0002] As the primary means of displaying merchandise, the rationality of shelf placement and its fill rate significantly impacts merchandise sales. This is particularly true in the fast-moving consumer goods (FMCG) sector, where certain shelf locations are often left empty for extended periods of time, or with a high number of empty spaces. This not only reduces the fill rate but also leads to inventory backlogs, ultimately impacting sales. In the past, determining the rationality of shelf placement relied primarily on on-site inspections by sales staff or remote camera monitoring. This process was not only time-consuming and inefficient, but also made it impossible to track the actual conditions of the shelves in a timely manner, dynamically adjust replenishment strategies, and maximize the effectiveness of shelf displays.

[0003] In the existing technical solutions, only the empty spaces on the shelf are counted, and it is difficult to obtain the SKU placement quantity in the empty spaces. Fast-moving consumer goods companies cannot understand the accurate shelf placement status in a timely manner. Summary of the Invention

[0004] Embodiments of the present invention provide a shelf management method, system, terminal, and medium based on target detection. These methods consider the impact of different degrees of overlap between product frames and empty space frames on shelf spaces, and analyze the relationship between the first product frames with different degrees of overlap and the empty space release on the shelf, so as to improve the accuracy of the assessment of the empty space release on the shelf.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a shelf management method based on target detection, comprising:

[0006] Performing target detection on a shelf image acquired in real time to obtain a plurality of first product frames and a plurality of first empty space frames within the shelf image, as well as first position information of each first product frame and second position information of each first empty space frame; wherein the first product frame is a minimum bounding rectangle of a SKU on the shelf;

[0007] The first product frame whose intersection-over-union ratio with any first empty frame is less than or equal to a first threshold is used as a second product frame. All the first product frames and all the first empty frames are layered according to all the first position information and all the second position information, and a width reference value of each layer is analyzed. The width reference value of each layer is the median of the widths of all the second product frames on each layer of the shelf.

[0008] The total number of SKUs placed in the shelf spaces is calculated based on the width reference of the layer where each of the first space frames is located. When the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

[0009] In the implementation of the embodiment of the present invention, target detection is performed on the shelf image acquired in real time to preliminarily identify the SKUs and vacant positions on the shelf, and multiple first product frames and multiple first vacant position frames in the shelf image, as well as first position information of each first product frame and second position information of each first vacant position frame are obtained. Then, since the farther the object is from the camera lens, the smaller the image of the object in the photo, and some SKUs in the shelf may be placed in the inner position, in the shelf image obtained by photographing the shelf, the first product frames corresponding to these SKUs have less overlapping parts with the first vacant position frames. Therefore, the first product frame whose intersection-with-union ratio with any first vacant position frame is less than or equal to the first threshold is used as the second product frame. According to all the first position information and all the second position information, all the first product frames and all the first vacant positions are layered, and The median width of all second product frames on each shelf layer is analyzed as a width reference for each layer. The total number of SKUs placed on the shelf vacant positions is then calculated based on the width reference of the layer where each first vacant position frame resides. Through the above-mentioned screening method, the degree of overlap between the first vacant position frame and the first product frame can be accurately analyzed based on the interaction ratio between the first vacant position frame and the first product frame. First product frames with a large overlap with the first vacant position frame are then filtered out, and only first product frames with a small overlap with the first vacant position frame are selected as second product frames. The total number of SKUs placed on the shelf vacant positions is calculated based on the median width of all second product frames on each shelf layer. This allows for the relationship between first product frames with different degrees of overlap and the amount of vacant shelf positions to be analyzed separately, thereby obtaining a more accurate total number of SKUs placed on the shelf vacant positions, thereby dynamically tracking the shelf vacancy status. Furthermore, when the total number of SKUs placed is greater than a second threshold, an alert signal and the total number of SKUs placed are fed back to the monitoring platform, which can promptly alert the monitoring platform's management personnel or the staff of the fast-moving consumer goods company using the monitoring platform to quickly understand the current shelf placement status and actual sales status of the shelf and make correct decisions.

[0010] As a preferred solution, the first product frame whose intersection-over-union ratio with any first empty frame is less than or equal to a first threshold is used as the second product frame. All the first product frames and all the first empty frames are layered according to all the first position information and all the second position information, and the width reference of each layer is analyzed. Specifically, the following steps are performed:

[0011] Obtaining an intersection-and-union (IoU) ratio between the first product frame and the first empty space frame, then selecting the first product frame whose IoU ratio is greater than a first threshold as the second product frame, selecting the first product frame whose IoU ratio is less than or equal to the first threshold as the third product frame, and selecting the first empty space frame whose IoU ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame;

[0012] Obtaining the maximum vertical coordinate value of each third product frame from all first position information, and obtaining the maximum vertical coordinate value of each first empty frame from all second position information, and then sorting all the first empty frames and all the third product frames simultaneously according to the maximum vertical coordinate values ​​of the first empty frames and the maximum vertical coordinate values ​​of the third product frames from largest to smallest, to obtain a first sorting result;

[0013] Traversing each first empty space frame and each third product frame in sequence according to the first sorting result, comparing the maximum vertical coordinate of the currently traversed frame with a preset vertical coordinate range of each layer on the shelf; if the maximum vertical coordinate of the currently traversed frame is within the preset vertical coordinate range of any layer on the shelf, determining that the layer of the currently traversed frame is the layer corresponding to the currently preset vertical coordinate range, thereby completing the layering of all first empty space frames and all third product frames;

[0014] Obtaining the hierarchical results of all the second empty frames from the hierarchical results of all the first empty frames, and assigning the hierarchical results of each second empty frame to the second product frame corresponding to each second empty frame to determine the layer where each second product frame is located;

[0015] The width reference of each layer on the shelf is analyzed based on all the first position information and all the second position information.

[0016] In a preferred embodiment of the present invention, an intersection-and-union ratio (IoU) of a first product frame and a first vacant frame is obtained. Then, a first product frame having an IoU ratio greater than a first threshold is selected as a second product frame. A first product frame having an IoU ratio less than or equal to the first threshold is selected as a third product frame. A first vacant frame having an IoU ratio greater than the first threshold with each second product frame is used as a second vacant frame corresponding to each second product frame. Then, all first vacant frames and all third product frames are sorted simultaneously based on the maximum value of the vertical coordinate of the first vacant frame and the maximum value of the vertical coordinate of the third product frame from large to small to obtain a first sorting result. Then, each first vacant frame and each third product frame are sequentially traversed based on the first sorting result. , compare the maximum vertical coordinate of the current traversal frame with the preset vertical coordinate range of each layer on the shelf. If the maximum vertical coordinate of the current traversal frame is within the preset vertical coordinate range of any layer on the shelf, then determine that the layer where the current traversal frame is located is the layer corresponding to the current preset vertical coordinate range, so that the empty frames and third product frames that meet the preset vertical coordinate range of each layer can be accurately allocated to the corresponding layer of the shelf, and obtain the stratification results of all second empty frames from the stratification results of all first empty frames, and assign the stratification results of each second empty frame to the second product frame corresponding to each second empty frame, so as to complete the stratification of all first empty frames, all second product frames and all third product frames in the shelf image.

[0017] As a preferred solution, the width reference of each layer on the shelf is analyzed based on all the first position information and all the second position information, specifically:

[0018] Obtaining the maximum and minimum horizontal coordinate values ​​of each second product frame, and the maximum and minimum horizontal coordinate values ​​of each third product frame, from all the first position information, then subtracting the maximum horizontal coordinate value of each second product frame from the minimum horizontal coordinate value of each second product frame to obtain the width of each second product frame, and subtracting the maximum horizontal coordinate value of each third product frame from the minimum horizontal coordinate value of each third product frame to obtain the width of each third product frame;

[0019] Traverse each layer on the shelf, sort the widths of all the second product boxes and the widths of all the third product boxes in the current traversal layer in ascending or descending order, and select the median of the sorting results as a reference for the width of the current traversal layer.

[0020] A preferred solution for implementing the embodiment of the present invention is to analyze and calculate the width of each second product box and each third product box based on the position information of each second product box and each third product box, and then traverse each layer on the shelf, and sort the widths of all second product boxes of the current traversed layer and the widths of all third product boxes of the current traversed layer in ascending or descending order, and select the median in the sorting results as the width reference of the current traversed layer. The median in the sorting results can describe the central trend of the widths of the product boxes of the current traversed layer. By using the median as the width reference, the influence of extreme data can be avoided, and the accuracy of the total SKU placement can be further improved.

[0021] As a preferred solution, the total number of SKUs placed in the empty shelf spaces is calculated based on the width reference of the layer where each first space frame is located. When the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform. Specifically,

[0022] Obtaining the maximum vertical coordinate value of each second product frame from all first position information, and obtaining the maximum vertical coordinate value of each second empty frame, as well as the maximum horizontal coordinate value and the minimum horizontal coordinate value of each first empty frame from all second position information. Then, subtracting the maximum vertical coordinate value of each second empty frame from the maximum vertical coordinate value of the second product frame corresponding to each second empty frame to obtain a bottom deviation value corresponding to each second empty frame.

[0023] Based on the bottom deviation value corresponding to each second empty space frame, the space type of each first empty space frame is determined, and then the SKU placement quantity of each first empty space frame is calculated according to the space type of each first empty space frame, the maximum and minimum values ​​of the horizontal coordinates of each first empty space frame, and the width reference of the layer where each first empty space frame is located;

[0024] Adding up the SKU placement quantities of all the first empty slots to obtain the total SKU placement quantity of the shelf slots;

[0025] When the total placement quantity of the SKU is greater than the second threshold, a reminder signal and the total placement quantity of the SKU are fed back to the monitoring platform.

[0026] According to a preferred solution of an embodiment of the present invention, the maximum vertical coordinate of each second vacant frame is subtracted from the maximum vertical coordinate of the second commodity frame corresponding to each second vacant frame to obtain the bottom deviation value corresponding to each second vacant frame, and the vacancy type of each first vacant frame is determined based on the bottom deviation value corresponding to each second vacant frame. Then, according to the vacancy type of each first vacant frame, the maximum horizontal coordinate and the minimum horizontal coordinate of each first vacant frame, and the width reference of the layer where each first vacant frame is located, the SKU placement quantity of each first vacant frame is calculated respectively. The bottom deviation value of the second vacant frame and the second commodity frame with less overlap can be analyzed, and the vacancy type of each first vacant frame can be determined based on the bottom deviation value corresponding to each second vacant frame, so as to facilitate the subsequent targeted analysis of the SKU placement quantity of the first vacant frames belonging to different vacancy types, thereby further improving the accuracy of the total SKU placement quantity of the shelf vacancy.

[0027] As a preferred solution, target detection is performed on the shelf image acquired in real time to obtain a plurality of first commodity frames and a plurality of first vacant frames within the shelf image, as well as first position information of each first commodity frame and second position information of each first vacant frame, specifically as follows:

[0028] Take real-time photos of the shelves to obtain shelf images;

[0029] Performing target detection on the shelf image using a target detection model to obtain a plurality of first product frames and a plurality of first space frames within the shelf image, as well as first position information of each of the first product frames and second position information of each of the first space frames;

[0030] The target detection model is obtained by training the target detector using a training data set, wherein the training data set includes several training images, multiple product frames and multiple empty space frames in each training image, and position information of each product frame and each empty space frame, and the training images are obtained by photographing the shelves.

[0031] A preferred solution for implementing an embodiment of the present invention utilizes a training data set including several training images, multiple product frames and multiple space frames in each training image, and position information of each product frame and each space frame to train a target detector to obtain a target detection model, and uses the trained target detection model to perform target detection on shelf images, which can improve the detection accuracy and efficiency of the first product frame, the first space frame, the first position information of the first product frame, and the second position information of the first space frame in the shelf image.

[0032] In order to solve the same technical problem, an embodiment of the present invention further provides a shelf management system based on target detection, comprising:

[0033] an object detection module, configured to perform object detection on a shelf image acquired in real time, and obtain a plurality of first product frames and a plurality of first empty space frames within the shelf image, as well as first position information of each first product frame and second position information of each first empty space frame; wherein the first product frame is a minimum bounding rectangle of a SKU on the shelf;

[0034] a layered analysis module, configured to treat as second product frames any first product frame whose intersection-over-union ratio with any first empty frame is less than or equal to a first threshold, stratify all first product frames and all first empty frames into layers based on all first position information and all second position information, and analyze a width reference value for each layer; wherein the width reference value for each layer is the median width of all second product frames on each layer of the shelf;

[0035] A calculation feedback module is used to calculate the total number of SKUs placed in the shelf space based on the width reference of the layer where each first space frame is located, and when the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

[0036] As a preferred solution, the hierarchical analysis module specifically includes:

[0037] a screening unit, configured to obtain an intersection-and-union (IoU) ratio between the first product frames and the first empty space frames, then select the first product frames whose IoU ratio is greater than a first threshold as second product frames, select the first product frames whose IoU ratio is less than or equal to the first threshold as third product frames, and select the first empty space frames whose IoU ratio with each second product frame is greater than the first threshold as second empty space frames corresponding to each second product frame;

[0038] a stratification unit, configured to obtain the maximum vertical coordinate of each third product frame from all first position information, and obtain the maximum vertical coordinate of each first empty frame from all second position information, and then simultaneously sort all the first empty frames and all the third product frames in descending order of the maximum vertical coordinates of the first empty frames and the maximum vertical coordinates of the third product frames to obtain a first sorting result; traverse each first empty frame and each third product frame in sequence according to the first sorting result, and compare the maximum vertical coordinate of the currently traversed frame with a preset vertical coordinate range of each layer on the shelf; if the maximum vertical coordinate of the currently traversed frame is within the preset vertical coordinate range of any layer on the shelf, determine that the layer of the currently traversed frame is the layer corresponding to the currently preset vertical coordinate range, thereby completing the stratification of all the first empty frames and all the third product frames; obtain the stratification results of all the second empty frames from the stratification results of all the first empty frames, and assign the stratification results of each second empty frame to the second product frame corresponding to each second empty frame to determine the layer of each second product frame;

[0039] An analyzing unit is used to analyze a width reference of each layer on the shelf based on all the first position information and all the second position information.

[0040] As a preferred solution, the calculation feedback module specifically includes:

[0041] a position analysis unit, configured to obtain the maximum vertical coordinate of each second product frame from all first position information, and obtain the maximum vertical coordinate of each second empty space frame, as well as the maximum horizontal coordinate and minimum horizontal coordinate of each first empty space frame from all second position information, and then subtract the maximum vertical coordinate of each second empty space frame from the maximum vertical coordinate of the second product frame corresponding to each second empty space frame to obtain a bottom deviation value corresponding to each second empty space frame;

[0042] an empty space quantity calculation unit, configured to determine the vacancy type of each first empty space frame based on the bottom deviation value corresponding to each second empty space frame, and then calculate the SKU placement quantity of each first empty space frame based on the vacancy type of each first empty space frame, the maximum and minimum values ​​of the horizontal coordinates of each first empty space frame, and the width reference of the layer where each first empty space frame is located; and add the SKU placement quantities of all the first empty spaces to obtain the total SKU placement quantity of the empty shelf space;

[0043] The feedback warning unit is used to feed back a reminder signal and the total SKU placement quantity to the monitoring platform when the total SKU placement quantity is greater than a second threshold.

[0044] In order to solve the same technical problem, the present invention also provides a terminal, including a processor, a memory and a computer program stored in the memory; wherein, the computer program can be executed by the processor to implement the target detection-based shelf management method.

[0045] In order to solve the same technical problem, the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the target detection-based shelf management method. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 : A flowchart of a shelf management method based on target detection provided in Example 1 of the present invention;

[0047] Figure 2 : A structural diagram of a shelf management system based on target detection provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] Embodiment one:

[0050] Please refer to Figure 1 , a shelf management method based on target detection provided by an embodiment of the present invention, the method includes steps S1 to S3, each step is specifically as follows:

[0051] Step S1, performing target detection on the shelf image acquired in real time, obtaining multiple first product boxes (sku_box_res) and multiple first empty boxes (empty_box_res) in the shelf image, as well as first position information of each first product box and second position information of each first empty box.

[0052] The first product frame is the minimum circumscribed rectangle of the SKU on the shelf, and the SKU refers to the shelf product with a specific business barcode.

[0053] In this embodiment, the position information of the box is (x_min, y_min, x_max, y_max, label); wherein x_min represents the minimum value of the horizontal coordinate of the box, y_min represents the minimum value of the vertical coordinate, x_max represents the maximum value of the horizontal coordinate, y_max represents the maximum value of the vertical coordinate, and label represents the label of the box, which is SKU or empty space.

[0054] As a preferred solution, step S1 includes steps S11 to S12, and the details of each step are as follows:

[0055] Step S11: photograph the shelf in real time to obtain a shelf image.

[0056] Step S12: Use the target detection model to perform target detection on the shelf image to obtain multiple first product boxes (sku_box_res) and multiple first empty boxes (empty_box_res) in the shelf image, as well as the first position information of each first product box and the second position information of each first empty box.

[0057] Among them, the target detection model is obtained by training the target detector using a training data set. The training data set includes several training images, multiple product frames and multiple empty space frames in each training image, and the position information of each product frame and each empty space frame. The training images are obtained by photographing the shelves.

[0058] In step S2, a first product frame whose intersection-over-union ratio with any first empty space frame is less than or equal to a first threshold is used as a second product frame. All first product frames and all first empty space frames are layered according to all first position information and all second position information, and a width reference value of each layer is analyzed.

[0059] The reference width of each layer is the median of the widths of all second commodity boxes on each layer of the shelf.

[0060] It should be noted that in actual application scenarios, SKUs on a shelf may be placed closer to the inside, while the shelf positions are empty. In this case, when performing object detection on the captured shelf image, the intersection-and-union ratio (IoU) of some product frames and the empty slot frames is low. These product frames are not normal shelf frames, while all the empty slot frames are normal shelf frames. Therefore, comparing the IoU ratio with the first threshold allows for subsequent differentiation and processing of different degrees of overlap.

[0061] As a preferred solution, step S2 includes steps S21 to S25, and each step is specifically as follows:

[0062] Step S21: Obtain the intersection-and-union (IoU) of the first product frame and the first empty space frame. Then, select the first product frame whose IoU is greater than a first threshold as the second product frame, select the first product frame whose IoU is less than or equal to the first threshold as the third product frame, and use the first empty space frame whose IoU with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame.

[0063] In this embodiment, all third product boxes and all first empty space boxes are boxes with normal layout. The set composed of all boxes with normal layout is recorded as sku_empty_box_res, and the set composed of all second product boxes and all second empty space boxes corresponding to the second product boxes is recorded as sku_empty_box_filter_res, and the information format of the set is (second empty space box, second SKU box, IOU), where IOU represents the intersection and union ratio of the second empty space box and its corresponding second product box.

[0064] Step S22: Obtain the maximum vertical coordinate value of each third product frame from all first position information, and obtain the maximum vertical coordinate value of each first empty space frame from all second position information. Then, sort all first empty space frames and all third product frames simultaneously according to the maximum vertical coordinate value y_max of the first empty space frame and the maximum vertical coordinate value y_max of the third product frame from large to small, thereby obtaining a first sorting result.

[0065] Step S23, traverse each first empty box and each third product box in turn according to the first sorting result, that is, traverse all boxes belonging to the set sku_empty_box_res in turn according to the first sorting result, and compare the maximum vertical coordinate y_max of the current traversal box with the preset vertical coordinate range of each layer on the shelf. If the maximum vertical coordinate y_max of the current traversal box is within the preset vertical coordinate range of any layer on the shelf, then determine that the layer where the current traversal box is located is the layer corresponding to the current preset vertical coordinate range, so as to complete the stratification of all first empty boxes and all third product boxes.

[0066] As an example, a shelf has three layers. The preset vertical coordinate range of the first layer on the shelf is less than a threshold value Q1, the preset vertical coordinate range of the second layer on the shelf is greater than or equal to the threshold value Q1 and less than a threshold value Q2, and the preset vertical coordinate range of the third layer on the shelf is greater than or equal to the threshold value Q2 and less than a threshold value Q3. If the maximum vertical coordinate y_max of the first empty box currently traversed is less than the threshold value Q1, then the layer where the third product box currently traversed is located is the first layer on the shelf; if the maximum vertical coordinate y_max of the third product box currently traversed is greater than or equal to the threshold value Q2 and less than the threshold value Q3, then the layer where the third product box currently traversed is located is the third layer on the shelf.

[0067] It should be noted that after the layering of all first empty boxes and all third product boxes is completed, the information format of the boxes belonging to the set sku_empty_box_res should be adjusted to (x_min, y_min, x_max, y_max, label) with the addition of the layer where the box is located, that is, (x_min, y_min, x_max, y_max, label, layer).

[0068] Step S24 , obtaining the hierarchical results of all second vacant frames from the hierarchical results of all first vacant frames, and assigning the hierarchical results of each second vacant frame to the second product frame corresponding to each second vacant frame to determine the layer where each second product frame is located.

[0069] Step S25: Analyze the width reference of each layer on the shelf based on all the first position information and all the second position information.

[0070] As a preferred solution, step S25 includes step S251 to step S252, and each step is specifically as follows:

[0071] Step S251, obtain the maximum horizontal coordinate x_max and the minimum horizontal coordinate x_min of each second product frame, as well as the maximum horizontal coordinate x_max and the minimum horizontal coordinate x_min of each third product frame from all the first position information, then subtract the maximum horizontal coordinate x_max of each second product frame from the minimum horizontal coordinate x_min of each second product frame to obtain the width of each second product frame, and subtract the maximum horizontal coordinate x_max of each third product frame from the minimum horizontal coordinate x_min of each third product frame to obtain the width of each third product frame.

[0072] In this embodiment, after executing step S251, a vacancy type threshold acquisition process is further included. The process includes steps S01 to S03. The details of each step are as follows:

[0073] Step S01, traverse each layer on the shelf, sort the widths of all second product boxes of the current traversal layer in ascending or descending order, then select the median of the sorting results as the first width statistic width_sku_empty_box of the current traversal layer, and count the number number1 of the widths of all second product boxes of the current traversal layer, sort the widths of all third product boxes of the current traversal layer in ascending or descending order, then select the median of the sorting results as the second width statistic width_sku_box_filter of the current traversal layer, and count the number number2 of the widths of all third product boxes of the current traversal layer.

[0074] In step S03, refer to formula (1). Based on the first width statistic width_sku_empty_box of each layer, the number of widths of all second product boxes number1, the second width statistic width_sku_box_filter, and the number of widths of all third product boxes number2 obtained in step S01, calculate the first vacancy type threshold empty_1_thresh and the second vacancy type threshold empty_2_thresh corresponding to the vacancy boxes of each layer.

[0075]

[0076] Where a1 and a2 are empirical setting values.

[0077] It should be noted that the number and definition of slot types can be adaptively adjusted based on actual application scenarios. In this embodiment, the slot types of the slot box are empty_1, empty_2, and empty_3. Empty_1 indicates that the slot box can accommodate one SKU in the depth direction, empty_2 indicates that the slot box can accommodate two SKUs in the depth direction, and empty_3 indicates that the slot box can accommodate two or more SKUs in the depth direction.

[0078] Step S252: traverse each layer on the shelf, sort the widths of all second commodity boxes and all third commodity boxes of the current traversal layer in ascending or descending order, and select the median of the sorting results as the width reference width_ref of the current traversal layer.

[0079] Step S3, calculating the total number of SKUs placed in the shelf space based on the width reference of the layer where each first space frame is located, and when the total number of SKUs placed is greater than the second threshold, feeding back a reminder signal and the total number of SKUs placed to the monitoring platform.

[0080] As a preferred solution, step S3 includes steps S31 to S25, and each step is specifically as follows:

[0081] Step S31: Obtain the maximum vertical coordinate value y_max of each second product frame from all first position information, and obtain the maximum vertical coordinate value y_max of each second empty frame, as well as the maximum horizontal coordinate value x_max and the minimum horizontal coordinate value x_min of each first empty frame from all second position information. Then, subtract the maximum vertical coordinate value y_max of each second empty frame from the maximum vertical coordinate value y_max of the second product frame corresponding to each second empty frame to obtain the bottom deviation value delta_depth corresponding to each second empty frame.

[0082] In step S32, the vacancy type of each first vacancy frame is determined based on the bottom deviation value delta_depth corresponding to each second vacancy frame. Then, the SKU placement amount sku_occupancy of each first vacancy frame is calculated based on the vacancy type of each first vacancy frame, the maximum horizontal coordinate value x_max and the minimum horizontal coordinate value x_min of each first vacancy frame, and the width reference value width_ref of the layer where each first vacancy frame is located.

[0083] In this embodiment, the execution process of step S32 may specifically include steps S321 to S322, and each step is specifically as follows:

[0084] In step S321, first, the slot type of all first slot boxes is set to empty_3. The second slot box is a special first slot box corresponding to the second product box. For each second empty space frame, if the bottom deviation value width_ref corresponding to the current second empty space frame is less than the first empty space type threshold corresponding to the layer where the current second empty space frame is located, the empty space type of the current second empty space frame is determined to be empty_1, and the empty space type of the current second empty space frame (the first empty space frame) is changed from empty_3 to empty_1. If the bottom deviation value width_ref corresponding to the current second empty space frame is greater than or equal to the first empty space type threshold corresponding to the layer where the current second empty space frame is located and less than the second empty space type threshold corresponding to the layer where the current second empty space frame is located, the empty space type of the current second empty space frame is determined to be empty_2, and the empty space type of the current second empty space frame (the first empty space frame) is changed from empty_3 to empty_2. If the bottom deviation value width_ref corresponding to the current second empty space frame is greater than the second empty space type threshold corresponding to the layer where the current second empty space frame is located, the empty space type of the current second empty space frame is determined to be empty_3, and the empty space type of the current second empty space frame (the first empty space frame) does not need to be changed.

[0085] In step S322, for the first empty frame of type empty_1, the type code is set to depth_number = 1; for the first empty frame of type empty_2, the type code is set to depth_number = 2; for the first empty frame of type empty_3, the type code is set to depth_number = 3. Then, referring to formula (2), the SKU placement amount sku_occupancy of the current first empty frame is calculated based on the width reference corresponding to the layer where the current first empty frame is located, the minimum horizontal coordinate value, the maximum horizontal coordinate value, and the type code of the current first empty frame.

[0086]

[0087] Step S33: Add up the SKU placement quantities sku_occupancy of all first vacant slots to obtain the total SKU placement quantity of the shelf slots.

[0088] Step S34: When the total SKU placement quantity is greater than the second threshold, a reminder signal and the total SKU placement quantity are fed back to the monitoring platform to remind the management personnel of the monitoring platform or the personnel of the enterprise or other terminals using the monitoring platform to track the total SKU placement quantity of the shelves in real time.

[0089] Please refer to Figure 2 , which is a schematic diagram of the structure of a shelf management system based on target detection provided by an embodiment of the present invention. The system includes a target detection module M1, a hierarchical analysis module M2, and a calculation feedback module M3. The details of each module are as follows:

[0090] The target detection module M1 is configured to perform target detection on the shelf image acquired in real time, and obtain a plurality of first product frames and a plurality of first space frames within the shelf image, as well as first position information of each first product frame and second position information of each first space frame; wherein the first product frame is the minimum circumscribed rectangle of the SKU on the shelf;

[0091] The layered analysis module M2 is configured to select a first product frame whose intersection-over-union ratio with any first empty space frame is less than or equal to a first threshold as a second product frame, stratify all first product frames and all first empty space frames based on all first position information and all second position information, and analyze a width reference value for each layer; wherein the width reference value for each layer is the median width of all second product frames on each layer of the shelf;

[0092] The calculation feedback module M3 is used to calculate the total number of SKUs placed in the shelf space based on the width reference of the layer where each first space frame is located. When the total number of SKUs placed is greater than the second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

[0093] As a preferred solution, the hierarchical analysis module M2 specifically includes a screening unit 21, a stratification unit 22, and an analysis unit 23. The details of each unit are as follows:

[0094] The screening unit 21 is configured to obtain an intersection-and-union (IoU) ratio between the first product frame and the first empty space frame, then select the first product frame whose IoU ratio is greater than a first threshold as the second product frame, select the first product frame whose IoU ratio is less than or equal to the first threshold as the third product frame, and select the first empty space frame whose IoU ratio with each second product frame is greater than the first threshold as the second empty space frame corresponding to each second product frame.

[0095] The stratification unit 22 is configured to obtain the maximum vertical coordinate value of each third product frame from all first position information and the maximum vertical coordinate value of each first vacant frame from all second position information, and then simultaneously sort all first vacant frames and all third product frames in descending order of the maximum vertical coordinate values ​​of the first vacant frames and the maximum vertical coordinate values ​​of the third product frames to obtain a first sorting result. The stratification unit 22 is configured to sequentially traverse each first vacant frame and each third product frame based on the first sorting result, and compare the maximum vertical coordinate value of the currently traversed frame with a preset vertical coordinate range for each layer on the shelf. If the maximum vertical coordinate value of the currently traversed frame is within the preset vertical coordinate range for any layer on the shelf, the layer on which the currently traversed frame resides is determined to be the layer corresponding to the currently preset vertical coordinate range, thereby completing the stratification of all first vacant frames and all third product frames. The stratification results of all second vacant frames are obtained from the stratification results of all first vacant frames, and the stratification results of each second vacant frame are assigned to the second product frame corresponding to each second vacant frame to determine the layer on which each second product frame resides.

[0096] The analyzing unit 23 is configured to analyze a width reference of each layer on the shelf according to all the first position information and all the second position information.

[0097] As a preferred solution, the calculation feedback module M3 specifically includes a position analysis unit 31, an empty position release calculation unit 32 and a feedback warning unit 33. The details of each unit are as follows:

[0098] The position analysis unit 31 is configured to obtain the maximum vertical coordinate value of each second product frame from all first position information, and obtain the maximum vertical coordinate value of each second empty space frame, as well as the maximum horizontal coordinate value and the minimum horizontal coordinate value of each first empty space frame from all second position information. The unit then subtracts the maximum vertical coordinate value of each second empty space frame from the maximum vertical coordinate value of the second product frame corresponding to each second empty space frame to obtain a bottom deviation value corresponding to each second empty space frame.

[0099] The empty space quantity calculation unit 32 is used to determine the empty space type of each first empty space frame based on the bottom deviation value corresponding to each second empty space frame, and then calculate the SKU placement quantity of each first empty space frame according to the empty space type of each first empty space frame, the maximum and minimum values ​​of the horizontal coordinates of each first empty space frame, and the width reference of the layer where each first empty space frame is located; the SKU placement quantities of all first empty space frames are added together to obtain the total SKU placement quantity of the empty space on the shelf.

[0100] The feedback warning unit 33 is used to feed back a reminder signal and the total number of SKUs placed to the monitoring platform when the total number of SKUs placed is greater than the second threshold.

[0101] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0102] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a shelf management method based on target detection in embodiment one.

[0103] In addition, an embodiment of the present invention further provides a terminal including a processor, a memory, and a computer program stored in the memory; wherein the computer program can be executed by the processor to implement a shelf management method based on target detection of the first embodiment.

[0104] Preferably, the computer program can be divided into one or more modules / units (e.g., computer program, computer program), one or more modules / units are stored in a memory and executed by a processor to implement the present invention. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal.

[0105] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the terminal, and various parts of the terminal are connected using various interfaces and lines.

[0106] The memory mainly includes a program storage area and a data storage area. The program storage area can store the operating system, at least one application required for a function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, and a flash memory card, etc., or the memory can also be other volatile solid-state memory devices.

[0107] It should be noted that the above-mentioned terminal may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above-mentioned terminal is merely an example and does not constitute a limitation on the terminal. It may include more or fewer components, or a combination of certain components, or different components.

[0108] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0109] The present invention provides a shelf management method, system, terminal and medium based on target detection, which performs target detection on shelf images acquired in real time to preliminarily identify the SKUs and vacancies on the shelf, and obtains multiple first product frames and multiple first vacancy frames in the shelf image, as well as first position information of each first product frame and second position information of each first vacancy frame. Then, since the farther the object is from the camera lens, the smaller the image of the object in the photo, and some SKUs in the shelf may be placed in the inner position, in the shelf image obtained by photographing the shelf, the first product frames corresponding to these part of the SKUs have less overlapping parts with the first vacancy frames. Therefore, the first product frame whose intersection-with-union ratio with any first vacancy frame is less than or equal to the first threshold is used as the second product frame, and all first product frames and all second position frames are compared based on all first position information and all second position information. The first empty space frames are layered, and the median of the widths of all the second product frames on each layer of the shelf is analyzed as a width reference for each layer. Then, the total SKU placement quantity of the empty shelf spaces is calculated based on the width reference of the layer where each first empty space frame is located. Through the above-mentioned screening means, the degree of overlap between the first empty space frame and the first product frame can be accurately analyzed based on the interaction ratio of the first empty space frame and the first product frame, and then the first product frames with more overlap with the first empty space frame can be filtered out, and only the first product frames with less overlap with the first empty space frame can be selected as the second product frames. The total SKU placement quantity of the empty shelf spaces is calculated based on the median of the widths of all the second product frames on each layer of the shelf. The relationship between the first product frames with different degrees of overlap and the empty shelf spaces can be analyzed separately, and a more accurate total SKU placement quantity of the empty shelf spaces can be obtained, so as to dynamically track the empty space status of the shelf. In addition, when the total SKU placement quantity is greater than the second threshold, a reminder signal and the total SKU placement quantity are fed back to the monitoring platform, which can promptly remind the managers of the monitoring platform or the staff of the fast-moving consumer goods companies using the monitoring platform to quickly understand the actual sales status of the shelves and make correct decisions.

[0110] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A shelf management method based on target detection, characterized in that: include: Performing target detection on a shelf image acquired in real time to obtain a plurality of first product frames and a plurality of first empty space frames within the shelf image, as well as first position information of each first product frame and second position information of each first empty space frame; wherein the first product frame is a minimum bounding rectangle of a SKU on the shelf; Obtaining an intersection-and-union ratio (IoU) of the first product frame and the first empty space frame, then selecting the first product frame whose IoU ratio is greater than a first threshold as the second product frame, and selecting the first product frame whose IoU ratio is less than or equal to the first threshold as the third product frame. Based on all the first position information and all the second position information, stratify all the first product frames and all the first empty space frames, and analyze a width reference value for each layer; wherein the width reference value for each layer is the median of the widths of all the second product frames and the third product frames on each layer of the shelf; The total number of SKUs placed in the shelf spaces is calculated based on the width reference of the layer where each of the first space frames is located. When the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

2. The shelf management method based on target detection according to claim 1, characterized in that: The step of layering all the first product boxes and all the first space boxes according to all the first position information and all the second position information, and analyzing the width reference of each layer, is as follows: The first empty frames whose intersection-over-union ratio with each second product frame is greater than the first threshold are used as the second empty frames corresponding to each second product frame; Obtaining the maximum vertical coordinate value of each third product frame from all first position information, and obtaining the maximum vertical coordinate value of each first empty frame from all second position information, and then sorting all the first empty frames and all the third product frames simultaneously according to the maximum vertical coordinate values ​​of the first empty frames and the maximum vertical coordinate values ​​of the third product frames from largest to smallest, to obtain a first sorting result; Traversing each first empty space frame and each third product frame in sequence according to the first sorting result, comparing the maximum vertical coordinate of the currently traversed frame with a preset vertical coordinate range of each layer on the shelf; if the maximum vertical coordinate of the currently traversed frame is within the preset vertical coordinate range of any layer on the shelf, determining that the layer of the currently traversed frame is the layer corresponding to the currently preset vertical coordinate range, thereby completing the layering of all first empty space frames and all third product frames; Obtaining the hierarchical results of all the second empty frames from the hierarchical results of all the first empty frames, and assigning the hierarchical results of each second empty frame to the second product frame corresponding to each second empty frame to determine the layer where each second product frame is located; The width reference of each layer on the shelf is analyzed based on all the first position information and all the second position information.

3. The shelf management method based on target detection according to claim 2, characterized in that: The analyzing the width reference of each layer on the shelf according to all the first position information and all the second position information is specifically as follows: Obtaining the maximum and minimum horizontal coordinate values ​​of each second product frame, and the maximum and minimum horizontal coordinate values ​​of each third product frame, from all the first position information, then subtracting the maximum horizontal coordinate value of each second product frame from the minimum horizontal coordinate value of each second product frame to obtain the width of each second product frame, and subtracting the maximum horizontal coordinate value of each third product frame from the minimum horizontal coordinate value of each third product frame to obtain the width of each third product frame; Traverse each layer on the shelf, sort the widths of all the second product boxes and the widths of all the third product boxes in the current traversal layer in ascending or descending order, and select the median of the sorting results as a reference for the width of the current traversal layer.

4. The shelf management method based on target detection according to claim 2, characterized in that: The total number of SKUs placed in the empty shelf spaces is calculated based on the width reference of the layer where each first space frame is located. When the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform. Specifically, Obtaining the maximum vertical coordinate value of each second product frame from all first position information, and obtaining the maximum vertical coordinate value of each second empty frame, as well as the maximum horizontal coordinate value and the minimum horizontal coordinate value of each first empty frame from all second position information. Then, subtracting the maximum vertical coordinate value of each second empty frame from the maximum vertical coordinate value of the second product frame corresponding to each second empty frame to obtain a bottom deviation value corresponding to each second empty frame. Based on the bottom deviation value corresponding to each second empty space frame, the space type of each first empty space frame is determined, and then the SKU placement quantity of each first empty space frame is calculated according to the space type of each first empty space frame, the maximum and minimum values ​​of the horizontal coordinates of each first empty space frame, and the width reference of the layer where each first empty space frame is located; Adding up the SKU placement quantities of all the first empty slots to obtain the total SKU placement quantity of the shelf slots; When the total placement quantity of the SKU is greater than the second threshold, a reminder signal and the total placement quantity of the SKU are fed back to the monitoring platform.

5. The shelf management method based on target detection according to claim 1, characterized in that: The target detection is performed on the shelf image acquired in real time to obtain a plurality of first commodity frames and a plurality of first vacant frames in the shelf image, as well as first position information of each of the first commodity frames and second position information of each of the first vacant frames, specifically: Take real-time photos of the shelves to obtain shelf images; Performing target detection on the shelf image using a target detection model to obtain a plurality of first product frames and a plurality of first space frames within the shelf image, as well as first position information of each of the first product frames and second position information of each of the first space frames; The target detection model is obtained by training the target detector using a training data set, wherein the training data set includes several training images, multiple product frames and multiple empty space frames in each training image, and position information of each product frame and each empty space frame, and the training images are obtained by photographing the shelves.

6. A shelf management system based on target detection, characterized in that: include: an object detection module, configured to perform object detection on a shelf image acquired in real time, and obtain a plurality of first product frames and a plurality of first empty space frames within the shelf image, as well as first position information of each first product frame and second position information of each first empty space frame; wherein the first product frame is a minimum bounding rectangle of a SKU on the shelf; a layered analysis module, configured to obtain an intersection-and-union ratio (IoU) of the first product frames and the first empty space frames, then select the first product frames whose IoU ratio is greater than a first threshold as second product frames, select the first product frames whose IoU ratio is less than or equal to the first threshold as third product frames, stratify all the first product frames and all the first empty space frames based on all the first position information and all the second position information, and analyze a width reference value for each layer; wherein the width reference value for each layer is the median of the widths of all the second product frames and the third product frames on each layer of the shelf; A calculation feedback module is used to calculate the total number of SKUs placed in the shelf space based on the width reference of the layer where each first space frame is located, and when the total number of SKUs placed is greater than a second threshold, a reminder signal and the total number of SKUs placed are fed back to the monitoring platform.

7. The target detection-based shelf management system according to claim 6, characterized in that: The hierarchical analysis module specifically includes: a screening unit, configured to obtain an intersection-and-union (IoU) ratio between the first product frames and the first empty space frames, then select the first product frames whose IoU ratio is greater than a first threshold as second product frames, select the first product frames whose IoU ratio is less than or equal to the first threshold as third product frames, and select the first empty space frames whose IoU ratio with each second product frame is greater than the first threshold as second empty space frames corresponding to each second product frame; a stratification unit, configured to obtain the maximum vertical coordinate of each third product frame from all first position information, and obtain the maximum vertical coordinate of each first empty frame from all second position information, and then simultaneously sort all the first empty frames and all the third product frames in descending order of the maximum vertical coordinates of the first empty frames and the maximum vertical coordinates of the third product frames to obtain a first sorting result; traverse each first empty frame and each third product frame in sequence according to the first sorting result, and compare the maximum vertical coordinate of the currently traversed frame with a preset vertical coordinate range of each layer on the shelf; if the maximum vertical coordinate of the currently traversed frame is within the preset vertical coordinate range of any layer on the shelf, determine that the layer of the currently traversed frame is the layer corresponding to the currently preset vertical coordinate range, thereby completing the stratification of all the first empty frames and all the third product frames; obtain the stratification results of all the second empty frames from the stratification results of all the first empty frames, and assign the stratification results of each second empty frame to the second product frame corresponding to each second empty frame to determine the layer of each second product frame; An analyzing unit is used to analyze a width reference of each layer on the shelf according to all the first position information and all the second position information.

8. The target detection-based shelf management system according to claim 7, characterized in that: The calculation feedback module specifically includes: a position analysis unit, configured to obtain the maximum vertical coordinate of each second product frame from all first position information, and obtain the maximum vertical coordinate of each second empty space frame, as well as the maximum horizontal coordinate and minimum horizontal coordinate of each first empty space frame from all second position information, and then subtract the maximum vertical coordinate of each second empty space frame from the maximum vertical coordinate of the second product frame corresponding to each second empty space frame to obtain a bottom deviation value corresponding to each second empty space frame; an empty space quantity calculation unit, configured to determine the vacancy type of each first empty space frame based on the bottom deviation value corresponding to each second empty space frame, and then calculate the SKU placement quantity of each first empty space frame based on the vacancy type of each first empty space frame, the maximum and minimum values ​​of the horizontal coordinates of each first empty space frame, and the width reference of the layer where each first empty space frame is located; and add the SKU placement quantities of all the first empty spaces to obtain the total SKU placement quantity of the empty shelf space; The feedback warning unit is used to feed back a reminder signal and the total SKU placement quantity to the monitoring platform when the total SKU placement quantity is greater than a second threshold.

9. A terminal, characterized in that: It comprises a processor, a memory and a computer program stored in the memory; wherein the computer program can be executed by the processor to implement a shelf management method based on target detection as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the target detection-based shelf management method according to any one of claims 1 to 5.

Images