Commodity out-of-stock detection method, device and system

Abstract

The present invention discloses a method, device and system for detecting out-of-stock goods. The method comprises: obtaining a display image of an object in a full-stock state at each level of a shelf; determining the type of each level, adopting a corresponding anchor frame detection strategy, and performing anchor frame detection on the display image in a full-stock state at each level; if the object is a single commodity, obtaining an empty-stock feature vector set and at least one sample feature vector of the commodity corresponding to each anchor frame according to multiple historical display images; after obtaining the image of the level to be analyzed, intercepting the anchor frame image of each anchor frame; for each anchor frame, comparing the feature vector of the anchor frame image of the anchor frame with at least one sample feature vector of the commodity corresponding to the anchor frame and the empty-stock feature vector set to determine the state of the anchor frame; for an out-of-stock anchor frame, determining and reporting the commodity information of the electronic price tag corresponding to the anchor frame according to the extended area of ​​the anchor frame position of the anchor frame. The present invention can efficiently and accurately perform out-of-stock detection on shelves.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a commodity out-of-stock detection method, device and system. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] Merchants are increasingly relying on efficient inventory management systems to ensure adequate supply of goods. However, traditional inventory management methods usually rely on manual inspection and manual recording, which is not only time-consuming and labor-intensive, but also prone to errors, resulting in out-of-stock goods not being discovered or replenished in a timely manner, which in turn affects customer experience and sales performance. With the continuous development of the modern retail industry, automated, efficient, and accurate out-of-stock detection methods have become an important demand in the current retail industry. Summary of the invention

[0004] The embodiment of the present invention provides a commodity out-of-stock detection method, which can automatically, efficiently and accurately perform out-of-stock detection on a shelf. The method includes:

[0005] Obtain display images of all objects at each shelf level in a fully stocked state;

[0006] According to the objects in each level of the shelf, the type of each level is determined, and the corresponding anchor frame detection strategy is adopted to perform anchor frame detection on the display images of each level in the full state to obtain the anchor frame of each object in each level;

[0007] If the object is a single commodity, a set of empty goods feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame are obtained according to a plurality of historical display images of each level of the shelf;

[0008] After obtaining the image of the level to be analyzed, the corresponding anchor frame detection strategy is adopted to obtain the anchor frame of each object in the level to be analyzed, and the anchor frame image of each anchor frame in the level to be analyzed is intercepted;

[0009] If the object is a single product, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and a set of out-of-stock feature vectors of the product corresponding to the anchor frame, and the state of the anchor frame is determined according to the comparison result, and the state is out of stock or in stock;

[0010] For an out-of-stock anchor frame, determine the electronic price tag corresponding to the anchor frame according to the extended area of ​​the anchor frame position of the anchor frame;

[0011] Report the product information corresponding to the out-of-stock electronic price tag.

[0012] The embodiment of the present invention provides a commodity out-of-stock detection device, which can automatically, efficiently and accurately detect out-of-stock on a shelf. The device includes:

[0013] A display image acquisition module is used to obtain display images of all objects at each level of the shelf in a fully stocked state;

[0014] The anchor frame detection module is used to determine the type of each shelf level according to the objects in each shelf level, and adopt the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each shelf level in the full state, and obtain the anchor frame of each object in each shelf level;

[0015] A feature vector acquisition module, for obtaining, if the object is a single commodity, a set of empty goods feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame according to a plurality of historical display images of each level of the shelf;

[0016] An anchor frame image capture module is used to obtain the image of the level to be analyzed, adopt the corresponding anchor frame detection strategy, obtain the anchor frame of each object in the level to be analyzed, and capture the anchor frame image of each anchor frame in the level to be analyzed;

[0017] An anchor frame state analysis module, for comparing, for each anchor frame, a feature vector of an anchor frame image of the anchor frame with at least one sample feature vector and a set of out-of-stock feature vectors of the product corresponding to the anchor frame, if the object is a single product, and determining the state of the anchor frame according to the comparison result, the state being out of stock or in stock;

[0018] An electronic price tag determination module is used to determine, for an out-of-stock anchor frame, the electronic price tag corresponding to the anchor frame according to an extension area of ​​the anchor frame position of the anchor frame;

[0019] The reporting module is used to report the product information corresponding to the out-of-stock electronic price tags.

[0020] The embodiment of the present invention provides a commodity out-of-stock detection system, which can automatically, efficiently and accurately perform out-of-stock detection on a shelf. The system includes: a visual sensor and the above-mentioned commodity out-of-stock detection device;

[0021] The camera is used to capture display images of all objects at each level of the shelf in a fully stocked state, as well as images of the level to be analyzed, and send them to the commodity out-of-stock detection device.

[0022] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned commodity out-of-stock detection method when executing the computer program.

[0023] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned commodity out-of-stock detection method is implemented.

[0024] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned commodity out-of-stock detection method is implemented.

[0025] In an embodiment of the present invention, a display image of all objects in each level of the shelf in a full-stock state is obtained; the type of each level is determined according to the objects in each level of the shelf, and a corresponding anchor frame detection strategy is adopted to perform anchor frame detection on the display image in a full-stock state of each level to obtain an anchor frame of each object in each level; if the object is a single commodity, a set of empty-stock feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame are obtained according to multiple historical display images of each level of the shelf; after obtaining the image of the level to be analyzed, a corresponding anchor frame detection strategy is adopted to obtain an anchor frame of each object in the level to be analyzed, and an anchor frame image of each anchor frame in the level to be analyzed is intercepted; if the object is a single commodity, for each anchor frame, a feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector of the commodity corresponding to the anchor frame and a set of empty-stock feature vectors, and the state of the anchor frame is determined according to the comparison result, and the state is out of stock or in stock; for an out-of-stock anchor frame, an electronic price tag corresponding to the anchor frame is determined according to an extended area of ​​the anchor frame position of the anchor frame; and commodity information corresponding to the out-of-stock electronic price tag is reported. Through the above process, the shelf display information of the full stock state is restored through the historical display information. The corresponding anchor frame detection strategy can be determined according to the type of level, making the anchor frame detection more accurate. For the object of a single commodity, a set of empty stock feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame can be obtained, which are then compared with the feature vectors of the anchor frame image of each anchor frame in the level to be analyzed to give an accurate out-of-stock judgment result. After determining the position of the out-of-stock anchor frame, one or more price tags in the area extending downward from the anchor frame are used to search for out-of-stock commodity information. This process is efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0027] Figure 1 Flow chart of a method for detecting a commodity out of stock in an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of dividing each historical display image into a plurality of blocks of equal proportions in an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of a reference mask image in an embodiment of the present invention;

[0030] Figure 4 Schematic diagram of a mask image corresponding to the display images of all objects in a fully stocked state in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the first type of shelf hierarchy in an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of the second type of shelf hierarchy in an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the third type of shelf levels in an embodiment of the present invention;

[0034] Figure 8 This is a schematic diagram of the first type of shelf-level out-of-stock judgment logic in an embodiment of the present invention;

[0035] Fig. 9 This is a schematic diagram of the second type of shelf-level out-of-stock judgment logic in an embodiment of the present invention;

[0036] Fig.10 This is a schematic diagram of the third type of shelf-level out-of-stock judgment logic in an embodiment of the present invention;

[0037] Fig.11 The principle of searching for an electronic price tag through the anchor frame position extension area of ​​the anchor frame in the embodiment of the present invention Figure 1 ;

[0038] Fig.12 The principle of searching for an electronic price tag through the anchor frame position extension area of ​​the anchor frame in the embodiment of the present invention Figure 2 ;

[0039] Fig.13 Schematic diagram of the structure of a commodity out-of-stock detection device in an embodiment of the present invention;

[0040] Fig.14 It is a structural schematic diagram of a commodity out-of-stock detection system in an embodiment of the present invention;

[0041] Fig.15 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0043] Figure 1 The flowchart of the commodity out-of-stock detection method in an embodiment of the present invention includes:

[0044] Step 101, obtaining a display image of all objects at each level of the shelf in a fully stocked state;

[0045] Step 102, according to the objects in each level of the shelf, determine the type of each level, and use the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level in the full-stock state to obtain the anchor frame of each object in each level;

[0046] Step 103: if the object is a single commodity, obtain a set of empty goods feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame according to a plurality of historical display images of each level of the shelf;

[0047] Step 104, after obtaining the image of the level to be analyzed, adopt the corresponding anchor frame detection strategy to obtain the anchor frame of each object in the level to be analyzed, and intercept the anchor frame image of each anchor frame in the level to be analyzed;

[0048] Step 105, if the object is a single product, for each anchor frame, compare the feature vector of the anchor frame image of the anchor frame with at least one sample feature vector and a set of out-of-stock feature vectors of the product corresponding to the anchor frame, and determine the state of the anchor frame according to the comparison result, the state being out of stock or in stock;

[0049] Step 106, for the out-of-stock anchor frame, determine the electronic price tag corresponding to the anchor frame according to the extension area of ​​the anchor frame position of the anchor frame;

[0050] Step 107: Report the product information corresponding to the out-of-stock electronic price tag.

[0051] Compared with the prior art, the present invention restores the shelf display information of the fully stocked state through the historical display information, and can determine the corresponding anchor frame detection strategy according to the type of level, so that the anchor frame detection is more accurate; for the object of a single commodity, a set of empty commodity feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame can be obtained, and then compared with the feature vector of the anchor frame image of each anchor frame in the level to be analyzed, to give an accurate out-of-stock judgment result, after determining the position of the out-of-stock anchor frame, one or more price tags in the area extending downward from the anchor frame are used to search for out-of-stock commodity information, and this process is highly efficient.

[0052] Each step is described in detail below.

[0053] In step 101, a display image of all objects at each level of the shelf in a fully stocked state is obtained;

[0054] In the embodiment of the present invention, there are two schemes for obtaining the display images of all objects in the fully stocked state.

[0055] In one embodiment, obtaining a display image of all objects at each level of the shelf in a fully stocked state includes:

[0056] Obtain a plurality of historical display images of equal size at each level of the shelf captured by the visual sensor;

[0057] Divide each historical display image into multiple tiles of equal proportions;

[0058] For multiple blocks at the same position in the historical display image, the block with the largest object detection area is selected as the sub-block to be spliced;

[0059] All the sub-image blocks to be stitched together are stitched together to form a display image of all objects at each level in a fully stocked state.

[0060] Specifically, the visual sensor can be a camera, which can be fixed to take pictures. For example, at a certain moment in the past, product A was out of stock, but product B was not out of stock. At a certain moment, product B was out of stock, but product A was not out of stock. Based on these two historical display images, we can stitch together the state where both product A and product B are fully stocked. Similarly, we can restore the state where the shelf is fully stocked. Figure 2 It is a schematic diagram of dividing each historical display image into a plurality of blocks of equal proportions in an embodiment of the present invention. Figure 2 In the figure, it is divided into 3×3=9 tiles. For the position corresponding to the first tile, the tile with the largest object detection area is selected from multiple tiles as the sub-tile to be stitched. After stitching, the display images of all objects at each level in the full stock state can be obtained.

[0061] In one embodiment, obtaining a display image of all objects at each level of the shelf in a fully stocked state includes:

[0062] Obtain a plurality of historical display images of equal size at each level of the shelf captured by the visual sensor;

[0063] The first historical display image is used as a reference image, and detection image recognition is performed on the reference image to obtain multiple detection frames;

[0064] The plurality of detection frames of the reference image are used to form a reference mask image, wherein the areas where there are objects in the reference mask image are marked with a first color, and the areas where there are no objects are marked with a second color;

[0065] Detection image recognition is performed on the historical display images except the benchmark image in turn to obtain multiple candidate detection frames. When the area of ​​the candidate detection frame is larger than a preset proportion of the area of ​​the first color identification part in the benchmark mask image at the corresponding position, the mask image of the candidate detection frame is updated with the benchmark mask image at the corresponding position, and the object frame image detected in the candidate detection frame is pasted at the corresponding position on the benchmark image. After the area judgment of the candidate detection frames of all historical display images is completed, the display images of all objects at each level of the shelf in the full stock state are obtained.

[0066] Figure 3 is a schematic diagram of a reference mask image in an embodiment of the present invention, Figure 3 For example, the areas with objects are marked with white, and the areas without objects are marked with black. Figure 4 Schematic diagram of a mask image corresponding to the display images of all objects in a fully stocked state according to an embodiment of the present invention. It can be seen that the mask image corresponding to the display images of all objects in a fully stocked state is more perfect on the basis of the original reference mask image.

[0067] In step 102, the type of each level is determined according to the objects in each level of the shelf, and the corresponding anchor frame detection strategy is used to perform anchor frame detection on the display image of each level in a full-stock state to obtain the anchor frame of each object in each level;

[0068] In one embodiment, the type of each level is determined based on the objects in each level of the shelf, including:

[0069] If the object in each level of the shelf is a single commodity, the level is determined to be the first type, and the anchor frame detection strategy is determined to place the single commodity in an anchor frame for anchor frame detection;

[0070] If the objects in each level of the shelf are a group of bulk commodities, the level is determined to be the second type, and the anchor frame detection strategy is determined to place all commodities that are completely identical and whose distance is less than the first distance into an anchor frame for anchor frame detection;

[0071] If the objects in each level of the shelf are boxed goods, the level is determined to be the third type, and the anchor frame detection strategy is determined to place a group of completely identical boxed goods in an anchor frame for anchor frame detection.

[0072] In the embodiment of the present invention, the levels include three situations, so when deploying cameras, it is necessary to classify the shelf levels. The classification can be performed according to the objects of the obtained levels.

[0073] In addition, an embodiment of the present invention supports that the classification is formed by manual configuration. That is to say, when the camera is deployed, the classification of the corresponding shelf level can be manually configured to the camera in advance, and the camera can perform automatic classification based on image recognition based on the captured content. For example, through the captured shelf level number, the goods category corresponding to the shelf level number in the background server is searched, and the category of the shelf level is determined based on different goods categories.

[0074] like Figure 5 This is a schematic diagram of the first type of shelf hierarchy in an embodiment of the present invention. For this type of shelf, the display is relatively standard and the merchandise is placed in a standardized manner. This type of shelf ensures that there is only one merchandise in each anchor frame and the merchandise has a clear display surface. Therefore, the anchor frame detection strategy is to place a single merchandise in an anchor frame for anchor frame detection. The red frame is the anchor frame. Figure 6 This is a schematic diagram of the second type of shelf hierarchy in an embodiment of the present invention. The products displayed on this type of shelf are generally some bulk products, such as yogurt, sausages, toothbrushes, etc. in paper boxes. In this case, it cannot be guaranteed that there is only a single product in the anchor frame, and the status of the product in the box is complex. The location where the customer takes the product may be random, and it cannot be guaranteed that the product is taken from the outside to the inside. The anchor frame detection strategy is to place all products that are exactly the same and less than the first distance into an anchor frame for anchor frame detection. The principle is to put the same products close to each other into an anchor frame. If there is a box, then one box is an anchor frame. Figure 6 The red one is the anchor box, and the blue one is the product detection box. Figure 7 This is a schematic diagram of the third type of shelf hierarchy in an embodiment of the present invention. This type of shelf generally places boxed products, such as large bottles of water, etc. The anchor frame detection strategy is to place a group of completely identical boxed products in an anchor frame for anchor frame detection. Figure 7 The red one is the anchor box, and the blue one is the product detection box.

[0075] In one embodiment, anchor frame detection is performed on the display image of each level in a fully stocked state to obtain the anchor frame of each object at each level, including:

[0076] Perform anchor frame detection on the display images of each level with full goods to obtain the anchor frames for preliminary detection;

[0077] The anchor frame detected by the preliminary detection is quality checked, and the anchor frame detected by the preliminary detection is corrected according to the quality inspection result to obtain the anchor frame of each object. The quality inspection includes the object integrity check in the anchor frame, the image absence check in the anchor frame, and the check whether there is an image without an anchor frame.

[0078] Specifically, the anchor frame of the object obtained according to the anchor frame detection strategy is the anchor frame of the preliminary detection. In order to improve its accuracy, quality inspection is required. The object integrity check in the anchor frame refers to checking whether the object is completely contained in the anchor frame. If not, the anchor frame of the complete object is captured again from the display image. The anchor frame can also be manually modified through the operation interface, including addition, deletion, area change, etc.; the image-free check in the anchor frame refers to checking whether there is no image in the anchor frame. If so, the anchor frame is deleted; the anchor frame-free image check refers to checking whether there is an object in the display image for which no anchor frame is generated. If so, anchor frame detection is performed again on the object in the display image.

[0079] In step 103, if the object is a single commodity, a set of empty goods feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame are obtained according to a plurality of historical display images of each level of the shelf;

[0080] Specifically, this step is for a single commodity, and feature vectors do not need to be collected for other objects. The multiple historical display images mentioned here are not the display images of the previous full goods state, but a large number of historical display images of each level of the actual shelves, from which sample feature vectors and empty goods feature vector sets are obtained.

[0081] In one embodiment, obtaining a set of empty goods feature vectors and at least one sample feature vector of a commodity corresponding to each anchor frame according to a plurality of historical display images of each level of the shelf includes:

[0082] Extract at least one sample image within each anchor frame from a plurality of historical display images of each level of the shelf;

[0083] For each sample image, if the sample image does not contain any product, extract the feature vector of the sample image as the empty product feature vector and add it to the empty product feature vector set;

[0084] For each sample image, if the sample image contains a product, the feature vector of the sample image is extracted as a sample feature vector of the product corresponding to the anchor frame.

[0085] Specifically, since there are multiple historical display images, multiple sample images can be obtained for each anchor frame. If there is a product in the sample image, the feature vector of the sample image is extracted as a sample feature vector of the product corresponding to the anchor frame. In this case, one anchor frame will often obtain multiple sample feature vectors; and all empty goods feature vectors, no matter which product's sample image they are obtained from, are added to the empty goods feature vector set.

[0086] In one embodiment, after obtaining the empty goods feature vector set and at least one sample feature vector of the commodity corresponding to each anchor frame, the method further includes:

[0087] A mapping relationship between an anchor frame and at least one sample feature vector is established, and the mapping relationship and at least one sample feature vector and a set of empty and cargo feature vectors are stored in a feature vector library, wherein the mapping relationship includes an anchor frame identifier, an anchor frame position and at least one sample feature vector.

[0088] Specifically, when comparing sample feature vectors later, they are read from the feature vector library, and an anchor frame identifier is set for each anchor frame to facilitate query.

[0089] In step 104, after obtaining the image of the level to be analyzed, the corresponding anchor frame detection strategy is adopted to obtain the anchor frame of each object in the level to be analyzed, and the anchor frame image of each anchor frame in the level to be analyzed is intercepted;

[0090] Specifically, each anchor frame in the level to be analyzed also has a corresponding anchor frame identifier according to the anchor frame position, and obtains its anchor frame image. Due to the different difficulty of identifying and detecting different shelf commodities, different out-of-stock judgment logics need to be adopted for different shelf levels.

[0091] In step 105, if the object is a single product, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and a set of out-of-stock feature vectors of the product corresponding to the anchor frame, and the state of the anchor frame is determined according to the comparison result, and the state is out of stock or in stock;

[0092] Since the difficulty of identifying and detecting commodities on different shelves is different, different out-of-stock judgment logics need to be adopted for different shelves. In the embodiment of the present invention, step 105 of the three levels is aimed at a single commodity, that is, the first type level.

[0093] Figure 8 This is a schematic diagram of the first type of shelf-level out-of-stock judgment logic in an embodiment of the present invention. In one embodiment, for each anchor frame, a feature vector of an anchor frame image of the anchor frame is compared with at least one sample feature vector and an out-of-stock feature vector set of the commodity corresponding to the anchor frame, and the state of the anchor frame is determined according to the comparison result, including:

[0094] For each anchor frame, a first similarity comparison is performed between a feature vector of the anchor frame image of the anchor frame and at least one sample feature vector of the product corresponding to the anchor frame;

[0095] When the maximum similarity of the first similarity comparison result is greater than the first similarity threshold, determining that the state of the anchor frame is in stock;

[0096] Otherwise, a second similarity comparison is performed between the feature vector of the anchor frame image of the anchor frame and each empty cargo feature vector in the empty cargo feature vector set;

[0097] When the maximum similarity of the second similarity comparison result is greater than the second similarity threshold, determining that the state of the anchor frame is out of stock;

[0098] Otherwise, perform commodity detection on the anchor frame image of the anchor frame to obtain a commodity detection frame;

[0099] If the intersection area between the anchor frame and the product detection frame exceeds the intersection threshold, the state of the anchor frame is determined to be in stock, otherwise it is out of stock.

[0100] Specifically, when the feature vector of the anchor frame image of the anchor frame is compared for the first similarity with at least one sample feature vector of the product corresponding to the anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with the sample feature vector in turn, the similarity between the vectors is calculated, and the maximum similarity is taken. If the maximum similarity is greater than the first similarity threshold, the status of the anchor frame is determined to be in stock, otherwise the next step of judgment is performed.

[0101] The next step is to compare empty goods. The empty goods feature vector set can be expressed as F empty ={F empty-1 ,F empty-2 ,F empty-3 ,F empty-4 , ……}, which includes multiple empty goods feature vectors. The feature vectors of the anchor frame image of the anchor frame are respectively compared with the empty goods feature vectors for the second similarity, that is, the similarity between the vectors is calculated, and the maximum similarity is taken. If the maximum similarity is greater than the second similarity threshold, the state of the anchor frame is determined to be out of stock, otherwise the next step of judgment is carried out, that is, product detection.

[0102] Through the above steps, the probability of executing product detection can be greatly reduced. Through the first similarity comparison and the second similarity comparison, a conclusion on whether the product is out of stock can be obtained with a high probability, thereby greatly improving the efficiency of out-of-stock judgment.

[0103] Fig. 9 This is a schematic diagram of the second type of shelf-level out-of-stock judgment logic in an embodiment of the present invention. In one embodiment, the method further includes:

[0104] If the object is a bulk commodity group, commodity detection is performed on the anchor frame image of each anchor frame in the level to be analyzed to obtain multiple commodity detection frames;

[0105] For each anchor frame, if the ratio of the total area of ​​multiple product detection frames to the anchor frame exceeds the area threshold, the state of the anchor frame is determined to be in stock.

[0106] The above embodiment is aimed at the second type of shelf level. The out-of-stock judgment strategy for this type of shelf is: only judge the product detection frame of the product in the anchor frame, take the total area S1 of all product detection frames in the anchor frame, and compare it with the current anchor frame area S2. If the detection result S1 / S2 exceeds the area threshold, it is considered that the current status of the goods does not require replenishment, otherwise it is necessary to report the out-of-stock situation.

[0107] Fig.10 This is a schematic diagram of the third type of shelf-level out-of-stock judgment logic in an embodiment of the present invention. In one embodiment, the method further includes:

[0108] If the object is a boxed commodity, determine the lower edge of each anchor box in the level to be analyzed as the base reference line of the anchor box;

[0109] Perform commodity detection on the anchor frame image of each anchor frame in the analysis level to obtain commodity detection results;

[0110] If the product detection result is that there is no product detection frame, the state of the anchor frame is determined to be out of stock;

[0111] If the product detection result is that there is a product detection frame, determine the product inspection frame closest to the baseline reference line in the product detection frame as the nearest product detection frame;

[0112] If the distance between the nearest product detection frame and the baseline reference line exceeds the distance threshold, the state of the anchor frame is determined to be out of stock; otherwise, the state of the anchor frame is determined to be in stock.

[0113] The above embodiment is for the third type of shelf level, defining the lower edge of the anchor frame as the baseline reference line of the current anchor frame. In the anchor frame, only the distance between the bottom product detection frame and the anchor frame baseline is determined. When the distance exceeds a certain range threshold, it is considered that the outermost product has been taken away and needs to be reported as out of stock. Another situation is that no product is detected in the current anchor frame, in which case the out of stock status is directly reported.

[0114] In step 106, for the out-of-stock anchor frame, the electronic price tag corresponding to the anchor frame is determined according to the extension area of ​​the anchor frame position of the anchor frame;

[0115] In one embodiment, for an out-of-stock anchor frame, determining the electronic price tag corresponding to the anchor frame according to an extension area of ​​the anchor frame position of the anchor frame includes:

[0116] For the out-of-stock anchor frame, the extension area is determined according to the anchor frame position of the anchor frame;

[0117] If there is at least one electronic price tag in the extended area, the at least one electronic price tag is used as the electronic price tag corresponding to the anchor frame;

[0118] If there is no electronic price tag in the extended area, at least one electronic price tag closest to both sides of the extended area is used as the electronic price tag corresponding to the anchor frame.

[0119] Specifically, the binding information of the position of the electronic price tag and the product information (such as sku) is stored in the background server. After the out-of-stock anchor frame (including the anchor frame identifier and the anchor frame position) is determined, the electronic price tag corresponding to the anchor frame is searched for in the downward extended area according to the anchor frame position. According to the product information bound to the electronic price tag, the product information corresponding to the out-of-stock anchor frame is queried from the background server, and the product information corresponding to the out-of-stock anchor frame (including the anchor frame identifier and the anchor frame position) is reported to the merchant, so that the merchant can take the specific product directly to the target location for replenishment. It should be noted that the downward extended area may cover multiple electronic price tags, such as Fig.11 The principle of searching for an electronic price tag through the extended area of ​​the anchor frame position of the anchor frame in the embodiment of the present invention Figure 1 At this time, based on the actual situation, the top three or top two possible product information can be reported, and the merchant can supplement the products.

[0120] It is also possible that the area extending downward from the anchor box does not cover the price tag, such as Fig.12 The principle of searching for an electronic price tag through the anchor frame position extension area of ​​the anchor frame in the embodiment of the present invention Figure 2 , at this time, it is necessary to search from the downward extension area to the left and right for at least one electronic price tag closest to the current extension area, and report at least one product information that may be out of stock. If it is necessary to report a unique out-of-stock sku, you can also compare the center position of the out-of-stock candidate price tag with the center position of the out-of-stock anchor frame, calculate the absolute value of the difference between the x-coordinate of the center of the price tag and the x-coordinate of the center of the anchor frame, select the product information (such as sku) corresponding to the price tag with the smallest absolute value, and report the out-of-stock product information.

[0121] In one embodiment, the method further comprises:

[0122] Obtaining a price tag image of the flashing electronic price tag captured by the visual server, wherein the backend server sends a flashing instruction to the electronic price tag when the binding relationship between the electronic price tag and the product information changes, so that the electronic price tag starts to flash after receiving the flashing instruction;

[0123] Detecting the price tag image to obtain a price tag detection frame position;

[0124] Update the position of the electronic price tag according to the position of the price tag detection frame.

[0125] Specifically, the method of determining the out-of-stock anchor frame and reporting out-of-stock products in the above manner depends on the accuracy of the price tag coordinates. In actual scenarios, price tags are frequently replaced with new products or moved, so it is crucial to develop a price tag position update strategy.

[0126] After determining the initial price tag coordinates, the latest price tag binding relationship is pulled from the price tag backend server at a certain frequency every day. If the binding relationship of a price tag has changed, or a new price tag has been bound, then the positions of these changed price tags need to be tracked. The backend server sends a flash command to the price tag whose binding relationship has changed. After sending the flash command, the camera takes pictures of all electronic price tags in the coverage area to obtain images of all electronic price tags, and then detects the images of the electronic price tags to obtain the price tag detection box position D = {(x1, y1, w1, h1), (x2, y2, w3, h4), ...}, where x i ,y i The pixel coordinate value of the center point of the price tag detection box, w i 、h i is the pixel width and height of the price tag detection frame. After the flash command is issued, the target electronic price tag starts flashing, and the flashing price tag detection frame position is captured from the camera. The price tag position that sends the flash command is determined based on the price tag detection frame position, and the price tag position is updated in the database. In this way, the latest position of the price tag is maintained at a fixed frequency to ensure that the out-of-stock anchor frame can be attributed to find the correct out-of-stock product information.

[0127] In step 107, the product information corresponding to the out-of-stock electronic price tag is reported. Specifically, it can be reported to the backend server for processing by the management personnel. In addition to reporting the out-of-stock product information, the out-of-stock anchor frame, that is, the anchor frame identifier and the anchor frame position can also be reported, so that the merchant can replenish the goods at the correct location.

[0128] The embodiment of the present invention further provides a commodity out-of-stock detection device, the principle of which is similar to that of the commodity out-of-stock detection method, and will not be described in detail here.

[0129] Fig.13 The schematic diagram of the structure of the commodity out-of-stock detection device in the embodiment of the present invention comprises:

[0130] The display image acquisition module 1301 is used to obtain the display images of all objects at each level of the shelf in a fully stocked state;

[0131] Anchor frame detection module 1302, used to determine the type of each level according to the objects in each level of the shelf, and adopt the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level in the full-stock state, and obtain the anchor frame of each object in each level;

[0132] A feature vector obtaining module 1303 is used to obtain, if the object is a single commodity, a set of empty goods feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame according to a plurality of historical display images of each level of the shelf;

[0133] An anchor frame image capture module 1304 is used to obtain the image of the level to be analyzed, adopt the corresponding anchor frame detection strategy, obtain the anchor frame of each object in the level to be analyzed, and capture the anchor frame image of each anchor frame in the level to be analyzed;

[0134] Anchor frame state analysis module 1305, for comparing, for each anchor frame, a feature vector of an anchor frame image of the anchor frame with at least one sample feature vector and a set of out-of-stock feature vectors of the product corresponding to the anchor frame, if the object is a single product, and determining the state of the anchor frame according to the comparison result, the state being out of stock or in stock;

[0135] The electronic price tag determination module 1306 is used to determine, for an out-of-stock anchor frame, the electronic price tag corresponding to the anchor frame according to the extension area of ​​the anchor frame position of the anchor frame;

[0136] The reporting module 1307 is used to report the commodity information corresponding to the out-of-stock electronic price tag.

[0137] In one embodiment, the display image acquisition module is used to:

[0138] Obtain a plurality of historical display images of equal size at each level of the shelf captured by the visual sensor;

[0139] Divide each historical display image into multiple tiles of equal proportions;

[0140] For multiple blocks at the same position in the historical display image, the block with the largest object detection area is selected as the sub-block to be spliced;

[0141] All the sub-image blocks to be stitched together are stitched together to form a display image of all objects at each level in a fully stocked state.

[0142] In one embodiment, the display image acquisition module is used to:

[0143] Obtain a plurality of historical display images of equal size at each level of the shelf captured by the visual sensor;

[0144] The first historical display image is used as a reference image, and detection image recognition is performed on the reference image to obtain multiple detection frames;

[0145] The plurality of detection frames of the reference image are used to form a reference mask image, wherein the areas where there are objects in the reference mask image are marked with a first color, and the areas where there are no objects are marked with a second color;

[0146] Detection image recognition is performed on the historical display images except the benchmark image in turn to obtain multiple candidate detection frames. When the area of ​​the candidate detection frame is larger than a preset proportion of the area of ​​the first color identification part in the benchmark mask image at the corresponding position, the mask image of the candidate detection frame is updated with the benchmark mask image at the corresponding position, and the object frame image detected in the candidate detection frame is pasted at the corresponding position on the benchmark image. After the area judgment of the candidate detection frames of all historical display images is completed, the display images of all objects at each level of the shelf in the full stock state are obtained.

[0147] In one embodiment, the anchor box detection module is used to:

[0148] If the object in each level of the shelf is a single commodity, the level is determined to be the first type, and the anchor frame detection strategy is determined to place the single commodity in an anchor frame for anchor frame detection;

[0149] If the objects in each level of the shelf are a group of bulk commodities, the level is determined to be the second type, and the anchor frame detection strategy is determined to place all commodities that are completely identical and whose distance is less than the first distance into an anchor frame for anchor frame detection;

[0150] If the objects in each level of the shelf are boxed goods, the level is determined to be the third type, and the anchor frame detection strategy is determined to place a group of completely identical boxed goods in an anchor frame for anchor frame detection.

[0151] In one embodiment, the anchor box detection module is used to:

[0152] Perform anchor frame detection on the display images of each level with full goods to obtain the anchor frames for preliminary detection;

[0153] The anchor frame detected by the preliminary detection is quality checked, and the anchor frame detected by the preliminary detection is corrected according to the quality inspection result to obtain the anchor frame of each object. The quality inspection includes the object integrity check in the anchor frame, the image absence check in the anchor frame, and the check whether there is an image without an anchor frame.

[0154] In one embodiment, the feature vector obtaining module is used to:

[0155] Extract at least one sample image within each anchor frame from a plurality of historical display images of each level of the shelf;

[0156] For each sample image, if the sample image does not contain any product, extract the feature vector of the sample image as the empty product feature vector and add it to the empty product feature vector set;

[0157] For each sample image, if the sample image contains a product, the feature vector of the sample image is extracted as a sample feature vector of the product corresponding to the anchor frame.

[0158] In one embodiment, the feature vector obtaining module is further used for:

[0159] After obtaining the empty goods feature vector set and at least one sample feature vector of the goods corresponding to each anchor frame, the method further includes:

[0160] A mapping relationship between an anchor frame and at least one sample feature vector is established, and the mapping relationship and at least one sample feature vector and a set of empty and cargo feature vectors are stored in a feature vector library, wherein the mapping relationship includes an anchor frame identifier, an anchor frame position and at least one sample feature vector.

[0161] In one embodiment, the anchor frame state analysis module is used to:

[0162] For each anchor frame, a first similarity comparison is performed between a feature vector of the anchor frame image of the anchor frame and at least one sample feature vector of the product corresponding to the anchor frame;

[0163] When the maximum similarity of the first similarity comparison result is greater than the first similarity threshold, determining that the state of the anchor frame is in stock;

[0164] Otherwise, a second similarity comparison is performed between the feature vector of the anchor frame image of the anchor frame and each empty cargo feature vector in the empty cargo feature vector set;

[0165] When the maximum similarity of the second similarity comparison result is greater than the second similarity threshold, determining that the state of the anchor frame is out of stock;

[0166] Otherwise, perform commodity detection on the anchor frame image of the anchor frame to obtain a commodity detection frame;

[0167] If the intersection area between the anchor frame and the product detection frame exceeds the intersection threshold, the state of the anchor frame is determined to be in stock, otherwise it is out of stock.

[0168] In one embodiment, the anchor frame state analysis module is used to:

[0169] If the object is a bulk commodity group, commodity detection is performed on the anchor frame image of each anchor frame in the level to be analyzed to obtain multiple commodity detection frames;

[0170] For each anchor frame, if the ratio of the total area of ​​multiple product detection frames to the anchor frame exceeds the area threshold, the state of the anchor frame is determined to be in stock.

[0171] In one embodiment, the anchor frame state analysis module is used to:

[0172] If the object is a boxed commodity, determine the lower edge of each anchor box in the level to be analyzed as the base reference line of the anchor box;

[0173] Perform commodity detection on the anchor frame image of each anchor frame in the analysis level to obtain commodity detection results;

[0174] If the product detection result is that there is no product detection frame, the state of the anchor frame is determined to be out of stock;

[0175] If the product detection result is that there is a product detection frame, determine the product inspection frame closest to the baseline reference line in the product detection frame as the nearest product detection frame;

[0176] If the distance between the nearest product detection frame and the baseline reference line exceeds the distance threshold, the state of the anchor frame is determined to be out of stock; otherwise, the state of the anchor frame is determined to be in stock.

[0177] In one embodiment, the electronic price label determination module is used to:

[0178] For the out-of-stock anchor frame, the extension area is determined according to the anchor frame position of the anchor frame;

[0179] If there is at least one electronic price tag in the extended area, the at least one electronic price tag is used as the electronic price tag corresponding to the anchor frame;

[0180] If there is no electronic price tag in the extended area, at least one electronic price tag closest to both sides of the extended area is used as the electronic price tag corresponding to the anchor frame.

[0181] In one embodiment, the device further includes a price tag position updating module, which is used to:

[0182] Obtaining a price tag image of the flashing electronic price tag captured by the visual server, wherein the backend server sends a flashing instruction to the electronic price tag when the binding relationship between the electronic price tag and the product information changes, so that the electronic price tag starts to flash after receiving the flashing instruction;

[0183] Detecting the price tag image to obtain a price tag detection frame position;

[0184] Update the position of the electronic price tag according to the position of the price tag detection frame.

[0185] The embodiment of the present invention also provides a commodity out-of-stock detection system, such as Fig.14 14 is a schematic diagram of the structure of a commodity out-of-stock detection system in an embodiment of the present invention, wherein the system comprises: a visual sensor 1401 and the above-mentioned commodity out-of-stock detection device 1402;

[0186] The visual sensor 1401 is used to capture display images of all objects at each level of the shelf in a fully stocked state, as well as images of the level to be analyzed, and send them to the commodity out-of-stock detection device.

[0187] The visual sensor may be a device capable of taking pictures, such as a camera. The visual sensor also captures the flashing price tag. The backend management system includes a feature vector library for storing the mapping relationship and at least one sample feature vector and an empty goods feature vector set.

[0188] The commodity out-of-stock detection system proposed in the embodiment of the present invention interacts with the background management system. The visual sensor can capture the image of the flashing price tag and store the current captured image in the background management system. The captured price tag position is uploaded to the background management module for unified update of the price tag position.

[0189] The main functions of the backend management system are: to pull the latest price tag product binding relationship, obtain the changed price tag representation, and issue the flashing light command of the specified price tag; to configure and modify the anchor frame position; to visually display the out-of-stock judgment results; to receive the price tag position information from the camera module, and to maintain the latest price tag position relationship, etc.

[0190] In summary, the method, device, and system proposed in the embodiments of the present invention have the following beneficial effects:

[0191] Obtain the display images of all objects in each level of the shelf in a full-stock state; determine the type of each level according to the objects in each level of the shelf, and adopt the corresponding anchor frame detection strategy to perform anchor frame detection on the display images in the full-stock state of each level to obtain the anchor frame of each object in each level; if the object is a single commodity, obtain a set of empty-stock feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame according to multiple historical display images of each level of the shelf; after obtaining the image of the level to be analyzed, adopt the corresponding anchor frame detection strategy to obtain the anchor frame of each object in the level to be analyzed, and intercept the anchor frame image of each anchor frame in the level to be analyzed; if the object is a single commodity, for each anchor frame, compare the feature vector of the anchor frame image of the anchor frame with at least one sample feature vector of the commodity corresponding to the anchor frame and the set of empty-stock feature vectors, and determine the state of the anchor frame according to the comparison result, which is out of stock or in stock; for the out-of-stock anchor frame, determine the electronic price tag corresponding to the anchor frame according to the extended area of ​​the anchor frame position of the anchor frame; and report the commodity information corresponding to the out-of-stock electronic price tag. Through the above process, the shelf display information of the full stock state is restored through the historical display information. The corresponding anchor frame detection strategy can be determined according to the type of level, making the anchor frame detection more accurate. For the object of a single commodity, a set of empty stock feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame can be obtained, which are then compared with the feature vectors of the anchor frame image of each anchor frame in the level to be analyzed to give an accurate out-of-stock judgment result. After determining the position of the out-of-stock anchor frame, one or more price tags in the area extending downward from the anchor frame are used to search for out-of-stock commodity information. This process is efficient.

[0192] An embodiment of the present invention further provides a computer device, Fig.1515 is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 1500 includes a memory 1510, a processor 1520, and a computer program 1530 stored in the memory 1510 and executable on the processor 1520. When the processor 1520 executes the computer program 1530, the above-mentioned commodity out-of-stock detection method is implemented.

[0193] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned commodity out-of-stock detection method is implemented.

[0194] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned commodity out-of-stock detection method is implemented.

[0195] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0199] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting a commodity out of stock, characterized in that: include: Obtain display images of all objects at each shelf level in a fully stocked state; According to the objects in each level of the shelf, the type of each level is determined, and the corresponding anchor frame detection strategy is adopted to perform anchor frame detection on the display images of each level in the full state to obtain the anchor frame of each object in each level; If the object is a single commodity, a set of empty goods feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame are obtained according to a plurality of historical display images of each level of the shelf; After obtaining the image of the level to be analyzed, the corresponding anchor frame detection strategy is adopted to obtain the anchor frame of each object in the level to be analyzed, and the anchor frame image of each anchor frame in the level to be analyzed is intercepted; If the object is a single product, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and a set of out-of-stock feature vectors of the product corresponding to the anchor frame, and the state of the anchor frame is determined according to the comparison result, and the state is out of stock or in stock; For an out-of-stock anchor frame, determine the electronic price tag corresponding to the anchor frame according to the extended area of ​​the anchor frame position of the anchor frame; Report the product information corresponding to the out-of-stock electronic price tag; Determine the type of each level based on the objects in each level of the shelf, including: If the object in each level of the shelf is a single commodity, the level is determined to be the first type, and the anchor frame detection strategy is determined to place the single commodity in an anchor frame for anchor frame detection; If the objects in each level of the shelf are a group of bulk commodities, the level is determined to be the second type, and the anchor frame detection strategy is determined to place all commodities that are completely identical and whose distance is less than the first distance into an anchor frame for anchor frame detection; If the objects in each level of the shelf are boxed goods, the level is determined to be the third type, and the anchor frame detection strategy is determined to place a group of completely identical boxed goods in an anchor frame for anchor frame detection.

2. The method according to claim 1, characterized in that Get the display images of all objects at each shelf level in the fully stocked state, including: Obtain a plurality of historical display images of equal size at each level of the shelf captured by the visual sensor; Divide each historical display image into multiple tiles of equal proportions; For multiple blocks at the same position in the historical display image, the block with the largest object detection area is selected as the sub-block to be spliced; All the sub-image blocks to be stitched together are stitched together to form a display image of all objects at each level in a fully stocked state.

3. The method according to claim 1, characterized in that Get the display images of all objects at each shelf level in the fully stocked state, including: Obtain a plurality of historical display images of equal size at each level of the shelf captured by the visual sensor; The first historical display image is used as a reference image, and detection image recognition is performed on the reference image to obtain multiple detection frames; The plurality of detection frames of the reference image are used to form a reference mask image, wherein the areas where there are objects in the reference mask image are marked with a first color, and the areas where there are no objects are marked with a second color; Detection image recognition is performed on the historical display images except the benchmark image in turn to obtain multiple candidate detection frames. When the area of ​​the candidate detection frame is larger than a preset proportion of the area of ​​the first color identification part in the benchmark mask image at the corresponding position, the mask image of the candidate detection frame is updated with the benchmark mask image at the corresponding position, and the object frame image detected in the candidate detection frame is pasted at the corresponding position on the benchmark image. After the area judgment of the candidate detection frames of all historical display images is completed, the display images of all objects at each level of the shelf in the full stock state are obtained.

4. The method according to claim 1, characterized in that Perform anchor frame detection on the display images of each level in the full-stock state to obtain the anchor frames of each object at each level, including: Perform anchor frame detection on the display images of each level with full goods to obtain the anchor frames for preliminary detection; The anchor frame detected by the preliminary detection is quality checked, and the anchor frame detected by the preliminary detection is corrected according to the quality inspection result to obtain the anchor frame of each object. The quality inspection includes checking the integrity of the object in the anchor frame, checking whether there is an image without an anchor frame in the anchor frame, and checking whether there is an image without an anchor frame.

5. The method according to claim 1, characterized in that According to multiple historical display images of each level of the shelf, a set of empty goods feature vectors and at least one sample feature vector of the goods corresponding to each anchor frame are obtained, including: Extract at least one sample image within each anchor frame from a plurality of historical display images of each level of the shelf; For each sample image, if the sample image does not contain any product, extract the feature vector of the sample image as the empty product feature vector and add it to the empty product feature vector set; For each sample image, if the sample image contains a product, the feature vector of the sample image is extracted as a sample feature vector of the product corresponding to the anchor frame.

6. The method according to claim 1, characterized in that After obtaining the empty goods feature vector set and at least one sample feature vector of the goods corresponding to each anchor frame, the method further includes: A mapping relationship between an anchor frame and at least one sample feature vector is established, and the mapping relationship and at least one sample feature vector and a set of empty and cargo feature vectors are stored in a feature vector library, wherein the mapping relationship includes an anchor frame identifier, an anchor frame position and at least one sample feature vector.

7. The method according to claim 1, characterized in that For each anchor frame, a feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and a set of empty feature vectors of the commodity corresponding to the anchor frame, and a state of the anchor frame is determined according to the comparison result, including: For each anchor frame, a first similarity comparison is performed between a feature vector of the anchor frame image of the anchor frame and at least one sample feature vector of the product corresponding to the anchor frame; When the maximum similarity of the first similarity comparison result is greater than the first similarity threshold, determining that the state of the anchor frame is in stock; Otherwise, a second similarity comparison is performed between the feature vector of the anchor frame image of the anchor frame and each empty cargo feature vector in the empty cargo feature vector set; When the maximum similarity of the second similarity comparison result is greater than the second similarity threshold, determining that the state of the anchor frame is out of stock; Otherwise, perform commodity detection on the anchor frame image of the anchor frame to obtain a commodity detection frame; If the intersection area between the anchor frame and the product detection frame exceeds the intersection threshold, the state of the anchor frame is determined to be in stock, otherwise it is out of stock.

8. The method according to claim 1, characterized in that Also includes: If the object is a bulk commodity group, commodity detection is performed on the anchor frame image of each anchor frame in the level to be analyzed to obtain multiple commodity detection frames; For each anchor frame, if the ratio of the total area of ​​multiple product detection frames to the anchor frame exceeds the area threshold, the state of the anchor frame is determined to be in stock.

9. The method according to claim 1, characterized in that Also includes: If the object is a boxed commodity, determine the lower edge of each anchor box in the level to be analyzed as the base reference line of the anchor box; Perform commodity detection on the anchor frame image of each anchor frame in the analysis level to obtain commodity detection results; If the product detection result is that there is no product detection frame, the state of the anchor frame is determined to be out of stock; If the product detection result is that there is a product detection frame, determine the product inspection frame closest to the baseline reference line in the product detection frame as the nearest product detection frame; If the distance between the nearest product detection frame and the baseline reference line exceeds the distance threshold, the state of the anchor frame is determined to be out of stock; otherwise, the state of the anchor frame is determined to be in stock.

10. The method according to claim 1, characterized in that For an out-of-stock anchor frame, the electronic price tag corresponding to the anchor frame is determined according to the extended area of ​​the anchor frame position of the anchor frame, including: For the out-of-stock anchor frame, the extension area is determined according to the anchor frame position of the anchor frame; If there is at least one electronic price tag in the extended area, the at least one electronic price tag is used as the electronic price tag corresponding to the anchor frame; If there is no electronic price tag in the extended area, at least one electronic price tag closest to both sides of the extended area is used as the electronic price tag corresponding to the anchor frame.

11. The method according to claim 1, characterized in that Also includes: Obtaining a price tag image of the flashing electronic price tag captured by the visual server, wherein the backend server sends a flashing instruction to the electronic price tag when the binding relationship between the electronic price tag and the product information changes, so that the electronic price tag starts to flash after receiving the flashing instruction; Detecting the price tag image to obtain a price tag detection frame position; Update the position of the electronic price tag according to the position of the price tag detection frame.

12. A commodity out-of-stock detection device, characterized in that: include: A display image acquisition module is used to obtain display images of all objects at each level of the shelf in a fully stocked state; The anchor frame detection module is used to determine the type of each shelf level according to the objects in each shelf level, and adopt the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each shelf level in the full state, and obtain the anchor frame of each object in each shelf level; A feature vector acquisition module, for obtaining, if the object is a single commodity, a set of empty goods feature vectors and at least one sample feature vector of the commodity corresponding to each anchor frame according to a plurality of historical display images of each level of the shelf; An anchor frame image capture module is used to obtain the image of the level to be analyzed, adopt the corresponding anchor frame detection strategy, obtain the anchor frame of each object in the level to be analyzed, and capture the anchor frame image of each anchor frame in the level to be analyzed; An anchor frame state analysis module, for comparing, for each anchor frame, a feature vector of an anchor frame image of the anchor frame with at least one sample feature vector and a set of out-of-stock feature vectors of the product corresponding to the anchor frame, if the object is a single product, and determining the state of the anchor frame according to the comparison result, the state being out of stock or in stock; An electronic price tag determination module is used to determine, for an out-of-stock anchor frame, the electronic price tag corresponding to the anchor frame according to an extension area of ​​the anchor frame position of the anchor frame; A reporting module is used to report the product information corresponding to the out-of-stock electronic price tags; Determine the type of each level based on the objects in each level of the shelf, including: If the object in each level of the shelf is a single commodity, the level is determined to be the first type, and the anchor frame detection strategy is determined to place the single commodity in an anchor frame for anchor frame detection; If the objects in each level of the shelf are a group of bulk commodities, the level is determined to be the second type, and the anchor frame detection strategy is determined to place all commodities that are completely identical and whose distance is less than the first distance into an anchor frame for anchor frame detection; If the objects in each level of the shelf are boxed goods, the level is determined to be the third type, and the anchor frame detection strategy is determined to place a group of completely identical boxed goods in an anchor frame for anchor frame detection.

13. A commodity out-of-stock detection system, characterized in that: include: A visual sensor and a commodity out-of-stock detection device as claimed in claim 12; The visual sensor is used to capture display images of all objects at each level of the shelf in a fully stocked state, as well as images of the level to be analyzed, and send them to the commodity out-of-stock detection device.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

16. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images