Method and device for detecting a merchandise restocking situation

Abstract

This invention discloses a method and apparatus for detecting product replenishment status, relating to the field of computer vision. The method includes the following steps: First, acquiring a first image and a second image of a retail cabinet; segmenting and reducing the weight of empty product aisles detected in the image before replenishment; calculating the ratio of the area of ​​empty product locations to the area of ​​the product aisles to determine N first product aisle information for the N product aisles included in the retail cabinet before replenishment; and determining N second product aisle information for the N product aisles included in the retail cabinet after replenishment; finally, determining the product replenishment status by obtaining N first information based on the N first and N second product aisle information. This invention addresses the product replenishment task of retail cabinets in new retail scenarios, accurately identifying whether retail cabinets are being replenished correctly, improving the accuracy of replenishment judgment, and enhancing review efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a method and apparatus for detecting product replenishment status. Background Technology

[0002] In recent years, with the iterative upgrades and innovations of artificial intelligence technology, revitalizing traditional industries through technological empowerment has become a development trend. Taking the retail industry, which is most frequently encountered in daily life, as an example, the empowerment of artificial intelligence technology has significantly changed consumer behavior patterns and greatly improved the user experience. With the mainstream consumer group becoming increasingly younger, consumers' demand for convenience is becoming stronger, and smart retail kiosks, as a major carrier of new retail, are a crucial future trend.

[0003] Current methods for detecting the replenishment status of retail lockers often rely on full manual review. However, as the scale of smart retail lockers grows, accurately determining whether a locker has been successfully replenished, improving the accuracy of replenishment judgments, and enhancing review efficiency have become challenges.

[0004] Therefore, determining whether retail cabinets are being replenished correctly is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This invention provides a method and apparatus for detecting product replenishment status, used to accurately identify whether a retail counter has been correctly replenished.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting product replenishment, comprising: acquiring a first image and a second image, wherein the first image is an image of a retail cabinet before replenishment, and the second image is an image of the retail cabinet after replenishment; determining, based on the first image, N first aisle information of N aisles included in the retail cabinet before replenishment, wherein the first aisle information indicates the proportion of the area of ​​empty product positions in the aisles to the area of ​​the aisles before replenishment; N is an integer greater than 0; and determining, based on the second image, N first aisle information of the N aisles included in the retail cabinet after replenishment. The second channel information indicates the proportion of the area of ​​the empty product location in the channel after replenishment to the channel area; based on the N first channel information and the N second channel information, N first information are obtained; if all N first information are greater than or equal to a preset threshold, it is determined that the replenishment of the retail cabinet is successful; or, if one or more of the N first information are less than the preset threshold, it is determined that the channel that obtained the first information failed to replenish the retail cabinet, and a first replenishment alarm is sent, the first replenishment alarm being used to indicate replenishment failure.

[0007] In the above scheme, by acquiring images of the retail cabinet before and after replenishment, and obtaining the area ratio of empty product positions in N aisles before and after replenishment, it is determined whether the retail cabinet is replenished correctly, and a replenishment alarm is sent if the replenishment is not performed correctly. Through the above method, this embodiment of the invention provides a method for detecting the replenishment status of retail cabinet products, which can accurately identify whether the retail cabinet is replenished correctly, improve the accuracy and efficiency of retail cabinet replenishment detection, and reduce the time cost of full manual review.

[0008] In one possible implementation, based on the second image, N third-channel information of the N channels included in the replenished retail cabinet is determined, wherein the third-channel information indicates the product type in the channel after replenishment; based on the N third-channel information, the quantity of goods added to each of the M product types in the N channels is determined; M is an integer greater than 0; for any of the M product types, if the quantity of goods added to the product type is less than the preset quantity of goods for the product type, it is determined that replenishment of the product type has failed, and a second replenishment alarm is sent, the second replenishment alarm indicating that replenishment of the product type has failed; or, if the quantity of goods added to the product type is greater than or equal to the preset quantity of goods for the product type, it is determined that replenishment of the product type has succeeded.

[0009] This solution determines whether a retail cabinet needs restocking by tracking changes in the quantity of goods before and after restocking. This helps to identify if the type of goods being restocked is correct, preventing errors caused by discrepancies between the required and actual restocked goods. This improves the accuracy of determining whether restocking was successful.

[0010] In one possible implementation, for any of the N first cargo channel information, the ratio m1 of the area of ​​the empty product location in the cargo channel before replenishment, as indicated by the first cargo channel information, satisfies the following form: Wherein, S1 represents the sum of the areas of the product positions and the empty product positions in the aisle, S2 represents the area of ​​the empty product positions in the aisle, and S3 represents the area of ​​the aisle.

[0011] In one possible implementation, for any of the N second cargo channel information, the ratio m2 of the area of ​​the empty product location in the cargo channel after replenishment, as indicated by the second cargo channel information, satisfies the following form: Wherein, S4 represents the sum of the areas of the product positions and the empty product positions in the channel, S5 represents the area of ​​the empty product positions in the channel, and S6 represents the area of ​​the channel.

[0012] By using the above method to calculate the proportion of empty product locations in the aisles before and after replenishment, the degree of product shortage in the aisles can be obtained more accurately. By calculating the proportion of empty product locations, the error in aisle area caused by factors such as the angle of the photograph can also be avoided when comparing the increase in product value before and after replenishment.

[0013] In one possible implementation, obtaining N pieces of first information based on the N first aisle information and the N second aisle information includes: subtracting the ratio m1 of the empty product positions to the aisle area in the N first aisle information from the ratio m2 of the empty product positions to the aisle area in the N second aisle information to obtain N pieces of first information; wherein, each piece of first information is obtained by subtracting the first aisle information corresponding to one of the N aisles from the second aisle information corresponding to that aisle; the first information represents the percentage increase in the number of products in the aisle before and after restocking the retail counter.

[0014] By using the above method, N first pieces of information representing the increase in the number of goods before and after replenishment are obtained based on the difference in the proportion of empty positions of goods in the aisle area before and after replenishment. The degree of replenishment for each aisle of the retail cabinet can be judged more intuitively based on the size of the first pieces of information, and at the same time, the accuracy of judging the degree of replenishment of the retail cabinet is improved.

[0015] Secondly, embodiments of the present invention provide a device for detecting product replenishment, comprising: an acquisition module, configured to acquire a first image and a second image, wherein the first image is an image of the retail cabinet before replenishment, and the second image is an image of the retail cabinet after replenishment; and a processing module, configured to, based on the first image, determine N first channel information of N channels in the retail cabinet before replenishment, wherein the first channel information indicates the proportion of the area of ​​empty product positions in the channels to the area of ​​the channels before replenishment; N is an integer greater than 0; and, based on the second image, determine N first channel information of the N channels in the retail cabinet after replenishment. The system includes two product lane information components, where the second product lane information indicates the proportion of the area of ​​empty product locations in the product lane after replenishment to the total area of ​​the product lane. Based on the N first product lane information components and the N second product lane information components, N first information components are obtained. A detection module is used to detect whether the retail cabinet has been successfully replenished. If all N first information components are greater than or equal to a preset threshold, the retail cabinet is determined to have been successfully replenished. Alternatively, if one or more of the N first information components are less than the preset threshold, the product lane that obtained the first information component is determined to have failed to replenish the retail cabinet, and a first replenishment alarm is sent, indicating that the replenishment has failed.

[0016] In one possible implementation, the processing module is further configured to determine, based on the second image, N third-channel information of the N channels included in the replenished retail cabinet, wherein the third-channel information indicates the type of goods in the channels after replenishment.

[0017] In one possible implementation, the detection module is further configured to: if the quantity of goods added to any of the M product types is less than a preset quantity, determine that replenishment of the product type has failed and send a second replenishment alarm, the second replenishment alarm indicating that replenishment of the product type has failed; or, if the quantity of goods added to the product type is greater than or equal to the preset quantity, determine that replenishment of the product type has succeeded.

[0018] Thirdly, an apparatus is provided, comprising a unit or module corresponding to performing the method described in any of the first aspects above, wherein the unit or module may be implemented by hardware circuitry, by software, or by a combination of hardware circuitry and software.

[0019] In one alternative embodiment, the apparatus includes an acquisition unit, a processing unit, and a detection unit, wherein the processing unit is configured to execute a program or instructions to enable the apparatus to perform the method described in any of the first aspects above.

[0020] Fourthly, an apparatus is provided, including a processor and interface circuitry, the processor being configured to perform the methods described in any of the first aspects above. The processor may include one or more processors.

[0021] Fifthly, an apparatus is provided, comprising a processor coupled to a memory for executing a program stored in the memory to perform the methods described in any of the first aspects above. The memory may be located within or outside the apparatus. Furthermore, there may be one or more processors.

[0022] A sixth aspect provides an apparatus including a processor and a memory; the memory is used to store computer instructions, and when the apparatus is in operation, the processor executes the computer instructions stored in the memory to cause the apparatus to perform the methods described in any of the first aspects above.

[0023] A seventh aspect provides a chip system comprising: a processor or circuitry for performing the method described in any of the first aspects above.

[0024] Eighthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a communication device, cause the methods described in any of the first aspects above to be performed.

[0025] Ninth aspect, a computer program product is provided, the computer program product including a computer program or instructions that, when the computer program or instructions are run by a device, cause the methods described in any of the first aspects above to be performed. Attached Figure Description

[0026] Figure 1 A schematic diagram of a retail cabinet provided for an embodiment of the present invention;

[0027] Figure 2 A schematic flowchart of a product replenishment detection method provided in an embodiment of the present invention;

[0028] Figure 3 A schematic flowchart of a product replenishment detection method provided in an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of a product replenishment detection device provided in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of a product replenishment detection device provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0032] Figure 1 This is a schematic diagram of a retail cabinet according to an embodiment of the present invention, such as... Figure 1 As shown, Figure 1 The images show the retail cabinet before and after restocking, with 1A representing the first image and 1B representing the second image. The retail cabinet images also include shelf layer 101, merchandise 102, and aisle 103.

[0033] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Furthermore, the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0034] based on Figure 1 The retail cabinet image shown Figure 2 An exemplary flowchart of an embodiment of the present invention is shown, which can be executed by a terminal device.

[0035] like Figure 2 As shown, the process of this method may include:

[0036] Step 201: Obtain the first image and the second image;

[0037] The first image is of the retail cabinet before restocking, and the second image is of the retail cabinet after restocking.

[0038] In one implementation, the retail cabinet can be a smart retail cabinet, a vending machine, an unmanned retail cabinet, or other types of retail cabinets. This embodiment of the invention does not limit the specific type of retail cabinet.

[0039] In one embodiment, the first image and the second image may be a front image of the retail cabinet taken by a restocking clerk, a front image of the retail cabinet taken by a camera installed inside the retail cabinet, or an image of the retail cabinet's merchandise from other angles taken by other means. This embodiment of the invention does not limit the scope of the images.

[0040] Step 202: Based on the first image, determine the information of the N first channels of the N channels included in the retail cabinet before replenishment; based on the second image, determine the information of the N second channels of the N channels included in the retail cabinet after replenishment.

[0041] Among them, N channels correspond one-to-one with N first channel information. The first channel information corresponding to a channel indicates the proportion of the area of ​​the empty position of goods in that channel to the area of ​​the channel before replenishment; N is an integer greater than 0.

[0042] Among them, N cargo channels correspond one-to-one with N second cargo channel information. The second cargo channel information corresponding to a cargo channel indicates the proportion of the empty space of goods in that cargo channel to the cargo channel area after replenishment.

[0043] For example, based on the first image, the total area of ​​the retail cabinet aisles before replenishment is obtained, and a list of aisles with product shortages before replenishment is obtained. The list of aisles is divided according to the width of a single aisle to obtain N aisles with product shortages.

[0044] Based on the second image, the aisle range is detected in the first and second images. The horizontal proportion of N aisles in the first image on their respective shelves is obtained, and the corresponding N aisles in the second image after replenishment are determined according to the proportion value.

[0045] In one implementation, the ratio m1 of the area of ​​empty product locations in the first cargo lane indicated by the first cargo lane information to the area of ​​the cargo lane before replenishment satisfies the following form: Where S1 represents the sum of the areas of the goods positions and the empty positions in the aisle, S2 represents the area of ​​the empty positions in the aisle, and S3 represents the area of ​​the aisle.

[0046] In one implementation, m1 can be used to represent the degree of vacancy of goods in the aisle. For example, the larger the value of m1, the more vacant the corresponding aisle is.

[0047] Specifically, the method for obtaining the area S1 of the product position and the empty product position in the channel can include filling the empty product area in each of the N channels respectively, and obtaining the channel area S1 after filling.

[0048] The method of obtaining the area S2 of the empty position of goods in the channel can include taking the union of the empty positions of goods in each of the N channels and obtaining the empty position area S2 of goods after taking the union.

[0049] The area S3 of the aisle can be obtained by taking the union of the filled aisle area S1 and the area S2 of the empty product position, and then obtaining the aisle area S3 after taking the union. It should be noted that the above method is only an example, and other methods can also be used to obtain the above area. This embodiment of the invention does not limit this.

[0050] In one implementation, for any of the N second cargo channel information, the ratio m2 of the area of ​​the empty space of goods in the cargo channel after replenishment, as indicated by the second cargo channel information, satisfies the following form: Where S4 represents the sum of the areas of the goods positions and the empty positions in the aisle, S5 represents the area of ​​the empty positions in the aisle, and S6 represents the area of ​​the aisle.

[0051] In one implementation, m2 can be used to represent the degree of vacancy of goods in the aisle. For example, the larger the value of m2, the more vacant the corresponding aisle is.

[0052] Specifically, the sum of the areas of the goods positions and empty positions in the aisle, S4, the area of ​​the empty positions in the aisle, S5, and the area of ​​the aisle, S6, are obtained in the same way as the areas S1, S2, and S3 mentioned above. It should be noted that the above method of obtaining the area is only an example, and other methods can also be used to obtain the above area. This embodiment of the invention does not limit this method.

[0053] Step 203: Obtain N first information items based on N first cargo channel information items and N second cargo channel information items.

[0054] For example, subtracting N first channel information from N second channel information yields N first information; where each first information is obtained by subtracting the first channel information corresponding to one of the N channels from the second channel information corresponding to that channel.

[0055] For a given first piece of information, the first cargo lane information and the second cargo lane information used to determine the first piece of information correspond to the same cargo lane.

[0056] Step 204: If all N first pieces of information are greater than or equal to the preset threshold, then the restocking of the retail cabinet is confirmed to be successful.

[0057] Step 205: If one or more of the N first pieces of information are less than a preset threshold, it is determined that the channel that obtained the first information failed to replenish the retail cabinet, and a first replenishment alarm is sent.

[0058] The first replenishment alarm is used to indicate that replenishment has failed.

[0059] The first replenishment alarm can be sent directly to the terminal device of the replenishment clerk who replenishes the retail counter, or it can be sent to the terminal device of the administrator who manages the retail counter. This application does not limit this.

[0060] In one implementation, the proportion m1 of the empty product locations in the N first aisle information items to the aisle area is subtracted from the proportion m2 of the empty product locations in the N second aisle information items to obtain N first information items; the first information items represent the proportion of product increase before and after restocking the retail counter.

[0061] Specifically, subtracting N second-order information from N first-order information results in N first-order information values. For each of the N first-order information values, its value is compared with a preset threshold. If the value of each of the N first-order information values ​​is greater than or equal to the preset threshold, the retail cabinet is successfully replenished. If one or more of the N first-order information values ​​are less than the preset threshold, the replenishment of the order in which the first-order information was obtained fails, and a replenishment alarm is sent.

[0062] In one implementation, if restocking of a retail cabinet fails, a first restocking alarm is sent. The alarm includes information about the storage channel where restocking failed, corresponding to the first piece of information. The administrator reviews the alarm and issues a restocking instruction. It should be noted that the above method is merely an example, and this embodiment of the invention does not limit the scope of the invention.

[0063] In this application, the third cargo channel information after replenishment can also be determined based on the second image, wherein the third cargo channel information indicates the product type in the cargo channel after replenishment; based on the N third cargo channel information, the quantity of each product type added in each of the M product types in the N cargo channels is determined, and compared with the preset quantity of the product type to determine whether the replenishment of the product is successful; M is an integer greater than 0;

[0064] In one implementation, for any of the M product types, if the quantity of the product corresponding to that product type increases is less than the preset quantity of that product type, then it is determined that the replenishment of that product type has failed, and a second replenishment alarm is sent. The second replenishment alarm is used to indicate that the replenishment of that product type to the retail counter has failed. The second replenishment alarm can be sent directly to the terminal device of the replenishment staff who replenished the retail counter, or it can be sent to the terminal device of the administrator who manages the retail counter. This application does not limit this.

[0065] Alternatively, if the quantity of goods added for a product type is greater than or equal to the preset quantity for that product type, then the replenishment of that product type is considered successful. In one implementation, if replenishment of the retail counter fails, the failure information is recorded in the work quality report of the corresponding replenishment personnel. The work quality reports of replenishment personnel are reviewed periodically. If the work quality issues of a replenishment personnel exceed a certain threshold within a certain period, a work quality alarm is issued to that personnel for the administrator to review and confirm, preventing replenishment personnel from improperly placing goods or damaging goods.

[0066] By using the above process as a dual criterion—the change in the area of ​​empty product locations in N aisles before and after replenishment, and the increase in the number of products in N aisles before and after replenishment—it is possible to not only accurately determine whether each shelf of the retail counter needs to be replenished, but also to determine whether the type of product being replenished is correct based on the increase in the quantity of each product. This improves the accuracy and efficiency of determining the replenishment status of the retail counter.

[0067] In specific implementation, it can be done according to the following: Figure 3 The process shown is for detecting product replenishment status, such as... Figure 3 The process shown is based on Figure 2 The flowchart shown is a specific embodiment. It should be noted that the flowchart is merely an example, and other methods can be used to implement it in practice; this embodiment of the invention does not limit the scope of the invention. Figure 3 The specific implementation process shown includes the following steps:

[0068] In one feasible approach, such as Figure 3As shown, the process includes the following steps:

[0069] S1: Acquire a first image and a second image of the retail cabinet, and perform shelf detection on the first and second images to obtain the range of each shelf in the retail cabinet and information on the locations of empty product slots. Perform product detection on the first and second images to obtain product location information and product category information on the shelves. Based on the obtained range of each shelf in the retail cabinet, arrange the obtained information on empty product slots, product location information, and product category information to obtain the retail cabinet information corresponding to the first and second images of the retail cabinet.

[0070] It should be noted that there may be problems with poor image quality between the first and second images, such as fog, blur, specular reflection, etc. There may also be problems such as incorrect shooting angle or other issues. Therefore, in one embodiment, the present invention will discard some images with extremely poor quality.

[0071] In one embodiment, the present invention first uses a glass detection model to detect glass regions in a first image and a second image. Then, a Huffman line detection model is used to obtain the pose of the retail cabinet from the detected glass regions. Based on the obtained retail cabinet pose estimation results, incomplete retail cabinet images and images with incorrect shooting angles are removed. It should be noted that the above-mentioned removal of extremely poor-quality images is only an example; other methods can also be used to remove poor-quality images, or no images may be removed at all. The embodiments of the present invention do not limit this. The glass detection model can be a glass image recognition algorithm, and the specific implementation method is not limited in this application.

[0072] In one implementation, the specific steps of S1 above include:

[0073] S11: Using an object detection model, acquire the first and second images of the retail cabinet, and perform shelf detection on the first and second images to obtain the range of each shelf in the retail cabinet and obtain information on the locations where goods are missing; specifically, the object detection model refers to a model that can separate the required target (such as shelf layer, aisle and goods, etc.) from other content in the image, identify object entities from the image and extract different image features.

[0074] S12: Using an object detection model, detect the location and category information of goods on the shelves in the first and second images;

[0075] S13: Based on the detection results in S11, calculate the vertical overlap rate between the detected range of each shelf in the retail cabinet and the information on the location of any empty product. If the overlap rate exceeds a certain threshold, determine that the empty product location is in the corresponding shelf, and map the empty product location information to that shelf layer in order from left to right. Perform the same operation on the product detection results to map the product location and product type information to the corresponding shelf layer.

[0076] In one implementation, the above detections all employ a target detection model, where the object categories are divided into three types: "shelf layer", "shopping aisle", and "shelf with empty product slots". The results obtained include category information and four coordinates representing the location.

[0077] In one implementation, the target detection model for the above-mentioned product detection uses a common target detection algorithm for product location detection. The object category is set to two classes, the image size is set to 960, and the result includes four coordinates representing the product's location. Based on product detection, the product category recognition uses a ReID model to identify the specific product category of the detected object, presenting it numerically. The final result is mapped to textual information representing the product category.

[0078] Through the above steps S11 to S13, the empty positions of goods on each shelf of the retail cabinet, as well as the goods location information and goods category information, are first obtained. Then, the above results are sorted in order from left to right according to the horizontal position to obtain the corresponding position information of each shelf.

[0079] S2: Segment and reduce the weight of the information on the detected empty product locations in the first image to obtain N product lanes with empty product locations in the first image. Then, calculate the proportion of the empty product location area to the total area of ​​the product lane in each of the N product lanes to obtain N first product lane information.

[0080] In one implementation, the specific steps of S2 above include:

[0081] S21: Based on the range information of each shelf and the product location information obtained in S1, estimate the approximate width of a single aisle for placing a column of products. Divide the areas in the first image where products are detected as empty according to the estimated width of a single aisle. Specifically, when dividing the aisles, detect the presence of product edges within a certain range based on the product location information detection results. If product edges exist within a certain threshold range, divide the aisles according to the product edges. If no product edges exist, divide the aisles according to the estimated width of a single aisle, resulting in a list of N divided aisles.

[0082] S22: Initialize one of the N channels after division. Traverse the N channels. If a channel has a missing product position, calculate the overlap area with the other channels in the N channels one by one. If the overlap area does not exceed the overlap threshold, add the channel to the N channels. If the overlap area exceeds the threshold, it is determined that it has been recorded in the N channels. Perform the above operation on all channels of the N channels after division until the N channels are de-weighted and a list of N channels without overlap is obtained.

[0083] S23: Fill the empty positions of the goods in the N channels after the weight reduction and segmentation to obtain the sum of the areas of the goods positions and the empty positions in the channels, S1. Take the union of the empty positions in the N channels to obtain the area of ​​the empty positions in the channels, S2. Take the union of S1 and S2 to obtain the total area of ​​the channels, S1∪S2. Fill the total area of ​​the channels to obtain S3.

[0084] S24: Calculate the number of pixels in S1+S2-S3 to obtain the overlapping area between the empty product location and the aisle. Finally, divide this overlapping area by S1 to obtain the ratio m1 of the empty product location area to the aisle area before replenishment. This ratio can be expressed by the formula: m1 can represent the information of the first cargo channel among N cargo channels. The larger m1 is, the emptier the cargo channel is.

[0085] In one implementation, the ratio of the area of ​​empty goods positions to the total area of ​​the aforementioned aisle is set with three different discrete values, representing the aisle status as "empty," "partially empty," and "partially full," respectively. This facilitates a more intuitive visual assessment of the aisle status, and the three statuses are represented by "red," "green," and "blue" rectangles, respectively. It should be noted that the use of three different statuses and colors for the aisle status in this embodiment is merely an example; other statuses or colors can also be used to represent the aisle status, and this embodiment does not limit this approach.

[0086] By using the steps S21 to S24 above, the product shortage status of the storage lane can be determined by the ratio of the area of ​​the empty product location to the total area of ​​the storage lane before replenishment.

[0087] S3: Based on the first and second images, match the aisles with empty stock before replenishment with the corresponding aisles after replenishment to determine the N aisles included in the retail cabinet after replenishment. Calculate the N second aisle information for each of the N aisles after replenishment, compare the first aisle information with the second aisle information before and after replenishment, and determine the N first aisle information.

[0088] In one implementation, the specific steps of S3 above include:

[0089] S31: Calculate the relative lateral position range of the N aisles with empty product locations in the first image, within their respective shelf layers. Based on the lateral proportion of their left and right edges within their respective shelf layers, obtain the relative lateral position range of each of the N aisles within the shelf layer.

[0090] S32: Locate the aisle areas in the second image that correspond to the aforementioned ranges in the first image. Find the shelf layer area in the second image that is on the same shelf layer as the aisle with empty product areas in the first image. Then, based on the lateral relative positions of the N aisles in the first image, obtain the corresponding areas in the second image and determine them as a pair of corresponding aisles.

[0091] S33: Following the methods of S21 to S22, segment and reduce the weight of the cargo lane region in the second image to obtain a list of N non-overlapping cargo lanes in the second image.

[0092] S34: Fill the empty positions of the goods in the N channels after the weight reduction and segmentation to obtain the sum of the areas of the goods positions and the empty positions in the channels, S4. Take the union of the empty positions in the N channels to obtain the area of ​​the empty positions in the channels, S5. Take the union of S4 and S5 to obtain the total area of ​​the channels, S4∪S5. Fill the total area of ​​the channels to obtain S6.

[0093] S35: Calculate the number of pixels in S4+S5-S6 to obtain the overlapping area between the empty product location and the aisle. Finally, divide this overlapping area by S4 to obtain the ratio m2 of the empty product location area to the aisle area before replenishment. This ratio can be expressed by the formula: m2 can represent the information of the second cargo channel out of N cargo channels. The larger m2 is, the emptier the cargo channel is.

[0094] S36: Subtract the ratio m1 of the empty product locations indicated by the N first aisle information to the ratio m2 of the empty product locations indicated by the N second aisle information to obtain N first information items. Each first information item is obtained by subtracting the first aisle information corresponding to one of the N aisles from the corresponding second aisle information. The first information item represents the degree of restocking of the retail counter. For a given first information item, the first and second aisle information items used to determine that the first information item corresponds to the same aisle are used.

[0095] In one implementation, if all N first pieces of information are greater than or equal to a preset threshold, then the replenishment of the retail cabinet is determined to be successful; or, if one or more of the N first pieces of information are less than the preset threshold, then the channel that obtained the first information is determined to have failed to replenish the retail cabinet, and a first replenishment alarm is sent to indicate that the replenishment has failed.

[0096] In one implementation, a grid search method is used to determine the first information and select the optimal threshold to reduce the discrepancy with the definition of large-scale replenishment as defined by manual annotation.

[0097] In one implementation, if restocking of a retail counter fails, a restocking alarm is sent. The alarm includes information about the shelf corresponding to the failed restocking attempt. The administrator reviews the alarm and issues a restocking instruction. It should be noted that the above method is merely an example, and this embodiment of the invention does not limit the scope of the invention.

[0098] In one implementation, if it is determined that restocking the retail counter has failed, the failure information is recorded in the work quality report of the corresponding restocking personnel. The work quality reports of the restocking personnel are reviewed periodically. If the work quality problems of a restocking personnel exceed the corresponding threshold within a certain period of time, a work quality alarm is issued to the restocking personnel for the administrator to review and confirm. This is to prevent restocking personnel from arranging goods carelessly or damaging goods.

[0099] Therefore, by comparing N pieces of first cargo channel information with N pieces of second cargo channel information through the above steps S31 to S36, N pieces of first information can be obtained.

[0100] S4: Based on the detected N pieces of initial information, record the area ratio and product category number of each type of product in the storage lane before and after replenishment. Finally, compare the area difference of each type of product in the N storage lanes before and after replenishment to determine the replenishment product category for the corresponding storage lane.

[0101] As a specific implementation of an embodiment of the present invention, the specific steps of S4 described above are as follows:

[0102] S41: Based on the detected N channels, detect the quantity and category number of all goods within the range of the N channels in the first image, and accumulate the quantity of goods of the same type;

[0103] S42: Based on the detected N channels, detect the quantity and category number of all goods within the N channels in the second image. If the goods are already in the first image, calculate the difference in the quantity of goods within the N channels in the first and second images. If the increased quantity of goods of this type is less than the preset quantity of goods of this type, it is determined that the replenishment of goods of this type has failed, and a second replenishment alarm is sent. The second replenishment alarm is used to indicate that the replenishment of goods of this type has failed. Alternatively, if the increased quantity of goods of this type is greater than or equal to the preset quantity of goods of this type, it is determined that the replenishment of goods of this type has succeeded.

[0104] In one implementation, a labeled dataset is used for testing, and a grid-based parameter tuning method is used to adjust the threshold for detecting the first information and the threshold for the product replenishment status, thereby selecting the threshold combination that yields the best test results.

[0105] In one implementation, if it is determined that replenishing the product type fails, a second replenishment alarm is sent. The administrator then reviews the replenishment alarm and issues a replenishment instruction. It should be noted that the above method is merely an example, and this embodiment of the invention does not limit the scope of the invention.

[0106] In one implementation, if restocking of a retail cabinet fails, the failure information is recorded in the work quality report of the corresponding restocking personnel. The work quality reports are periodically reviewed; if a restocking personnel's work quality issues exceed a certain threshold within a certain period, a work quality alarm is issued for that personnel, allowing the administrator to review and confirm the alarm, thus preventing restocking personnel from improperly arranging goods or damaging them. In a specific implementation, this invention is tested on a dataset of first and second images of retail cabinets in a real-world scenario provided by Xinyi Technology Group. For a given set of 32,895 pairs of first and second images from real-world scenarios, images with poor quality, such as blurriness, incorrect shooting angles, or tilted retail cabinets, are first removed. The model is then trained using the remaining 21,365 images. Due to the high cost of manually labeling the product types after restocking, the actual algorithm test used 228 images of the retail cabinet before and after restocking for testing.

[0107] To objectively evaluate the performance of this method, two test metrics, recall and precision, are used. Recall represents the first piece of information correctly identified by the model, while precision represents the accuracy in correctly identifying the actual replenishment status of a product among all samples labeled as replenishment items by the model. The ground truth for product replenishment status is determined based on image data before and after replenishment and the actual replenishment status. A higher recall value indicates that the model can effectively identify product replenishment status, while a higher precision value indicates that there are very few false positives in the product replenishment status labeled by the model. This invention aims to maximize the precision of the model.

[0108] Furthermore, model performance is affected by the shooting angle. Generally, the less tilted the image is, the higher the accuracy of lane and product detection, resulting in better performance in detecting product replenishment. Therefore, to better demonstrate the model's performance, the test dataset is divided into frontal and side-view images based on the angle of the photos, with all-view images representing all test images.

[0109] The experimental results are shown in Table 1. The results show that the present invention can detect the replenishment status of goods relatively accurately, and can accurately provide information such as the product category, location, and shelf layer of the replenished goods.

[0110] Table 1. Images before and after replenishment, product replenishment status, and accuracy of the detection results.

[0111] Full-view recall rate Full-view precision Positive perspective recall rate Precision of positive angle 67.9％ 71.1％ 85.4％ 79.3％

[0112] The above embodiments are merely one solution of the present invention, and the embodiments of the present invention are not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

[0113] In accordance with the above method and process, this embodiment of the invention also provides a device, an acquisition module, a processing module, and a detection module. The specific details of these devices and systems are as described in the above embodiments and will not be repeated here.

[0114] Based on the same inventive concept, such as Figure 4 As shown, an embodiment of the present invention provides an apparatus that can execute the process of a product replenishment detection method.

[0115] like Figure 4 As shown, the device is used to perform the following functions:

[0116] The acquisition module 401 is used to acquire a first image and a second image, wherein the first image is an image of the retail cabinet before restocking and the second image is an image of the retail cabinet after restocking;

[0117] Processing module 402 is used to determine, based on the first image, N first aisle information of N aisles included in the retail cabinet before replenishment, wherein the first aisle information indicates the proportion of the area of ​​empty product positions in the aisles to the area of ​​the aisles before replenishment; N is an integer greater than 0.

[0118] Based on the second image, determine the N second channel information of the N channels included in the retail cabinet after replenishment, wherein the second channel information indicates the proportion of the area of ​​the empty position of the goods in the channel after replenishment to the area of ​​the channel.

[0119] Based on N pieces of information about the first freight lane and N pieces of information about the second freight lane, obtain N pieces of first information;

[0120] The detection module 403 is used to detect whether the retail cabinet has been successfully replenished. If all N first information are greater than or equal to a preset threshold, it is determined that the retail cabinet has been successfully replenished; or, if one or more of the N first information are less than the preset threshold, it is determined that the channel that obtained the first information failed to replenish the retail cabinet, and a first replenishment alarm is sent. The first replenishment alarm is used to indicate that the replenishment has failed.

[0121] In one embodiment, the processing module 402 is further configured to: determine, based on the second image, N third channel information of the N channels included in the replenished retail cabinet, wherein the third channel information indicates the type of goods in the replenished channels.

[0122] In one embodiment, the detection module 403 is further configured to:

[0123] If the quantity of goods added to any of the M product types is less than the preset quantity, then replenishment for that product type is determined to have failed, and a second replenishment alarm is sent to indicate the failure. Alternatively, if the quantity of goods added to that product type is greater than or equal to the preset quantity, then replenishment for that product type is determined to have succeeded.

[0124] Based on the same technical concept, this invention also provides a terminal device 500, see reference. Figure 5 As shown, the terminal device 500 is used to implement the product replenishment detection method described in the above method embodiment. The terminal device 500 in this embodiment may include: a processor 501, an interface 502, a memory 503, and a computer program stored in the memory and executable on the processor, such as a replenishment detection program.

[0125] The specific connection medium between the processor 501, interface 502, and memory 503 described above is not limited in the embodiments of this invention. The embodiments of this application... Figure 5 The processor 501, interface 502, and memory 503 are connected via a bus. The connection methods between other components are only illustrative and should not be considered as limiting. The bus can be divided into address bus, data bus, control bus, etc.

[0126] Processor 501 is used to call computer programs stored in memory 503 to execute, such as Figure 2 The method for detecting the replenishment status of goods shown is illustrated.

[0127] Interface 502 is used to acquire images of the retail cabinet.

[0128] Memory 503 may be volatile memory, such as random-access memory (RAM); memory 503 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 503 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 503 may be a combination of the above-mentioned memories.

[0129] This application also provides a computer-readable storage medium storing a computer program or instructions, which, when executed by a user device, implement... Figure 2 Any one of the methods.

[0130] In some possible implementations, various aspects of the table entry update method provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on an electronic device, causes the electronic device to perform the steps in the table entry update method according to various exemplary embodiments of the present invention described above.

[0131] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting the replenishment status of goods, characterized in that, include: Acquire a first image and a second image, wherein the first image is an image of the retail cabinet before restocking, and the second image is an image of the retail cabinet after restocking; Based on the first image, determine the information of N first aisles of the N aisles included in the retail cabinet before replenishment, wherein the information of the first aisle indicates the proportion of the area of ​​the empty position of the goods in the aisle to the area of ​​the aisle before replenishment; N is an integer greater than 0. Based on the second image, determine the N second channel information of the N channels included in the retail cabinet after replenishment, wherein the second channel information indicates the proportion of the area of ​​the empty position of the goods in the channel after replenishment to the area of ​​the channel. Based on the N first cargo channel information and the N second cargo channel information, N first information pieces are obtained; If all N first pieces of information are greater than or equal to a preset threshold, then the replenishment of the retail cabinet is determined to be successful; or, if one or more of the N first pieces of information are less than the preset threshold, then the channel that obtained the first information is determined to have failed to replenish the retail cabinet, and a first replenishment alarm is sent, the first replenishment alarm being used to indicate replenishment failure. The step of obtaining N pieces of first information based on the N pieces of first cargo channel information and the N pieces of second cargo channel information includes: The proportion of the empty positions of the goods indicated by the N first cargo lane information to the cargo lane area. The proportion of the empty positions of the goods in the N second cargo lane information to the area of ​​the cargo lane. Subtracting the two information values ​​yields N pieces of the first information; wherein each piece of the first information is obtained by subtracting the first information of one of the N information channels from the second information of the corresponding channel.

2. The method as described in claim 1, characterized in that, The method further includes: Based on the second image, determine the information of N third channels of the N channels included in the retail cabinet after replenishment, wherein the information of the third channels indicates the type of goods in the channels after replenishment; Based on the N third-channel information, determine the additional quantity of goods for each of the M product types in the N channels; M is an integer greater than 0. For any of the M product types, if the quantity of goods added to the product type is less than the preset quantity of goods for the product type, then it is determined that the replenishment of goods for the product type has failed, and a second replenishment alarm is sent. The second replenishment alarm is used to indicate that the replenishment of goods for the product type has failed; or, if the quantity of goods added to the product type is greater than or equal to the preset quantity of goods for the product type, then it is determined that the replenishment of goods for the product type has succeeded.

3. The method as described in claim 1, characterized in that, For any one of the N first cargo channel information, the proportion of the area of ​​the empty product location in the cargo channel before replenishment, as indicated by the first cargo channel information, to the total area of ​​the cargo channel. It must meet the following form: ； in, This represents the sum of the areas of the goods positions and the empty positions in the cargo aisle. This represents the area of ​​the empty space for goods in the cargo aisle. This indicates the area of ​​the cargo channel.

4. The method as described in claim 1, characterized in that, For any of the N second cargo channel information, the proportion of the area of ​​the empty product location in the cargo channel after replenishment, as indicated by the second cargo channel information, to the total area of ​​the cargo channel. It must meet the following form: ； in, This represents the sum of the areas of the goods positions and the empty positions in the cargo aisle. This represents the area of ​​the empty space for goods in the cargo aisle. This indicates the area of ​​the cargo channel.

5. A device for detecting the replenishment status of goods, characterized in that, include: The acquisition module is used to acquire a first image and a second image, wherein the first image is an image of the retail cabinet before restocking, and the second image is an image of the retail cabinet after restocking; The processing module is configured to, based on the first image, determine N first aisle information for the N aisles of the retail cabinet before replenishment, wherein the first aisle information indicates the proportion of the area of ​​empty product positions in the aisles to the aisle area before replenishment; N is an integer greater than 0; based on the second image, determine N second aisle information for the N aisles of the retail cabinet after replenishment, wherein the second aisle information indicates the proportion of the area of ​​empty product positions in the aisles to the aisle area after replenishment; and obtain N first information based on the N first aisle information and the N second aisle information. The detection module is used to detect whether the retail cabinet has been successfully restocked. If all of the N first pieces of information are greater than or equal to a preset threshold, it is determined that the retail cabinet has been successfully restocked; or, if one or more of the N first pieces of information are less than the preset threshold, it is determined that the retail cabinet has failed to be restocked, and a first restocking alarm is sent. The first restocking alarm is used to indicate that the restocking has failed. The processing module is specifically used to calculate the proportion of the empty positions of the goods in the N first channel information to the channel area. The proportion of the empty positions of the goods in the N second cargo lane information to the area of ​​the cargo lane. Subtracting the two information values ​​yields N pieces of the first information; wherein each piece of the first information is obtained by subtracting the first information of one of the N information channels from the second information of the corresponding channel.

6. The apparatus as claimed in claim 5, characterized in that, The processing module is also used for: Based on the second image, determine the information of N third channels of the N channels included in the retail cabinet after replenishment, wherein the information of the third channels indicates the type of goods in the channels after replenishment.

7. The apparatus as described in claim 5 or 6, characterized in that, The detection module is also used for: If the quantity of goods added to any of the M product types is less than the preset quantity, then it is determined that the replenishment of goods for that product type has failed, and a second replenishment alarm is sent to indicate that the replenishment of goods for that product type has failed; or, if the quantity of goods added to that product type is greater than or equal to the preset quantity, then it is determined that the replenishment of goods for that product type has succeeded.

8. A computer device, characterized in that, Includes a program or instruction, which, when executed, performs the method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, Includes a program or instruction, which, when executed, performs the method as described in any one of claims 1 to 4.

Images