Recognition Method, System, Device and Storage Medium for Stacked Items

Abstract

The present disclosure provides a method for identifying stacked objects. This method marks objects in an image containing stacked objects to obtain marked regions corresponding to the objects in the image; rotates the marked regions so that each marked region is used as a reference region at least once; filters and processes the marked regions to obtain intermediate regions corresponding to each reference region with the highest detection confidence; and identifies the objects marked by the reference region and the corresponding intermediate regions as stacked objects. This method can effectively improve the detection rate of stacked objects that are missed due to row detection failures.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of image processing in computer vision, and in particular to a method, system, device, and storage medium for identifying stacked objects. Background Art

[0002] Stacking items on shelves is common. Since these stacked items are often the same as the products at the bottom, they effectively take up ineffective brand space. Therefore, manufacturers actively ignore these stacked items when calculating the brand space percentage of their products.

[0003] In the prior art, stacking recognition is usually implemented based on row detection, where a row refers to each layer of a shelf. Summary of the Invention

[0004] One aspect of the present disclosure provides a method for identifying stacked objects. The method for identifying stacked objects comprises the following steps:

[0005] Acquire an image with stacked objects, identify and mark the objects in the image to obtain marked areas corresponding to the objects in the image;

[0006] rotating the marked areas so that each of the marked areas is used as a reference area at least once;

[0007] The marked areas are screened and processed to obtain an intermediate area corresponding to each reference area and having the highest detection confidence; and the objects marked by the reference area and the corresponding intermediate area are identified as stacked objects.

[0008] According to an embodiment of the present disclosure, the step of identifying the objects marked in the reference area and the corresponding middle area as stacked objects further includes:

[0009] Rescreening the marked areas to obtain target areas corresponding to each of the intermediate areas and having the highest detection confidence;

[0010] Comparing the detection confidence levels to obtain comparison results of the detection confidence levels of the reference area and the target area respectively corresponding to the intermediate area;

[0011] The comparison result is identified to identify the objects marked in the reference area and the corresponding target area as stacked objects; wherein the detection confidence of the reference area is not lower than the detection confidence of the target area.

[0012] According to an embodiment of the present disclosure, before the step of rotating the marked area, the step further includes:

[0013] The marked areas are sorted so as to rotate the marked areas in sequence.

[0014] According to an embodiment of the present disclosure, the value range of the second threshold in the screening process is 0.6-1.

[0015] According to an embodiment of the present disclosure, two or more marking areas identified as stacked objects are merged into one marking area.

[0016] Another aspect of the present disclosure provides a stacked object identification system. The stacked object identification system is used to implement the stacked object identification method described above. The identification system includes:

[0017] an object marking module, configured to acquire an image with stacked objects, identify and mark the objects in the image, and obtain marked areas corresponding to the objects in the image;

[0018] an area rotation module, configured to rotate the marked areas so that each of the marked areas is used as a reference area at least once;

[0019] a screening and processing module, configured to screen and process the marked areas to obtain an intermediate area corresponding to each reference area and having the highest detection confidence;

[0020] The stacking identification module is used to identify the objects marked in the reference area and the corresponding middle area as stacked objects.

[0021] Another aspect of the present disclosure provides a device for identifying stacked items. The device includes:

[0022] A memory and a processor, wherein instructions are stored in the memory and the memory and the processor are interconnected via a line;

[0023] The processor calls the instructions in the memory to implement the stacked object identification method in an embodiment of the present disclosure.

[0024] The final aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the stacked object identification method of an embodiment of the present disclosure.

[0025] Due to the adoption of the above technical solution, the present disclosure has the following advantages and positive effects compared with the prior art:

[0026] 1) The stacked object recognition method in one embodiment of the present disclosure marks items in an image containing stacked objects to obtain marked regions corresponding to the objects in the image; rotates the marked regions so that each marked region is used as a reference region at least once; filters and processes the marked regions to obtain an intermediate region corresponding to each reference region with the highest detection confidence; and identifies the objects marked by the reference region and the corresponding intermediate region as stacked objects. Compared to existing technologies, this method further improves the success rate of stacked object detection.

[0027] 2) The stacked object identification method in one embodiment of the present disclosure combines two or more marked areas identified as stacked objects into one marked area, so that when calculating the brand face ratio, objects that are stacked will not be counted repeatedly.

[0028] 3) In an embodiment of the present disclosure, the stacked object recognition method sorts the marked areas to facilitate rotation of the marked areas so that the marked areas are rotated in sequence to avoid duplication or omission, thereby improving the recognition rate of stacked objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a method for identifying stacked objects in one embodiment of the present disclosure;

[0030] Figure 2 This is a block diagram of a stacked object recognition system in one embodiment of the present disclosure;

[0031] Figure 3 Schematic diagram of a device for identifying stacked objects in one embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a detailed description of the stacked object identification method, system, device and storage medium proposed in the present disclosure in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present disclosure will become more apparent from the following description and claims.

[0034] Example 1

[0035] Current stacking detection is based on row detection, where a "row" in existing technology refers to each layer of the shelf. When multiple products are in the same row and have a certain degree of vertical overlap on the shelf level, they are considered stacked.

[0036] During the specific implementation process, existing detection solutions may fail to detect rows. Generally, in order to clearly capture large packaged products in warehouse-style supermarkets or objects with large row spacing such as rolls of paper, the camera frame may lose some shelf features, such as shelves and columns, causing the detection model in the existing detection solution to be unable to effectively detect the row features, ultimately leading to frequent failure of row detection.

[0037] To solve the above problems, this embodiment provides a method for identifying stacked objects. Figure 1 , the stacked object identification method comprises the following steps:

[0038] Step S100: Acquire an image with stacked objects;

[0039] Step S200: marking objects in the image to obtain marked areas corresponding to the objects in the image;

[0040] Step S300: rotating the marked areas so that each marked area is used as a reference area at least once;

[0041] Step S400: screening and processing the marked areas to obtain the middle area corresponding to each reference area and having the highest detection confidence;

[0042] Step S500: Identify the objects marked in the reference area and the corresponding middle area as stacked objects.

[0043] Specifically, in step S100, a real-time captured on-site image of a scene where items are stacked can be obtained. The real-time image is an image of item storage on a shelf, including the shelf and items on the shelf.

[0044] In step S200, the objects in the image are marked to obtain marked regions corresponding to the objects in the image. In practical applications, an image of stacked objects acquired on-site can be input into a pre-set object recognition model for object recognition, generating a bounding box (i.e., a marked region) surrounding each object.

[0045] The object recognition model can be a MASK R-CNN model, which includes a ResNet-101 network, an RPN network, a ROI Alignment layer, and a classification network. The recognition principle of the MASK R-CNN model is as follows:

[0046] The image is input into the preset MASK R-CNN model, and the image features are extracted through the ResNet-101 network to obtain a first feature map; the first feature map is input into the RPN network, and the prediction box is extracted from the first feature map through the preset selective search algorithm to obtain a prediction box corresponding to the first feature map; the first feature map and the prediction box are input into the ROI Align layer for prediction to obtain a second feature map containing the prediction box; the second feature map is input into the fully connected layer for classification processing to obtain the predicted probability that the prediction box contains the object; based on the predicted probability that the prediction box contains the object, an image marked with the marked area of the object is obtained.

[0047] In practical applications, different object recognition models can be selected according to the application scenario. The model is not limited to the MASK R-CNN model in this embodiment, and can also be an object recognition model based on the YOLO series.

[0048] Alternatively, a static image detector such as DeepID-Net or CRAFT can be used to label objects in the image and generate bounding boxes for each object.

[0049] In step S300, the marked areas of the object are rotated so that each marked area is used as a reference area at least once. Rotation here means that each marked area is used as a reference area for screening in turn.

[0050] In order to facilitate the rotation of the marked areas, the marked areas can be sorted so that the marked areas can be rotated in sequence to avoid duplication or omission, thereby improving the recognition rate of subsequent stacked items.

[0051] In step S400 , the marked areas of the processed object are screened to obtain the middle area corresponding to each reference area and having the highest detection confidence.

[0052] In order to more clearly describe the solution provided by the present disclosure, the present disclosure gives a unified definition in each embodiment, defining the extension direction of the stacked items on the shelf as the first direction, and the second direction as the direction opposite to the first direction.

[0053] For the concept of "direction" involved in the various embodiments of the present disclosure, more specific directional nouns are usually used in practical applications. For example, in the present disclosure, the shelves referred to are usually conventional shelves placed on the ground in supermarkets, convenience stores, etc. Since each layer of the shelf is parallel to each other, and in order to make the items placed thereon more stable, each layer of the shelf is also parallel to the ground. It is easy to understand that the ground of supermarkets, convenience stores, etc. put into commercial use can often be considered to be in a horizontal state, so in the present disclosure, those skilled in the art can also regard each layer of the shelf as a horizontal state. Relative to the ground where the shelf is located, the vertical upward direction, that is, the vertical positive direction is consistent with the extension direction of the stacked items. Therefore, in the various embodiments of the present disclosure, the first direction defined above can be specifically expressed as the vertical positive direction, and the second direction can be specifically expressed as the vertical negative direction.

[0054] The screening process in this disclosure refers to selecting other marked areas whose distance from the reference area in the first direction is no greater than a first threshold as candidate screening areas; continuing to select candidate screening areas whose projection overlap on a plane perpendicular to the first direction with the reference area is no less than a second threshold and whose shapes are similar as candidate screening areas; and selecting the marked area with the highest detection confidence among the candidate screening areas to obtain the intermediate area. Similar shape refers to areas where the difference in aspect ratio and area size between the marked areas is no greater than a third threshold and a fourth threshold, respectively.

[0055] The first threshold, second threshold, third threshold, and fourth threshold can be set according to actual needs. In combination with actual use results, when facing conventional supermarket shelves, when the value range of the second threshold in the present disclosure is set to 0.6 to 1, it can not only bring more accurate detection results, which is applicable to images taken from most supermarket shelves on the market, but also reduce the processing intensity of the computer and improve recognition efficiency. When the value of the third threshold and / or the fourth threshold is set larger, it can be considered that the shapes of the selected marked areas are more consistent.

[0056] Based on the definition of "direction," in this embodiment, the marked boxes containing the items can be sorted vertically to construct a connectivity graph of the marked areas. The marked areas are then rotated within the connectivity graph so that at least one marked area is used as a reference area. When rotating the marked areas within the connectivity graph, each marked area can also be used as a reference area at least once.

[0057] The following describes step S400 using the object bounding box a as an example:

[0058] For each box a (reference area), search for an item bounding box whose distance from box a is less than a first threshold along the vertical positive direction, and use it as a candidate screening area. When the bounding box is rectangular, this distance is the distance between the top edge of the lower rectangular box and the bottom edge of the upper rectangular box. The first threshold can be adjusted according to the height of different items, such as by multiplying the size of the current bounding box (only the height of the box is used to represent the size of the box) by a coefficient reasonably set based on actual data. For example, the row spacing of toothpaste will be smaller than the row spacing of toilet paper.

[0059] From the candidate screening areas, the marked objects whose horizontal overlap with the marked objects is greater than a second threshold and whose shapes are similar are selected as the candidate screening areas. The marked area with the highest detection confidence is selected from the candidate screening areas to obtain the middle area.

[0060] That is, from the candidate regions, select the object bounding boxes whose horizontal overlap with box a is greater than the second threshold and whose shape is similar to box a as the candidate regions. Select the marked region with the highest detection confidence within the candidate regions to obtain the intermediate region a_i.

[0061] It should be noted that horizontal overlap here can be understood as overlap of projections onto a horizontal plane. If all items are projected onto the same horizontal plane (i.e., projected vertically), significant horizontal overlap generally occurs only when they are arranged one above the other with the target item, a. Therefore, setting a second threshold can filter out items with a high degree of overlap. After analysis, setting the second threshold to 0.7 in the disclosed embodiment yields relatively ideal screening results.

[0062] The overlap degree in this disclosure is used to measure the degree of overlap between two or more objects. Generally, the overlap degree value can be considered to be between 0 and 1, where 0 indicates no overlap at all and 1 indicates complete overlap.

[0063] In step S500 , the objects marked in the reference area and the corresponding middle area are identified as stacked objects.

[0064] Specifically, the marked areas are rescreened to obtain a target area corresponding to each intermediate area and having the highest detection confidence; the detection confidences are compared to obtain a comparison result of the detection confidences of the reference area and the target area corresponding to the intermediate areas respectively; the comparison result is identified to identify the objects marked by the reference area and the corresponding target area as stacked objects; wherein the detection confidence of the reference area is not lower than the detection confidence of the target area.

[0065] The re-screening process here refers to selecting other marked areas whose distance from the middle area in the second direction is not greater than the first threshold as re-screening candidate areas; continuing to select re-screening candidate areas whose projection overlap on the plane perpendicular to the second direction with the middle area is not less than the second threshold and whose shape is similar, as re-screening pending areas; selecting the marked area with the highest detection confidence in the re-screening pending areas to obtain the target area.

[0066] The shape similarity is consistent with the shape similarity requirement in the screening process in the above step S400.

[0067] In practical applications, starting from the middle area a_i, along the vertical negative direction, other marked areas whose distance from it is not greater than the first threshold are selected as re-screening candidate areas; continue to select re-screening candidate areas whose projection overlap on the plane in the vertical negative direction of the middle area is not less than the second threshold and whose shape is similar as re-screening pending areas; select the marked area with the highest detection confidence in the re-screening pending areas to obtain the target area a_j.

[0068] The vertical negative direction and the vertical positive direction in the present disclosure are relative concepts, that is, if the vertical positive direction is in step S400, then the direction in this step must be opposite to that in step S400, that is, the vertical negative direction.

[0069] After the above steps of screening and elimination, the detection confidence is verified. If the detection confidence is not lower than a_j, it can be considered that a~a_i belong to the same group of stacked items.

[0070] After stacked items are identified, two or more marked areas identified as stacked items are merged into one marked area. When calculating the brand's face value ratio, the stacked items will not be counted repeatedly due to duplication. The stacked items can be automatically ignored, thereby improving the calculation speed and the accuracy of the face value ratio.

[0071] Based on the above description, it can be seen that the stacked object identification method provided by the present disclosure can effectively reduce the problem of missed detection caused by row detection failure.

[0072] Example 2

[0073] This embodiment provides a stacked object recognition system. Figure 2 , the stacked object recognition system includes:

[0074] The object marking module 110 is used to obtain an image with stacked objects, identify and mark the objects in the image, and obtain marked areas corresponding to the objects in the image;

[0075] a region rotation module 120 for rotating the marked regions so that each marked region is used as a reference region at least once;

[0076] A screening processing module 130 is used to screen and process the marked areas to obtain an intermediate area corresponding to each reference area and having the highest detection confidence;

[0077] The stacking identification module 140 is configured to identify the objects marked in the reference area and the corresponding middle area as stacked objects.

[0078] The stacked object recognition system also includes a rescreening unit and a comparison unit. The rescreening unit is configured to rescreen the marked areas to obtain a target area corresponding to each intermediate area and having the highest detection confidence; compare the detection confidence levels to obtain a comparison result of the detection confidence levels of the reference area and the target area corresponding to the intermediate area; and identify the comparison result to identify the marked items between the reference area and the corresponding target area as stacked items. The detection confidence level of the reference area is not lower than that of the target area.

[0079] The comparison unit is used to compare the detection confidence levels to obtain comparison results of the detection confidence levels of the reference area and the target area respectively corresponding to the middle area.

[0080] In addition, the stacking identification module 140 can also be used to identify the comparison result obtained by the comparison unit to identify the objects marked in the reference area and the corresponding target area as stacked objects. It should be noted that the stacking identification module 140 identifies the stacked objects referred to in this paragraph only if the comparison result shows that the detection confidence level of the reference area is not lower than the detection confidence level of the target area.

[0081] Compared with the prior art, the stacked object recognition system provided in this embodiment can more quickly obtain effective features of stacked objects by utilizing the screening of object spacing and detection confidence, thereby improving the execution efficiency of the system in identifying stacked objects; in addition, by rotating and sorting the marked areas, the corresponding recognition operations are performed in an orderly manner, avoiding the problems of repeated and missed recognition, thereby reducing the proportion of computing resources occupied.

[0082] Example 3

[0083] This embodiment provides a device for identifying stacked objects. Figure 3The stacked object recognition device 500 may vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 510 (e.g., one or more processors), a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing application programs 533 or data 532. The memory 520 and storage media 530 may be either transient or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), each of which may include a series of instructions for operating the stacked object recognition device 500.

[0084] Furthermore, the processor 510 may be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the stacked item identification device 500 .

[0085] The stacked item identification device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Vista, etc.

[0086] Those skilled in the art will understand that Figure 3 The structure of the stacked object identification device shown does not constitute a limitation on the stacked object identification device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0087] Another embodiment of the present disclosure further provides a computer-readable storage medium.

[0088] Figure 4 A program storage product 600 for implementing the identification method disclosed in the aforementioned embodiment is described. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the stacked object identification method described in Example 1.

[0089] If the stacked object identification method is implemented in the form of program instructions and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of software. The computer software is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the identification content specifically executed by the above-described system and device can refer to the corresponding process in the aforementioned method embodiment.

[0091] The embodiments of the present disclosure are described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to the above embodiments. Even if various changes are made to the present disclosure, if these changes fall within the scope of the claims of the present disclosure and their equivalents, they will still fall within the scope of protection of the present disclosure.

Claims

1. A method for identifying stacked objects, characterized in that: include: Get an image with stacked items; Marking the objects in the image to obtain marked areas respectively corresponding to the objects in the image; rotating the marked areas so that at least one of the marked areas is used as a reference area; Screening and processing the marked areas to obtain an intermediate area corresponding to each reference area and having the highest detection confidence; identifying the objects marked in the reference area and the corresponding middle area as stacked objects; The step of identifying the objects marked in the reference area and the corresponding middle area as stacked objects further includes: Rescreening the marked areas to obtain target areas corresponding to each of the intermediate areas and having the highest detection confidence; Comparing the detection confidence levels to obtain comparison results of the detection confidence levels of the reference area and the target area respectively corresponding to the intermediate area; The comparison result is identified to identify the objects marked in the reference area and the corresponding target area as stacked objects; wherein the detection confidence of the reference area is not lower than the detection confidence of the target area.

2. The method for identifying stacked objects according to claim 1, wherein: The step of rotating the marked areas so that at least one marked area is used as a reference area further comprises: sorting the marked regions along a first direction to construct a connectivity graph of the marked regions; The marked regions are rotated in the connectivity map so that at least one of the marked regions is used as a reference region.

3. The method for identifying stacked objects according to claim 2, wherein: The marked regions are rotated in the connectivity map so that each of the marked regions is used as a reference region at least once.

4. The method for identifying stacked objects according to claim 1, wherein: The value range of the second threshold in the screening process is 0.6-1.

5. The method for identifying stacked objects according to claim 4, wherein: The second threshold is 0.

7.

6. The method for identifying stacked objects according to claim 1, wherein: Two or more marked areas identified as stacked objects are merged into one marked area.

7. A stacked object identification system, used to implement the steps of the stacked object identification method according to any one of claims 1 to 6, characterized in that: The identification system comprises: an object marking module, configured to acquire an image with stacked objects, identify and mark the objects in the image, and obtain marked areas corresponding to the objects in the image; an area rotation module, configured to rotate the marked areas so that at least one of the marked areas is used as a reference area; a screening and processing module, configured to screen and process the marked areas to obtain an intermediate area corresponding to each reference area and having the highest detection confidence; a stacking identification module, configured to identify the objects marked in the reference area and the corresponding middle area as stacked objects; Wherein, the stacking identification module is further used for: Rescreening the marked areas to obtain target areas corresponding to each of the intermediate areas and having the highest detection confidence; Comparing the detection confidence levels to obtain comparison results of the detection confidence levels of the reference area and the target area respectively corresponding to the intermediate area; The comparison result is identified to identify the objects marked in the reference area and the corresponding target area as stacked objects; wherein the detection confidence of the reference area is not lower than the detection confidence of the target area.

8. A device for identifying stacked objects, characterized in that: include: A memory and a processor, wherein instructions are stored in the memory and the memory and the processor are interconnected via a line; The processor calls the instructions in the memory to implement the stacked object identification method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying stacked objects according to any one of claims 1 to 6 is implemented.

Images