Goods shelf detection device and method

Abstract

The embodiment of the invention provides a goods shelf detection device and method, and the device comprises a first detection unit which detects and recognizes the positions of a plurality of goods in an input image from the input image; the first determining unit is used for determining the information of the preset position of each goods and classifying the information of the preset position of each goods; and the second determining unit is used for fitting the information of the preset positions of the goods in each category, and determining the straight line obtained by fitting in each category as the position of the goods shelf layer.

Description

Technical Field

[0001] The present application relates to the field of image processing. Background Art

[0002] Traditional retail supermarkets and convenience stores require dedicated staff, resulting in high labor costs. Related technologies require staff to monitor shelves to see if items are restocked, out of stock, or moved by customers. Currently, automatic detection of product status can be performed based on captured images. However, when products are densely stacked, it is difficult to identify shelf layers from the image, and thus it is impossible to use the position information of the products on the corresponding shelf layer to detect product status. Alternatively, methods for detecting shelf layers based on vanishing points have been proposed, but these methods are unstable when complex textures are present in the image.

[0003] It should be noted that the above technical background is merely provided to provide a clear and complete description of the technical solutions of this application and to facilitate understanding by those skilled in the art. Simply because these solutions are described in the background technology section of this document, it should not be assumed that the above technical solutions are well known to those skilled in the art. Summary of the Invention

[0004] To address at least one of the above problems, embodiments of the present application provide a shelf detection device and method.

[0005] According to a first aspect of an embodiment of the present application, a shelf detection device is provided, the device comprising:

[0006] a first detection unit, configured to detect and identify positions of a plurality of items in an input image;

[0007] a first determining unit, which determines information about a predetermined location of each item and classifies the information about the predetermined location of each item;

[0008] The second determining unit fits the information of the predetermined positions of the goods in each category, and determines the straight line obtained by fitting in each category as the position of the shelf layer.

[0009] According to a second aspect of an embodiment of the present application, a shelf detection method is provided, the method comprising:

[0010] Detecting and identifying locations of a plurality of items in an input image;

[0011] Determining the information of the scheduled location of each product and classifying the information of the scheduled location of each product;

[0012] The information of the predetermined positions of the goods in each category is fitted, and the straight line obtained by fitting in each category is determined as the position of the shelf layer.

[0013] According to a third aspect of an embodiment of the present application, a storage medium storing a computer-readable program is provided, wherein the computer-readable program causes a processor coupled to the storage medium to execute the following method:

[0014] Detecting and identifying locations of a plurality of items in an input image;

[0015] Determining information about a predetermined location of each item and classifying the information about the predetermined location of each item;

[0016] The information of the predetermined positions of the goods in each category is fitted, and the straight line obtained by fitting in each category is determined as the position of the shelf layer.

[0017] One of the beneficial effects of the embodiments of the present application is that the positions of multiple goods in the input image are detected and identified from the input image, and after classifying the detected positions of the goods, a straight line is fitted to the position information within each category as the position of the shelf layer. Therefore, accurate detection of the shelf layer can be achieved based only on the image obtained by the monocular camera, and the cost is relatively low.

[0018] With reference to the following description and accompanying drawings, specific embodiments of the present application are disclosed in detail, indicating the manner in which the principles of the present application can be employed. It should be understood that the embodiments of the present application are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of the present application include many variations, modifications and equivalents.

[0019] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0020] It should be emphasized that the term "include / comprising" when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0022] Figure 1is a schematic diagram of a shelf detection device according to an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of identifying the positions of multiple goods in the input image according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of information on the predetermined location of each product in an embodiment of the present application;

[0025] Figure 4 Schematic diagram of a nested clustering tree obtained by hierarchical clustering according to an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of determining cluster distances in an embodiment of the present application;

[0027] Figure 6 Schematic diagram of a straight line fitted in the embodiment of the present application;

[0028] Figure 7 is a schematic diagram of determining a predetermined number of straight lines based on a gradient image according to an embodiment of the present application;

[0029] Figure 8 Schematic diagram of the detection window of an embodiment of the present application;

[0030] Figure 9 is a schematic diagram of a product detection device according to an embodiment of the present application;

[0031] Figure 10 is a schematic diagram of an electronic device according to an embodiment of the present application;

[0032] Figure 11 is a schematic block diagram of the system structure of the electronic device according to an embodiment of the present application;

[0033] Figure 12 is a schematic diagram of a shelf detection method according to an embodiment of the present application;

[0034] Figure 13 2 is a schematic diagram of a shelf detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements from the name, but do not indicate the spatial arrangement or temporal order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0036] In the embodiments of this application, the singular forms "a," "the," etc. include plural forms and should be broadly understood to mean "a" or "a type" rather than being limited to "one." Furthermore, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. Furthermore, the term "according to" should be understood to mean "at least in part based on...", and the term "based on" should be understood to mean "at least in part based on...", unless the context clearly indicates otherwise.

[0037] The above and other features of the present application will become apparent through the following description with reference to the accompanying drawings. In the description and the accompanying drawings, specific embodiments of the present application are disclosed in detail, which illustrate some embodiments in which the principles of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the present application includes all modifications, variations and equivalents that fall within the scope of the appended claims.

[0038] Embodiments of the first aspect

[0039] An embodiment of the present application provides a shelf detection device. Figure 1 FIG. 1 is a schematic diagram of a shelf detection device according to an embodiment of the present application. Figure 1 As shown, the shelf detection device 100 includes:

[0040] A first detection unit 101 detects and identifies positions of a plurality of items in an input image;

[0041] a first determining unit 102, which determines information about a predetermined location of each product and classifies the information about the predetermined location of each product;

[0042] The second determining unit 103 fits the information of the predetermined positions of the goods in each category, and determines the straight line obtained by fitting in each category as the position of the shelf layer.

[0043] In some embodiments, the above-mentioned input image can be a frame image in a video stream obtained by a camera, which can be obtained by a camera shooting from the area where the shelf is located, for example, by a set surveillance camera, and the camera is a monocular RGB camera. The input image can be a two-dimensional color image (RGB) or a two-dimensional grayscale image. The embodiments of the present application are not limited to this.

[0044] In some embodiments, the first detection unit may use a target detection model, including but not limited to a YOLO model, to detect multiple goods from an input image. For example, the input image is input into the YOLO model, and the output result is a bounding box of the multiple detected goods superimposed on the input image. The positions of the bounding boxes of the multiple goods in the input image are used as the positions of the multiple goods in the input image.

[0045] In some embodiments, the first determination unit 102 determines the information of the predetermined position of each product. The predetermined position can be the pixel position of the lower left corner, the pixel position of the upper right corner, the position of the bottom center, or other positions of the bounding box of the multiple products. The embodiments of the present application are not limited to this. The information of the predetermined position, that is, the information of the pixel position can be represented by the pixel coordinates in the input image coordinate system. The following is an example of the pixel position of the lower left corner of the bounding box as the predetermined position. The information of the predetermined position of each product can be the pixel coordinates (x, y) of the pixel point of the lower left corner of the bounding box corresponding to each product in the input image.

[0046] In some embodiments, the first determination unit may classify the information of the predetermined location of each of the goods, for example, using a hierarchical clustering method to classify the information of the predetermined location of each of the goods according to a set cluster distance. That is, for the information of the predetermined locations of the N determined goods, a hierarchical clustering method is used to classify (x1, y1), (x2, y2), ..., (xN, yN) according to the set cluster distance. The hierarchical clustering is to create a hierarchical nested cluster tree by calculating the similarity between data points, and there is no need to specify the number of cluster categories. After the clustering is completed, the cluster tree can be cut at the level corresponding to the set cluster distance according to the set cluster distance to obtain the number of categories corresponding to the level, and then determine the data points within each category. For the hierarchical clustering, reference can be made to the relevant technology, and the embodiments of the present application are not limited thereto. The cluster distance can be a pre-set fixed value, or determined using the following method.

[0047] In some embodiments, the number of categories obtained after hierarchical clustering is equivalent to the number of shelf layers. Therefore, the size of the cluster distance has a direct impact on the accuracy of shelf detection. To further improve shelf detection accuracy, embodiments of the present application also propose a method for setting the cluster distance. This cluster distance is directly related to the shelf layer distance. A first range of cluster distances can be determined based on the size of the input image in a first direction. Multiple distance values ​​within the first range are used to perform clustering processing in sequence. The second range of distances corresponding to the result with the most occurrences in the clustering results is counted. The minimum distance in the second range is used as the cluster distance. Detailed description is given below.

[0048] For example, the number of shelves in a supermarket is usually between 3 and 7. The first direction of the input image can be the height direction of the image that is approximately perpendicular to the shelves. The height dimension H of the input image is divided by 3 and 7 respectively to obtain the upper and lower limits of the first range 3 / H and 7 / H, that is, the first range is [7 / H, 3 / H]. Within this first range, sampling is performed at predetermined intervals to determine multiple distance values ​​H1, H2, ..., H P , first use H1,H2,...,H P Perform clustering processing to obtain clustering results, which are the number of categories. Count the number of categories that appear the most times in the clustering results in different clustering processes, Maxnum. According to Maxnum, the second range of the corresponding distance value is determined. That is, assuming H P-3 ,H P-2 ,H P-1 ,H P The clustering results of these four distance values ​​are all Maxnum, so the second range is H P-3 ,H P-2 ,H P-1 ,H P , the minimum distance H in the second range P-3 as the clustering distance.

[0049] Figure 2 is a schematic diagram of identifying the positions of multiple goods in the input image according to an embodiment of the present application. Figure 3 This is a schematic diagram of information about the predetermined location of each product (taking the lower left corner as an example) in an embodiment of the present application. Figure 4 Schematic diagram of a nested clustering tree obtained by hierarchical clustering in an embodiment of the present application. Figure 5 This is a schematic diagram of determining cluster distances in an embodiment of the present application, such as Figure 5 As shown, sampling is performed within the first range [200, 1000] to determine multiple distance values ​​of 200, 300, 400, 500, 600, 700, 800, 900, and 1000. For each distance value, according to the nested clustering tree, the corresponding number of categories is 9, 7, 7, 6, 5, 5, 5, 5, 5, among which 5 appears the most times, that is, 5 is the stable number of clustered categories. The second range corresponding to 5 is 600, 700, 800, 900, and 1000. The smallest distance 600 in the second range is used as the clustering distance, and the final number of clustered categories is 5, that is, the number of shelf layers is 5.

[0050] In some embodiments, the number of cluster categories is determined based on a set cluster distance, and the information about the predetermined locations of each item is then classified to obtain information about the predetermined locations of Maxnum categories. The second determination unit 103 performs a linear fit (fitting) on ​​the information about the predetermined locations of the items within each category, and determines the fitted straight line within each category as the shelf layer location. The linear fitting method can be a least squares method, but this embodiment of the present application is not limited thereto. The specific fitting method can be referenced to related technologies and will not be further described here. Figure 6 is a straight line diagram obtained by fitting in the embodiment of the present application, such as Figure 6 As shown, the positions of the straight lines represent the positions of the detected shelf layers in the input image.

[0051] In some embodiments, after determining the shelf layer in the input image by linear fitting, the direction of the shelf layer (the direction of the fitted straight line) may optionally be corrected. The apparatus may further include:

[0052] The shelf direction determining unit 104 generates a depth image according to the input image, determines the shelf direction according to the depth image, and corrects the position of the shelf layer according to the shelf direction.

[0053] In some embodiments, the shelf direction determination unit can use an existing monocular depth estimation model, such as AdaBins, to obtain a depth image (grayscale image, different grayscale values ​​reflect different depths), that is, input the input image into the monocular depth estimation model to obtain the depth image, perform edge feature detection in the depth image, obtain a first edge feature image corresponding to the depth image, and determine the shelf direction based on the first edge feature image, but the embodiment of the present application is not limited to this. The shelf direction determination unit can also calculate the gradient image of the depth image in the first direction (height direction). Compared with the original input image and depth image, the depth gradient in the height direction can more significantly reflect the depth change of the image. Therefore, the shelf layer can be found more clearly, and then edge feature detection is performed on the gradient image to obtain a second edge feature image, and the shelf direction is determined based on the second edge feature image.

[0054] In some embodiments, determining the shelf direction based on the second edge feature image includes: processing edge features detected in the second edge feature image to generate a predetermined number of straight lines; determining vanishing points based on the predetermined number of straight lines, determining the direction of each shelf layer based on the vanishing points, and correcting the position of each shelf layer based on the direction of each shelf layer (determining the shelf direction based on the first edge feature image is a similar implementation method and will not be repeated here). The following details are described in detail.

[0055] In some embodiments, the edge features may be extracted using an edge detection operator, wherein the edge detection operator includes at least one of the following operators: a Prewitt operator, a Laplacian operator, a Roberts crossover operator, a Canny operator, and a Sobel operator. Figure 7 is a schematic diagram of determining a predetermined number of straight lines based on a gradient image in an embodiment of the present application, Figure 7 Where a is the aforementioned gradient image, and b is the second edge feature image, wherein the edge features include multiple scattered contours, and the edge features need to be processed first. The processing includes: Hough detection, through which the detected scattered contours are merged into longer line segments. For the Hough detection, you can refer to the relevant technology and will not go into details here. Since there are some edge features in the merged line segments that obviously do not belong to the shelf (such as vertical line segments, or line segments with larger slopes), these edge features need to be filtered out. For example, the slope of the merged line segment can be compared with the preset slope range. If the slope of the merged line segment is not within the slope range, the line segment is deleted. After filtering, the line segment merging process can be performed again to merge short lines and long lines that are close in distance and have similar slopes to generate a predetermined number of straight lines (or line segments), such as Figure 7 As shown in c.

[0056] In some embodiments, the predetermined number may not be completely consistent with the number of categories classified by the first determination unit 102 (i.e., the number of shelf layers). Therefore, in order to accurately correct the direction of each shelf layer, the vanishing point can be determined based on the predetermined number of straight lines, the direction of each shelf layer can be determined based on the vanishing point, and the position of the shelf layer can be corrected based on the direction of each shelf layer. Figure 7 As shown in c in FIG, the predetermined number of straight lines are not parallel lines. Therefore, the vanishing points of the predetermined number of straight lines in the depth image or the gradient image can be determined. Based on the vanishing points, the vanishing line closest to the straight lines fitted by the second determination unit 103 in the height direction is determined. The direction of the closest vanishing line is used as the direction of the shelf, and the direction of the corresponding straight line is corrected, that is, the slope of the corresponding straight line is adjusted to be the same as the slope of the closest vanishing line.

[0057] For example, for the fitted straight line L1, the vanishing line V1 closest to the straight line L1 in the height direction is determined based on the vanishing points determined by a predetermined number of straight lines, and the slope of the straight line L1 is adjusted to be consistent with the slope of the vanishing line V1. For the fitted straight line L2, the vanishing line V2 closest to the straight line L2 in the height direction is determined based on the vanishing points determined by a predetermined number of straight lines, and the slope of the straight line L2 is adjusted to be consistent with the slope of the vanishing line V2. And so on. Maxnum, the vanishing point determined by a predetermined number of straight lines is determined to be the same as the straight line L Maxnum The nearest vanishing line V in the height direction Maxnum , the straight line L Maxnum The slope is adjusted to match the vanishing line V Maxnum The slope of is consistent, thus completing the correction of the shelf layer position.

[0058] In some embodiments, if the predetermined number of straight lines are completely parallel and there is no vanishing point, the directions of the straight lines fitted by the second determination unit 103 can be corrected according to the directions of the parallel straight lines, that is, the straight lines fitted by the second determination unit 103 are also parallel, and their directions (slopes) are the same as the directions (slopes) of the predetermined number of straight lines, which will not be repeated here.

[0059] In some embodiments, after correcting the shelf layer position, it is also possible to detect whether the goods on the shelf layer are out of stock and the quantity of goods out of stock. Optionally, the device may further include:

[0060] The product detection unit 105 uses the edge of the area of ​​the product detected by the first detection unit in the input image as the starting position of the detection window, determines the overlapping area between the area occupied by another adjacent product on one side of the product and the detection window; and determines whether there is a shortage of the product between the product and the other product based on the relationship between the overlapping area and a first threshold.

[0061] In some embodiments, the size of the detection window can be predetermined. The width and height of the detection window can be approximately the same as, or slightly smaller than, the width and height of the aforementioned bounding box of the product. This embodiment of the present application is not limited thereto. The edge of the region of the product in the input image can be a side of the bounding box of the product, such as the right side in the height direction. The region occupied by another adjacent product can be the region occupied by the bounding box of another product adjacent to the right side in the height direction of the bounding box of the product. Figure 8 This is a schematic diagram of the detection window of the embodiment of the present application. Figure 8As shown, the bounding boxes of two adjacent items detected by the first detection unit are bounding box B1 and bounding box B2, respectively. The right side of bounding box B1 in the height direction serves as the starting position of the detection window. The overlapping area S between the detection window and bounding box B2 is determined, and the relationship between S and a first threshold is used to determine whether the item B1 and the other item B2 are out of stock. Items B1 and B2 are merely two examples of items detected by the first detection unit. The above method is used to determine whether each pair of adjacent items among the multiple items detected by the first detection unit is out of stock. Furthermore, the adjacent items can be adjacent items on the same shelf layer as previously detected, or in other words, adjacent items along the length of the shelf.

[0062] In some embodiments, whether the product B1 and the other product B2 are out of stock is determined based on the relationship between S and a first threshold value. When S is greater than or equal to the first threshold value, it is determined that the product B1 and the other product B2 are not out of stock. When S is less than the first threshold value, or S is equal to 0 (there is no overlapping area or the detection window does not overlap with the bounding box B2), it is determined that the product B1 and the other product B2 are out of stock. The first threshold value can be determined as needed, for example, set to the area of ​​a bounding box, or slightly smaller than the area of ​​a bounding box, or set to other values. The embodiments of the present application are not limited to this.

[0063] In some embodiments, the product detection unit is further configured to calculate a first distance between the center of the area occupied by the product and the other product and a distance to the farthest edge, and determine the number of out-of-stock items between the product and the other product based on the first distance and the farthest edge distance. For example, the product detection unit calculates a ratio of the first distance to the farthest edge distance, and determines the number of out-of-stock items between the product and the other product based on the relationship between the ratio and a second threshold. Figure 8As shown, the first distance is equal to the distance D1 between the center of the bounding box B1 and the center of the bounding box B2, and the farthest edge distance is equal to the distance D2 between the left side of the bounding box B1 in the height direction and the right side of the bounding box B2 in the height direction. Assuming that the out-of-stock quantity is n, and assuming that the horizontal dimensions of the goods are roughly the same, the distance D1 / D2 is between n / n+1 and n+1 / n+2. Therefore, the number of out-of-stock items between the product and the other product can be determined based on the relationship between D1 / D2 and the second threshold (n / n+1, where n is an integer greater than or equal to 1). Different second thresholds can be pre-set to correspond to different out-of-stock quantities. For example, when D1 / D2 is greater than 0.5 but less than 0.67, it is determined that one unit is out of stock; when D1 / D2 is greater than or equal to 0.67 but less than 0.75, it is determined that two units are out of stock; when D1 / D2 is greater than or equal to 0.75 but less than 0.8, it is determined that three units are out of stock; and so on. Examples are not given here one by one. The larger the value of D1 / D2, the greater the out-of-stock quantity. The second threshold can be determined as needed and is not limited to this embodiment of the present application.

[0064] In some embodiments, the product detection unit can be provided independently of the shelf detection device. That is, product detection and shelf detection can be implemented independently or in combination. The embodiments of the present application are not limited to this.

[0065] Figure 9 Schematic diagram of a product inspection device according to an embodiment of the present application. Figure 9 As shown, the product detection device 900 includes:

[0066] A first detection unit 901 detects and identifies positions of multiple items in an input image;

[0067] The product detection unit 902 uses the edge of the area of ​​the detected product in the input image as the starting position of the detection window, determines the overlapping area between the area occupied by another adjacent product on one side of the product and the detection window, and determines whether there is a shortage of the product between the product and the other product based on the relationship between the overlapping area and a first threshold.

[0068] The implementation of the first detection unit 901 and the product detection unit 902 may refer to the aforementioned first detection unit 101 and the product detection unit 105 , and will not be repeated here.

[0069] It is worth noting that the above description only describes the components or modules related to the present application, but the present application is not limited thereto. The shelf detection device 100 and the product detection device 900 may also include other components or modules. For the specific contents of these components or modules, please refer to the relevant art.

[0070] To keep it simple, Figure 1 and Figure 9 The connection relationships and signal paths between various components or modules are shown for illustrative purposes only. However, those skilled in the art will appreciate that various related technologies, such as bus connections, can be employed. The aforementioned components or modules can be implemented using hardware such as processors and memory; this is not a limitation of the present invention.

[0071] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0072] It can be seen from the above embodiments that the positions of multiple goods in the input image are detected and identified from the input image, and after the detected positions of the goods are classified, the straight line obtained by fitting the position information within each category is used as the position of the shelf layer. Therefore, accurate detection of the shelf layer (especially for scenes where goods are densely stacked) can be achieved only based on the image obtained by the monocular camera, and the cost is relatively low.

[0073] Embodiments of the second aspect

[0074] The embodiment of the present application also provides an electronic device, Figure 10 A schematic diagram of an electronic device according to an embodiment of the present application. Figure 10 As shown, the electronic device 1000 includes a shelf detection device 1001 (or includes a product detection device 900, not shown). The structure and function of the shelf detection device 1001 are the same as those of the shelf detection device 100 in the embodiment of the first aspect. For specific details, please refer to the description in the embodiment of the first aspect, and will not be repeated here.

[0075] Figure 11 1 is a schematic block diagram of the system structure of the electronic device according to the embodiment of the present application. Figure 11 As shown, electronic device 1100 may include a processor 1101 and a memory 1102; the memory 1102 is coupled to the processor 1101. This figure is exemplary; other types of structures may be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0076] In some embodiments, processor 1101 includes at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0077] like Figure 11As shown, the electronic device 1100 may further include: an input device 1103, a display 1104, and a power supply 1105. In one embodiment, the functionality of the shelf detection device may be integrated into the processor 1101. The processor 1101 may be configured to: detect and identify the positions of multiple items in an input image; determine information about the predetermined position of each item and classify the information about the predetermined position of each item; fit the information about the predetermined positions of items within each category, and determine the fitted straight line within each category as the position of the shelf layer. Alternatively, the processor 1101 may be configured to: detect and identify the positions of multiple items in an input image; use the edge of the detected area of ​​the item in the input image as the starting position of a detection window, determine the overlapping area between the area occupied by another item adjacent to the item and the detection window; and determine whether the item is out of stock between the item and the other item based on the relationship between the overlapping area and a first threshold. For other embodiments, please refer to the first aspect of the embodiment for details and will not be further described here.

[0078] In another embodiment, the shelf detection device or the product detection device can be configured separately from the processor 1101. For example, the shelf detection device or the product detection device can be configured as a chip connected to the processor 1101, and the functions of the shelf detection device or the product detection device can be realized through the control of the processor 1101.

[0079] In this embodiment, the electronic device 1100 does not necessarily have to include Figure 11 All parts shown in .

[0080] like Figure 11 As shown, the processor 1101 is sometimes also referred to as a controller or operation control, which may include a microprocessor or other processor device and / or logic device. The processor 1101 receives inputs and controls the operations of various components of the electronic device 1100 .

[0081] The memory 1102 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. Furthermore, the processor 1101 may execute the program stored in the memory 1102 to implement information storage or processing. The functions of the other components are similar to those of the prior art and are not described in detail here. The various components of the electronic device 1100 may be implemented using dedicated hardware, firmware, software, or a combination thereof without departing from the scope of this application.

[0082] It can be seen from the above embodiments that the positions of multiple goods in the input image are detected and identified from the input image. After the detected positions of the goods are classified, the straight line obtained by fitting the position information within each category is used as the position of the shelf layer. Therefore, accurate detection of the shelf layer can be achieved only based on the image obtained by the monocular camera, and the cost is relatively low.

[0083] Embodiments of the third aspect

[0084] An embodiment of the present application also provides a shelf detection method, which corresponds to the shelf detection device of the embodiment of the first aspect. Figure 12 FIG. 1 is a schematic diagram of a shelf detection method according to an embodiment of the present application. Figure 12 As shown, the method includes:

[0085] 1201, detecting and identifying positions of a plurality of goods in an input image;

[0086] 1202, determining the information of the predetermined location of each product and classifying the information of the predetermined location of each product;

[0087] 1203. Fit the information of the predetermined positions of the goods in each category, and determine the straight line obtained by fitting in each category as the position of the shelf layer.

[0088] In this embodiment, the specific implementation method of each of the above steps is the same as that described in the embodiment of the first aspect and will not be repeated here.

[0089] Figure 13 FIG. 1 is a schematic diagram of a shelf detection method according to an embodiment of the present application. Figure 13 As shown, the method includes:

[0090] 1301, obtain input image, execute 1302 and 1304 respectively;

[0091] 1302, detecting and identifying positions of a plurality of (N) items in the input image, where N is an integer greater than 1;

[0092] 1303, determining the information of the predetermined location of each product and classifying the information of the predetermined location of each product; fitting the information of the predetermined location of the products in each category, and determining the straight line obtained by fitting in each category as the location of the shelf layer;

[0093] 1304, generating a depth image according to the input image, and determining a shelf direction according to the depth image;

[0094] 1305 , correcting the position of the shelf layer obtained in 1303 according to the shelf direction determined in 1304 .

[0095] 1306, determine whether X is less than N. If the result is yes, execute 1307, otherwise terminate;

[0096] 1307, according to the product B detected in 1302 X The edge of the area in the input image is used as the starting position of the detection window to determine another adjacent product B on one side of the product. X+1 The overlap area between the occupied area and the detection window; initially X=1;

[0097] 1308 , determining whether the product and the other product are out of stock based on the relationship between the overlapping area and the first threshold value. If the product is out of stock, execute 1309 ; otherwise, X=X+1 and return to 1306 ;

[0098] 1309. Calculate a first distance between the centers of the area occupied by the product and the other product and a distance to the farthest edge, and determine the number of out-of-stock items between the product and the other product based on the first distance and the farthest edge distance.

[0099] The implementation of 1301 to 1309 can refer to the embodiment of the first aspect and will not be repeated here.

[0100] The above 1302, 1306 to 1309 can be implemented independently as product inspection methods and will not be described in detail here.

[0101] The above only describes the various steps or processes related to the present application, but the present application is not limited thereto. The shelf detection method may also include other steps or processes. For the specific content of these steps or processes, reference may be made to the prior art. In addition, the above only uses some structures of the model used in the shelf detection method as an example to illustrate the embodiments of the present application, but the present application is not limited to these structures. These structures may also be appropriately modified, and the implementation of these modifications should all be included within the scope of the embodiments of the present application.

[0102] The above embodiments are merely exemplary of the present invention, but the present invention is not limited thereto. Appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0103] It can be seen from the above embodiments that the positions of multiple goods in the input image are detected and identified from the input image. After the detected positions of the goods are classified, the straight line obtained by fitting the position information within each category is used as the position of the shelf layer. Therefore, accurate detection of the shelf layer can be achieved only based on the image obtained by the monocular camera, and the cost is relatively low.

[0104] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a shelf detection device, a product detection device, or an electronic device, the program causes the computer to execute the shelf detection method or product detection method described in the embodiment of the third aspect in the shelf detection device, the product detection device, or the electronic device.

[0105] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the shelf detection method or product detection method described in the embodiment of the third aspect in a shelf detection device or product detection device or electronic device.

[0106] The shelf / product detection device or electronic device described in the embodiments of the present application may be directly implemented as hardware, a software module executed by a processor, or a combination of the two. Figure 1 and Figure 9 One or more of the functional block diagrams shown in and / or one or more combinations of functional block diagrams may correspond to various software modules of a computer program flow or to various hardware modules. These software modules may correspond to Figures 12 to 13 These hardware modules can be implemented by solidifying these software modules using, for example, a field programmable gate array (FPGA).

[0107] The software module may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to the processor so that the processor can read information from and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and storage medium may be located in an ASIC. The software module may be stored in the memory of the mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the electronic device uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0108] against Figure 1 and Figure 9 One or more of the functional block diagrams and / or one or more combinations of the functional block diagrams described herein may be implemented as a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described in this application. Figure 1 and Figure 9 One or more of the described functional block diagrams and / or one or more combinations of the functional block diagrams may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0109] The present application has been described above in conjunction with specific embodiments. However, those skilled in the art should understand that these descriptions are merely illustrative and are not intended to limit the scope of protection of the present application. Those skilled in the art may make various modifications and variations to the present application based on the spirit and principles of the present application, and such modifications and variations are also within the scope of the present application.

[0110] This application also provides the following notes:

[0111] 1. A shelf detection method, characterized in that the method comprises:

[0112] a first detection unit, configured to detect and identify positions of a plurality of items in an input image;

[0113] a first determining unit, which determines information about a predetermined location of each item and classifies the information about the predetermined location of each item;

[0114] The second determining unit fits the information of the predetermined positions of the goods in each category, and determines the straight line obtained by fitting in each category as the position of the shelf layer.

[0115] 2. The method according to Note 1, wherein a hierarchical clustering method is used to classify the information of the predetermined location of each product according to a set clustering distance.

[0116] 3. The method according to Note 2, wherein the method further comprises: determining the cluster distance, comprising:

[0117] determining a first range of clustering distances according to a size of the input image in a first direction;

[0118] Perform clustering processing in sequence using multiple distance values ​​within the first range, and count the second range of distances corresponding to the result with the largest number of occurrences in the clustering results;

[0119] The minimum distance in the second range is used as the clustering distance.

[0120] 4. The method according to Supplementary Note 1, further comprising:

[0121] A depth image is generated according to the input image, a shelf direction is determined according to the depth image, and a position of the shelf layer is corrected according to the shelf direction.

[0122] 5. The method according to Supplementary Note 4, wherein generating a depth image based on the input image, determining a shelf direction based on the depth image, and correcting the position of the shelf layer based on the shelf direction comprises:

[0123] Calculating a gradient image of the depth image in a first direction, performing edge feature detection on the gradient image, processing the detected edge features, and generating a predetermined number of straight lines;

[0124] According to the predetermined number of straight lines, a vanishing point is determined, the direction of each shelf layer is determined according to the vanishing point, and the position of the shelf layer is corrected according to the direction of each shelf layer.

[0125] 6. The method according to Supplementary Note 1, wherein the method further comprises:

[0126] The edge of the area of ​​the detected goods in the input image is used as the starting position of the detection window, and the overlapping area between the area occupied by another adjacent product on one side of the goods and the detection window is determined; and whether there is a shortage of goods between the goods and the other goods is determined based on the relationship between the overlapping area and a first threshold.

[0127] 7. The method according to Note 6, wherein the method further includes: calculating a first distance and a farthest edge distance between the center of the area occupied by the product and the other product, and determining the number of out-of-stock items between the product and the other product based on the first distance and the farthest edge distance.

[0128] 8. The method according to Note 7, wherein determining the number of out-of-stock items between the product and the other product based on the first distance and the farthest edge distance includes: calculating the ratio of the first distance to the farthest edge distance, and determining the number of out-of-stock items between the product and the other product based on the relationship between the ratio and a second threshold value.

[0129] 9. A method for detecting the status of goods, characterized in that the method comprises:

[0130] Detecting and identifying locations of a plurality of items in an input image;

[0131] Using the edge of the detected product area in the input image as the starting position of the detection window, determining the overlapping area between the area occupied by another adjacent product on one side of the product and the detection window;

[0132] Whether the product and the other product are out of stock is determined based on a relationship between the overlapping area and a first threshold.

[0133] 10. A storage medium storing a computer-readable program, wherein the computer-readable program causes a processor coupled to the storage medium to execute the following method:

[0134] Detecting and identifying locations of a plurality of items in an input image;

[0135] Using the edge of the detected product area in the input image as the starting position of the detection window, determining the overlapping area between the area occupied by another adjacent product on one side of the product and the detection window;

[0136] Whether the product and the other product are out of stock is determined based on a relationship between the overlapping area and a first threshold.

Claims

1. A shelf detection device, characterized in that: The device comprises: A first detection unit, which detects and identifies positions of a plurality of goods in an input image from the input image; a first determining unit, which determines information of a predetermined location of each commodity and classifies the information of the predetermined location of each commodity; The second determining unit fits the information of the predetermined positions of the goods in each category, and determines the straight line obtained by fitting in each category as the position of the shelf layer.

2. The device according to claim 1, wherein: The first determination unit uses a hierarchical clustering method to classify the information of the predetermined location of each product according to a set clustering distance.

3. The device according to claim 2, wherein: The first determining unit is further used to determine the cluster distance, including: Determine a first range of clustering distances according to a size of the input image in a first direction; Using multiple distance values ​​within the first range to perform clustering processing in sequence, and counting the second range of distances corresponding to the result with the largest number of occurrences in the clustering results; The minimum distance in the second range is used as the clustering distance.

4. The device according to claim 1, wherein: The device also includes: A shelf direction determining unit generates a depth image according to the input image, determines the shelf direction according to the depth image, and corrects the position of the shelf layer according to the shelf direction.

5. The device according to claim 4, wherein: The shelf direction determination unit further calculates a gradient image of the depth image in the first direction, performs edge feature detection on the gradient image, processes the detected edge features, and generates a predetermined number of straight lines; According to the predetermined number of straight lines, a vanishing point is determined, the direction of each shelf layer is determined according to the vanishing point, and the position of the shelf layer is corrected according to the direction of each shelf layer.

6. The device according to claim 1, wherein: The device also includes: A product detection unit, which uses the edge of the area of ​​the product detected by the first detection unit in the input image as the starting position of the detection window, determines the overlapping area between the area occupied by another adjacent product on one side of the product and the detection window; and determines whether there is a shortage of the product between the product and the other product based on the relationship between the overlapping area and a first threshold.

7. The device according to claim 6, wherein: The product detection unit is further used to calculate a first distance between the centers of the areas occupied by the product and the other product and a farthest edge distance, and determine the number of out-of-stock items between the product and the other product based on the first distance and the farthest edge distance.

8. The device according to claim 7, wherein: The product detection unit calculates a ratio of the first distance to the farthest edge distance, and determines the number of out-of-stock products between the product and the other product according to a relationship between the ratio and a second threshold.

9. A shelf detection method, characterized in that: The method comprises: Detecting and identifying locations of a plurality of items in an input image; Determine the information of the scheduled location of each item and classify the information of the scheduled location of each item; The information of the predetermined positions of the goods in each category is fitted, and the straight line obtained by fitting in each category is determined as the position of the shelf layer.

10. A storage medium storing a computer-readable program, wherein the computer-readable program causes a processor coupled to the storage medium to execute the following method: Detecting and identifying locations of a plurality of items in an input image; Determine the information of the predetermined location of each product, and classify the information of the predetermined location of each product; The information of the predetermined positions of the goods in each category is fitted, and the straight line obtained by fitting in each category is determined as the position of the shelf layer.

Images