Detection device and detection method for commodity shelf

Abstract

To provide a detection device and a detection method for a commodity shelf.SOLUTION: A detection device includes a first detection unit that detects and recognizes, from an input image, the positions of a plurality of commodities in the input image, a first determination unit that determines information about prescribed positions of the respective commodities and classifies information about the prescribed positions of the respective commodities, and a second determination unit that performs fitting of information about the prescribed position of the commodities in each category, and determines, as the position of a shelf stage, a straight line obtained by fitting in each category.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the field of image processing. [Background technology]

[0002] Conventional retail supermarkets and convenience stores all require dedicated employees, and labor costs are relatively high. In related technologies, employees are required to check the status of products on the shelves, whether they are replenished, out of stock, or moved by customers. Currently, it is possible to automatically detect the status of products based on captured images, but when products are densely stacked, it is difficult to recognize the shelf level from the image, and it is also not possible to detect the status of products using position information corresponding to the products on the shelf. In addition, a method for detecting shelf levels based on vanishing points has also been proposed, but when an image has a complex texture, the detection result is not stable.

[0003] It should be noted that the above description of the background art is provided for the purpose of making the technical solutions of the present invention clear and complete, and is described for the understanding of persons skilled in the art, and these technical solutions are merely described as part of the background art of the present invention, and are not well known to persons skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of at least one of the above problems, embodiments of the present invention provide a shelf detection apparatus and method. [Means for solving the problem]

[0005] In a first aspect of an embodiment of the present invention, a product shelf detection device is provided, which includes a first detection unit that detects and recognizes positions of a plurality of products in an input image from the input image, a first determination unit that determines information of a predetermined position of each product and classifies the information of the predetermined position of each product, and a second determination unit that fits the information of the predetermined positions of products in each category and determines a straight line obtained by fitting in each category as the position of the shelf level.

[0006] In a second aspect of an embodiment of the present invention, a method for detecting product shelves is provided, the method including the steps of detecting and recognizing positions of a plurality of products in an input image from the input image, determining information of a predetermined position of each product and classifying the information of the predetermined position of each product, and fitting the information of the predetermined position of products in each category and determining a straight line obtained by fitting in each category as the position of the shelf level.

[0007] In a third aspect of an embodiment of the present invention, there is provided a storage medium having a computer-readable program stored therein, the computer-readable program causing a processor connected to the storage medium to execute a method including the steps of: detecting and recognizing positions of a plurality of products in an input image from the input image; determining information of a predetermined position of each product and classifying the information of the predetermined position of each product; and fitting the information of the predetermined position of the products in each category and determining a straight line obtained by fitting in each category as the position of the shelf level.

[0008] One of the advantageous effects of the embodiment of the present invention is as follows: By detecting and recognizing the positions of multiple products in an input image from the input image, classifying the positions of the detected products, and fitting the position information for each category to obtain a straight line as the position of the shelf level, the shelf level can be detected accurately and at low cost based only on the image captured by the monocular camera.

[0009] Certain embodiments of the invention are disclosed in detail, as set forth in the following description and the drawings, which show how the principles of the invention may be employed, but are not intended to be limiting in scope, and include all modifications, alterations, and equivalents within the spirit and content of the appended claims.

[0010] Features described and / or shown in one embodiment may be used in the same or similar manner in one or many other embodiments and may be combined with or substituted for features in the other embodiments.

[0011] It should be noted that the term "comprises" when used in the present text means the presence of a feature, element, step or component and does not exclude the presence or addition of one or more other features, elements, steps or components. [Brief description of the drawings]

[0012] The drawings included herein are for understanding the embodiments of the present invention, constitute a part of this specification, and are for illustrating the embodiments of the present invention, and together with the description of the text, explain the principles of the present invention. Note that the drawings described herein are merely for illustrating the embodiments of the present invention, and those skilled in the art can easily obtain other drawings based on these drawings. [Figure 1] 1 is a schematic diagram of an example of a product shelf detection device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram of the positions of multiple recognized products in an input image according to an embodiment of the present invention; [Diagram 3] FIG. 4 is a schematic diagram of information on a predetermined position of each commodity according to an embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram of a nested cluster tree obtained by hierarchical clustering according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a schematic diagram of determining cluster distance according to an embodiment of the present invention; [Figure 6]FIG. 13 is a schematic diagram of a straight line obtained by fitting according to an embodiment of the present invention. [Figure 7] FIG. 4 is a schematic diagram of determining a predetermined number of straight lines based on a gradient image according to an embodiment of the present invention; [Figure 8] 4 is a schematic diagram of a detection window according to an embodiment of the present invention. [Figure 9] 1 is a schematic diagram of a commodity detection device according to an embodiment of the present invention; [Figure 10] 1 is a schematic diagram of an example of an electronic device according to an embodiment of the present invention. [Figure 11] 1 is a schematic block diagram of an example of a system configuration of an electronic device according to an embodiment of the present invention. [Figure 12] FIG. 2 is a schematic diagram of an example of a product shelf detection method according to an embodiment of the present invention. [Figure 13] FIG. 2 is a schematic diagram of an example of a product shelf detection method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In the embodiments of the present invention, the terms "first" and "second" are used to distinguish different elements by name, and do not imply the spatial arrangement or temporal order of these elements, and the elements are not limited to these terms. The term "and / or" includes any one or more of the listed terms and combinations thereof. The terms "including", "comprising", and "having" refer to the presence of stated features, elements, components or members, but do not exclude the presence or addition of one or more other features, elements, components or members.

[0014] In the present embodiment, the singular forms "a", "the", "the" and the like include the plural, meaning "a kind" or "a class", and are not limited to "one". Additionally, the term "said" includes both the singular and the plural, unless the context clearly dictates otherwise. Additionally, the term "according to" means "at least in part depending on" and the term "based on" means "at least in part based on", unless the context clearly dictates otherwise.

[0015] The above and other features of the present invention can be understood from the drawings and the following description. The specification and drawings disclose certain embodiments of the present invention, i.e., some embodiments according to the principles of the present invention. It is to be understood that the present invention is not limited to the described embodiments, and the present invention includes all modifications, variations, and equivalents within the scope of the appended claims.

[0016] <Example 1> An embodiment of the present invention provides a product shelf detection device. Fig. 1 is a schematic diagram of an example of a product shelf detection device according to an embodiment of the present invention. As shown in Fig. 1, the product shelf detection device 100 includes the following units.

[0017] The first detection unit 101 detects and recognizes the positions of a plurality of products in an input image.

[0018] The first determination unit 102 determines information on the predetermined position of each product, and classifies the information on the predetermined position of each product.

[0019] The second determination unit 103 performs fitting on information on a predetermined position of the product in each category, and determines a straight line obtained by fitting in each category as the position of the shelf level.

[0020] In some embodiments, the input image may be a frame image in a video stream captured by a camera, or may be captured from a camera capturing an area in which the shelves are located, for example, a security camera installed in the area. The camera may be a monocular RGB camera. The input image may be a two-dimensional color image (RGB) or a two-dimensional grayscale image, but the embodiments of the present invention are not limited thereto.

[0021] In some embodiments, the first detector may detect multiple products from the input image using a target detection model. The target detection model may include, but is not limited to, a YOLO model. For example, the input image is input to the YOLO model, and the output result displays bounding boxes of the detected multiple products superimposed on the input image. The positions of the bounding boxes of the multiple products in the input image are the positions of the multiple products in the input image.

[0022] In some embodiments, the first determination unit 102 determines information of a predetermined position of each product. The predetermined position may be a pixel position of the lower left corner of the bounding frame of a plurality of products, a pixel position of the upper right corner, a position of the center of the lower side, or other positions, but the embodiments of the present invention are not limited thereto. The information of the predetermined position, i.e., the information of the pixel position, may be expressed by pixel coordinates in the coordinate system of the input image. The following describes as an example that the predetermined position is the pixel position of the lower left corner of the bounding frame, and the information of the predetermined position of each product may be the pixel coordinates (x, y) in the input image of the pixel point of the lower left corner of the bounding frame corresponding to each product.

[0023] In some embodiments, the first determination unit may classify the information of the predetermined position of each product, for example, using a hierarchical clustering method to classify the information of the predetermined position of each product based on the set cluster distance. In other words, for the determined information of the predetermined positions of the N products, using a hierarchical clustering method, classify (x1, y1), (x2, y2), ..., (xN, yN) based on the set cluster distance. The hierarchical clustering creates a hierarchical nested cluster tree by calculating the similarity between data points, and it is not necessary to specify the number of clustering categories. After the clustering is completed, a cluster tree is cut out at a hierarchy corresponding to the cluster distance based on the set cluster distance, a number of categories corresponding to the hierarchy is obtained, and the data points in each category are determined. The hierarchical clustering may refer to related art, and the embodiment of the present invention is not limited thereto, and the cluster distance may be a fixed value set in advance, or may be determined as follows.

[0024] In some embodiments, the number of categories obtained after hierarchical clustering corresponds to the number of shelves, so the size of the cluster distance directly affects the accuracy of product shelf detection. In order to further improve the accuracy of product shelf detection, the embodiment of the present invention further provides a method for setting a cluster distance, and the cluster distance is directly correlated to the distance of the shelves. Based on the size of the input image in the first direction, a first range of cluster distances is determined, and a clustering process is sequentially performed using a plurality of distance values ​​within the first range, and statistics of a second range of distances corresponding to the result with the highest frequency of occurrence among the clustering results are taken, and the minimum distance within the second range is set as the cluster distance. The following will be described in detail.

[0025] For example, the number of shelves in a supermarket is usually 3 to 7, and the first direction of the input image may be the height direction of the image that is approximately perpendicular to the shelves. The size H of the input image in the height direction is divided by 3 and 7 to obtain the upper and lower limits of the first range, 3 / H and 7 / H, respectively, i.e., the first range is [7 / H, 3 / H]. Sampling is performed at predetermined intervals within the first range to obtain a plurality of distance values ​​H 1 ,H 2 ,…,H P Determine and H 1 ,H 2 ,…,H P Then, in the different clustering processes, statistics are taken of the number Maxnum of the most frequently occurring categories in the clustering results, and a second range of the corresponding distance values ​​is determined based on the Maxnum. That is, H P-3 , H P-2 , H P-1 , H P Assuming that the clustering results of the four distance values ​​are all Maxnum, the second range is H P-3 , H P-2 , H P-1 , H P The smallest distance H P-3 Let be the cluster distance.

[0026] FIG. 2 is a schematic diagram of the positions of recognized multiple products in an input image according to an embodiment of the present invention. FIG. 3 is a schematic diagram of the information of the predetermined position (taking the lower left corner as an example) of each product according to an embodiment of the present invention. FIG. 4 is a schematic diagram of a nested cluster tree obtained by hierarchical clustering according to an embodiment of the present invention. FIG. 5 is a schematic diagram of determining a cluster distance according to an embodiment of the present invention. As shown in FIG. 5, sampling is performed within the first range [200, 1000], and a number of distance values ​​200, 300, 400, 500, 600, 700, 800, 900, 1000 are determined, and the corresponding number of categories obtained based on the nested cluster tree for each distance value is 9, 7, 7, 6, 5, 5, 5, 5, 5 respectively. Here, 5 has the highest occurrence frequency, that is, 5 is the stable number of categories obtained by clustering. The second range corresponding to 5 is 600, 700, 800, 900, 1000, and if the smallest distance within the second range, 600, is taken as the cluster distance, the final number of categories in the cluster is 5, and the number of shelves is 5.

[0027] In some embodiments, the number of categories of the cluster is determined based on the set cluster distance, the information of the predetermined position of each product is classified, and the information of the predetermined position of Maxnum categories is obtained. The second determination unit 103 performs fitting (linear fitting) on ​​the information of the predetermined position of the product in each category, and determines the straight line obtained by fitting in each category as the position of the shelf. The linear fitting method may be the least squares method, but the embodiment of the present invention is not limited to this, and the fitting method may refer to related art, and the description thereof will be omitted here. FIG. 6 is a schematic diagram of a straight line obtained by fitting according to an embodiment of the present invention. As shown in FIG. 6, the position of the straight line represents the position of the shelf in the detected input image.

[0028] In some embodiments, after determining the plateau in the input image by linear fitting, the plateau direction (the direction of the straight line obtained by fitting) may be further corrected. The apparatus may further include the following units.

[0029] The shelf direction determination unit 104 generates a depth image based on the input image, determines a shelf direction based on the depth image, and corrects the position of the shelf level based on the shelf direction.

[0030] In some embodiments, the shelf direction determination unit may use a conventional monocular depth estimation model, such as AdaBins, to obtain a depth image (a grayscale image in which different grayscale values ​​reflect different depths). That is, an input image is input to the monocular depth estimation model to obtain the depth image. Edge feature detection may be performed on the depth image to obtain a first edge feature image corresponding to the depth image, and the shelf direction may be determined based on the first edge feature image. However, the embodiments of the present invention are not limited to this. The shelf direction determination unit may further calculate a gradient image in a first direction (height direction) of the depth image. Compared with the original input image and depth image, the gradient of the depth in the height direction can more significantly reflect the change in the depth of the image, so that the shelf tier can be found more clearly. Then, edge feature detection is performed on the gradient image to obtain a second edge feature image, and the shelf direction is determined based on the second edge feature image.

[0031] In some embodiments, determining the shelf direction based on the second edge feature image includes: processing the edge features in the detected second edge feature image to generate a predetermined number of straight lines; determining a vanishing point based on the predetermined number of straight lines; determining the direction of each shelf level based on the vanishing point; and correcting the position of the shelf level based on the direction of each shelf level (determining the shelf direction based on the first edge feature image is also implemented in a similar manner, and the description thereof is omitted here). The following will be described in detail.

[0032] In some embodiments, the edge feature may be extracted using an edge detection operator. The edge feature may include at least one of a Prewitt operator, a Laplacian operator, a Roberts cross operator, a Canny operator, and a Sobel operator. FIG. 7 is a schematic diagram of determining a predetermined number of lines based on a gradient image according to an embodiment of the present invention. In FIG. 7, a is the gradient image, and b is a second edge feature image, in which the edge feature includes multiple divergent contours, so the edge feature needs to be processed first. The processing includes Hough detection. The Hough detection merges the detected divergent contours into a long line segment. For the Hough detection, reference may be made to related art, and the description thereof will be omitted here. Since the merged line segment has edge features that are obviously not part of the shelf (e.g., vertical line segments or line segments with a large inclination), the edge features of this part need to be removed by filtering. For example, the slope of the merged line segment is compared with a predetermined slope range, and if the slope of the merged line segment is not within the slope range, the line segment is deleted. After the filtering process, the line segment merging process may be performed again to generate a predetermined number of straight lines (or line segments), as shown in Fig. 7c, and short and long straight lines that are close in distance and have similar slopes may be merged.

[0033] In some embodiments, since the predetermined number may not completely match the number of categories (i.e., the number of shelves) classified by the first determination unit 102, a vanishing point may be determined based on the predetermined number of straight lines, the direction of each shelf may be determined based on the vanishing point, and the position of each shelf may be corrected based on the direction of each shelf, in order to accurately correct the direction of each shelf. As shown in Fig. 7c, since the predetermined number of straight lines are not parallel lines, a vanishing point in the depth image or gradient image of the predetermined number of straight lines may be determined, and a vanishing line that is closest in distance to each straight line filtered by the second determination unit 103 in the height direction may be determined based on the vanishing point, and the direction of the closest vanishing line may be taken as the direction of the shelf to correct the direction of the corresponding straight line, i.e., the inclination of the corresponding straight line may be adjusted to be the same as the inclination of the closest vanishing line.

[0034] For example, the fitted line L 1 Based on the vanishing points determined by a certain number of straight lines, 1 and the vanishing line V that is closest in height 1 Determine the line L 1 The slope of the vanishing line V 1 The fitted line L is adjusted to match the slope of 2 Based on the vanishing points determined by a certain number of straight lines, 2 and the vanishing line V that is closest in height 2 Determine the line L 2 The slope of the vanishing line V 2 The same is true for the other lines, and the fitted line L Maxnum Based on the vanishing points determined by a certain number of straight lines, Maxnum and the vanishing line V that is closest in height Maxnum Determine the line L Maxnum The slope of the vanishing line V Maxnum In this way, the shelf position is corrected.

[0035] In some embodiments, when the predetermined number of straight lines are completely parallel and there is no vanishing point, the direction of each straight line fitted by the second determination unit 103 may be corrected based on the direction of the parallel straight lines. That is, each straight line fitted by the second determination unit 103 is also parallel, and its direction (slope) is the same as the direction (slope) of the predetermined number of straight lines. Here, the description is omitted.

[0036] In some embodiments, after correcting the position of the shelf, the apparatus may further detect whether the product on the shelf is out of stock and the number of products that are out of stock.

[0037] The product detection unit 105 determines the overlap area between the detection window and an area occupied by another product adjacent to one side of the product, using the edge of the area in the input image of the product detected by the first detection unit as the starting position of a detection window, and determines whether a shortage has occurred between the product and the other product based on the relationship between the overlap area and a first threshold value.

[0038] In some embodiments, the size of the detection window may be determined first. The width and height of the detection window may be approximately the same as the width and height of the bounding box of the product, or may be slightly smaller than the width and height of the bounding box, and the embodiments of the present invention are not limited thereto. The edge of the area of ​​the product in the input image may be one side of the bounding box of the product, for example, the right side in the height direction. The area occupied by the adjacent product may be an area occupied by the bounding box of the other product adjacent to the right side in the height direction of the bounding box of the product. FIG. 8 is a schematic diagram of a detection window according to an embodiment of the present invention. As shown in FIG. 8, the bounding boxes of the two adjacent products detected by the first detection unit are bounding box B1 and bounding box B2, respectively, and the right side in the height direction of the bounding box B1 is set as the starting position of the detection window. The overlapping area S between the detection window and the bounding box B2 is determined, and whether a shortage has occurred between the product B1 and the other product B2 is determined based on the relationship between S and a first threshold value. The products B1 and B2 are merely two examples of the products detected by the first detection unit, and the above method is adopted to determine whether or not a shortage has occurred for each two adjacent products among the multiple products detected by the first detection unit. The adjacent products may be adjacent products on the same shelf level that has been detected, in other words, the adjacent products refer to products that are adjacent in the length direction of the shelf.

[0039] In some embodiments, it is determined whether a shortage occurs between the product B1 and the other product B2 according to the relationship between S and a first threshold value, and if S is equal to or greater than the first threshold value, it is determined that a shortage occurs between the product B1 and the other product B2, and if S is smaller than the first threshold value or S is equal to 0 (no overlapping area or the detection window and the bounding box B2 do not overlap), it is determined that a shortage does not occur between the product B1 and the other product B2. The first threshold value may be determined according to demand, for example, the area of ​​one bounding box, or an area slightly smaller than the area of ​​one bounding box, or other values, and the embodiments of the present invention are not limited thereto.

[0040] In some embodiments, the product detection unit calculates a first distance between the center of the area occupied by the product and the area occupied by the other product and the farthest edge distance, and determines the number of missing parts between the product and the other product based on the first distance and the farthest edge distance. For example, the product detection unit calculates a ratio between the first distance and the farthest edge distance, and determines the number of missing parts between the product and the other product based on the relationship between the ratio and a second threshold. As shown in FIG. 8, the first distance is equal to the distance D1 between the center of the bounding box B1 and the center of the bounding box B2, and the farthest edge distance is equal to the distance D2 between the left side of the bounding box B1 in the height direction and the right side of the bounding box B2 in the height direction. Assuming that the number of out-of-stock items is n and that the horizontal size of each item is approximately the same, the distance D1 / D2 is between n / n+1 and n+1 / n+2, so the number of out-of-stock items between the item and the other item may be determined based on the relationship between D1 / D2 and a second threshold (n / n+1, where n is an integer of 1 or more). Correspondence between each different second threshold and different number of out-of-stock items may be preset. For example, if D1 / D2 is greater than 0.5 and less than 0.67, the number of out-of-stock items is determined to be 1; if D1 / D2 is greater than 0.67 and less than 0.75, the number of out-of-stock items is determined to be 2; if D1 / D2 is greater than 0.75 and less than 0.8, the number of out-of-stock items is determined to be 3; and the same applies to other cases, and the description thereof will be omitted here. The larger the value of D1 / D2, the greater the number of out-of-stock items. The second threshold may be determined according to demand, and the embodiment of the present invention is not limited thereto.

[0041] In some embodiments, the product detection unit may be provided independently of the product shelf detection device, in other words, product detection may be performed independently of product shelf detection or may be performed in combination, and embodiments of the present invention are not limited thereto.

[0042] 9 is a schematic diagram of a commodity detection device according to an embodiment of the present invention. As shown in FIG. 9, the commodity detection device 900 includes the following components:

[0043] The first detection unit 901 detects and recognizes the positions of a plurality of products in an input image.

[0044] The product detection unit 902 determines the overlap area between the detection window and an area occupied by another product adjacent to one side of the product, using the edge of the area of ​​the detected product in the input image as the starting position of a detection window, and determines whether a shortage has occurred between the product and the other product based on the relationship between the overlap area and a first threshold value.

[0045] The first detection unit 901 and the commodity detection unit 902 may be implemented by referring to the first detection unit 101 and the commodity detection unit 105 described above, and the description thereof will be omitted here.

[0046] Note that the above merely describes each component or module related to the present invention, and the present invention is not limited thereto. The product shelf detection device 100 and the product detection device 900 may include other components or modules, and reference may be made to related art for specific contents of these components or modules.

[0047] 1 and 9 merely exemplarily show connection relationships or signal directions between each component or module, and it is clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned various components or modules may be realized by hardware devices such as a processor and a memory, and the embodiment of the present invention is not limited thereto.

[0048] The above-mentioned embodiments are merely illustrative of the embodiments of the present invention, and the present invention is not limited thereto, and appropriate modifications may be made based on the above-mentioned various embodiments. For example, the above-mentioned embodiments may be used alone, or one or more of the above-mentioned embodiments may be used in combination.

[0049] According to this embodiment, the positions of multiple products in an input image are detected and recognized from the input image, the positions of the detected products are classified, and the straight line obtained by fitting the position information in each category is set as the position of the shelf level. This makes it possible to accurately and low-costly detect shelf levels (especially when products are densely stacked) based solely on images acquired by a monocular camera.

[0050] <Example 2> An embodiment of the present invention further provides an electronic device. Fig. 10 is a schematic diagram of an example of an electronic device according to an embodiment of the present invention. As shown in Fig. 10, an electronic device 1000 includes a product shelf detection device 1001 (or a product detection device 900, not shown), and the configuration and function of the product shelf detection device 1001 are similar to the product shelf detection device 100 described in the first embodiment. For details, the description in the first embodiment may be referred to, and the description thereof will be omitted here.

[0051] Fig. 11 is a schematic block diagram of an example of a system configuration of an electronic device according to an embodiment of the present invention. As shown in Fig. 11, an electronic device 1100 may include a processor 1101 and a memory 1102, and the memory 1102 is connected to the processor 1101. The diagram is merely exemplary, and other types of configurations may be used to supplement or replace the configurations to realize electrical communication functions or other functions.

[0052] In some aspects, the processor 1101 includes at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0053] As shown in FIG. 11, the electronic device 1100 may further include an input device 1103, a display 1104, and a power source 1105. In one embodiment, the function of the product shelf detection device may be integrated into the processor 1101. The processor 1101 may be configured to detect and recognize the positions of multiple products in an input image from the input image, determine information on a predetermined position of each product, classify the information on the predetermined position of each product, fit the information on the predetermined position of the product in each category, and determine a straight line obtained by fitting in each category as the position of the shelf. Alternatively, the processor 1101 may be configured to detect and recognize the positions of multiple products in the input image from the input image, determine an overlapping area between an area occupied by another product adjacent to one side of the product and the detection window, using an edge of an area in the input image of the detected product as a start position of a detection window, and determine whether a shortage has occurred between the product and the other product based on a relationship between the overlapping area and a first threshold value. Other embodiments may refer to Example 1, and a description thereof will be omitted here.

[0054] In another aspect of an embodiment of the present invention, the product shelf detection device or the product detection device may be respectively arranged with the processor 1101, for example, the product shelf detection device or the product detection device may be a chip connected to the processor 1101 and configured to realize the functions of the product shelf detection device or the product detection device under the control of the processor 1101.

[0055] In this embodiment, electronic device 1100 need not include all of the components shown in FIG.

[0056] As shown in FIG. 11, processor 1101, also referred to as a controller or operation control unit, may include a microprocessor or other processing device and / or logic device, and processor 1101 accepts input and controls the operation of each part of electronic device 1100.

[0057] The memory 1102 may be, for example, one or more of a buffer, a flash memory, a hard disk, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. The processor 1101 may execute a program stored in the memory 1102 to realize information storage or processing, etc. The other members are similar to the prior art, and therefore the description thereof is omitted here. Each part of the electronic device 1100 may be realized by dedicated hardware, firmware, software, or a combination thereof without departing from the scope of the present invention.

[0058] According to this embodiment, the positions of multiple products in an input image are detected and recognized from the input image, the positions of the detected products are classified, and the straight line obtained by fitting the position information in each category is set as the position of the shelf level, thereby making it possible to detect the shelf level accurately and at low cost based solely on the image acquired by the monocular camera.

[0059] <Example 3> An embodiment of the present invention provides a product shelf detection method. The method corresponds to the product shelf detection device of embodiment 1. Fig. 12 is a schematic diagram of an example of the product shelf detection method according to an embodiment of the present invention. As shown in Fig. 12, the method includes the following steps:

[0060] Step 1201: The positions of a plurality of products in an input image are detected and recognized from the input image.

[0061] Step 1202: Determine the information of the predetermined position of each product, and classify the information of the predetermined position of each product.

[0062] Step 1203: Information on the predetermined position of the product in each category is fitted, and the straight line obtained by fitting in each category is determined as the shelf position.

[0063] In this embodiment, the specific method of implementing each of the above steps is the same as that described in the first embodiment, and the description thereof will be omitted here.

[0064] 13 is a schematic diagram of an example of a method for detecting a store shelf according to an embodiment of the present invention. As shown in FIG. 13, the method includes the following steps:

[0065] Step 1301: An input image is obtained, and steps 1302 and 1304 are executed, respectively.

[0066] Step 1302: Detect and recognize the positions of a plurality of (N) products in an input image, where N is an integer greater than 1.

[0067] Step 1303: Determine information on the predetermined position of each product, classify the information on the predetermined position of each product, fit the information on the predetermined position of the products in each category, and determine the straight line obtained by fitting in each category as the shelf position.

[0068] Step 1304: Generate a depth image based on the input image, and determine a shelf direction based on the depth image.

[0069] Step 1305: Based on the shelf direction determined in step 1304, the shelf position obtained in step 1303 is corrected.

[0070] Step 1306: Determine whether X is smaller than N. If the result of the determination is YES, execute step 1307; if not, end the process.

[0071] Step 1307: Item B detected in step 1302 X The edge of the region in the input image is set as the starting position of the detection window, and another product B adjacent to one side of the product is X+1 Determine the overlap area between the area occupied by and the detection window. Initially, X=1.

[0072] Step 1308: Based on the relationship between the overlapping area and a first threshold, determine whether a shortage has occurred between the product and the other product, and if a shortage has occurred, execute step 1309; if not, X=X+1, and return to step 1306.

[0073] Step 1309: Calculate a first distance between the center of the area occupied by the product and the area occupied by the other product and a farthest edge distance, and determine the number of stockouts between the product and the other product based on the first distance and the farthest edge distance.

[0074] The embodiment of steps 1301 to 1309 may refer to the first embodiment, and the description thereof will be omitted here.

[0075] The above steps 1302, 1306 to 1309 may be independently performed as a product detection method, and the description thereof will be omitted here.

[0076] The above merely describes steps or processes related to the present invention, and the present invention is not limited thereto. The product shelf detection method may further include other steps or processes, and the specific contents of these steps or processes may refer to the prior art. In addition, the above merely exemplifies the embodiments of the present invention by taking some structures of the model used in the lane division method as examples, and the present invention is not limited to these structures, and appropriate modifications may be made to these structures, and these modifications should be included in the scope of the embodiments of the present invention.

[0077] The above-mentioned embodiments are merely illustrative of the embodiments of the present invention, and the present invention is not limited thereto, and appropriate modifications may be made based on the above-mentioned various embodiments. For example, the above-mentioned embodiments may be used alone, or one or more of the above-mentioned embodiments may be used in combination.

[0078] According to this embodiment, the positions of multiple products in an input image are detected and recognized from the input image, the positions of the detected products are classified, and the straight line obtained by fitting the position information in each category is set as the position of the shelf level, thereby making it possible to detect the shelf level accurately and at low cost based solely on the image acquired by the monocular camera.

[0079] An embodiment of the present invention further provides a computer-readable program that, when executed in a product shelf detection device, a product detection device, or an electronic device, causes a computer to execute the product shelf detection method or product detection method described in Example 3 above in the product shelf detection device, product detection device, or electronic device.

[0080] An embodiment of the present invention further provides a storage medium having stored thereon a computer-readable program for causing a computer to execute the product shelf detection method or product detection method described in Example 3 above in a product shelf detection device, a product detection device, or an electronic device.

[0081] The forklift status detection method performed in the product shelf / product detection device or electronic device described with reference to the embodiments of the present invention may be implemented in hardware, a software module executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in FIG. 1 or FIG. 9, or one or more combinations of the functional block diagrams, may correspond to each software module of a computer program flow or each hardware module. These software modules may correspond to each step shown in FIG. 12 to FIG. 13, respectively. These hardware modules may be realized by implementing these software modules in hardware using, for example, a field programmable gate array (FPGA).

[0082] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor reads information from the storage medium or writes information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal, or in a memory card inserted in the mobile terminal. For example, if the electronic device uses a MEGA-SIM card or a large-capacity flash memory device with a relatively large capacity, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0083] One or more of the functional blocks and / or one or more combinations of functional blocks described in Figure 1 or Figure 9 may be implemented in a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein. One or more of the functional blocks and / or one or more combinations of functional blocks described in Figure 1 or Figure 9 may be implemented in, for example, a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with a DSP communication, or any other configuration.

[0084] Although the present invention has been described above with reference to specific embodiments, the above description is merely illustrative and does not limit the scope of protection of the present invention. Various modifications and changes may be made to the present invention without departing from the spirit and principles of the present invention, and these modifications and changes also fall within the scope of the present invention.

[0085] The present invention further provides the following notes. (Appendix 1) A method for detecting a product shelf, comprising: detecting and recognizing positions of a plurality of products in an input image; determining predetermined position information for each product and classifying the predetermined position information for each product; fitting information on predetermined positions of products in each category, and determining a straight line obtained by fitting for each category as the position of the shelf. (Appendix 2) The method described in Appendix 1, wherein a hierarchical clustering method is used to classify the information of the predetermined location of each product based on the set cluster distance. (Appendix 3) determining said cluster distance; The step of determining the cluster distance comprises: determining a first range of cluster distances based on a size of the input image in a first direction; Sequentially performing clustering processing using a plurality of distance values ​​within the first range, and taking statistics of a second range of distances corresponding to the result having the highest frequency of occurrence among the clustering results; and determining the smallest distance within the second range as the cluster distance. (Appendix 4) The method of claim 1, further comprising the steps of generating a depth image based on the input image, determining a shelf orientation based on the depth image, and correcting the position of the shelf level based on the shelf orientation. (Appendix 5) The steps of generating a depth image based on the input image, determining a shelf direction based on the depth image, and correcting the position of the shelf level based on the shelf direction include: calculating a gradient image in a first direction of the depth image, performing edge feature detection on the gradient image, and processing the detected edge features to generate a predetermined number of straight lines; determining a vanishing point based on the predetermined number of straight lines, determining a direction of each shelf based on the vanishing point, and correcting a position of the shelf based on the direction of each shelf. (Appendix 6) The method described in Appendix 1 further includes a step of determining an overlap area between an area occupied by another product adjacent to one side of the product and the detection window, using an edge of an area of ​​the detected product in the input image as a starting position for a detection window, and determining whether a shortage has occurred between the product and the other product based on a relationship between the overlap area and a first threshold value. (Appendix 7) 7. The method of claim 6, further comprising the steps of: calculating a first distance and a farthest edge distance between the centers of an area occupied by the product and an area occupied by the other product; and determining a number of stockouts between the product and the other product based on the first distance and the farthest edge distance. (Appendix 8) determining a number of stockouts between the product and the other products based on the first distance and the farthest edge distance, 8. The method of claim 7, further comprising: calculating a ratio between the first distance and the farthest edge distance; and determining a number of stockouts between the product and the other products based on a relationship between the ratio and a second threshold. (Appendix 9) A method for detecting a product shelf, comprising: detecting and recognizing positions of a plurality of products in an input image; determining an overlapping area between an area of ​​the detected product in the input image, the area being a starting position of a detection window, and an area of ​​another product adjacent to one side of the detected product; and determining whether a shortage has occurred between the product and the other product based on a relationship between the overlapping area and a first threshold value. (Appendix 10) A storage medium having a computer-readable program stored therein, the computer-readable program being configured to cause a processor connected to the storage medium to: detecting and recognizing positions of a plurality of products in an input image; determining an overlapping area between an area of ​​the detected product in the input image, the area being a starting position of a detection window, and an area of ​​another product adjacent to one side of the detected product; and determining whether a shortage has occurred between the product and the other product based on a relationship between the overlapping area and a first threshold value.

Claims

1. A product shelf detection device, a first detection unit that detects and recognizes positions of a plurality of products in an input image; A first determination unit that determines information of a predetermined position of each product and classifies the information of the predetermined position of each product; a second determination unit that performs fitting to information on a specific position of a product in each category and determines a straight line obtained by fitting for each category as the position of the shelf level.

2. The device according to claim 1 , wherein the first determination unit classifies the information of the predetermined position of each product based on a set cluster distance using a hierarchical clustering method.

3. When determining the cluster distance, the first determination unit determining a first range of cluster distances based on a size of the input image in a first direction; Sequentially performing clustering processing using a plurality of distance values ​​within the first range, and taking statistics of a second range of distances corresponding to the result having the highest frequency of occurrence among the clustering results; The apparatus of claim 2 , wherein the cluster distance is the smallest distance within the second range.

4. The apparatus of claim 1 , further comprising a shelf direction determination unit that generates a depth image based on the input image, determines a shelf direction based on the depth image, and corrects a position of the shelf level based on the shelf direction.

5. The shelf direction determination unit is calculating a gradient image in a first direction of the depth image, performing edge feature detection on the gradient image, and processing the detected edge features to generate a predetermined number of straight lines; The apparatus according to claim 4 , further comprising: determining a vanishing point based on the predetermined number of straight lines; determining a direction of each shelf based on the vanishing point; and correcting a position of the shelf based on the direction of each shelf.

6. 2. The device according to claim 1, further comprising a product detection unit that determines an overlap area between an area of ​​the product in the input image detected by the first detection unit and the detection window, using an edge of the area in the input image of the product detected by the first detection unit as a start position of a detection window, and determines whether a shortage has occurred between the product and the other product based on a relationship between the overlap area and a first threshold value.

7. 7. The device of claim 6, wherein the product detection unit calculates a first distance and a farthest edge distance between the centers of an area occupied by the product and an area occupied by the other product, and determines a number of missing items between the product and the other product based on the first distance and the farthest edge distance.

8. 8. The apparatus of claim 7, wherein the product detection unit calculates a ratio between the first distance and the farthest edge distance, and determines a number of shortages between the product and the other products based on a relationship between the ratio and a second threshold.

9. A method for detecting a product shelf, comprising: detecting and recognizing positions of a plurality of products in an input image; determining predetermined position information for each product and classifying the predetermined position information for each product; fitting information on predetermined positions of products in each category, and determining a straight line obtained by fitting for each category as the position of the shelf.

10. A storage medium having a computer-readable program stored therein, the computer-readable program being configured to cause a processor connected to the storage medium to: detecting and recognizing positions of a plurality of products in an input image; determining predetermined position information for each product and classifying the predetermined position information for each product; fitting information on a predetermined position of the product in each category, and determining a straight line obtained by the fitting for each category as the position of the shelf.

Images