Setting device

Abstract

A setting device 600 comprises: a clustering unit 621 that, on the basis of the feature amounts of a plurality of objects extracted from image data, the subject of which includes the objects, and the positions of the objects, performs clustering, which is a process for classifying the objects; and a setting unit 622 that, on the basis of the results of the clustering performed by the clustering unit 621 and the image data, sets a target frame indicating an area which is to be subjected to item shortage detection.

Description

Setting device

[0001] The present invention relates to a setting device, a setting method, and a recording medium.

[0002] 2. Description of the Related Art Techniques used to detect shortages of products on shelves and the like are known.

[0003] For example, Patent Document 1 describes a merchandise monitoring device that monitors the display status of merchandise based on video captured of a display area in a store. According to Patent Document 1, a status monitoring area is set on the video of the display area in response to a user's input operation using an input device. The merchandise monitoring device then determines the display status of merchandise for each status monitoring area. Patent Document 1 also discloses using a merchandise occupancy rate, which is the percentage of merchandise that occupies the video of the display area, as a numerical value representing the display status.

[0004] Another related technology is, for example, Patent Document 2. Patent Document 2 describes an item recognition device that extracts item regions of items from an image, extracts feature information indicating the features of the item regions from each object region, and identifies items of the same type based on the feature information. Patent Document 2 also discloses, as an example, that determining whether feature information for adjacent item regions matches determines whether the regions contain the same type of item.

[0005] JP 2016-57952 A JP 2020-68006 A

[0006] In the case of the technology described in Patent Document 1, it is necessary to manually set the range for monitoring the condition of the product. This has resulted in problems such as high setup costs. In addition, the technology described in Patent Document 2 simply identifies an object region where the same type of object exists. Therefore, even if the technology described in Patent Document 2 is used, it is difficult to set an appropriate range for product monitoring, making it difficult to solve the above problem. For example, as described above, there has been a problem in that it is difficult to accurately set an appropriate range for product monitoring.

[0007] Therefore, one of the objects of the present invention is to provide a setting device, a setting method, and a program that can solve the above-mentioned problems.

[0008] In order to achieve this object, a setting device according to one embodiment of the present disclosure has a configuration including: a clustering unit that performs clustering, which is a process of classifying objects, based on feature amounts of the objects extracted from image data containing multiple objects as subjects and the positions of the objects; and a setting unit that sets a target frame indicating a range for detecting missing items, based on the result of the clustering by the clustering unit and the image data.

[0009] In addition, a setting method that is another embodiment of the present disclosure is configured such that an information processing device performs clustering, which is a process of classifying objects based on feature amounts of the objects extracted from image data containing multiple objects as subjects and the positions of the objects, and sets a target frame indicating the range in which missing items are detected based on the results of the clustering and the image data.

[0010] Another aspect of the present disclosure is a computer-readable recording medium having recorded thereon a program for causing an information processing device to perform the following processes: clustering, which is a process of classifying objects based on the feature quantities of the objects extracted from image data containing multiple objects as subjects and the positions of the objects; and setting a target frame indicating the range within which missing items are to be detected based on the results of the clustering and the image data.

[0011] According to the above-mentioned configurations, the above-mentioned problems can be solved.

[0012] FIG. 1 is a diagram for explaining an overview of a detection system in a first embodiment of the present disclosure. FIG. 1 is a diagram for explaining an overview of a detection system. FIG. 2 is a diagram for explaining an example configuration of a detection system. FIG. 3 is a block diagram showing an example configuration of a detection device. FIG. 4 is a diagram for explaining an example of an object detection process. FIG. 5 is a diagram for explaining an example of a clustering process. FIG. 6 is a diagram for explaining an example of a clustering process. FIG. 7 is a diagram for explaining an example of a clustering process. FIG. 8 is a diagram for explaining an example of a target frame setting process. FIG. 9 is a diagram for explaining an example of a target frame setting process. FIG. 10 is a diagram showing an example output by a detection device. A flowchart showing an example operation of a detection device when setting an target frame. A flowchart showing an example operation of a detection device when detecting a missing item. A block diagram showing another example configuration of a detection device. FIG. 11 is a diagram showing an example hardware configuration of a setting device in a second embodiment of the present disclosure. A block diagram showing an example configuration of a setting device.

[0013] [First Embodiment] A first embodiment of the present invention will be described with reference to FIGS. 1 to 17. FIGS. 1 and 2 are diagrams for explaining an overview of a detection system 100. FIG. 3 is a diagram showing an example of the configuration of the detection system 100. FIG. 4 is a block diagram showing an example of the configuration of a detection device 300. FIG. 5 is a diagram showing an example of object detection processing. FIGS. 6 to 10 are diagrams for explaining an example of clustering processing. FIGS. 11 to 13 are diagrams for explaining an example of target frame setting processing. FIG. 14 is a diagram showing an example of output by the detection device 300. FIG. 15 is a flowchart showing an example of the operation of the detection device 300 when setting a target frame. FIG. 16 is a flowchart showing an example of the operation of the detection device 300 when detecting missing items. FIG. 17 is a block diagram showing another example of the configuration of a detection device.

[0014] In a first embodiment of the present disclosure, a detection system 100 that performs out-of-stock detection using image data of a product shelf 400 containing multiple products is described. As shown in FIG. 1 , the detection system 100 of the present disclosure pre-sets an object frame r for out-of-stock detection based on the results of object detection for the image data. Furthermore, upon acquiring image data for detection, the detection system 100 detects object regions in the image data, which are regions where products exist, and calculates the proportion of the object regions for each of the pre-set object frames r. The detection system 100 then performs out-of-stock detection for each of the object frames r based on the calculated results. In this manner, the detection system 100 of the present disclosure automatically sets an object frame r that indicates the range within which out-of-stock detection is performed, and performs out-of-stock detection for each of the object frames r using the pre-set object frame r. In other words, the detection system 100 performs out-of-stock detection by detecting whether the proportion of products within the object frame r is lower than a predetermined standard based on the pre-set object frame r.

[0015] 2 is a diagram illustrating an overview of the target frame setting process in the present disclosure. For example, when setting the target frame r, the detection system 100 performs clustering based on the feature amounts and positions of products extracted by a predetermined process. Then, the detection system 100 sets the target frame r for out-of-stock detection based on the clustering results. In this case, the detection system 100 does not simply set the target frame r based on the clustering results, but can set the target frame r based on the clustering results and image data so as to include as many products as possible that are the target of out-of-stock detection.

[0016] For example, in a situation where multiple rows of products are displayed on a shelf 400, as illustrated in FIG. 2 , products at the rear are obscured by products displayed further forward. Therefore, the detection system 100 is unable to properly extract features of the products hidden at the rear, and instead extracts features from products located at the front for which features can be extracted, and performs clustering. If the target frame r were simply set based solely on the clustering results, the target frame r would be set to include only a portion of the displayed products, as illustrated by the target frame r1 in FIG. 2 . As a result, performing out-of-stock detection using the target frame r1 would result in detecting an out-of-stock situation simply because a single product is missing from the front row, making it difficult to properly detect out-of-stock items. On the other hand, the detection system 100 of the present disclosure sets the target frame r to include the detection target based on the clustering results and image data. For example, the detection system 100 sets the target frame r based on the clustering results, the results of shelf line detection based on image data, and the results of detection of the range in which objects such as products exist. This allows the detection system 100 to set the target frame r so that it more appropriately includes products that are the target of out-of-stock detection, as in the target frame r2 illustrated in Figure 2. As a result, more appropriate out-of-stock detection becomes possible.

[0017] In the present disclosure, an example will be given in which the object of missing-out detection is any product displayed on the product shelf 400. However, the object of missing-out detection does not necessarily have to be a product. For example, the object of missing-out detection may be any object other than a product, such as a promotional item displayed on the shelf.

[0018] Fig. 3 shows an example of the overall configuration of the detection system 100. Referring to Fig. 3, for example, the detection system 100 includes an imaging device 200 and a detection device 300. As illustrated in Fig. 3, the imaging device 200 and the detection device 300 are connected to each other so as to be able to communicate with each other via wire or wirelessly.

[0019] 3 illustrates an example in which the detection system 100 includes one imaging device 200. However, the number of imaging devices 200 included in the detection system 100 is not limited to one. The detection system 100 may include one imaging device 200 or multiple imaging devices 200.

[0020] The imaging device 200 is installed in advance at a predetermined location in a store or the like, and acquires image data. The imaging device 200 may be a general device such as a surveillance camera. In the case of the present disclosure, the imaging device 200 acquires image data of at least a portion of a product shelf 400 on which a plurality of products are displayed. In other words, the imaging device 200 acquires image data that includes at least a product that is the target of out-of-stock detection. Furthermore, the imaging device 200 can acquire image data in a time series with different image capture times for the same predetermined area.

[0021] In the present disclosure, the imaging device 200 can acquire image data for setting a target frame. For example, the imaging device 200 acquires image data for setting a target frame in response to an operation by an operator on the imaging device 200 or the detection device 300, an instruction from the detection device 300, or the like. The imaging device 200 may acquire image data for setting a target frame in response to any other trigger, such as time. Furthermore, the imaging device 200 can transmit image data for setting a target frame to the detection device 300 along with any information indicating that the image data is for setting a target frame, so that the detection device 300 can determine that the image data is for setting a target frame. Furthermore, the imaging device 200 can acquire image data for detection, which is used to check whether or not a stockout has occurred. For example, the imaging device 200 can start acquiring image data for detection in response to an operation by an operator on the imaging device 200 or the detection device 300, an instruction from the detection device 300, or the like. The imaging device 200 may acquire image data for detection based on time, such as during opening hours. Furthermore, the imaging device 200 can transmit image data for detection to the detection device 300 together with any information indicating that the image data is for detection, so that the detection device 300 can determine that the image data is for detection.

[0022] The detection device 300 may be configured to determine the purpose of the image data. In this case, the imaging device 200 may transmit image data to the detection device 300 without distinguishing between image data for setting a target frame and image data for detection.

[0023] The detection device 300 is an information processing device that performs out-of-stock detection using a preset target frame r. The detection device 300 can also set the target frame r for performing out-of-stock detection based on image data for setting the target frame. In other words, the detection device 300 also functions as a setting device that sets the target frame r for performing out-of-stock detection.

[0024] Fig. 4 shows an example of the configuration of the detection device 300. Referring to Fig. 4, the detection device 300 has, as main components, for example, an operation input unit 310, a screen display unit 320, a communication I / F unit 330, a storage unit 340, and an arithmetic processing unit 350.

[0025] 4 illustrates an example in which the functions of the detection device 300 are realized using one information processing device. However, the detection device 300 may be realized using multiple information processing devices, for example, on the cloud. For example, the detection device 300 may be composed of a setting device having a configuration for setting a target frame, which will be described later, and a detection device having a configuration for detection, which will be described later. Furthermore, the detection device 300 may not include some of the configurations exemplified above, such as not having the operation input unit 310 or the screen display unit 320, or may have configurations other than those exemplified above.

[0026] The operation input unit 310 is made up of operation input devices such as a keyboard, a mouse, etc. The operation input unit 310 detects operations of the operator operating the detection device 300 and outputs the operations to the arithmetic processing unit 350.

[0027] The screen display unit 320 is composed of a screen display device such as an LCD (Liquid Crystal Display), etc. The screen display unit 320 can display various information stored in the storage unit 340 on the screen in response to instructions from the arithmetic processing unit 350.

[0028] The communication I / F unit 330 includes a data communication circuit, etc. The communication I / F unit 330 performs data communication with an external device connected via a communication line.

[0029] The storage unit 340 is a storage device such as a hard disk or memory. The storage unit 340 stores processing information and a program 344 required for various processes in the calculation processing unit 350. The program 344 is read into the calculation processing unit 350 and executed to realize various processing units. The program 344 is read in advance from an external device or recording medium via a data input / output function such as the communication I / F unit 330, and is stored in the storage unit 340. Main information stored in the storage unit 340 includes, for example, reference image information 341, comparison image information 342, and target frame information 343.

[0030] The reference image information 341 includes image data for setting a target frame, which is acquired by the imaging device 200. The reference image information 341 may include multiple image data acquired at different times. For example, the reference image information 341 is updated in response to an image data acquisition unit 351 (described later) acquiring image data for setting a target frame from the imaging device 200.

[0031] The comparison image information 342 includes detection image data that is acquired by the imaging device 200 and is used to check whether or not there is a shortage. The comparison image information 342 may include multiple image data acquired at different times. For example, the comparison image information 342 is updated in response to an image data acquisition unit 351 (described later) acquiring detection image data from the imaging device 200.

[0032] The target frame information 343 includes information about the position of the target frame r in the image data. The target frame information 343 may include information about the position of each of multiple target frames r set by a target frame setting unit 355 (described later). For example, the target frame information 343 is updated in response to the setting of the target frame r by the target frame setting unit 355 (described later).

[0033] The arithmetic processing unit 350 has an arithmetic device such as a CPU (Central Processing Unit) and its peripheral circuits. The arithmetic processing unit 350 reads and executes a program 344 from the storage unit 340, thereby realizing various processing functions by causing the above hardware and the program 344 to work together. Major processing units realized by the arithmetic processing unit 350 include, for example, an image data acquisition unit 351, an object detection unit 352, a feature extraction unit 353, a clustering unit 354, a target frame setting unit 355, an object region detection unit 356, a ratio calculation unit 357, and an output unit 358.

[0034] In addition, instead of the above-mentioned CPU, the arithmetic processing unit 350 may have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination of these.

[0035] The image data acquisition unit 351 acquires image data from the imaging device 200. For example, the image data acquisition unit 351 acquires image data from the imaging device 200 via the communication I / F unit 330 or the like.

[0036] For example, the image data acquisition unit 351 acquires image data for setting a target frame from the imaging device 200. Then, the image data acquisition unit 351 stores the acquired image data in the storage unit 340 as reference image information 341. The image data acquisition unit 351 also acquires image data for detection from the imaging device 200. Then, the image data acquisition unit 351 stores the acquired image data in the storage unit 340 as comparison image information 342.

[0037] The image data acquisition unit 351 may be configured to store the image data in the storage unit 340 according to the result of determining the use of the acquired image data. For example, the image data acquisition unit 351 may determine the use of the image data according to the time when the image data was acquired.

[0038] The object detection unit 352 performs an object detection process to detect product portions in the image data based on the image data for setting the target frame stored in the storage unit 340 as the reference image information 341. The object detection unit 352 may detect product portions in the image data by inputting the image data for setting the target frame into a pre-trained model such as a general-purpose product detector. The object detection unit 352 may also perform object detection using other known methods.

[0039] Fig. 5 shows an example of object detection processing by the object detection unit 352. Referring to Fig. 5, for example, the object detection unit 352 can acquire rectangles including the detected product portion by performing the object detection processing. In the example shown in Fig. 5, the object detection unit 352 acquires a total of 21 rectangles, seven per shelf of the product shelf 400.

[0040] The feature extraction unit 353 extracts the feature of the product, which is the object, by extracting the feature from the rectangle detected by the object detection unit 352. For example, the feature extraction unit 353 extracts feature according to the appearance of the product, such as the product's outer shape, color, or pattern, as the feature of the product detected by the object detection unit 352. The feature extraction unit 353 may extract the feature of the product by inputting the rectangle acquired by the object detection unit 352 into a feature extractor that has been trained in advance. The feature extraction unit 353 may also extract the feature using other known methods.

[0041] Note that the acquisition of rectangles by the object detection unit 352 and the extraction of features by the feature extraction unit 353 may be performed in a single process. For example, the detection device 300 may perform the acquisition of rectangles and the extraction of features in a single process by inputting image data to a trained model that simultaneously detects objects and extracts features based on the input image data.

[0042] The clustering unit 354 performs clustering based on the feature amounts extracted by the feature amount extraction unit 353 and the positions of the products from which the feature amounts were extracted. In other words, the clustering unit 354 classifies the products in the image data into multiple clusters based on the feature amounts extracted by the feature amount extraction unit 353, the positional relationship of the products, and the like. Generally, products of the same type are displayed on the same shelf in the product shelf 400. Therefore, the clustering unit 354 can perform clustering for each shelf level of the product shelf 400. For example, FIG. 6 shows an example of clustering by the clustering unit 354. In the example shown in FIG. 6, the clustering unit 354 performs clustering for each shelf level of the product shelf 400, thereby classifying each level into two clusters. Note that the clustering unit 354 may also perform clustering for multiple levels together.

[0043] In the present disclosure, the clustering unit 354 can perform clustering for dividing products present on the same shelf of the product shelf 400 into a predetermined number of clusters, such as a preset number, based on feature amounts. The clustering unit 354 may perform the clustering by comparing feature amounts within the same shelf and classifying products that are determined to be similar, such as products whose similarity, such as the distance between feature amounts, is equal to or less than a predetermined threshold, into the same cluster. In other words, the number of clusters into which the clustering unit 354 classifies products may or may not be predetermined. Note that during the above processing, the clustering unit 354 may be configured to classify products according to their positional relationships, such as classifying products into the same cluster only when it is determined that the distance between products is equal to or less than a predetermined value.

[0044] The clustering unit 354 may also be configured to perform at least one of the methods described below.

[0045] For example, when it is determined that products belong to the same cluster based on feature amounts or the positional relationship of the products, the clustering unit 354 can be configured to classify each product into a cluster according to the determination result. For example, as shown in FIG. 7 , in a situation where some products are oriented differently, the extracted feature amounts may differ even for the same product. For example, in the example shown in FIG. 7 , products 511 and 512 are the same type of product, and products 517 and 518 are the same type of product. Furthermore, products 513, 514, 515, and 516 are the same type of product, but only product 515 is displayed with a different orientation. In such a situation, the extracted feature amounts differ between product 513, product 514, product 516, and product 515. As a result, the clustering unit 354 classifies product 513, product 514, product 516, and product 515 into different clusters. To deal with such cases, the clustering unit 354 can be configured to classify non-adjacent products into one cluster and then determine that any products displayed between them belong to the same cluster. In other words, the clustering unit 354 determines, based on the positional relationship of the products, that product 515 sandwiched between product 514 and product 516 that belong to the same cluster belongs to the same cluster as product 514 and product 516. As a result, the clustering unit 354 can classify product 513, product 514, product 515, and product 516 into one cluster.

[0046] Furthermore, when certain conditions are met, such as when there are missing products, the clustering unit 354 can perform clustering based on the positional relationships of the features by reordering the missing products and then comparing adjacent features. For example, as shown in FIG. 8 , some products may be missing. For example, in the example shown in FIG. 8 , products 521, 522, 523, 524, 525, 526, and 527 are present on the same row, but there is a gap between products 524 and 525. In such a case, the clustering unit 354 sorts the products in ascending order of coordinates in the x-axis direction, for example. The clustering unit 354 then compares adjacent features and can classify products that are determined to be similar into the same cluster. As a result, as shown in FIG. 8 , even when there are missing products, the products can be classified into appropriate clusters.

[0047] Furthermore, when there are gaps as described above, the clustering unit 354 may be configured to perform clustering using features extracted from past image data instead of the above-described sorting. For example, in the example shown in FIG. 9 , products 531, 532, 533, 534, 535, 536, and 537 are present on the same level, but there is a gap between products 534 and 535. In such a case, the clustering unit 354 identifies product 538, which is a product present in a location corresponding to the gap in the past image data, by referencing the reference image information 341 and the comparison image information 342. The clustering unit 354 can then perform clustering using the feature extraction results for the identified product 538. As a result, as shown in FIG. 9 , even when there are gaps in a product, it is possible to classify the product into an appropriate cluster. In this way, the clustering unit 354 may be configured to perform clustering using features extracted from past image data when a predetermined condition is met, such as when it is determined that there is a gap in a product. It is desirable to use past image data that can be determined to be close in time, such as image data acquired at the time closest to the current time when a product exists in the gap. Furthermore, when clustering is performed using features extracted from past image data, as shown in the example of Figure 9, even if there is a gap at the edge of the product, it can be classified into an appropriate cluster without any problems.

[0048] The clustering unit 354 may also be configured to perform classification based on the detection results of the shelf tags 410 placed on the product shelves 400. For example, as illustrated in FIG. 10 , there may be a discrepancy between the number of clusters and the number of shelf tags 410, such as when multiple shelf tags 410 exist despite being classified into one cluster as a result of feature comparison. In such cases, the clustering unit 354 can increase or decrease the number of clusters to match the number of shelf tags 410. In other words, the clustering unit 354 can control the number of clusters into which products are classified based on the number of shelf tags 410. For example, in the example illustrated in FIG. 10 , the clustering unit 354 classifies products 541 to 548 into one cluster. Meanwhile, two shelf tags 410 are detected at the bottom of the product shelves 400 using an arbitrary detector. Thus, in the example illustrated in FIG. 10 , there is a discrepancy between the number of clusters and the number of shelf tags 410. Therefore, the clustering unit 354 can reclassify the products so that the number of clusters matches the number of shelf tags. As a result, products 541 to 548 are classified into one of two clusters. For example, in the example shown in Fig. 10, the clustering unit 354 classifies products 541, 542, 543, and 544 into one cluster, and classifies products 545, 546, 547, and 548 into another cluster.

[0049] For example, the clustering unit 354 can be configured to perform at least one of the above processes. When performing clustering based on the results of comparing features within the same row, the clustering unit 354 can determine whether the features are similar based on similarity, such as the distance between the features, and a predetermined threshold. The predetermined threshold compared with the similarity may be a fixed value determined in advance, or may be a value dynamically determined by the clustering unit 354 or the like. For example, when dynamically determining the threshold, the clustering unit 354 may determine the threshold based on the similarity between adjacent products and the similarity between non-adjacent products. For example, the clustering unit 354 calculates the similarity, such as the distance between the features, between adjacent products in the entire image data. The clustering unit 354 also calculates the similarity between non-adjacent products in the entire image data. The clustering unit 354 can then determine the threshold based on the distribution of the calculated similarity. For example, the clustering unit 354 may determine, as the threshold, an arbitrary value that lies between the average value of the similarity between adjacent products and the average value of the similarity between non-adjacent products. The clustering unit 354 may dynamically determine the threshold by a method other than the above examples.

[0050] 11 , the target frame setting unit 355 sets a target frame r indicating the range in which out-of-stock detection is performed based on the image data and the clustering results by the clustering unit 354. For example, the target frame setting unit 355 identifies a range in which the product may be present based on the image data, and sets the target frame r to include the identified range. In addition, the target frame setting unit 355 stores information indicating the position of the set target frame r in the image data, etc., in the storage unit 340 as target frame information 343.

[0051] FIG. 12 shows an example of the target frame setting process. Referring to FIG. 12 , the target frame setting unit 355 detects shelf lines, which are areas where shelves constituting the product shelf 400 are located, based on the clustered image data. The target frame setting unit 355 then sets a target frame r based on the clustering results and the detected shelf lines. That is, the target frame setting unit 355 identifies an area where products may be located based on the detected shelf lines, and sets the target frame r to include the identified area. As an example, the target frame setting unit 355 creates a target frame r that circumscribes each rectangle classified into the same cluster, and then extends the top and bottom of the target frame r so that it aligns with the detected shelf lines. As a result, the target frame setting unit 355 can set a target frame r to more appropriately include products that are the target of missing items detection, such as the target frame r2 illustrated in FIG. 12 .

[0052] FIG. 13 also shows another example of the target frame setting process. For example, as illustrated in FIG. 13 , instead of detecting shelf board lines, the target frame setting unit 355 may use the results of object detection using an arbitrary object detector or the like to identify an area where a product may be present. For example, the detection device 300 uses an arbitrary object detector or the like to identify an area where an object may be present. The target frame setting unit 355 then sets the target frame r to include the identified area. This allows the target frame setting unit 355 to set the target frame r to more appropriately include products that are the target of missing item detection, such as the target frame r3 illustrated in FIG. 13 .

[0053] The target frame setting unit 355 may set the target frame r using any of the above-exemplified methods, or may set the target frame r by combining the above-exemplified methods. The target frame setting unit 355 may also set the target frame r based on the clustering results and image data using a method other than the above-exemplified methods.

[0054] The object region detection unit 356 detects an object region based on the image data for detection stored in the storage unit 340 as comparison image information 342. For example, the object region detection unit 356 can detect an object region based on the image data by performing processing using learning results such as semantic segmentation, which detects an object region on a pixel-by-pixel basis. Note that the object region detection unit 356 may detect an object region using a method different from that used by the object detection unit 352, or may detect an object region using a method similar to that used by the object detection unit 352.

[0055] The ratio calculation unit 357 calculates the ratio of the object region for each target frame r based on the target frame information 343 and the detection result by the object region detection unit 356. The ratio calculation unit 357 also functions as a detection unit that detects out-of-stock for each target frame r based on the ratio calculation result. For example, if the calculated ratio falls below a predetermined first threshold, the ratio calculation unit 357 determines that a shortage has occurred in the target frame r for which the ratio was calculated. If the calculated ratio exceeds a predetermined second threshold, the ratio calculation unit 357 may also determine that the number of products has increased. The first threshold and the second threshold may be set arbitrarily. The first threshold and the second threshold may be the same for each target frame r or may be different for each target frame r.

[0056] The output unit 358 performs output according to the determination result by the ratio calculation unit 357. For example, the output unit 358 displays information according to the determination result on the screen display unit 320. The output unit 358 may transmit the information according to the determination result to an external device via the communication I / F unit 330.

[0057] For example, if the ratio calculation unit 357 determines that a shortage has occurred, the output unit 358 outputs that information. In this case, as illustrated in FIG. 14 , the output unit 358 can cut out the portion of the target frame r of the image data included in the reference image information 341 for which it has determined that a shortage has occurred, and output the cut-out image portion. By performing this output, the output unit 358 can present the details of the out-of-stock product to the output destination without specifying the product name, etc., present within the target frame r. The output unit 358 can also perform similar processing when it determines that an increase in the number of products has occurred. When it determines that an increase in the number of products has occurred, the output unit 358 may notify the results of cutting out the image data for detection.

[0058] The above is an example of the configuration of the detection device 300. Next, an example of the operation of the detection device 300 will be described with reference to Figs.

[0059] 15 is a flowchart showing an example of the operation of the detection device 300 when setting the object frame. Referring to Fig. 15, the image data acquisition unit 351 acquires image data for setting the object frame from the imaging device 200 (step S101).

[0060] The object detection unit 352 performs an object detection process to detect product portions in the image data based on the image data for setting the target frame (step S102). The object detection unit 352 may perform object detection in the image data by inputting the image data for setting the target frame into a pre-trained model such as a general-purpose product detector. The object detection unit 352 may also perform object detection using other known methods. For example, the object detection unit 352 can obtain a rectangle including the detected product by performing the object detection process.

[0061] The feature extraction unit 353 extracts feature amounts of the product, which is the object, by extracting feature amounts from the rectangle detected by the object detection unit 352 (step S103). For example, the feature extraction unit 353 can extract feature amounts according to the appearance of the product, such as the shape, color, and pattern of the product, as feature amounts of the product detected by the object detection unit 352.

[0062] The clustering unit 354 performs clustering based on the feature amounts extracted by the feature amount extraction unit 353 and the positions of the products from which the feature amounts were extracted (step S104). In other words, the clustering unit 354 classifies the products in the image data into multiple clusters based on the feature amounts extracted by the feature amount extraction unit 353 and the relative positions of the products. Generally, products of the same type are placed on the same shelf in the product shelf 400. Therefore, the clustering unit 354 may perform clustering for each shelf level of the product shelf 400.

[0063] The object frame setting unit 355 sets an object frame r to include missing-item detection targets based on the clustering results by the clustering unit 354 and the image data (step S105). For example, the object frame setting unit 355 sets the object frame r to include missing-item detection targets based on the clustering results, the shelf line detection results based on the image data, the detection results of the range where objects such as products exist, etc. The object frame setting unit 355 can also store the setting results in the storage unit 340 as object frame information 343.

[0064] The above is an example of the operation of the detection device 300 when setting the target frame. Next, an example of the operation of the detection device 300 when detecting a shortage will be described with reference to FIG.

[0065] 16 is a flowchart showing an example of the operation of the detection device 300 when detecting a missing item. Referring to Fig. 16, the image data acquisition unit 351 acquires image data for detection from the imaging device 200 (step S201).

[0066] The object region detection unit 356 detects an object region based on the image data for detection stored in the storage unit 340 as the comparison image information 342 (step S202). For example, the object region detection unit 356 may detect an object region based on the image data by performing processing using a learning result such as semantic segmentation that detects an object region on a pixel-by-pixel basis.

[0067] The ratio calculation unit 357 calculates the ratio of the object region for each object frame r based on the object frame information 343 and the detection result by the object region detection unit 356 (step S203). Furthermore, the ratio calculation unit 357 performs missing-item detection for each object frame r based on the ratio calculation result (step S204).

[0068] If there is a target frame r where the calculated ratio is below the predetermined first threshold (step S204, Yes), the ratio calculation unit 357 detects that a shortage has occurred. In response, the output unit 358 outputs information that a shortage has occurred (step S205). In this case, the output unit 358 may cut out the portion of the target frame r where it has been determined that a shortage has occurred from the image data included in the reference image information 341, and output the cut-out image portion. On the other hand, if there is no target frame r where the calculated ratio is below the predetermined first threshold (step S204, No), the detection device 300 ends the processing.

[0069] The above is an example of the operation of the detection device 300 when detecting a shortage.

[0070] As described above, the detection device 300 includes a clustering unit 354 and a target frame setting unit 355. With this configuration, the target frame setting unit 355 can set a target frame r to include a missing item detection target based on the clustering results by the clustering unit 354 and the image data. In other words, with the above configuration, it is possible to accurately set an appropriate range for product monitoring. As a result, it is possible to more appropriately detect missing items.

[0071] The configuration of the detection device 300 is not limited to the example exemplified in this disclosure. For example, Fig. 17 shows another example configuration of the detection device 300. Referring to Fig. 17, the arithmetic processing unit 350 of the detection device 300 can implement an execution feasibility determination unit 359 in addition to the configuration exemplified in Fig. 4 by reading and executing the program 344 stored in the storage unit 340.

[0072] The execution possibility determination unit 359 determines whether or not to set the target frame r based on the image data for setting the target frame. For example, the execution possibility determination unit 359 may determine to set the target frame r depending on whether or not a predetermined condition is satisfied.

[0073] For example, the execution feasibility determination unit 359 can determine to set the target frame r in response to receiving input indicating that stocking is complete from an external device, etc. In response to this determination, the object detection unit 352, the feature extraction unit 353, the clustering unit 354, and the target frame setting unit 355 can be configured to start processing to set the target frame r. Furthermore, in response to receiving input indicating that stocking is complete from an external device, etc., the execution feasibility determination unit 359 can instruct the imaging device 200 to acquire image data for setting the target frame.

[0074] The execution feasibility determination unit 359 may be configured to make a determination based on conditions other than those exemplified above. For example, the execution feasibility determination unit 359 may determine whether to set the target frame r based on the results of image analysis of image data acquired from the imaging device 200, such as image data for setting the target frame. For example, the execution feasibility determination unit 359 may determine to set the target frame r based on, for example, the detection of the end of stocking as a result of the image analysis. The image analysis may be achieved using any method, such as using a pre-trained model.

[0075] Furthermore, the execution feasibility determination unit 359 may determine to perform the target frame setting process subsequent to the process by the feature amount extraction unit 353, on the condition that a preset number of products or more have been detected by the object detection unit 352. In this case, the execution feasibility determination unit 359 may determine whether or not to perform the target frame setting process for each shelf constituting the product shelf 400.

[0076] Second Embodiment Next, a second embodiment of the present disclosure will be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a diagram illustrating an example of the hardware configuration of a setting device 600. Fig. 19 is a block diagram illustrating an example of the configuration of the setting device 600.

[0077] In the second embodiment of the present disclosure, a setting device 600 that sets a target frame for detecting missing items will be described. Fig. 18 shows an example of the hardware configuration of the setting device 600. Referring to Fig. 18, the setting device 600 has the following hardware configuration, for example: - CPU (Central Processing Unit) 601 (arithmetic unit) - ROM (Read Only Memory) 602 (storage device) - RAM (Random Access Memory) 603 (storage device) - Programs 604 loaded into RAM 603 - Storage device 605 that stores the programs 604 - Drive device 606 that reads and writes from / to a storage medium 610 outside the information processing device - Communication interface 607 that connects to a communication network 611 outside the information processing device - Input / output interface 608 that inputs and outputs data - Bus 609 that connects the various components

[0078] 19 by the CPU 601 acquiring and executing the program group 604. The program group 604 is stored in advance in the storage device 605 or the ROM 602, for example, and is loaded into the RAM 603 or the like by the CPU 601 for execution as needed. The program group 604 may be supplied to the CPU 601 via the communication network 611, or may be stored in advance in the recording medium 610, and the drive device 606 may read out the program and supply it to the CPU 601.

[0079] 18 shows an example of the hardware configuration of the setting device 600. The hardware configuration of the setting device 600 is not limited to the above-described case. For example, the setting device 600 may be configured with only a part of the above-described configuration, such as excluding the drive device 606. Furthermore, the CPU 601 may be a GPU or the like exemplified in the first embodiment.

[0080] The clustering unit 621 performs clustering, which is a process of classifying objects, based on the feature amounts of the objects extracted from image data containing a plurality of objects as subjects and the positions of the objects.

[0081] The setting unit 622 sets a target frame indicating the range in which missing items are detected, based on the result of clustering by the clustering unit 621 and the image data.

[0082] As described above, the setting device 600 includes a clustering unit 621 and a setting unit 622. With this configuration, the setting unit 622 can set a target frame indicating the range in which out-of-stock detection is performed based on the clustering results by the clustering unit 621 and the image data. In other words, with the above configuration, it is possible to accurately set an appropriate range for product monitoring. As a result, it is possible to more appropriately perform out-of-stock detection.

[0083] The setting device 600 described above can be realized by incorporating a predetermined program into an information processing device such as the setting device 600. Specifically, a program according to another aspect of the present invention is a program for causing an information processing device such as the setting device 600 to perform clustering, which is a process of classifying objects based on feature amounts of the objects extracted from image data containing a plurality of objects as subjects and the positions of the objects, and to set a target frame indicating a range for detecting missing items based on the clustering results and the image data.

[0084] Furthermore, a setting method executed by an information processing device such as the setting device 600 described above is a method in which clustering, which is a process of classifying objects, is performed based on the feature quantities of the objects extracted from image data containing multiple objects as subjects and the positions of the objects, and a target frame indicating the range in which missing items are detected is set based on the results of the clustering and the image data.

[0085] Even if the invention is a program having the above-mentioned configuration, or a computer-readable recording medium having the program recorded thereon, or a setting method, it can achieve the same functions and effects as the above-mentioned setting device 600, and therefore can achieve the above-mentioned objective of the present disclosure.

[0086] <Supplementary Notes> Part or all of the above-described embodiments can be described as follows: Below, an outline of the setting device and the like in the present invention will be described. However, the present invention is not limited to the following configuration.

[0087] (Supplementary Note 1) A setting device comprising: a clustering unit that performs clustering, which is a process of classifying objects based on feature amounts of the objects extracted from image data containing a plurality of objects as subjects and the positions of the objects; and a setting unit that sets an object frame that indicates an area for detecting missing items based on a result of the clustering by the clustering unit and the image data. (Supplementary Note 2) The setting device according to Supplementary Note 1, wherein the setting unit sets the object frame so as to include an area of ​​the object determined based on the image data. (Supplementary Note 3) The setting device according to Supplementary Note 1 or Supplementary Note 2, wherein the setting unit detects, based on the image data, a shelf line that is an area where shelves that constitute a shelf on which the objects are displayed are located, and sets the object frame based on a result of clustering by the clustering unit and the detected shelf line. (Supplementary Note 4) The setting device according to any one of Supplementary Notes 1 to 3, wherein the setting unit sets the target frame based on the result of clustering by the clustering unit and the range in which the object exists, determined based on the result of object detection for the image data. (Supplementary Note 5) The setting device according to any one of Supplementary Notes 1 to 4, wherein the clustering unit performs clustering for each shelf on which the objects are displayed. (Supplementary Note 6) The setting device according to any one of Supplementary Notes 1 to 5, wherein, when it is determined that the objects belong to the same cluster based on their positional relationships, the clustering unit classifies the determined objects into a cluster according to the determination result. (Supplementary Note 7) The setting device according to any one of Supplementary Notes 1 to 6, wherein the clustering unit performs clustering by rearranging the objects so that they are packed tightly together and then comparing feature amounts of the objects.(Supplementary Note 8) The setting device described in any one of Supplementary Notes 1 to 7, wherein the clustering unit acquires, from a past image, feature amounts of portions corresponding to gaps in the display of the objects, and performs clustering based on the feature amounts of the objects extracted from the image data and the feature amounts acquired from the past image. (Supplementary Note 9) The setting device described in any one of Supplementary Notes 1 to 8, wherein the clustering unit performs classification based on a result of detecting shelf labels placed on a shelf on which the objects are displayed. (Supplementary Note 10) The setting device described in Supplementary Note 9, wherein the clustering unit controls the number of clusters into which the objects are classified depending on the number of detected shelf labels. (Supplementary Note 11) The setting device described in any one of Supplementary Notes 1 to 10, wherein the clustering unit determines a threshold based on a similarity calculated based on feature amounts of adjacent objects in the image data and a similarity calculated based on feature amounts of non-adjacent objects, and performs clustering using the determined threshold. (Supplementary Note 12) The setting device according to any one of Supplementary Notes 1 to 11, comprising a determination unit that determines whether to set an object frame, wherein the clustering unit performs clustering according to a result of the determination by the determination unit. (Supplementary Note 13) The setting device according to any one of Supplementary Notes 1 to 12, comprising a detection unit that performs missing item detection using the object frame set by the setting unit, and an output unit that operates in response to a detection result by the detection unit, wherein the output unit outputs an image corresponding to the object frame in which a missing item is detected, according to a result of the missing item detection by the detection unit. (Supplementary Note 14) A setting method comprising an information processing device that performs clustering, which is a process of classifying objects based on feature amounts of the objects extracted from image data including multiple objects as subjects and the positions of the objects, and sets an object frame that indicates a range for missing item detection based on a result of the clustering and the image data.(Supplementary Note 15) A computer-readable recording medium having recorded thereon a program for causing an information processing device to perform the following processes: clustering, which is a process of classifying objects based on feature amounts of the objects extracted from image data containing multiple objects as subjects and the positions of the objects; and setting a target frame indicating an area within which missing items are to be detected based on the results of the clustering and the image data.

[0088] The programs described in the above embodiments and appendices may be stored in a storage device or a computer-readable recording medium, such as a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0089] Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0090] 100 Detection system 200 Imaging device 300 Detection device 310 Operation input unit 320 Screen display unit 330 Communication I / F unit 340 Memory unit 341 Reference image information 342 Comparison image information 343 Object frame information 344 Program 350 Arithmetic processing unit 351 Image data acquisition unit 352 Object detection unit 353 Feature extraction unit 354 Clustering unit 355 Object frame setting unit 356 Object region detection unit 357 Ratio calculation unit 358 Output unit 359 Execution feasibility determination unit 400 Product shelf 511 Product 512 Product 513 Product 514 Product 515 Product 516 Product 517 Product 518 Product 521 Product 522 Product 523 Product 524 Product 525 Product 526 Product 527 Product 531 Product 532 Product 533 Product 534 Product 535 Product 536 Product 537 Product 538 Product 541 Product 542 Product 543 Product 544 Product 545 Product 546 Product 547 Product 548 Product 600 Setting device 601 CPU 602 ROM 603 RAM 604 Program group 605 Storage device 606 Drive device 607 Communication interface 608 Input / output interface 609 Bus 610 Recording medium 611 Communication network 621 Clustering unit 622 Setting unit

Claims

1. a clustering unit that performs clustering, which is a process of classifying objects, based on feature amounts of the objects extracted from image data containing a plurality of objects as subjects and positions of the objects; a setting unit that sets a target frame indicating a range in which missing item detection is performed based on the clustering result by the clustering unit and the image data; have Setting device.

2. 2. The setting device according to claim 1, The setting unit sets the target frame so as to include a range of the object determined based on the image data. Setting device.

3. 2. The setting device according to claim 1, The setting unit detects a shelf board line, which is an area where shelf boards constituting the shelf on which the object is displayed exist, based on the image data, and sets the target frame based on the clustering result by the clustering unit and the detected shelf board line. Setting device.

4. 2. The setting device according to claim 1, The setting unit sets the target frame based on a result of clustering by the clustering unit and a range in which the object exists determined according to a result of object detection for the image data. Setting device.

5. 2. The setting device according to claim 1, The clustering unit performs clustering for each shelf on which the objects are displayed. Setting device.

6. 2. The setting device according to claim 1, When it is determined that the objects belong to the same cluster based on their positional relationships, the clustering unit classifies the determined objects into clusters according to the determination result. Setting device.

7. 2. The setting device according to claim 1, The clustering unit performs clustering by comparing feature amounts of the objects after rearranging the objects in a packed state. Setting device.

8. 2. The setting device according to claim 1, The clustering unit determines a threshold based on a similarity calculated based on feature amounts of the adjacent objects in the image data and a similarity calculated based on feature amounts of the non-adjacent objects, and performs clustering using the determined threshold. Setting device.

9. The information processing device performing clustering, which is a process of classifying the objects, based on feature amounts of the objects extracted from image data containing a plurality of objects as subjects and positions of the objects; A target frame indicating the range in which missing items are to be detected is set based on the clustering result and the image data. How to set it up.

10. In the information processing device, performing clustering, which is a process of classifying the objects, based on feature amounts of the objects extracted from image data containing a plurality of objects as subjects and positions of the objects; A target frame indicating the range in which missing items are to be detected is set based on the clustering result and the image data. A program to realize the processing.