Processing device, processing method, and program

The processing device addresses the increased load by identifying product groups on shelves and matching only their feature amounts, enhancing efficiency and reducing processing load in product recognition.

JP7590009B2Active Publication Date: 2024-11-26NEC CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022529218
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-06-03
Publication Date
2024-11-26
Estimated Expiration
2040-06-03

AI Technical Summary

Technical Problem

Existing systems face increased processing load on computers as the number of products to be recognized increases, leading to inefficiencies in product recognition processes.

Method used

A processing device that identifies product groups on shelves using shelf-by-shelf display information and recognizes products by matching only the feature amounts of these groups, reducing the number of verification targets.

Benefits of technology

Reduces the processing load on computers by narrowing down the comparison targets in product recognition, thereby improving efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590009000001
    Figure 0007590009000001
  • Figure 0007590009000002
    Figure 0007590009000002
  • Figure 0007590009000003
    Figure 0007590009000003
Patent Text Reader

Abstract

The present invention provides a processing device (10) having: an acquisition unit (11) that acquires a product removal image showing a customer removing a product from a first product shelf; an identifying unit (12) that identifies, on the basis of shelf-by-shelf display information which indicates the products displayed on each product shelf, a product group displayed on the first product shelf; and a first recognition unit (13) that recognizes the product included in the product removal image by means of recognition processing in which the product group that has been identified is subjected to matching with product feature amount information indicating feature amounts for the appearance of a plurality of products.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a processing device, a processing method, and a program. [Background technology]

[0002] Non-Patent Documents 1 and 2 disclose a store system that eliminates payment processing (such as product registration and payment) at the cash register counter. This technology recognizes the products held by customers based on images generated by a camera that photographs the inside of the store, and automatically processes the payment based on the recognition results when the customer leaves the store.

[0003] Patent Document 1 discloses a technology that identifies the number of products a customer has taken from a shelf based on images captured by a surveillance camera, and then compares the number of products registered in a POS terminal at the time of payment with the number of products taken that was identified in advance to detect fraud. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2004-171241 A [Non-patent literature]

[0005] [Non-Patent Document 1] Takuya Miyata, "Amazon Go's mechanism: a cashierless supermarket realized using cameras and microphones," [online], December 10, 2016, [Retrieved December 6, 2019], Internet<URL:https: / / www.huffingtonpost.jp / tak-miyata / amazon-go_b_13521384.html> [Non-Patent Document 2] "NEC opens cashierless store "NEC SMART STORE" in headquarters--Facial recognition used, payment is made simultaneously when leaving the store", [online], February 28, 2020, [Retrieved March 27, 2020], Internet<URL: https: / / japan.cnet.com / article / 35150024 / > Summary of the Invention [Problem to be solved by the invention]

[0006] In a technology for recognizing products by comparing information (appearance features, etc.) of each of a number of products registered in advance with information (appearance features, etc.) of an object shown in an image, there is a problem that the processing load on the computer increases as the number of products handled in a store increases and the number of products to be compared increases. Non-Patent Documents 1 and 2, and Patent Document 1 do not disclose this problem.

[0007] An object of the present invention is to reduce the processing load on a computer in a technology for recognizing products by matching information on each of a plurality of pre-registered products with information on an object shown in an image. [Means for solving the problem]

[0008] According to the present invention, An acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating the feature amount of the appearance of a plurality of commodities; A processing apparatus is provided having:

[0009] Further, according to the present invention, The computer acquiring a product removal image showing a customer removing a product from a first product shelf; Identifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; A processing method is provided for recognizing products included in the product-taken-out image by performing a recognition process in which the identified group of products is compared against product feature information indicating the appearance features of a plurality of products.

[0010] Further, according to the present invention, Computer, an acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating the feature amounts of the appearances of a plurality of commodities; A program is provided to function as a Effect of the Invention

[0011] According to the present invention, in a technique for recognizing a product by matching information on each of a plurality of pre-registered products with information on an object shown in an image, the processing load on a computer is reduced. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a processing device according to the present embodiment. [Diagram 2] FIG. 2 is a functional block diagram of a processing apparatus according to an embodiment of the present invention; [Diagram 3] FIG. 2 is a diagram for explaining an example of camera installation according to the present embodiment. [Figure 4] FIG. 2 is a diagram for explaining an example of camera installation according to the present embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of an image generated by the camera of the present embodiment. [Figure 6]FIG. 2 is a diagram for explaining the relationship between the processing device, the camera, and the product shelf of the embodiment. [Figure 7] FIG. 4 is a diagram illustrating an example of information processed by the processing device of the present embodiment. [Figure 8] FIG. 4 is a diagram illustrating an example of information processed by the processing device of the present embodiment. [Figure 9] 4 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. [Figure 10] 4 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. [Figure 11] FIG. 4 is a diagram illustrating an example of information processed by the processing device of the present embodiment. [Figure 12] FIG. 2 is a functional block diagram of a processing apparatus according to an embodiment of the present invention; [Figure 13] 4 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. [Figure 14] FIG. 2 is a functional block diagram of a processing apparatus according to an embodiment of the present invention; [Figure 15] 4 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. [Figure 16] 4 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. [Figure 17] 4 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] <First embodiment> "overview" When the processing device of this embodiment acquires a product removal image showing a customer removing a product from a product shelf, the processing device identifies a group of products displayed on the product shelf from which the customer removed the products, based on previously generated shelf-by-shelf display information.The processing device then recognizes the products included in the product removal image by performing a recognition process in which the appearance feature values ​​of the identified group of products are compared with product feature value information indicating the appearance feature values ​​of multiple products.

[0014] According to such a processing device, only a portion of the product feature amount information indicating the appearance features of a plurality of products can be used as the target for matching, thereby reducing the processing load on the processing device.

[0015] "Hardware Configuration" Next, an example of the hardware configuration of the processing device will be described.

[0016] Each functional part of the processing device is realized by any combination of hardware and software, centered on the CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk that stores the programs (in addition to programs that are stored before the device is shipped, it can also store programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet), and a network connection interface.Those skilled in the art will understand that there are various variations in the methods and devices for realizing the above.

[0017] FIG. 1 is a block diagram illustrating a hardware configuration of a processing device. As shown in FIG. 1, the processing device has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device does not have to have the peripheral circuit 4A. The processing device may be composed of multiple devices that are physically and / or logically separated, or may be composed of a single device that is physically and / or logically integrated. When the processing device is composed of multiple devices that are physically and / or logically separated, each of the multiple devices can have the above hardware configuration.

[0018] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of the input device include a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. Examples of the output device include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0019] "Function Configuration" 2 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an acquisition unit 11, an identification unit 12, a first recognition unit 13, and a storage unit .

[0020] The acquiring unit 11 acquires a product removal image showing a customer removing a product from a product shelf.

[0021] Here, the camera that generates the product removal image will be described. The camera is installed in a position and orientation that captures the state of a customer removing a product from a product shelf. The camera may be installed on a product shelf, on a ceiling, on a floor, on a wall, or in other locations.

[0022] In addition, the number of cameras that capture the customer taking out a product from one product shelf may be one or more. When the customer taking out a product from one product shelf is captured by multiple cameras, it is preferable that the multiple cameras are installed so as to capture the customer taking out the product from the product shelf from different positions and directions.

[0023] In addition, a camera may be installed on each product shelf, or on every multiple product shelves, or on every tier of a product shelf, or on every multiple tiers of a product shelf.

[0024] The camera may capture moving images continuously (e.g., during business hours), or it may continuously capture still images at time intervals longer than the frame interval of the moving images, or it may perform these captures only while a motion sensor or the like detects a person in a specified position (such as in front of a product shelf).

[0025] The product take-out image generated by the camera may be input to the processing device 10 in real-time processing or in batch processing. Which processing is to be used can be determined depending on, for example, the usage of the product recognition result.

[0026] Here, an example of camera installation is shown. Note that the camera installation example described here is merely an example, and is not limited to this. In the example shown in Fig. 3, two cameras 2 are installed for each product shelf 1. Fig. 4 is a diagram in which the frame 4 of Fig. 3 is extracted. Each of the two parts that make up the frame 4 is provided with a camera 2 and lighting (not shown).

[0027] The light emission surface of the lighting extends in one direction, and has a light-emitting unit and a cover that covers the light-emitting unit. The lighting mainly emits light in a direction perpendicular to the extension direction of the light-emitting surface. The light-emitting unit has a light-emitting element such as an LED, and emits light in a direction that is not covered by the cover. When the light-emitting element is an LED, multiple LEDs are lined up in the direction in which the lighting extends (up and down in the figure).

[0028] Camera 2 is provided at one end of the linearly extending frame 4 component, and its imaging range is the direction in which the illumination light is emitted. For example, in the frame 4 component on the left side of Fig. 4, camera 2's imaging range is downward and diagonally downward to the right. Also, in the frame 4 component on the right side of Fig. 4, camera 2's imaging range is upward and diagonally upward to the left.

[0029] As shown in FIG. 3, the frame 4 is attached to the front frame (or the front of both side walls) of the product shelf 1 that constitutes the product placement space. One of the components of the frame 4 is attached to one of the front frames in a direction in which the camera 2 is located downward. The other of the components of the frame 4 is attached to the other front frame in a direction in which the camera 2 is located upward. The camera 2 attached to one of the components of the frame 4 takes pictures of the upper and diagonally upward so that the opening of the product shelf 1 is included in the shooting range. On the other hand, the camera 2 attached to the other of the components of the frame 4 takes pictures of the lower and diagonally downward so that the opening of the product shelf 1 is included in the shooting range. With this configuration, the two cameras 2 can shoot the entire range of the opening of the product shelf 1. When the configuration shown in FIG. 3 and FIG. 4 is adopted, it is possible to shoot the two cameras 2 of a customer taking out a product from the product shelf 1 as shown in FIG. 5. Images 7 and 8 generated by such cameras 2 include the product taken out by the customer from the product shelf 1.

[0030] As shown in Fig. 6, in this embodiment, multiple cameras 2 are installed to capture images of customers taking products from multiple product shelves 1. Then, the acquisition unit 11 acquires multiple product taking images generated by the multiple cameras 2. Note that in the example shown in Fig. 6, one camera 2 is installed corresponding to one product shelf 1, but as described above, this configuration is merely an example and is not limited to this configuration.

[0031] Returning to Figure 2, the identification unit 12 first identifies which product shelf among multiple product shelves the first product removal image, which is one of the multiple product removal images acquired by the acquisition unit 11, is an image showing the state of removing a product from.

[0032] For example, in the case where the installation positions of the multiple cameras that capture the multiple product shelves are fixed, the identification unit 12 may identify which product shelf the first product take-out image is from by identifying the camera that generated the first product take-out image. In addition, in the case where information indicating the shooting position is attached to the product take-out image, the identification unit 12 may identify which product shelf the first product take-out image is from by comparing the information indicating the shooting position with information indicating the installation positions of each of the multiple product shelves that is prepared in advance. In addition, in the case where information (such as a shelf number) for identifying each product shelf is physically attached and the product take-out image includes the information, the identification unit 12 may identify which product shelf the first product take-out image is from by analyzing the product take-out image and recognizing the information attached to the product shelf.

[0033] In the following description, a "first product removal image" is defined as an image showing a state in which a product is removed from a "first product shelf" among a plurality of product shelves.

[0034] After identifying that the first product removal image is an image showing a state in which a product is removed from the first product shelf, the identifying unit 12 identifies a group of products displayed on the first product shelf based on the shelf-by-shelf display information. The shelf-by-shelf display information indicates the products displayed on each of the multiple product shelves.

[0035] An example of the shelf-by-shelf display information is shown in Fig. 7. The shelf-by-shelf display information shown in the figure is information that links shelf identification information for identifying multiple product shelves with product identification information of products displayed on each product shelf. For example, the storage unit 14 stores the shelf-by-shelf display information.

[0036] The shelf display information may be information generated by an input operation of an operator, or may be information automatically generated by image analysis, etc. An example of the latter automatic generation will be described in detail in the following embodiment.

[0037] Returning to Fig. 2, the first recognition unit 13 recognizes the products included in the first product taken-out image by a recognition process that targets a portion of the product feature amount information indicating the external appearance feature amounts of a plurality of products as a comparison target. In the recognition process, the feature amounts of other products that are not included in the portion are not targeted for comparison. The portion that is targeted for comparison is the feature amounts of the group of products identified by the identification unit 12 as products displayed on the first product shelf.

[0038] 8 is a schematic diagram showing an example of product feature amount information indicating the appearance feature amounts of a plurality of products. The product feature amount information indicates, for example, the appearance feature amounts of a plurality of products handled in the store. For example, the storage unit 14 stores the product feature amount information.

[0039] Next, an example of the flow of processing by the processing device 10 will be described with reference to the flowchart of FIG.

[0040] First, when the acquisition unit 11 acquires a product removal image to be processed (S10), the identification unit 12 identifies which product shelf among multiple product shelves the product removal image is from (S11). Next, the identification unit 12 identifies a group of products displayed on the product shelf identified in S11 based on shelf-by-shelf display information (see FIG. 7) indicating products displayed on each of the multiple product shelves (S12).

[0041] Next, the first recognition unit 13 recognizes the products included in the product-taken-out image acquired in S10 by performing a recognition process using the appearance features of the product group identified in S12 as the target of comparison among the product feature information (see FIG. 8) indicating the appearance features of a plurality of products (S13).Then, the processing device 10 outputs the recognition result (S14).

[0042] Next, another example of the processing flow of the processing device 10 will be described with reference to the flowchart of FIG.

[0043] First, when the acquisition unit 11 acquires a product removal image to be processed (S20), the identification unit 12 identifies which product shelf among multiple product shelves the product removal image is from (S21). Next, the identification unit 12 identifies a group of products displayed on the product shelf identified in S21 based on shelf-by-shelf display information (see FIG. 7) indicating products displayed on each of the multiple product shelves (S22).

[0044] Next, the first recognition unit 13 recognizes the products included in the product-taken-out image acquired in S20 by a recognition process in which the external feature quantities of the products identified in S22 in the product feature quantity information (see FIG. 8) indicating the external feature quantities of a plurality of products are collated (S23). 3 If the recognition process is successful (Yes in S24), the recognition result is output (S26).

[0045] "Successful recognition processing" means that a recognition result with a reliability equal to or higher than a reference value is obtained. The reliability is calculated based on, for example, the number of matched features or the ratio of the number of matched features to the number of preregistered features.

[0046] For example, when a customer returns a product that he or she has taken out of a shelf to a shelf different from the original shelf, the product may not be displayed on the shelf where it should be. That is, a situation may occur where a product different from the product displayed on a certain shelf is displayed on that shelf as indicated by the shelf-by-shelf display information. In this case, a situation may occur where the product cannot be correctly recognized (recognition process failure) by a recognition process that matches only the features of a product group identified based on the shelf-by-shelf display information.

[0047] In this processing example, S2 3 If the recognition process in S20 fails (No in S24), the first recognition unit 13 recognizes the product included in the product taken-out image acquired in S20 by performing a recognition process that compares the appearance features of all the products indicated by the product feature information (see FIG. 8) (S25). Then, the processing device 10 outputs the recognition result (S26).

[0048] In this embodiment, the contents of subsequent processes performed on the result of the recognition process by the first recognition unit 13 (product recognition result) are not particularly limited.

[0049] For example, the product recognition result may be used in payment processing in a store system that does not require payment processing (product registration, payment, etc.) at a checkout counter as disclosed in Non-Patent Documents 1 and 2. An example will be described below.

[0050] First, the store system registers the output product recognition result (product identification information) by linking it to information that identifies the customer holding the product. For example, a camera that captures the face of a customer holding a product may be installed in the store, and the store system may extract the external features of the customer's face from the image generated by the camera. Then, the store system may register the product identification information of the product held by the customer and other product information (unit price, product name, etc.) by linking it to the external features of the face (information that identifies the customer). The other product information can be obtained from a product master (information that links product identification information with other product information) that is stored in advance in the store system.

[0051] Alternatively, the customer's customer identification information (membership number, name, etc.) may be linked to the appearance features of the customer's face and registered in advance in any location (store system, center server, etc.). Then, when the store system extracts the appearance features of the customer's face from an image including the face of the customer holding a product, the store system may identify the customer identification information of the customer based on the previously registered information. Then, the store system may register the product identification information of the product held by the customer and other product information in association with the identified customer identification information.

[0052] The store system also calculates the payment amount based on the registered details and executes the payment process. For example, the payment process is executed when the customer leaves the store through the gate or when the customer leaves the store through the exit. The detection of these timings may be realized by detecting the customer's exit from an image generated by a camera installed at the gate or exit, by inputting customer identification information of the customer leaving the store into an input device (such as a reader that performs short-range wireless communication) installed at the gate or exit, or by other methods. The details of the payment process may be a payment process using a credit card based on pre-registered credit card information, a payment based on pre-charged money, or other methods.

[0053] Other examples of applications of the results of the recognition process (product recognition results) by the first recognition unit 13 include customer preference surveys and marketing surveys. For example, by linking the products picked up by each customer to each customer and registering them, it is possible to analyze the products that each customer is interested in. Also, by registering that the customer picked up each product, it is possible to analyze which products customers are interested in. Furthermore, by estimating customer attributes (gender, age, nationality, etc.) using conventional image analysis technology and registering the attributes of the customers who picked up each product, it is possible to analyze what attributes of customers are interested in each product.

[0054] "Action and effect" According to the processing device 10 of the present embodiment described above, it is possible to recognize products contained in an image by a recognition process that does not target the appearance feature quantities of "all products" in the product feature quantity information indicating the appearance feature quantities of multiple products as the target for comparison, but only targets the appearance feature quantities of "the group of products displayed on the product shelf from which the products have been removed." According to such a processing device 10, it is possible to reduce the number of targets for comparison, thereby reducing the processing load on the processing device 10.

[0055] The processing device 10 also identifies which product shelf the image to be processed is from and sets the features of the products displayed on the identified product shelf as the objects of comparison. The processing device 10 can appropriately narrow down the objects to be compared, thereby preventing the occurrence of a problem in which matching information is not included in the objects to be compared.

[0056] Furthermore, as shown in the flowchart of FIG. 10, for example, the processing device 10 has a two-step configuration of a first recognition process that compares the feature amounts of some products and a second recognition process that compares the feature amounts of all products, and can perform the second recognition process if the first recognition process is unable to correctly recognize a product. In this case, even if a situation arises in which a product is not displayed on the shelf where it should be because, for example, a customer has taken a product off a shelf and put it back on a shelf different from the original shelf, the product taken off the shelf by the customer can be correctly recognized. Furthermore, the two-step configuration reduces the processing load of the second recognition process, which places a heavy burden on the processing device 10. recognition This can prevent the inconvenience of performing processing unnecessarily.

[0057] <Second embodiment> The shelf-by-shelf display information of this embodiment indicates products displayed on each level of each of a plurality of product shelves, as shown in Fig. 11. The shelf-by-shelf display information shown in the figure is information that links together shelf identification information for identifying a plurality of product shelves from one another, level identification information for identifying a plurality of levels of each product shelf from one another, and product identification information of products displayed on each level of each product shelf.

[0058] After identifying that the first product removal image is an image showing a product being removed from the first product shelf, the identification unit 12 analyzes the first product removal image to identify from which shelf of the first product shelf the product was removed. The algorithm for achieving this identification is not particularly limited. For example, as shown in images 7 and 8 in FIG. 5, when the first product shelf is included in the first product removal image, the area occupied by each shelf in the image is registered in advance, and it is possible to identify from which shelf each product was removed based on the relationship between the area of ​​each shelf in the image and the position of the removed product.

[0059] In the following description, a "first product removal image" is defined as an image showing a state in which a product is removed from a "first shelf" of a "first product shelf."

[0060] After the identification unit 12 determines that the first product removal image is an image showing a product being removed from the first tier of the first product shelf, it identifies the group of products displayed on the first tier of the first product shelf based on "shelf-by-shelf display information indicating the products displayed on each tier of each of the multiple product shelves (see Figure 11)."

[0061] The first recognition unit 13 recognizes the products included in the first product taken-out image by a recognition process that uses as a comparison target a portion of the product feature amount information indicating the appearance feature amounts of the multiple products. The portion to be compared is the feature amount of the product group identified by the identification unit 12 as the products displayed on the first shelf of the first product shelf.

[0062] As described in the first embodiment, the first recognition unit 13 may configure the recognition process to include multiple steps.

[0063] For example, the system may be configured in two steps, including a first recognition process that compares the features of a group of products identified as products displayed on the first shelf of a first product shelf, and a second recognition process that compares the features of all products, and the second recognition process may be performed if the first recognition process is unable to correctly recognize the products.

[0064] Alternatively, the system may be configured in three steps, including a first recognition process that compares the features of a group of products identified as products displayed on the first shelf of the first product shelf, a second recognition process that compares the features of the group of products identified as products displayed on the first product shelf, and a third recognition process that compares the features of all products, and the second recognition process is performed if the first recognition process is unable to correctly recognize a product, and the third recognition process is performed if the second recognition process is unable to correctly recognize a product.

[0065] The other configuration of the processing device 10 is similar to that of the first embodiment.

[0066] According to the processing device 10 of this embodiment, the same action and effect as in the first embodiment is realized. Moreover, according to the processing device 10 of this embodiment, the matching target for the recognition process can be further narrowed down, so that the processing load on the processing device 10 is further reduced.

[0067] <Third embodiment> The processing device 10 of this embodiment has a function of automatically generating shelf-by-shelf display information. An example of a functional block diagram of the processing device 10 of this embodiment is shown in Fig. 12. As shown in the figure, the processing device 10 has an acquisition unit 11, an identification unit 12, a first recognition unit 13, a storage unit 14, a second recognition unit 15, and a shelf-by-shelf display information generation unit 16.

[0068] The acquisition unit 11 acquires a plurality of product replenishment images showing a store clerk replenishing products on each of a plurality of product shelves. The configuration of the camera that generates the product replenishment images is the same as the configuration of the camera that generates the product removal images described in the first embodiment. For example, a camera 2 as shown in Figs. 3 to 5 can be used to capture an image of a store clerk replenishing products on a product shelf 1. The camera that generates the product replenishment images may be the same camera as the camera that generates the product removal images, or it may be a different camera.

[0069] The merchandise replenishment images generated by the camera may be input to the processing device 10 in a real-time manner or in a batch manner. However, in order to eliminate discrepancies between the shelf-by-shelf display information and the actual display situation, it is preferable to input the merchandise replenishment images to the processing device 10 in a real-time manner and update the shelf-by-shelf display information with less time loss after replenishment.

[0070] The second recognition unit 15 recognizes products included in the product replenishment image by a recognition process that compares the appearance features of all products indicated by the product feature information. The first recognition unit 13 that recognizes products taken out from the product shelf by a customer compares only the appearance features of some products in the product feature information, whereas the second recognition unit 15 that recognizes products to be replenished on the product shelf by a store clerk is different in that it compares the appearance features of all products indicated by the product feature information.

[0071] The shelf-by-shelf display information generation unit 16 identifies which product shelf among the multiple product shelves a first product replenishment image, which is one of the multiple product replenishment images acquired by the acquisition unit 11, is an image showing how products are being replenished on.

[0072] For example, in the case where the installation positions of the multiple cameras that capture the multiple product shelves are fixed, the shelf-by-shelf display information generating unit 16 may identify which product shelf the first product replenishment image shows by identifying the camera that generated the first product replenishment image. In addition, in the case where information indicating the shooting position is attached to the product replenishment image, the shelf-by-shelf display information generating unit 16 may identify which product shelf the first product replenishment image shows by comparing the information indicating the shooting position with information indicating the installation position of each of the multiple product shelves that is prepared in advance. In addition, in the case where information (such as a shelf number) for identifying each product shelf is physically attached and the product replenishment image includes the information, the shelf-by-shelf display information generating unit 16 may identify which product shelf the first product replenishment image shows by analyzing the product replenishment image and recognizing the information attached to the product shelf.

[0073] In the following description, the "first product restocking image" is defined as an image showing a state in which products are being restocked on a "first product shelf" among a plurality of product shelves.

[0074] After the shelf-by-shelf display information generation unit 16 identifies that the first product replenishment image is an image showing the state of replenishing products on the first product shelf, it registers the products recognized by the second recognition unit 15 as being included in the first product replenishment image in the shelf-by-shelf display information (see Figure 7) as products displayed on the first product shelf.

[0075] Next, an example of the flow of processing by the processing device 10 will be described with reference to the flowchart of FIG.

[0076] First, when the acquisition unit 11 acquires the product replenishment image to be processed (S30), the second recognition unit 15 recognizes the products included in the product replenishment image acquired in S30 by performing a recognition process in which the external features of all products indicated by the product feature information are compared (S31).

[0077] Next, the shelf-by-shelf display information generating unit 16 identifies which of the multiple product shelves the product replenishment image acquired in S30 is an image showing how products are being replenished (S32). Next, the shelf-by-shelf display information generating unit 16 registers the products recognized as included in the product replenishment image acquired in S30 in the shelf-by-shelf display information as products displayed on the product shelf identified in S32 (S33). Note that if the products have already been registered, the registration process may not be performed, and the registration process may be performed only if the products have not been registered.

[0078] Here, a modified example of this embodiment will be described. After determining that the first product replenishment image shows the state of replenishing products on the first product shelf, the shelf-by-shelf display information generating unit 16 may analyze the first product replenishment image to determine which stage of the first product shelf the product has been replenished on. The algorithm for achieving this determination is not particularly limited. For example, as shown in images 7 and 8 of FIG. 5, when the first product shelf is included in the first product replenishment image, the area occupied by each stage in the image is registered in advance, and based on the relationship between the area of ​​each stage in the image and the replenishment position of the product, it is possible to determine which stage each product has been replenished on.

[0079] The shelf-by-shelf display information generation unit 16 identifies which level of the first product shelf the first product replenishment image shows the product being replenished on, and then registers the product recognized by the second recognition unit 15 as being included in the first product replenishment image in the shelf-by-shelf display information (see Figure 11) as a product displayed on the identified level of the first product shelf.

[0080] The other configuration of the processing device 10 is similar to that of the first and second embodiments.

[0081] According to the processing device 10 of this embodiment, the same operational effects as those of the first and second embodiments are realized. In addition, according to the processing device 10, display information for each shelf can be automatically created without manual intervention. This reduces the workload of store clerks.

[0082] Moreover, the processing device 10 analyzes product replenishment images showing the store clerk replenishing products on the product shelves to identify the products replenished on each product shelf, and generates shelf-by-shelf display information based on the identification results. With this type of processing device 10, it is possible to generate shelf-by-shelf display information with fewer errors.

[0083] Furthermore, the processing device 10 can recognize products that the store clerk is replenishing on the shelves by performing a recognition process that compares the external features of all products indicated by the product feature information. Therefore, even if the products displayed on each shelf change, the products can be correctly recognized and the correct information can be registered in the shelf-by-shelf display information.

[0084] <Fourth embodiment> The processing device 10 of this embodiment generates shelf-by-shelf display information using the method described in the third embodiment. The same camera captures "a customer taking a product from the product shelf" and "a store clerk restocking products on the product shelf" to generate a product removal image and a product restocking image. This configuration reduces the number of cameras to be installed and the cost required for equipment.

[0085] As described in the first to third embodiments, the processing contents executed by the processing device 10 differ depending on whether the image generated by the camera is a product take-out image or a product replenishment image. Therefore, the processing device 10 has a function of classifying the images generated by the camera into a product take-out image and a product replenishment image.

[0086] 14 shows an example of a functional block diagram of the processing device 10 of this embodiment. As shown in the figure, the processing device 10 has an acquisition unit 11, an identification unit 12, a first recognition unit 13, a storage unit 14, a second recognition unit 15, a shelf-by-shelf display information generation unit 16, and a classification unit 17.

[0087] The classification unit 17 classifies the images generated by the camera into product removal images and product replenishment images. For example, the classification unit 17 performs the classification based on the time the image was taken, the feature amount of the appearance of the person included in the image, or the mode set when the image was taken. An example of the classification process will be described below.

[0088] "Classification processing example 1" In this example, the time period for replenishing products on the product shelves is determined in advance. For example, the time period before the store opens is the time period for replenishing products on the product shelves. The classification unit 17 classifies images whose shooting time falls within the time period for replenishing products on the product shelves as product replenishing images, and classifies other images as product removal images.

[0089] "Classification processing example 2" In this example, a store clerk wears a predetermined uniform and replenishes merchandise. The classification unit 17 judges whether a "person holding a product" in an image is a store clerk based on whether the person in the image is wearing the uniform. The classification unit 17 then classifies an image that includes a store clerk holding a product as a product replenishment image, and classifies the other images as a product removal image.

[0090] "Classification processing example 3" In this example, the facial appearance feature of the store clerk is registered in advance in a database. The classification unit 17 compares the facial appearance feature of the person included in the image with the feature in the database to determine whether or not the "person holding a product" included in the image is a store clerk. The classification unit 17 then classifies an image that includes a store clerk holding a product as a product replenishment image, and classifies the other images as a product removal image.

[0091] "Classification processing example 4" In this example, the camera has a normal shooting mode and a product replenishment shooting mode. The mode is switched by an input from a store clerk. The input may be made via an input device (physical button, microphone, touch panel, keyboard, mouse, etc.) installed near the product shelf or in the back room of the store, or may be made by gesture input that causes a camera installed near the product shelf to capture a predetermined gesture, or may be made by input via a portable device such as a smartphone, tablet terminal, smart watch, or mobile phone. The camera that captures the predetermined gesture may be the same camera that captures the "situation of a customer taking a product from the product shelf" and the "situation of a store clerk replenishing products on the product shelf."

[0092] The classification unit 17 then classifies the images generated in the product replenishment photography mode as product replenishment images, and classifies the images generated in the normal photography mode as product removal images.

[0093] For example, information indicating in what mode the image was generated may be added to the image generated by the camera, and the classification unit 17 may then identify in what mode each image was generated based on the information.

[0094] Additionally, the camera may store mode history information, which indicates the time periods when each mode was in use. Then, the classification unit 17 may determine in which mode each image was generated based on the history information and the time each image was taken.

[0095] Next, an example of the flow of processing by the processing device 10 will be described with reference to the flowchart of FIG.

[0096] First, when the acquisition unit 11 acquires an image to be processed (S40), the classification unit 17 executes a classification process to classify the image into a product removal image or a product replenishment image (S41).

[0097] If the image is classified as a merchandise replenishment image ("product replenishment image" in S42), the processing device 10 performs the processes of S43 to S45. The processes of S43 to S45 are the same as the processes of S31 to S33 in FIG.

[0098] On the other hand, if the image is classified as a product take-out image ("product take-out image" in S42), the processing device 10 performs the processes of S46 to S49. The processes of S46 to S49 are the same as the processes of S11 to S14 in FIG.

[0099] The other configurations of the processing device 10 are similar to those of the first to third embodiments.

[0100] The processing device 10 of this embodiment achieves the same effects as those of the first to third embodiments. Moreover, the processing device 10 can capture images of "a customer taking a product from a shelf" and "a store clerk restocking products on the shelf" with the same camera, and generate a product taking image and a product restocking image. This configuration reduces the number of cameras to be installed, and the cost required for the equipment is reduced.

[0101] Furthermore, the processing device 10 can automatically classify images generated by the camera into product removal images and product replenishment images. This eliminates the need for human labor, thereby reducing the workload of store clerks.

[0102] <Fifth embodiment> The processing device 10 of this embodiment has a function of automatically generating shelf-by-shelf display information using a method different from that of the third and fourth embodiments. An example of a functional block diagram of the processing device 10 of this embodiment is shown in Fig. 12. As shown in the figure, the processing device 10 has an acquisition unit 11, an identification unit 12, a first recognition unit 13, a storage unit 14, a second recognition unit 15, and a shelf-by-shelf display information generation unit 16.

[0103] The acquisition unit 11 acquires a plurality of product shelf images generated by photographing each of a plurality of product shelves. The product shelf images include products displayed on the product shelves. The configuration of the camera that generates the product shelf images is the same as the configuration of the camera that generates the product removal image described in the first embodiment. For example, by adjusting the shooting direction of the camera 2 as shown in Figs. 3 to 5, it is possible to capture an image that includes products displayed on the product shelves.

[0104] The camera that generates the product shelf image may be the same as the camera that generates the product removal image, or it may be a different camera. If the same camera is used, for example, by adjusting the shooting direction of camera 2 as shown in Figures 3 to 5, it is possible to capture an image that includes both "a customer removing a product from the product shelf" and "the product displayed on the product shelf."

[0105] The product shelf images generated by the camera may be input to the processing device 10 by real-time processing, or may be input to the processing device 10 by batch processing. However, in order to eliminate discrepancies between the shelf-by-shelf display information and the actual display situation, it is preferable to input the product shelf images to the processing device 10 by real-time processing and update the shelf-by-shelf display information with as little time loss as possible from the time the product shelf image is photographed.

[0106] The second recognition unit 15 recognizes products included in the product shelf image by a recognition process that compares the external appearance features of all products indicated by the product feature information. The first recognition unit 13, which recognizes products taken out from the product shelf by a customer, compares only the external appearance features of some products in the product feature information, whereas the second recognition unit 15, which recognizes products displayed on the product shelf, differs in that it compares only the external appearance features of all products indicated by the product feature information.

[0107] The shelf-by-shelf display information generating section 16 specifies which product shelf among the multiple product shelves is included in the first product shelf image, which is one of the multiple product shelf images acquired by the acquiring section 11.

[0108] For example, in the case where the installation positions of multiple cameras that capture multiple product shelves are fixed, the shelf-by-shelf display information generating unit 16 may identify which product shelf the first product shelf image includes by identifying the camera that generated the first product shelf image. In addition, in the case where information indicating the shooting position is attached to the product shelf image, the shelf-by-shelf display information generating unit 16 may identify which product shelf the first product shelf image includes by collating the information indicating the shooting position with information indicating the installation position of each of the multiple product shelves that is prepared in advance. In addition, in the case where information (such as a shelf number) for identifying each product shelf is physically attached to each product shelf and the product shelf image includes this information, the shelf-by-shelf display information generating unit 16 may identify which product shelf the first product shelf image includes by analyzing the product shelf image and recognizing the information attached to the product shelf.

[0109] In the following description, a "first product shelf image" is defined as an image that includes a "first product shelf" among a plurality of product shelves.

[0110] After the shelf-by-shelf display information generation unit 16 identifies that the first product shelf image is an image including the first product shelf, it registers the products recognized by the second recognition unit 15 as being included in the first product shelf image in the shelf-by-shelf display information (see Figure 7) as products displayed on the first product shelf.

[0111] Next, an example of the flow of processing by the processing device 10 will be described with reference to the flowchart of FIG.

[0112] First, when the acquisition unit 11 acquires a product shelf image to be processed (S50), the second recognition unit 15 recognizes the products included in the product shelf image acquired in S50 by performing a recognition process in which the external appearance features of all products indicated by the product feature information are compared (S51).

[0113] Next, the shelf-by-shelf display information generation unit 16 specifies which product shelf among the multiple product shelves is included in the product shelf image acquired in S50 (S52). Next, the shelf-by-shelf display information generation unit 16 registers the products recognized as being included in the product shelf image acquired in S50 in the shelf-by-shelf display information as products displayed on the product shelf specified in S52 (S53). Note that if the products have already been registered, the registration process may not be performed, and the registration process may be performed only if the products have not been registered.

[0114] Here, a modified example of this embodiment will be described. After identifying that the first product shelf image is an image including the first product shelf, the shelf-by-shelf display information generating unit 16 may analyze the first product shelf image to identify which tier of the first product shelf the product recognized to be included in the first product shelf image is displayed on. The algorithm for realizing this identification is not particularly limited. For example, when the first product shelf image includes multiple tiers, the area occupied by each tier in the image is registered in advance, and it is possible to identify which tier each product is displayed on based on the relationship between the area of ​​each tier in the image and the position of the recognized product in the image. Also, when an image is taken for each tier and a product shelf image is generated for each tier, it is possible to identify which tier of each product shelf each product shelf image includes by, for example, a process similar to the above-mentioned "process for identifying which product shelf among multiple product shelves the first product shelf image includes."

[0115] The shelf-by-shelf display information generating unit 16 specifies which shelf of the first product shelf the first product shelf image includes, and then the second recognition unit 15 recognizes the first product shelfThe product that is recognized as being included in the image is registered in the shelf-by-shelf display information (see FIG. 11) as a product displayed on the specified shelf of the first product shelf.

[0116] The other configuration of the processing device 10 is similar to that of the first and second embodiments.

[0117] According to the processing device 10 of this embodiment, the same operational effects as those of the first and second embodiments are realized. In addition, according to the processing device 10, display information for each shelf can be automatically created without manual intervention. This reduces the workload of store clerks.

[0118] Moreover, the processing device 10 identifies the products displayed on each product shelf by analyzing the product shelf image including the products displayed on the product shelf, and generates shelf-by-shelf display information based on the identification results. With such processing device 10, it is possible to generate shelf-by-shelf display information with fewer errors.

[0119] Furthermore, the processing device 10 can recognize products displayed on the display shelves by performing a recognition process that compares the external feature quantities of all products indicated by the product feature quantity information. Therefore, even if the products displayed on each product shelf change, the products can be correctly recognized and the correct information can be registered in the shelf-by-shelf display information.

[0120] Sixth embodiment The processing device 10 of this embodiment generates display information for each shelf using the method described in the fifth embodiment. The same camera captures images including "a customer taking a product from the product shelf" and "the products displayed on the product shelf." In other words, the image generated by the camera shows both "a customer taking a product from the product shelf" and "the products displayed on the product shelf." This configuration reduces the number of cameras to be installed, and reduces the cost of equipment.

[0121] However, if the second recognition unit 15 and shelf-by-shelf display information generation unit 16 described in the fifth embodiment were to execute the processing on all images generated by the camera, the processing load on the processing device 10 would increase.

[0122] Therefore, when a predetermined display content confirmation condition is met, the processing device 10 of this embodiment executes processing by the second recognition unit 15 and the shelf-by-shelf display information generation unit 16 on the image generated by the camera at the time when the condition is met, but does not execute processing by the second recognition unit 15 and the shelf-by-shelf display information generation unit 16 on images generated by the camera at other times.

[0123] Examples of the display content confirmation condition include "a predetermined time has come," "a display content confirmation instruction has been input from a store clerk," etc. For example, a store clerk inputs a display content confirmation instruction after displaying products on a display shelf.

[0124] Next, an example of the flow of processing by the processing device 10 will be described with reference to the flowchart of FIG.

[0125] First, when the acquisition unit 11 acquires an image to be processed (S60), the processing device 10 specifies which product shelf among multiple product shelves is included in the image acquired in S60 (S61).

[0126] Next, if the display contents confirmation condition is met (Yes in S62), the processing device 10 performs the processes of S63 and S64. The processes of S63 and S64 are the same as the processes of S51 and S53 in Fig. 16. Thereafter, the processing device 10 performs the processes of S65 to S67. The processes of S65 to S67 are the same as the processes of S12 to S14 in Fig. 9.

[0127] On the other hand, if the display content confirmation condition is not satisfied (No in S62), the processing device 10 performs the processes of S65 to S67 without performing the processes of S63 and S64.

[0128] The other configuration of the processing apparatus 10 is similar to that of the first, second and fifth embodiments.

[0129] The processing device 10 of this embodiment achieves the same effects as those of the first, second and fifth embodiments. In addition, the processing device 10 can capture images of "customers taking products off the shelves" and "products displayed on the shelves" with the same camera. This configuration reduces the number of cameras to be installed, and reduces the cost of equipment.

[0130] Furthermore, according to the processing device 10, when a predetermined condition for checking the display contents of a product shelf is satisfied, processing can be executed by the second recognition unit 15 and the shelf-by-shelf display information generation unit 16 to execute processing for generating shelf-by-shelf display information. In this case, it is possible to suppress the inconvenience of unnecessarily and frequently executing processing by the second recognition unit 15 and the shelf-by-shelf display information generation unit 16, which places a large processing burden on the computer. As a result, the processing burden on the processing device 10 is reduced.

[0131] Seventh embodiment The shelf-by-shelf display information generation unit 16 of the processing device 10 of this embodiment removes a product that has been removed from a product shelf and is therefore no longer displayed on the product shelf from the products displayed on that product shelf indicated by the shelf-by-shelf display information (see Figures 7 and 11).

[0132] For example, the shelf-by-shelf display information generation unit 16 counts the number of items removed from each product shelf based on the recognition results of the first recognition unit 13, and removes any items whose number of items reaches a reference value from the items displayed on each product shelf indicated by the shelf-by-shelf display information.

[0133] As another example, the shelf-by-shelf display information generation unit 16 can delete products whose sales volume indicated in the sales information managed by a POS (point of sales) system has reached a threshold value from the products displayed on each product shelf indicated by the shelf-by-shelf display information.

[0134] The reference value is the number of products displayed on the product shelf. A store clerk may input and register the reference value for each product into the processing device 10. Alternatively, the processing device 10 may specify the number of products displayed on the product shelf based on the recognition results of the second recognition unit 15 described in the third to sixth embodiments, and set the number as the reference value.

[0135] The other configuration of the processing apparatus 10 is similar to that of the first to sixth embodiments.

[0136] According to the processing device 10 of this embodiment, the same action and effect as those of the first to sixth embodiments can be realized. In addition, according to the processing device 10, the feature amount of a product that has disappeared from the product shelf can be excluded from the target of matching. As a result, the processing load of the processing device 10 can be further reduced.

[0137] Furthermore, the processor 10 can identify products that have disappeared from product shelves without manual intervention and automatically update the display information for each shelf, thereby reducing the workload of store clerks.

[0138] Furthermore, the processing device 10 can identify products that have disappeared from the shelves based on the recognition results of the first recognition unit 13 that recognizes products that customers have taken out of the shelves, sales information managed by the POS system, etc. Therefore, it is possible to identify products that have disappeared from the shelves with high accuracy.

[0139] In this specification, "acquisition" includes at least one of the following: "the device itself goes to retrieve data stored in another device or storage medium (active acquisition)" based on user input or based on program instructions, such as receiving data by making a request or inquiry to another device, or accessing another device or storage medium and reading it, and "the device itself inputs data output from another device (passive acquisition)" based on user input or based on program instructions, such as receiving data that is distributed (or transmitted, push notification, etc.), and selecting and acquiring data from among the received data or information, and "editing data (converting it to text, rearranging data, extracting some data, changing the file format, etc.) to generate new data and acquiring the new data."

[0140] Although the present invention has been described above with reference to the embodiments (and examples), the present invention is not limited to the above-mentioned embodiments (and examples). Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0141] A part or all of the above-described embodiments can be described as follows, but is not limited to the following. 1. An acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating the feature amount of the appearance of a plurality of commodities; A processing device having 2. The acquiring means further acquires a product replenishment image showing a state in which a store clerk replenishes products to the first product shelf, A second recognition means for recognizing products included in the product replenishment image by a recognition process that targets all products indicated by the product feature amount information as a comparison target; a shelf-by-shelf display information generating means for registering a product recognized as being included in the product replenishment image in the shelf-by-shelf display information as a product displayed on the first product shelf; 1. The processing device according to claim 1 . 3. The same camera generates the product removal image and the product replenishment image; 3. The processing device according to claim 1 or 2, further comprising a classification means for classifying the images generated by the camera into the product removal images and the product replenishment images. 4. The processing device according to claim 3, wherein the classification means performs the classification based on the time the image was taken, the external appearance features of the person contained in the image, or the mode that was set when the image was taken. 5. The acquiring means further acquires a product shelf image including products displayed on the first product shelf, a second recognition means for recognizing products included in the product shelf image by a recognition process that targets all products indicated by the product feature amount information as matching targets; a shelf-by-shelf display information generating means for registering a product recognized as being included in the product shelf image in the shelf-by-shelf display information as a product displayed on the first product shelf; 1. The processing device according to claim 1 . 6. The identification means is Identifying a shelf on which the product removed from the first product shelf was displayed based on the product removal image; Identifying a group of products displayed on the identified shelf of the first product shelf based on the shelf-by-shelf display information indicating products displayed on each shelf of each of the plurality of product shelves; The first recognition means is a processing device described in any one of 1 to 5, which recognizes the products included in the product removal image by performing a recognition process that uses the group of products identified as being displayed on the identified shelf from the product feature information as a matching target. 7. A processing device described in any one of 1 to 6, wherein the shelf-by-shelf display information generation means counts the number of items removed from the first product shelf based on the recognition result of the first recognition means, and removes items whose number of items reaches a reference value from the items displayed on the first product shelf indicated by the shelf-by-shelf display information. 8. A processing device described in any one of 1 to 6, wherein the shelf-by-shelf display information generating means removes products whose sales volume indicated in sales information managed by a POS system has reached a reference value from the products displayed on the first product shelf indicated by the shelf-by-shelf display information. 9. The computer: acquiring a product removal image showing a customer removing a product from a first product shelf; Identifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; A processing method for recognizing products included in the product-taken-out image by performing a recognition process in which the identified group of products is used as a comparison target among product feature information indicating the appearance features of multiple products. 10. Computer an acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating the feature amounts of the appearances of a plurality of commodities; A program that functions as a

Claims

1. an acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating an appearance feature amount of a plurality of commodities; having The acquiring means further acquires a product replenishment image showing a state in which a store clerk replenishes products to the first product shelf, A second recognition means for recognizing a commodity included in the commodity replenishment image by a recognition process using the commodity indicated by the commodity feature amount information as a collation target; a shelf-by-shelf display information generating means for registering a product recognized as being included in the product replenishment image in the shelf-by-shelf display information as a product displayed on the first product shelf; A processing device having

2. The same camera generates the product removal image and the product replenishment image; 2. The processing device according to claim 1, further comprising a classification means for classifying the images generated by the camera into the product removal images and the product replenishment images.

3. The processing device according to claim 2 , wherein the classification means performs the classification based on a shooting time of the image, a feature amount of an appearance of a person included in the image, or a mode set when the image was shot.

4. an acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; an identification means for identifying a target shelf, which is a shelf on which the product removed from the first product shelf was displayed, based on the product removal image, and for identifying a group of products displayed on the target shelf of the first product shelf, based on shelf-by-shelf display information indicating products displayed on each shelf of a plurality of product shelves; a first recognition means for recognizing a commodity included in the commodity removal image by a recognition process in which the commodity group identified as being displayed on the target shelf is used as a collation target among commodity feature amount information indicating feature amounts of the appearances of a plurality of commodities; A processing device having

5. an acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating the feature amount of the appearance of a plurality of commodities; a means for counting the number of items removed from the first product shelf based on the recognition result of the first recognition means, and removing an item whose number has reached a reference value from the items displayed on the first product shelf indicated by the shelf-by-shelf display information; A processing device having

6. an acquisition means for acquiring a product removal image showing a customer removing a product from a first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating the feature amount of the appearance of a plurality of commodities; a means for removing a product whose sales volume, indicated by sales information managed by a POS (point of sales) system, has reached a reference value from the products displayed on the first product shelf indicated by the shelf-by-shelf display information; A processing device having

7. The acquisition means further acquires a product shelf image including products displayed on the first product shelf, a second recognition means for recognizing products included in the product shelf image by a recognition process that targets all products indicated by the product feature amount information as matching targets; a shelf-by-shelf display information generating means for registering a product recognized as being included in the product shelf image in the shelf-by-shelf display information as a product displayed on the first product shelf; 7. The processing device according to claim 4, further comprising:

8. The computer acquiring a product removal image showing a customer removing a product from a first product shelf; Identifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; Recognizing the products included in the product-taken-out image by performing a recognition process using the identified product group as a collation target among product feature amount information indicating the feature amounts of the appearances of a plurality of products; further acquiring a product replenishment image showing a state in which a store clerk replenishes products on the first product shelf; Recognizing the products included in the product replenishment image by performing a recognition process using the products indicated by the product feature amount information as a collation target; A processing method for registering a product recognized as being included in the product replenishment image in the shelf-by-shelf display information as a product displayed on the first product shelf.

9. The computer acquiring a product removal image showing a customer removing a product from a first product shelf; Identifying a target shelf on which the product removed from the first product shelf was displayed based on the product removal image; Identifying a group of products displayed on the target shelf of the first product shelf based on shelf-by-shelf display information indicating products displayed on each shelf of each of a plurality of product shelves; A processing method for recognizing products contained in the product removal image by performing a recognition process that compares the group of products identified as being displayed on the target shelf from product feature information indicating the external features of multiple products.

10. The computer acquiring a product removal image showing a customer removing a product from a first product shelf; Identifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; Recognizing the products included in the product-taken-out image by performing a recognition process using the identified product group as a collation target among product feature amount information indicating the feature amounts of the appearances of a plurality of products; A processing method in which the number of items removed from the first product shelf is counted for each product based on the results of the recognition, and any product for which the number of items removed reaches a reference value is removed from the products displayed on the first product shelf indicated by the shelf-by-shelf display information.

11. The computer acquiring a product removal image showing a customer removing a product from a first product shelf; Identifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; Recognizing the products included in the product-taken-out image by performing a recognition process using the identified product group as a collation target among product feature amount information indicating the feature amounts of the appearances of a plurality of products; A processing method for removing a product whose sales volume, indicated by sales information managed by a POS system, has reached a reference value from the products displayed on the first product shelf indicated by the shelf-by-shelf display information.

12. Computer, an acquisition means for acquiring a product removal image showing a customer removing a product from the first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating feature amounts of the appearances of a plurality of commodities; Function as a The acquiring means further acquires a product replenishment image showing a state in which a store clerk replenishes products to the first product shelf, The computer further comprises: a second recognition means for recognizing a commodity included in the commodity replenishment image by a recognition process using the commodity indicated by the commodity feature amount information as a collation target; a shelf-by-shelf display information generating means for registering a commodity recognized as being included in the commodity replenishment image in the shelf-by-shelf display information as a commodity displayed on the first commodity shelf; A program that functions as a

13. Computer, an acquisition means for acquiring a product removal image showing a customer removing a product from the first product shelf; an identification means for identifying a target shelf, which is a shelf on which the product removed from the first product shelf was displayed, based on the product removal image, and for identifying a group of products displayed on the target shelf of the first product shelf, based on shelf-by-shelf display information indicating the products displayed on each shelf of each of a plurality of product shelves; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process in which the commodity group identified as being displayed on the target shelf is used as a collation target among commodity feature amount information indicating feature amounts of the appearances of a plurality of commodities; A program that functions as a

14. Computer, an acquisition means for acquiring a product removal image showing a customer removing a product from the first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating feature amounts of the appearances of a plurality of commodities; a means for counting the number of items taken out of the first product shelf based on the recognition result of the first recognition means, and removing an item whose number of items has reached a reference value from the items displayed on the first product shelf indicated by the shelf-by-shelf display information; A program that functions as a

15. Computer, an acquisition means for acquiring a product removal image showing a customer removing a product from the first product shelf; A specifying means for specifying a group of products displayed on the first product shelf based on shelf-by-shelf display information indicating products displayed on each product shelf; a first recognition means for recognizing a commodity included in the commodity-taken-out image by a recognition process using the identified commodity group as a collation target among commodity feature amount information indicating feature amounts of the appearances of a plurality of commodities; a means for removing a product whose sales volume, indicated by sales information managed by a POS system, has reached a reference value from the products displayed on the first product shelf indicated by the shelf-by-shelf display information; A program that functions as a

Citation Information

Patent Citations

  • Illegality monitoring system and program

    JP2004171241A

  • Shelving-allocation evaluation system, portable terminal, server and shelving-allocation evaluation method

    JP2018151842A

  • POS terminal device, POS system, product recognition method, and non-transient computer-readable medium having program stored thereon

    WO2015140853A1

  • Storefront device, storefront management method, and program

    WO2019038965A1