Information processing system, control method for information processing system, and program

Abstract

To conduct management of articles related to entry into and exit from a shelf by using an imaging part.SOLUTION: An information processing device that communicates to an imaging part to achieve the purpose mentioned above includes: moving direction identifying means that identifies the moving direction of an article based on an image taken of the front direction of the shelf by the imaging part; entry and exit identifying means that identifies the entry or exit of the article by the direction identified by the moving direction detection means; managing means that manages the number related to the articles on the shelf based on the entry and exit identified by the identifying means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program for managing inventory on shelves using an imaging unit. [Background technology]

[0002] In recent years, the logistics industry, including retail and distribution, has been facing a shortage of human resources and the need for more precise and efficient inventory management than ever before.In particular, inventory counts on shelves where product inventory is stored are still largely done manually.

[0003] RFID is known as a solution to these problems. RFID is a system in which an IC chip called an RF tag is attached to each item in stock, and product inventory is counted via wireless communication.

[0004] However, in actual operation, attaching RF tags to each and every product incurs physical and human costs, and the current situation is that most retail and logistics sites are unable to achieve cost-effectiveness that justifies the implementation.

[0005] Another technique is known in which weight sensors are attached to shelves to count the number of products, but this method has problems such as the cost of attaching weight sensors to each shelf being prohibitive and the inaccuracy of inventory counts due to variations in product weight.

[0006] In this context, a technology for managing shelf inventory using computer vision (image processing) technology has been devised. Patent Document 1 discloses a system for tracking the removal or placement of items in an inventory location having a material handling facility. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Table 2016-532932 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0008] Patent Document 1 describes a technology for determining which product has been taken from a shelf by detecting the position of the customer's hand from an image of the customer taking the product from the shelf.

[0009] However, the above technology requires an imaging device to capture images of the inventory location or a presence detection device to detect the inventory status, which poses a problem of high costs because an imaging device or presence detection device must be installed for each shelf level or each row of shelves.

[0010] Therefore, the present invention provides Based on the image The purpose is to manage items that are taken in and out of shelves. [Means for solving the problem]

[0011] The present invention provides an image capturing means for capturing an image including an area in front of an item take-out opening on a shelf within a photographing range; The image acquired by the image acquisition means Within If the first item is detected, Based on an image of the first article acquired by the image acquisition means, which is an image of the first article acquired before a first time point at which the first article is detected, The detection line relating to the shelf Before the first point in time That I passed judgement do judgement Means and The aforementioned judgement By means before Item 1 passed the detection line before the first time point. of judgement If you do, Applicable a storage / retrieval simulation means for determining that the first item has been removed from a shelf associated with the detection line; The present invention is characterized by having the following. [Effects of the Invention]

[0012] According to the present invention, Based on the image This has the effect of enabling management of items being taken in and out of the shelves. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a configuration diagram showing an outline of an inventory management system according to an embodiment of the present invention; [Figure 2] 1 is a configuration diagram showing the hardware configuration of an information processing apparatus 102 and various servers according to an embodiment of the present invention. [Figure 3] 1 is a configuration diagram showing a hardware configuration of a network camera 101 according to an embodiment of the present invention. [Figure 4] 10 is a flowchart showing an outline of a process for inventory management in the information processing device 102 according to the embodiment of the present invention. [Figure 5] 10 is a flowchart showing an outline of a product inventory tracking process in an information processing device 102 according to an embodiment of the present invention. [Figure 6] 10 is a flowchart showing an outline of a product removal process in the information processing device 102 according to the embodiment of the present invention. [Figure 7] 10 is a flowchart showing an outline of a product return process in the information processing device 102 according to the embodiment of the present invention. [Figure 8] FIG. 2 is a schematic diagram illustrating an image of a product shelf and an area (virtual grid area) from which products are taken out in an embodiment of the present invention. [Figure 9] 1 is a schematic diagram illustrating a product shelf and a processing image at the start of product pick-up in an embodiment of the present invention. FIG. [Figure 10] 1 is a schematic diagram illustrating a product shelf and a process image during product removal according to an embodiment of the present invention. FIG. [Figure 11] FIG. 10 is a schematic diagram illustrating a product shelf and a processing image at the end of product removal in an embodiment of the present invention. [Figure 12] 10 is a schematic diagram illustrating a product shelf and a processing image at the start of product return in an embodiment of the present invention. FIG. [Figure 13] 1 is a schematic diagram illustrating a product shelf and a process image during product return in an embodiment of the present invention. FIG. [Figure 14]FIG. 10 is a schematic diagram illustrating a product shelf and a processing image at the end of product return in an embodiment of the present invention. [Figure 15] FIG. 10 is a schematic diagram illustrating an image of detecting a different type of product within an area (virtual grid area) from which a product is taken out in an embodiment of the present invention. [Figure 16] FIG. 10 is a schematic diagram illustrating an image of detecting another product of the same type within an area (virtual grid area) from which a product is taken out in an embodiment of the present invention. [Figure 17] FIG. 1 is a schematic diagram illustrating an image of a product shelf having a height direction and an area (virtual grid area) from which products are taken out in an embodiment of the present invention. [Figure 18] 10 is an example of data held by an information processing device according to an embodiment of the present invention, for identifying a shelf from which a product is to be stored or retrieved based on the relationship between the coordinates and depth of a virtual grid where the product is detected. [Figure 19] 10 is an example of a data table storing the inventory quantity of a product shelf according to an embodiment of the present invention. [Figure 20] 10 is a flowchart showing an outline of a process of a second embodiment for performing inventory management in an information processing device 102 according to an embodiment of the present invention. [Figure 21] 10 is a flowchart showing an outline of a process of a second embodiment of product inventory tracking in the information processing device 102 according to the embodiment of the present invention. [Figure 22] 1 is a schematic diagram illustrating an image of a product shelf having a plurality of shelves in the height direction and an area (detection area) from which products are taken out in an embodiment of the present invention. [Figure 23] 10 is an example of a series of product images stored in the information processing device 102 according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] First Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] FIG. 1 is a diagram showing an outline of a configuration of an inventory management system according to an embodiment of the present invention.

[0016] Inventory monitoring system 105 is connected to network camera 101 and information processing device 102 that processes images captured by the camera, and network camera 101 captures an image of the access opening of product shelf 103 from top to bottom. Products 104 are displayed on product shelf 103. Note that although the example in which network camera 101 captures an image of the access opening from top to bottom has been described, the image of the access opening may also be captured from the side or from diagonally above. In this embodiment, an example in which the image of the access opening is captured from top to bottom will be described.

[0017] The status of products monitored by the inventory monitoring system 105 is counted via a network 106, for example, by an inventory management server 107 on the cloud, and the inventory status is provided to the user. Note that although the inventory management server 107 is illustrated in FIG. 1 as being housed separately from the information processing device 102, they may be housed in a single housing. That is, the information processing device 102 may have the functionality of the inventory management server 107, or the inventory management server 107 may have a function of processing captured images held by the information processing device 102.

[0018] The information processing device 102 may acquire information from one network camera 101 as shown in Figure 1 and send it to the inventory management server 107, or may be connected to multiple network cameras 101 and collect information from the multiple network cameras 101 and send it to the inventory management server 107.

[0019] Next, the schematic hardware configuration of the information processing device 102 will be described with reference to FIG.

[0020] FIG. 2 is a diagram showing the hardware configuration of the information processing device 102 and various servers according to the embodiment of the present invention.

[0021] The CPU 201 comprehensively controls each device and controller connected to the system bus 204 .

[0022] The ROM 202 or the external memory 211 also stores a BIOS (Basic Input / Output System), which is a control program for the CPU 201, an operating system program (hereinafter referred to as OS), and various programs (described later) required to realize the functions executed by the image processing server 108. The RAM 203 functions as the main memory, work area, etc. of the CPU 201.

[0023] The CPU 201 loads programs and the like required for executing processes into the RAM 203 and executes the programs to realize various operations.

[0024] An input controller (input C) 205 controls input from a keyboard serving as an input unit 209 or a pointing device such as a mouse (not shown).

[0025] A video controller (VC) 206 controls the display on a display device such as a CRT display (CRT) serving as a display unit 210. The display device is not limited to a CRT, but may be a liquid crystal display. These are used by the administrator as needed, and are not directly related to the present invention.

[0026] The memory controller (MC) 207 controls access to an external memory 211 such as a hard disk (HD), a floppy disk (FD) (registered trademark), or a CompactFlash (registered trademark) memory connected to a PCMCIA card slot via an adapter, which stores a boot program, browser software, various applications, font data, user files, edited files, various data, etc.

[0027] The communication I / F controller (communication I / FC) 208 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, internet communication using TCP / IP is possible. It also has the function of a communication I / F controller that can connect to the network camera 101 via the network.

[0028] The CPU 201 enables display on the display unit 210 by, for example, executing a process of expanding (rasterizing) an outline font into a display information area in the RAM 203. The CPU 201 also enables user instructions using a mouse cursor (not shown) or the like on the display unit 210.

[0029] A program for realizing the present invention is recorded in the external memory 211, and is loaded into the RAM 203 as needed and executed by the CPU 201. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 211, and will be described in detail later.

[0030] Next, the schematic hardware configuration of the network camera 101 will be described with reference to FIG.

[0031] FIG. 3 is a diagram showing the hardware configuration of the network camera 101. As shown in FIG.

[0032] The CPU 301 controls all devices and controllers connected to the system bus 304 .

[0033] The ROM 302 also stores a BIOS (Basic Input / Output System), which is a control program for the CPU 301, an operating system program (hereinafter referred to as OS), and various programs required to realize the function of generating data to be transmitted to the information processing device 102. The RAM 303 functions as the main memory, work area, etc. of the CPU 301.

[0034] The CPU 301 loads programs and the like required for executing processes into the RAM 303 and executes the programs to realize various operations.

[0035] The RGB camera unit 307 is connected to the image processing unit 308, and after the light obtained by passing through the lens directed toward the monitored object is photoelectrically converted by a light receiving cell such as a CCD or CMOS, the RGB signal and complementary color signal are output to the image processing unit 308.

[0036] The image processing unit 308 performs white balance adjustment, gamma processing, and sharpness processing based on the RGB signal and complementary color signal, and further performs YC signal processing to generate a luminance signal Y and a chrominance signal (hereinafter referred to as YC signal), compresses the YC signal in a predetermined compression format (for example, JPEG format or MotionJPEG format), and this compressed data is temporarily stored in RAM 303 as image data.

[0037] The TOF sensor 305 is an image sensor that measures the distance to an object using the TOF (Time-of-Flight) method, and measures the distance to the object by measuring the time it takes for light (infrared laser or LED) to reflect off the object and return to the sensor together with the depth measurement unit 306. Note that the means for measuring the depth of the product may be a stereo camera configured with two RGB cameras 307, or a method may be used in which the depth is estimated from the results of two-dimensional image analysis using image processing or deep learning.

[0038] The communication I / F controller (communication I / FC) 309 connects and communicates with external devices via a network, and performs communication control processing on the network. Image data stored in RAM 303 is transmitted by the communication I / F controller 309 to the information processing device 102, which is an external device.

[0039] Next, the flow of inventory management processing according to the first embodiment of the present invention will be described with reference to FIGS.

[0040] 4 is a flowchart showing an outline of a process for inventory management in the information processing device 102 according to an embodiment of the present invention, where S401 to S407 indicate each step. The process of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

[0041] 4 shows the flow of processing that starts when the network camera 101 and the information processing device 102 are started up. Note that in the following flowcharts, it is assumed that the information processing device 102 can acquire data from the network camera 101 via the network at any time.

[0042] Before starting the processing of the flowchart in FIG. 4, the CPU 201 of the information processing device 102 first acquires a captured image of the product outlet area of the product shelf obtained by the RGB camera 307 of the network camera 101.

[0043] Next, in step S401, the CPU 201 of the information processing device 102 sets a virtual grid that matches the lanes of the product shelves based on the acquired captured image. An example of setting the virtual grid will be described with reference to Figs. 8, 17, and 18.

[0044] FIG. 8 is a schematic diagram illustrating an image of a product shelf and an area (virtual grid area) from which products are taken out in the embodiment of the present invention.

[0045] In Fig. 8, 103 is a product shelf, 104 is an example of a product, and 404, which is an imaging range, corresponds to the product removal opening. In the following Figs. 8 to 17, the image of the product removal opening from the product shelf is taken from above, but it is also acceptable to take an image of the product removal opening from the product shelf from the side. In that case, the coordinates and depth of the product will be swapped in Fig. 18, which will be described later.

[0046] The product shelf 103 in FIG. 8 has lanes 410 arranged for each product, and positions 405 and 406 exist as boundaries between the lanes.

[0047] Boundary positions 405 and 406 are extended into image capture range 404 to set boundaries 407 and 408, and virtual grids 401 that separate these boundaries are created in image capture range 404. In other words, virtual grids are set as pick-up lanes for each lane on each product shelf.

[0048] Figure 17 shows the results of performing these processes on multiple shelves.

[0049] FIG. 17 is a schematic diagram illustrating an image of a product shelf having a height direction and an area (virtual grid area) from which products are taken out in an embodiment of the present invention.

[0050] The product shelf 103 in FIG. 17 is made up of multiple tiers, and FIG. 17 shows an example in which there are three tiers of shelves.

[0051] The virtual grid for the three-tiered shelf in Figure 17 varies depending on the number of tiers, with the lane boundary for the first tier being 1204, the lane boundary for the second tier being 1205, and the lane boundary for the third tier being 1206. The virtual grid boundaries for each tier are set as 1207 for the first tier, 1208 for the second tier, and 1209 for the third tier.

[0052] A virtual grid of three shelves is set as shown in FIG. 17, and data relating the positions of the shelves will be explained with reference to FIG.

[0053] FIG. 18 shows an example of data for identifying shelves to be accessed from the relationship between the coordinates and depth of a virtual grid where a commodity is detected, which is held by an information processing device according to an embodiment of the present invention.

[0054] Reference numeral 1800 in Fig. 18 denotes data that identifies the position of the product shelf that has entered or exited from the depth 1801 obtained from the depth measurement unit 306 and the coordinates 1802 of the product position acquired by the RGB camera. For example, if a product has entered or exited at a depth of 20 cm and its grid coordinates (horizontal coordinates in Fig. 17, with the center as the origin) are at 70, it is determined that the product has entered or exited from grid C on the first shelf (i.e., grid C-1). Also, if a product has entered or exited at a depth of 80 cm and its grid coordinates are at -60, it is determined that the product has entered or exited from grid A on the third shelf (i.e., grid A-3). The direction of entry or exit will be described later in the next step S402.

[0055] As shown in Fig. 18, a table is provided for storing the positions of shelves where products are moved in and out. This table may be entered manually by the user, or may be automatically set based on the boundaries (dividers) of the product shelves in the image captured before the processing of step S401. Returning to the explanation of the flowchart in Fig. 4.

[0056] 4, the CPU 201 of the information processing device 102 sets the direction in which the product is being taken out based on the orientation of the product moving within the virtual grid. This will be described with reference to FIG.

[0057] In Fig. 8, the product shelf 103 is located at the bottom of the drawing, and the removal opening side (virtual grid side) 404 is located at the top of the drawing, so the product removal direction is set to be upward as shown by 402, which is the positive direction in the Y-axis direction of coordinate axis 420. This removal direction may be set manually by the user, or may be set by specifying the position of the product shelf 103 on the captured screen (Fig. 8). Returning to the explanation of the flowchart in Fig. 4.

[0058] 4, the CPU 201 of the information processing device 102 receives an input of the inventory quantity of the lane of the product shelf corresponding to the virtual grid set in step S401. This will be described with reference to FIGS.

[0059] 8 is set, if the inventory of the lane of the product shelf corresponding to each virtual grid is, for example, 10 units (411) in grid A, 5 units (412) in grid B, and 1 unit (413) in grid C, the user sets the inventory quantity of each lane by inputting the inventory quantity from the input unit 209 of the information processing device 102. An example of data for storing the set inventory quantity will be described with reference to FIG.

[0060] FIG. 19 is an example of a data table that stores the inventory quantity of a product shelf according to an embodiment of the present invention.

[0061] The table in Fig. 19 has the inventory quantities set in the first row in Fig. 8, with data registered such that 10 units are in grid A, 5 units are in grid B, and 1 unit is in grid C. The table in Fig. 19 may be stored in the information processing device 102 or in the inventory management server 107. Returning to the explanation of the flowchart in Fig. 4.

[0062] In the next step S404 in FIG. 4, the CPU 201 of the information processing device 102 transmits a command to start image capture (image capture) to the network camera 101, and starts image capture.

[0063] In the next step S405, the CPU 201 of the information processing device 102 determines whether a product has been detected within the captured image of the virtual grid 401. For example, when a network camera is attached to the top as shown in FIG. 8, the product may be detected by storing a pre-stored top view of the product and performing image matching using a SIFT (Scale-Invariant Feature Transform) algorithm. Alternatively, various top-view images of the product may be registered in advance as learning data and machine learning may be performed to detect the product using image recognition AI.

[0064] In step S405, if a product is detected, the process proceeds to step S406, and if a product is not detected, the process proceeds to step S407.

[0065] When the process proceeds to step S406, the CPU 201 of the information processing device 102 performs a product tracking process to grasp the inventory status of the product detected in step S405. The process content of step S406 will be described with reference to FIG.

[0066] 5 is a flowchart showing an outline of the product inventory tracking process in the information processing device 102 according to the embodiment of the present invention, where S501 to S514 indicate each step. The process of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

[0067] The flowchart in FIG. 5 shows the flow of processing that starts when the process transitions to step S406 in the flowchart in FIG.

[0068] 5, the CPU 201 of the information processing device 102 acquires the location (coordinates) of the detected product from the captured image (capture) acquired from the network camera 101, and stores the location (coordinates) as the start coordinates. A description with reference to specific product images will be given later with reference to FIGS. 9 to 14.

[0069] Next, in step S502, the CPU 201 of the information processing device 102 acquires the height (depth) of the product detected by the TOF sensor 305 provided in the network camera 101, and stores it as the start depth.

[0070] Next, in step S503, the CPU 201 of the information processing device 102 acquires the next captured image (capture) from the network camera 101.

[0071] Next, in step S504, the CPU 201 of the information processing device 102 determines whether the same type of product as the previous one has been detected in the virtual grid 401 captured in step S503. If the same type of product has been detected, the process proceeds to step S505. If the same type of product has not been detected, the process proceeds to step S509. Note that if a different type of product has been detected in the virtual grid 401 captured in step S503 from the previous one (as in the case of FIG. 15), the flowchart of FIG. 5 is executed for the different type of product, starting from step S501.

[0072] When the process proceeds to step S505, the CPU 201 of the information processing device 102 acquires the location (coordinates) of the detected product from the captured image (capture) captured in step S503 and stores the location (coordinates) as the current coordinates. The CPU 201 also stores the previously acquired current coordinates and product type as the "previously acquired current coordinates" and the previous product type.

[0073] Next, in step S506, the CPU 201 of the information processing device 102 acquires the height (depth) of the product detected when it was imaged in step S503, and stores it as the current depth.

[0074] Next, in step S507, the CPU 201 of the information processing device 102 compares the distance between the "current coordinates acquired one time previously" and the current coordinates newly acquired in step S505, and if there are only products whose distance is equal to or greater than a predetermined threshold, the process proceeds to step S510. On the other hand, if there are products whose distance between the "current coordinates acquired one time previously" and the current coordinates newly acquired in step S505 is within the threshold, the process proceeds to step S508. A specific example will be described later with reference to FIG. 16.

[0075] The decision branch in step S507 occurs when multiple items of the same type enter virtual grid 401 at the same time, and the fact that the two distances are equal to or greater than the threshold means that the individual items are determined to be different. If the two distances are within the threshold, the item at the "previously acquired current coordinates" and the item newly detected in step S505 are recognized as the same individual item (processing in step S508).

[0076] After the process of step S508, the process returns to step S503, and the process of acquiring the captured image within the virtual grid 401 is repeated.

[0077] On the other hand, in step S504, if the same type of product as the previous one is not detected in the virtual grid 401, the process proceeds to step S509.

[0078] When the process proceeds to step S509, the CPU 201 of the information processing device 102 determines whether the same product has not been detected a predetermined number of times or more. If the same product has not been detected a predetermined number of times or more, the process proceeds to step S510, and if the same product has been detected within the predetermined number of times, the process returns to step S503.

[0079] The decision branch in step S509 is a decision branch that prevents the system from determining that the product has gone outside the virtual grid 401 even if the product is within the virtual grid 401 but cannot be detected temporarily due to afterimages or light disturbances in the image captured by the imaging unit.

[0080] The process proceeds to the next step S510 if the current coordinates have been stored at least once in step S509 (YES in step S509), or if the distance between the "previously acquired current coordinates" in step S507 and the new current coordinates detected in step S505 is greater than or equal to a threshold value.

[0081] In step S510, the CPU 201 of the information processing device 102 calculates the direction of movement of the product from the start coordinates, which are the first detection position of the product obtained in step S501, and the "last current coordinates," which are the last detection position of the product obtained in step S505.

[0082] Next, in step S511, the CPU 201 of the information processing apparatus 102 divides the processing depending on the orientation calculated in step S510.

[0083] If the orientation of the start coordinate and the "last current coordinate" is the same, for example, if the orientation from the center of the virtual grid in the Y-axis direction of coordinate axis 420 in Fig. 8 is the same, the processing of the flowchart in Fig. 5 ends. This decision is made when a product enters the virtual grid but does not leave (for example, when a customer takes a product off the shelf and considers purchasing it, but then decides not to buy it and returns it to the shelf).

[0084] Also, in step S511, if the direction of movement of the product is the direction of removal 402, the process proceeds to step S512.

[0085] On the other hand, in step S511, if the direction of movement of the product is the return direction (opposite to the direction of removal), the process proceeds to step S513.

[0086] When the process proceeds to step S512, the CPU 201 of the information processing device 102 performs a product removal process, and then the process proceeds to step S514. The detailed process will be described later with reference to FIG.

[0087] Furthermore, when the process proceeds to step S513, the CPU 201 of the information processing device 102 performs a product return process, and then proceeds to step S514. The detailed process will be described later with reference to FIG.

[0088] When the process proceeds to step S514, the CPU 201 of the information processing device 102 notifies the inventory management server to change the inventory quantity on the shelf identified in step S512 or step S513, and ends the flowchart in Fig. 5. Returning to the description of the flowchart in Fig. 4.

[0089] In step S406, product tracking processing is performed to grasp the inventory status of the product detected in step S405, and then in the next step S407, it is determined whether or not to end the image capture processing of the network camera 101 for checking the product inventory. If the image capture processing is to be continued, the processing returns to step S404, and the processing from step S404 is repeated. On the other hand, if an instruction to end the image capture processing is received from the user, the image capture processing is stopped and the processing of the present invention is ended.

[0090] Next, with reference to FIG. 6, a process for identifying the position of the shelf from which the product is to be taken out will be described.

[0091] 6 is a flowchart showing an overview of the product removal process in the information processing device 102 according to the embodiment of the present invention, where S601 to S602 indicate each step. The process of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

[0092] The flowchart in FIG. 6 shows the flow of processing that starts when the process transitions to step S512 in the flowchart in FIG.

[0093] First, in step S601 of Fig. 6, the CPU 201 of the information processing device 102 identifies the number of shelves from which the product has been taken out based on the start depth acquired in step S502. Specifically, with reference to Fig. 18, if the product depth acquired in step S502 is 20 cm, for example, it is identified from the data 1801 and 1803 that the height of the taken-out shelf is the first shelf.

[0094] Next, in step S602, the CPU 201 of the information processing device 102 identifies the position of the shelf taken out from the start coordinates acquired in step S501. To explain this in detail with reference to Fig. 18, for example, if the grid coordinate of the product acquired in step S501 is 80, it is identified from the data 1802 that the product is C-1, whose grid coordinate of the first shelf is 80, i.e., shelf C on the first shelf.

[0095] In the above example, it is determined that the product was taken out from shelf C on the first shelf.

[0096] Through the above process, the location of the shelf from which the customer took out the product can be identified.

[0097] Next, with reference to FIG. 7, a process for identifying the position of the shelf to which the product has been returned will be described.

[0098] 7 is a flowchart showing an overview of the product removal process in the information processing device 102 according to the embodiment of the present invention, where S701 to S702 indicate each step. The process of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

[0099] The flowchart in FIG. 7 shows the flow of processing that starts when the process transitions to step S513 in the flowchart in FIG.

[0100] First, in step S701 of Fig. 7, the CPU 201 of the information processing device 102 identifies the number of shelves to which the product has been returned from the latest current depth acquired in step S506. To explain this in detail with reference to Fig. 18, for example, if the depth of the product acquired in step S502 is 80 cm, it is identified from the data 1801 and 1803 that the returned shelf is the third shelf.

[0101] Next, in step S702, the CPU 201 of the information processing device 102 identifies the position of the shelf that was taken out from the latest current coordinates acquired in step S505. To explain this in detail with reference to Fig. 18, for example, if the grid coordinate of the product acquired in step S501 is 80, it is identified from the data 1802 that the grid coordinate of the third row is C-3, which is 80, i.e., shelf C on the third row.

[0102] In the above example, it is identified that the product has been returned to shelf C on the third shelf.

[0103] By performing the above process, the location of the shelf to which the customer returned the product can be identified.

[0104] Next, a process for identifying the position of a shelf where a product enters or leaves a shelf will be described in detail with reference to FIGS. 9 to 14, based on an image of a product entering or leaving a shelf.

[0105] FIG. 9 is a schematic diagram illustrating a product shelf and a processing image at the start of product pick-up in an embodiment of the present invention.

[0106] 9, when a product 104 is detected as 801 in the virtual grid 401 (processing of step S405), first, the start coordinates 802 of the product are identified (processing of step S501). Next, the start depth of the product is also identified (processing of step S502), and the respective values are stored.

[0107] FIG. 10 is a schematic diagram illustrating a product shelf and a process image during product pick-up in an embodiment of the present invention.

[0108] In FIG. 10, when the product 104 moves within the virtual grid, the current coordinates 802 to 804 are continuously acquired until the product 104 disappears from the virtual grid (processing in step S505).

[0109] FIG. 11 is a schematic diagram illustrating a product shelf and a processing image at the end of product pick-up in an embodiment of the present invention.

[0110] In FIG. 11, when the current coordinates of the product 104 no longer exist in the virtual grid as in 805 (transition to NO in the decision branch of step S504), 804 is set as the current coordinates in the last virtual grid (processing of step S510).

[0111] Next, the direction of movement 806 of the product is calculated from the information on the product's start coordinates 801 and the latest current coordinates 804. In the case of Fig. 11, this is the same as the take-out direction 402 in Fig. 8, so it is identified as the take-out direction (processing of step S510).

[0112] When the product has moved as shown in the image from FIG. 9 to FIG. 11, it is determined in step S511 that the product has moved in the direction of removal, and the process proceeds to step S512 (ie, the process in FIG. 6).

[0113] In the case of Figure 11, the position of the shelf from which the item was removed is identified from the start depth acquired at the same time as the start coordinate 801 (processing of steps S601 and S602 in Figure 6). In Figure 11, one item has been removed from grid B (the center shelf), so the inventory quantity is reduced from 5 to 4 as shown in 807. Information on this reduction in inventory quantity is sent to the inventory management server 107 (processing of step S514).

[0114] As shown in the image above, by capturing an image of the shelf's access point, it is possible to grasp the product removal status.

[0115] Next, a process image when a product is returned to a shelf will be described with reference to FIGS.

[0116] FIG. 12 is a schematic diagram illustrating a product shelf and a processing image at the start of returning a product in an embodiment of the present invention.

[0117] 12, when a product 104 is first detected as 901 in the virtual grid 401 (processing of step S405), the start coordinates 901 of the product are first identified (processing of step S501). Next, the start depth of the product is also identified (processing of step S502), and each value is stored.

[0118] FIG. 13 is a schematic diagram illustrating a product shelf and a process image during product return in an embodiment of the present invention.

[0119] 13, when the product 104 moves within the virtual grid, the current coordinates of the current coordinates 902 to 905 are continuously acquired until the product 104 disappears from the virtual grid (processing of step S505). Subsequently, the value of the current depth is also continuously acquired.

[0120] FIG. 14 is a schematic diagram illustrating a product shelf and a processing image at the end of returning a product in the embodiment of the present invention.

[0121] In FIG. 14, when the current coordinates of the product 104 are no longer present in the virtual grid as in 906 (transition to NO in the decision branch at step S504), 905 is set as the current coordinates in the last virtual grid (processing at step S510).

[0122] Next, the direction of movement 907 of the product is calculated from the information on the product's start coordinates 901 and the latest current coordinates 905. In the case of Fig. 14, the direction of movement 907 of the product is opposite to the direction of removal 402 in Fig. 8, so it is identified as the direction of return (processing of step S510).

[0123] When the product has moved as shown in the image from FIG. 12 to FIG. 14, it is determined in step S511 that the product has moved in the direction of returning, and the process proceeds to step S513 (that is, the process in FIG. 7).

[0124] In the case of Figure 14, the position of the shelf from which the item was removed is identified from the start depth acquired at the same time as the last current coordinates 905 (processing of steps S701 and S702 in Figure 7). In the example of Figure 14, one item has been returned to grid A (left shelf), so the inventory quantity is increased from the original 10 to 11, as shown in 908. Information about this increase in inventory quantity is sent to the inventory management server 107 (processing of step S514).

[0125] As shown in the image above, by capturing an image of the shelf's access point, it is possible to grasp the status of product returns.

[0126] Next, with reference to FIG. 16, an overview of the processing in steps S507 and S508 will be described.

[0127] FIG. 16 is a schematic diagram illustrating an image of detecting another product of the same type within the virtual grid area 401 according to an embodiment of the present invention.

[0128] The image in FIG. 16 shows an example in which, after a product 104 is detected at 1102 (step S501), the same product is detected at 1103 and 1104 in the next imaging process (step S503).

[0129] In Figure 16, when the threshold value indicating product movement is 1101, the distance between the start coordinate 1102 and the current coordinate 1104 is equal to or greater than the threshold (1106), while the current coordinate 1103 is within the threshold (1105), so it is determined that the product moved from the start coordinate 1102 has moved to 1103. The threshold value is set based on the speed at which the customer moves the product and the time interval for performing image capture processing. This threshold value must be shorter than the lane interval on the product shelf (the distance between 405 and 406 in Figure 8), so the time interval for performing image capture processing may be set in relation to the speed at which the customer moves the product.

[0130] With the above processing, even if the image capture frame rate is slow and there is a limit to the speed at which products can be tracked, if two products are recognized but are separated by more than the threshold value 1101, the two products can be identified and their movement on and off the shelves can be managed. <Second embodiment> In the first embodiment, the product is tracked after identifying the product from the image captured by the imaging unit (network camera) 101. In the second embodiment, by identifying the product after tracking the product, the process of identifying the product from the captured image each time an image is captured is omitted, and the frame rate of the imaging unit can be increased. That is, in the second embodiment, when a product leaves the detection area (virtual grid) 401, the product is identified collectively from the accumulated images, thereby preventing the product identification process from being performed each time an image is captured and increasing the frame rate of the imaging unit. The flow of inventory management processing in the second embodiment of the present invention will be described with reference to Figures 20 and 21.

[0131] 20 is a flowchart showing an outline of a process for inventory management in the information processing device 102 according to the second embodiment of the present invention, in which S2001 to S2003 indicate each step. The process of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

[0132] 20 shows the flow of processing that starts when the imaging unit 101 and the information processing device 102 are started up. Note that in the following flowcharts, it is assumed that the information processing device 102 can acquire data from the network camera 101 via the network at any time.

[0133] Before starting the processing of the flowchart in FIG. 20, the CPU 201 of the information processing device 102 first acquires a captured image of the product take-out opening portion of the product shelf obtained by the RGB camera 307 of the imaging unit 101.

[0134] Next, in step S2001, the CPU 201 of the information processing device 102 accepts the setting of a detection area 401 that matches the lane and height of the product shelf based on the acquired captured image. The method of setting the detection area is similar to that shown in FIGS. 8, 17, and 18, and therefore description thereof will be omitted. Note that in the first embodiment, since it is assumed that the frame rate of the imaging unit is low, the width of the detection area 401 in the product removal direction (Y-axis direction) is ensured to be wide as shown in FIGS. 8, 17, and 18. However, in the second embodiment, since the frame rate of the imaging unit is high, the width of the detection area 401 in the product removal direction (Y-axis direction) can be set narrower as shown in detection area 2210 in FIG. 22.

[0135] Next, in step S2002, the CPU 201 of the information processing device 102 accepts the setting of a detection line corresponding to the entrance / exit of the product shelf based on the acquired captured image. The setting of the detection line will be described with reference to FIG.

[0136] FIG. 22 is a schematic diagram illustrating an image of a product shelf having a plurality of shelves in the height direction and an area (detection area) from which products are taken out in an embodiment of the present invention.

[0137] In Figure 22, 103 is a product shelf, 104 is an example of a product, and 404, which is an imaging range, corresponds to the product removal port. Note that since the detection of the lateral direction (each lane) of each shelf is the same as in the first embodiment, the following explanation will be omitted. In the example of FIG. 22, the product shelf 103 has three shelves.

[0138] In the detection area 2210 of FIG. 22, detection lines 2201-2203 are set for each of the three shelves. While the detection lines 2201-2203 are shown in different positions in FIG. 22 to facilitate understanding, the detection lines are set at the edges of the detection area 2210 (in FIG. 22, the bottom edge of the rectangle of the detection area 2210), and the widths of the shelves vary. The widths are, for example, the widths set in FIG. 18. Whether a product has entered or left the corresponding shelf can be determined by passing through this detection line, preventing erroneous detection of products that have entered or left the adjacent shelf, or preventing erroneous detection of products that happen to pass through the detection area (for example, when a customer crosses the detection area from right to left with a product in hand). While the embodiment is described using one inventory monitoring system 105, it is also possible that multiple inventory monitoring systems 105 with imaging units 101 are installed, and product detection is performed on the racks of each inventory monitoring system 105. In this case, erroneous detection of products on adjacent racks can be prevented. Returning to the explanation of the flowchart in FIG.

[0139] Next, when the process proceeds to step S2003 in Fig. 20, the CPU 201 of the information processing device 102 starts a process for managing product inventory. The process content of step S2003 will be described with reference to Fig. 21.

[0140] 21 is a flowchart showing an outline of the processing of a second embodiment of product inventory tracking in the information processing device 102 according to the embodiment of the present invention, in which S2101 to S2118 indicate each step. The processing of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

[0141] The flowchart in FIG. 21 shows the flow of processing that starts when the process transitions to step S2003 in the flowchart in FIG.

[0142] First, in step S2101 of Fig. 21, the CPU 201 of the information processing device 102 acquires a captured image (capture) from the imaging unit 101. This acquisition of the captured image may be performed continuously during the processing of the flowchart of Fig. 21. Note that the processing of step S2101 may perform object detection by the TOF sensor 305 at the same time as acquiring the captured image.

[0143] Next, in step S2102, the CPU 201 of the information processing device 102 determines whether an object such as a commodity or a hand has been detected within the detection area 2210 in the shooting area 404. If an object has been detected, the process proceeds to step S2103; if an object has not been detected, the process returns to step S2101 and is repeated until an object is detected. Whether an object has been detected may be determined by object detection using a TOF sensor, detection based on image difference information, or object detection using a stereo camera.

[0144] Next, in step S2103, the CPU 201 of the information processing device 102 identifies the position where the object is detected by the TOF sensor, etc. This position information is acquired as needed, and the position information can be tracked at all times.

[0145] Next, in step S2104, the CPU 201 of the information processing device 102 stores an image of the area around the position where the object is detected. An example of the stored image will be described with reference to FIG.

[0146] FIG. 23 shows an example of a series of product images stored in the information processing device 102 according to the embodiment of the present invention.

[0147] Images 2301 to 2312 in Figure 23 are displayed in chronological order, in the order of detected objects. Specifically, Figure 23 shows a series of images of a plastic bottle being returned to a shelf by a customer. By limiting the range of images captured and stored as in this process, or by using this range for product identification, it is possible to reduce the image storage memory and shorten the image processing time required for product identification compared to the method of identifying objects from the entire image as in the first embodiment. Returning to the explanation of the flowchart in Figure 21.

[0148] Next, in step S2105 of FIG. 21, the CPU 201 of the information processing device 102 acquires height information (depth information) of an object from the TOF sensor or the stereo camera.

[0149] Next, in step S2106, the CPU 201 of the information processing device 102 determines whether the object has passed through the detection lines 2201 to 2203 based on the tracking information acquired in step S2103. Whether the object has passed through the detection lines is determined based on the object height information acquired in step S2105 and the tracking information acquired in step S2103. For example, if the object height information is detected as the height of the first shelf (20 cm in the example of FIG. 18), the detection line for the first shelf is "-150 to 150" in size on the screen in FIG. 18. Similarly, if the object height information is detected as the height of the third shelf (80 cm in the example of FIG. 18), the detection line for the third shelf is "-90 to 90" in size on the screen in FIG. 18. If the object has passed between these, it is determined that the object has passed through the detection lines. In other words, this means that a product or a hand has entered or exited the area between the shelf and the area in front of the shelf. On the other hand, if the object's height information is detected as the height of the third shelf and the horizontal detection line passes, for example, a position of "110", it is determined that the object is being taken in or out of the adjacent shelf, and it is not considered to have passed the corresponding detection line.

[0150] If it is determined in step S2106 that the object has passed the detection line, a flag (not shown) indicating that the object has passed the detection line is set, and the process proceeds to step S2108. At the same time, height information is stored. Thereafter, as long as the detection line passing flag is set, the process proceeds to Yes in step S2106. On the other hand, if it is determined that the object has not passed the detection line, the process proceeds to step S2107.

[0151] When the process moves to step S2107, the CPU 201 of the information processing device 102 determines whether the tracked object has left (OUT) the detection area 2210. If it is determined that the object has left the detection area 2210, this means that the object has left the detection area without passing through the detection line (i.e., has not touched the inside of the shelf), so a process of deleting the stored image and height information (step S2118) is executed, and the process returns to the initial process. On the other hand, if the object has not left the detection area, this means that the object is within the detection area, so the stored image is not deleted, and the process returns to the initial process.

[0152] If the process proceeds to step S2108, that is, if the object has passed the detection line at least once, the CPU 201 of the information processing device 102 determines whether the object has left the detection area in the opposite direction to the detection line. If the object has left the detection area in the opposite direction to the detection line, the process proceeds to step S2111, and if the object has not left the detection area in the opposite direction to the detection line, the process proceeds to step S2109. In the example of FIG. 22, leaving the detection area in the opposite direction to the detection line is determined by whether the object has left the detection area in the upward direction as indicated by 2211. In other words, the process of step S2108 is a process for determining whether a product or a hand has been removed from the shelf.

[0153] When the process proceeds to step S2109, the CPU 201 of the information processing device 102 determines whether or not the object has passed through the detection line and exited the detection area. If the object has passed through the detection line, the process proceeds to step S2110, and if the object has not passed through the detection line, the process proceeds to step S2117. In the example of FIG. 22, whether or not the object has passed through the detection line is determined by whether or not the object has passed through detection lines 2201 to 2203 and exited the detection area. In other words, the process of step S2109 is a process for determining whether or not a product or hand has entered the shelf.

[0154] Note that even if an object exits the detection line, if the object entered from the same detection line and is being tracked, the process transitions to No and the decision branch at step S2117. The reason for transitioning to No is to prevent the product from being counted up if, for example, a customer takes a product off a shelf and returns it to the shelf without removing it from the detection area.

[0155] When the process proceeds to step S2110, the CPU 201 of the information processing device 102 sets a count-up flag (not shown) to execute a process to increase inventory if the object is a commodity. Thereafter, the process proceeds to step S2112.

[0156] On the other hand, if the process proceeds to step S2111, the CPU 201 of the information processing device 100 sets a countdown flag (not shown) to execute a process to reduce inventory if the object is a commodity. Thereafter, the process proceeds to step S2112.

[0157] When the process proceeds to step S2112, the CPU 201 of the information processing device 100 reads the plurality of captured images stored in step S2104 and acquires the images.

[0158] Next, in step S2113, the CPU 201 of the information processing device 100 performs a process of identifying a product through image recognition using AI. Specifically, several thousand images (several thousand images of a plastic bottle taken from various angles when inferring an image such as that shown in FIG. 23) are registered as training data for the AI, and the AI is trained using a deep learning algorithm such as VGG or MobileNet. For example, multiple images 2301 to 2312 shown in FIG. 23 are input to the trained model, and the inferred plastic bottle product is identified. In this way, since a single product is inferred from multiple images of the object's surroundings, the image analysis speed is increased compared to the process of identifying a product from a single overall image as in the first embodiment, and the frame rate of the imaging unit can be increased. Note that product identification may also be performed by capturing, identifying, and matching tags attached to the product without using AI. Alternatively, multiple images may be stored and the product may be identified simply based on matching conditions. In such cases, multiple images of the object's surroundings can be used to make judgments, thereby facilitating image analysis processing and increasing processing speed.

[0159] Next, in step S2114, CPU 201 of information processing device 100 determines whether the object identified in step S2113 is a product or not. If it is a product, the process proceeds to step S2115, and if it is a hand and not a product, or if the product cannot be identified, the process proceeds to step S2118.

[0160] When the process transitions to step S2118, the CPU 201 of the information processing device 100 deletes the stored image, object height information, detection line crossing flag, inventory change count, etc. The transition to step S2118 indicates that an object has left the detection area and the detected object is a hand or an unidentifiable product, or the object has left the detection area without putting its hand inside the shelf. Therefore, the stored image, object height information, detection line crossing flag, and inventory change count are no longer needed, and therefore these data are deleted. The process then returns to the initial processing of this flowchart. Note that if it is possible to distinguish between a user's hand and an unrecognizable product in step S2113, a configuration may be adopted in which, when an unrecognizable product is placed on the shelf, an alert is issued to notify the store's product manager that an unrecognizable product is present.

[0161] On the other hand, if the process transitions to step S2115 as a result of the decision branch in step S2114, the CPU 201 of the information processing device 100 sets information to increase or decrease the inventory of the product inferred in step S2114 on the shelf of the height memorized in step S2106 by the inventory increase / decrease amount in step S2110 or step S2111.

[0162] Next, in step S2116, CPU 201 of information processing device 100 transmits the shelves, products, and increase / decrease information set in step S2115 to inventory management server 107, and the process proceeds to step S2117.

[0163] When the process proceeds to step S2117, the process receives a command from the user (not shown) to terminate the flowchart of this embodiment. If the command to terminate is received from the user, the process terminates. If the command to terminate is not received from the user, the process returns to the first step in the flowchart.

[0164] As in the above process, object detection and tracking are performed first, and when an object leaves the detection area, the captured image is used to identify the product and manage inventory information. This eliminates the process of identifying the product from the entire image each time it is captured, increases the frame rate of the imaging unit, and enables finer tracking.

[0165] In the second embodiment, whether a product has been taken in or out of a shelf is determined based on whether the product has passed the detection line, but it may also be determined that the product has been taken in or out of a shelf when the product has moved in the removal direction (Y coordinate) by more than a predetermined threshold. In this case, the height information of the shelf where the product is taken in or out is set from the height when the product passed the detection line (step 2106).

[0166] By the above processing, the present invention has the effect of managing the entry and exit of products from shelves consisting of multiple rows with a small number of imaging devices, regardless of whether or not images of the inventory locations are taken.

[0167] In addition to inventory management, data on shelf height and horizontal position can be acquired sequentially, which has the effect of enabling acquisition of information for appropriate allocation of shelf layouts.

[0168] In addition, since the location of products and their entry and exit can be identified, information can be obtained that makes it easy to allocate shelves appropriately to increase sales.

[0169] Furthermore, inventory management requires identifying the type of item, but identifying the type of item from each captured image would delay processing and reduce the frame rate. This solves the problem of delayed item detection, making it difficult to track items.

[0170] Furthermore, since the product placement location can be determined based on the {position / height / product image} identified by the image, users can reduce the effort required to maintain product placement locations, and inventory management and product inventory information can be performed more accurately.

[0171] Furthermore, the program in the present invention is a program that can be executed (readable) by a computer for each processing method, and the storage medium of the present invention stores a program that can be executed by a computer for each processing method.

[0172] The program in the present invention may be a program for each processing method of each device.

[0173] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0174] In this case, the program itself read from the recording medium realizes the novel functions of the present invention, and the recording medium storing the program constitutes the present invention. Examples of recording media that can be used to supply the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon disks.

[0175] Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing a program it has read, but also cases are included in which the OS or the like running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0176] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to the computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0177] It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or device. In this case, by reading a recording medium storing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0178] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0179] 101 Network Camera 102 Information processing equipment 103 Product shelf 104 items 105 Inventory Monitoring System 106 Network 107 Inventory Management Server 201 CPU 202 ROM 203 RAM 204 System Bus 205 Input Controller 206 Video Controller 207 Memory Controller 208 Communication I / F Controller 209 Input section 210 Display section 211 External Memory 301 CPU 302 ROM 303 RAM 304 System Bus 305 TOF sensor 306 Depth measurement unit 307 RGB camera unit 308 Image Processing Unit 309 Communication I / F Controller

Claims

1. an image capturing means for capturing an image including an area in front of an item take-out opening on a shelf within a photographing range; a determination means for determining, when a first item is detected in an image acquired by the image acquisition means, that the first item passed through a detection line associated with the shelf before the first time point, based on an image of the first item acquired by the image acquisition means that is taken before the first time point at which the first item is detected; a storage / retrieval simulation means for determining, when the determination means determines that the first item has passed the detection line before the first time point, that the first item has been removed from the shelf associated with the detection line; An information processing system comprising:

2. The information processing system described in Claim 1, characterized in that when the determination means detects the first item, it determines that the first item passed the detection line related to the shelf before the first point in time based on position information of the first item obtained based on an image related to the first item before the first point in time at which the first item was detected.

3. The information processing system described in claim 1 or 2 is characterized in that, when the determination means detects the first item, it registers an image of the first item linked to the time of photographing, and determines that the first item passed the detection line related to the shelf before the first time point based on the position information of the first item obtained based on the image of the first item registered in chronological order.

4. The information processing system described in any one of claims 1 to 3, characterized in that when the first item is detected within a predetermined range set in front of the removal port among the images acquired by the image acquisition means, the determination means determines that the first item passed the detection line related to the shelf before the first point in time based on an image of the first item taken before the first point in time at which the first item was detected.

5. the determination means determines that a detection line related to the shelf has been passed after a second time point at which a second item was detected in the predetermined range in the image acquired by the image acquisition means, When the determination means detects the second item, the storage / retrieval virtual means determines that the second item has been placed on the shelf related to the detection line.

5. The information system according to claim 4, wherein:

6. The determining means acquires height information of the article, A plurality of detection lines are set for shelves of different heights, The storage / retrieval simulation means selects a detection line set for each of the plurality of shelves having different heights based on height information at the time when the first item passed through the detection line, and considers that the first item has been removed from the shelf at the height corresponding to the detection line.

6. The information processing system according to claim 1, wherein:

7. The determining means acquires height information of the article, A plurality of detection lines are set for shelves of different heights, The storage / retrieval simulation means selects a detection line set for each of the plurality of shelves having different heights based on height information at the time when the second item passes the detection line within the predetermined range, and considers that the second item has been placed on the shelf at the height corresponding to the detection line.

6. The information processing system according to claim 5,

8. 8. The information processing system according to claim 6, wherein the detection lines have different lengths for the shelves at different heights.

9. 9. The information processing system according to claim 1, wherein the detection line is set in a direction along the access opening of the shelf.

10. The storage / retrieval simulation means, when an article detected by the determination means enters a predetermined range set in front of the take-out opening from the detection line and exits the detection line, does not increase or decrease the number of articles that have entered or left the shelf.

10. The information processing system according to claim 1, wherein:

11. An image acquisition means for acquiring an image including an area in front of an item removal opening on a shelf within a photographing range; a detection means for detecting, when a first item is detected in a predetermined range set in front of the take-out opening in the image acquired by the image acquisition means, that the first item has passed through a detection line related to the shelf before the first time point at which the first item is detected; a storage / retrieval simulation means for determining, when the detection means detects the first item, that the first item has been removed from the shelf associated with the detection line that has passed; The detection means acquires height information of the article, A plurality of detection lines are set for shelves of different heights, The storage / retrieval simulation means selects a detection line set for each of the plurality of shelves having different heights based on height information at the time when the first item passes the detection line within the predetermined range, and considers that the first item has been removed from the shelf at the height corresponding to the detection line. An information processing system characterized by:

12. An image acquisition means for acquiring an image including an area in front of an item removal opening on a shelf within a photographing range; a detection means for detecting that a second item has passed through a detection line associated with the shelf after a second point in time at which a second item has been detected in a predetermined range set in front of the take-out opening in the image acquired by the image acquisition means; and a storage / retrieval simulation means for determining, when the second item is detected by the detection means, that the second item has been placed on a shelf related to the detection line that has been passed; The detection means acquires height information of the article, A plurality of detection lines are set for shelves of different heights, The storage / retrieval simulation means selects a detection line set for each of the plurality of shelves having different heights based on height information at the time when the second item passes the detection line within the predetermined range, and considers that the second item has been placed on the shelf at the height corresponding to the detection line. An information processing system characterized by:

13. An information processing system as described in Claim 11 or 12, characterized in that the detection line has a different length for each shelf of a different height.

14. An image acquisition means for acquiring an image including an area in front of an item removal opening on a shelf within a photographing range; a detection means for detecting, when a first item is detected in a predetermined range set in front of the take-out opening in the image acquired by the image acquisition means, that the first item has passed through a detection line related to the shelf before the first time point at which the first item is detected; a storage / retrieval simulation means for determining, when the detection means detects the first item, that the first item has been removed from the shelf associated with the detection line that has passed; The storage / retrieval simulation means is configured to not increase or decrease the number of articles that have entered or left the shelf when an article detected by the detection means enters the predetermined range from the detection line and leaves the detection line. An information processing system characterized by:

15. an image acquisition step of acquiring an image including an area in front of an item takeout opening on a shelf in an imaging range; a determination step of determining, when a first item is detected in the image acquired in the image acquisition step, that the first item passed through a detection line related to the shelf before the first time point, based on an image related to the first item acquired in the image acquisition step that is taken before the first time point at which the first item is detected; a storage / retrieval simulation step of assuming that the first item has been removed from a shelf associated with the detection line when it is determined in the determination step that the first item has passed through the detection line before the first time point; 1. A method for controlling an information processing system, comprising:

16. A program for causing at least one computer to function as each of the means of the information processing system according to any one of claims 1 to 14.

Images