Stock management system, stock management method and program

Abstract

To accurately calculate stock quantities of products.SOLUTION: A stock management system comprises: a second camera that acquires a distance image including the distance from a predetermined position to the product in the last row by capturing a product row from the predetermined position; a recording unit with pre-registered first information indicating the correspondence relationship between an appearance of the product and identification information of the product; a database server with pre-recorded second information indicating the correspondence relationship between the identification information and detailed information about the product; and an information processing unit. The information processing unit acquires the identification information corresponding to the appearance of the product included in a visible light image from the first information, acquires the detailed information corresponding to the identification information from the second information, calculates the empty space distance from a rear end of a display shelf to the products in the last row on the basis of the distance image, and calculates stock quantities of the products on the display shelf on the basis of a depth size of the product included in the detailed information, the empty space distance, and a dimension of the display shelf in a depth direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an inventory management system, an inventory management method, and a program for managing inventory of products. [Background technology]

[0002] Patent Document 1 discloses a technology for detecting decreases and increases in the number of loaded products by comparing and analyzing current image data with previous image data. Specifically, Patent Document 1 discloses a system that has a control unit learn egg shape images in advance, and automatically determines that "four eggs have been sold" based on the fact that four eggs were loaded in the area where eggs were loaded in the previous image data, and that no eggs exist in the area where eggs were loaded in the current image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7040126 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 may not be able to accurately calculate the inventory quantity if the entire product is not captured in the image data. For example, if a product is located in a blind spot of the camera, the technology may not count the product as inventory, resulting in an inaccurate calculation of the inventory quantity.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technique that can accurately calculate the inventory quantity of a product. [Means for solving the problem]

[0006] An inventory management system according to a first aspect of the present invention comprises a first camera that photographs a row of products, each of which is lined up in the depth direction of a display shelf, to obtain a visible light image including the appearance of the product; a second camera that photographs the row of products from a predetermined position to obtain a distance image including the distance from the predetermined position to the product in the last row; a recording unit in which first information indicating a correspondence between the appearance of the products and identification information of the products is pre-registered; a database server in which second information indicating a correspondence between the identification information and detailed information about the products is pre-recorded; and an information processing unit, wherein the information processing unit obtains the identification information corresponding to the appearance of the products included in the visible light image from the first information, obtains the detailed information corresponding to the identification information from the second information in the database server, calculates the free space distance from the back end of the display shelf to the product in the last row based on the distance image, and calculates the inventory number of the products on the display shelf based on the depth size of the products, the free space distance, and the dimensions of the display shelf in the depth direction, which are included in the detailed information.

[0007] With this configuration, the inventory count of a product is calculated based on the product depth, the distance between the back end of the display shelf and the last product, and the dimension of the display shelf in the depth direction. Therefore, even if not all of the products are captured in the visible light image and the distance image, the inventory count can be accurately calculated.

[0008] In the inventory management system of the second aspect of the present invention, in the first aspect, the specified position may be behind the row of products within the display shelf, or above the front of the display shelf, or the ceiling above the front of the display shelf.

[0009] According to this configuration, the second camera is installed behind the row of products on the display shelf, or above the front of the display shelf, or on the ceiling above the front of the display shelf, so that the second camera can reliably acquire a distance image including the accurate distance to the last row of products.

[0010] In the inventory management system according to a third aspect of the present invention, in the first or second aspect, the first camera and the second camera may be housed in a common housing and formed as a unit.

[0011] With this configuration, the first camera and the second camera are unitized, so that the first camera and the second camera are independent of each other, and the installation work for the first camera and the second camera is easier than when both cameras are installed separately.

[0012] In the inventory management system according to a fourth aspect of the present invention, in any one of the first to third aspects, the second camera may be a ranging camera that measures the distance to the last row of products and acquires the distance image based on that distance.

[0013] With this configuration, the second camera measures the distance to the last row of products, thereby reducing the processing load on the information processing unit compared to when the information processing unit performs a predetermined image analysis process on the image acquired by the second camera to measure the above distance.

[0014] An inventory management system according to a fifth aspect of the present invention, in any one of the first to fourth aspects, may further include a determination unit that determines whether an obstacle exists between the first camera and the row of products, and if the determination unit determines that the obstacle exists, the first camera may re-acquire the visible light image.

[0015] If an obstacle exists between the first camera and the product row, there is a high possibility that the obstacle will be captured in the visible light image acquired by the first camera. If the inventory quantity is calculated based on a visible light image that captures the obstacle, there is a possibility that the inventory quantity will not be calculated accurately. In contrast, with the above configuration, if the determination unit determines that an obstacle exists between the first camera and the product row, the first camera re-acquires a visible light image. This makes it possible to acquire a visible light image that does not capture the obstacle. This allows for a more accurate calculation of the inventory quantity of products.

[0016] A sixth aspect of the present invention provides an inventory management method for an inventory management system including a first camera, a second camera, a recording unit, a database server, and an information processing unit, wherein the information processing unit acquires a visible light image including the appearance of one or more products by using the first camera to capture an image of a row of products in which at least one product is lined up in the depth direction of a display shelf, and acquires a distance image including the distance from a predetermined position to a product in a last row by using the second camera to capture an image of the row of products from the predetermined position, and acquires a first information registered in the recording unit that indicates a correspondence between the appearance of the products and identification information of the products. acquires first information indicating the appearance of the product, acquires second information recorded in the database server and indicating a correspondence between the identification information and detailed information about the product, acquires the identification information corresponding to the appearance of the product included in the visible light image from the first information, acquires the detailed information corresponding to the identification information from the second information, calculates the free space distance from the back end of the display shelf to the product in the last row based on the distance image, and calculates the inventory number of the product on the display shelf based on the depth size of the product included in the detailed information, the free space distance, and the dimension of the display shelf in the depth direction.

[0017] According to this inventory management method, it is possible to obtain the same effects as the above-mentioned inventory management system.

[0018] A seventh aspect of the present invention provides a program for causing an information processing unit to execute processing in an inventory management system including a first camera, a second camera, a recording unit, a database server, and an information processing unit, the program comprising: the first camera photographing a row of products in which at least one product is lined up in the depth direction of a display shelf, thereby acquiring a visible light image including the appearance of the product; the second camera photographing the row of products from a predetermined position, thereby acquiring a distance image including the distance from the predetermined position to the product in the last row; and recording first information registered in the recording unit, the first information including the appearance of the product and identification information of the product. acquires second information recorded in the database server, the second information indicating the correspondence between the identification information and detailed information about the product; acquires the identification information corresponding to the appearance of the product included in the visible light image from the first information; acquires the detailed information corresponding to the identification information from the second information; calculates the free space distance from the back end of the display shelf to the product in the last row based on the distance image; and calculates the inventory quantity of the product on the display shelf based on the depth size of the product included in the detailed information, the free space distance, and the dimension of the display shelf in the depth direction.

[0019] This program can provide the same effects as the above-mentioned inventory management system. [Effects of the Invention]

[0020] According to the present invention, the inventory quantity of a product can be calculated accurately. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a simplified diagram showing the overall configuration of an inventory management system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing the appearance of a display shelf installed in a store. [Figure 3] FIG. [Figure 4]FIG. 1 is a diagram showing a simplified configuration of an information processing device. [Figure 5] FIG. 2 is a simplified diagram showing functions of an information processing unit. [Figure 6] FIG. 2 is a simplified diagram illustrating the configuration of a database server. [Figure 7] FIG. 3 is a diagram showing the data structure of first information. [Figure 8] FIG. 10 is a diagram showing the data structure of second information. [Figure 9] 4 is a flowchart showing the flow of processing executed by the inventory management system according to the first embodiment. [Figure 10] 10 is a flowchart illustrating the process in step S04. [Figure 11] FIG. 10 is a diagram showing a schematic arrangement of a color camera and a depth camera according to a modified example. [Figure 12] FIG. 10 is a diagram showing the interior of a store according to a second embodiment. [Figure 13] FIG. 10 is a diagram showing, in a simplified form, the functions of an information processing unit according to the second embodiment. [Figure 14] FIG. 11 is a diagram showing the interior of a store according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] The inventory management system according to the present invention will be described below. Each of the embodiments described below represents a specific example of the present invention. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept will be described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined. Note that elements with the same reference numerals in different drawings represent the same or corresponding elements.

[0023] (Embodiment 1) 1 is a simplified diagram showing the overall configuration of an inventory management system 1 according to a first embodiment of the present invention. The inventory management system 1 includes a color camera 11 (an example of a first camera), a depth camera 12 (an example of a second camera), an information processing device 2, and a database server 3. The color camera 11, the depth camera 12, the information processing device 2, and the database server 3 are all connected to a communication network NW. The communication network NW is any dedicated line network or public line network such as an IP (Internet Protocol) network.

[0024] The inventory management system 1 is a system that calculates the inventory quantity of products 100 sold in a store 10. A store 10 refers to a building that has one or more display shelves 14 on its premises. Examples of stores 10 include supermarkets, convenience stores, electronics retailers, and department stores.

[0025] Fig. 2 is a diagram showing the appearance of a display shelf 14 installed in the store 10. As shown in Fig. 2, the display shelf 14 according to the first embodiment is a fixture having a base 15 extending in the vertical direction (the direction indicated by H in Fig. 2) and a plurality of shelves 16 extending from the base 15 in a depth direction (the direction indicated by D in Fig. 2) perpendicular to the vertical direction. The display shelf 14 is placed facing an aisle 105 of the store 10. Hereinafter, in the depth direction, the side closer to the aisle 105 of the store 10 is referred to as the front direction, and the side farther from the aisle 105 is referred to as the rear direction.

[0026] FIG. 3 is an enlarged view of the display shelf 14. As shown in FIG. 3, one or more products 100 are lined up in a row along the depth direction on the shelf boards 16 of the display shelf 14 to form a product row 101. The product row 101 is arranged so as to be close to the aisle 105 within the display shelf 14 by a so-called forward-pushing operation performed by an employee of the store 10. The product row 101 is formed by one or more products 100 of the same type. When the product row 101 is formed by a plurality of products 100, the plurality of products 100 are arranged closely packed with no gaps between them. The products 100 according to the first embodiment are, for example, confectioneries or seasonings.

[0027] 3, in the first embodiment, a color camera 11 and a depth camera 12 are placed behind a row of products 101 in a display shelf 14. These cameras 11 and 12 are placed independently of each other.

[0028] The color camera 11 captures an image of a product row 101 in which at least one product 100 is lined up in the depth direction of the display shelf 14, thereby acquiring an RGB image D3 (an example of a visible light image) including the appearance of the product 100. The color camera 11 refers to a camera that has the function of irradiating visible light and absorbing the reflected light in an element to generate the RGB image D3.

[0029] The depth camera 12 photographs the product row 101 from a predetermined position P1, thereby acquiring a distance image D4 including the distance from the predetermined position P1 to the product 100 in the last row. In the first embodiment, the predetermined position P1 refers to the rear of the product row 101 within the display shelf 14. The depth camera 12 is configured with a distance measurement camera that measures the distance to the product 100 in the last row and acquires the distance image D4 based on the measured distance. Specifically, the depth camera 12 according to the first embodiment is configured with a ToF (Time of Flight) camera that emits invisible light such as near-infrared light, absorbs the reflected light in an element, and generates data called a depth value that quantifies the distance between the element and the subject. However, the depth camera 12 may also be configured with a stereo camera that uses multiple lenses to photograph the same subject from different angles to acquire multiple images, and calculates the distance to the subject by analyzing the parallax between the multiple images.

[0030] The color camera 11 and the depth camera 12 transmit the acquired RGB image D3 and distance image D4 to the information processing device 2 via the communication network NW.

[0031] FIG. 4 is a simplified diagram showing the configuration of the information processing device 2. The information processing device 2 is, for example, a cloud server or an on-premise server constructed by one or more computers, and includes a communication unit 21, a processing unit 22 (an example of an information processing unit), and a recording unit 23. The communication unit 21 is configured with a communication module compatible with any communication method such as IP, and transmits and receives information such as data or commands to and from other devices via the communication network NW. The processing unit 22 is configured with a processor such as a CPU. The recording unit 23 is configured with an HDD, SSD, semiconductor memory, or the like. A program 24 is recorded in the recording unit 23.

[0032] 5 is a simplified diagram showing the functions of the processing unit 22. When the CPU executes the program 24 read from the recording unit 23, the processing unit 22 functions as a data acquisition unit 221 and a data processing unit 222. In other words, the program 24 is a program for causing a processor such as a CPU mounted on the information processing device 2 to function as the data acquisition unit 221 and the data processing unit 222. The processing of the data acquisition unit 221 and the data processing unit 222 will be described later. The program 24 may be stored on a recording medium and made available for distribution as a recording medium or made downloadable from the recording medium.

[0033] FIG. 6 is a simplified diagram showing the configuration of the database server 3. As shown in FIG. 6, the database server 3 includes a communication unit 31, a processing unit 32, a first recording unit 33 (an example of a recording unit), and a second recording unit 34. The communication unit 31 includes a communication module compatible with communication standards such as IP, and transmits and receives information such as data or commands to and from other devices via the communication network NW. The processing unit 32 includes a processor such as a CPU. The first recording unit 33 includes a recording medium such as an HDD, SSD, or semiconductor memory. The second recording unit 34 includes a recording medium such as an HDD, SSD, or semiconductor memory. The first recording unit 33 has first information D1 registered in advance, which indicates a correspondence between the appearance of a product and the identification information of the product. The second recording unit 34 has second information D2 recorded in advance, which indicates a correspondence between the identification information and detailed product information.

[0034] FIG. 7 is a diagram showing the data configuration of the first information D1. As shown in FIG. 7, the first information D1 has a column C1 of product appearances and a column C2 of identification information. The product appearance column C1 stores front images showing the front appearances of various products. Note that the product appearance column C1 may further store rear images showing the rear appearances of various products, top images showing the top appearances of various products, bottom images showing the bottom appearances of various products, and side images showing the side appearances of various products. The product appearance column C1 may also store information showing feature quantities of the appearances of various products. The identification information column C2 stores identification information of various products. Identification information refers to information assigned to each product to distinguish it from other products. In the first embodiment, the identification information is, for example, a JAN code. However, the identification information is not limited to a JAN code, and various barcodes, QR codes (registered trademark), etc. may also be used as identification information.

[0035] FIG. 8 is a diagram showing the data configuration of second information D2 indicating the correspondence between identification information and detailed information. As shown in FIG. 8, the second information D2 has a column C3 of identification information, a column C4 of depth size, a column C5 of width size, a column C6 of product name, a column C7 of price, and a column C8 of weight. The information stored in columns C4 to C8 corresponds to detailed information. The column C4 of depth size stores the size of each product in the depth direction (depth dimension). The column C5 of width size stores the size of each product in the width direction W (width dimension). The column C6 of product name, column C7 of price, and column C8 of weight store the product name, price, and weight of each product. Note that the detailed information is not limited to that shown in FIG. 8, and more information may be stored.

[0036] FIG. 9 is a flowchart showing the flow of processing executed by the processing unit 22 according to the first embodiment.

[0037] First, color camera 11 photographs product row 101 and acquires RGB image D3. Depth camera 12 photographs product row 101 from a predetermined position P1 and acquires distance image D4. Color camera 11 and depth camera 12 transmit the acquired RGB image D3 and distance image D4 to information processing device 2 via communications network NW. Communication unit 21 of information processing device 2 receives RGB image D3 and distance image D4, and temporarily records the received RGB image D3 and distance image D4 in recording unit 23.

[0038] In step S01, the data acquisition unit 221 acquires the RGB image D3 recorded in the recording unit .

[0039] In step S02, the data acquisition unit 221 acquires the distance image D4 recorded in the recording unit .

[0040] In step S03, the data acquisition unit 221 acquires the first information D1 pre-registered in the first recording unit 33 and the second information D2 pre-recorded in the second recording unit 34. Then, the data acquisition unit 221 inputs the first information D1, the second information D2, the RGB image D3, and the distance image D4 to the data processing unit 222.

[0041] In step S04, the data processing unit 222 acquires, from the first information D1, identification information corresponding to the appearance of the product 100 included in the RGB image D3. The processing of step S04 will be described in detail with reference to FIG.

[0042] 10 is a flowchart for explaining the processing in step S04. First, in step S41, the data processing unit 222 extracts the feature amounts of the appearance of the product 100 included in the RGB image D3. Also in step S41, the data processing unit 222 extracts the feature amounts of each image stored in the product appearance column C1 (FIG. 7) in the first information D1.

[0043] Next, in step S42, the data processing unit 222 identifies the product in the appearance column C1 that has the appearance most similar to the appearance of the product 100 included in the RGB image D3 based on the features extracted in step S41.

[0044] Then, in step S43, the data processing unit 222 acquires the identification information (JAN code in this example) corresponding to the product 100 identified in step S43 from the identification information column C2. In this way, the data processing unit 222 acquires the identification information of the product 100 from the first information D1.

[0045] Returning to FIG. 9, in step S05, the data processing unit 222 acquires detailed information corresponding to the identification information from the second information D2. For example, assume that the processing in step S04 has revealed that the identification information of the product 100 reflected in the RGB image D3 is "XXX." In this case, the data processing unit 222 references the column shown in R1 in FIG. 8 and recognizes that the depth size of the product 100 is 2 cm, the width size is 5 cm, the product name is A, the price is 100 yen, and the weight is 50 g. Note that the data processing unit 222 does not necessarily need to acquire all of the information included in the detailed information, and may acquire only information related to the depth size of the product.

[0046] In step S06, the data processing unit 222 calculates the empty space distance L3 from the innermost end P3 of the display shelf 14 to the product 100 in the last row, based on the distance image D4.

[0047] A method for calculating the empty space distance L3 will be described below with reference to FIG. 3. The data processing unit 222 first acquires a first distance L1 (an example of a distance) in the depth direction from the predetermined position P1 to the last product 100 in the last row. This first distance L1 is measured by the depth camera 12. Next, the data processing unit 222 acquires a second distance L2 in the depth direction from the back end P3 of the display shelf 14 to the predetermined position P1. The second distance L2 is pre-recorded, for example, in a predetermined recording area of ​​the recording unit 23. The data processing unit 222 then sums the first distance L1 and the second distance L2. This calculates the empty space distance L3, which is the distance from the back end P3 of the display shelf 14 to the last product 100 in the last row. Note that in the first embodiment, the back end P3 of the display shelf 14 refers to the inner surface of the base 15 of the display shelf 14. In the first embodiment, the empty space distance L3, more specifically, refers to the distance in the depth direction from the rear end P3 to the back of the product 100 in the last row.

[0048] Before executing the process of step S06, the data processing unit 222 may perform preprocessing. In the preprocessing, the data processing unit 222 performs calibration processing to adjust the parallax between the color camera 11 and the depth camera 12. For example, the data processing unit 222 uses a known image processing technique such as an affine transformation matrix to calculate parameters for aligning the coordinate system of the RGB image D3 with the coordinate system of the distance image D4, and aligns the RGB image D3 with the distance image D4 based on the parameters. By performing this preprocessing, the data processing unit 222 can accurately determine what type of product 100 is located and how far it is away from the predetermined position P1.

[0049] Returning to FIG. 9, in step S07, the data processing unit 222 calculates the inventory quantity of the product 100 on the display shelf 14 based on the depth size L6 of the product 100, the free space distance L3, and the dimension L5 of the display shelf 14 in the depth direction.

[0050] 3 again, the process for calculating the stock quantity of product 100 will be described below. The stock quantity of product 100 is calculated by dividing the dimension L4 of product row 101 in the depth direction (hereinafter referred to as product row depth dimension L4) by the depth size L6 of product 100 (depth size L6 per product 100). In other words, if the stock quantity of product 100 is X, then stock quantity X is expressed by the following formula 1.

[0051] X=L4 / L6...(Formula 1) Here, the product row depth dimension L4 is found by calculating the difference between the dimension L5 of the display shelf 14 in the depth direction (hereinafter referred to as display shelf depth dimension L5) and the empty space distance L3. In other words, the product row depth dimension L4 is expressed by the following formula 2.

[0052] L4=L5-L3...(Formula 2) The display shelf depth dimension L5 is recorded in advance in a predetermined recording area of ​​the recording unit 23, for example.

[0053] By rearranging the above formulas 1 and 2, the inventory quantity X is expressed by the following formula 3.

[0054] X = (L5 - L3) / L6 (Equation 3) The data processing unit 222 calculates the inventory quantity by calculating Equation 3 above. For example, assume that the display shelf depth dimension L5 is 40 cm, the free space distance L3 is 30 cm, and the product 100 depth dimension L6 is 2 cm. In this case, the solution to (L5 - L3) is 10 cm. That is, the product row depth dimension L4 is calculated to be 10 cm. Then, by dividing the product row depth dimension L4 (10 cm) by the product 100 depth dimension L6 (2 cm), X = 5 is obtained. That is, the inventory quantity on the display shelf 14 is calculated to be 5. In this way, the data processing unit 222 calculates the inventory quantity of the product 100.

[0055] The inventory quantity calculated by the data processing unit 222 may be transmitted to a store terminal (not shown) installed in the store 10 via the communication network NW. The store terminal is, for example, an information terminal such as a PC, tablet, or smartphone. The store terminal may then output the inventory quantity through an output device (not shown) such as a display or speaker connected to the store terminal. This allows the employee of the store 10 to know the inventory quantity calculated by the data processing unit 222.

[0056] As described above, the inventory management system 1 according to this embodiment calculates the inventory quantity based on the depth dimension L6 of the product 100, the empty space distance L3, and the display shelf depth dimension L5. Specifically, the inventory quantity is calculated by dividing the product row depth dimension L4, which is calculated from the difference between the display shelf depth dimension L5 and the empty space distance L3, by the depth dimension L6 of the product 100. In this way, the inventory quantity of the product 100 can be accurately calculated even if the entire product 100 is not captured in the RGB image D3 and the distance image D4.

[0057] Furthermore, according to the inventory management system 1 of this embodiment, the depth camera 12 is installed behind the product row 101 within the display shelf 14. Therefore, even if the display shelf 14 has multiple shelves 16 arranged vertically, making it difficult to photograph the product row 101 from above, the inventory number of the products 100 can be accurately calculated.

[0058] The following describes various modifications to the above-described embodiment 1. These modifications can be applied in any combination.

[0059] (Variation 1-1) The inventory management system 1 may be equipped with various cameras instead of the color camera 11. For example, the inventory management system 1 may be equipped with a binary camera (an example of a first camera) capable of acquiring a binary image (an example of a visible light image). Alternatively, the inventory management system 1 may be equipped with a camera capable of acquiring an HSV image (an example of a visible light image).

[0060] (Variant 1-2) In embodiment 1, an example was described in which the depth camera 12 and the color camera 11 are arranged independently of each other, but the depth camera 12 and the color camera 11 may also be housed in a common housing and formed into a unit.

[0061] (Variation 1-3) In the first embodiment, an example in which one color camera 11 and one depth camera 12 are arranged behind the product row 101 has been described. However, multiple color cameras 11 and multiple depth cameras 12 may be arranged behind the product row 101. FIG. 11 is a diagram schematically illustrating the arrangement of the color cameras 11 and depth cameras 12 according to this variation, showing a shelf 16, multiple color cameras 11, and multiple depth cameras 12 as viewed from above. Note that the product row 101 is not shown in FIG. 11. In the example shown in FIG. 11, three color cameras 11 and three depth cameras 12 are arranged at intervals in the width direction W, which is perpendicular to both the up-down direction and the depth direction. With this configuration, substantially the entire width of the shelf 16 can be captured within the angle of view of the multiple color cameras 11 and depth cameras 12. In this way, even if multiple rows of products are lined up in the width direction W, the stock quantity of the products 100 that make up each row of products 101 can be calculated without omission.

[0062] (Variation 1-4) The color camera 11 is not always able to capture an RGB image D3 that includes the appearance of the product 100. For example, the color camera 11 may capture an RGB image D3 that is so-called blacked out or out of focus. Hereinafter, such an image will be referred to as an error image. If an error image is captured, the data processing unit 222 may acquire, in the process of step S43 in FIG. 9, identification information for a product 100 that is different from the product 100 actually displayed on the display shelf 14. This may result in an inaccurate inventory count being calculated.

[0063] To avoid such a situation, third information may be recorded in the recording unit 23, which is a pre-registered correspondence between the display shelf 14 on which the color camera 11 is installed and the appearance of the product 100 displayed on the display shelf 14. In other words, the third information may be recorded in the recording unit 23, which indicates the appearance of the product 100 displayed on the display shelf 14 on which the color camera 11 is located.

[0064] The data processing unit 222 according to this modification performs a predetermined image analysis process to determine whether an error image has been input when the RGB image D3 is input from the data acquisition unit 221. If it is determined that an error image has been input, the data processing unit 222 identifies the appearance of the product 100 arranged on the display shelf 14 by referring to the third information.

[0065] In this way, even if an error image is acquired, the data processing unit 222 can acquire accurate identification information of the product 100. As a result, it becomes possible to more reliably calculate the inventory quantity of the product 100.

[0066] (Variation 1-5) In the first embodiment, the data processing unit 222 may calculate the number of products in stock by referring to the dimension of the top surface of the shelf board 16 in the depth direction instead of the display shelf depth dimension L5.

[0067] (Embodiment 2) An inventory management system 1A according to embodiment 2 will be described. In inventory management system 1A, a color camera and a depth camera are installed on the ceiling above the front of the display shelves. The following description will focus on the differences from embodiment 1 above.

[0068] Fig. 12 is a diagram showing the interior of a store 10A according to embodiment 2. As shown in Fig. 12, store 10A has a display shelf 14A, and a color camera 11A and a depth camera 12A installed on ceiling 110 above the front of display shelf 14A. In embodiment 2, color camera 11A and depth camera 12A are unitized.

[0069] The display shelf 14A according to the second embodiment is a platform having a shelf 16A, a support 17, and a base 18. The shelf 16A is positioned above the base 18 and extends in an oblique direction that intersects both the vertical and depth directions. One or more products 100A are lined up in a row along the depth direction on the shelf 16A, forming a product row 101. The products 100A may be, for example, vegetables or fruits. The support 17 is interposed between the base 18 and the shelf 16A and supports the shelf 16A from below. The base 18 is placed on a floor 120 of the store 10A.

[0070] The color camera 11A and the depth camera 12A according to the second embodiment are installed at positions that allow the bottom 300 and the top 400 of the shelf 16A to be within the angle of view A of both cameras 11A and 12A. The installation positions of the cameras 11A and 12A are not particularly limited as long as the bottom 300 and the top 400 are within the angle of view A of both cameras 11A and 12A. That is, the installation heights of the cameras 11A and 12A and the distances of the cameras 11A and 12A in the depth direction from the display shelf 14A may be adjusted as appropriate based on the average height of customers using the store 10A, the height of the ceiling 110, the interior design of the store 10A, the types of products 100A displayed on the display shelf 14A, the vertical and depth dimensions of the display shelf 14A, and the like. For example, the positions of the cameras 11A and 12A may be adjusted along the camera arrangement axis AX shown in FIG. 12 .

[0071] 13 is a simplified diagram showing the functions of a processing unit 22A according to embodiment 2. The processing unit 22A includes a data acquisition unit 230, a determination unit 240, and a data processing unit 250.

[0072] The data acquisition unit 230 inputs the acquired RGB image D3 to the determination unit 240. Except for this point, the processing of the data acquisition unit 230 is the same as that of the data acquisition unit 221 according to the first embodiment, and therefore a description thereof will be omitted.

[0073] The determination unit 240 determines whether or not an obstacle exists between the color camera 11A and the row of products 101. An obstacle is, for example, a person or an object (such as a shopping basket, shopping cart, or dolly) that exists between the color camera 11A and the row of products 101. The determination unit 240, for example, performs a known image recognition process on the RGB image D3 received from the data acquisition unit 230, and determines whether or not an obstacle appears in the RGB image D3. If an obstacle appears in the RGB image D3, the determination unit 240 determines that an obstacle exists between the color camera 11A and the row of products 101.

[0074] If it is determined that an obstacle exists between color camera 11A and product row 101, determination unit 240 generates a reacquisition signal to cause color camera 11A to reacquire RGB image D3, and transmits this signal to color camera 11A. Upon receiving the reacquisition signal, color camera 11A reacquires RGB image D3. RGB image D3 reacquired by color camera 11A is sent to information processing device 2, acquired by data acquisition unit 230, and input to determination unit 240, where it is processed again.

[0075] If it is determined that no obstacle exists between the color camera 11A and the product row 101, the determination unit 240 inputs the RGB image D3 to the data processing unit 250.

[0076] The above-described processing by the determination unit 240 may be performed after the processing of step S03 and before the processing of step S04 in the flow shown in FIG.

[0077] If the obstacle is also captured in the RGB image D3 recaptured by the color camera 11A, the determination unit 240 may generate a recapture signal again and transmit this signal to the color camera 11A again. If this process is repeated a predetermined number of times and the obstacle is still captured in the RGB image D3, the determination unit 240 may input the RGB image D3 in which the obstacle is captured to the data processing unit 250. In this case, the data processing unit 250 performs the process described in step S04 of the first embodiment based on the appearance of the product 100A that can be captured from an area other than the area in which the obstacle is captured.

[0078] Additionally, when the determination unit 240 determines that an obstacle exists between the color camera 11A and the product row 101, it may send a signal to the store terminal to display a message such as "Please move the obstacle" on the store terminal. Then, when information indicating that the obstacle has been moved is entered into the store terminal by an employee or the like and the determination unit 240 receives this information, the determination unit 240 may send a reacquisition signal to the color camera 11A.

[0079] The data processing unit 250 according to the second embodiment calculates the inventory quantity of the product 100A by performing substantially the same processing as the data processing unit 250 according to the first embodiment. However, the method of calculating the free space distance L3 differs from that of the data processing unit 250 according to the first embodiment. This will be described below with reference to FIG. 12.

[0080] The data processing unit 250 according to the second embodiment acquires a third distance L7 from the predetermined position P1 to the rear end P3 of the display shelf 14A and a first distance L1 (an example of a distance) from the predetermined position P1 to the last product 100A in the row, both of which are included in the distance image D4 captured by the depth camera 12A. The data processing unit 250 then performs, for example, triangulation based on the first distance L1 and the third distance L7 to calculate the empty space distance L3 from the rear end P3 of the display shelf 14A to the last product 100A in the row. Note that in the second embodiment, the rear end P3 of the display shelf 14A coincides with the top 400 of the display shelf 14A.

[0081] The data processing unit 250 calculates the inventory quantity of the product 100A based on the thus calculated free space distance L3, the depth dimension L5 of the display shelf 14A, and the depth size L6 of the product 100A. Note that the data processing unit 250 may calculate the inventory quantity of the product 100A based on the depth dimension of the shelf 16A instead of the depth dimension L5 of the display shelf 14A. The depth dimension of the shelf 16A refers to the dimension in the depth direction from the bottommost part 300 to the topmost part 400 of the shelf 16A. This dimension may be recorded in advance in the recording unit 23, for example.

[0082] As described above, according to the inventory management system 1A of the second embodiment, the color camera 11A and the depth camera 12A are installed on the ceiling 110 above the front of the display shelf 14A. Therefore, even if the display shelf 14A is made up of a platform and the product 100A is made up of vegetables and fruits, the inventory quantity of the product 100A can be accurately calculated.

[0083] Furthermore, in the inventory management system 1A according to the second embodiment, if the determination unit 240 determines that an obstacle exists between the color camera 11A and the product row 101, the color camera 11A reacquires the RGB image D3. In this way, an RGB image D3 that does not include the obstacle can be acquired. This allows the inventory quantity of the product 100A to be calculated more accurately.

[0084] The following describes various modifications of the above-described embodiment 2. These modifications can be applied in any combination.

[0085] (Variation 2-1) The average depth of each product may be recorded in the depth size column C4 shown in Fig. 8. In this way, even if the product 100A, such as fruits or vegetables, whose depth size is not uniquely determined, is placed on the display shelf 14A, it is possible to calculate the inventory quantity of the product 100A.

[0086] (Variation 2-2) The inventory management system 1A may further include a detection device that detects whether or not an obstacle exists between the color camera 11A and the product row 101. In this case, the determination unit 240 may determine whether or not an obstacle exists between the color camera 11A and the product row 101 based on information received from the detection device. Specifically, this is as follows.

[0087] The detection device according to this modification is configured, for example, with an optical sensor having a light-receiving unit and a light-emitting unit, and detecting the presence of an obstacle when light projected from the light-emitting unit is blocked or reflected. The detection device is installed, for example, on the floor 120 or ceiling 110 between the color camera 11A and the product row 101. When the detection device detects an obstacle, it transmits obstacle detection information to the information processing device 2 via the communication network NW. The obstacle detection information includes obstacle detection time information that indicates the time the obstacle was detected.

[0088] When acquiring an RGB image D3 through the communication network NW, the color camera 11A according to this modification additionally acquires photographing time information indicating the time at which the RGB image D3 was acquired. Then, when transmitting the RGB image D3 to the information processing device 2, the color camera 11A also additionally transmits the photographing time information.

[0089] Based on the obstacle detection time information and the photography time information, the determination unit 240 according to this modification determines whether or not an obstacle exists between the color camera 11A and the commodity row 101. For example, if the RGB image D3 is acquired within a predetermined time from the time the detection device detected the obstacle, the determination unit 240 determines that an obstacle exists between the color camera 11A and the commodity row 101.

[0090] According to the above configuration, the same effects as those of the inventory management system 1A according to the second embodiment can be obtained.

[0091] (Modification 2-3) In the second embodiment, the shelf boards 16A of the display shelf 14A extend in an inclined direction, but the shelf boards 16A may extend in the depth direction. In other words, the shelf boards 16A do not necessarily have to be inclined.

[0092] (Embodiment 3) An inventory management system 1B according to embodiment 3 will be described. In inventory management system 1B, color camera 11B and depth camera 12B are installed in positions above and in front of display shelf 14B. The following description will focus on differences from embodiment 1 above.

[0093] Fig. 14 is a diagram showing the interior of a store 10B according to embodiment 3. As shown in Fig. 14, store 10B has a display shelf 14B, a wall 130 disposed above and in front of display shelf 14B, and a color camera 11B and a depth camera 12B installed on wall 130. In embodiment 3, color camera 11B and depth camera 12B are unitized.

[0094] The display shelf 14B according to the third embodiment is a refrigerated display shelf for displaying one or more products 100B, and includes a plurality of shelves 16B, a bottom 19, and a refrigerant system (not shown). The products 100B are, for example, meat or fresh fish. In this example, the plurality of shelves 16B are three shelves spaced apart in the vertical direction and arranged vertically above the bottom 19. The bottom 19 is placed on the floor 120 of the store 10B. The products 100B are arranged in a row in the depth direction on the upper surface of the bottom 19 and on the upper surfaces of each of the plurality of shelves 16B, forming a product row 101. The refrigerant system (not shown) is built into the display shelf 14B and generates cool air, which is circulated by a blower fan (not shown) to maintain the freshness of the products 100B.

[0095] The color camera 11B and depth camera 12B according to the third embodiment are installed on a wall 130 above the front of the display shelf 14B. The cameras 11B and 12B are installed at positions that allow the bottom 300 and top 400 of the display shelf 14B to be included within the angle of view A.

[0096] The installation positions of both cameras 11B and 12B are not limited to the wall 130 above the front of display shelf 14B. For example, if a pillar is installed above the front of display shelf 14B, both cameras 11B and 12B may be installed on this pillar. Furthermore, if there is another display shelf in front of display shelf 14B, both cameras 11B and 12B may be installed on that other display shelf. Furthermore, as with color camera 11A and depth camera 12A according to embodiment 2, the installation positions of color camera 11B and depth camera 12B can be adjusted as appropriate.

[0097] The data processing unit according to the third embodiment, like the data processing unit 250 according to the second embodiment, calculates the inventory quantity of the product 100B based on the free space distance L3, the depth dimension L5 of the display shelf 14B, and the depth size L6 of the product 100B. Note that the data processing unit 250 may calculate the inventory quantity of the product 100B based on the depth dimension of the top surface of the bottom 19 instead of the depth dimension L5 of the display shelf 14B. The data processing unit 250 may calculate the inventory quantity of the product 100B based on the depth dimension of the top surface of the shelf 16B instead of the depth dimension L5 of the display shelf 14B. The depth dimension of the top surface of the bottom 19 and the depth dimension of the top surface of the shelf 16B may be recorded in a predetermined recording area of ​​the recording unit 23.

[0098] As described above, according to the inventory management system 1B of the third embodiment, the color camera 11B and the depth camera 12B are installed above the front of the display shelf 14B. Therefore, even if the display shelf 14B is a refrigerated display shelf and the product 100B is made up of dressed meat, fresh fish, or the like, the inventory quantity of the product 100B can be accurately calculated.

[0099] The following describes modifications of the third embodiment. These modifications can be applied in any combination.

[0100] (Variation 3-1) The color camera 11 and depth camera 12 described in the first embodiment may be installed behind the product row 101 on the display shelf 14B. In this way, by arranging the color cameras 11, 11B and the depth cameras 12, 12B in various locations and acquiring RGB images D3 and distance images D4 taken from various angles, it becomes possible to calculate the number of products in stock more accurately.

[0101] (Modification 3-2) Color camera 11B and depth camera 12B may be installed, for example, on floor 120. Alternatively, they may be attached to a shopping basket or shopping cart.

[0102] (Variation 3-3) Color camera 11B and depth camera 12B may be built into various information terminals, such as smartphones and tablets, operated by employees of store 10B. That is, RGB image D3 and distance image D4 described above may be acquired by an employee photographing product row 101.

[0103] (Variation 3-4) If the product row 101 is not displayed forward, for example, if there is a gap in the depth direction between the product in the front row of the product row 101 and the end of the upper surface of the bottom 19 on the aisle side, the dimension of the gap in the depth direction may be added to the free space distance L3. [Explanation of symbols]

[0104] 1: Inventory management system 2: Information processing equipment 3: Database server 10: Store 11: Color camera (example of the first camera) 12: Depth camera (example of a second camera) 14:Display shelf 16: Shelf 22: Information Processing Department 33: First recording unit (an example of a recording unit) 100:Product 101: Product column 110: Ceiling D1: First information D2 :Second information D3: RGB image (example of visible light image) D4: Range image L1: First distance (an example of distance) L3: Free space distance L5: Display shelf depth dimension (example of display shelf dimension in the depth direction) L6: Depth size P1 :Predetermined position P3: Back end

Claims

1. a first camera that captures a row of at least one product lined up in the depth direction of a display shelf, thereby acquiring a visible light image including an appearance of the product; a second camera that captures an image of the row of products from a predetermined position to obtain a distance image including a distance from the predetermined position to the product in the last row; a recording unit in which first information indicating a correspondence relationship between the appearance of the product and the identification information of the product is registered in advance; a database server in which second information indicating a correspondence between the identification information and detailed information of the product is pre-recorded; an information processing unit, The information processing unit obtaining the identification information corresponding to the appearance of the product included in the visible light image from the first information; obtaining the detailed information corresponding to the identification information from the second information; calculating an empty space distance from the innermost end of the display shelf to the product in the last row based on the distance image; calculating the inventory quantity of the product on the display shelf based on the depth size of the product included in the detailed information, the empty space distance, and the dimension of the display shelf in the depth direction; Inventory management system.

2. the predetermined position is behind the row of products in the display shelf, above the front of the display shelf, or the ceiling above the front of the display shelf; The inventory management system of claim 1 .

3. The first camera and the second camera are housed in a common housing and formed as a unit. The inventory management system according to claim 1 or 2.

4. the second camera is a distance measuring camera that measures the distance to the product in the last row and acquires the distance image based on the measured distance; The inventory management system according to claim 1 or 2.

5. a determination unit that determines whether an obstacle exists between the first camera and the product row, When the determination unit determines that the obstacle is present, the first camera re-acquires the visible light image. The inventory management system according to claim 1 or 2.

6. An inventory management method in an inventory management system including a first camera, a second camera, a recording unit, a database server, and an information processing unit, The information processing unit The first camera captures a row of at least one product lined up in a depth direction of a display shelf, thereby acquiring a visible light image including an appearance of the product; a distance image including a distance from a predetermined position to the last row of the products is acquired by the second camera photographing the row of products from the predetermined position; acquiring first information registered in the recording unit, the first information indicating a correspondence relationship between the appearance of the product and identification information of the product; acquiring second information recorded in the database server, the second information indicating a correspondence relationship between the identification information and detailed information about the product; obtaining the identification information corresponding to the appearance of the product included in the visible light image from the first information; obtaining the detailed information corresponding to the identification information from the second information; calculating an empty space distance from the innermost end of the display shelf to the product in the last row based on the distance image; calculating the inventory quantity of the product on the display shelf based on the depth size of the product included in the detailed information, the empty space distance, and the dimension of the display shelf in the depth direction; Inventory management methods.

7. In an inventory management system including a first camera, a second camera, a recording unit, a database server, and an information processing unit, a program for causing the information processing unit to execute processing, The process comprises: The first camera captures a visible light image including an appearance of the product by photographing a row of products in which at least one product is lined up in a depth direction of the display shelf; a distance image including a distance from a predetermined position to the last row of the products is acquired by the second camera photographing the row of products from the predetermined position; acquiring first information registered in the recording unit, the first information indicating a correspondence relationship between the appearance of the product and identification information of the product; acquiring second information recorded in the database server, the second information indicating a correspondence relationship between the identification information and detailed information about the product; obtaining the identification information corresponding to the appearance of the product included in the visible light image from the first information; obtaining the detailed information corresponding to the identification information from the second information; calculating an empty space distance from the innermost end of the display shelf to the product in the last row based on the distance image; calculating the inventory quantity of the product on the display shelf based on the depth size of the product included in the detailed information, the empty space distance, and the dimension of the display shelf in the depth direction; program.

Images