Product sales system

Abstract

To provide an unattended service without allowing a customer to enter a store after business hours with store staff.SOLUTION: A commodity selling system includes a store server 4 and a picking robot 6. The store server 4 includes: an order receiving unit 42 which receives an order of a commodity from a customer terminal 2; and a picking processing unit 44 which issues an instruction to the picking robot 6 to pick up a commodity related to the order from a display shelf 5 and transport it to a handover locker 7. The picking robot 6 includes an arm, picking means arranged at a tip of the arm, and a sensor arranged near the tip of the arm. The system detects objects by image processing, from images of display stages captured by the sensor where commodities are displayed, identifies the commodities with the objects by image recognition processing based on a recognition model set for each of the commodities with respect to the objects, and moves the picking means to a position so as to pick up the commodity.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for selling products in a physical store, and more particularly to a technology that is effective when applied to a product sales system for selling products in an unmanned physical store. [Background technology]

[0002] Convenience stores (hereinafter referred to as "convenience stores") and other retail businesses have extended their business hours, such as late-night and early-morning hours and 24-hour operations, to provide convenience to customers and increase sales and profits by increasing customer opportunities. However, due to the declining birthrate and aging population, which has led to a decline in the working population, especially among young people, the retail industry is also facing a serious labor shortage, and it is becoming increasingly difficult to secure personnel who can work late-night and early-morning hours. As the cost of securing personnel during these hours is high, while there are fewer customers and sales are lower than during the day, there is a growing trend to end 24-hour operations and shorten business hours by closing late-night and early-morning hours.

[0003] In response to this situation, consideration is being given to operating stores unmanned without the need for human intervention during late-night and early-morning hours. For example, Japanese Patent Laid-Open Publication No. 2019-021283 (Patent Document 1) describes an unmanned store system that includes an authentication device that performs biometric authentication of a user, a gate control device that opens a gate to allow the user to enter and exit the store if authenticated, and an unmanned cash register that performs biometric authentication of the user to settle the product price. Furthermore, Non-Patent Document 1 describes a demonstration experiment of unmanned operation of a convenience store during late-night hours. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-021283 [Non-patent literature]

[0005] [Non-Patent Document 1] "Experimental Start of Unmanned Late-Night Operations," [online], March 29, 2019, Lawson, Inc., [Retrieved October 18, 2020], Internet<URL:https: / / www.lawson.co.jp / company / news / detail / 1369017_2504.html> Summary of the Invention [Problem to be solved by the invention]

[0006] According to conventional technology, it is possible for existing retail stores to operate normally during the day and operate unmanned only during late night and early morning hours when it is difficult to secure staff, without the need to install dedicated sales areas, product pickup areas, vending machines, etc. However, with a system that allows customers to enter an unmanned store, there is a limit to how much security and crime prevention risks can be reduced, no matter how many measures are taken.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a merchandise sales system that enables unmanned sales outside of manned business hours without requiring customers to enter the store.

[0008] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0009] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0010] A representative embodiment of the present invention is a product sales system that sells products displayed on display shelves in a store to customers outside of the store's manned business hours, and includes a store server consisting of an information processing system and a picking robot that can move within the store.The store server includes an order reception unit that receives product orders from the customer's information processing terminal via a network, and a picking processing unit that instructs the picking robot to pick the ordered product from the display shelves and transport it to a specified delivery location.

[0011] The picking robot also has an arm whose tip can be moved to a desired position, a picking means installed at the tip of the arm to pick up products, and a sensor with an imaging function installed near the tip of the arm, and detects one or more objects using image processing from an image of the display shelf on which the products related to the order are displayed, taken by the sensor, and identifies which of the objects is the product related to the order using image recognition processing based on a recognition model set for each product for each object, and moves the picking means to a position where it can pick the product related to the order. [Effects of the Invention]

[0012] The effects obtained by the representative inventions disclosed in this application can be briefly explained as follows.

[0013] That is, according to the representative embodiment of the present invention, it is possible to operate an unmanned store outside of staffed business hours without requiring customers to enter the store. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram showing an outline of a configuration example of a product sales system according to an embodiment of the present invention; [Figure 2]FIG. 1 is a diagram schematically illustrating an example of the configuration of a picking robot according to an embodiment of the present invention. [Figure 3] 1A and 1B are diagrams showing an outline of a typical configuration example of a product conveying table according to an embodiment of the present invention. [Figure 4] 1A and 1B are diagrams schematically illustrating an example of installation of lockers according to an embodiment of the present invention. [Figure 5] FIG. 1 is a diagram schematically illustrating an example of a movement route of a picking robot according to an embodiment of the present invention. [Figure 6] FIG. 1 is a diagram illustrating an example of a method for identifying the display position of a commodity on a display shelf according to an embodiment of the present invention. [Figure 7] 1A and 1B are diagrams showing an outline of an example of an image recognition method according to an embodiment of the present invention. [Figure 8] FIG. 2 is a diagram illustrating an example of a data configuration of a product master DB according to an embodiment of the present invention. [Figure 9] FIG. 2 is a diagram illustrating an example of a data configuration of an inventory DB according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram outlining an example of a data configuration of a locker DB according to an embodiment of the present invention. [Figure 11] FIG. 2 is a diagram outlining an example of a data configuration of an order DB according to an embodiment of the present invention. [Figure 12] FIG. 10 is a process flow diagram outlining an example of the flow of a product sales process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all drawings used to explain the embodiments, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. However, parts that have been designated and explained in one drawing may be referred to by the same reference numerals in the explanation of other drawings, although they will not be shown again.

[0016] <Summary> A product sales system according to one embodiment of the present invention realizes a service that enables off-hours (e.g., late at night, early in the morning, on weekends, national holidays, the New Year holidays, etc.) operations to be automated using robots in brick-and-mortar stores that sell products to customers, such as convenience stores, department stores, supermarkets, drug stores, shopping complexes, bookstores, and other retail outlets. Here, "business hours" refers to the time periods during which store personnel, such as store clerks, perform operations such as selling products, and "off-hours" refers to the time periods other than the aforementioned "business hours." According to this embodiment, "off-hours" can also be considered "business hours" because robots can sell products to customers. Therefore, hereinafter, the time periods during which store clerks perform operations (corresponding to conventional "business hours") may be referred to as "staffed business hours," and the time periods during which robots perform operations (corresponding to conventional "off-hours") may be referred to as "unmanned business hours."

[0017] In this embodiment, a customer uses an information processing terminal outside the store in advance (or during manned business hours) to order a product via a network. During unmanned business hours, the store's entrances and exits are locked, preventing customers from entering the store. However, a robot moves around the unmanned store on behalf of the customer, picking and collecting the ordered products from the shelves, and transporting them to a dedicated delivery location. This delivery location may consist of, for example, a small window or locker that allows delivery of products inside or outside the store. The customer makes a cashless payment using electronic money, a credit card, or the like (or completes an electronic payment in advance) and receives the product from this delivery location. This allows the store to sell products inside the store without allowing customers into the store, even during "off-hours."

[0018] <System configuration> 1 is a diagram showing an overview of an example of the configuration of a product sales system according to one embodiment of the present invention. The product sales system 1 is composed of, for example, one or more picking robots 6 that move around a store 3, picking products ordered by customers from display shelves 5 and transporting them to a dedicated delivery location, delivery lockers 7 that are dedicated locations for delivering products inside and outside the store 3, and a store server 4.

[0019] [Picking robot] 2 is a diagram schematically illustrating an example of the configuration of a picking robot 6 according to an embodiment of the present invention. The picking robot 6 includes various components, such as an automatic guided vehicle (AGV) 61, a control unit 62, an arm 63, a suction pad 64, a vacuum pump 65, a barometer 66, a depth sensor 67, and a product transport platform 68.

[0020] The automated guided vehicle 61 is configured to move and travel on a magnetic tape 31 laid on the floor along a passageway for accessing the display shelves 5 in the store 3. Specifically, for example, a rail-guided AGV manufactured by Sharp Corporation<XFシリーズ> The automatic guided vehicle 61 may be configured to move and travel while automatically recognizing its own position within the store 3 using so-called SLAM (Simultaneous Localization and Mapping) technology, without being limited to the use of magnetic tape.

[0021] The control unit 62 includes, for example, a CPU (Central Processing Unit), a recording device such as memory, a communication device, etc., and communicates with the store server 4 (described later) via a network to receive instructions related to product picking and the like and to notify the transport status. It also controls the overall operation of the picking robot 6 based on instructions from the store server 4, and picks and transports products from the display shelves 5. After receiving instructions from the store server 4, the control unit 62 may independently control all of the picking robot 6, or it may communicate with the store server 4 in real time and adjust the control content as appropriate in response to instructions from the store server 4.

[0022] A suction pad 64 is attached to the tip of the arm 63 as a product picking means, and the suction pad 64 can be moved to a desired position by bending and extending each movable part of the arm 63 based on instructions from the control unit 62. The suction pad 64 is connected to a barometer 66 and a vacuum pump 65 via a hose, and the vacuum suction function of the suction pad 64 is realized by controlling the vacuum pump 65 based on instructions from the control unit 62 (the success or failure of suction can be determined by the barometer 66). Specifically, the arm 63 can be configured, for example, by a UR5 from Universal Robots, the suction pad 64 can be configured, for example, by a bell-shaped vacuum pad from Schmalz Corporation, and the vacuum pump 65 can be configured, for example, by a Linicon LV660 from Nitto Kohki Co., Ltd.

[0023] In this embodiment, the display shelf 5 is configured to include two suction pads 64, but the number of suction pads 64 is not limited to this. Also, in this embodiment, the vacuum suction function of the suction pads 64 and the vacuum pump 65 is used as a means for picking a desired product from the display shelf 5, but the picking means is not limited to this. For example, it may be a mechanical gripping means with a mechanism simulating a magic hand or a human hand, or it may be a means for scooping up a product with a fork-shaped mechanism or the like.

[0024] A depth sensor 67 is also attached near the tip of the arm 63. The depth sensor 67 is a sensor having a depth detection function and an image capturing function, and specifically, can be configured, for example, by Intel's RealSense (registered trademark) Depth Camera. As will be described later, when the control unit 62 of the picking robot 6 identifies the position of the product to be picked from the display shelf 5, it identifies the position by image recognition using AI (artificial intelligence) from the image of the display shelf 5 captured by the depth sensor 67. In addition, the control unit 62 determines the depth (depth) to the target product based on the depth information obtained from the depth sensor 67, and based on this information, controls the arm 63 and suction pad 64 to suck and pick the product, and then places it on the product transport table 68.

[0025] The product transport table 68 is a component that functions as a "shopping basket" that holds products picked up by the picking robot 6 from the display shelf 5 until they are transported to the delivery locker 7. In this embodiment, the surface on which products are placed on the product transport table 68 is configured as a belt conveyor, so that the products being held can be transported to the delivery locker 7 easily and smoothly.

[0026] FIG. 3 is a diagram schematically illustrating an example of the configuration of a product conveying platform 68 according to an embodiment of the present invention. FIG. 3(a) is a diagram illustrating an example of the configuration of a picking robot 6, mainly focusing on the product conveying platform 68. In this embodiment, a belt conveyor 681 is used as the loading surface of the product conveying platform 68. Specifically, for example, the platform can be configured using a Mini Mini X2 (MMX2) manufactured by Maruyasu Machinery Co., Ltd. The picking robot 6 controls the arm 63 to pick up a product by suction with a suction pad 64 from the display shelf 5 and place the picked product on the upper surface (loading surface) of the belt conveyor 681 as shown in the figure.

[0027] A protective wall 682 is provided on the outer periphery of the loading surface of the belt conveyor 681 to prevent the held products from falling, etc. Of the four sides of the periphery, one side corresponding to the downstream side of the belt conveyor 681 (the left side in the example of FIG. 3(a)) is provided with a transport gate 683 as a product transport opening instead of a protective wall 682. The transport gate 683 is normally fixed in a closed state as shown in the figure to prevent the held products from falling, etc.

[0028] FIG. 3(b) is a diagram outlining an example of a configuration when transporting products from the product transport platform 68 to a storage space 71 in the delivery locker 7. Like FIG. 3(a), FIG. 3(b) also shows an excerpt of the product transport platform 68 of the picking robot 6. The picking robot 6 moves itself so that the product transport platform 68 is positioned so that the product can be transported to the desired storage space 71 in the delivery locker 7, and adjusts the height of the product transport platform 68 as necessary using a servo mechanism (not shown). The belt conveyor 681 then rotates to move the products held on the upper surface downstream (to the left in the example of FIG. 3(b)). At this time, the transport gate 683 is opened as shown in the figure, allowing the products to exit the product transport platform 68.

[0029] 3(b), the transport gate 683 is configured to open like a double door, but is not limited to this, and may be configured to open like a sliding door, or may be configured to open by moving the transport gate 683 as a whole upward or downward.Furthermore, the transport gate 683 may be configured to open by tilting outward from the product transport platform 68.

[0030] Meanwhile, a product receiving belt conveyor 72 may also be installed on the bottom portion of the storage space 71 of the delivery locker 7 as shown in the figure, and may be rotated in conjunction with the belt conveyor 681 of the picking robot 6 to receive products that have come out from the product transport table 68 and move and transport them to the back side of the storage space 71. In this case, the belt conveyor 72 may start rotating when it detects that a product has been received (or received) by a sensor (not shown), for example, or may be controlled to start rotating in response to an instruction from the picking robot 6. It is also possible to configure the storage space 71 without the belt conveyor 72, so that the picking robot 6 moves the product transport table 68 to insert it into the storage space 71 and then comes out while placing products sequentially from the back side.

[0031] There may be multiple products of various types placed on the belt conveyor 681 of the product conveying platform 68, and depending on the product, there is a possibility that the product may interfere with other products when being transported to the storage space 71 of the delivery locker 7, resulting in damage to the product itself or other products. For this reason, for example, a weight sensor may be installed on the belt conveyor 681 to grasp the total weight of the products being held, and products exceeding a predetermined weight may not be held (in this case, for example, all of the products related to the order may be transported in multiple trips rather than being picked in one go).

[0032] Furthermore, the weight distribution and tilt of the upper surface of the products held on the upper surface of the belt conveyor 681 may be detected, and the arm 63 may be controlled to adjust the placement position of the products to maintain overall balance when placing the products on the product conveying table 68, thereby preventing heavy products from moving or rolling and interfering with other products. Alternatively, when placing products on the product conveying table 68, heavy products may be preferentially placed downstream so that they are transported before lighter products when transported to the storage space 71, preventing them from interfering with lighter products. Furthermore, the posture of the product when placed on the product conveying table 68 may be adjusted to be as stable as possible depending on the shape of the product.

[0033] [Delivery Locker] In this embodiment, delivery lockers 7, which are used to deliver merchandise to and from customers, are installed, for example, on a wall or window in a location within store 3 that customers can access from outside the store (for example, on a wall facing the parking lot on the first floor). Figure 4 is a diagram that schematically illustrates an example of the installation of delivery lockers 7 in one embodiment of the present invention. The example in Figure 4 shows an example in which 3 x 3 = 9 delivery lockers 7 are installed.

[0034] FIG. 4(a) is a schematic diagram showing a delivery locker 7 installed on a wall or window of the store 3, as seen from inside the store. The portion of the delivery locker 7 indicated by the dotted line indicates that it is located outside the store 3. In the example of FIG. 4(a), the inside of each delivery locker 7 is open and has no doors, and products can be freely placed in each storage space 71 from inside the store using the method shown in FIG. 3(b) above. However, this configuration is not limited to this. Each delivery locker 7 may have a door or the like on the inside of the store, and the door or the like is normally closed to prevent access to the storage space 71. However, the door or the like may be opened to allow access to the storage space 71 when the picking robot 6 transports products.

[0035] FIG. 4(b) is a diagram showing a typical view of the delivery lockers 7 as seen from outside the store. The portion of the delivery lockers 7 shown by the dotted line indicates that they are located inside the store 3. Each delivery locker 7 has an openable door 72 on the outside of the store. This door 72 can be configured as a door that can be electronically locked and unlocked, such as those used in known electronic lockers.

[0036] For example, the outside of the delivery locker 7 may be equipped with a short-range wireless communication device such as an IC card reader (not shown), and the door 72 may be unlocked and opened by the customer holding up a mobile terminal with short-range wireless communication capabilities, such as a smartphone, associated with the customer. If the customer has not yet made a prepayment by credit card or the like, the customer may make an electronic payment on the spot before the door is unlocked. When the customer opens the door 72 of the delivery locker 7 assigned to their order, the ordered items have already been transported to the storage space 71 by the in-store picking robot 6, and the customer can pick up the items from outside the store via the delivery locker 7 without having to enter the store.

[0037] [Store Server] In the example of FIG. 1, the store server 4 is an information processing system that receives orders from customers, instructs the picking of ordered products from the display shelves 5 and their transport to the delivery locker 7, and processes payments by customers. For example, it is implemented by a server device or a virtual server built on a cloud computing service, and is configured to be connected to a customer terminal 2, which is an information processing terminal such as a PC (Personal Computer) or smartphone that customers use to process orders, via a network such as the Internet (not shown). The customer terminal 2 may be an information processing terminal owned by the customer, or may be a dedicated ordering device or ordering terminal installed outside the store 3.

[0038] The store server 4, for example, uses a CPU (not shown) to execute middleware such as an OS (Operating System) and a Web server, which are loaded from a storage device such as a hard disk drive (HDD) onto memory, and software running on the OS and middleware, thereby realizing various functions (described below) related to product sales during unmanned business hours. The store server 4 includes various units implemented by software, such as an available product presentation unit 41, an order reception unit 42, a picking preparation processing unit 43, a picking processing unit 44, and a payment processing unit 45. The store server 4 also includes various data stores, such as a product master database (DB) 46, an inventory DB 47, a locker DB 48, and an order DB 49, which are configured using databases, files, etc.

[0039] The available product presentation unit 41 has a function of receiving a request from a customer via the customer terminal 2, extracting information on products that are available for sale during the current or upcoming unmanned business hours from among the products displayed on the shelves 5 of the store 3, and presenting the information on the customer terminal 2. For example, via a web server program (not shown), the available products are displayed as a list on a web browser (not shown) or a dedicated application on the customer terminal 2. Products may be narrowed down and displayed based on conditions specified by the customer.

[0040] When extracting products that can be sold, based on information from the product master DB46 and inventory DB47 described below, products with a display quantity of zero, i.e. products that are not currently displayed on the display shelves 5 (even if there is inventory in the back room or warehouse, this also applies as they cannot be sold during unmanned business hours unless there is a system in place to unmannedly replenish products from inventory onto the display shelves 5), and products that, due to their nature, cannot be sold unmanned or are not suitable for unmanned sales in the first place, are excluded from the products that can be sold.

[0041] Whether or not a product is inherently incapable of or unsuitable for unmanned vending due to its characteristics can be set in advance as attribute information for each product itself in the product master DB 46. For example, there may be cases where a product is incapable of or unsuitable for unmanned vending due to the characteristics of the product itself or the specifications and performance of the picking robot 6, such as a product that poses a risk of liquid spilling or being damaged during transportation, or a product that is too small or has an irregular shape and therefore cannot (or is difficult to) be picked by the picking robot 6.

[0042] Furthermore, for example, products whose weight exceeds a predetermined upper limit may be excluded from products that can be sold unmanned, and products whose price is higher than a predetermined upper limit may be excluded from products that can be sold unmanned from the standpoint of security and crime prevention. Also, products that are not necessarily incapable of unmanned sales as a result of their attributes, but that are effectively impossible or difficult to pick because they cannot be accessed by the picking robot 6 due to, for example, the location of the display shelf 5 on which the product is displayed, the display position within the display shelf 5, or the type of display shelf 5 (refrigerator, freezer, thermal case, etc.), may be excluded from products that can be sold.

[0043] The order receiving unit 42 has the function of receiving order information from the customer terminal 2, including the products that the customer has requested to purchase from the products available for sale displayed on the customer terminal 2, the quantity of the products, and payment information, and recording the information in the order DB 49 described below.

[0044] The picking preparation processing unit 43 has a function of performing preparation processing for the picking robot 6 to actually pick up an order received by the order receiving unit 42. The preparation processing includes, for example, a process of assigning a delivery locker 7 to be used for delivery of products to and from a customer. The usage status of each delivery locker 7 is recorded in a locker DB 48, which will be described later. The preparation processing also includes a process of obtaining information about each product related to the order from the product master DB 46 and inventory DB 47, and determining a movement route that the picking robot 6 will take to sequentially pick each product from the display shelf 5 and transport it to the delivery locker 7.

[0045] FIG. 5 is a diagram that schematically illustrates an example of a movement route of a picking robot 6 according to an embodiment of the present invention. The example in FIG. 5 shows a plan view of the interior of a store 3, showing the installation of multiple display shelves 5 (A1-A4, B1-B7, C1-C7, D1-D3, E1-E2) and multiple delivery lockers 7. The example also shows that the picking robot 6 is waiting at a predetermined waiting location. The picking preparation processing unit 43 of the store server 4 (not shown) references the inventory DB 47 to determine on which display shelf 5 the ordered items are displayed. Based on the location of each item on the display shelf 5, the picking preparation processing unit 43 picks each item, transports it to the delivery locker 7, and searches for and determines a movement route for returning to the waiting location.

[0046] It is desirable that the movement route be the shortest route, but it does not necessarily have to be the shortest route. For example, when picking multiple products, the picking order may be adjusted depending on the type of product, and the movement route may be determined accordingly. Also, one movement route may be determined for multiple orders (or multiple customers). In this case, a different delivery locker 7 is assigned to each order, but multiple orders from the same customer may be transported together to a single delivery locker 7.

[0047] In FIG. 5, curved arrows indicate an example of a movement route when the ordered products are displayed on the "A4" and "C2" display shelves 5. According to this movement route, the picking robot 6 first moves to the "C2" display shelf 5 via the route indicated by the arrow "a", picks the target product from there, and places it on the product conveying tray 68. Then, it moves to the "A4" display shelf 5 via the route indicated by the arrow "b", picks the target product from there, and places it on the product conveying tray 68. Then, it moves to the delivery locker 7 assigned to the customer via the route indicated by the arrow "c" (the shaded area in the figure), and transports the product placed on the product conveying tray 68 into the storage space 71 by the belt conveyor 681. Then, it returns to the designated waiting area via the route indicated by the arrow "d".

[0048] 1, the picking processing unit 44 has the function of issuing instructions and commands to the picking robot 6 regarding a series of processes, such as moving along the movement route determined by the picking preparation processing unit 43, sequentially picking up products displayed on the display shelves 5, transporting them to the assigned delivery locker 7, and returning to the waiting area, as well as managing the picking status. As described above, the necessary information regarding the series of picking processes (information on the products to be picked and the display shelves 5, information on the display positions on the display shelves 5, the assigned delivery lockers 7, the movement route, etc.) may be initially transmitted all at once to the picking robot 6, and thereafter the picking robot 6 may perform the picking process autonomously, or instructions may be issued sequentially while communicating with the picking robot 6 as needed in real time or in a similar manner.

[0049] When the picking robot 6 picks a desired product from the display shelf 5 (in this embodiment, it picks up the product using the suction pad 64), it is necessary to identify where on the display shelf 5 the product is displayed. Generally, in store 3, where on the display shelf 5 a product will be displayed can be set and fixed to some extent in advance, up to which shelf (hereinafter referred to as "display shelf"; for example, "top shelf," "middle shelf," "bottom shelf," etc.), but it is not possible to set in advance which position within that display shelf (for example, "xxth from the left," etc.) the product will be displayed, or even if it can, it is thought that this is fluid and the position will often change later.

[0050] Therefore, in this embodiment, display position information, such as which display shelf of which display shelf 5 each product is displayed on, is held in a database (for example, the product master DB 46 and inventory DB 47, which will be described later), while the actual display position of each product on the display shelf is determined in real time based on information obtained by the depth sensor 67 of the picking robot 6. That is, the depth sensor 67 captures an image of the vicinity of the target display shelf, and the target product is identified and specified by image recognition processing using AI based on the captured image data, and the depth information is further used to determine its position (up / down, left / right, and depth) and orientation. This makes it possible to identify and specify the target product and determine its display position even if the product's display position is shifted, or if the product is tilted or tipped over.

[0051] FIG. 6 is a diagram outlining an example of a method for identifying the display position of a product on a display shelf 5 in one embodiment of the present invention. FIG. 6 shows an example in which the product to be picked is "Product C," and the product is on the upper shelf of the display shelf 5. The picking robot 6 moves close to the display shelf 5 and captures an image of the vicinity of the upper shelf of the display shelf 5 using the depth sensor 67. Then, object detection is performed on the captured image using a known contour extraction technique or the like (in the example of FIG. 6, four objects surrounded by dotted squares are detected), and image recognition is performed using AI on each of the detected objects to identify objects that match "Product C."

[0052] The recognition model 441, which is a learning model used by the AI ​​at this time, is generated in advance by learning based on image data captured of each product for sale in the store 3, and is stored in the store server 4. Note that the image recognition process by the AI ​​may be performed by the picking processing unit 44 of the store server 4, or the control unit 62 of the picking robot 6 may acquire the recognition model 441 from the store server 4 and perform it itself. In addition, there are no particular limitations on the AI ​​engine, library, etc. used, and any publicly available AI engine, library, etc. may be used as appropriate.

[0053] [Product identification using image recognition] 7 is a diagram outlining an example of an image recognition method according to an embodiment of the present invention. Fig. 7(a) shows a method in which, for example, for each shelf (each level) of a display shelf 5, one recognition model 441 capable of identifying all products displayed on that shelf (in the example shown, "Product A," "Product B," "Product C," "Product D," ...) is created and applied. In this case, this recognition model 441 is sequentially applied to each object identified from the image of the display shelf by image processing such as contour extraction, to identify which product each object is.

[0054] Although such image recognition techniques are common, for example, in a chain of convenience stores or the like, there are many cases where the products handled by each store 3 are similar but the lineup and display positions of the products displayed on each display shelf 5 or display shelf are different, which ultimately results in the need to create an individual recognition model 441 for each store 3. This is not a problem when there are only a few stores 3 or when the positions and types of products displayed on the display shelves 5 are not changed frequently, but as the number of stores 3 increases, the burden of handling increases and it becomes unrealistic.

[0055] Therefore, in this embodiment, as shown in FIG. 7(b), a recognition model 441 for identifying each product is created in advance. In this case, the identification model 441 created for each product is applied to each object identified from an image of a display shelf by image processing such as contour extraction, to identify which object is the target product. In this method, since the identification model 441 is created for each product, it can be created in advance in bulk at the headquarters of a convenience store chain and provided to each store 3 as part of the product information stored in the product master DB 46. This allows each store 3 to directly use the recognition model 441 created for each product, regardless of which display shelf 5 or display shelf each product is displayed on. This significantly reduces the burden associated with creating and preparing the recognition model 441 and also improves the accuracy of product identification through image recognition.

[0056] To identify the product to be picked from among the objects identified from the image of the display shelf, for example, a recognition model 441 for the target product is obtained, and this is applied to each object in turn to distinguish them using AI, and the object with the highest score can be determined to be the target product. In this embodiment, to further improve accuracy, a recognition model 441 is obtained for each product displayed on the target display shelf (including not only the product to be picked but also other products displayed on the display shelf), and the recognition model 441 for each product is applied to each object in turn to distinguish them using AI, and the product with the highest score is determined to be the product related to the object. Note that information about the other products displayed on the target display shelf can be obtained, for example, from the inventory DB 47.

[0057] Generally, similar products of the same type or products of the same type or grade are often displayed side by side on a shelf, and products with similar appearances such as size, shape, pattern, and color are often displayed side by side. Therefore, in order to further improve the accuracy of product identification, in addition to each product displayed on the shelf in question, recognition models 441 related to products with similar appearances may also be acquired and applied to further improve accuracy. Note that information on other products with similar appearances for each product can be set and registered in advance in the product master DB 46, for example.

[0058] To identify the product to be picked from each object identified by image processing such as contour extraction, as described above, it is necessary to acquire a recognition model 441 for each product, including other products displayed on the target shelf (which may include products similar in appearance to the target product). Regarding these recognition models 441, regardless of whether the image recognition process is performed by the picking robot 6 or the picking processing unit 44 of the store server 4, including when multiple products are to be picked, in this embodiment, when the picking robot 6 moves and approaches within a predetermined distance from the target shelf 5, the recognition models 441 for products related to the product to be picked from the shelf 5 (such as other products displayed on the same shelf or products similar in appearance) are identified and acquired as needed. However, this is not limited to this. All recognition models 441 that will be needed later may be identified and acquired in advance at the first stage of the picking process.

[0059] For example, when image recognition processing is performed by the picking robot 6, the recognition model 441 once acquired from the store server 4 may be associated with the target product and display shelf 5 and stored as a history. This makes it possible to avoid the burden of having to communicate with the store server 4 to acquire the recognition model 441 that has already been acquired once again when a product that has already been picked needs to be picked again for another order. During unmanned business hours, the arrangement of products displayed on the display shelves 5 does not generally change, so the recognition model 441 of the products displayed on each display shelf 5 can be reused for that display shelf 5.

[0060] <Data structure> 8 is a diagram outlining an example of the data configuration of the product master DB 46 in one embodiment of the present invention. The product master DB 46 is a table that holds master information about each product sold in the store 3, and includes items such as product ID, product name, type, unit price, weight, whether unmanned sales are possible, recognition model, and similar product ID.

[0061] The product ID field holds identification information such as an ID or number that uniquely identifies each product. The product name, type, and unit price fields hold the name of the target product, the name or code value that identifies the product type (for example, "instant noodles" or "detergent"), and the unit sales price of the target product, respectively. The weight field holds information on the weight per sales unit of the target product. The unmanned sales availability field holds information such as a flag indicating whether the target product is eligible for unmanned sales according to this embodiment. This makes it possible to forcibly set the target product as ineligible for unmanned sales.

[0062] The recognition model item holds information about the recognition model 441 for identifying the target product through image recognition by AI. The recognition model 441 itself may be held, or instruction information such as a path or link to the recognition model 441 managed in a file or on another recording medium may be held. The similar product ID item holds the product IDs of other products (if there are multiple products) that have similar appearances to the target product.

[0063] 9 is a diagram outlining an example of the data configuration of inventory DB 47 in one embodiment of the present invention. Inventory DB 47 is a table that holds information related to the inventory and display status of each product sold in store 3, and has items such as product ID, inventory quantity, display quantity, display shelf ID, and display shelf.

[0064] The product ID item is identification information such as an ID that uniquely identifies each product, and has the same content as the product ID item in the product master DB 46 described above. The inventory quantity item holds information on the inventory quantity of the target product at the store 3 at that time. The display quantity item holds information on the display quantity of the target product on the display shelf 5 at that time at the store 3. If the display quantity is zero, the product will be replenished from inventory and displayed during manned business hours unless the inventory quantity is zero, but during unmanned business hours, the product will be treated as out of stock unless there is a means of unmanned replenishment and display using a robot or the like.

[0065] The display shelf ID field holds identification information such as an ID or number that identifies the display shelf 5 on which the target product is actually displayed (for example, "C2" in the example of FIG. 5) when the value of the display quantity described above is not zero. By referencing this information, the picking robot 6 can move to the target display shelf 5. The display shelf field holds information that identifies the display shelf on which the target product is actually displayed within the display shelf 5 identified by the display shelf ID field described above (for example, "top shelf," "middle shelf," "bottom shelf," "first shelf," "second shelf," "third shelf," etc.) when the value of the display quantity is not zero. This field makes it possible to determine on which display shelf the target product is displayed, as well as which products are displayed on a specific display shelf.

[0066] 10 is a diagram outlining an example of the data configuration of the locker DB 48 in one embodiment of the present invention. The locker DB 48 is a table that stores information such as availability and usage status of the delivery lockers 7 installed in the store 3, and has items such as the locker number, status, status update date and time, and order ID.

[0067] The locker number item holds identification information such as a number or ID that uniquely identifies each delivery locker 7 installed in the store 3. This locker number can be used to identify the location of the delivery locker 7 (for example, "XXXth row, XXXth locker" in the configuration shown in the example of FIG. 4), but a separate item may also be included to hold information for identifying the location of the delivery locker 7.

[0068] The status field holds information such as a string or code value indicating the availability of the target delivery locker 7 (for example, "available," "allocated," "delivered," etc.). For orders from customers, a delivery locker 7 to be used to deliver the products for that order is assigned from among those for which the field indicates "available." The status update date / time field holds information on the timestamp when the value of the above status field was last updated. For example, if a predetermined amount of time has passed since the status became "delivered," a notification may be sent to the target customer urging them to retrieve the products. The order ID field holds information on the ID of the order to which the delivery locker 7 has been assigned when the status of the target delivery locker 7 is "allocated" or "delivered." This order ID has the same content as the order ID field in the order DB 49 described below.

[0069] 11 is a diagram outlining an example of the data configuration of the order DB 49 in one embodiment of the present invention. The order DB 49 is a table that holds information related to the contents of orders from customers, and includes items such as an order ID, a user ID, a product ID, a purchase quantity, a payment amount, a payment status, an order date and time, a corresponding robot ID, a scheduled delivery date and time, and an order status.

[0070] The order ID field holds identification information such as an ID or sequential number that uniquely identifies each order at the target store 3. The user ID field holds identification information such as an ID that uniquely identifies the account of the customer who placed the target order. The product ID and purchase quantity fields hold the product ID and quantity information that identify the product instructed to be purchased in the target order. The product ID field has the same content as the product ID field in the product master DB 46 described above. Note that if the purchase of multiple products is instructed in the target order, these fields are held for each requested product.

[0071] The payment amount and payment status fields hold information on the total payment amount for the order and information on a flag indicating whether the customer has completed payment. The order date and time field holds information on the timestamp when the store server 4 accepted the order. The corresponding robot ID field holds identification information such as an ID or number that identifies the picking robot 6 assigned to pick up the items for the order and transport them to the delivery locker 7. The scheduled delivery date and time field holds information on the date and time estimated by the picking processing unit 44 of the store server 4, etc., when the picking robot 6 will have completed transporting the items for the order to the delivery locker 7 and the customer will be able to receive them. The order status field holds information such as a character string or code value that indicates the status of the order (e.g., "ordered," "transported," "delivered," etc.).

[0072] <Processing flow> 12 is a process flow diagram outlining an example of the flow of a product sales process in one embodiment of the present invention. First, the store server 4 uses the available product presentation unit 41 to refer to the product master DB 46 and inventory DB 47 to extract products that can be sold (ordered) during unmanned business hours, i.e., products that are available for unmanned sales and are actually displayed on the display shelves 5, and presents these to the customer via the customer terminal 2 (S01). The customer terminal 2 displays available products using a web browser or dedicated application, and when the customer specifies the desired product, payment method, etc. and instructs purchase, order information is sent to the store server 4. The order reception unit 42 of the store server 4 receives the order information sent from the customer terminal 2 and records it in the order DB 49 (S02).

[0073] When an order is accepted, the picking preparation processing unit 43 refers to the locker database 48 and assigns an "empty" delivery locker 7 to be used for delivery of the products for that order (S03). If there are no available delivery lockers 7, the customer terminal 2 is notified of this, and the customer may be able to cancel the order if they wish. If the customer does not cancel, the delivery locker 7 may be assigned after waiting until an available delivery locker 7 becomes available. In this case, the completion of delivery of the products to the delivery locker 7 will be delayed, and as a result, the time when the customer can receive the products will be delayed. Once the delivery locker 7 is assigned, the picking preparation processing unit 43 determines a movement route for the picking robot 6 to sequentially pick up each item related to the order from the display shelf 5, transport it to the delivery locker 7, and return to the waiting area in preparation for picking (S04).

[0074] Thereafter, the picking processing unit 44 of the store server 4 issues a picking instruction to the picking robot 6 via wireless or wired communication (S05). The picking instruction includes at least information about the product to be picked, information about the delivery locker 7 assigned in step S03, and information about the movement route determined in step S04. After issuing the picking instruction, the delivery status of the product is periodically checked (S06) and it is determined whether the delivery status has become "delivered" (S07). The delivery status may be determined, for example, by whether the order status item in the order DB 49 is "delivered," or information about the delivery status may be obtained by inquiring the picking robot 6 each time.

[0075] If the transport status is not "transported" in step S07, it is determined that picking is still in progress, and the picking processing unit 44 calculates the estimated time required until the status becomes "transported" (S08), and by adding this to the current time, obtains the estimated time when the status will become "transported", and presents this to the customer via the customer terminal 2 (S09). After that, the process returns to step S07 and repeats.

[0076] The estimated required time can be calculated based on, for example, the time required for the picking robot 6 to move along the movement route (calculated from the distance of the movement route and the movement speed of the picking robot 6), the time required from moving to the target display shelf 5 to picking up the product (calculated from the time required for image recognition processing and the time required for the picking robot 6 to move the arm 63 to the desired position, etc.), and the time required for the picking robot 6 to transport the product from the product transport table 68 to the storage space 71 of the delivery locker 7 (calculated from the rotation speed and size of the belt conveyor 681, etc.).

[0077] If the delivery status becomes "delivered" in step S07, it is determined that delivery of the product to the delivery locker 7 has been completed, and the picking processing unit 44 notifies the customer via the customer terminal 2 that delivery to the customer is now possible due to the completion of delivery (S10). When a customer visits the store 3 and attempts to unlock or open the delivery locker 7 in question from outside the store, the payment processing unit 45 of the store server 4 refers to the order DB 49 and determines whether the payment status of the order in question is "paid" (S11). As described above, payment may be made online in advance using a credit card or the like, or may be made on the spot at the delivery locker 7 using electronic money or the like.

[0078] If the payment status is not "paid", the process of step S11 is repeated until it becomes "paid", and when it becomes "paid", the door 72 of the delivery locker 7 is unlocked and opened (S12). When the customer removes the product from the storage space 71 and closes the door 72, the delivery locker 7 notifies the store server 4 of this, and the store server 4 updates the order status item for that order in the order DB 49 to "delivered", thereby completing the series of processes.

[0079] Meanwhile, when a picking instruction is issued from the store server 4 to the picking robot 6 in step S05 described above, the picking robot 6 moves from a predetermined waiting location to the target display shelf 5 along the instructed movement route (S21). At this time, when the picking robot 6 moves close to the target display shelf 5, it acquires from the store server 4 the recognition models 441 of all products (which may further include products similar in appearance to the target product) displayed on the display shelf where the product to be picked is displayed on the target display shelf 5 (S22). Note that the necessary recognition models 441 may be acquired from the store server 4 each time the picking robot moves close to the display shelf 5 in this way, or all of the recognition models 441 required for all of the ordered products may be acquired in advance when the picking instruction is received in step S05.

[0080] Thereafter, the picking robot 6 moves to the target display shelf 5, moves the arm 63 to control the position of the depth sensor 67 for the display shelf on which the product to be picked is displayed, takes an image, and identifies the display position of the target product on the display shelf through image recognition by AI (S23).Then, the arm 63 is moved to control the position of the suction pad 64, and the product is picked up by suction pad 64, and placed on the product transport table 68 (S24).

[0081] More specifically, when the picking robot 6 moves to the front of the target display shelf 5, it determines the position where the depth sensor 67 should be placed, depending on the height and width of the display shelf on which the product to be picked is displayed, so that the entire product on that display shelf falls within the image capture range of the depth sensor 67, moves the arm 63 to place the depth sensor 67 at the desired position, and captures an image of the display shelf. Then, based on the captured image, the position of the product to be picked is identified using an image recognition technique such as that shown in the example of Figure 6 above.

[0082] Once the position of the product to be picked is identified, the control unit 62 of the picking robot 6 calculates the relative positional relationship between the target product and the suction pad 64 attached to the tip of the arm 63 based on the position information and the depth information of the target product obtained by the depth sensor 67. From the calculated positional relationship, the control unit 62 of the picking robot 6 generates a movement trajectory for bringing the suction surface of the suction pad 64 into contact with the target product, i.e., for moving the suction pad 64 to a position where it can be picked, and moves the arm 63 to move the suction pad 64 according to the generated movement trajectory. Thereafter, the vacuum pump 65 is turned on to adsorb the target product onto the suction pad 64, and in this state, the arm 63 is moved to move the suction pad 64 to a desired position on the product conveyance table 68. The vacuum pump 63 is then turned off, and the product is released from the suction pad 64 and placed on the product conveyance table 68.

[0083] If the order contains multiple types of products, the processing of steps S21 to S24 above is repeated for each product as necessary. If multiple purchases of the same product are requested, the processing of steps S23 to S24 above is repeated as many times as necessary. Once all the products included in the order have been picked, the picking robot 6 moves to the delivery locker 7 assigned to the order in step S03 above (S25), and transports the products from the product transport platform 68 to the storage space 71 of the delivery locker 7 using the method shown in the example of Figure 3 above (S26). Thereafter, the picking robot 6 returns to the predetermined waiting location following the movement route determined in step S04 above (S27), thereby completing the series of processes.

[0084] As described above, according to the product sales system 1 which is one embodiment of the present invention, even when the store 3 is locked during unmanned business hours and customers cannot enter the store 3, the picking robot 6 moves around the unmanned store 3 on behalf of the customer, picks and collects the ordered products from the display shelves 5 in order, and transports them to the delivery locker 7. This makes it possible to sell products that are actually displayed on the display shelves 5 without allowing customers into the store 3 even during "off-hours," reducing security and crime prevention risks.

[0085] Furthermore, for ordered products, the display position, posture, etc. are identified by AI image recognition from a captured image of the display shelf on the target display shelf 5, so it is possible to identify the display position of the target product regardless of the product's actual display situation. Furthermore, by preparing recognition model 441 used for image recognition for each product rather than for each display shelf, even in a convenience store or the like with many affiliated stores, for example, if the headquarters prepares recognition model 441 for each product in advance, there is no need to individually create and prepare recognition model 441 according to the actual product display situation at each store 3, and product sales system 1 can be easily deployed to many stores 3.

[0086] In addition, by using a belt conveyor 681 on the product transport platform 68 of the picking robot 6, it becomes possible to easily and smoothly transport products to the delivery locker 7. Furthermore, by providing a belt conveyor 72 in the storage space 71 of the delivery locker 7, it becomes possible to transport products even more smoothly.

[0087] The invention made by the inventor has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. Furthermore, the above embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the above embodiments with other configurations.

[0088] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a storage medium such as an IC card, SD card, or DVD.

[0089] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. In reality, it can be assumed that almost all components are interconnected. [Industrial Applicability]

[0090] The present invention can be used in a product sales system that sells products in unmanned brick-and-mortar stores. [Explanation of symbols]

[0091] 1...Product sales system, 2...Customer terminal, 3...Store, 4...Store server, 5...Display shelf, 6...Picking robot, 7...Delivery locker, 41...Available product presentation unit, 42...Order reception unit, 43...Picking preparation processing unit, 44...Picking processing unit, 45...Payment processing unit, 46...Product master DB, 47...Inventory DB, 48...Locker DB, 49...Order DB, 61...automated guided vehicle, 62...control unit, 63...arm, 64...suction pad, 65...vacuum pump, 66...barometer, 67...depth sensor, 68...product transport table, 71...storage space, 72...belt conveyor, 441...cognitive model, 681...Belt conveyor, 682...Protection wall, 683...Transport gate

Claims

1. A merchandise sales system that sells merchandise displayed in a store to customers, A store server; a picking robot that is movable within the store, The store server an order receiving unit that receives product orders; a picking processing unit that instructs the picking robot to pick up the product related to the order and transport it to a predetermined delivery location, The picking robot is an arm capable of moving the tip to a desired position; a suction pad that is installed at the tip of the arm and that picks up products; a depth sensor having an imaging function and installed near the tip of the arm; a product conveyance table on which the product picked by the suction pad is placed; and a product sales system that detects one or more objects from an image of the ordered products displayed, taken by the depth sensor, by image processing; identifies the position of each of the objects; determines the relative positional relationship between the ordered products and the suction pad and the attitude of the ordered products based on the position information of the ordered products and the depth information obtained by the depth sensor; moves the arm to move the suction pad to a position where the ordered products can be picked; adsorbs the ordered products onto the suction pad; moves the arm to move the ordered products to a predetermined position on the product conveying table; and removes the ordered products from the suction pad and places them on the product conveying table.

2. The product sales system according to claim 1, The picking robot is A product sales system that identifies which product each of the objects related to the order is by sequentially applying a recognition model set for each product to each of the objects detected by image processing.

3. The product sales system according to claim 1, The picking robot is A product sales system that identifies which object the ordered product is by image processing in which recognition models set for the ordered product and products similar in appearance to the ordered product are sequentially applied to each of the objects detected by image processing.

4. The product sales system according to claim 1, The picking robot is A product sales system that obtains a recognition model set up to recognize the ordered product through image processing from the store server when the ordered product moves a predetermined distance from the display shelf on which it is displayed.

5. The product sales system according to claim 1, The store server further a sales product presentation unit that presents sales products to the customer based on inventory information and / or master information of products displayed in the store; a picking preparation processing unit that identifies a display shelf on which the product related to the order received by the order receiving unit is displayed from among the products available for sale, and determines a route for the picking robot to move to the display shelf, pick the product, transport it to the specified delivery location, and return to a waiting location.

6. The product sales system according to claim 1, The picking processing unit When the picking robot has completed transporting the ordered merchandise to the predetermined delivery location, the merchandise sales system notifies the customer of this fact.

7. The product sales system according to claim 1, A product sales system in which the loading surface of the product transport table of the picking robot and the bottom surface of the designated delivery location each consist of a belt conveyor, and the belt conveyors are operated in conjunction with each other to transport the product related to the order from the product transport table to a designated position at the designated delivery location.

8. A merchandise sales system that sells merchandise displayed in a store to customers, A store server; a picking robot that is movable within the store, The store server an order receiving unit that receives product orders; a picking processing unit that instructs the picking robot to pick up the product related to the order and transport it to a predetermined delivery location, The picking robot is an arm capable of moving the tip to a desired position; a suction pad that is installed at the tip of the arm and that picks up products; a depth sensor installed near the tip of the arm; a product conveyance table on which the product picked by the suction pad is placed; and The product sales system detects and identifies the position of the ordered product based on the image captured by the depth sensor, and determines the relative positional relationship between the ordered product and the suction pad and the attitude of the ordered product based on the position information of the ordered product and the depth information obtained by the depth sensor, moves the arm to move the suction pad to a position where the ordered product can be picked, adsorbs the ordered product to the suction pad, moves the arm to move the ordered product to a predetermined position on the product transport table, and removes the ordered product from the suction pad and places it on the product transport table.

Images