Commodity selling system

Abstract

To provide an unattended service without allowing a customer to enter a store after business hours with store staff.SOLUTION: A commodity selling system includes a store server 4 and a picking robot 6. The store server 4 includes: an order receiving unit 42 which receives an order of a commodity from a customer terminal 2; and a picking processing unit 44 which issues an instruction to the picking robot 6 to pick up a commodity related to the order from a display shelf 5 and transport it to a handover locker 7. The picking robot 6 includes an arm, picking means arranged at a tip of the arm, and a sensor arranged near the tip of the arm. The system detects objects by image processing, from images of display stages captured by the sensor where commodities are displayed, identifies the commodities with the objects by image recognition processing based on a recognition model set for each of the commodities with respect to the objects, and moves the picking means to a position so as to pick up the commodity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for selling products in a brick-and-mortar store, and more particularly to a technique that is effective when applied to a product sales system for selling products in an unmanned brick-and-mortar store. [Background technology]

[0002] Convenience stores (hereafter referred to as "convenience stores") and other retail businesses have expanded their business hours by opening late at night or early in the morning, or operating 24 hours a day, to provide convenience to customers and increase sales and profits by increasing the number of customers. However, due to the declining birthrate and aging population, especially the younger generation, the labor force has decreased, and the labor shortage in the retail industry has become serious, making it difficult to secure personnel who can work late at night or early in the morning. As the cost of securing personnel is high during these hours, while there are fewer customers and sales are lower than during the day, there is a growing trend to stop 24-hour operations and shorten business hours by closing late at night or early in the morning.

[0003] In response to such circumstances, it is being considered to operate a store unmanned without the need for manpower during late night and early morning hours. For example, Japanese Patent Laid-Open Publication No. 2019-021283 (Patent Document 1) describes an unmanned store system that includes an authentication device that performs biometric authentication of a user, a gate control device that opens the gate to allow the user to enter and exit the store if authentication is successful, and an unmanned cash register device that settles the product price by performing biometric authentication of the user. In addition, Non-Patent Document 1 describes a demonstration experiment of unmanned operation of a convenience store during late night hours. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-021283 A [Non-patent literature]

[0005] [Non-Patent Document 1] “Experiment begins for unmanned operation during late night hours”, [online], March 29, 2019, Lawson Co., Ltd., [Retrieved October 18, 2020], Internet<URL:https: / / www.lawson.co.jp / company / news / detail / 1369017_2504.html> Summary of the Invention [Problem to be solved by the invention]

[0006] According to conventional technology, it is possible for existing retail stores to operate normally during the day and operate unmanned only during late night and early morning hours when it is difficult to secure manpower, without the need to set up dedicated sales areas, product receiving areas, vending machines, etc. However, in a system where customers are forced to enter an unmanned store, there is a limit to how much security and crime prevention risks can be reduced, no matter how many measures are taken.

[0007] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a product sales system that enables unmanned business outside manned business hours without requiring customers to enter the store.

[0008] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0009] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.

[0010] A representative embodiment of the present invention is a product sales system that sells products displayed on display shelves in a store to customers outside of the store's manned business hours, and has a store server consisting of an information processing system and a picking robot that can move around the store, and the store server has an order reception unit that receives product orders from the customer's information processing terminal via a network, and a picking processing unit that instructs the picking robot to pick the ordered product from the display shelves and transport it to a specified delivery location.

[0011] The picking robot also has an arm whose tip can be moved to a desired position, a picking means for picking products that is installed at the tip of the arm, and a sensor with an imaging function that is installed near the tip of the arm, and detects one or more objects through image processing from an image of the display shelf on which the products related to the order are displayed that is taken by the sensor, identifies which of the objects is the product related to the order through image recognition processing based on a recognition model set for each product for each object, and moves the picking means to a position where it can pick the product related to the order. Effect of the Invention

[0012] The effects obtained by the representative inventions disclosed in this application can be briefly explained as follows.

[0013] That is, according to the representative embodiment of the present invention, it is possible to operate an unmanned store outside of manned business hours without having customers enter the store. [Brief description of the drawings]

[0014] [Figure 1] 1 is a diagram showing an overview of a configuration example of a product sales system according to an embodiment of the present invention. [Diagram 2]FIG. 1 is a diagram illustrating an example of the configuration of a picking robot according to an embodiment of the present invention. [Diagram 3] 1A and 1B are schematic diagrams illustrating an example of the configuration of a product conveying table according to an embodiment of the present invention. [Figure 4] 1A and 1B are schematic diagrams illustrating an example of installation of lockers according to an embodiment of the present invention. [Diagram 5] 1 is a diagram showing an outline of a schematic diagram of an example of a movement route of a picking robot according to an embodiment of the present invention; [Figure 6] 1 is a diagram outlining an example of a method for identifying a display position of a commodity on a display shelf according to an embodiment of the present invention; [Figure 7] 1A and 1B are diagrams showing an overview of an example of an image recognition method according to an embodiment of the present invention. [Figure 8] 2 is a diagram showing an overview of an example of a data configuration of a product master DB in one embodiment of the present invention. FIG. [Figure 9] 2 is a diagram showing an overview of an example of a data configuration of an inventory DB in one embodiment of the present invention. FIG. [Figure 10] 2 is a diagram showing an overview of an example of a data configuration of a locker DB in one embodiment of the present invention. FIG. [Figure 11] 2 is a diagram showing an overview of an example of a data configuration of an order DB in one embodiment of the present invention. FIG. [Figure 12] FIG. 1 is a process flow diagram outlining an example of the flow of a product sales process in one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In all the drawings for explaining the embodiment, the same parts are generally given the same reference numerals, and the repeated explanations will be omitted. On the other hand, a part that has been described with a reference numeral in a certain drawing may be referred to with the same reference numeral in the explanation of another drawing, although it will not be shown again.

[0016] <Summary> A product sales system according to an embodiment of the present invention realizes a service in which a robot performs unmanned operations at a brick-and-mortar store that sells products to customers, such as a convenience store, department store, supermarket, drug store, commercial complex, bookstore, or shop, during non-business hours (for example, late at night or early in the morning, on holidays, national holidays, or during the year-end and New Year holidays). Here, "business hours" refers to the time period during which store staff, such as store clerks, perform operations such as selling products, and "non-business hours" refers to the time period other than the above-mentioned "business hours." According to this embodiment, since robots can sell products to customers during "non-business hours," they can also be called "time periods during which "business" is being conducted. Therefore, hereinafter, the time period during which store staff perform operations (corresponding to conventional "business hours") may be referred to as "manned business hours," and the time period during which robots perform operations (corresponding to conventional "non-business hours") may be referred to as "unmanned business hours."

[0017] In this embodiment, a customer uses an information processing terminal outside the store in advance (may be during manned business hours) to order a product via a network. During unmanned business hours, the entrances and exits of the store are locked and customers cannot enter the store, but inside the store, a robot moves around the unmanned store on behalf of the customer, picks up the ordered products from the shelves one by one, and transports them to a dedicated delivery location. This delivery location is, for example, a small window or locker that allows delivery of products inside and outside the store, and the customer receives the product from this delivery location by making a cashless payment using electronic money, a credit card, or the like (or by completing an electronic payment in advance). This makes it possible for the store to sell products inside the store without letting customers into the store, even when it is "outside business hours."

[0018] <System configuration> 1 is a diagram showing an overview of an example of the configuration of a product sales system according to an embodiment of the present invention. The product sales system 1 is composed of, for example, one or more picking robots 6 that move around a store 3, pick up products ordered by customers from display shelves 5, and transport them to a dedicated delivery location, delivery lockers 7 as dedicated locations for delivering products inside and outside the store 3, and a store server 4.

[0019] [Picking robot] 2 is a diagram showing an outline of a configuration example of a picking robot 6 according to an embodiment of the present invention. The picking robot 6 includes various parts such as an automatic guided vehicle (AGV) 61, a control unit 62, an arm 63, a suction pad 64, a vacuum pump 65, an atmospheric pressure gauge 66, a depth sensor 67, and a product transport platform 68.

[0020] The unmanned guided vehicle 61 is configured to move and travel on a magnetic tape 31 laid on the floor along a passageway for accessing the display shelves 5 in the store 3. Specifically, for example, a rail-guided AGV<XFシリーズ> The automatic guided vehicle 61 may be configured to move and travel while automatically recognizing its own position within the store 3 using so-called SLAM (Simultaneous Localization and Mapping) technology, without being limited to the one using magnetic tapes.

[0021] The control unit 62 includes, for example, a CPU (Central Processing Unit), a storage device such as a memory, a communication device, etc., and communicates with the store server 4 described later via a network to receive instructions related to picking products and the like and to notify the transport status. Also, based on instructions from the store server 4, the control unit 62 controls the operation of the entire picking robot 6 to pick and transport products from the display shelves 5. After receiving instructions from the store server 4, the control unit 62 may independently control all of the picking robot 6, or may communicate with the store server 4 in real time and adjust the control content accordingly by receiving instructions from the store server 4.

[0022] A suction pad 64 is attached to the tip of the arm 63 as a product picking means, and the suction pad 64 can be moved to a desired position by bending and extending each movable part of the arm 63 based on instructions from the control unit 62. The suction pad 64 is connected to an atmospheric pressure gauge 66 and a vacuum pump 65 via a hose, and the vacuum suction function of the suction pad 64 is realized by controlling the vacuum pump 65 based on instructions from the control unit 62 (the success or failure of suction can be grasped by the atmospheric pressure gauge 66). Specifically, the arm 63 can be configured, for example, by a UR5 from Universal Robots, the suction pad 64 can be configured, for example, by a bell-shaped vacuum pad from Schmalz Corporation, and the vacuum pump 65 can be configured, for example, by a Linicon LV660 from Nitto Kohki Corporation.

[0023] In this embodiment, the configuration includes two suction pads 64, but the number of suction pads 64 is not limited to this. Also, in this embodiment, a vacuum suction function using the suction pads 64 and the vacuum pump 65 is used as a means for picking a desired product from the display shelf 5, but the picking means is not limited to this. For example, it may be a mechanical gripping means using a mechanism that imitates a magic hand or a human hand, or it may be a means that scoops up a product using a mechanism shaped like a fork or the like.

[0024] A depth sensor 67 is also attached near the tip of the arm 63. The depth sensor 67 is a sensor having a depth detection function and an image capturing function, and specifically, for example, can be configured by Intel's RealSense (registered trademark) Depth Camera. As described later, when identifying the position of a product to be picked from the display shelf 5, the control unit 62 of the picking robot 6 identifies the position by image recognition using AI (artificial intelligence) from an image of the display shelf 5 captured by the depth sensor 67. In addition, the depth (depth) to the target product is grasped based on the depth information obtained from the depth sensor 67, and based on this information, the arm 63 and the suction pad 64 are controlled to suction and pick the product, and the product is placed on the product transport table 68.

[0025] The product transport table 68 is a component that plays the role of a "shopping basket" that holds the products picked up by the picking robot 6 from the display shelves 5 until they are transported to the delivery locker 7. In this embodiment, the surface on which the products are placed on the product transport table 68 is configured by a belt conveyor, so that the held products can be transported to the delivery locker 7 easily and smoothly.

[0026] FIG. 3 is a diagram showing a schematic overview of a configuration example of a product conveying platform 68 in one embodiment of the present invention. FIG. 3(a) is a diagram showing an overview of a configuration example by mainly extracting the product conveying platform 68 part of the picking robot 6. In this embodiment, a belt conveyor 681 is used for the placement surface of the product conveying platform 68. Specifically, for example, it can be configured by a Mini Mini X2 type (MMX2) made by Maruyasu Machine Co., Ltd. The picking robot 6 controls the arm 63 to place the product picked up by suction from the display shelf 5 with the suction pad 64 on the upper surface (placement surface) of the belt conveyor 681 as shown in the figure.

[0027] A protective wall 682 is provided on the outer periphery of the placement surface of the belt conveyor 681 to prevent the held products from falling, etc. Of the four sides of the periphery, one side corresponding to the downstream side of the belt conveyor 681 (the left side in the example of FIG. 3(a)) is provided with a transport gate 683 as a product transport opening instead of a protective wall 682. The transport gate 683 is normally fixed in the closed state as shown in the figure to prevent the held products from falling, etc.

[0028] FIG. 3(b) is a diagram showing an overview of a configuration example when a product is transported from the product transport platform 68 to the storage space 71 of the delivery locker 7. FIG. 3(b) also shows an excerpt of the product transport platform 68 of the picking robot 6, similar to FIG. 3(a). The picking robot 6 moves itself so that the product transport platform 68 is positioned at a position where the product can be transported to the desired storage space 71 in the delivery locker 7, and adjusts the height of the product transport platform 68 as necessary using a servo mechanism (not shown). Thereafter, the belt conveyor 681 is rotated to move the product held on the upper surface side downstream (to the left in the example of FIG. 3(b)). At that time, the transport gate 683 is opened as shown in the figure, so that the product can be taken out of the product transport platform 68.

[0029] In the example of Fig. 3(b), the transport gate 683 is configured to open like a double door, but is not limited to this, and may be configured to open like a sliding door, or may be configured to open by the transport gate 683 moving upward or downward as a whole. Also, the transport gate 683 may be configured to open by falling down to the outside of the product transport platform 68.

[0030] On the other hand, a product receiving belt conveyor 72 may also be installed on the bottom surface of the storage space 71 of the delivery locker 7 as shown in the figure, and may be rotated in conjunction with the belt conveyor 681 of the picking robot 6 to receive products that have come out from the product transport platform 68 and move and transport them to the back side of the storage space 71. In this case, the belt conveyor 72 may start rotating when it detects that a product has been received (or has been received) by a sensor (not shown), or may be controlled to start rotating upon receiving an instruction from the picking robot 6. It is also possible to configure the storage space 71 without the belt conveyor 72, and for the picking robot 6 to move the product transport platform 68 to insert it into the storage space 71 and come out while placing products one by one from the back side.

[0031] There may be multiple products of various kinds placed on the belt conveyor 681 of the product transport platform 68, and some products may interfere with other products when transported to the storage space 71 of the delivery locker 7, resulting in damage to the product itself or the other products. For this reason, for example, a weight sensor may be installed on the belt conveyor 681 to grasp the total weight of the products being held, and products exceeding a predetermined weight may not be held (in this case, for example, all the products related to an order are not transported in a single picking trip, but rather are transported in multiple trips).

[0032] Also, the weight distribution of the products held on the upper surface of the belt conveyor 681 and the inclination of the upper surface may be detected, and the arm 63 may be controlled to adjust the placement position of the products so that the products are balanced as a whole when they are placed on the product transport table 68, thereby preventing the heavy products from moving or rolling and interfering with other products. Alternatively, when placing products on the product transport table 68, heavy products may be preferentially placed downstream so that they are transported before lighter products when transported to the storage space 71, and thus do not interfere with the lighter products. Also, the posture of the products when placed on the product transport table 68 may be adjusted to be as stable as possible depending on the shape of the products.

[0033] [Delivery Locker] In this embodiment, delivery lockers 7, which are used as a place for delivery of merchandise to and from customers, are provided, for example, on a wall or window in store 3 at a location accessible to customers from outside the store (for example, the wall facing the parking lot on the first floor). FIG. 4 is a schematic diagram outlining an example of the installation of delivery lockers 7 in one embodiment of the present invention. The example in FIG. 4 shows an example in which 3×3=9 delivery lockers 7 are installed.

[0034] FIG. 4(a) is a schematic diagram showing a state in which a delivery locker 7 installed on a wall or window of a store 3 is seen from inside the store. The portion of the delivery locker 7 shown by a dotted line indicates that it is located outside the store 3. In the example of FIG. 4(a), the inside of the store of each delivery locker 7 is open and has no door, and products can be freely placed in each storage space 71 from inside the store by the method shown in FIG. 3(b) above, but this is not limited to the configuration. Each delivery locker 7 has a door on the inside of the store, and the door is normally closed so that the storage space 71 cannot be accessed. However, the door may be opened when the picking robot 6 transports products, allowing access to the storage space 71.

[0035] Fig. 4(b) is a schematic diagram showing the delivery locker 7 as seen from outside the store. The portion of the delivery locker 7 shown in dotted lines indicates that it is located inside the store 3. Each delivery locker 7 is provided with an openable and closable door 72 on the outside of the store. This door 72 can be configured as a door that can be electronically locked and unlocked, such as that used in known electronic lockers.

[0036] For example, the delivery locker 7 may be provided with a short-range wireless communication device such as an IC card reader (not shown) on the outside of the store, and the door 72 may be unlocked and opened by a customer holding up a mobile terminal with a short-range wireless communication function such as a smartphone associated with the customer over the door 72 to communicate. If the customer has not made a prepayment by credit card or the like, the door may be unlocked after the customer makes a payment by electronic money on the spot. When the customer opens the door 72 of the delivery locker 7 assigned to his / her order, the product related to the order has already been transported to the storage space 71 by the picking robot 6 in the store, and the customer can receive the product from outside the store via the delivery locker 7 without entering the store.

[0037] [Store server] The store server 4 in the example of FIG. 1 is an information processing system having a function of receiving an order from a customer, instructing picking of the ordered product from the display shelf 5 and transporting it to the delivery locker 7, and processing the payment by the customer. For example, it is implemented by a server device or a virtual server constructed on a cloud computing service, and has a configuration in which a customer terminal 2, which is an information processing terminal such as a PC (Personal Computer) or a smartphone that processes the order, is connected via a network such as the Internet (not shown). The customer terminal 2 may be an information processing terminal owned by the customer himself, or may be a dedicated ordering device or ordering terminal installed outside the store 3.

[0038] The store server 4 realizes various functions related to product sales during unmanned business hours, which will be described later, by executing middleware such as an OS (Operating System) and a Web server, which are expanded from a recording device such as an HDD (Hard Disk Drive) onto a memory, and software running thereon, using a CPU (not shown). The store server 4 has various units, such as a saleable product presentation unit 41, an order reception unit 42, a picking preparation processing unit 43, a picking processing unit 44, and a payment processing unit 45, which are implemented by software. The store server 4 also has various data stores, such as a product master database (DB) 46, an inventory DB 47, a locker DB 48, and an order DB 49, which are configured by databases, files, etc.

[0039] The saleable product presentation unit 41 has a function of receiving a request from a customer via the customer terminal 2, extracting information on products that are available for sale during the current or upcoming unmanned business hours from among the products displayed on the shelves 5 of the store 3, and presenting the information on the customer terminal 2. For example, via a web server program or the like not shown, the saleable products are displayed as a list on a web browser (not shown) or a dedicated application on the customer terminal 2. Products may be narrowed down and displayed based on conditions specified by the customer.

[0040] When extracting sellable products, based on information from the product master DB 46 and inventory DB 47 described below, for example, products with a display quantity of zero, i.e. products that are not currently displayed on the display shelves 5 (even if there is inventory in the back yard or warehouse, etc., this is the same as products that cannot be sold during unmanned business hours unless there is a system in place to replenish products from the inventory to the display shelves 5 unmanned), and products whose nature makes them impossible or unsuitable for unmanned sales in the first place, are excluded from the sellable products.

[0041] Whether or not a product is incapable of or suitable for unmanned sales due to its characteristics can be set in advance as attribute information of each product itself in the product master DB 46. For example, there may be cases where a product is incapable of or suitable for unmanned sales due to the characteristics of the product itself and the specifications and performance of the picking robot 6, such as a product that poses a risk of liquid spilling or being damaged during transportation, or a product that is too small or has an irregular shape and cannot (or is difficult to) be picked by the picking robot 6.

[0042] Furthermore, for example, products whose weight exceeds a predetermined upper limit may be excluded from products that can be sold unmanned, and products whose price is higher than a predetermined upper limit may be excluded from products that can be sold unmanned from the viewpoint of security and crime prevention. Also, products that are not necessarily incapable of unmanned sales as an attribute of the product itself, but are inaccessible to the picking robot 6 due to the location of the display shelf 5 on which the product is displayed, the display position within the display shelf 5, the type of display shelf 5 (refrigerator, freezer, thermal case, etc.), etc., based on information in the inventory DB 47, may be excluded from products that can be sold.

[0043] The order receiving unit 42 has the function of receiving order information from the customer terminal 2, including the products that the customer has requested to purchase from the products available for sale displayed on the customer terminal 2, the quantity of the products, and payment information, and recording the information in the order DB 49 described later.

[0044] The picking preparation processing unit 43 has a function of performing preparation processing for the picking robot 6 to actually pick up an order received by the order receiving unit 42. The preparation processing includes, for example, a process of allocating a delivery locker 7 to be used for delivery of products to and from a customer. The usage status of each delivery locker 7 is recorded in a locker DB 48, which will be described later. The preparation processing also includes a process of acquiring information about each product related to the order from the product master DB 46 and the inventory DB 47, and determining a movement route for the picking robot 6 to sequentially pick up each product from the display shelf 5 and transport it to the delivery locker 7.

[0045] FIG. 5 is a diagram showing an outline of an example of a movement route of the picking robot 6 in one embodiment of the present invention. The example of FIG. 5 shows a plan view of the inside of the store 3, and shows that a plurality of display shelves 5 ("A1" to "A4", "B1" to "B7", "C1" to "C7", "D1" to "D3", "E1" to "E2") and a plurality of delivery lockers 7 are installed. It also shows that the picking robot 6 is waiting at a predetermined waiting place. The picking preparation processing unit 43 of the store server 4 (not shown) refers to the inventory DB 47 to grasp which display shelf 5 the ordered product is displayed on, and searches and determines a movement route for picking up each product based on the position of the display shelf 5 of each product, transporting it to the delivery locker 7, and returning to the waiting place.

[0046] Although it is desirable for the movement route to be the shortest route, it is not necessary to use the shortest route. For example, when picking multiple products, the picking order may be adjusted according to the type of product, and the movement route may be determined accordingly. Also, one movement route may be determined for multiple orders (or multiple customers). In this case, a different delivery locker 7 is assigned to each order, but multiple orders from the same customer may be transported together to one delivery locker 7.

[0047] In FIG. 5, curved arrows show an example of a movement route when the ordered product is displayed on the display shelves 5 of "A4" and "C2". According to this movement route, the picking robot 6 first moves to the display shelf 5 of "C2" via the route of the arrow "a", picks the target product from there and places it on the product conveying table 68. Then, it further moves to the display shelf 5 of "A4" via the route of the arrow "b", picks the target product from there and places it on the product conveying table 68. Then, it moves to the delivery locker 7 (shaded part in the figure) assigned to the customer via the route of the arrow "c", and conveys the product placed on the product conveying table 68 into the storage space 71 by the belt conveyor 681. Then, it returns to a predetermined waiting place via the route of the arrow "d".

[0048] Returning to Fig. 1, the picking processing unit 44 has a function of issuing instructions and commands to the picking robot 6 regarding a series of processes, such as moving along the movement route determined by the picking preparation processing unit 43, sequentially picking up products displayed on the display shelves 5, transporting them to the assigned delivery locker 7, and returning to the waiting area, and also managing the picking status. As described above, the necessary information regarding the series of picking processes (information on the products to be picked and the display shelves 5, information on the display positions on the display shelves 5, the assigned delivery lockers 7, the movement route, etc.) may be initially sent all at once to the picking robot 6, and thereafter the picking robot 6 may autonomously perform the picking process, or instructions may be issued sequentially while communicating with the picking robot 6 in real time or in a state equivalent thereto.

[0049] When the picking robot 6 picks a desired product from the display shelf 5 (in this embodiment, it is adsorbed by the suction pad 64), it is necessary to identify where on the display shelf 5 the product is displayed. Generally, in the store 3, with regard to where on the display shelf 5 a product is to be displayed, it is possible to set and fix to some extent which shelf (hereinafter referred to as the "display shelf"; for example, the "top shelf", the "middle shelf", the "bottom shelf", etc.) on which the product will be displayed, but it is not possible to set in advance which position on that display shelf (for example, "xxth from the left") the product will be displayed in, or even if it can, it is likely to be fluid and the position will change thereafter.

[0050] In this embodiment, display position information on which display shelf 5 and which display level each product is displayed on is stored in a database (for example, the product master DB 46 and the inventory DB 47 described later), while the actual display position of the product on the display level is grasped in real time based on information obtained by the depth sensor 67 of the picking robot 6. That is, the depth sensor 67 captures an image of the vicinity of the target display level, and the target product is identified and specified by image recognition processing by AI based on the captured image data, and the position (up, down, left, right, and depth) and posture are further grasped by using the depth information. This makes it possible to identify and specify the target product and grasp its display position even if the display position of the product is shifted, tilted, or tipped over.

[0051] FIG. 6 is a diagram outlining an example of a method for identifying the display position of an item on a display shelf 5 in one embodiment of the present invention. FIG. 6 shows an example in which the item to be picked is "Item C" and is on the top shelf of the display shelf 5. The picking robot 6, which has moved close to the display shelf 5, captures an image of the area near the top shelf of the display shelf 5 using a depth sensor 67. Then, object detection is performed on the captured image using a known contour extraction technique or the like (in the example of FIG. 6, four objects surrounded by dotted squares are detected), and image recognition is performed by AI on each detected object to identify an object that matches "Item C."

[0052] At this time, the recognition model 441, which is a learning model used by the AI, is generated in advance by learning based on image data captured on each product for sale in the store 3, and is stored in the store server 4. Note that the image recognition process by the AI ​​may be performed by the picking processing unit 44 of the store server 4, or the control unit 62 of the picking robot 6 may acquire the recognition model 441 from the store server 4 and perform it by itself. In addition, there are no particular limitations on the AI ​​engine, library, etc. used, and any publicly available one may be used as appropriate.

[0053] [Product identification using image recognition] Fig. 7 is a diagram outlining an example of an image recognition method in an embodiment of the present invention. Fig. 7(a) shows a method in which, for example, for each shelf (one level) of a display shelf 5, one recognition model 441 capable of identifying all products displayed on that shelf (in the example in the figure, "Product A", "Product B", "Product C", "Product D", ...) is created and applied. In this case, this recognition model 441 is applied sequentially to each object identified by image processing such as contour extraction from the image of the display level to identify which product each object is.

[0054] Although such image recognition techniques are common, for example, in a chain such as a convenience store, although the products handled by each store 3 are similar, there are many cases where the lineup and display positions of the products displayed on each display shelf 5 or shelf are different, and as a result, it becomes necessary to create a recognition model 441 individually for each store 3. In this case, there is no problem when the number of stores 3 is small or when the positions and types of products displayed on the display shelves 5 are not frequently changed, but as the number of stores 3 increases, the burden of handling increases and it becomes unrealistic.

[0055] Therefore, in this embodiment, as shown in FIG. 7(b), a method is adopted in which a recognition model 441 for identifying each product is created for each product. In this case, the identification model 441 created for each product is applied to each object identified by image processing such as contour extraction from the image of the display shelf to identify which object is the target product. In this method, since the identification model 441 is created for each product, it can be created collectively in advance at the headquarters of the convenience store chain, for example, and provided to each store 3 as one of the product information held in the product master DB 46. As a result, in each store 3, the recognition model 441 created for each product can be used as it is, regardless of which display shelf 5 and which display shelf each product is displayed on. As a result, the burden of creating and preparing the recognition model 441 can be significantly reduced, and the accuracy of product identification by image recognition can be improved.

[0056] In order to identify the product to be picked from among the objects identified from the image of the display shelf, for example, a recognition model 441 for the target product is acquired, and this is applied to each object in sequence to be discriminated by AI, and the object with the highest score can be regarded as the target product. In this embodiment, in order to further improve accuracy, a recognition model 441 for each product displayed on the target display shelf (including other products displayed on the display shelf in addition to the product to be picked) is acquired, and the recognition model 441 for each product is applied to each object in sequence to be discriminated by AI, and the product with the highest score is regarded as the product related to the object. Note that information on other products displayed on the target display shelf can be obtained, for example, from the inventory DB 47.

[0057] Generally, on a shelf, products with similar appearances such as size, shape, pattern, color, etc., such as similar products of the same type or products of the same product but different types or grades, are often displayed side by side. Therefore, in order to further improve the accuracy of identifying products, in addition to each product displayed on the shelf in question, a recognition model 441 related to a product with a similar appearance to these products may be acquired and applied to further improve the accuracy. Note that information on other products with a similar appearance to each product can be set and registered in advance in the product master DB 46, for example.

[0058] In order to perform the process of identifying the product to be picked from each object identified by image processing such as contour extraction, as described above, it is necessary to obtain the recognition model 441 of each product including other products displayed on the target shelf (which may include products similar in appearance to the target product). Regardless of whether the image recognition process for these recognition models 441 is performed by the picking robot 6 or the picking processing unit 44 of the store server 4, including the case of picking multiple products, in this embodiment, when the picking robot 6 moves and approaches the target shelf 5 within a predetermined distance, the recognition model 441 of the product related to the product to be picked from the shelf 5 (other products displayed on the same shelf, products similar in appearance, etc.) is specified and obtained as needed, but is not limited to this. At the first stage of a series of picking processes, all recognition models 441 that will be required later may be specified and obtained in advance.

[0059] For example, when image recognition processing is performed by the picking robot 6, the recognition model 441 once acquired from the store server 4 may be associated with the target product and display shelf 5 and stored as a history. This makes it possible to avoid the burden of acquiring the recognition model 441 that has already been acquired once again from the store server 4 via communication when a product that has already been picked is to be picked again for another order. During unmanned business hours, the arrangement of the products displayed on the display shelves 5 basically does not change, so the recognition model 441 of the products displayed on each display shelf 5 can be reused for that display shelf 5.

[0060] <Data Structure> 8 is a diagram outlining an example of the data configuration of the product master DB 46 in one embodiment of the present invention. The product master DB 46 is a table that holds master information about each product sold in the store 3, and has items such as product ID, product name, type, unit price, weight, whether unmanned sales are possible, recognition model, and similar product ID.

[0061] The product ID field holds identification information such as an ID or number that uniquely identifies each product. The product name, type, and unit price fields hold the name of the target product, name and code value information that identifies the product type (for example, "instant noodles" or "detergent"), and unit price information of the target product. The weight field holds information on the weight per sales unit of the target product. The unmanned sales availability field holds information such as a flag indicating whether the target product is eligible for unmanned sales according to this embodiment. This makes it possible to forcibly set the target product as ineligible for unmanned sales.

[0062] The recognition model item holds information on the recognition model 441 for identifying the target product through image recognition by AI. The recognition model 441 itself may be held, or instruction information such as a path or link to the recognition model 441 managed in a file or on another recording medium may be held. The similar product ID item holds the product IDs of other products (if there are multiple products) that are similar in appearance to the target product.

[0063] 9 is a diagram outlining an example of the data configuration of inventory DB 47 in one embodiment of the present invention. Inventory DB 47 is a table that holds information related to the stock and display status of each product sold in store 3, and has items such as product ID, stock quantity, display quantity, display shelf ID, and display level.

[0064] The product ID item is identification information such as an ID that uniquely identifies each product, and has the same content as the product ID item in the product master DB 46 described above. The stock quantity item holds information on the stock quantity of the target product in the store 3 at the relevant time. The display quantity item holds information on the display quantity of the target product on the display shelves 5 in the store 3 at the relevant time. If the display quantity is zero, the product will be replenished from stock and displayed during manned business hours as long as the stock quantity is not zero, but during unmanned business hours, the product will be treated as out of stock unless there is a means of unmanned replenishment and display using a robot or the like.

[0065] The display shelf ID field holds identification information such as an ID or number that identifies the display shelf 5 on which the target product is actually displayed (for example, "C2" in the example of FIG. 5) when the display quantity value described above is not zero. By referring to this information, the picking robot 6 can move to the target display shelf 5. The display shelf field holds information that identifies the display shelf on which the target product is actually displayed within the display shelf 5 identified by the display shelf ID field described above (for example, "top shelf," "middle shelf," "bottom shelf," "first shelf," "second shelf," "third shelf," etc.) when the display quantity value is not zero. This field makes it possible to know which display shelf the target product is displayed on, as well as which product is displayed on a specific display shelf.

[0066] 10 is a diagram outlining an example of the data configuration of the locker DB 48 in one embodiment of the present invention. The locker DB 48 is a table that holds information such as availability and usage status of the delivery lockers 7 installed in the store 3, and has items such as the locker number, status, status update date and time, and order ID.

[0067] The locker number item holds identification information such as a number or ID that uniquely identifies each delivery locker 7 installed in the store 3. This locker number is assumed to be able to identify the location of the delivery locker 7 (for example, "xxth on xxth row" in the configuration of the example in FIG. 4), but a separate item may be included to hold information for identifying the location of the delivery locker 7.

[0068] The status item holds information such as a character string or a code value indicating the availability of the target delivery locker 7 (for example, "available", "allocated", "delivered", etc.). For an order from a customer, a delivery locker 7 to be used for delivery of the product for that order is allocated from among those for which the item indicates "available". The status update date and time item holds information on the timestamp when the value of the above status item was last updated. For example, if a predetermined time has passed since the status became "delivered", a notification may be sent to the target customer urging them to pick up the product. The order ID item holds information on the ID of the order to which the delivery locker 7 is allocated when the status of the target delivery locker 7 is "allocated" or "delivered". This order ID is the same as the order ID item in the order DB 49 described later.

[0069] 11 is a diagram outlining an example of the data configuration of the order DB 49 in one embodiment of the present invention. The order DB 49 is a table that holds information related to the contents of orders from customers, and has items such as an order ID, a user ID, a product ID, a purchase quantity, a payment amount, a payment status, an order date and time, a corresponding robot ID, a scheduled delivery date and time, and an order status.

[0070] The order ID field holds identification information such as an ID or sequential number that uniquely identifies each order at the target store 3. The user ID field holds identification information such as an ID that uniquely identifies the account of the customer who placed the target order. The product ID and purchase quantity fields hold the product ID and quantity information that specify the product instructed to be purchased in the target order. The product ID field has the same content as the product ID field in the above-mentioned product master DB 46. Note that if the purchase of multiple products is instructed in the target order, these fields are held for each requested product.

[0071] The payment amount and payment status items hold information on the total payment amount for the target order and flag information indicating whether the payment by the customer has been completed. The order date and time item holds information on the timestamp when the target order is accepted by the store server 4. The corresponding robot ID item holds identification information such as an ID or number that specifies the picking robot 6 assigned to pick up the product for the target order and transport it to the delivery locker 7. The scheduled delivery date and time item holds information on the date and time estimated by the picking processing unit 44 of the store server 4, etc., as the date and time when the picking robot 6 will complete transporting the product for the target order to the delivery locker 7 and the customer will be able to receive the product. The order status item holds information such as a character string or code value that indicates the status of the target order (for example, "ordered", "transported", "delivered", etc.).

[0072] <Processing flow> Fig. 12 is a process flow diagram outlining an example of the flow of a product sales process in an embodiment of the present invention. First, in the store server 4, the available product presentation unit 41 refers to the product master DB 46 and the inventory DB 47 to extract products that can be sold (ordered) during unmanned business hours, i.e., products that can be sold unmanned and are actually displayed on the display shelves 5, and presents them to the customer via the customer terminal 2 (S01). In the customer terminal 2, available products are displayed using a Web browser or a dedicated application, and when the customer specifies the desired product, payment method, etc. and instructs purchase, order information is sent to the store server 4. The order reception unit 42 of the store server 4 receives the order information sent from the customer terminal 2 and records it in the order DB 49 (S02).

[0073] When an order is accepted, the picking preparation processing unit 43 refers to the locker DB 48 and allocates a delivery locker 7 in an "empty" state to be used for delivery of the product for the order (S03). If there is no empty delivery locker 7, the customer terminal 2 is notified of this, and the customer may cancel the order if he / she wishes. If the customer does not cancel, the delivery locker 7 may be allocated after waiting until an empty delivery locker 7 becomes available. In this case, the delivery of the product to the delivery locker 7 is delayed, and as a result, the time when the customer can receive the product is delayed. When the delivery locker 7 is allocated, the picking preparation processing unit 43 determines a movement route for the picking robot 6 to sequentially pick up each product related to the order from the display shelf 5, deliver it to the delivery locker 7, and return to the waiting area in preparation for picking (S04).

[0074] Thereafter, the picking processing unit 44 of the store server 4 issues a picking instruction to the picking robot 6 through wireless or wired communication (S05). The picking instruction includes at least information on the product to be picked, information on the delivery locker 7 assigned in step S03, and information on the movement route determined in step S04. After issuing the picking instruction, the transport status of the product is periodically checked (S06) and it is determined whether the transport status has become "transported" (S07). The transport status may be determined, for example, by whether the order status item in the order DB 49 is "transported", or information on the transport status may be obtained by inquiring of the picking robot 6 each time.

[0075] If the transport status is not "transported" in step S07, it is determined that picking is still in progress, and the picking processing unit 44 calculates the expected time required until the status becomes "transported" (S08), and by adding this to the current time, obtains the scheduled time when the status will become "transported", and presents this to the customer via the customer terminal 2 (S09). After that, the process returns to step S07 and is repeated.

[0076] The expected required time can be calculated based on, for example, the time required for the picking robot 6 to move along the movement route (calculated from the distance of the movement route and the movement speed of the picking robot 6), the time required from moving to the target display shelf 5 to picking up the product (calculated from the time required for image recognition processing and the time required for the picking robot 6 to move the arm 63 to the desired position, etc.), and the time required for the picking robot 6 to transport the product from the product transport table 68 to the storage space 71 of the delivery locker 7 (calculated from the rotation speed and size of the conveyor belt 681, etc.).

[0077] If the delivery status becomes "delivered" in step S07, it is determined that the delivery of the product to the delivery locker 7 is completed, and the picking processing unit 44 notifies the customer via the customer terminal 2 that the delivery of the product to the customer is now possible due to the completion of the delivery (S10). If a customer visits the store 3 and attempts to unlock / open the target delivery locker 7 from outside the store, the payment processing unit 45 of the store server 4 refers to the order DB 49 and determines whether the payment status of the target order is "paid" (S11). As described above, the payment may be made online in advance using a credit card or the like, or may be made on the spot at the delivery locker 7 using electronic money or the like.

[0078] If the payment status is not "Paid", the process of step S11 is repeated until it becomes "Paid", at which point the door 72 of the delivery locker 7 is unlocked and opened (S12). When the customer removes the item from the storage space 71 and closes the door 72, the delivery locker 7 notifies the store server 4, and the store server 4 updates the order status item for that order in the order DB 49 to "Paid", completing the series of processes.

[0079] On the other hand, when the picking robot 6 is instructed to pick from the store server 4 in step S05, the picking robot 6 moves from a predetermined waiting location to the target display shelf 5 according to the instructed movement route (S21). At that time, when the picking robot 6 moves close to the target display shelf 5, the picking robot 6 acquires from the store server 4 the recognition models 441 of all products (which may further include products similar in appearance to the target product) displayed on the display shelf where the product to be picked is displayed in the target display shelf 5 (S22). Note that the recognition models 441 required at the stage of moving close to the display shelf 5 in this way may be acquired from the store server 4 each time, or all the recognition models 441 required for all products related to the order may be acquired in advance when the picking instruction is received in step S05.

[0080] After that, when the picking robot 6 moves to the target display shelf 5, it moves the arm 63 to control the position of the depth sensor 67 for the display shelf where the product to be picked is displayed, takes an image, and identifies the display position of the target product on the display shelf by image recognition using AI (S23).Then, it moves the arm 63 to control the position of the suction pad 64 to pick up the product by suction pad 64, and places it on the product transport table 68 (S24).

[0081] More specifically, when the picking robot 6 moves to the front of the target display shelf 5, it determines the position of the depth sensor 67 according to the height and width of the display shelf on which the product to be picked is displayed so that the entire product on the display shelf falls within the imaging range of the depth sensor 67, moves the arm 63 to position the depth sensor 67 at the desired position, and captures an image of the display shelf. Then, based on the captured image, the position of the product to be picked is identified by an image recognition technique such as that shown in the example of Fig. 6 above.

[0082] When the position of the product to be picked is specified, the relative positional relationship between the target product and the suction pad 64 installed at the tip of the arm 63 is calculated based on the position information and the depth information of the position obtained by the depth sensor 67. The control unit 62 of the picking robot 6 generates a movement trajectory for bringing the suction surface of the suction pad 64 into contact with the target product, i.e., for moving the suction pad 64 to a position where it can be picked, from the calculated positional relationship, and moves the arm 63 to move the suction pad 64 according to the generated movement trajectory. Thereafter, the vacuum pump 65 is turned on to cause the target product to be sucked onto the suction pad 64, and in this state, the arm 63 is moved to move the suction pad 64 to a desired position on the product conveying platform 68. Then, the vacuum pump 63 is turned off, and the product is removed from the suction pad 64 and placed on the product conveying platform 68.

[0083] If the order contains multiple types of products, the above steps S21 to S24 are repeated for each product as necessary. If multiple purchases of the same product are requested, the above steps S23 to S24 are repeated as many times as necessary. After picking all the products included in the order, the picking robot 6 moves to the delivery locker 7 assigned to the order in step S03 (S25), and transports the products from the product transport platform 68 to the storage space 71 of the delivery locker 7 by the method shown in the example of FIG. 3 (S26). After that, the robot returns to a predetermined waiting area following the movement route determined in step S04 (S27), and the series of processes ends.

[0084] As described above, according to the product sales system 1 which is one embodiment of the present invention, even if the store 3 is locked during unmanned business hours and customers cannot enter the store 3, the picking robot 6 moves around the unmanned store 3 on behalf of the customer, picks and collects the ordered products from the display shelves 5 in order, and transports them to the delivery locker 7. This makes it possible to sell the products that are actually displayed on the display shelves 5 without allowing customers into the store 3 even during "off-hours," while reducing security and crime prevention risks.

[0085] In addition, for the ordered product, the display position, posture, etc. are identified from the captured image of the display shelf in the target display shelf 5 by AI image recognition, so it is possible to identify the display position of the target product regardless of the actual display situation of the product. In addition, by preparing the recognition model 441 used for image recognition for each product, not for each display shelf, even in a convenience store or the like with many affiliated stores, if the headquarters prepares the recognition model 441 for each product in advance, it is not necessary to individually create and prepare the recognition model 441 according to the actual display situation of the product at each store 3, and the product sales system 1 can be easily deployed to many stores 3.

[0086] In addition, by using a belt conveyor 681 on the product transport platform 68 of the picking robot 6, it becomes possible to easily and smoothly transport products to the delivery locker 7. Furthermore, by providing a belt conveyor 72 in the storage space 71 of the delivery locker 7, it becomes possible to transport products even more smoothly.

[0087] The invention made by the present inventor has been specifically described above based on the embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention. Furthermore, the above embodiment has been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all of the configurations described. Furthermore, it is possible to add, delete, or replace part of the configuration of the above embodiment with other configurations.

[0088] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0089] In addition, in each of the above figures, the control lines and information lines are shown as those considered necessary for explanation, and do not necessarily show all the control lines and information lines in the actual implementation. In reality, it can be considered that almost all the components are connected to each other. [Industrial Applicability]

[0090] The present invention can be used in a product sales system that sells products in an unmanned brick-and-mortar store. [Explanation of symbols]

[0091] 1...Product sales system, 2...Customer terminal, 3...Store, 4...Store server, 5...Display shelf, 6...Picking robot, 7...Delivery locker, 41...Available product presentation unit, 42...Order reception unit, 43...Picking preparation processing unit, 44...Picking processing unit, 45...Payment processing unit, 46...Product master DB, 47...Inventory DB, 48...Locker DB, 49...Order DB, 61...automatic guided vehicle, 62...control unit, 63...arm, 64...suction pad, 65...vacuum pump, 66...barometer, 67...depth sensor, 68...product transport platform, 71...storage space, 72...conveyor belt, 441…Recognition model, 681...belt conveyor, 682...protective wall, 683...transport gate

Claims

1. A product sales system that sells products displayed in a store to customers, A store server; A picking robot capable of moving within the store, The store server includes: an order receiving unit that receives product orders; A picking processing unit that instructs the picking robot to pick up the product related to the order and transport it to a predetermined delivery location, The picking robot includes: An arm capable of moving a tip portion to a desired position; A suction pad for picking up a product is provided at the tip of the arm; A depth sensor having an imaging function and installed near the tip of the arm; a product conveying table on which the product picked by the suction pad is placed; having a product sales system which detects one or more objects from an image of the ordered product displayed, taken by the depth sensor, by image processing; identifies the position of each object; and based on the position information of the ordered product and the depth information obtained by the depth sensor, determines the relative positional relationship between the ordered product and the suction pad and the posture of the ordered product; moves the arm to move the suction pad to a position where the ordered product can be picked, adsorbs the ordered product to the suction pad, moves the arm to move the ordered product to a predetermined position on the product transport table, and removes the ordered product from the suction pad and places it on the product transport table.

2. The product sales system according to claim 1, The picking robot includes: A product sales system which identifies which product each of the objects related to the order is by sequentially applying a recognition model set for each product to each of the objects detected by image processing.

3. The product sales system according to claim 1, The picking robot includes: A product sales system that identifies which object the ordered product is by image processing that sequentially applies recognition models set for the ordered product and products similar in appearance to the ordered product to each of the objects detected by image processing.

4. The product sales system according to claim 1, The picking robot includes: A product sales system that obtains a recognition model set for recognizing the ordered product by image processing from the store server when the ordered product moves a predetermined distance from the display shelf on which it is displayed.

5. The product sales system according to claim 1, The store server further a sales available product presentation unit that presents sales available products to the customer based on inventory information and / or master information of products displayed in the store; a picking preparation processing unit that identifies a display shelf on which the product related to the order accepted by the order acceptance unit is displayed from among available products, and determines a route for the picking robot to move to the display shelf, pick the product, transport it to the specified delivery location, and return to a waiting location.

6. The product sales system according to claim 1, The picking processing unit includes: When the picking robot has completed transportation of the ordered product to the specified delivery location, the product sales system notifies the customer of this completion.

7. The product sales system according to claim 1, A product sales system in which the loading surface of the product transport table of the picking robot and the bottom surface of the designated delivery location are each made of a conveyor belt, and the product related to the order is transported from the product transport table to a designated position at the designated delivery location by operating each of the conveyor belts in conjunction with one another.

8. A product sales system that sells products displayed in a store to customers, A store server; A picking robot capable of moving within the store, The store server includes: an order receiving unit that receives product orders; A picking processing unit that instructs the picking robot to pick up the product related to the order and transport it to a predetermined delivery location, The picking robot includes: An arm capable of moving a tip portion to a desired position; A suction pad for picking up a product is provided at the tip of the arm; A depth sensor installed near the tip of the arm; a product conveying table on which the product picked by the suction pad is placed; having A product sales system which detects and identifies the position of the product related to the order based on the image captured by the depth sensor, and determines the relative positional relationship between the product related to the order and the suction pad and the posture of the product related to the order based on the position information of the product related to the order and the depth information obtained by the depth sensor, moves the arm to move the suction pad to a position where the product related to the order can be picked, adsorbs the product related to the order to the suction pad, moves the arm to move the product related to the order to a predetermined position on the product transport table, and removes the product related to the order from the suction pad and places it on the product transport table.

Images