Product identification system and product identification method

The integration of a product image acquisition device and machine learning within a product replenishment robot enables efficient and automated product identification and restocking on inventory shelves, addressing the inefficiencies of manual replenishment and enhancing operational efficiency.

WO2024237240A1PCT designated stage expired Publication Date: 2024-11-21TELEXISTENCE INC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017682
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-17
Filing Date
2024-05-13
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Current product replenishment systems in stores, particularly in convenience stores, require manual intervention by store clerks to identify and restock products on inventory shelves, which is time-consuming and labor-intensive, especially in low-temperature environments, and limits the operational efficiency of the product movement system.

Method used

A product identification system integrated with a product replenishment robot that uses a product image acquisition device with a rotating table, elevating mechanism, and image acquisition unit, along with a control device that employs machine learning to identify products based on captured images, allowing the robot to autonomously recognize and restock products on inventory shelves.

Benefits of technology

The system enables efficient and automated product replenishment on inventory shelves, reducing human error and labor costs, while improving the operational rate of the product movement system by allowing the robot to accurately identify and restock products without manual intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017682_21112024_PF_FP_ABST
    Figure JP2024017682_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided is a product identification system for identifying the type of a product to be gripped and moved by a product replenishment robot that moves products. The product identification system includes: a product image acquisition apparatus 200 for acquiring an image of a product; and a control apparatus for identifying the type of the product on the basis of the image of the product acquired by the product image acquisition apparatus. The product image acquisition apparatus 200 includes: a rotary base 210 provided with a rotary table 212 for rotating the product placed thereon; a lifting mechanism 230 for lifting the rotary base 210; and an image acquisition unit 260 for acquiring an image of the product. The control apparatus includes a control unit and a storage unit. The storage unit stores image data of various products, or a trained model generated using image data through execution of machine learning by a learning device. The control unit identifies the type of the product related to the image acquired by the image acquisition part 260, on the basis of the image data or the trained model.
Need to check novelty before this filing date? Find Prior Art

Description

Product identification system and product identification method

[0001] The present disclosure relates to a product identification system and a product identification method.

[0002] Patent Document 1 discloses a product moving system including a product moving device that moves products placed on inventory shelves in a store such as a convenience store to a display shelf different from the inventory shelf. This product moving device includes a gripping unit that grips the product, an arm unit that moves the gripping unit to the inventory shelf and the display shelf, a first imaging unit that photographs the display shelf, a second imaging unit that photographs the inventory shelf, and a control unit that controls the operation of the arm unit, gripping unit, and first and second imaging units.

[0003] The control unit of the product moving device identifies replenishment target products, which are products that need to be replenished on the display shelves, based on the display shelf image data generated by the first imaging unit by photographing the display shelves, and identifies the positions of inventory products in the inventory shelf image data that correspond to the replenishment target products, based on the inventory shelf image data generated by the second imaging unit by photographing the inventory shelves. The control unit further causes the gripper to grip the replenishment target products based on the inventory shelf image data, and after the gripper has gripped the replenishment target products, moves the gripper toward the display shelf and places the replenishment target products in product placement positions on the display shelves based on the display shelf image data.

[0004] In this way, the product moving device disclosed in Patent Document 1 performs the operation of moving products from inventory shelves to display shelves to replenish them.

[0005] International Publication No. 2023 / 022214

[0006] According to the system disclosed in Patent Document 1, it is possible to replenish products on inventory shelves by moving products from inventory shelves to display shelves, but the replenishment of products on inventory shelves still requires the store clerk to do so.

[0007] Each shelf of a stock shelf has multiple rows of products separated by partitions, and each row is stocked with a specific product. The system recognizes which products are stocked in each row on each shelf of the stock shelf. For example, a label listing the product name and other information to be replenished is placed in front of each row on each shelf of the stock shelf, and store clerks repeatedly check the labels and replenish the rows with the products they need. This task of replenishing products on stock shelves while searching for the appropriate row takes a lot of time and effort. Furthermore, stock shelves that handle beverages at convenience stores and other stores are located in back-of-house areas where low temperatures are controlled, and working in such a low-temperature environment places a great physical strain on store clerks.

[0008] Furthermore, while a store clerk is replenishing products on the shelves, the product moving device must be moved to a position where it will not interfere with the clerk's work and must be stopped for safety reasons, so the product moving device cannot be operated during that time. Therefore, the replenishing of products on the shelves by a store clerk is one factor that reduces the operating rate of the product movement system.

[0009] An object of the present disclosure is to provide a means for enabling a product replenishment robot to replenish products on a stock shelf after identifying the products to be replenished on the stock shelf.

[0010] According to one aspect of the present disclosure, there is provided a product identification system that identifies the type of product grasped and moved by a product replenishment robot that moves products. The product identification system includes a product image capture device mounted on the product replenishment robot and configured to capture product images, and a control device that controls the operation of the product image capture device and identifies the type of product based on the product images captured by the product image capture device. The product image capture device includes a turntable equipped with a rotary table that rotates products placed on it, an elevation mechanism that raises and lowers the turntable, and an image capture unit that captures product images. The control device includes a control unit and a memory unit, and the memory unit stores image data of various products or a trained model generated by a learner using the image data to perform machine learning, and the control unit is configured to identify the type of product associated with the image captured by the image capture unit based on the image data or the trained model.

[0011] Other features and advantages of the present disclosure can be seen from the following description and the accompanying drawings, which are given by way of example and are not exhaustive.

[0012] According to the present disclosure, a means is provided that allows an item replenishment robot to replenish the items on the inventory shelves after identifying the items to be replenished on the inventory shelves.

[0013] 1 is a diagram showing the overall configuration of a product replenishment system according to an embodiment of the present disclosure; FIG. 1 is a plan view schematically showing the arrangement of shelves in a store and a product replenishment robot arranged in the store; FIG. 1(a) is a view of the display shelf as seen from the front side, and FIG. 1(b) is a view of the display shelf as seen from the rear side; FIG. 1(b) is a schematic diagram for explaining the functions of an inventory shelf; FIG. 1 is a side view schematically showing the configuration of a product replenishment robot; FIG. 2 is a perspective view showing the peripheral structure of the tip of the arm of the product replenishment robot; FIG. 3 is a block diagram showing the configuration of the product replenishment robot; FIG. 4 is an image of the display shelf captured by a first camera (left side) of the product replenishment robot; FIG. 5 is a flowchart showing the product replenishment operation by the product replenishment robot; FIG. 6 is a perspective view showing the overall configuration of a product image acquisition device; FIG. 7 is a diagram showing details of the configuration of each part of the product image acquisition device; FIG. 8 is a block diagram showing the configuration of a management device; FIG. 9 is a block diagram showing the configuration of a terminal device; FIG. 10 is a diagram showing an example of a display screen for placement layout information displayed on the display operation unit of the terminal device; FIG. 11 is a flowchart for explaining an example operation of an image acquisition device and a control device that function as a product identification system in this embodiment; FIG. 12 is a diagram showing a series of operations for acquiring product images using the product image acquisition device; FIG. 13 is a diagram showing an example in which a turntable is raised to various heights depending on the size of the product.

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0015] <Overall configuration of product replenishment system>

[0016] First, the overall configuration of a product replenishment system according to an embodiment of the present disclosure will be described. Fig. 1 is a diagram showing the overall configuration of a product replenishment system according to an embodiment of the present disclosure.

[0017] As shown in FIG. 1, a merchandise replenishment system 1 in one embodiment of the present disclosure includes a merchandise replenishment robot 100 that moves merchandise, a merchandise image acquisition device 200 provided on the merchandise replenishment robot 100, a management device 300 that manages the schedule of merchandise replenishment work performed by the merchandise replenishment robot 100, and a terminal device 400 that is operated by a store clerk or the like involved in merchandise replenishment.

[0018] Merchandise replenishment robot 100 is placed in a store such as a convenience store, and replenishes merchandise onto inventory shelves SL20 and from inventory shelves SL20 to display shelves SL10. A merchandise image acquisition device 200 provided on merchandise replenishment robot 100 captures images of merchandise placed on merchandise image acquisition device 200 by merchandise replenishment robot 100. Based on the captured merchandise images, a control device 150 of merchandise replenishment robot 100 identifies the type of merchandise placed on merchandise image acquisition device 200.

[0019] The management device 300 transmits to the terminal device 400 product replenishment information generated by scheduling the types and quantities of products to be replenished on the inventory shelves SL20 and the timing of replenishment. The product replenishment information includes stacking layout information for placing various products on the product replenishment shelves SL30 for transporting the products to be replenished to the inventory shelves SL20. The management device 300 also causes the product replenishment robot 100 to perform an operation to replenish products from the product replenishment shelves SL30 to the inventory shelves SL20 based on the product replenishment schedule.

[0020] The terminal device 400 displays the product replenishment information transmitted from the management device 300 on the display screen of the terminal device 400, and presents the product replenishment information, such as the timing for replenishing products on the inventory shelf SL20 and the type and number of products to be replenished on the inventory shelf SL20, to the store clerk using the terminal device 400. The store clerk places designated quantities of various designated products in predetermined positions on the product replenishment shelf SL30 in accordance with the loading layout information displayed on the screen of the terminal device 400.

[0021] The store clerk moves the replenishment shelf SL30 on which the replenishment items are placed to a position accessible by the replenishment robot 100 at the timing indicated in the replenishment information. The replenishment robot 100 moves the items on the replenishment shelf SL30 to predetermined positions on the inventory shelf SL20, replenishing the items on the inventory shelf SL20. At this time, the replenishment robot 100 may, as necessary, place the items on the replenishment image acquisition device 200 to identify the type of item to be replenished. Between replenishment operations on the inventory shelf SL20, or after the replenishment operations on the inventory shelf SL20 have been completed, the replenishment robot 100 moves the items from the inventory shelf SL20 to the display shelf SL10, replenishing the items on the display shelf.

[0022] In this way, the product replenishment system 1 of the present disclosure replenishes products from the product replenishment shelf SL30 to the inventory shelf SL20 in accordance with product replenishment information that indicates the type and quantity of products to be replenished on the inventory shelf SL20 and the timing of replenishment, and further replenishes products from the inventory shelf SL20 to the display shelf SL10.

[0023] Next, the layout of the store will be described. Fig. 2 is a plan view showing the shelves and product replenishment robots arranged in the store. Fig. 3(a) is a view of the display shelves from the front side, and Fig. 3(b) is a view of the display shelves from the rear side. Fig. 4 is a schematic diagram for explaining the functions of the inventory shelves.

[0024] As shown in Figure 2, the store is divided into an in-store space SH1 and a backyard space SH2. The in-store space SH1 is a space where customers select and purchase products T. The backyard space SH2 is a space where inventory of products T is stored. The store is equipped with display shelves SL10, inventory shelves SL20, product replenishment shelves SL30, a product replenishment robot 100, and a product image acquisition device 200 on the product replenishment robot 100. A track consisting of two rails R is laid on the floor between the display shelves SL10 and the inventory shelves SL20 in the store, and the product replenishment robot 100 is configured to move on these rails R in the left-right direction as shown in the figure.

[0025] The product T is, for example, a beverage container or a cylindrical beverage can having a main body Ta and a cap Tb attached to the upper end of the main body Ta. The product T may be made of various materials, such as PET, glass, aluminum, steel, or other metals.

[0026] As shown in Figures 3(a) and (b), the display shelf SL10 has a plurality of shelf boards SL11 (also referred to as display shelf tiers). A plurality of types of products T are arranged on the shelf boards SL11. For example, products T of the same type are arranged in two or three rows. The products T may also be arranged in only one row. A plurality of partition boards SL12 are provided on the upper surface of each shelf board SL11 to separate the products T in each row. Figure 3 shows an example in which products T of the same type are arranged in two rows, and in this example, a partition board SL12 is provided every two rows of products T. The arrangement of the partition boards SL12 is not limited to this, and the partition boards SL12 can be arranged at intervals of one row or two or more rows.

[0027] The front of the display shelf SL10 faces the in-store space SH1, allowing customers to pick up products T from the front side of the display shelf SL10. The shelf board SL11 is inclined so that the front side of the display shelf SL10 is relatively lower than the rear side. As a result, when a customer picks up a product T, the other products T lined up behind that product T slide along the shelf board SL11 and move to the front side.

[0028] The rear of the display shelf SL10 faces the backyard space SH2, and a store clerk or a product replenishment robot 100 replenishes products T onto the display shelf SL10 from the rear side of the display shelf SL10. Although not shown, doors may be provided on the front and rear of the display shelf SL10. Although only one display shelf SL10 is shown in Fig. 3 for the sake of simplicity, multiple display shelves SL10 may be arranged in the store.

[0029] The inventory shelf SL20 is arranged in the backyard space SH2 facing the display shelf SL10, with the front surface of the inventory shelf SL20 facing the rear surface of the display shelf SL10. Similar to the display shelf 10, the inventory shelf SL20 has multiple shelves (tiers) arranged in the vertical direction. Products to be replenished on the display shelf SL10 are arranged on the shelves of the inventory shelf SL20. A store clerk or a product replenishment robot 100 replenishes products T onto the inventory shelf SL20 from the front side of the inventory shelf SL20.

[0030] The product replenishment shelf SL30 has multiple shelves (tiers) arranged in the vertical direction. The bottom of the product replenishment shelf SL30 is provided with multiple casters that allow the product replenishment shelf SL30 to be moved in any direction. The product replenishment shelf SL30 may also be affixed with a label bearing characters, a two-dimensional code, or the like as identification information for identifying the product replenishment shelf SL30. The identification information written on the label can be read using a camera provided in the terminal device 400.

[0031] The configuration and function of the inventory shelf SL20 in this embodiment will now be described with reference to FIG. 4 . Each shelf 21 of the inventory shelf SL20 is configured to be able to switch between a first position, in which the front edge (right side in the figure) is higher than the rear edge (left side in the figure), as shown by the solid line in FIG. 4 , and a second position, in which the front edge is lower than the rear edge, as shown by the dotted line in FIG. 4 . The inventory shelf SL20 is equipped with a drive mechanism (not shown) for switching each shelf SL21 between the first position and the second position. The drive mechanism includes, for example, an electric motor as a drive source. The drive mechanism switches the position of each shelf SL21 between the first position and the second position in response to a control signal transmitted, for example, from a communication unit 195 (see FIG. 7 ) equipped in the product replenishment robot 100. The drive mechanism may be provided individually for each shelf SL21 of the inventory shelf SL20 so as to drive each shelf SL21 independently, or may be provided as a single mechanism so as to drive multiple shelves SL21 together via a connecting mechanism or the like.

[0032] When the product replenishment robot 100 replenishes products on each shelf SL21 of the inventory shelf SL20, the shelf SL21 is placed in a first position in which the front edge is higher than the rear edge. In the first position, the shelf SL21 is inclined downward from the front to the rear. As a result, products placed on the front side of a specific product row on the shelf SL21 for replenishment slide down the inclination from the front to the rear toward the rear of the shelf SL21. Therefore, by repeatedly placing products on the front side of the shelf SL21, products can be sequentially replenished in the product row on the shelf SL21.

[0033] On the other hand, when the product replenishment robot 100 moves products from the inventory shelf SL20 to the display shelf SL10 to replenish the products on the display shelf SL10, the shelf SL21 of the inventory shelf SL20 is placed in a second position in which the front edge is lower than the rear edge. In the second position, the shelf SL21 slopes downward from the rear to the front. As a result, products placed in each product row on the shelf SL21 slide down the slope from the rear to the front toward the front of the shelf SL21. Each time the product replenishment robot 100 removes the product located at the front of the front side of the inventory shelf SL20 for replenishment, the products behind it slide down sequentially toward the front of the shelf SL21. This allows the product replenishment robot 100 to remove the product located at the front of the front side of the inventory shelf SL20 and move it to the display shelf SL10 for replenishment. <Configuration and Operation of the Product Replenishment Robot 100>

[0034] The product replenishment robot 100 will be described below with reference to Figures 2 and 5 to 7. Figure 5 is a side view showing a schematic configuration of the product replenishment robot 100. Figure 6 is a perspective view showing the peripheral structure of the tip of the arm of the product replenishment robot 100. Figure 7 is a block diagram showing the configuration of the product replenishment robot 100.

[0035] The product replenishment robot 100 has a gripper 10, an arm 20, a contact detection sensor 30, first cameras 50R and 50L, a second camera 60, a third camera 70, a horizontal movement mechanism 80, an elevator mechanism 90, and a control device 150. The product replenishment robot 100 is a robot that moves in the space between a display shelf SL10 and an inventory shelf SL20. The product replenishment robot 100 performs the following operations: gripping a product T on a product replenishment shelf SL30 with the gripper 10 and moving it to a storage position of that product on the inventory shelf SL20 (a lane in which that product T is stored); and gripping a product T in the inventory shelf SL20 with the gripper 10 and moving the gripped product T to a display position of that product on the display shelf SL10 (a lane in which that product T is displayed).

[0036] As shown in FIG. 6 , the gripping portion 10 has a pair of fingers 11 a, 11 b for gripping an object. The pair of fingers 11 a, 11 b are shaped to be able to grip the vicinity of the cap portion Tb of a container of the product T, such as a PET bottled beverage, and to grip the outer periphery of the product T, such as a canned beverage. The gripping portion 10 may be configured with a pair of fingers 11 a, 11 b, or may have an attraction structure for attracting and holding the object, or a structure for holding the object using adhesive force, magnetic force, or the like. It is preferable that a flexible holding member (not shown) is provided on the gripping surface of each finger 11 a, 11 b to better hold the product T when gripped. The holding member may be formed of, for example, a rubber sheet material, a urethane resin sheet material, or the like.

[0037] The arm unit 20 has a plurality of link members 21, 22, and 23. The plurality of link members 21, 22, and 23 constitute an articulated robot arm. As an example, the articulated robot arm may be a six-axis arm having degrees of freedom in linear directions along the X-axis, Y-axis, and Z-axis, and degrees of freedom around the X-axis, Y-axis, and Z-axis. The articulated robot arm may also have any other mechanism, such as a Cartesian coordinate system robot arm, a polar coordinate system robot arm, a cylindrical coordinate system robot arm, or a SCARA robot arm. One end of the arm unit 20 is fixed to the lifting mechanism 90. A gripper 10 is provided at the tip of the arm unit 20. The operation of the arm unit 20 is controlled by a control device 150.

[0038] The arm unit 20 can move the gripper 10 toward the inventory shelf SL20 or toward the display shelf SL10 by moving the respective link members 21, 22, and 23. The orientation of the arm unit 20 is not fixed to a specific direction, but for convenience of explanation, the direction in which the respective link members 21, 22, and 23 are extended will be referred to as the extension direction Ax of the arm unit 20 (see FIG. 6 ). The arm unit 20 grips the product T by moving the gripper 10 forward toward the product T. The extension direction Ax of the arm unit 20 corresponds to the forward direction of the gripper 10 in the gripping operation.

[0039] The contact detection sensor 30 is a sensor that detects that the gripping unit 10 has come into contact with the product T when gripping the product T, and that when placing the product T gripped by the gripping unit 10 on a shelf SL21 of the inventory shelf SL20 or a shelf SL11 of the display shelf SL10, the product T gripped by the gripping unit 10, the gripping unit 10, or the arm unit 20 has come into contact with an obstacle such as a wall or a support of the shelf SL10 or SL20. The contact detection sensor 30 may be, for example, a torque sensor, an acceleration sensor, an inertial measurement unit (IMU), a motor input current sensor, or the like.

[0040] The torque sensor may be, for example, a strain gauge that detects torque generated at the axis of each joint of the arm unit 20. The acceleration sensor may be any of various types of acceleration sensors, such as capacitance type or piezo-resistance type, that are installed in the gripping unit 10 or the arm unit 20.

[0041] An inertial measurement unit (IMU) is a device that detects three-dimensional inertial motion (translational and rotational motion in three orthogonal axes). It includes an acceleration sensor that detects translational motion and an angular velocity (gyro) sensor that detects rotational motion. The gyro sensor detects angular velocity, allowing the angle or angular change of an object to be acquired. For example, if a gear or a gear and a toothed belt are used as a drive transmission mechanism for the wrist portion of the gripper 10, redundancy in the movement of the wrist portion of the gripper 10 can occur due to play between the gears or stretching of the toothed belt. Therefore, for example, during the operation of placing a product T gripped by the gripper 10 on a shelf SL11 or SL21 of a shelf SL10 or SL20, if the arm 20 is further operated with the product T in contact with the shelf SL11 or SL21, applying additional force to the gripper 10, some displacement occurs in the wrist portion of the gripper 10, causing a change in the attitude (i.e., angle) of the gripper 10. Therefore, by detecting such angle changes that may occur in the gripping portion 10 during the operation of placing the product T on the shelf SL11, SL21 using an IMU installed in the gripping portion 10, it is possible to detect that the product T has come into contact with the shelf SL11, SL21.

[0042] Furthermore, in this embodiment, by using an inertial measurement unit (IMU) as the contact detection sensor 30, it is possible to detect the relative height position of the PET bottled beverage to be grasped and the gripping portion 10 by detecting the pitch angle that occurs in the gripping portion 10 when the height of the gripping portion 10 is lowered and the gripping portion 10 is brought into contact with the vicinity of the neck of the PET bottled beverage in the series of actions when the gripping portion 10 grasps the cap portion Tb of the PET bottled beverage of the product T. Then, by adjusting the height of the gripping portion 10 from the state in which the gripping portion 10 is in contact with the vicinity of the neck of the PET bottled beverage, it becomes possible for the gripping portion 10 to grasp the cap portion Tb of the PET bottled beverage at an appropriate height position.

[0043] When a servo motor or the like is used to drive each joint of the arm unit 20, when an external force occurs that causes an angle deviation from the angle at which the gripping posture is maintained, the servo motor operates to zero the angle deviation in order to maintain the original angle. When the arm unit 20 is further moved with the product T gripped by the gripper 10 in contact with the shelf SL11 or SL21 of the shelf SL10 or SL20, a current is input to the servo motor to drive the servo motor against the load. Therefore, by detecting the current input to the servo motor for such operation with a motor input current sensor, it is possible to detect that the product T is in contact with the shelf SL11 or SL21. The motor input current sensor can be configured, for example, as the control unit 151 (see FIG. 7 ), which will be described later.

[0044] The two first cameras 50R, 50L are disposed on the left and right sides of the arm unit 20, respectively. The first camera 50L, attached to the first side surface 23a, which is the left side of the arm unit 20, faces in a first direction A1, which is perpendicular to the extension direction Ax of the arm unit 20 (see FIG. 6 ). The first camera 50L is mainly used to photograph the display shelf SL10. The first camera 50R, attached to the second side surface 23b, which is the right side of the arm unit 20 and parallel to the first side surface 23a, opposite the first side surface 23a, faces in a second direction A2, which is opposite to the first direction A1. The first camera 50R is mainly used to photograph the inventory shelf SL20. By arranging the first cameras 50R and 50L facing in opposite directions in this manner, the first cameras 50R and 50L can simultaneously photograph the display shelf SL10 and the inventory shelf SL20, respectively, while keeping the arm section 20 in the same position.

[0045] The performance of the first cameras 50R, 50L may be the same or different, but for simplicity, an example in which both cameras have the same performance will be described below. However, since the purpose and conditions for photographing the display shelf SL10 are different from the purpose and conditions for photographing the inventory shelf SL20, it is of course possible to use cameras with different performance to suit the respective purposes and conditions.

[0046] The first cameras 50R and 50L may include, for example, an imaging element that generates an image (e.g., an RGB image) in which pixels are arranged two-dimensionally, and a depth sensor that is a distance detection device that generates distance data. The depth sensor is not limited to a specific type as long as it can acquire distance data to an object. For example, a stereo lens type or a LiDAR (Light Detection and Ranging) type can be used. The depth sensor may generate, for example, a depth image. In another aspect of the present invention, one or both of the first cameras 50R and 50L may acquire distance data using, for example, an ultrasonic element.

[0047] Note that the first camera 50L is directed in a first direction A1, which means that the imaging direction of the imaging element and depth sensor of the first camera 50L is direction A1. Similarly, the first camera 50R is directed in a second direction A2, which means that the imaging direction of the imaging element and depth sensor of the first camera 50R is direction A2. The directions A1 and A2 do not necessarily have to be 180° opposite, but may be directions that allow images of the product display shelf SL10 and the inventory shelf SL20 to be captured.

[0048] One or both of the first cameras 50R, 50L may be provided on the grip portion 10. The first cameras 50R, 50L do not necessarily have to be provided on the same member; for example, the first camera 50R may be attached to one of the link members 21 to 23, and the first camera 50L may be attached to another of the link members 21 to 23. However, when the first cameras 50R, 50L are provided on the same member, a common coordinate system is used, which has the advantage of simplifying the calculations for image processing compared to when the cameras 50R, 50L are provided on separate link members.

[0049] The second camera 60 is used to capture an image of the gripper 10 gripping the product T, and the relative positional relationship between the product T gripped by the gripper 10 and the shelf boards SL11, SL21 of the shelves SL10, SL20. Similar to the first cameras 50R, 50L, the second camera 60 may have, for example, an imaging element that generates an image in which pixels are arranged two-dimensionally (an RGB image, for example), and a depth sensor that generates distance data.

[0050] As an example, the second camera 60 is installed in a position close to the gripping unit 10, below the link member 23 that is closest to the gripping unit 10 among the link members 21 to 23 of the arm 20, with the imaging direction of its imaging element and depth sensor directed directly below the link member 23 and the gripping unit 10 (the −z direction in FIGS. 5 and 6) or downward and forward (slightly further toward the +x direction than the −z direction in FIGS. 5 and 6). This enables the second camera 60 to capture images of at least the lower portion of the product T gripped by the gripping unit 10 and the shelves SL11 and SL21 located in front of the gripping unit 10.

[0051] The third camera 70 is a camera that changes direction in response to operation by, for example, a remote operator to photograph a predetermined object. The third camera 70 is attached to a part of the lifting mechanism 90, for example. The third camera 70 is capable of moving horizontally and vertically in the space between the display shelf SL10 and the inventory shelf SL20 in accordance with the operation of the horizontal movement mechanism 80 and the lifting mechanism 90. The part of the lifting mechanism 90 to which the third camera 70 is attached is rotatable about a support column 95, and the third camera 70 is configured to rotate left and right about the support column 95 in accordance with the rotation of that part, so that it can photograph the display shelf SL10 or the inventory shelf SL20 as needed.

[0052] The third camera 70 may be, for example, a stereo lens type. Although not limited thereto, the third camera 70 may have a wider angle of view than the first cameras 50R and 50L and the second camera 60.

[0053] The horizontal movement mechanism 80 has a base plate 81 and a drive mechanism (not shown). The base plate 81 supports the lifting mechanism 90 and allows it to slide along rails (not shown) laid between the display shelves SL10 and the inventory shelves SL20 in the store. The drive mechanism (not shown) includes a motor, rollers, etc., and operates based on control signals from the control device 150 (see FIG. 7 ) to move the lifting mechanism 90 to a predetermined position along the rails. The lifting mechanism 90 has a support column 95, a first lifting mechanism 91, and a second lifting mechanism 92. The support column 95 is fixed to the base plate 81 and extends vertically.

[0054] The first lifting mechanism 91 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., and operates based on a control signal from a control device 150 (see FIG. 7). By operating the drive mechanism (not shown), the first lifting mechanism 91 moves up and down in the vertical direction along a support column 95. An upper portion of the first lifting mechanism 91, to which the third camera 70 is attached, is configured to be rotated left and right around the support column 95.

[0055] The second lifting mechanism 92 is held by the first lifting mechanism 91. One end of the arm unit 20 is attached to the second lifting mechanism 92. The second lifting mechanism 92 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., and operates based on a control signal from the control device 150 (see FIG. 7). By operating the drive mechanism (not shown), the second lifting mechanism 92 also moves up and down in the vertical direction.

[0056] When grasping a product T that is located at a predetermined height, the lifting mechanism 90 uses the first lifting mechanism 91 to move the arm unit 20 and the gripping unit 10 to a height near where the product T can be grasped, and then uses the second lifting mechanism 92 to fine-tune the height of the arm unit 20 and the gripping unit 10.

[0057] In this embodiment, the first lifting mechanism 91 and the second lifting mechanism 92 are provided as the lifting mechanism, but in another aspect of the present invention, a configuration in which only one lifting mechanism is provided may be used.

[0058] 7, the control device 150 includes a control unit 151, a storage unit 160, an input unit 191, an output unit 193, and a communication unit 195. Although the control device 150 is depicted as a single element in FIG. 7, the control device 150 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0059] The input unit 191 is a device for receiving input from an operator. The input unit 191 may be configured with devices for inputting to a computer, such as a keyboard, a mouse, or a touch panel. The input unit 191 may also have a voice input device, such as a microphone. The input unit 191 may also have a gesture input device that recognizes and identifies the operator's movements through image recognition.

[0060] The output unit 193 is used by the product replenishment robot 100 to output an alert to a store clerk or the like, and is configured, for example, by one or a combination of a speaker, a display, a light-emitting device, and a vibration device. The communication unit 195 has the function of receiving data from an external device and the function of transmitting data to an external device. If the product replenishment robot 100 is configured to be remotely operable, the communication unit 195 receives input from the operator via the operation unit of an external device (not shown), and the control device 150 causes the product replenishment robot 100 to perform a predetermined operation based on the input. Note that communication between the operation unit of the external device and the communication unit 195 may be wired or wireless.

[0061] When product replenishment robot 100 is configured to be remotely operable, operation unit 191 may be a device worn by the operator. This device includes a display device (not shown) and an operation device (not shown). The display device may be, for example, a head-mounted display (HMD) having a display visible to the operator. The operation device may include, for example, one or more input sensors capable of detecting the movement of a body part (e.g., a hand or arm) of the operator.

[0062] Storage unit 160 includes temporary or non-temporary storage media such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). Storage unit 160 stores computer programs executed by control unit 151, trained models (described below), and the like. The computer programs stored in storage unit 160 include instructions for implementing a method for controlling commodity replenishment robot 100 by control unit 151 (described below with reference to FIG. 9 and the like).

[0063] The storage unit 160 has an acquired data storage unit 160a and a reference data storage unit 160b. The acquired data storage unit 160a stores, for example, image data captured by the cameras 50R, 50L, 60, and 70. The reference data storage unit 160b stores various data necessary for the operation of the product replenishment robot 100. This various data includes, for example, data on each shelf SL10 to SL30 (shape data, position data, lane coordinate data, etc.) and data on the product T (shape data, position data, surrounding image data, etc.).

[0064] The control unit 151 is configured with, for example, one or more central processing units (CPUs). The control unit 151 executes computer programs stored in the storage unit 160 to function as an operation control unit 152, an imaging control unit 153, and an image data processing unit 155.

[0065] The operation control unit 152 generates control signals for operating the gripping unit 10, the arm unit 20, the horizontal movement mechanism 80, the lifting mechanism 90, and each unit of the control device 150. The operation control unit 152 generates the control signals by referring to input signals from the operation unit 191 and various data stored in the storage unit 160. The control signals may be generated using the processing results of the image data processing unit 155. The operation control unit 152 also transmits and receives data via the communication unit 195, and performs predetermined output via the output unit 193.

[0066] The imaging control unit 153 controls the operation of each of the cameras 50R, 50L, 60, and 70. The imaging timing of each of the cameras 50R, 50L, 60, and 70 may be determined using data stored in advance in the reference data storage unit 160b, for example.

[0067] The image data processing unit 155 performs various information processing using the captured image data and distance data (depth data) captured by each of the cameras 50R, 50L, 60, and 70. As an example, the image data processing unit 155 performs image analysis of the image data captured by the first camera 50R to identify products on the inventory shelf SL20 and the product replenishment shelf SL30. The image data processing unit 155 has a replenishment possibility determination unit 155a and a grasping target identification unit 155b. The image data processing unit 155 also performs various information processing using captured image data captured by an image acquisition unit 260 of the product image acquisition device 200 (described later). As an example, the image data processing unit 155 performs processing to identify the type of product using image data of the outer periphery of the product captured by the image acquisition unit 260 (an image including a product label, barcode, etc. surrounding the product).

[0068] The replenishment possibility determination unit 155a determines, for example, whether a space Sp (see FIG. 8 ) exists behind the last product T lined up on the display shelf SL10 and the inventory shelf SL20, where an additional product can be placed, based on at least one of the image data captured by the first camera 50L and the distance data. FIG. 8 shows an image of the display shelf SL10 captured by the first camera 50L of the product replenishment robot 100. If a space Sp exists, this product T needs to be replenished. Therefore, if a space Sp exists, the replenishment possibility determination unit 155a sends a notification to the operation control unit 151 to notify the operation control unit 151 that the product T should be replenished on the shelf SL11 below the space Sp. Upon receiving this notification, the operation control unit 151 performs a replenishment operation for the product T.

[0069] The gripping target specifying unit 155b performs at least one of the following processes based on at least one of the image data captured by the first camera 50R and the distance data: determining whether or not there are replenishment target products to be gripped on the inventory shelf SL20 and the replenishment shelf SL30; specifying the size or shape of the replenishment target products; and determining the gripping position of the replenishment target products. In the case of a product T having a cap Tb as shown in FIG. 2, the gripping target specifying unit 155b sets the gripping position, for example, near the cap Tb. On the other hand, in the case of a product T such as a canned beverage without a cap, the gripping target specifying unit 155b may set the gripping position to the side of the container. (Product Replenishment Operation by the Product Replenishment Robot 100)

[0070] Next, a description will be given of the product replenishment operation by the product replenishment robot 100. An example in which products T from the inventory shelf SL20 are replenished to the display shelf SL10 will be described below. Fig. 9 is a flowchart showing the product replenishment operation by the product replenishment robot 100.

[0071] First, in step S11, the rear surface of the display shelf SL10 is photographed by the first camera 50L attached to the arm unit 20 of the product replenishment robot 100. The product replenishment robot 100 moves the arm unit 20, horizontal movement mechanism 80, and lifting mechanism 90 so that the first camera 50L can photograph each level of the display shelf SL10. Next, the product replenishment robot 100 operates the first camera 50L to capture an image of the rear surface of the display shelf SL10 and also acquire distance data to the products T lined up on the display shelf SL10.

[0072] Next, in step S12, a determination is made as to whether or not replenishment is possible. In the "determination of whether or not replenishment is possible," the replenishment possibility determination unit 155a in the image data processing unit 155 of the product replenishment robot 100 analyzes the captured image of the rear surface of the display shelf and determines which product T can be displayed on which shelf SL11 (in other words, which product T needs to be replenished). The product replenishment robot 100 acquires an image of the rear surface of the display shelf SL10 as shown in FIG. 6.

[0073] The image data processing unit 155 of the product replenishment robot 100 identifies the lane on the shelf SL11 to which the product T is to be displayed on the display shelf SL10 based on the image data captured by the first camera 50L. Furthermore, the image data processing unit 155 of the product replenishment robot 100 can acquire distance data to the product T (depth data Dep visualized in FIG. 8 ), and therefore can recognize that there is space Sp behind the product T, such as the product T in lanes "7" and "8," where the number of products displayed has decreased. The replenishment availability determination unit 155a of the product replenishment robot 100 determines whether there is space Sp for the product T based on the distance data to the product T, and determines whether more products T can be displayed in lanes "7" and "8." Step S12 determines which product T needs to be replenished on which shelf SL11 of the display shelf SL10.

[0074] The method for determining whether a product needs to be replenished is not limited to the above method, and various other methods can be used. For example, it may be possible to determine the percentage of the product T that is allocated within a predetermined three-dimensional space, and if this value is equal to or less than a predetermined reference value, it may be determined that the product T needs to be replenished.

[0075] Next, in step S13, the inventory shelf SL20 is photographed. The product replenishment robot 100 operates the arm unit 20, the horizontal movement mechanism 80, the lifting mechanism 90, and the first camera 50R to photograph the inventory shelf SL20 from the front side. As an example, the product replenishment robot 100 photographs the inventory shelf SL20 one shelf at a time to obtain a photographed image showing the inventory status of the products T. The photograph of the inventory shelf SL20 does not need to be taken after the photograph of the display shelf SL10; the photograph of the inventory shelf SL20 may be taken before the photograph of the display shelf SL10.

[0076] Next, in step S14, the image data processing unit 155 of the product replenishment robot 100 analyzes the captured image of the inventory shelf SL20 acquired in step S13 and identifies the products lined up on the inventory shelf SL20. In addition, the grasping target identification unit 155b of the image data processing unit 155 identifies the grasping position of the product. Through the steps up to this point, the product replenishment robot 100 acquires information indicating which product T needs to be replenished at which position on which shelf SL11 of the display shelf SL10, as well as information indicating the position on the inventory shelf SL20 of the product to be replenished corresponding to that product T and the grasping position of that product to be replenished.

[0077] Next, in step S15, the product replenishment robot 100 performs a product replenishment operation (pick-and-place operation) based on the acquired information. Specifically, the operation control unit 152 of the control unit 151 (see FIG. 7 ) of the product replenishment robot 100 operates the arm unit 20, horizontal movement mechanism 80, and lifting mechanism 90 to move the gripper 10 toward a predetermined product to be replenished on the inventory shelf SL20. The gripper 10 then grips the predetermined gripping position of the product to be replenished (for example, the cap Tb of the product T if the product T is a PET bottled beverage) and lifts the product. The operation control unit 152 then operates the arm unit 20, horizontal movement mechanism 80, and lifting mechanism 90 to move the gripped product T to a predetermined placement position on the display shelf SL10 and releases the product T from the gripper 10 to place the product T in the predetermined position. Thereafter, the product replenishment robot 100 repeats the same pick-and-place operation to complete the replenishment of the product T.

[0078] While the above describes an example of replenishing the display shelf SL10 with products T from the inventory shelf SL20, the replenishing operation when replenishing the inventory shelf SL20 with products T from the replenishing shelf SL30 is also performed according to the same process. In this case, as an example, the first camera 50L captures an image of the front of the inventory shelf SL20, and the image data processing unit 155 of the product replenishment robot 100 identifies the lane on the shelf SL21 to which the products from the inventory shelf SL20 are to be replenished based on the image data captured by the first camera 50L. Furthermore, the first camera 50R captures an image of the front of the replenishing shelf SL30, and the image data processing unit 155 of the product replenishment robot 100 identifies the position of each product placed on the replenishing shelf SL30 based on the image data captured by the first camera 50R. When replenishing products T from the product replenishment shelf to the inventory shelf SL20, the shelf board SL21 of the inventory shelf SL20 is first placed in a first position in which the front end (right side in the figure) is higher than the rear end (left side in the figure), and after replenishing is completed, the shelf board SL21 is placed in a second position in which the front end is lower than the rear end. <Configuration of the product image acquisition device 200>

[0079] Fig. 10 is a perspective view showing the overall configuration of the commodity image acquisition device 200 included in the commodity replenishment system 1 of this embodiment. The commodity image acquisition device 200 includes a turntable 210, a rotation drive unit 220, a lifting mechanism 230, a lifting drive mechanism 240, a lifting drive unit 250, and an image acquisition unit 260. Fig. 11 is a diagram showing the details of the configuration of each unit of the commodity image acquisition device 200, with Fig. 11(a) being a plan view, Fig. 11(b) being a side view showing the turntable 210 positioned at the lowest position by the lifting mechanism 230, and Fig. 11(b) being a side view showing the turntable 210 positioned at the highest position by the lifting mechanism 230.

[0080] The turntable 210 has a turntable 212 having a circular periphery, and a storage section 214 that supports a rotation drive section 220 that rotates the turntable 212 and stores the turntable 212 in the circumferential direction. The turntable 212 is configured to be rotatable around a central axis, and can rotate around the central axis together with the products placed thereon.

[0081] The turntable 210 is also configured so that the turntable 212 can move between a first position, in which it is biased by a biasing member such as a spring so that it protrudes somewhat above the turntable 210, and a second position, in which it is pushed from the first position against the biasing force of the biasing member and abuts against the turntable 210. As a result, the turntable 212 is located at the first position when no product is abutting on the turntable 212, and is located at the second position when a product is abutting on or placed on the turntable 212. A contact switch (not shown) is provided between the turntable 210 and the turntable 212, and the contact switch is off when the turntable 212 is in the first position, and the contact switch is on when the turntable 212 is in the second position.

[0082] The rotation drive unit 220 is supported and fixed to the underside of the accommodation portion 214 of the turntable 210, and supports the turntable 212, the periphery of which is accommodated in the accommodation portion 214, so that the turntable 212 can be rotated about its central axis. The rotation drive unit 220 includes a mechanism and a drive source for rotationally driving the turntable 210, and the drive source is, for example, an electric motor. The turntable 210 may be configured to be directly coupled to the rotation shaft of an electric motor of the drive source so as to be rotationally driven, or may be configured to transmit the rotational driving force of the electric motor to the turntable 212 via drive transmission means such as gears, thereby rotationally driving the turntable 212.

[0083] The lifting mechanism 230 is a mechanism for raising and lowering the rotating table 210 and is, for example, a scissors-type link mechanism that expands and contracts in the vertical direction. The link mechanism of the lifting mechanism 230 is configured by connecting multiple link members 232 with connecting portions 234 that allow the link members 232 to rotate relative to one another. In this example, two link mechanisms, each configured by connecting both end portions and a middle portion of four link members 232 with connecting portions 234, connect the rotating table 210 and the lifting drive mechanism 240 on both sides. The upper ends of the two upper link members 232 of the link mechanism are connected to the underside of the storage portion 214 of the rotating table 210 via the connecting portions 234, and the lower ends of the two lower link members 232 of the link mechanism are connected to the lifting drive mechanism 240 via the connecting portions 234. With this configuration, the lifting mechanism 230 can be transformed to expand and contract between a configuration in which the turntable 210 is positioned at the lowest position (Figure 11(b)) and a configuration in which the turntable 210 is positioned at the highest position (Figure 11(c)).

[0084] The lifting drive mechanism 240 is a drive mechanism for extending and retracting the lifting mechanism 230 to lift and lower the rotating table 210, and is configured to, for example, reciprocate the lower end of one link member 232 on the lower side of the scissors mechanism of the lifting mechanism 230 in a direction (first direction) toward the lower end of the other link member 232 and in a direction (second direction) away from the other end. More specifically, the lifting drive mechanism 240 includes a slider 242 connected by a connecting portion 234 to the lower end of one link member 232 on the lower side of each scissors mechanism of the lifting mechanism 230, a slider drive unit 244 that reciprocates the slider 242 in the first and second directions, and a guide rail 246 that supports the slider 242 so that it can reciprocate in the first and second directions. As an example, the slider drive unit 244 has a shaft member 244a with a male thread formed on its outer circumferential surface, and the slider 242 has a female thread formed therein through which the male thread passes, and is configured so that rotating the shaft member 244a in one rotation direction moves the slider 242 in a first direction (for example, the right side in the figure), and rotating the shaft member 244a in the other rotation direction moves the slider 242 in a second direction (for example, the left side in the figure). The lift drive mechanism 240 also functions as a base member on which the turntable 210, the rotation drive unit 220, the lift mechanism 230, and the lift drive unit 250 are mounted.

[0085] The lifting / lowering drive unit 250 includes a drive source, such as an electric motor, that rotates the shaft member 244a of the lifting / lowering drive mechanism 240. The lifting / lowering drive mechanism 240 may be configured such that the rotation shaft of the electric motor of the drive source is directly coupled to the shaft member 244a of the lifting / lowering drive mechanism 240 to rotate the shaft member 244a, or may be configured such that the rotational driving force of the electric motor is transmitted to the shaft member 244a of the lifting / lowering drive mechanism 240 via a drive transmission means such as a gear to rotate the shaft member 244a.

[0086] The electric motor that is the drive source of the lifting / lowering drive unit 250 is configured to be able to switch its rotation direction between one rotation direction and the other rotation direction and to appropriately change the rotation speed in each rotation direction. As a result, the electric motor that is the drive source of the lifting / lowering drive unit 250 can rotate the shaft member 244a of the lifting / lowering drive mechanism 240 in one rotation direction to move the slider 242 in a first direction (e.g., to the right in the figure) to extend the lifting mechanism 230 and raise the turntable 210, and can rotate the shaft member 244a in the other rotation direction to move the slider 242 in a second direction (e.g., to the left in the figure) to retract the lifting mechanism 230 and lower the turntable 210, and can also change the speed of these raising and lowering movements.

[0087] The main body of commodity image acquisition device 200, which is made up of turntable 210, rotation drive unit 220, lifting mechanism 230, lifting drive mechanism 240, and lifting drive unit 250, is installed, for example, on base plate 81 of commodity replenishment robot 100. Furthermore, rotation drive unit 220 and lifting drive unit 250 are connected by wired or wireless connection to communication unit 195 of control device 150 of commodity replenishment robot 100, and control device 150 controls the operation of these drive units (e.g., electric motors).

[0088] Image acquisition unit 260 has a function of acquiring an image of the outer periphery of a product placed on turntable 210, and includes, for example, an imaging element that generates a captured image in which pixels are arranged two-dimensionally (an RGB image, for example). Image acquisition unit 260 is installed, for example, on lifting mechanism 90 that forms the housing of product replenishment robot 100.

[0089] The image acquisition unit 260 captures an image of a product T held by the gripper 10 of the commodity replenishment robot 100 while the product T is positioned in front of the image acquisition unit 260 (the image acquisition position for the image acquisition unit 260), or while a product placed on the turntable 210 is being rotated by the rotation drive unit 220. The image acquisition unit 260 then captures an image of the product's side or outer periphery (including product labels, barcodes, etc., surrounding the product) and acquires the image data. The image acquisition unit 260 then transmits the acquired image data to the control device 150 of the commodity replenishment robot 100 via the communication unit 195 of the control device 150. Upon receiving the image data of the product, the image data processing unit 155 of the control device 150 performs processing to identify the type of the product using the image data of the product's side or outer periphery (including product labels, barcodes, etc., surrounding the product), as described above. In this manner, the commodity image acquisition device 200 and the control device 150 function as a commodity identification system for identifying the type of product held by the commodity replenishment robot 100. By mounting the commodity image acquisition device 200 on the commodity replenishing robot 100, the commodity replenishing robot 100 can place the commodity at a position where an image is acquired by the image acquisition unit 260 simply by operating the arm unit 20 after grasping the commodity.

[0090] 12, the management device 300 includes a processor 310, a storage unit 320, an input unit 330, a display unit 340, and a communication unit 350. Although the management device 300 is depicted as a single element in FIG. 12, the management device 300 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0091] The input unit 330 is a device for receiving input from an operator of the management device 300. The input unit 330 may be configured with a keyboard, a mouse, and a touch panel. The display unit 340 is a display device that displays a display screen generated by the processor 310, and may be, for example, a liquid crystal display or an organic EL display device. The communication unit 350 communicates with the control device 150 and terminal device 400 of the commodity replenishment robot 100 via wired or wireless communication, and inputs and outputs information, control signals, and the like between these components.

[0092] The storage unit 320 includes a temporary or non-temporary storage medium such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a solid state drive (SSD). The storage unit 320 stores a computer program executed by the processor 155. The computer program stored in the storage unit 320 includes instructions for performing information processing by the processor 310, which will be described later with reference to FIG. 14 and the like. The storage unit 160 also at least temporarily stores information received from the control device 150 or the terminal device 400, and various data (including intermediate data) generated by processing operations by the processor 310.

[0093] The storage unit 320 also stores, for example, past sales data for each product at each store. The sales data includes information such as the sales date and time and the quantity of each product. This sales data can be used as statistical information for predicting which products should be replenished on which time schedule on the display shelves SL10 and inventory shelves SL20 of each store. The storage unit 320 may also store a trained model generated by performing machine learning using such sales data with a learning device. This trained model can be used as a sales prediction model for generating a schedule for replenishing products in a store.

[0094] The processor 310 is configured, for example, with one or more CPUs (Central Processing Units). The processor 310 operates to realize the processes and functions performed by the processor 310 as described with reference to Fig. 14. In particular, the processor 310 uses past sales data or a learned sales prediction model stored in the storage unit 320 to predict which products should be replenished on which time schedule on the display shelves SL10 and inventory shelves SL20 of each store, and based on the prediction, schedules the types and quantities of products to be replenished on the inventory shelves SL20 and the timing of replenishment, thereby generating product replenishment information.

[0095] The product replenishment information includes loading layout information regarding products to be replenished from the product replenishment shelf SL30 to the inventory shelf SL20 at a certain replenishment timing, including the loading position and loading quantity of each product in each area on the product replenishment shelf SL30 where the products are loaded. The loading layout information may include shelf designation information that designates the product replenishment shelf SL30 on which the products should be loaded.

[0096] The generated product replenishment information is transmitted from the communication unit 350 to the control device 150 and the terminal device 400. The terminal device 400, which is the user of the terminal device 400, displays the received loading layout information on its display screen. A store clerk places designated quantities of each type of designated product in each designated area on a specific designated product replenishment shelf SL30 in accordance with the loading layout information displayed on the terminal device 400. <Configuration of the terminal device 400>

[0097] 14, the terminal device 400 includes a processor 410, a storage unit 420, a display operation unit 430, a communication unit 440, and a camera 450. Note that this configuration is an example, and other configurations may be included.

[0098] The processor 410 controls the overall operation of the terminal device 400 by executing a program stored in the storage unit 420, and is configured from a CPU, a GPU, etc. The processor 410 operates to realize the processes and functions performed by the processor 410, as will be described with reference to FIG.

[0099] The storage unit 420 stores various control processes, programs for the processor 410 to execute each function, input data, etc., and is composed of RAM, ROM, and other storage. The storage unit 420 also temporarily stores the content of communications with other terminals. The storage unit 420 particularly stores a program for causing the processor 410 to execute the processes performed by the terminal device 400, which will be described with reference to FIG. 14.

[0100] The display operation unit 430 is a user interface used by the user of the terminal device 400 to input instructions and to display text, images, etc. in accordance with the display processing of the processor 410, and is composed of a touch panel, etc., if the terminal device 400 is a tablet terminal, etc.

[0101] The communication unit 440 is a communication interface for communicating with the control device 150 and the management device 300 via a network.

[0102] Camera 450 is primarily used to read labels affixed to product replenishment shelves SL30, which contain characters, two-dimensional codes, and the like as identification information for identifying product replenishment shelves SL30. Storage unit 420 also stores a character recognition program that recognizes characters, two-dimensional codes, and the like read by camera 450, and a program that decodes codes read by camera 450.

[0103] The processor 410 displays the product replenishment information received from the management device 300 on its display operation unit 430. The store clerk places specified quantities of various specified products in specified positions on a specified specific product replenishment shelf SL30 in accordance with the loading layout information displayed on the terminal device 400. The store clerk photographs the identification labels affixed to the specified specific product replenishment shelf SL30 with the camera 450 of the terminal device 400. The processor 410 transmits image data of the identification labels photographed by the camera 450 to the management device 300 via the communication unit 440.

[0104] (Placement layout information)

[0105] Here, the loading layout information including the loading position and loading quantity of each product in each area on the product replenishment shelf SL30 will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of a display screen of the loading layout information displayed on the display operation unit 430 of the terminal device 400.

[0106] The placement layout information displayed on the display operation unit 430 indicates the type and number of products to be placed in each area (areas 1 to 10 in the example shown in Figure 14) divided on each shelf of the product replenishment shelf SL30 where the products are to be placed.

[0107] The store clerk retrieves various products stored in the store's product storage area (usually at room temperature) and places the specified number of products in the designated areas on the product replenishment shelf SL30 according to the placement layout information displayed on the display operation unit 430 of the terminal device 400 as shown in Figure 14. As an example, when placing one delivery package of products in a designated area, the upper and front portions of the delivery package can be removed, and the remaining part of the delivery package, including the products inside, can be placed in the designated area on the product replenishment shelf SL30 (e.g., areas 1 to 6 and 8 shown in Figure 14). Alternatively, multiple types of products may be placed together in a designated area (e.g., area 7 shown in Figure 14).

[0108] When various products are placed on the product replenishment shelf SL30 in accordance with the placement layout information, the product placement work on the product replenishment shelf SL30 is completed, and the product replenishment shelf SL30 is ready to be placed in a predetermined position within the store.

[0109] 14 , on the display screen of the placement layout information displayed on the display operation unit 430 of the terminal device 400, the display portion of each product can be selected by clicking on the display operation unit 430. For example, if a product specified in the placement layout information is out of stock, such as when the product is not present in the product storage location in the store or when fewer than the specified number of products are stored, the store clerk clicks on the product on the display screen of the placement layout information displayed on the display operation unit 430 to select it. The processor 410 of the terminal device 400 then displays a column for accepting changes to the number of products for that product on the display operation unit 430 and accepts numerical input on the display operation unit 430. When the store clerk inputs the numerical value of the number of products that can be placed on the display operation unit 430, the processor 410 of the terminal device 400 transmits numerical information about the product to the management device 300. When the processor 310 of the management device 300 receives the numerical information about the product, it updates the number of units of the product in the placement layout information according to the numerical value, and sends product replenishment information including the updated placement layout information to the control device 150.

[0110] According to the product replenishment system of the present disclosure, a placement layout including the various products and the numbers of products to be placed on the replenishment shelf SL30, which holds the products to be replenished on the inventory shelf SL20, is generated by the management device 300 in accordance with a product replenishment schedule for the inventory shelf SL20, and is presented to a store clerk via the terminal device 400. The store clerk places the various products on the replenishment shelf SL30 in accordance with the placement layout displayed on the display operation unit 430 of the terminal device 400, and places the replenishment shelf SL30 in a predetermined position within the store. Thereafter, the product replenishment robot 100, controlled by the control device 150, sequentially moves the various products placed on the replenishment shelf SL30 to the corresponding product rows on the inventory shelf SL20, thereby replenishing products from the replenishment shelf SL30 to the inventory shelf SL20. After the product replenishment robot 100 has completed replenishment of products from the replenishment shelf SL30 to the stock shelf SL20, it resumes replenishment of products from the stock shelf SL20 to the display shelf SL10 until the next scheduled replenishment of products from the replenishment shelf SL30 to the stock shelf SL20 begins. In this way, according to the product replenishment system of the present disclosure, in addition to replenishing products from the stock shelf SL20 to the display shelf SL10, the product replenishment robot can also replenish products to the stock shelf SL20.

[0111] Here, various products are placed on the product replenishment shelf SL30 by the store clerk according to the placement layout displayed on the display operation unit 430 of the terminal device 400 as described above. However, if a product different from the product specified in the placement layout is placed, or if the specified product is placed in a different area of ​​the product replenishment shelf SL30 from the specified area, and the product is replenished from the product replenishment shelf SL30 to the inventory shelf SL20 without checking the placed product, the product different from the product that should have been replenished from the inventory shelf SL20 to the display shelf SL10 will ultimately be replenished. The present disclosure provides a means to address such inconveniences. <Operation Example>

[0112] Next, an example of the operation of the image acquisition device 200 and the control device 150 that function as a commodity identification system for identifying the type of commodity grasped by the commodity replenishment robot 100 in this embodiment will be described. Fig. 15 is a flowchart for explaining an example of the operation of the image acquisition device 200 and the control device 150 that function as a commodity identification system in this embodiment. Fig. 16 is a diagram showing a series of operations for acquiring commodity images by the commodity image acquisition device.

[0113] First, in step S21, the control device 150 receives from the management device 300 an instruction to cause the product replenishment robot 100 to perform an operation to replenish products on the product replenishment shelf SL30 located at a predetermined position within the store onto the inventory shelf SL20.The control device 150 then causes the product replenishment robot 100 to sequentially move the various products placed on the product replenishment shelf SL30 to the corresponding product rows on the inventory shelf SL20, thereby commencing the product replenishment operation from the product replenishment shelf SL30 to the inventory shelf SL20.

[0114] In the product replenishment system 1 of this embodiment, when a store clerk places various products on the replenishment shelf SL30 according to the placement layout displayed on the display operation unit 430 of the terminal device 400 and places the replenishment shelf SL30 in a predetermined location within the store, the store clerk operates the display operation unit 430 of the terminal device 400 to send completion confirmation information from the terminal device 400 to the management device 300, indicating that the replenishment shelf SL30 has been placed in the predetermined location within the store. Upon receiving the completion confirmation information, the management device 300 sends an instruction to the control device 150 of the product replenishment robot 100 to cause the product replenishment robot 100 to replenish the products on the replenishment shelf SL30 placed in the predetermined location within the store onto the inventory shelf SL20. The product replenishment robot 100, controlled by the control device 150, then sequentially moves the various products placed on the replenishment shelf SL30 to the corresponding product rows on the inventory shelf SL20, thereby replenishing the products from the replenishment shelf SL30 to the inventory shelf SL20.

[0115] The operation of replenishing products from the product replenishment shelf SL30 to the inventory shelf SL20 may be performed, for example, sequentially for each area in the placement layout described with reference to Fig. 14 for all products in each area, or may be performed by repeatedly replenishing one or several products in each area in the placement layout for each area. Explaining the latter more specifically, as an example, the operation of replenishing three products in each area may be performed sequentially from area 1 to area 8, and this may be repeated multiple times until all products on the product replenishment shelf SL30 are used up.

[0116] Next, in step S22, the control device 150 causes the product replenishment robot 100 to perform an operation of placing the product that the product replenishment robot 100 has grasped and removed from the product replenishment shelf SL30 in a position in front of the image acquisition unit 260 where an image can be acquired.

[0117] 16(a), the product T grasped by the gripper 10 of the product replenishment robot 100 and removed from the product replenishment shelf SL30 is positioned above the turntable 210 of the product image acquisition device 200 at a height position that allows the image acquisition unit 260 of the product image acquisition device 200 to capture an image of the side of the product T. This makes it possible for the image acquisition device 200 to acquire an image of the product T grasped by the product replenishment robot 100.

[0118] Next, in step S23, the control device 150 executes a process to cause the image acquisition device 200 to acquire an image (first image) of one side of the product that has been grasped by the product replenishment robot 100 and placed in front of the image acquisition device 200.

[0119] The control device 150 causes the image acquisition unit 260 to capture an image (first image) of the side of the product T that is gripped by the product replenishment robot 100 and placed in front of the image acquisition device 200, facing the image acquisition unit 260 (see FIG. 16( a)). Therefore, in step S23, the image of the product T captured by the image acquisition unit 260 is an image of the side of the product T that faces the image acquisition unit 260, and no images of the product T viewed from other directions (such as the side or rear) are captured. The image data (first image data) of the product T captured by the image acquisition unit 260 is transmitted to the control device 150 of the product replenishment robot 100 via the communication unit 195 of the control device 150, as described above.

[0120] Next, in step S24, the control device 150 executes a process (first product identification process) to identify the type of product T based on the first image data received from the image acquisition unit 260 (the first image, which is an image of the side of the product T facing the image acquisition unit 260), and determines whether the type of product T was successfully identified by the first product identification process.

[0121] As an example, the control device 150 can identify the type of product T by performing image analysis processing in which image data transmitted to the control device 150 is compared with image data of various products stored in the storage unit 160 of the control device 150. The surrounding image data of the various products may include, for each product T, multiple image data captured from various directions or a complete circumference of the product T. Alternatively, the control device 150 can identify the type of product T by using a trained model generated by machine learning using an arbitrary learning device using the image data of the various products described above. The trained model is stored in the storage unit 160 of the control device 150.

[0122] The control device 150 can determine whether the type of product T has been successfully identified by the first product identification process, for example, based on a matching score calculated in an image analysis process that compares image data of the product T to be identified with image data to be compared, or a determination process that uses a trained model. For example, the control device 150 determines that the type of product T has been successfully identified (Y) if the matching score is equal to or greater than a predetermined value, and determines that the type of product T has been unsuccessful (N) if the matching score is less than the predetermined value. If the control device 150 determines that the type of product T has been successfully identified (Y), the product identification process is terminated, and if it determines that the type of product T has been unsuccessful (N), the process proceeds to step S25.

[0123] In the first product identification process of step S24, an image of one side of a product T that has been gripped by the gripper 10 of the product replenishment robot 100 and placed opposite the image acquisition unit 260 is captured, and a process is performed to identify the type of product T based on the image data. Since it is arbitrary which part of the side of the product T gripped by the gripper 10 faces the image acquisition unit 260 and is imaged, in some cases a side of the product T other than the front of the label may be imaged, which may prevent the control device 150 from identifying the product T. In that case, a second product identification process is performed in steps S25 and S26, which will be described later, to acquire image data of the entire periphery of the side of the product T.

[0124] Next, in step S25, the control device 150 executes a process to cause the image acquisition unit 260 to acquire an image (second image) of the entire side of the product that has been grasped by the product replenishment robot 100 and placed in front of the image acquisition unit 260.

[0125] In step S25, first, if the commodity T placed in a position in front of the image acquisition unit 260 where an image can be acquired in step S22 is not placed on the turntable 210 of the commodity image acquisition device 200, that is, if the bottom surface of the commodity T is not in contact with the turntable 212 of the turntable 210, the control device 150 drives the lifting drive unit 250 to rotate the shaft member 244a of the lifting drive mechanism 240 in one rotational direction to move the slider 242 in a first direction (to the right in FIG. 11 ), thereby moving the lower end of one link member 232 below the scissors mechanism of the lifting mechanism 230 in a direction (first direction) approaching the lower end of the other link member 232, thereby extending the lifting mechanism 230 and raising the turntable 210 until the turntable 212 of the turntable 210 contacts the bottom surface of the commodity T. This places the bottom surface of the commodity T on the turntable 210. The contact of the turntable 210 with the bottom surface of the product T can be detected when the turntable 212 of the turntable 210 comes into contact with the bottom surface of the product T, the turntable 212 displaces from the first position to the second position against the biasing force of the biasing member as described above, and the contact switch turns on.

[0126] It should be noted here that how far the lifting mechanism 230 is extended, in other words, how far the turntable 210 is raised, depends on the size of the product T grasped by the product replenishment robot 100 and placed above the turntable 210. Figure 17 is a diagram showing an example in which the turntable 210 is raised to various heights depending on the size of the product T. As shown in each diagram in Figure 17, the height of the upper end of the product T grasped by the gripper 10 is constant regardless of the size of the product T, and in the example shown, the product T is placed in front of the image acquisition unit 260 so that the height of the upper end of the product T is approximately the same as the height of the upper end of the image acquisition unit 260.

[0127] FIG. 17( a) shows a state in which the turntable 212 of the turntable 210 is in contact with the bottom of a relatively large product T, such as a 2000 ml PET bottle. In this example, the turntable 212 is in contact with the bottom of the product T, with the product T placed on it, simply by the turntable 210 being lifted by the lifting mechanism 230 from its lowest position by a certain amount. FIG. 17( b) shows a state in which the turntable 212 of the turntable 210 is in contact with the bottom of a relatively small product T, such as a 500 ml PET bottle. In this example, the turntable 210 is further lifted by the lifting mechanism 230 from the height shown in FIG. 17( a), with the turntable 212 in contact with the bottom of the product T, with the product T placed on it. FIG. 17( c) shows a state in which the turntable 212 of the turntable 210 is in contact with the bottom of a small product T, such as a 180 ml can. In this example, the turntable 210 is further raised from the height shown in FIG. 17(b) by the lifting mechanism 230, so that the turntable 212 comes into contact with the bottom of the product T and places the product T on it.

[0128] In addition, when the product T grasped by the gripping unit 10 of the product replenishment robot 100 is moved above the turntable 210 to a height position where the image acquisition unit 260 can acquire an image, if the bottom surface of the product T is placed on the turntable 210, the operations shown in Figures 16 (b) and 17 are not performed. Furthermore, when the turntable 210 is positioned at a position higher than the height of the bottom surface of the product T when it is moved to a height position where an image can be acquired by the image acquisition unit 260, the control device 150 drives the lifting drive unit 250, rotates the shaft member 244a of the lifting drive mechanism 240 in a rotational direction opposite to the above-mentioned one rotational direction, and moves the slider 242 in a second direction (left side as shown in Figure 11), thereby moving the lower end of one of the link members 232 below the scissors mechanism of the lifting mechanism 230 in a direction (second direction) away from the lower end of the other link member 232, thereby contracting the lifting mechanism 230, and once lowering the turntable 210, the lifting mechanism 230 is extended until it reaches a height where the rotating table 212 of the turntable 210 contacts the bottom surface of the product T, thereby raising the turntable 210.

[0129] Once the product T has been placed on the turntable 210 as described above, the control device 150 then widens the gap between the fingers 11 of the gripping unit 10 to temporarily release the grip of the product T, and then drives the rotation drive unit 220 to rotate the turntable 212 of the turntable 210. The control device 150 then causes the image acquisition unit 260 to capture an image (second image) of the periphery of the side of the product T as it rotates together with the turntable 212. The image of the periphery of the side of the product T can be captured, for example, by the control device 150 controlling the image capture operation of the image acquisition unit 260 so that the image acquisition unit 260 captures an image only during one rotation of the turntable 212. The image (second image) of the periphery of the side of the product T captured in this manner is an image of the entire periphery of the side of the product T, and corresponds to an image of the product T when a label attached to the periphery of the side of the product T is peeled off and the product is laid out flat. Image data (second image data) of the area around the side of the product T captured by the image acquisition unit 260 is transmitted to the control device 150 of the product replenishment robot 100 via the communication unit 195 of the control device 150, as described above.

[0130] Next, in step S26, the control device 150 executes a process (second product identification process) to identify the type of product T based on the second image data (image data of the image around the side of the product T (second image)) received from the image acquisition unit 260, and determines whether the type of product T was successfully identified by the second product identification process.

[0131] As in the first commodity identification process in step S24 described above, in the second commodity identification process in step S26, the type of commodity T identified by the control device 150 can be identified by, for example, an image analysis process in which the image data transmitted to the control device 150 is compared with image data of various commodities stored in the storage unit 160 of the control device 150, or the above-described trained model generated by machine learning with an arbitrary learning device using the image data of various commodities. In the second commodity identification process, a process in which the type of commodity T is identified based on an image of the periphery of the side of the commodity T is performed, and therefore the accuracy of commodity identification is higher than in the first commodity identification process in which the type of commodity T is identified based on an image of only one side.

[0132] The control device 150 can also determine whether the type of product T has been successfully identified by the second product identification process, for example, based on a matching score calculated in an image analysis process that compares image data of the product T to be identified with image data to be compared, or a determination process that uses a trained model. For example, the control device 150 determines that the type of product T has been successfully identified (Y) if the matching score is equal to or greater than a predetermined value, and determines that the type of product T has failed (N) if the matching score is less than the predetermined value. The predetermined value of the matching score used for determination in the second product identification process may be the same as or different from the predetermined value of the matching score used for determination in the first product identification process. If the control device 150 determines that the type of product T has been successfully identified (Y), the product identification process is terminated. If the control device 150 determines that the type of product T has failed (N), the process proceeds to step S27.

[0133] Finally, in step S27, the control device 150 obtains information regarding the type of product T for which the first and second product identification processes have been performed from an external device (not shown) operated by an operator who remotely controls the product replenishment robot 100.

[0134] In step S27, first, information indicating that the control device 150 failed to identify the type of product T for which the first and second product identification processes were performed, and at least one of the first and second image data of the product T captured by the image acquisition unit 260, are transmitted via the communication unit 195 to an external device (not shown) operated by an operator remotely operating the product replenishment robot 100. The external device (not shown) presents the information received from the control device 150 to the operator on a display unit, accepts input by the operator of information regarding the type of product indicated by the image data, and transmits the information to the control device 150. In this way, the control device 150 acquires information regarding the type of product T for which the product identification processes were performed.

[0135] The above processing operations identify the type of product T that the product replenishment robot 100 has grasped, removed from the replenishment shelf SL30, and placed in front of the image acquisition unit 260 of the image acquisition device 220. Then, if the control device 150 has temporarily released its grip on the product T by widening the spacing between the fingers 11 of the gripper 10, the control device 150 closes the fingers 11 to cause the gripper 10 to grip the product T again, and causes the product replenishment robot 100 to move the product T to a predetermined replenishment position on the inventory shelf SL20 where the product T should be placed. In this way, according to this embodiment, a means is provided that enables the product replenishment robot 100 to replenish the product from the replenishment shelf SL30 to the inventory shelf SL20 after identifying the product T to be replenished on the inventory shelf SL20.

[0136] The above-described product identification process may be performed on all products removed by the product replenishment robot 100 from the product replenishment shelf SL30, or on only a portion of the products on the product replenishment shelf SL30. Referring to the placement layout information shown in FIG. 14 , for example, areas 1 to 6 and 8 of the product replenishment shelf SL30 each contain a single product, reducing the likelihood of a store clerk accidentally placing the wrong product during placement. Therefore, the product identification process may be omitted for products placed in these areas, or the product identification process may be performed only on the first product removed by the product replenishment robot 100. When a single product is placed in a single box, all products contained in the box are of the same type. Therefore, by performing the product identification process only on the first product, the type of all products contained in that box can be identified. On the other hand, because multiple types of products are mixed in area 7 of the product replenishment shelf SL30, it is preferable to perform the product identification process individually for all products placed in area 7.

[0137] Although the present disclosure has been described above through the embodiments of the invention, the above-described embodiments do not limit the invention according to the claims. Furthermore, combinations of features described in the embodiments of the present disclosure may also fall within the technical scope of the present invention. Furthermore, it will be apparent to those skilled in the art that various modifications and improvements can be made to the above-described embodiments.

Claims

1. A product identification system that identifies the type of product grasped and moved by a product replenishment robot that moves products, the product identification system comprising: a product image capture device mounted on the product replenishment robot and capturing images of the products; and a control device that controls the operation of the product image capture device and identifies the type of the product based on the images of the products captured by the product image capture device, the product image capture device comprising: a turntable equipped with a turntable that rotates the products placed on it; a lifting mechanism that raises and lowers the turntable; and an image capture unit that captures images of the products, the control device comprising a control unit and a memory unit, the memory unit storing image data of various products or a trained model generated by running machine learning in a learning device using the image data, and the control unit configured to execute, based on the image data or the trained model, identifying the type of product associated with the image captured by the image capture unit.

2. The product identification system of claim 1, wherein the control unit is configured to cause the image acquisition unit to acquire an image of the product that is grasped by the product replenishment robot and placed at an image acquisition position of the image acquisition unit.

3. The product identification system of claim 2, wherein the image is an image of one side of the product.

4. The product identification system of claim 1, wherein the control unit is configured to perform the following: operate the product replenishment robot to place the product grasped by the product replenishment robot on the turntable; rotate the turntable on which the product is placed; and cause the image acquisition unit to acquire the image of the surroundings of the side of the product while the turntable is rotating.

5. The product identification system of claim 1, wherein the control unit is configured to operate the lifting mechanism to raise or lower the turntable to a height at which the turntable contacts the bottom surface of the product that has been grasped by the product replenishment robot and placed at the image acquisition position of the image acquisition unit.

6. A product identification method using a product identification system that identifies the type of product grasped and moved by a product replenishment robot that moves products, wherein the product identification system comprises: a product image capture device mounted on the product replenishment robot and captures images of the products; and a control device that controls the operation of the product image capture device and identifies the type of the product based on the images of the products captured by the product image capture device, wherein the product image capture device comprises: a turntable equipped with a turntable that rotates the products placed on it, a lifting mechanism that raises and lowers the turntable, and an image capture unit that captures images of the products; the control device has a control unit and a memory unit, and the memory unit stores image data of various products or a trained model generated by running machine learning on a learning device using the image data, and the method executed by the control unit includes: operating the product replenishment robot to place the products grasped by the product replenishment robot at an image capture position of the image capture unit; causing the image capture unit to capture images of the products; and identifying the type of the product associated with the image captured by the image capture unit based on the image data or the trained model.

7. The method of claim 6, wherein the image is an image of one side of the product.

8. The product identification method described in claim 6, wherein the method executed by the control unit further includes: operating the product replenishment robot to place the product grasped by the product replenishment robot on the turntable; rotating the turntable on which the product is placed; and causing the image acquisition unit to acquire the image around the side of the product while the turntable is rotating.

9. The product identification method described in claim 6, wherein the method executed by the control unit further includes operating the lifting mechanism to raise or lower the turntable to a height at which the turntable contacts the bottom surface of the product that is grasped by the product replenishment robot and placed at the image acquisition position of the image acquisition unit.

10. A program for controlling a product identification system that identifies the type of product grasped and moved by a product replenishment robot that moves products, wherein the product identification system comprises: a product image capture device mounted on the product replenishment robot and captures images of the products; and a control device that controls the operation of the product image capture device and identifies the type of the product based on the images of the products captured by the product image capture device, wherein the product image capture device comprises: a turntable equipped with a turntable that rotates the products placed on it, a lifting mechanism that raises and lowers the turntable, and an image capture unit that captures images of the products; the control device has a control unit and a memory unit, and the memory unit stores image data of various products or a trained model generated by running machine learning with a learning device using the image data, and the program causes the control unit to operate the product replenishment robot to place the products grasped by the product replenishment robot at the image capture position of the image capture unit, cause the image capture unit to capture images of the products, and identify the type of product associated with the images captured by the image capture unit based on the image data or the trained model.

11. The program of claim 10, wherein the image is an image of one side of the product.

12. The program described in claim 10, further causing the control unit to operate the product replenishment robot to place the product grasped by the product replenishment robot on the turntable; rotating the turntable on which the product is placed; and causing the image acquisition unit to acquire the image of the surroundings of the side of the product while the turntable is rotating.

13. The program described in claim 10, further causing the control unit to operate the lifting mechanism to raise or lower the turntable to a height at which the turntable contacts the bottom surface of the product that is grasped by the product replenishment robot and placed at an image acquisition position of the image acquisition unit.

Citation Information

Patent Citations

  • Inverted pendulum type carriage robot

    JP2004345030A

  • Robot system, information processing device, control method of the same, and program

    JP2021017324A

  • Method and apparatus for controlling robot and program

    JP2022018307A

  • Commodity selling system

    JP2022084417A

  • Picking truck, picking system, and picking program

    JP2022187044A