Commodity recognition system and commodity recognition method

Abstract

Provided is a product recognition system for recognizing the type of a product held and moved by a product supplement robot for moving the product. A product recognition system is provided with: a product image acquisition device (200) that acquires an image of a product; and a control device that recognizes the type of the product on the basis of the image of the product acquired by the product image acquisition device. A product image acquisition device (200) is provided with: a rotary table (210) provided with a rotary table (212) for rotating a placed product; a lifting mechanism (230) that lifts and lowers the rotating table (210); and an image acquisition unit (260) that acquires an image of the product. The control device includes a control unit and a storage unit that stores image data of various commodities or a learning completion model generated by performing machine learning by the learning device using the image data. The control unit performs a process for identifying the type of a product related to the image acquired by the image acquisition unit (260) on the basis of the image data or the learned model.

Description

Technical Field

[0001] This disclosure relates to a product identification system and a product identification method. Background Technology

[0002] Patent Document 1 discloses a merchandise moving system including a merchandise moving device that moves merchandise from a stock shelf in a convenience store or similar shop to a display shelf different from the stock shelf. The merchandise moving device includes: a gripping part for gripping merchandise; an arm for moving the gripping part toward the stock shelf and the display shelf; a first camera for photographing the display shelf; a second camera for photographing the stock shelf; and a control unit for controlling the movements of the arm, gripping part, first camera, and second camera.

[0003] The control unit of the merchandise moving device determines the merchandise that needs to be replenished to the display shelf, i.e., the replenishment target merchandise, based on display shelf camera image data generated by the first camera unit capturing images of the display shelf. It also determines the position of the inventory merchandise corresponding to the replenishment target merchandise in the inventory shelf camera image data, based on inventory shelf camera image data. The control unit further uses the inventory shelf camera image data to cause the holding unit to grasp the replenishment target merchandise. After the holding unit grasps the replenishment target merchandise, it moves the holding unit toward the display shelf and, based on the display shelf camera image data, places the replenishment target merchandise on the display shelf at the designated merchandise placement position.

[0004] Thus, the merchandise moving device disclosed in Patent Document 1 performs the action of moving merchandise from the storage shelf to the display shelf for replenishment.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: International Publication No. 2023 / 022214 Summary of the Invention

[0008] The problem the invention aims to solve

[0009] According to the system disclosed in Patent Document 1, although it is possible to move goods from the inventory shelf to the display shelf and replenish the goods on the inventory shelf, it is still necessary for store staff to replenish the goods on the inventory shelf.

[0010] The shelves in the inventory unit are divided into multiple product rows by partitions, each row containing a specific product. The system identifies which product is in each row on each shelf. For example, labels indicating the product to be replenished are placed in front of each product row on each shelf. Employees check these labels and repeatedly replenish the required product to the row. This process of searching for the correct row and replenishing the inventory unit requires significant time and labor. Furthermore, inventory units for beverages in convenience stores are often located in refrigerated backyards, placing a heavy physical burden on employees working in such cold environments.

[0011] Furthermore, during the process of store clerks replenishing goods to the shelves, for safety reasons, the merchandise movement device needs to be moved to a position that does not obstruct the clerks' work and its operation needs to be stopped. Therefore, the merchandise movement device cannot operate during this period. Consequently, the replenishment of goods to the shelves by store clerks becomes one reason for the reduced operational efficiency of the merchandise movement system.

[0012] The purpose of this disclosure is to provide a solution that enables a replenishment robot to replenish the inventory shelves after identifying items to be replenished.

[0013] Solution for solving the problem

[0014] According to one aspect of this disclosure, a product identification system is provided for identifying the type of goods handled and moved by a replenishment robot used to move goods. The product identification system includes: a product image acquisition device mounted on the replenishment robot for acquiring images of goods; and a control device for controlling the operation of the product image acquisition device and identifying the type of goods based on the images of goods acquired by the product image acquisition device. The product image acquisition device includes: a rotary table having a rotating worktable for rotating the goods it carries; a lifting mechanism for raising and lowering the rotary table; and an image acquisition unit for acquiring images of goods. The control device includes a control unit and a storage unit. The storage unit stores image data of various goods or a learned model generated by a learner using the image data to perform machine learning. The control unit is configured to perform the following process: identifying the type of goods involved in the images acquired by the image acquisition unit based on the image data or the learned model.

[0015] Other features and advantages of this disclosure can be understood from the following illustrative and non-exclusive description and accompanying drawings.

[0016] Invention Effects

[0017] According to this disclosure, a scheme is provided that enables a merchandise replenishment robot to replenish the merchandise on the warehouse shelves after identifying merchandise to be replenished. Attached Figure Description

[0018] Figure 1 This is a diagram illustrating the overall structure of a merchandise replenishment system according to one embodiment of the present disclosure.

[0019] Figure 2 It is a schematic top view showing the layout of shelves in the store and the merchandise replenishment robots deployed in the store.

[0020] Figure 3 (a) is a view of the display shelf from the front surface side. Figure 3 (b) is a view of the display shelf from the rear surface side.

[0021] Figure 4 It is a schematic diagram used to illustrate the functions of the storage rack.

[0022] Figure 5 This is a side view schematically showing the structure of the merchandise replenishment robot.

[0023] Figure 6 This is a perspective view showing the peripheral structure of the front end of the arm of a merchandise replenishment robot.

[0024] Figure 7 This is a block diagram showing the structure of the merchandise replenishment robot.

[0025] Figure 8 This is an image of the display shelf taken by the first camera (left side) of the merchandise replenishment robot.

[0026] Figure 9 This is a flowchart illustrating the replenishment actions performed by a replenishment robot.

[0027] Figure 10 This is a perspective view showing the overall structure of the product image acquisition device.

[0028] Figure 11 This is a diagram showing details of the various components of the product image acquisition device.

[0029] Figure 12 This is a block diagram showing the structure of the management device.

[0030] Figure 13 This is a block diagram showing the structure of the terminal device.

[0031] Figure 14 This is an example of a display screen showing the layout information displayed on the display operation section of a terminal device.

[0032] Figure 15 This is a flowchart illustrating the operation of the image acquisition device and control device, which function as a commodity recognition system in this embodiment.

[0033] Figure 16 This diagram illustrates a series of actions performed by a product image acquisition device to acquire an image of a product.

[0034] Figure 17 This diagram shows examples of rotating platforms being raised to various heights depending on the size of the goods. Detailed Implementation

[0035] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0036] <Overall Structure of the Product Replenishment System>

[0037] First, the overall structure of the product replenishment system in one embodiment of this disclosure will be described. Figure 1 This is a diagram illustrating the overall structure of a merchandise replenishment system according to one embodiment of the present disclosure.

[0038] like Figure 1 As shown, a merchandise replenishment system 1 in one embodiment of this disclosure includes: a merchandise replenishment robot 100 that moves merchandise; a merchandise image acquisition device 200 disposed on the merchandise replenishment robot 100; a management device 300 that manages a schedule or similar data for merchandise replenishment operations performed by the merchandise replenishment robot 100; and a terminal device 400 that is operated by store clerks or others involved in merchandise replenishment.

[0039] A merchandise replenishment robot 100 is installed in convenience stores and other shops to replenish merchandise to the inventory shelf SL20 and to replenish merchandise from the inventory shelf SL20 to the display shelf SL10. A merchandise image acquisition device 200 installed on the merchandise replenishment robot 100 captures images of the merchandise placed on it by the replenishment robot 100. Based on the captured images, the control device 150 of the merchandise replenishment robot 100 determines the type of merchandise placed on the merchandise image acquisition device 200.

[0040] The management device 300 sends merchandise replenishment information to the terminal device 400. This merchandise replenishment information is generated by scheduling the types and quantities of merchandise to be replenished to the inventory shelf SL20 and the timing of replenishment. The merchandise replenishment information includes loading layout information for placing various merchandise on the merchandise replenishment shelf SL30, which is used to transport merchandise to be replenished to the inventory shelf SL20. The management device 300 also causes the merchandise replenishment robot 100 to perform actions based on the merchandise replenishment schedule to replenish merchandise from the merchandise replenishment shelf SL30 to the inventory shelf SL20.

[0041] The terminal device 400 displays the merchandise replenishment information sent from the management device 300 on its display screen, presenting to the store clerk using the terminal device 400 the timing of replenishing merchandise on the inventory shelf SL20, as well as the types and quantities of merchandise to be replenished. The store clerk, following the loading layout information displayed on the terminal device 400 screen, places the specified quantities of various specified merchandise into the designated positions on the merchandise replenishment shelf SL30.

[0042] The store clerk moves the merchandise replenishment rack SL30, which contains replenished goods, to a location accessible to the merchandise replenishment robot 100 at the indicated replenishment time. The merchandise replenishment robot 100 moves the goods from the replenishment rack SL30 to designated positions on the inventory rack SL20 to replenish the inventory rack SL20. At this time, the merchandise replenishment robot 100 may also, as needed, place goods on the merchandise image acquisition device 200 and determine the type of goods to be replenished. During the interval between replenishments to the inventory rack SL20, or after the replenishment to the inventory rack SL20 is completed, the merchandise replenishment robot 100 moves the goods from the inventory rack SL20 to the display shelf SL10 to replenish the display shelf.

[0043] Thus, the merchandise replenishment system 1 of this disclosure replenishes merchandise from the merchandise replenishment shelf SL30 to the storage shelf SL20 according to merchandise replenishment information indicating the type and quantity of merchandise to be replenished to the storage shelf SL20 and the timing of replenishment, and then replenishes merchandise from the storage shelf SL20 to the display shelf SL10.

[0044] <Store's internal layout>

[0045] Next, the configuration structure of the store will be explained. Figure 2 It is a schematic top view showing the shelves and merchandise replenishment robots arranged in the store. Figure 3 (a) is a diagram of the display shelf viewed from the front surface side. Figure 3 (b) is a view of the display shelf from the rear surface side. Figure 4 It is a schematic diagram used to illustrate the functions of the storage rack.

[0046] like Figure 2As shown, the store is divided into a store space SH1 and a warehouse space SH2. Store space SH1 is where customers select and purchase goods T. Warehouse space SH2 is where goods T are stored. The store is equipped with display shelves SL10, inventory shelves SL20, replenishment shelves SL30, a replenishment robot 100, and a product image acquisition device 200 on the replenishment robot 100. A track consisting of two guide rails R is laid on the floor between the display shelves SL10 and the inventory shelves SL20. The replenishment robot 100 is configured to move left and right along these guide rails R as shown in the diagram.

[0047] Product T is, for example, a beverage container or a cylindrical beverage can having a main body Ta and a cap Tb disposed at the upper end of the main body Ta. Product T can be formed from various raw materials such as PET, glass, aluminum, steel, and other metals.

[0048] like Figure 3 As shown in (a) and (b), the display shelf SL10 has multiple shelves SL11 (also referred to as shelves of the display shelf). Multiple types of merchandise T are arranged on the shelves SL11. For example, merchandise T of the same type is arranged in two or three columns. Merchandise T may also be arranged in only one column. Multiple dividers SL12 are provided on the upper surface of each shelf SL11 to separate the merchandise T in each column. Figure 3 The diagram illustrates an example where identical product T is arranged in two columns, with a divider 12 placed every two columns of product T. The configuration of the divider 12 is not limited to this; the divider 12 can be configured with one or more columns at intervals.

[0049] The front surface of the display shelf SL10 faces the store space SH1, allowing customers to retrieve product T from the front surface side of the display shelf SL10. The shelf SL11 is tilted such that the front surface side of the display shelf SL10 is relatively lower than the rear surface side. Thus, when a customer retrieves product T, other products T arranged after that product T slide on the shelf SL11 and move to the front surface side.

[0050] The rear surface of display shelf SL10 faces the warehouse space SH2. A salesperson or a merchandise replenishment robot 100 replenishes merchandise T from the rear surface of display shelf SL10. Although not illustrated, doors can also be installed on the front and rear surfaces of display shelf SL10. Figure 3 For the sake of simplicity, only one display rack SL10 is depicted in the illustration, but multiple display racks SL10 can also be configured in the store.

[0051] Inventory shelf SL20 is configured within warehouse space SH2, facing display shelf SL10, with the front surface of inventory shelf SL20 and the rear surface of display shelf SL10 facing each other. Like display shelf 10, inventory shelf SL20 has multiple shelves (layers) arranged along its height. Goods to be replenished to display shelf SL10, i.e., replenishment targets, are arranged on the shelves of inventory shelf SL20. A store clerk or merchandise replenishment robot 100 replenishes the inventory shelf SL20 from the front surface side.

[0052] The merchandise replenishment rack SL30 has multiple shelves (layers) arranged along its height. Multiple casters are provided at the bottom of the merchandise replenishment rack SL30, enabling it to move in any direction. The merchandise replenishment rack SL30 may also have a label affixed thereto, containing characters, QR codes, or other identification information for identifying the merchandise replenishment rack SL30. The identification information recorded on the label can be read using a camera included with the terminal device 400.

[0053] Here, refer to Figure 4 The structure and function of the storage rack SL20 in this embodiment will be described. Each shelf 21 of the storage rack SL20 is configured to switch between a first posture and a second posture, wherein the first posture is as follows: Figure 4 The second posture, as shown by the solid line, is such that the end of the front surface side (right side of the diagram) is higher than the end of the rear surface side (left side of the diagram). Figure 4 The front surface end is lower than the rear surface end, as shown by the dashed line. The storage rack SL20 includes a drive mechanism (not shown) for switching each shelf SL21 to a first and a second posture. The drive mechanism includes, for example, an electric motor as the drive source. The drive mechanism responds, for example, to the communication unit 195 (see reference 100) provided by the merchandise replenishment robot 100. Figure 7 The control signal sent is used to switch the posture of each shelf SL21 between a first posture and a second posture. Multiple drive mechanisms can be set for each shelf SL21 to drive each shelf SL21 of the storage rack SL20 independently, or a single drive mechanism can be set to drive multiple shelves SL21 together via a connecting mechanism, etc.

[0054] When the merchandise replenishment robot 100 replenishes merchandise onto each shelf SL21 of the inventory rack SL20, the shelf SL21 is positioned in a first posture where the end of the front surface is higher than the end of the rear surface. In this first posture, the shelf SL21 tilts downward from the front surface towards the rear surface. As a result, the merchandise on the front surface of the designated merchandise column placed on the shelf SL21 for replenishment moves towards the rear surface of the shelf SL21 along the tilting motion from the front surface towards the rear surface. Therefore, by repeatedly performing the action of placing merchandise on the front surface of the shelf SL21, merchandise can be sequentially replenished to the merchandise column of the shelf SL21.

[0055] On the other hand, when the merchandise replenishment robot 100 moves the merchandise from the inventory shelf SL20 to the display shelf SL10 to replenish the merchandise in the display shelf SL10, the shelf SL21 of the inventory shelf SL20 is positioned in a second posture where the end of the front surface is lower than the end of the rear surface. In this second posture, the shelf SL21 is tilted downwards from the rear surface towards the front surface. As a result, the merchandise in each column placed on the shelf SL21 slides down toward the front surface of the shelf SL21 along the tilt from the rear surface towards the front surface. Whenever the merchandise at the very front of the front side of the inventory shelf SL20 is removed from the inventory shelf SL20 by the merchandise replenishment robot 100 for replenishment, the merchandise following it moves sequentially toward the front surface of the shelf SL21, so that the merchandise replenishment robot 100 can remove the merchandise at the very front of the front side of the inventory shelf SL20 and move it to the display shelf SL10 for replenishment.

[0056] <Structure and Operation of Product Replenishment Robot 100>

[0057] Reference Figure 2 , Figures 5-7 The following is a description of the product replenishment robot 100. Figure 5 This is a side view schematically showing the structure of the merchandise replenishment robot 100. Figure 6 This is a perspective view showing the peripheral structure of the front end of the arm of the merchandise replenishment robot 100. Figure 7 This is a block diagram showing the structure of the merchandise replenishment robot 100.

[0058] The merchandise replenishment robot 100 includes a gripping part 10, an arm 20, a contact detection sensor 30, first cameras 50R and 50L, a second camera 60, a third camera 70, a horizontal movement mechanism 80, a lifting mechanism 90, and a control device 150. The merchandise replenishment robot 100 is a robot that moves in the space between a display shelf SL10 and a storage shelf SL20. The merchandise replenishment robot 100 performs the following actions: using the gripping part 10 to grip merchandise T on the merchandise replenishment shelf SL30 and move the merchandise T towards its storage position (the channel for storing merchandise T) on the storage shelf SL20; and using the gripping part 10 to grip merchandise T within the storage shelf SL20 and move the gripped merchandise T towards its display position (the channel for displaying merchandise T) on the display shelf SL10.

[0059] like Figure 6 As shown, the gripping portion 10 has a pair of fingers 11a and 11b for gripping an object. The pair of fingers 11a and 11b have a shape capable of gripping the vicinity of the cap Tb of a product T, such as a plastic bottle beverage, and capable of gripping the outer periphery of the product T, which is a canned beverage. In addition to being composed of the pair of fingers 11a and 11b, the gripping portion 10 may also have an adsorption structure for adsorbing and holding the object, or a structure that uses adhesive force, magnetism, or the like to hold the object. Preferably, a flexible holding member (not shown) is provided on the gripping surface of each finger 11a and 11b to better hold the product T when gripping it. The holding member can be formed, for example, from a rubber sheet or a polyurethane resin sheet.

[0060] The arm 20 has multiple linkage components 21, 22, and 23. These linkage components 21, 22, and 23 constitute a multi-joint robotic arm. As an example, a multi-joint robotic arm can also be a six-axis arm with degrees of freedom along the X, Y, and Z axes, and also with degrees of freedom in each of the X, Y, and Z axes. In addition, multi-joint robotic arms can also be orthogonal coordinate system robotic arms, polar coordinate system robotic arms, cylindrical coordinate system robotic arms, SCARA (Selective Compliance Assembly Robot Arm) type robotic arms, and any other mechanism. One end of the arm 20 is fixed to the lifting mechanism 90. A gripping part 10 is provided at the front end of the arm 20. The movement of the arm 20 is controlled by a control device 150.

[0061] The arm 20 can move the gripping part 10 to the storage shelf SL20 side or the display shelf SL10 side by manipulating the various linkage members 21, 22, and 23. Although the orientation of the arm 20 is not fixed to a specific direction, for ease of explanation, the direction in which the linkage members 21, 22, and 23 extend is referred to as the extension direction Ax of the arm 20 (see reference). Figure 6 The arm 20 moves the gripping part 10 toward the product T to grip the product T. The extension direction Ax of the arm 20 corresponds to the forward direction of the gripping part 10 during the gripping action.

[0062] The contact detection sensor 30 is used to detect situations where the gripping part 10 contacts the product T when it is holding the product T, and when the product T held by the gripping part 10 is placed on the shelf SL21 of the storage shelf SL20 or the shelf SL11 of the display shelf SL10, and the product T held by the gripping part 10, the gripping part 10, or the arm 20 contacts obstacles such as the wall or support of the shelf SL10 or SL20. The contact detection sensor 30 can be, for example, a torque sensor, an acceleration sensor, an inertial measurement unit (IMU), or a motor input current sensor.

[0063] As a torque sensor, a strain gauge can be used, for example, to detect the torque generated by the shaft of each joint of the arm 20. As an acceleration sensor, various types of acceleration sensors, such as electrostatic capacitive type and piezoelectric type, can be used and installed on the gripping part 10 or the arm 20.

[0064] An inertial measurement unit (IMU) is a device that detects three-dimensional inertial motion (translational and rotational motion along three orthogonal axes). This IMU includes an accelerometer for detecting translational motion and an angular velocity (gyroscope) sensor for detecting rotational motion. By using the gyroscope sensor to detect angular velocity, the angle or change in angle of an object can be obtained. For example, if gears, or gears and toothed belts, are used as the drive transmission unit for the wrist part of the gripping unit 10, redundancy may occur in the movement of the wrist part of the gripping unit 10 due to gaps between gears, tension of the toothed belt, etc. Therefore, for example, during the action of placing the product T held by the gripping part 10 onto the shelves SL11 and SL21 of the shelves SL10 and SL20, if the arm part 20 is further moved to apply more force to the gripping part 10 while the product T is in contact with the shelves SL11 and SL21, some displacement occurs in the wrist portion of the gripping part 10, and the posture (i.e., angle) of the gripping part 10 changes. Therefore, by using an IMU provided on the gripping part 10 to detect such angle changes that may occur in the gripping part 10 during the action of placing the product T onto the shelves SL11 and SL21, it is possible to detect the situation where the product T contacts the shelves SL11 and SL21.

[0065] Furthermore, in this embodiment, by using an inertial measurement unit (IMU) as a contact detection sensor 30, the pitch angle generated by the grip 10 when lowering its height to bring it into contact with the vicinity of the neck of the plastic bottle beverage (product T) during a series of actions when the grip 10 is used to hold the cap Tb of the plastic bottle beverage (product T), can be detected. This allows for the detection of the relative height position between the plastic bottle beverage (the object to be held) and the grip 10. Moreover, by adjusting the height of the grip 10 from the state where the grip 10 contacts the vicinity of the neck of the plastic bottle beverage, the grip 10 can hold the cap Tb of the plastic bottle beverage at an appropriate height.

[0066] When servo motors or the like are used to drive the joints of the arm 20, if an external force is generated that causes an angular deviation relative to the angle maintaining the gripping posture, the operation is performed to maintain the original angle and keep the angular deviation zero. When the arm 20 is moved further while the product T held by the gripping part 10 is in contact with the shelves SL11 and SL21 of the shelves SL10 and SL20, a current is input to the servo motor to drive it against the load. Therefore, by using a motor input current sensor to detect the current input to the servo motor for this operation, it is possible to detect when the product T is in contact with the shelves SL11 and SL21. The motor input current sensor can be, for example, from the control unit 151 described later (see reference). Figure 7)constitute.

[0067] Two first cameras 50R and 50L are respectively disposed on the left and right sides of the arm 20. The first camera 50L mounted on the first side 23a faces the first direction A1, which is orthogonal to the extension direction Ax of the arm 20. The first side 23a is the left side of the arm 20 (see reference). Figure 6 The first camera 50L is primarily used to film the display shelf SL10. The first camera 50R, mounted on the second side 23b, faces the direction opposite to the first direction A1, i.e., the second direction A2. This second side 23b is a plane parallel to the first side 23a and is the right side of the arm 20 opposite to the first side 23a. The first camera 50R is primarily used to film the storage shelf SL20. Thus, the first cameras 50R and 50L are configured to face opposite directions, allowing the first cameras 50R and 50L to simultaneously film both the display shelf SL10 and the storage shelf SL20 while the arm 20 is held in the same posture.

[0068] The performance of the first cameras 50R and 50L can be the same or different. However, for the sake of simplicity, the following example will be given where the two cameras have the same performance. However, the purpose and shooting conditions for photographing the display shelf SL10 are different from those for photographing the inventory shelf SL20. Therefore, it is of course possible to use cameras with different performance to suit their respective purposes and conditions.

[0069] The first cameras 50R and 50L may, for example, include: an imaging element that generates a camera image (in one example, an RGB image) composed of pixels arranged in a two-dimensional manner; and a depth sensor, which is a distance detection device for generating distance data. The depth sensor is not limited to any particular method as long as it can acquire distance data up to the object. For example, it can utilize a stereo lens or a LiDAR (Light Detection and Ranging) method. The depth sensor may also generate a depth image. In other embodiments of the invention, either or both of the first cameras 50R and 50L may, for example, utilize an ultrasonic element to acquire distance data.

[0070] Furthermore, the first camera 50L faces the first direction A1; however, this refers to the imaging direction of the image sensor and depth sensor of the first camera 50L being direction A1. Similarly, the first camera 50R faces the second direction A2, meaning that the imaging direction of the image sensor and depth sensor of the first camera 50R is direction A2. Directions A1 and A2 do not necessarily need to be 180° opposite; any direction that allows for capturing images of the merchandise display shelf SL10 and the inventory shelf SL20 is acceptable.

[0071] One or both of the first cameras 50R and 50L can be disposed on the holding part 10. The first cameras 50R and 50L do not necessarily need to be disposed on the same component; for example, the first camera 50R can be mounted on one of the connecting members 21-23, and the first camera 50L can be mounted on other components of the connecting members 21-23. However, when the first cameras 50R and 50L are disposed on the same component, compared to the case where each camera 50R and 50L is disposed on different connecting members, the coordinate system is shared, thus simplifying image processing operations.

[0072] The second camera 60 is used to capture the state of the product T held by the gripping part 10, and the relative positional relationship between the product T held by the gripping part 10 and the shelves SL11 and SL21 of the shelves SL10 and SL20. The second camera 60 is also similar to the first cameras 50R and 50L described above, and may, for example, include: an image sensor that generates a two-dimensional image (in one example, an RGB image); and a depth sensor that generates distance data.

[0073] As an example, the second camera 60 is located on the lower side of the link member 23 closest to the gripping part 10 among the link members 21-23 of the arm 20, close to the gripping part 10. The imaging direction of the second camera 60's imaging element and depth sensor is directly below the link member 23 and the gripping part 10. Figure 5 , Figure 6 (in the -z direction) or below and in front (than) Figure 5 , Figure 6 (The -z direction is slightly closer to the +x direction side). Thus, the second camera 60 is able to capture at least the lower part of the product T held by the gripping part 10 and the shelves SL11 and SL21 located in front of the gripping part 10.

[0074] The third camera 70 is, for example, a camera that changes direction based on the operation of a remote operator to capture images of a designated object. As an example, the third camera 70 is mounted as part of the lifting mechanism 90. The third camera 70 can move horizontally and vertically within the space between the display shelf SL10 and the storage shelf SL20, in conjunction with the movement of the horizontal movement mechanism 80 and the lifting mechanism 90. Furthermore, the portion of the lifting mechanism 90 in which the third camera 70 is mounted can rotate around a support column 95, and the third camera 70 is configured to rotate and move left and right around the support column 95 as this portion rotates, thereby capturing images of either the display shelf SL10 or the storage shelf SL20 as needed.

[0075] The third camera 70 may also be a stereo lens camera, for example. Although not limited, the third camera 70 may also have a wider field of view than the first cameras 50R and 50L and the second camera 60.

[0076] The horizontal moving mechanism 80 has a base plate 81 and a drive mechanism (not shown). The base plate 81 supports the lifting mechanism 90 and slides along a guide rail (not shown) laid between the display shelf SL10 and the inventory shelf SL20 in the store. The drive mechanism (not shown) includes a motor, rollers, etc., based on a control device 150 (see reference). Figure 7 The control signal is used to move the lifting mechanism 90 to a specified position along the guide rail.

[0077] The lifting mechanism 90 has a support column 95, a first lifting mechanism 91, and a second lifting mechanism 92. The support column 95 is fixed to the base plate 81 and extends in the vertical direction.

[0078] The first lifting mechanism 91 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., based on the control device 150 (see reference). Figure 7 The first lifting mechanism 91 moves vertically along the support column 95 by actuating the drive mechanism (not shown). The upper part of the first lifting mechanism 91, on which the third camera 70 is mounted, is configured to be rotated and driven in the left-right direction around the support column 95.

[0079] The second lifting mechanism 92 is held by the first lifting mechanism 91. One end of the arm 20 is mounted on the second lifting mechanism 92. The second lifting mechanism 92 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., based on a signal from the control device 150 (see reference 150). Figure 7 The second lifting mechanism 92 moves up and down in the vertical direction by controlling the drive mechanism (not shown).

[0080] When holding a product T at a specified height, the lifting mechanism 90 moves the arm 20 and the holding part 10 to a height where the product T can be held via the first lifting mechanism 91, and then makes a fine adjustment to the height of the arm 20 and the holding part 10 via the second lifting mechanism 92.

[0081] Furthermore, in this embodiment, a first lifting mechanism 91 and a second lifting mechanism 92 are provided as lifting mechanisms, but in other embodiments of the present invention, a structure with only one lifting mechanism may also be provided.

[0082] <Structure of Control Device 150>

[0083] like Figure 7 As shown, the control device 150 includes a control unit 151, a storage unit 160, an input unit 191, an output unit 193, and a communication unit 195. Figure 7 In this context, the control device 150 is depicted as a single element, but the control device 150 does not necessarily need to be a single element physically; it can also consist of multiple physically separate elements.

[0084] The input unit 191 is a device for receiving input from the operator. The input unit 191 can be composed of devices for inputting data into a computer, such as a keyboard, mouse, or touch panel. The input unit 191 may also include a voice input device such as a microphone. Additionally, the input unit 191 may include a gesture input device that recognizes the operator's movements using image recognition.

[0085] The output unit 193 is a device for the merchandise replenishment robot 100 to issue warnings to store clerks, etc., and may consist of one or a combination of a speaker, a display, a light-emitting device, and a vibration device. The communication unit 195 has the function of receiving data from the outside and transmitting data to the outside. In a configuration where the merchandise replenishment robot 100 can be operated remotely, the communication unit 195 receives input from the operator via the operation unit of an external device (not shown), and the control device 150 uses this input to cause the merchandise replenishment robot 100 to perform predetermined actions. Furthermore, the communication between the operation unit of the external device and the communication unit 195 can be either wired or wireless communication.

[0086] In the case of a structure capable of remotely operating the merchandise replenishment robot 100, the operation unit 191 can also be a device worn by an operator. This device includes a display device (not shown) and an operation device (not shown). The display device can also be a head-mounted display (HMD) with a visually recognizable screen for the operator. The operation device can also include, for example, one or more input sensors capable of detecting movements of parts of the operator's body (e.g., hands, arms).

[0087] Storage unit 160 includes transient or non-transient storage media such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). Storage unit 160 stores the computer program executed by control unit 151, the learning completion model (described later), etc. The computer program stored in storage unit 160 includes references... Figure 9 Commands, as described later, for implementing the control method of the merchandise replenishment robot 100 performed by the control unit 151.

[0088] The storage unit 160 includes an acquisition data storage unit 160a and a reference data storage unit 160b. The acquisition data storage unit 160a stores, for example, video image data captured by each camera 50R and 50L, 60, and 70. The reference data storage unit 160b stores various data required for the operation of the product replenishment robot 100. This various data includes, for example, data related to each of the SL10 to SL30 (its shape data, position data, or channel coordinate data, etc.) and data related to the product T (shape data, position data, surrounding image data, etc.).

[0089] The control unit 151 may consist of one or more CPUs (Central Processing Units). The control unit 151 functions as the motion control unit 152, the camera control unit 153, and the image data processing unit 155 by executing computer programs stored in the storage unit 160.

[0090] The motion control unit 152 generates control signals for operating the gripping unit 10, arm 20, horizontal movement mechanism 80, lifting mechanism 90, and control device 150. The motion control unit 152 generates control signals by referring to input signals from the operation unit 191 and various data stored in the storage unit 160. The generation of control signals can also utilize processing results from the image data processing unit 155. The motion control unit 152 also transmits and receives data via the communication unit 195 and performs specified outputs via the output unit 193.

[0091] The camera control unit 153 controls the operation of each camera 50R, 50L, 60, and 70. The timing of each camera's recording, such as the recording time, can be determined, for example, using data pre-stored in the reference data storage unit 160b.

[0092] The image data processing unit 155 uses video image data and distance data (depth data) captured by cameras 50R, 50L, 60, and 70 to perform various information processing. For example, the image data processing unit 155 performs image analysis on the image data captured by the first camera 50R to identify the goods on the inventory shelf SL20 and the replenishment shelf SL30. The image data processing unit 155 includes a replenishment determination unit 155a and a holding object determination unit 155b. The image data processing unit 155 also uses video image data captured by the image acquisition unit 260 of the product image acquisition device 200 (described later) to perform various information processing. For example, the image data processing unit 155 performs the following processing: using image data of the product's periphery captured by the image acquisition unit 260 (including images of the product's surrounding labels, barcodes, etc.) to determine the type of the product.

[0093] Can the determination unit 155a, based on at least one of the video image data and distance data captured by the first camera 50L, determine, for example, whether there is space Sp behind the last item T arranged in the display shelf SL10 and the storage shelf SL20 that can still accommodate other items (refer to...) Figure 8 ). Figure 8 The image is of the display shelf SL10 captured by the first camera 50L of the merchandise replenishment robot 100. If space Sp exists, the merchandise T needs to be replenished. Therefore, the replenishment determination unit 155a sends a notification to the motion control unit 151 indicating that the merchandise T should be replenished to the shelf SL11 below space Sp if space Sp exists. Upon receiving this notification, the motion control unit 151 performs the replenishment operation for the merchandise T.

[0094] The object-holding determination unit 155b performs at least one of the following processes based on at least one of the camera image data and distance data captured by the first camera 50R: determining whether a replenishment object product, which is an object to be held, exists in the inventory shelf SL20 and the replenishment shelf SL30; determining the size or shape of the replenishment object product; and determining the holding position of the replenishment object product. Figure 2 In the case of a product T with a lid Tb as shown, the gripping object determining unit 155b sets the gripping position, for example, near the lid Tb. On the other hand, if the product T is a canned beverage or the like without a lid, the gripping object determining unit 155b may also set the side of the container as the gripping position.

[0095] (The replenishment of goods by the replenishment robot 100)

[0096] Next, the product replenishment operation performed by the product replenishment robot 100 will be explained. An example of replenishing product T from inventory shelf SL20 to display shelf SL10 will be described below. Figure 9 This is a flowchart illustrating the replenishment action of product T performed by the product replenishment robot 100.

[0097] First, in step S11, a first camera 50L mounted on the arm 20 of the merchandise replenishment robot 100 captures an image of the rear surface of the display shelf SL10. The merchandise replenishment robot 100 moves the arm 20, the horizontal movement mechanism 80, and the lifting mechanism 90 so that the first camera 50L can capture images of each layer of the display shelf SL10. Next, the merchandise replenishment robot 100 moves the first camera 50L to acquire an image of the rear surface of the display shelf SL10 and acquires distance data up to the merchandise T arranged on the display shelf SL10.

[0098] Next, in step S12, a determination is made as to whether the item can be replenished. This "determination of replenishment capability" involves the following steps: The replenishment determination unit 155a in the image data processing unit 155 of the replenishment robot 100 analyzes the captured image of the back surface of the display shelf to determine which item T can be displayed on which shelf SL11 (in other words, which item T needs replenishment). The replenishment robot 100 acquires... Figure 6 The image on the back surface of that display shelf SL10.

[0099] The image data processing unit 155 of the merchandise replenishment robot 100 determines the aisle on the shelf SL11 of the display rack SL10, which serves as the display destination, based on the video data captured by the first camera 50L. Furthermore, the image data processing unit 155 of the merchandise replenishment robot 100 is able to acquire distance data up to the merchandise T (in... Figure 8 The depth data (Dep) is displayed in a visual manner. Therefore, in cases where the number of items T displayed is reduced, such as in aisles "7" and "8", it is possible to identify the existence of space Sp behind item T. The replenishment robot 100's replenishment determination unit 155a determines whether there is space Sp for item T based on the distance data up to item T, thereby determining whether item T can still be displayed in aisles "7" and "8". Through the process of step S12, it is determined which item T needs to be replenished on which shelf SL11 of the display rack SL10.

[0100] Furthermore, the method for determining whether a product needs to be replenished is not limited to the methods described above, and various methods can be used. For example, it can be determined that product T is configured at approximately a certain percentage of its occupancy within a specified three-dimensional space, and if this value is below a specified baseline value, then product T needs to be replenished.

[0101] Next, in step S13, the inventory shelf SL20 is photographed. The merchandise replenishment robot 100 moves its arm 20, horizontal movement mechanism 80, lifting mechanism 90, and first camera 50R to photograph the inventory shelf SL20 from the front surface side. As an example, the merchandise replenishment robot 100 photographs the inventory shelf SL20 layer by layer to obtain video images representing the inventory status of merchandise T. It is not necessary to photograph the inventory shelf SL20 after photographing the display shelf SL10; the inventory shelf SL20 can also be photographed before photographing the display shelf SL10.

[0102] Next, in step S14, the image data processing unit 155 of the merchandise replenishment robot 100 analyzes the camera image of the inventory shelf SL20 acquired in step S13 to identify the merchandise arranged in the inventory shelf SL20. Additionally, the grasping object determination unit 155b of the image data processing unit 155 determines the grasping position of the merchandise. Through these processes, the merchandise replenishment robot 100 acquires information indicating which merchandise T needs to be replenished on which shelf SL11 of the display shelf SL10, and information indicating the location of the corresponding replenishing object merchandise on the inventory shelf SL20 and the grasping position of that replenishing object merchandise.

[0103] Next, in step S15, the merchandise replenishment robot 100 performs merchandise replenishment actions (pick-up and placement actions) based on the acquired information. Specifically, the control unit 151 of the merchandise replenishment robot 100 (see...) Figure 7 The motion control unit 152 actuates the arm 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripping unit 10 toward the designated replenishment item on the storage shelf SL20. Then, the gripping unit 10 grasps the replenishment item at a predetermined gripping position (the cap Tb in the case of a plastic bottle beverage) and picks up the replenishment item. Afterward, the motion control unit 152 actuates the arm 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripped item T to the designated placement position on the display shelf SL10 and place the item T from the gripping unit 10, thereby placing the item T in the designated position. Thereafter, the replenishment robot 100 repeatedly performs the same picking and placing actions to replenish the item T.

[0104] Furthermore, while the example described above of replenishing product T from inventory shelf SL20 to display shelf SL10 was explained, the replenishment operation of replenishing product T from replenishment shelf SL30 to inventory shelf SL20 is also performed using the same process. In this case, as an example, the image data processing unit 155 of the replenishment robot 100 determines the aisle on shelf SL21 of inventory shelf SL20, which serves as the destination for replenishment, based on the image data captured by the first camera 50L. Additionally, the image data processing unit 155 of the replenishment robot 100 determines the position of each product placed on replenishment shelf SL30 based on the image data captured by the first camera 50R, capturing the front surface of replenishment shelf SL30. When replenishing product T from the replenishment shelf to the inventory shelf SL20, the shelf SL21 of the inventory shelf SL20 is initially set in a first position where the end of the front surface side (right side of the figure) is higher than the end of the rear surface side (left side of the figure), and after replenishment is completed, it is set in a second position where the end of the front surface side is lower than the end of the rear surface side.

[0105] <Structure of the Product Image Acquisition Device 200>

[0106] Figure 10 This is a perspective view showing the overall structure of the product image acquisition device 200 included in the product replenishment system 1 of this embodiment. The product image acquisition device 200 includes a rotary table 210, a rotation drive unit 220, a lifting mechanism 230, a lifting drive mechanism 240, a lifting drive unit 250, and an image acquisition unit 260. Furthermore, Figure 11 This is a diagram showing details of the structure of each part of the product image acquisition device 200. Figure 11 (a) is a top view. Figure 11 (b) is a side view showing the rotary table 210 positioned at its lowest position by the lifting mechanism 230. Figure 11 (b) is a side view showing the state in which the rotary table 210 is configured at its highest position by the lifting mechanism 230.

[0107] The rotary table 210 includes: a rotary worktable 212 having a circular circumference; and a receiving portion 214 that supports a rotary drive unit 220 for rotating the rotary worktable 212 and receives the circumference of the rotary worktable 212. The rotary worktable 212 is configured to rotate about a central axis and to rotate about the central axis together with the goods placed on it.

[0108] The rotary table 210 is also configured to allow the rotary worktable 212 to move between a first position and a second position. The first position is where the rotary worktable 212 is slightly protruding upwards from the rotary table 210 due to a force applied by a spring or other force-applying member. The second position is where the rotary worktable 212 is pressed in from the first position against the force applied by the force-applying member and comes into contact with the rotary table 210. Thus, when the product is not in contact with the rotary worktable 212, the rotary worktable 212 is positioned in the first position; when the product is in contact with or placed on the rotary worktable 212, the rotary worktable 212 is positioned in the second position. A contact switch (not shown) is provided between the rotary table 210 and the rotary worktable 212. When the rotary worktable 212 is in the first position, the contact switch is open; when the rotary worktable 212 is in the second position, the contact switch is closed.

[0109] The rotary drive unit 220 is supported and fixed to the lower surface of the receiving portion 214 of the rotary table 210, and supports the rotary table 212, which is housed in the receiving portion 214, in a manner that allows the rotary table 212 to rotate around its central axis. The rotary drive unit 220 includes a mechanism for rotating the rotary table 210 and a drive source, such as an electric motor. The rotary table 210 can be configured to be directly connected to the rotating shaft of the electric motor, which serves as the drive source, and thus rotated, or it can be configured to transmit the rotational driving force of the electric motor to the rotary table 212 via a drive transmission unit such as gears to rotate the rotary table 212.

[0110] The lifting mechanism 230 is a mechanism for raising and lowering the rotary table 210, and is, for example, composed of a scissor-type linkage mechanism that extends and retracts in the vertical direction. The linkage mechanism of the lifting mechanism 230 is configured by connecting portions 234 that allow multiple linkage members 232 to rotate relative to each other. In this example, the linkage mechanisms, consisting of four linkage members 232 connected at both ends and in the middle by the connecting portions 234, are grouped in pairs, connecting the rotary table 210 to the lifting drive mechanism 240 on both sides. The upper ends of the two upper linkage members 232 of the linkage mechanism are connected to the lower surface of the receiving portion 214 of the rotary table 210 via the connecting portions 234, and the lower ends of the two lower linkage members 232 of the linkage mechanism are connected to the lifting drive mechanism 240 via the connecting portions 234. With this structure, the lifting mechanism 230 can be configured such that the rotary table 210 is positioned at its lowest position. Figure 11 (b) and the method of positioning the rotary table 210 at its highest position ( Figure 11 The deformation occurs by stretching or contracting between (c) and (c).

[0111] The lifting drive mechanism 240 is a drive mechanism for extending and retracting the lifting mechanism 230 to raise and lower the rotary table 210. For example, it is configured to reciprocate the lower end of one of the lower connecting rod members 232 of the scissor mechanism of the lifting mechanism 230 in a direction closer to the lower end of the other connecting rod member 232 (first direction) and in a direction farther from the other end (second direction). More specifically, the lifting drive mechanism 240 includes: a slider 242, which is connected to the lower end of the lower connecting rod member 232 of each scissor mechanism of the lifting mechanism 230 via a connecting portion 234; a slider drive portion 244, which reciprocates the slider 242 in the aforementioned first and second directions; and a guide rail 246, which supports the slider 242 in such a way that the slider 242 can reciprocate in the aforementioned first and second directions. As an example, the slider drive unit 244 has a shaft member 244a with an external thread formed on its outer peripheral surface, and an internal thread formed on the slider 242 through which the external thread passes. This configuration allows the slider 242 to move in a first direction (e.g., to the right in the diagram) if the shaft member 244a is rotated in one direction, and to move in a second direction (e.g., to the left in the diagram) if the shaft member 244a is rotated in another direction. The lifting drive mechanism 240 also functions as a base member for mounting the rotary table 210, the rotary drive unit 220, the lifting mechanism 230, and the lifting drive unit 250.

[0112] The lifting drive unit 250 includes a drive source for rotating the shaft member 244a of the lifting drive mechanism 240, such as an electric motor. The lifting drive mechanism 240 can be configured to directly connect the rotating shaft of the electric motor, which serves as the drive source, to the shaft member 244a of the lifting drive mechanism 240 to rotate the shaft member 244a, or it can be configured to transmit the rotational driving force of the electric motor to the shaft member 244a of the lifting drive mechanism 240 via a drive transmission unit such as gears to rotate the shaft member 244a.

[0113] The electric motor, which serves as the drive source for the lifting drive unit 250, is configured to switch the rotation direction between one rotation direction and another, and is also configured to appropriately change the rotational speed in each rotation direction. Thus, the electric motor, as the drive source for the lifting drive unit 250, can cause the slider 242 to move in a first direction (e.g., to the right in the figure) by rotating the shaft member 244a of the lifting drive mechanism 240 in one rotation direction, thereby extending the lifting mechanism 230 and raising the rotary table 210. Alternatively, it can cause the slider 242 to move in a second direction (e.g., to the left in the figure) by rotating the shaft member 244a in the other rotation direction, thereby retracting the lifting mechanism 230 and lowering the rotary table 210, and can change the speed of their ascent and descent.

[0114] The main body of the product image acquisition device 200, which consists of a rotary table 210, a rotary drive unit 220, a lifting mechanism 230, a lifting drive mechanism 240, and a lifting drive unit 250, is mounted on the base plate 81 of the product replenishment robot 100, for example. Furthermore, the rotary drive unit 220 and the lifting drive unit 250 are connected to the communication unit 195 of the control device 150 of the product replenishment robot 100 via a wired or wireless connection, and the control device 150 performs drive control on these drive units (e.g., electric motors).

[0115] The image acquisition unit 260 has the function of acquiring an image of the outer periphery of a product placed on the rotary table 210, and includes, for example, a camera element that generates a camera image (an RGB image in one example) with pixels arranged in a two-dimensional manner. The image acquisition unit 260 is provided, for example, on the lifting mechanism 90 that forms the housing of the product replenishment robot 100.

[0116] While an item T is being held by the gripping unit 10 of the merchandise replenishment robot 100 before the image acquisition unit 260 (at the image acquisition position of the image acquisition unit 260), or while an item placed on the turntable 210 is being rotated by the turntable 210 via the rotation drive unit 220, the image acquisition unit 260 captures an image of the item. Thus, the image acquisition unit 260 captures an image of the side or outer periphery of the item (including images of the surrounding product labels, barcodes, etc.) and acquires its image data. The image acquisition unit 260 transmits the acquired image data to the control device 150 via the communication unit 195 of the control device 150 of the merchandise replenishment robot 100. The image data processing unit 155 of the control device 150, upon receiving the image data of the item, performs the following processing as described above: using the image data of the side or outer periphery of the item (including images of the surrounding product labels, barcodes, etc.), it determines the type of the item. In this way, the product image acquisition device 200 and the control device 150 function as a product identification system for identifying the type of product held by the product replenishment robot 100. Furthermore, by mounting the product image acquisition device 200 on the product replenishment robot 100, the product can be positioned at the image acquisition position of the image acquisition unit 260 simply by moving the arm 20 after the product replenishment robot 100 has grasped the product.

[0117] <Structure of Management Device 300>

[0118] like Figure 12 As shown, the management device 300 includes a processor 310, a storage unit 320, an input unit 330, a display unit 340, and a communication unit 350. Figure 12In this context, the management device 300 is depicted as a single element, but the management device 300 does not necessarily need to be a single element physically; it can also consist of multiple physically separate elements.

[0119] The input unit 330 is a device for receiving input from the operator of the management device 300. The input unit 330 can be composed of a keyboard, mouse, or touch panel. The display unit 340 is a display device that displays the screen generated by the processor 310; for example, it can be a liquid crystal display (LCD) or an organic EL display. The communication unit 350 communicates with the control device 150 and terminal device 400 of the merchandise replenishment robot 100 via wired or wireless communication, and performs input and output of information, control signals, etc., with these structural elements.

[0120] Storage unit 320 includes transient or non-transitory storage media such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive). Storage unit 320 stores the computer program executed by processor 155. The computer program stored in storage unit 320 includes references... Figure 14 Commands for implementing information processing performed by processor 310, as described later. Storage unit 160 also temporarily stores information received from control device 150 and terminal device 400, as well as various data (including intermediate generated data) generated by processing actions performed by processor 310.

[0121] Storage unit 320 also stores, for example, past sales data for each product in each store. The sales data includes information such as the sales date, time, and quantity of each product. This sales data can be used as statistical information to predict which product should be replenished to the display shelves SL10 and inventory shelves SL20 in each store according to which schedule. Storage unit 320 may also store a learned model generated by a learner using such sales data to perform machine learning. Such a learned model can be used as a sales forecasting model for generating a schedule for replenishing products in the store.

[0122] Processor 310 may consist of one or more CPUs (Central Processing Units). Processor 310 performs operations to achieve the reference. Figure 14To explain the processing performed by such processor 310 and its functions. In particular, processor 310 uses past sales data or sales forecasts stored in storage unit 320 to learn a model to predict which product should be replenished to the display shelves SL10 and inventory shelves SL20 of each store according to which time schedule, and based on the prediction, schedules the types and quantities of products to be replenished to inventory shelves SL20 and the timing of the replenishment, and generates product replenishment information.

[0123] The merchandise replenishment information relates to the merchandise that should be replenished from the merchandise replenishment rack SL30 to the inventory rack SL20 at a given replenishment time. This includes loading layout information, which specifies the loading position and quantity of each merchandise in its respective area on the merchandise replenishment rack SL30. The loading layout information may also include rack designation information for specifying the merchandise replenishment rack SL30 on which the merchandise should be loaded.

[0124] The generated merchandise replenishment information is sent from the communication unit 350 to the control device 150 and the terminal device 400. The terminal device 400, as the user of the terminal device 400, displays the received loading layout information on its display screen. The store clerk, according to the loading layout information displayed on the terminal device 400, places a specified number of specified various merchandise items in the designated areas of the specified merchandise replenishment shelf SL30.

[0125] <Structure of Terminal Device 400>

[0126] like Figure 14 As shown, the terminal device 400 includes a processor 410, a storage unit 420, a display operation unit 430, a communication unit 440, and a camera 450. Furthermore, this structure is one example, and other structures may also be included.

[0127] The processor 410 controls the overall operation of the terminal device 400 by executing a program stored in the storage unit 420. The processor 410 comprises a CPU, GPU, etc. The processor 410 performs actions to achieve the reference... Figure 14 To explain the processing performed by such processor 410 and its functions.

[0128] The storage unit 420 stores programs, input data, etc., used for various control processes and for the processor 410 to execute various functions. The storage unit 420 is composed of RAM, ROM, and other storage devices. Additionally, the storage unit 420 temporarily stores communication content with other terminals. Specifically, the storage unit 420 stores information for causing the processor 410 to execute references... Figure 14 The procedure for processing performed by the terminal device 400 is explained below.

[0129] The display operation unit 430 is a user interface for providing user input instructions to the terminal device 400 and displaying text, images, etc. according to the display processing of the processor 410. When the terminal device 400 is composed of a tablet terminal or the like, the display operation unit 430 is composed of a touch panel or the like.

[0130] The communication unit 440 is a communication interface used to communicate with the control device 150 and the management device 300 via a network.

[0131] The camera 450 is primarily used to read labels affixed to the merchandise refill shelf SL30. These labels contain characters, QR codes, and other identification information for recognizing the merchandise refill shelf SL30. The storage unit 420 also stores a character recognition program for recognizing the characters and QR codes read by the camera 450, and a program for deciphering the codes read by the camera 450.

[0132] The processor 410 causes the display operation unit 430 to display the merchandise replenishment information received from the management device 300. The store clerk, according to the loading layout information displayed on the terminal device 400, places a specified quantity of various merchandise at designated positions on the designated merchandise replenishment rack SL30. The store clerk uses the camera 450 of the terminal device 400 to photograph the identification tags attached to the designated merchandise replenishment rack SL30. The processor 410 transmits the image data of the identification tags captured by the camera 450 to the management device 300 via the communication unit 440.

[0133] (Layout information)

[0134] Here, refer to Figure 14 This describes the loading layout information, which includes the loading position and quantity of each item in each area of ​​the merchandise replenishment shelf SL30. Figure 14 This is an example of a display screen showing the layout information displayed on the display operation unit 430 of the terminal device 400.

[0135] The placement layout information displayed on the display operation unit 430 shows the areas divided into on each shelf of the merchandise replenishment rack SL30, which is the object of merchandise placement. Figure 14 The example shown represents the types of goods and their quantities in regions 1 to 10.

[0136] Shop staff retrieve various goods stored in the shop's designated storage area (usually at room temperature) and process them according to... Figure 14The placement layout information displayed on the display operation unit 430 of the terminal device 400, as shown, places a specified number of specified items into designated areas on the product replenishment shelf SL30. For example, when placing a certain item in a box of delivery packaging into designated areas, the top and front surfaces of the delivery packaging can be removed, and the remaining portion of the delivery packaging can be placed into designated areas on the product replenishment shelf SL30 according to each item inside (e.g., ...). Figure 14 (As shown in areas 1-6, 8). Alternatively, sometimes multiple goods may be mixed and displayed in a designated area (e.g., Figure 14 Area 7 is shown.

[0137] If various products are placed on the merchandise replenishment rack SL30 according to the placement layout information, the merchandise placement operation on the merchandise replenishment rack SL30 is completed, and the merchandise replenishment rack SL30 can be placed in the designated position in the store.

[0138] In addition, such as Figure 14 As shown, in the display screen of the display operation unit 430 on the terminal device 400 displaying the layout information, the display area of ​​each product can be selected by clicking on the display operation unit 430. For example, if the product specified by the layout information does not exist in the store's storage area, if only a smaller quantity of the product is stored, or if the specified product is out of stock, the store clerk can click on the product in the display screen of the layout information on the display operation unit 430 to select it. Then, the processor 410 of the terminal device 400 displays a column for handling quantity changes for that product on the display operation unit 430 to handle numerical input on the display operation unit 430. If the store clerk inputs a value for the number of products that can be displayed on the display operation unit 430, the processor 410 of the terminal device 400 sends the numerical information for that product to the management device 300. When the processor 310 of the management device 300 receives numerical information for the product, it updates the number of items for the product in the placement layout information according to the value, and sends supplementary product information containing the updated placement layout information to the control device 150.

[0139] According to the merchandise replenishment system disclosed herein, the management device 300 generates a layout of various merchandise and their quantities on the replenishment shelf SL30 (used for replenishing merchandise to the inventory shelf SL20) based on a merchandise replenishment plan for the inventory shelf SL20, and presents this layout to store clerks via the terminal device 400. Store clerks place various merchandise on the replenishment shelf SL30 according to the layout displayed on the display operation unit 430 of the terminal device 400, and position the replenishment shelf SL30 in a designated location within the store. Then, the merchandise replenishment robot 100, controlled by the control device 150, moves the various merchandise placed on the replenishment shelf SL30 sequentially to the corresponding merchandise column on the inventory shelf SL20, replenishing merchandise from the replenishment shelf SL30 to the inventory shelf SL20. After replenishing goods from replenishment shelf SL30 to inventory shelf SL20, the replenishment robot 100 resumes replenishing goods from inventory shelf SL20 to display shelf SL10 until the next scheduled replenishment from replenishment shelf SL30 to inventory shelf SL20 begins. Thus, according to the replenishment system disclosed herein, in addition to replenishing goods from inventory shelf SL20 to display shelf SL10, the replenishment robot can also replenish goods from inventory shelf SL20.

[0140] Here, as described above, store clerks place various goods onto the replenishment shelf SL30 according to the placement layout displayed on the display operation unit 430 of the terminal device 400. However, if goods different from those indicated in the placement layout are placed, or if the indicated goods are placed in an area of ​​the replenishment shelf SL30 that is different from the indicated area, and replenishment from the replenishment shelf SL30 to the inventory shelf SL20 is performed without confirming these goods, goods different from those that should have been replenished from the inventory shelf SL20 to the display shelf SL10 will ultimately be replenished. This disclosure provides a solution to address such adverse situations.

[0141] <Action Example>

[0142] Next, an example of the operation of the image acquisition device 200 and the control device 150, which function as a product identification system for identifying the type of product held by the product replenishment robot 100 in this embodiment, will be described. Figure 15 This is a flowchart illustrating an example of the operation of the image acquisition device 200 and the control device 150, which function as a product recognition system in this embodiment. Additionally, Figure 16 This diagram illustrates a series of actions performed by a product image acquisition device to acquire an image of a product.

[0143] First, in step S21, when the control device 150 receives an instruction from the management device 300 to cause the merchandise replenishment robot 100 to perform the action of replenishing merchandise from the merchandise replenishment shelf SL30 located in a specified position in the store to the inventory shelf SL20, the control device 150 causes the merchandise replenishment robot 100 to move the various merchandise placed on the merchandise replenishment shelf SL30 to the corresponding merchandise column on the inventory shelf SL20 in sequence, thereby starting the merchandise replenishment action from the merchandise replenishment shelf SL30 to the inventory shelf SL20.

[0144] In the merchandise replenishment system 1 of this embodiment, when a store clerk places various merchandise on the merchandise replenishment rack SL30 according to the placement layout displayed on the display operation unit 430 of the terminal device 400 and positions the merchandise replenishment rack SL30 at a predetermined location within the store, the store clerk operates the display operation unit 430 of the terminal device 400 to send a completion confirmation message indicating that the merchandise replenishment rack SL30 has been positioned at the predetermined location within the store from the terminal device 400 to the management device 300. Upon receiving the completion confirmation message, the management device 300 sends an instruction to the control device 150 of the merchandise replenishment robot 100 to perform the following action: replenish the merchandise on the merchandise replenishment rack SL30, positioned at the predetermined location within the store, to the inventory rack SL20. Then, the merchandise replenishment robot 100, controlled by the control device 150, moves the various merchandise placed on the merchandise replenishment rack SL30 sequentially to the corresponding merchandise column on the inventory rack SL20, performing the merchandise replenishment action from the merchandise replenishment rack SL30 to the inventory rack SL20.

[0145] Regarding the replenishment of goods from the replenishment shelf SL30 to the inventory shelf SL20, for example, it can be done by referring to... Figure 14 The described placement layout can be implemented by sequentially replenishing all items in each area, or by repeatedly replenishing one or more items in each area of ​​the placement layout. To explain the latter more specifically, as an example, the replenishment action of three items in each area from area 1 to area 8 can be performed repeatedly until the items on the replenishment shelf SL30 disappear.

[0146] Next, in step S22, the control device 150 causes the merchandise replenishment robot 100 to perform the following action: placing the merchandise held and taken out by the merchandise replenishment robot 100 from the merchandise replenishment rack SL30 at a position in front of the image acquisition unit 260 where an image can be acquired.

[0147] like Figure 16As shown in (a), the product T, held by the gripping part 10 of the product replenishment robot 100 and taken out from the product replenishment rack SL30 in a gripping manner, is positioned above the rotary table 210 of the product image acquisition device 200, and the image acquisition part 260 of the product image acquisition device 200 is able to capture the height position of the side of the product T. Thus, an image of the product T held by the product replenishment robot 100 can be acquired by the image acquisition device 200.

[0148] Next, in step S23, the control device 150 performs the following process: causing the image acquisition device 200 to acquire an image (first image) of one side of the product held by the product replenishment robot 100 and positioned in front of the image acquisition device 200.

[0149] The control device 150 causes the image acquisition unit 260 to capture an image (first image) of the side of the product T that is held by the product replenishment robot 100 and positioned in front of the image acquisition device 200, facing the image acquisition unit 260. (Refer to...) Figure 16 (a) Therefore, in step S23, the image of product T captured by the image acquisition unit 260 in step S23 is an image of the side of product T facing the image acquisition unit 260, and images obtained from viewing product T from other directions (side or back) are not acquired. The image data of product T captured by the image acquisition unit 260 (first image data) is transmitted to the control device 150 via the communication unit 195 of the control device 150 of the product replenishment robot 100 as described above.

[0150] Next, in step S24, the control device 150 performs a process to determine the type of the product T (first product identification process) based on the first image data received from the image acquisition unit 260 (the image of the side of the product T facing the image acquisition unit 260, i.e., the first image), and determines whether the determination of the type of the product T based on the first product identification process is successful.

[0151] As an example, the type of product T determined by the control device 150 can be achieved by comparing image data sent to the control device 150 with image data of various products stored in the storage unit 160 of the control device 150 through image analysis processing. The surrounding image data of various products can include multiple images of each product T taken from various directions, a circumferential view of the surrounding area, etc. Alternatively, the control device 150 can use a learned model to determine the type of product T, which is generated by machine learning using the aforementioned image data of various products by any learner. The learned model is stored in the storage unit 160 of the control device 150.

[0152] The control device 150 can, for example, determine whether the determination of the type of product T based on the first product recognition process is successful based on a matching score. This matching score is calculated in image analysis processing that compares the image data of the product T to be determined with the image data of the reference product, or in a determination process using a learned model. For example, the control device 150 determines that the type of product T is successfully determined (yes) if the matching score is above a predetermined value, and determines that the type of product T is determined to be unsuccessfully determined (no) if the matching score is below a predetermined value. If the control device 150 determines that the type of product T is successfully determined (yes), the product recognition process ends; if the control device 150 determines that the type of product T is determined to be unsuccessfully determined (no), the process proceeds to step S25.

[0153] In the first product recognition process of step S24, the following processing is performed: an image of a side of a product T held by the gripping part 10 of the product replenishment robot 100 and positioned opposite the image acquisition unit 260 is captured, and the type of product T is determined based on the image data. Which part of the side of the product T held by the gripping part 10 faces the image acquisition unit 260 is arbitrary when it is captured; therefore, depending on the situation, it is possible that the side of the product T captured is not the front of the label, and thus sometimes the product T cannot be determined by the control device 150. In this case, a second product recognition process based on steps S25 and S26, described later, is performed to acquire image data of the entire circumference of the side of the product T.

[0154] Next, in step S25, the control device 150 performs the following process: the image acquisition unit 260 acquires an image (second image) of the entire side circumference of the product that has been held and positioned in front of the image acquisition unit 260 by the product replenishment robot 100.

[0155] In step S25, firstly, if the product T, which was positioned before the image acquisition unit 260 in step S22 and capable of acquiring images, is not placed on the rotary table 210 of the product image acquisition device 200, that is, if the bottom surface of the product T does not contact the rotary worktable 212 of the rotary table 210, the control device 150 drives the lifting drive unit 250 to rotate the shaft member 244a of the lifting drive mechanism 240 in one rotation direction, causing the slider 242 to move in the first direction ( Figure 11The lifting mechanism 230 moves (as shown on the right side of the diagram), thereby causing the lower end of one of the lower links 232 of the scissor mechanism of the lifting mechanism 230 to move towards the lower end of the other link 232 (in the first direction). This causes the lifting mechanism 230 to extend, raising the rotary table 210 to a height where the rotary worktable 212 of the rotary table 210 contacts the bottom surface of the product T. Thus, the bottom surface of the product T is placed on the rotary table 210. When the rotary worktable 212 of the rotary table 210 contacts the bottom surface of the product T, the rotary worktable 212, as described above, resists the force applied by the force-applying member and is displaced from the first position to the second position to activate the contact switch, thereby enabling the detection that the rotary table 210 is in contact with the bottom surface of the product T.

[0156] It should be noted here that the extent to which the lifting mechanism 230 extends, or in other words, how much the turntable 210 rises, depends on the size of the product T held and positioned above the turntable 210 by the product replenishment robot 100. Figure 17 This diagram illustrates examples of the rotating platform 210 being raised to various heights according to the size of the product T. (Example) Figure 17 As shown in the figures, the height of the upper end of the product T held by the holding part 10 is constant regardless of the size of the product T. In the example shown, the product T is arranged in front of the image acquisition part 260 such that the height of the upper end of the product T is approximately the same as the height of the upper end of the image acquisition part 260.

[0157] Figure 17 (a) shows the state in which the rotating worktable 212 of the rotating platform 210 is in contact with the bottom of a relatively large product T, such as a plastic bottle container with a capacity of 2000 ml. In this example, the rotating worktable 212 is in contact with the bottom of the product T and thus is in a state where the product T is placed, simply by using the lifting mechanism 230 to slightly raise the rotating platform 210 from its lowest position. Figure 17 (b) shows the state in which the rotating worktable 212 of the rotary table 210 is in contact with the bottom of a relatively small product T, such as a 500ml plastic bottle container. In this example, the rotary table 210 is moved from a height of 210 to a height of 212 by means of the lifting mechanism 230. Figure 17 The height shown in (a) is further increased, so that the rotating worktable 212 comes into contact with the bottom of the product T and becomes a state in which the product T is placed. Figure 17 (c) shows the state in which the rotating worktable 212 of the rotary table 210 is in contact with the bottom of a small product T, such as a can with a capacity of 180 ml. In this example, the rotary table 210 is moved from a height of 100 ml using the lifting mechanism 230. Figure 17 The height shown in (b) is further increased, so that the rotating worktable 212 comes into contact with the bottom of the product T and becomes a state in which the product T is placed.

[0158] Furthermore, when the product T held by the gripping unit 10 of the product replenishment robot 100 is moved above the rotary table 210 and the image acquisition unit 260 is able to acquire the height position of the image, and the bottom surface of the product T is placed on the rotary table 210, the operation is not performed. Figure 16 (b) and Figure 17 The operation is shown. Additionally, when the rotary table 210 is positioned higher than the height of its bottom surface when the product T is moved to a height position where the image acquisition unit 260 can acquire an image, the control device 150 drives the lifting drive unit 250, causing the shaft member 244a of the lifting drive mechanism 240 to rotate in a direction opposite to the aforementioned rotation direction, thus causing the slider 242 to move in the second direction (…). Figure 11 The left side of the diagram is moved, thereby causing the lower end of the lower link member 232 of the scissor mechanism of the lifting mechanism 230 to move away from the lower end of the other link member 232 (second direction), causing the lifting mechanism 230 to retract. After the rotary table 210 is temporarily lowered, the lifting mechanism 230 is extended, causing the rotary table 210 to rise to the height at which the rotating worktable 212 of the rotary table 210 contacts the bottom surface of the product T.

[0159] When the product T is placed on the rotary table 210 as described above, the control device 150 then widens the spacing of the fingers 11 of the gripping part 10 to temporarily release the grip of the product T, thereby driving the rotation drive unit 220 to rotate the rotary worktable 212 of the rotary table 210. Then, the control device 150 causes the image acquisition unit 260 to capture an image (second image) of the periphery of the side of the product T as it rotates together with the rotary worktable 212. For example, the image acquisition unit 260 can be controlled by the control device 150 to capture an image of the periphery of the side of the product T only during the time it takes for the rotary worktable 212 to rotate one revolution. The image (second image) of the periphery of the side of the product T captured in this way is an image of a circle around the side of the product T, which can be said to be equivalent to peeling off a label pasted around the side of the product T and unfolding the label flat. Image data (second image data) of the side and surrounding area of ​​the product T captured by the image acquisition unit 260 is transmitted to the control unit 150 via the communication unit 195 of the control unit 150 of the product replenishment robot 100, as described above.

[0160] Next, in step S26, the control device 150 performs a process (second product identification process) to determine the type of product T based on the second image data (image data of the image around the side of the product T (second image)) received from the image acquisition unit 260, and determines whether the determination of the type of product T based on the second product identification process is successful.

[0161] Similar to the first product identification process in step S24 described above, in the second product identification process in step S26, for example, image analysis processing and the aforementioned learned model can also be used to determine the type of product T by the control device 150. In this image analysis process, the image data sent to the control device 150 is compared with the image data of various products stored in the storage unit 160 of the control device 150. The aforementioned learned model is generated by machine learning using image data of various products by an arbitrary learner. The second product identification process determines the type of product T based on images surrounding the side of product T. Therefore, compared to the first product identification process which determines the type of product T based on images obtained by only capturing one side view, the accuracy of product identification is higher.

[0162] The control device 150 can, for example, determine whether the product type determination based on the second product recognition process is successful based on a matching score. This matching score is calculated in image analysis processing that compares the image data of the product T to be determined with the image data of the control object, or in a determination process using a learned model. For example, the control device 150 determines that the product type determination is successful (Yes) if the matching score is above a predetermined value, and determines that the product type determination is unsuccessful (No) if the matching score is below a predetermined value. The predetermined value of the matching score used for determination in the second product recognition process can be the same as or different from the predetermined value of the matching score used for determination in the first product recognition process. If the control device 150 determines that the product type determination is successful (Yes), the product recognition process ends; if the control device 150 determines that the product type determination is unsuccessful (No), the process proceeds to step S27.

[0163] Finally, in step S27, the control device 150 obtains information related to the type of product T obtained by performing the first product identification process and the second product identification process from an external device (not shown) operated by the operator of the remotely operated product replenishment robot 100.

[0164] In step S27, firstly, the control device 150 sends, via the communication unit 195, a message indicating that the type determination of product T obtained from the first and second product identification processes failed, and at least one of the first and second image data of product T captured by the image acquisition unit 260, to an external device (not shown) operated by the operator of the remotely operated product replenishment robot 100. The external device (not shown) displays this information received from the control device 150 to the operator on the display unit, accepts input from the operator regarding information related to the type of product represented by the image data, and sends this information to the control device 150. Thus, the control device 150 acquires information related to the type of product T obtained from the product identification process.

[0165] Through the above processing actions, the type of product T before the product replenishment robot 100 holds and removes it from the product replenishment shelf SL30 and places it in the image acquisition unit 260 of the image acquisition device 220 is determined. Then, when the control device 150 temporarily releases the grip on product T by widening the spacing of the fingers 11 of the gripping part 10, it closes the fingers 11 so that the gripping part 10 holds product T again, and causes the product replenishment robot 100 to move product T to the designated replenishment position on the storage shelf SL20 where product T should be placed. Thus, according to this embodiment, a scheme is provided in which the product replenishment robot 100 can replenish products from the product replenishment shelf SL30 to the storage shelf SL20 after recognizing product T to be replenished to the storage shelf SL20.

[0166] Furthermore, the aforementioned product identification process can be performed on all products retrieved by the product replenishment robot 100 from the product replenishment rack SL30, or it can be performed on only a portion of the products on the product replenishment rack SL30. (Refer to...) Figure 14 To illustrate with the display layout information shown, since single items are displayed in areas 1-6 and 8 of the merchandise replenishment rack SL30 in units of one box each, the possibility of errors such as picking up and placing the wrong item during the loading operation performed by the store clerk is low. Therefore, product identification processing can be omitted for items placed in these areas, or product identification processing can be performed only on the first item initially retrieved by the merchandise replenishment robot 100. When single items are displayed in units of one box, all items contained in the box are of the same type, so by performing product identification processing only on the first item, the types of all items contained in that box can be determined. On the other hand, since various types of items are mixed in area 7 of the merchandise replenishment rack SL30, it is preferable to perform product identification processing individually on each item placed in area 7.

[0167] The present disclosure has been described above through embodiments of the invention; however, these embodiments do not limit the invention as defined in the claims. Furthermore, combinations of features described in the embodiments of the present disclosure can also be included within the scope of the present invention. Moreover, various modifications or improvements can be made to the above embodiments, as will be readily apparent to those skilled in the art.

Claims

1. A product identifying system that identifies a kind of a product held and moved by a product replenishment robot that is used to move a product, the product identifying system having: a product image acquisition device that is mounted on the product replenishment robot and that acquires an image of the product; and a control device that controls an operation of the product image acquisition device and that identifies the kind of the product on the basis of the image of the product acquired by the product image acquisition device.

2. The product identifying system according to claim 1, wherein the control device is configured to perform a process of causing the product image acquisition device to acquire the image of the product held by the product replenishment robot and arranged at an image acquisition position of the product image acquisition device.

3. The product identifying system according to claim 2, wherein the image is an image of one side of the product. wherein 4. The product identifying system according to claim 1, wherein the control device is configured to perform a process of: causing the product replenishment robot to operate to place the product held by the product replenishment robot on a rotation table; causing the rotation table on which the product is placed to rotate; and causing the product image acquisition device to acquire the image of the surroundings of the side of the product during the rotation of the rotation table.

5. The product identifying system according to claim 1, wherein the control device is configured to perform a process of causing the lifting mechanism to operate to lift or lower the rotation table to a height at which the rotation table contacts a bottom surface of the product held by the product replenishment robot and arranged at an image acquisition position of the product image acquisition device.

6. A product identifying method that uses a product identifying system that identifies a kind of a product held and moved by a product replenishment robot that is used to move a product, the product identifying system having: a product image acquisition device that is mounted on the product replenishment robot and that acquires an image of the product; and a control device that controls an operation of the product image acquisition device and that identifies the kind of the product on the basis of the image of the product acquired by the product image acquisition device.

7. The product identifying method according to claim 6, wherein the control device is configured to perform a process of causing the product image acquisition device to acquire the image of the product held by the product replenishment robot and arranged at an image acquisition position of the product image acquisition device.

8. The product identifying method according to claim 7, wherein the image is an image of one side of the product.

9. The product identifying method according to claim 6, wherein the control device is configured to perform a process of: causing the product replenishment robot to operate to place the product held by the product replenishment robot on a rotation table; causing the rotation table on which the product is placed to rotate; and causing the product image acquisition device to acquire the image of the surroundings of the side of the product during the rotation of the rotation table.

10. The product identifying method according to claim 6, wherein the control device is configured to perform a process of causing the lifting mechanism to operate to lift or lower the rotation table to a height at which the rotation table contacts a bottom surface of the product held by the product replenishment robot and arranged at an image acquisition position of the product image acquisition device. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ wherein, ​ ​ ​ ​ The control device has a control section and a storage section, The storage section stores image data of various goods or a learned model generated by performing machine learning using the image data by a learner, The method executed by the control section includes: causing the goods replenishment robot to act so as to arrange the goods held by the goods replenishment robot at an image acquisition position of the image acquisition section; causing the image acquisition section to acquire an image of the goods; and identifying the kind of the goods involved in the image acquired by the image acquisition section based on the image data or the learned model.

7. The goods identification method according to claim 6, wherein The image is an image of one side of the goods.

8. The goods identification method according to claim 6, wherein The method executed by the control section further includes: causing the goods replenishment robot to act so as to place the goods held by the goods replenishment robot on the rotary table; causing the rotary table on which the goods are placed to rotate; and causing the image acquisition section to acquire the image of the periphery of the side of the goods during the rotation of the rotary table.

9. The goods identification method according to claim 6, wherein The method executed by the control section further includes: causing the lifting mechanism to work so as to raise or lower the rotary table to a height at which the rotary table contacts the bottom surface of the goods held by the goods replenishment robot and arranged at the image acquisition position of the image acquisition section.

10. A program for controlling a goods identification system that identifies the kind of goods held and moved by a goods replenishment robot that moves the goods, The goods identification system has: a goods image acquisition device mounted on the goods replenishment robot for acquiring an image of the goods; and a control device that controls the action of the goods image acquisition device and identifies the kind of the goods based on the image of the goods acquired by the goods image acquisition device, wherein The goods image acquisition device has: a rotary table provided with a rotary work table that rotates the goods placed thereon; a lifting mechanism that raises and lowers the rotary table; and an image acquisition section that acquires an image of the goods, The control device has a control section and a storage section, The storage section stores image data of various goods or a learned model generated by performing machine learning using the image data by a learner, The program causes the control section to execute the following processing: causing the goods replenishment robot to act so as to arrange the goods held by the goods replenishment robot at an image acquisition position of the image acquisition section; causing the image acquisition section to acquire an image of the goods; and identifying the kind of the goods involved in the image acquired by the image acquisition section based on the image data or the learned model.

11. The program according to claim 10, wherein The image is an image of one side of the goods. ​ 12. The program according to claim 10, wherein the program causes the control section to further execute the following process: causing the article replenishment robot to act so as to place the article held by the article replenishment robot on the rotary table; causing the rotary table on which the article is placed to rotate; and causing the image acquisition section to acquire the image of the surroundings of the side of the article during the rotation of the rotary table.

13. The program according to claim 10, wherein the program causes the control section to further execute the following process: causing the lifting mechanism to operate so as to raise or lower the rotary table to a height at which the rotary table contacts the bottom surface of the article held by the article replenishment robot and disposed at the image acquisition position of the image acquisition section.

Images