Management device, product movement device, and product movement system

Abstract

One embodiment of the present disclosure provides a management device 200 that is communicably connected to one or more product movement devices that move products T placed on an inventory shelf to a display shelf different from the inventory shelf. The product movement device 1 includes imaging units 50, 60 for acquiring image data including the products T. The management device 200 includes a processor configured to generate a trained model by executing machine learning using image data including each product T, transmitted from at least one product movement device 1, as training data.

Description

Management device, product movement device, and product movement system

[0001] The present invention relates to a management device, a product movement device, and a product movement system.

[0002] Patent Document 1 discloses a product moving system including a product moving device that moves products placed on inventory shelves in a store such as a convenience store to a display shelf different from the inventory shelf. This product moving device includes a gripping unit that grips the product, an arm unit that moves the gripping unit to the inventory shelf and the display shelf, a first imaging unit that photographs the display shelf, a second imaging unit that photographs the inventory shelf, and a control unit that controls the operation of the arm unit, gripping unit, and first and second imaging units.

[0003] The control unit of the product moving device identifies replenishment target products, which are products that need to be replenished on the display shelves, based on the display shelf image data generated by the first imaging unit by photographing the display shelves, and identifies the positions of inventory products in the inventory shelf image data that correspond to the replenishment target products, based on the inventory shelf image data generated by the second imaging unit by photographing the inventory shelves. The control unit further causes the gripper to grip the replenishment target products based on the inventory shelf image data, and after the gripper has gripped the replenishment target products, moves the gripper toward the display shelf and places the replenishment target products in product placement positions on the display shelves based on the display shelf image data.

[0004] In this way, the product moving device disclosed in Patent Document 1 performs the operation of moving products from inventory shelves to display shelves to replenish them.

[0005] International Publication No. 2023 / 022214

[0006] The system disclosed in Patent Document 1 can identify the type of product on a display shelf using image recognition technology, but it may not be able to accurately identify the type of product if the product is placed in a different orientation from the reference image of the product that serves as the basis for identification. If the product replenishment operation continues with an error in product identification, the gripper will grab a product from the product lane that is different from the product that was originally intended to be replenished, and the grabbed product will be placed in a position on the display shelf that is different from the position where it should have been placed. In other words, due to an error in the product's placement on the inventory shelf, the product will be displayed in the wrong position on the display shelf.

[0007] An object of the present disclosure is to provide a means for improving the ability of a product moving device to identify the type of product.

[0008] According to one aspect of the present disclosure, there is provided a management device communicably connected to one or more product moving devices that move products placed on an inventory shelf to a display shelf different from the inventory shelf. The product moving devices include an imaging unit that acquires image data including the products, and the management device includes a processor configured to generate a trained model by performing machine learning using the image data including each product transmitted from at least one product moving device as training data.

[0009] Other features and advantages of the present disclosure can be seen from the following description and the accompanying drawings, which are given by way of example and are not exhaustive.

[0010] According to the present disclosure, it is possible to provide a means for further improving the ability of a product moving device to identify the type of product.

[0011] FIG. 1 is a plan view schematically showing the arrangement of shelves in a store and a product moving device arranged in the store. FIG. 1(a) is a view of the display shelf as seen from the front side, and FIG. 1(b) is a view of the display shelf as seen from the rear side. FIG. 2 is a side view schematically showing the configuration of the product moving device. FIG. 3 is a perspective view showing the peripheral structure of the tip of the arm part of the product moving device. FIG. 4 is a block diagram showing the configuration of the product moving device. FIG. 5 is an image of the display shelf taken by the first camera (left side) of the product moving device. FIG. 6 is a block diagram showing the configuration of the management device. FIG. 7 is a flowchart showing the product replenishing operation by the product moving device. FIG. 8 is a flowchart for explaining an example of the operation of the product moving system of this embodiment. FIG. 9 is a diagram showing various examples in which the appearance of a label changes depending on the orientation of the product.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] <Store layout configuration> First, the layout configuration of the store will be described. Fig. 1 is a plan view showing the layout of shelves in the store and the product moving devices arranged in the store. Fig. 2(a) is a view of the display shelves as seen from the front side, and Fig. 2(b) is a view of the display shelves as seen from the rear side.

[0014] As shown in FIG. 1 , the store is divided into an in-store space SH1 and a backyard space SH2. The in-store space SH1 is a space where customers select and purchase products T. The backyard space SH2 is a space where inventory of products T is stored. A display shelf 410, an inventory shelf 420, and a product moving device 1 are arranged in the store. The product T is, for example, a container having a main body Ta and a cap Tb attached to the upper end of the main body Ta. A neck portion is formed on the upper side of the main body Ta, the outer diameter of which decreases from the outer diameter of the main body Ta to the outer diameter of the cap Tb. The product T can be made of various materials, such as PET, glass, aluminum, steel, and other metals.

[0015] As shown in Figures 2(a) and (b), the display shelf 410 has a plurality of shelves 411 (also referred to as display shelf tiers). A plurality of types of products T are arranged on the shelves 411. For example, the same type of products T are arranged in two or three rows. The products T may also be arranged in only one row. A plurality of partitions 412 are provided on the upper surface of each shelf 411 to separate the products T in each row. Figure 2 shows an example in which the same type of products T are arranged in two rows, and in this example, a partition 412 is provided for every two rows of products T. The arrangement of the partitions 412 is not limited to this, and the partitions 412 can be arranged at intervals of one row or two or more rows.

[0016] The front of the display shelf 410 faces the in-store space SH1, allowing customers to pick up products T from the front side of the display shelf 410. The shelf board 411 is inclined so that the front side of the display shelf 410 is relatively lower than the rear side. As a result, when a customer picks up a product T, the other products T lined up behind that product T slide on the shelf board 411 and move to the front side.

[0017] The rear of the display shelf 410 faces the backyard space SH2, and a store clerk or the product moving device 1 replenishes the products T onto the display shelf 410 from the rear side of the display shelf 410. Although not shown, doors may be provided on the front and rear of the display shelf 410. Although only one display shelf 410 is shown in Fig. 1 for the sake of simplicity, multiple display shelves 410 may be arranged in the store.

[0018] The inventory shelf 420 is arranged in the backyard space SH2 facing the display shelf 410, with the front of the inventory shelf 420 facing the rear of the display shelf 410. Like the display shelf 410, the inventory shelf 420 has multiple shelves (tiers) arranged in the height direction. Products to be replenished, which are products to be replenished on the display shelf 410, are arranged on the shelves of the inventory shelf 420. The products to be replenished may be arranged by a store clerk or by the product moving device 1. <Configuration of the product moving device 1>

[0019] The product moving device 1 will be described below with reference to Fig. 1 and Fig. 3 to Fig. 5. Fig. 3 is a side view showing a schematic configuration of the product moving device 1. Fig. 4 is a perspective view showing the peripheral structure of the tip of the arm portion of the product moving device 1. Fig. 5 is a block diagram showing the configuration of the product moving device 1.

[0020] The product moving device 1 has a gripping unit 10, an arm unit 20, a contact detection sensor 30, first cameras 50R and 50L, a second camera 60, a third camera 70, a horizontal movement mechanism 80, a lifting mechanism 90, and a control device 150. The product moving device 1 is a robot that moves in the space between a display shelf 410 and an inventory shelf 420. The product moving device 1 grips a product T in the inventory shelf 420 with the gripping unit 10, and then moves the gripped product T to the display position of that product on the display shelf 410 (the lane where that product T is displayed).

[0021] As shown in FIG. 4 , the gripping portion 10 has a pair of fingers 11 a, 11 b for gripping an object. The pair of fingers 11 a, 11 b are shaped to be able to grip the vicinity of the cap portion Tb of a container of the product T, such as a PET bottled beverage, and to grip the outer periphery of the product T, such as a canned beverage. The gripping portion 10 may be configured with a pair of fingers 11 a, 11 b, or may have an attraction structure for attracting and holding the object, or a structure for holding the object using adhesive force, magnetic force, or the like. It is preferable that a flexible holding member (not shown) is provided on the gripping surface of each finger 11 a, 11 b to better hold the product T when gripped. The holding member may be formed of, for example, a rubber sheet material, a urethane resin sheet material, or the like.

[0022] The arm unit 20 has a plurality of link members 21, 22, and 23. The plurality of link members 21, 22, and 23 constitute an articulated robot arm. As an example, the articulated robot arm may be a six-axis arm having degrees of freedom in linear directions along the X-axis, Y-axis, and Z-axis, and degrees of freedom around the X-axis, Y-axis, and Z-axis. The articulated robot arm may also have any other mechanism, such as a Cartesian coordinate system robot arm, a polar coordinate system robot arm, a cylindrical coordinate system robot arm, or a SCARA robot arm. One end of the arm unit 20 is fixed to the lifting mechanism 90. A gripper 10 is provided at the tip of the arm unit 20. The operation of the arm unit 20 is controlled by a control device 150.

[0023] The arm unit 20 can move the gripper 10 toward the inventory shelf 420 or toward the display shelf 410 by moving the respective link members 21, 22, and 23. The orientation of the arm unit 20 is not fixed to a specific direction, but for convenience of explanation, the direction in which the respective link members 21, 22, and 23 are extended will be referred to as the extension direction Ay of the arm unit 20 (see FIG. 4). The arm unit 20 moves the gripper 10 forward toward the product T to grip the product T. The extension direction Ay of the arm unit 20 corresponds to the forward direction of the gripper 10 in the gripping operation.

[0024] The contact detection sensor 30 is a sensor that detects that the gripping unit 10 has come into contact with the product T when gripping the product T, and that when placing the product T gripped by the gripping unit 10 on a shelf 411 of the display shelf 410, the product T gripped by the gripping unit 10, the gripping unit 10, or the arm unit 20 has come into contact with an obstacle such as a wall or a support of the display shelf 410. The contact detection sensor 30 may be, for example, a torque sensor, an acceleration sensor, an inertial measurement unit (IMU), a motor input current sensor, or the like.

[0025] The torque sensor may be, for example, a strain gauge that detects torque generated at the axis of each joint of the arm unit 20. The acceleration sensor may be any of various types of acceleration sensors, such as capacitance type or piezo-resistance type, that are installed in the gripping unit 10 or the arm unit 20.

[0026] An inertial measurement unit (IMU) is a device that detects three-dimensional inertial motion (translational and rotational motion in three orthogonal axes) and includes an acceleration sensor that detects translational motion and an angular velocity (gyro) sensor that detects rotational motion. The gyro sensor detects angular velocity, allowing the angle or angular change of an object to be acquired. For example, if a gear or a gear and a toothed belt are used as a drive transmission means for the wrist portion of the gripper 10, redundancy in the movement of the wrist portion of the gripper 10 can occur due to play between the gears or stretching of the toothed belt. Therefore, for example, while placing a product T held by the gripper 10 on a shelf 411 of a display shelf 410, if the arm 20 is further operated with the product T in contact with the shelf 411 and further force is applied to the gripper 10, some displacement occurs in the wrist portion of the gripper 10, causing a change in the attitude (i.e., angle) of the gripper 10. Therefore, by detecting such angle changes that may occur in the gripping unit 10 during the operation of placing the product T on the shelf 411 using an IMU installed in the gripping unit 10, it is possible to detect that the product T has come into contact with the shelf 411.

[0027] Furthermore, in this embodiment, by using an inertial measurement unit (IMU) as the contact detection sensor 30, it is possible to detect the relative height position of the PET bottled beverage to be grasped and the gripping portion 10 by detecting the pitch angle that occurs in the gripping portion 10 when the height of the gripping portion 10 is lowered and the gripping portion 10 is brought into contact with the vicinity of the neck of the PET bottled beverage in the series of actions when the gripping portion 10 grasps the cap portion Tb of the PET bottled beverage of the product T. Then, by adjusting the height of the gripping portion 10 from the state in which the gripping portion 10 is in contact with the vicinity of the neck of the PET bottled beverage, it becomes possible for the gripping portion 10 to grasp the cap portion Tb of the PET bottled beverage at an appropriate height position.

[0028] When a servo motor or the like is used to drive each joint of the arm unit 20, when an external force occurs that causes an angle deviation from the angle at which the gripping posture is maintained, the servo motor operates to reduce the angle deviation to zero in order to maintain the original angle. When the arm unit 20 is further moved with the product T gripped by the gripper 10 in contact with the shelf 411 of the display shelf 410, a current that drives the servo motor against the load is input to the servo motor. Therefore, by detecting the current input to the servo motor for such an operation with a motor input current sensor, it is possible to detect that the product T is in contact with the shelf 411. The motor input current sensor can be configured, for example, as the control unit 151 (see FIG. 5 ), which will be described later.

[0029] The two first cameras 50R, 50L are disposed on the left and right sides of the arm unit 20, respectively. The first camera 50L, attached to the first side surface 23a, which is the left side of the arm unit 20, faces in a first direction A1 along a direction perpendicular to the extension direction Ax of the arm unit 20 (see FIG. 4 ). The first camera 50L is mainly used to photograph the display shelves 410. The first camera 50R, attached to the second side surface 23b, which is the right side of the arm unit 20 and parallel to the first side surface 23a, opposite the first side surface 23a, faces in a second direction A2 opposite to the first direction A1. The first camera 50R is mainly used to photograph the inventory shelves 420. By arranging the first cameras 50R and 50L facing in opposite directions in this manner, the display shelf 410 and the inventory shelf 420 can be photographed simultaneously by the first cameras 50R and 50L, respectively, while keeping the arm section 20 in the same position.

[0030] The performance of the first cameras 50R, 50L may be the same or different, but for simplicity, an example in which both cameras have the same performance will be described below. However, because the purpose and conditions for photographing the display shelves 410 are different from the purpose and conditions for photographing the inventory shelves 420, it goes without saying that cameras with different performance may be used to suit the respective purposes and conditions.

[0031] The first cameras 50R and 50L may include, for example, an imaging element that generates an image (e.g., an RGB image) in which pixels are arranged two-dimensionally, and a depth sensor that is a distance detection device that generates distance data. The depth sensor is not limited to a specific type as long as it can acquire distance data to an object. For example, a stereo lens type or a LiDAR (Light Detection and Ranging) type can be used. The depth sensor may generate, for example, a depth image. In another aspect of the present invention, one or both of the first cameras 50R and 50L may acquire distance data using, for example, an ultrasonic element.

[0032] Note that first camera 50L is pointed in a first direction A1, which means that the imaging direction of the imaging element and depth sensor of first camera 50L is direction A1. Similarly, first camera 50R is pointed in a second direction A2, which means that the imaging direction of the imaging element and depth sensor of first camera 50R is direction A2. Directions A1 and A2 do not necessarily have to be 180° opposite, but may be any direction that allows images of product display shelves 410 and inventory shelves 420 to be captured.

[0033] One or both of the first cameras 50R, 50L may be provided on the grip portion 10. The first cameras 50R, 50L do not necessarily have to be provided on the same member; for example, the first camera 50R may be attached to one of the link members 21 to 23, and the first camera 50L may be attached to another of the link members 21 to 23. However, when the first cameras 50R, 50L are provided on the same member, a common coordinate system is used, which has the advantage of simplifying the calculations for image processing compared to when the cameras 50R, 50L are provided on separate link members.

[0034] The second camera 60 is used to capture an image of the state in which the gripping unit 10 is gripping the product T, and the relative positional relationship between the product T gripped by the gripping unit 10 and the shelf 411 of the display shelf 410. Similar to the first cameras 50R and 50L, the second camera 60 may also have, for example, an imaging element that generates an image in which pixels are arranged two-dimensionally (an RGB image, for example), and a depth sensor that generates distance data.

[0035] As an example, the second camera 60 is installed in a position close to the gripping unit 10, below the link member 23 that is closest to the gripping unit 10 among the link members 21 to 23 of the arm 20, with the imaging direction of its imaging element and depth sensor directed directly below the link member 23 and the gripping unit 10 (the −z direction in FIGS. 3 and 4) or downward and forward (slightly further in the +y direction than the −z direction in FIGS. 3 and 4). This enables the second camera 60 to capture an image of at least the lower portion of the product T gripped by the gripping unit 10 and the shelf board 411 located in front of the gripping unit 10.

[0036] The third camera 70 is a camera that changes direction in response to operation by, for example, a remote operator to capture an image of a predetermined object. The third camera 70 is attached to a part of the lifting mechanism 90, for example. The third camera 70 is capable of horizontal and vertical movement in the space between the display shelf 410 and the inventory shelf 420 in accordance with the operation of the horizontal movement mechanism 80 and the lifting mechanism 90. The part of the lifting mechanism 90 to which the third camera 70 is attached is rotatable about a support 95, and the third camera 70 is configured to rotate left and right about the support 95 in accordance with the rotation of that part, thereby capturing an image of the display shelf 410 or the inventory shelf 420 as needed.

[0037] The third camera 70 may be, for example, a stereo lens type camera. Although not limited thereto, the third camera 70 may have a wider angle of view than the first cameras 50R and 50L and the second camera 60.

[0038] The horizontal movement mechanism 80 has a base plate 81 and a drive mechanism (not shown). The base plate 81 supports the lifting mechanism 90 and allows it to slide along rails (not shown) laid between the display shelves 410 and the inventory shelves 420 in the store. The drive mechanism (not shown) includes a motor, rollers, etc., and operates based on control signals from the control device 150 (see FIG. 5 ) to move the lifting mechanism 90 to a predetermined position along the rails.

[0039] The lifting mechanism 90 has a support column 95, a first lifting mechanism 91, and a second lifting mechanism 92. The support column 95 is fixed onto the base plate 81 and extends in the vertical direction.

[0040] The first lifting mechanism 91 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., and operates based on a control signal from a control device 150 (see FIG. 5). By operating the drive mechanism (not shown), the first lifting mechanism 91 moves up and down in the vertical direction along the support column 95. The upper portion of the first lifting mechanism 91, to which the third camera 70 is attached, is configured to be rotated left and right around the support column 95.

[0041] The second lifting mechanism 92 is held by the first lifting mechanism 91. One end of the arm unit 20 is attached to the second lifting mechanism 92. The second lifting mechanism 92 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., and operates based on a control signal from the control device 150 (see FIG. 5). By operating the drive mechanism (not shown), the second lifting mechanism 92 also moves up and down in the vertical direction.

[0042] When grasping a product T that is located at a predetermined height, the lifting mechanism 90 uses the first lifting mechanism 91 to move the arm unit 20 and the gripping unit 10 to a height near where the product T can be grasped, and then uses the second lifting mechanism 92 to fine-tune the height of the arm unit 20 and the gripping unit 10.

[0043] In this embodiment, a first lifting mechanism 91 and a second lifting mechanism 92 are provided as lifting mechanisms, but in other aspects of the present invention, a configuration in which only one lifting mechanism is provided may be used.

[0044] <Configuration of control device 150> As shown in Fig. 5, control device 150 has a control unit 151, a storage unit 160, an input unit 191, an output unit 193, and a communication unit 195. Although control device 150 is depicted as a single element in Fig. 5, control device 150 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0045] The input unit 191 is a device for receiving input from an operator. The input unit 191 may be configured with devices for inputting to a computer, such as a keyboard, a mouse, or a touch panel. The input unit 191 may also have a voice input device, such as a microphone. The input unit 191 may also have a gesture input device that recognizes and identifies the operator's movements through image recognition.

[0046] The output unit 193 is used by the product moving device 1 to output an alert to a store clerk or the like, and is configured, for example, by one or a combination of a speaker, a display, a light-emitting device, and a vibration device. The communication unit 195 has a function of receiving data from an external device and a function of transmitting data to an external device. Communication between the operation unit of the external device and the communication unit 195 may be via wired or wireless communication. The product moving device 1 may be connected via the communication unit 195 to a management device 200 that monitors the operating status and occurrence of various errors of each product moving device 1 installed in multiple stores and remotely controls the product moving device 1 in each store. The product moving device 1 and the management device 200 communicate with each other via a network such as the Internet. The management device 200 monitors the operating status and occurrence of various errors of the product moving device 1 in each store based on data transmitted from the product moving device 1 in each store as needed. Furthermore, if the product movement device 1 is configured to be remotely operable, input from an operator via the input operation unit 230 (see FIG. 6 ) of the management device 200 is received by the communication unit 195, and the control device 150 causes the product movement device 1 to perform a predetermined operation based on the input. The management device 200 is further configured to accumulate image data of each product transmitted from the product movement device 1 of each store, perform machine learning using the image data, and generate a learning model for identifying the type of each product. Such a management device 200 and the product movement device 1 installed in each store collectively constitute a product movement system. The configuration of the management device 200 will be described later with reference to FIG. 7.

[0047] When the product moving device 1 is configured to be remotely operable, the operation unit 191 may be a device worn by the operator. This device includes a display device (not shown) and an operation device (not shown). The display device may be, for example, a head-mounted display (HMD) having a display visible to the operator. The operation device may include, for example, one or more input sensors capable of detecting the movement of a body part (e.g., a hand or arm) of the operator.

[0048] The storage unit 160 includes temporary or non-temporary storage media such as a read-only memory (ROM), a random access memory (RAM), and a hard disk drive (HDD). The storage unit 160 stores computer programs executed by the control unit 151, trained models (described later), and the like. The computer programs stored in the storage unit 160 include instructions for implementing a method for controlling the product moving device 1 by the control unit 151 (described later with reference to Figures 8 and 9, etc.).

[0049] The storage unit 160 has an acquired data storage unit 160a and a reference data storage unit 160b. The acquired data storage unit 160a stores, for example, image data captured by the cameras 50R, 50L, 60, and 70. The reference data storage unit 160b stores various data necessary for the operation of the product moving device 1. These various data include, for example, data on the product display shelves 410 and inventory shelves 420 (shape data, position data, lane coordinate data, etc.) and data on the products T (shape data, position data, etc.).

[0050] The control unit 151 is configured with, for example, one or more central processing units (CPUs). The control unit 151 executes computer programs stored in the storage unit 160 to function as an operation control unit 152, an imaging control unit 153, and an image data processing unit 155.

[0051] The operation control unit 152 generates control signals for operating the gripping unit 10, the arm unit 20, the horizontal movement mechanism 80, the lifting mechanism 90, and each unit of the control device 150. The operation control unit 152 generates the control signals by referring to input signals from the operation unit 191 and various data stored in the storage unit 160. The control signals may be generated using the processing results of the image data processing unit 155. The operation control unit 152 also transmits and receives data via the communication unit 195, and performs predetermined output via the output unit 193.

[0052] The imaging control unit 153 controls the operation of each of the cameras 50R, 50L, 60, and 70. The imaging timing of each of the cameras 50R, 50L, 60, and 70 may be determined using data stored in advance in the reference data storage unit 160b, for example.

[0053] The image data processing unit 155 performs various information processing using the captured image data and distance data (depth data) captured by each of the cameras 50R, 50L, 60, and 70. As an example, the image data processing unit 155 performs image analysis on the image data captured by the first camera 50R to identify the products lined up on the inventory shelf 420. Alternatively, the image data processing unit 155 may be configured to perform image analysis on the image data captured by any of the cameras 50R, 50L, and 60 to identify the products included in the captured image. Furthermore, the image data processing unit 155 may be configured to identify the type of product included in the image data captured by any of the cameras 50R, 50L, and 60 using a trained model for product type recognition generated by machine learning performed by the management device 200 and stored in the storage unit 160.

[0054] The image data processing unit 155 has a display feasibility determination unit 155a and a grasp target identification unit 155b. The display feasibility determination unit 155a determines, for example, whether a space Sp (see FIG. 6 ) exists behind the last product T lined up on the display shelf 410, where an additional product can be placed, based on at least one of the image data captured by the first camera 50L and the distance data. FIG. 6 shows an image of the display shelf 410 captured by the first camera 50L of the product moving device 1. If the space Sp exists, this product T needs to be replenished. Therefore, if the space Sp exists, the display feasibility determination unit 155a sends a notification to the operation control unit 151 to replenish the product T on the shelf 411 below the space Sp. Upon receiving this notification, the operation control unit 152 performs a replenishment operation for the product T.

[0055] The gripping target specifying unit 155b performs at least one of the following processes based on at least one of the image data captured by the first camera 50R and the distance data: determining whether or not a replenishment target product to be gripped is present on the inventory shelf 420; specifying the size or shape of the replenishment target product; and determining the gripping position of the replenishment target product. In the case of a product T having a cap Tb as shown in FIG. 1, the gripping target specifying unit 155b sets the gripping position, for example, near the cap Tb. On the other hand, in the case of a product T such as a canned beverage without a cap, the gripping target specifying unit 155b may set the gripping position to the side portion of the container. <Configuration of the management device 200>

[0056] 7, the management device 200 includes a processor 210, a storage unit 220, an input operation unit 230, a display unit 240, and a communication unit 250. Although the management device 200 is depicted as a single element in FIG. 7, the management device 200 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0057] The input operation unit 230 is a device for receiving input from an operator of the management device 200. The input unit 230 may be composed of a keyboard, a mouse, and a touch panel. The input operation unit 230 may also have a gesture input device that identifies the operator's movements through image recognition. The display unit 240 is a display device that displays a display screen generated by the processor 210, and may be, for example, a liquid crystal display, an organic EL display device, or a head-mounted display device. The communication unit 250 communicates with the control device 150 of the product moving device 1 via wired or wireless communication, and inputs and outputs information, control signals, and the like between these components.

[0058] The storage unit 220 includes a temporary or non-temporary storage medium such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a solid state drive (SSD). The storage unit 220 stores a computer program executed by the processor 155. The computer program stored in the storage unit 220 includes instructions for performing information processing by the processor 210, which will be described later with reference to FIG. 9 and the like. The storage unit 220 also at least temporarily stores information received from the control device 150 and various data (including intermediate data) generated by processing operations by the processor 210.

[0059] The storage unit 220 also accumulates and stores, for example, image data of each product T received from the control device 150 of the product moving device 1 in each store. The image data of each product T is used as learning data for machine learning to generate a trained model that performs image recognition of each product T.

[0060] The processor 210 is configured, for example, with one or more central processing units (CPUs). The processor 210 operates to realize the processes and functions performed by the processor 210 as described with reference to FIG. 9. In particular, the processor 210 is configured to function as a learner that performs machine learning using image data of each product T stored in the storage unit 220 and generates a learned model for identifying the type of each product T based on the image. Such a learned model may be generated for each type of product T, or may be generated in an integrated manner for multiple types of product T.

[0061] (Product Replenishment Operation by Product Moving Device 1) FIG. 8 is a flowchart showing the product T replenishing operation by the product moving device 1.

[0062] First, in step S11, the rear surface of the display shelf 410 is photographed by the first camera 50L attached to the arm unit 20 of the product moving device 1. The product moving device 1 moves the arm unit 20, horizontal movement mechanism 80, and lifting mechanism 90 so that the first camera 50L can photograph each level of the display shelf 410. Next, the product moving device 1 operates the first camera 50L to capture an image of the rear surface of the display shelf 410 and also acquire distance data to the products T lined up on the display shelf 410.

[0063] Next, in step S12, a determination is made as to whether or not the product can be displayed. The "determination as to whether or not the product can be displayed" is a step in which the display possibility determination unit 155a in the image data processing unit 155 of the product moving device 1 analyzes the captured image of the rear surface of the display shelf and determines which product T can be displayed on which shelf 411 (in other words, which product T needs to be replenished). The product moving device 1 acquires an image of the rear surface of the display shelf 410 as shown in FIG. 6.

[0064] The image data processing unit 155 of the product moving device 1 identifies the lane on the shelf 411 where the product T is to be displayed on the display shelf 410 based on the image data captured by the first camera 50L. Furthermore, the image data processing unit 155 of the product moving device 1 can acquire distance data to the product T (depth data Dep visualized in FIG. 6 ), and therefore can recognize that there is space Sp behind the product T for products whose number of displays has decreased, such as the product T in lanes "7" and "8." The display feasibility determination unit 155a of the product moving device 1 determines whether there is space Sp for the product T based on the distance data to the product T, and determines whether or not more product T can be displayed in lanes "7" and "8." Step S12 determines which product T needs to be replenished on which shelf 411 of the display shelf 410.

[0065] The method for determining whether a product needs to be replenished is not limited to the above method, and various other methods can be used. For example, it may be possible to determine the percentage of the product T that is allocated within a predetermined three-dimensional space, and if this value is equal to or less than a predetermined reference value, it may be determined that the product T needs to be replenished.

[0066] Next, in step S13, the inventory shelf 420 is photographed. The product moving device 1 operates the arm unit 20, the horizontal movement mechanism 80, the lifting mechanism 90, and the first camera 50R to photograph the inventory shelf 420 from the front side. As an example, the product moving device 1 photographs the inventory shelf 420 one shelf at a time to obtain a photographed image showing the inventory status of the products T. The photograph of the inventory shelf 420 does not need to be taken after the photograph of the display shelf 410; the photograph of the inventory shelf 420 may be taken before the photograph of the display shelf 410.

[0067] Next, in step S14, the image data processing unit 155 of the product moving device 1 analyzes the captured image of the inventory shelf 420 acquired in step S13, and identifies the products lined up on the inventory shelf 420. In addition, the grasping target specifying unit 155b of the image data processing unit 155 specifies the grasping position of the product. Through the steps up to this point, the product moving device 1 acquires information indicating which product T needs to be replenished at which position on which shelf 411 of the display shelf 410, as well as information indicating the position on the inventory shelf 420 of the product to be replenished that corresponds to that product T, and the grasping position of that product to be replenished.

[0068] Next, in step S15, the product moving device 1 performs a product replenishment operation (pick-and-place operation) based on the acquired information. Specifically, the operation control unit 152 of the control unit 151 (see FIG. 5 ) of the product moving device 1 operates the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripper 10 toward a predetermined product to be replenished on the inventory shelf 420. The gripper 10 then grips the predetermined gripping position of the product to be replenished (for example, the cap Tb of the product T if the product T is a PET bottle beverage) and lifts the product. The operation control unit 152 then operates the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripped product T to a predetermined placement position on the display shelf 410, and releases the product T from the gripper 10 to place the product T in the predetermined position. Thereafter, the product moving device 1 repeats the same pick-and-place operation to complete the replenishment of the product T.

[0069] The above series of steps are the basic operations performed by the product moving device 1 to automatically replenish products T from the inventory shelf 420 to the display shelf 410. In the product replenishment operation (pick-and-place operation) in step S15 of this series of steps, if, for example, a product of a different type than the type that should be placed in a product lane on the inventory shelf 420 is placed in that product lane, and the type of product is not correctly identified, a product different from the product that should originally be replenished will be placed in the specified placement position (product lane) on the display shelf 410. In response to this, the present embodiment provides a means for further improving the product type identification capability of the product moving device.

[0070] <Operation Example> Next, an operation example of the product movement system of this embodiment will be described. Fig. 9 is a flowchart for explaining one operation example of the product movement system of this embodiment.

[0071] First, in step S21, the imaging control unit 153 of the control unit 151 (see Figure 5) of the product moving device 1 operates at least one of the cameras 50R, 50L, and 60 to capture an image of the product T placed on the product shelf 410 or the inventory shelf 420, etc.

[0072] The operation of capturing an image of product T in step S21 can be performed, as an example, by having the first camera 50L capture an image of product T on the display shelf 410, the first camera 50R capture an image of product T on the inventory shelf 420, and / or by having the second camera 60 capture an image of product T on the inventory shelf 420 when the product T is grasped by the gripping portion 10.

[0073] As described above, the reference data storage unit 160b of the storage unit 160 contains data (shape data, position data, etc.) related to the product T, and the position data of the product T includes product lane information for each of the product display shelves 410 and inventory shelves 420 on which a specific type of product T (product brand, content volume, etc.) should be placed. As a result, the control unit 151 associates information indicating the type of product T corresponding to the product lane on which the product T is placed (type information) with the image data of the captured product T, based on which product lane on the product display shelves 410 and inventory shelves 420 the product T is imaged on by the cameras 50 and 60, and at least temporarily stores the image data of the product T in the acquired data storage unit 160a. As an example, the image data of each product T includes type information indicating the type of the product T as metadata. During product replenishment operations, the imaging control unit 153 captures images of each product T on each product lane of the product display shelf 410 and the inventory shelf 420 using cameras 50 and 60, and stores multiple image data for each product T in the acquired data memory unit 160a.

[0074] Note that even in images of the same type of product T, the appearance of the label of the product T differs depending on the orientation when the product is placed on the product display shelf 410 or the inventory shelf 420. Fig. 10 is a diagram showing various examples in which the appearance of the label differs depending on the orientation of the product T. Fig. 10(a) is a schematic diagram showing the product T labeled with the product name "ABC" as viewed from the front, Fig. 10(b) is a schematic diagram showing the product T labeled with the product name "ABC" as viewed from the side, and Fig. 10(a) is a schematic diagram showing the product T labeled with the product name "ABC" as viewed from the back.

[0075] 10 , the appearance of the labels of the products T captured by the cameras 50 and 60 varies depending on the orientation of the products T when they are placed on the product display shelf 410 or the inventory shelf 420. Therefore, image data acquired by capturing images of multiple products T of the same type on a certain product lane of the product display shelf 410 or the inventory shelf 420 includes image data in which the labels of the products T appear differently when viewed from various directions.

[0076] Next, in step S22, the control unit 151 in the control device 150 of the product moving device 1 transmits the image data of each product stored in the acquired data storage unit 160a of the storage unit 160 to the management device 200 via the communication unit 195.

[0077] When the management device 200 receives image data of each product, it at least temporarily stores the image data in the storage unit 220. The management device 200 receives image data of each product from each control device 150 installed in each store. Therefore, the storage unit 220 of the management device 200 stores not only image data of each product acquired by the product movement device 1 of a certain store, but also image data of each product acquired by the product movement device 1 of each store in a consolidated manner. The control device 150 of the product movement device 1 of each store, for example, periodically transmits the image data of each product accumulated in the acquired data storage unit 160a to the management device 200. As an example, the control device 150 collectively transmits the image data of each product accumulated in the acquired data storage unit 160a to the management device 200 daily or weekly.

[0078] Next, in step S23, the processor 210 of the management device 200 performs machine learning using the image data of each product stored in the storage unit 220 to generate a trained model.

[0079] The processor 210 functioning as a learner generates a trained model used to identify each product, for example, by performing machine learning on the image data of each product associated with type information of the product T as training data. Alternatively, the storage unit 220 of the management device 200 may store label images of each product, and the processor 210 functioning as a learner may use the label images of each product as training data and perform machine learning on the image data of each product associated with type information of the product T as training data to generate a trained model. The label images of each product stored in the storage unit 220 may be label images of the product viewed from the front, or may be label images covering the entire circumference of the product.

[0080] As mentioned above, each image data for each product is associated with type information of the product T contained in the image, so the processor 210 of the management device 200 can extract and process images of a specific type of product from the large number of image data stored in the memory unit 220 based on the type information.

[0081] Furthermore, the processor 210 of the management device 200 may be configured to perform machine learning by omitting, from the image data of each product used as learning data, image data in which the captured image is unclear or image data in which an object different from the label image of the corresponding product type is captured. For example, the processor 210 can compare the image data of each product with the label image of the product type corresponding to that product using any known image processing technology, and perform a process to identify image data in which the value indicating the degree of association between the two is below a predetermined value, thereby excluding the identified image data from the target of machine learning.

[0082] The trained model described above can be generated, for example, by performing machine learning using a neural network composed of multiple layers, each containing neurons. Such a neural network may be a deep neural network, such as a convolutional neural network (CNN) with 20 or more layers. Machine learning using such a deep neural network is referred to as deep learning. Alternatively, the trained model described above can be generated using a "visual transformer," which applies a transformer, a type of deep neural network primarily based on a self-attention mechanism, to the field of computer vision. The trained model generated in this manner is at least temporarily stored in the memory unit 220 of the management device 200.

[0083] The processor 210 of the management device 200 may periodically generate the trained model. For example, the processor 210 may periodically generate a trained model at a frequency (e.g., daily or weekly) at which the control device 150 transmits image data of each product stored in the acquired data storage unit 160a to the management device 200, thereby updating the trained model. Since image data of each product is periodically added to the storage unit 220 of the management device 200, generating a trained model at the timing when image data is added makes it possible to generate a trained model with higher accuracy based on a larger amount of image data.

[0084] Next, in step S24, the processor 210 of the management device 200 transmits the learned model stored in the memory unit 220 to the control device 150 of the product movement device 1 in each store.

[0085] The processor 210 may transmit the generated trained model to the control devices 150 of the product movement devices 1 in each store all at once, or may transmit the trained model only to the control devices 150 of one or more product movement devices 1 in each store that have requested the trained model. Upon receiving the trained model, the control device 150 of the product movement device 1 stores it in the storage unit 160.

[0086] Next, in step S25, the image data processing unit 155 in the control device 150 of the product moving device 1 uses the trained model stored in the memory unit 160 to identify the product contained in the image data captured by one of the cameras 50R, 50L, and 60.

[0087] More specifically, the image data processing unit 155 of the control device 150 analyzes image data captured by the first camera 50R using the trained model stored in the storage unit 160 to identify the products lined up on the inventory shelf 420 and determine whether the products placed on a specific product lane on the inventory shelf 420 are of the type associated with that product lane. The image data processing unit 155 also analyzes image data captured by the first camera 50L using the trained model stored in the storage unit 160 to identify the products lined up on the product display shelf 410 and determine whether the products placed on a specific product lane on the product display shelf 410 are of the type associated with that product lane. Furthermore, the image data processing unit 155 analyzes image data captured by the second camera 60 using the trained model stored in the storage unit 160 to identify the products to be grasped by the gripper 10 and determine whether the products to be grasped by the gripper 10 are of the type of product to be grasped.

[0088] When the image data processing unit 155 of the control unit 151 in the control device 150 determines that the type of product in the image data to be identified is different from the actual type of product, the operation control unit 152 of the control unit 151 outputs information (misplacement information) indicating the possibility of misplacement of the product type on a specific product lane to the output unit 193, or transmits it to the management device 200 via the communication unit 195, or performs both.

[0089] A store clerk who sees the misplacement information displayed on the output unit 193 of the control device 150 can check the products placed on a specific product lane on the product display shelf 410 or inventory shelf 420, and if a product of a different type from the type that should be placed on that product lane has been placed, can take action to remove that product from that product lane. Furthermore, when misplacement information is transmitted to the management device 200, the administrator (operator) of the management device 200 can remotely operate the product movement device 1 of the store that transmitted the misplacement information to check the products placed on a specific product lane on the product display shelf 410 or inventory shelf 420, and if a product of a different type from the type that should be placed on that product lane has been placed, can take action to remove that product from that product lane.

[0090] As described above, according to the product movement system of this embodiment, image data of each product acquired by the product movement device 1 in each store is collected in the management device 200, and the management device 200 performs machine learning using this image data as training data to generate a trained model for identifying the type of product. The generated trained model is transmitted from the management device 200 to the control device 150 of the product movement device 1 in each store, and is used by the control device 150 to identify the type of product included in images captured by the cameras 50, 60 of the product movement device 1.

[0091] In this way, by generating a trained model based on a large amount of image data collected from each store in the management device 200, it becomes possible to generate a trained model that can more accurately identify the type of product.

[0092] Furthermore, for example, it is possible to store a large amount of image data for each product on a cloud server, and have the image data processing unit 155 in the control device 150 of the product movement device 1 in each store refer to the image data in the cloud server and compare it with the image to be identified, thereby identifying the type of product. However, in this case, a large amount of data needs to be stored and maintained, and a large amount of calculation cost is required because it is necessary to compare it with many image data to identify the type of one product. In contrast, by using a trained model to identify the type of product as in the present embodiment, it is possible to reduce the amount of data to be maintained and the calculation cost.

[0093] Although the above describes an example in which the control unit 151 in the control device 150 of the product moving device 1 identifies the type of product placed on the product display shelf 410 or the inventory shelf 420 using the learned model stored in the memory unit 160, the types of products that can be identified by the control device 150 are not limited to the products placed on them. For example, a store may be equipped with a product replenishment shelf (not shown) in addition to the product display shelf 410 and the inventory shelf 420, and the type of product may be identified by capturing images of the products placed on the product replenishment shelf with the cameras 50 and 60. The product replenishment shelf may have, for example, multiple shelves (tiers) arranged in the vertical direction and multiple casters at the bottom that enable the product replenishment shelf to be moved in any direction. The product replenishment shelf holds products to be replenished on the product display shelf 410 or the inventory shelf 420, and the control device 150 of the product moving device 1 identifies the type of each product placed on the product replenishment shelf and causes the product moving device 1 to perform an operation to move the products to a specific product lane on the product display shelf 410 or the inventory shelf 420 where those products should be placed.

[0094] Although the present invention has been described above through the embodiments of the invention, the above-described embodiments do not limit the scope of the invention according to the claims. Furthermore, combinations of features described in the embodiments of the present invention may also fall within the technical scope of the present invention. Furthermore, it will be apparent to those skilled in the art that various modifications and improvements can be made to the above-described embodiments.

Claims

1. A management device communicatively connected to one or more product moving devices that move products placed on an inventory shelf to a display shelf different from the inventory shelf, wherein the product moving device has an imaging unit that acquires image data including the products, and the management device has a processor configured to generate a trained model by performing machine learning using the image data including each of the products transmitted from at least one of the product moving devices as training data.

2. The management device according to claim 1, wherein the image data includes type information relating to the type of the product included in the image data.

3. The management device of claim 1, further comprising a memory unit in which label images of each of the products are stored, and the processor is configured to use the image data as learning data and the label images as teacher data to perform machine learning to generate the trained model.

4. The management device of claim 1, wherein the product moving device periodically transmits the image data including the product acquired by the imaging unit to the management device, the management device further includes a memory unit in which the image data transmitted from the product moving device is stored, and the processor of the management device is configured to periodically generate and update the trained model by performing machine learning using the image data stored in the memory unit as training data.

5. A product moving device that moves products placed on an inventory shelf to a display shelf different from the inventory shelf, the product moving device comprising: an arm unit having a gripping unit that grips the product; an imaging unit that acquires image data including the product; a control unit that controls the operation of the gripping unit and the imaging unit; and a memory unit that stores a trained model generated by the management device described in claim 1 and transmitted to the product moving device, wherein the control unit is configured to identify the type of the product contained in the image data acquired by the imaging unit based on the trained model.

6. The product moving device according to claim 5, wherein the imaging unit acquires the image data of the product placed on the inventory shelf or the display shelf.

7. A product moving device as described in claim 5, wherein the imaging unit acquires the image data of the product placed on a product replenishment shelf on which the product to be moved to the inventory shelf or display shelf is placed.

8. A product movement system comprising the management device according to claim 1 and the product movement device according to claim 5.

Images