Product transfer apparatus and method for controlling same

Abstract

According to one embodiment of the present disclosure, provided is a product transfer apparatus 1 that transfers a container of a product which has been placed on an inventory shelf to a display shelf. A control unit 150 of the product transfer apparatus 1 is configured to execute the operations of: acquiring a position of a product T on the basis of a position of a holding part 10 when a sensor 30 detects that a change in a pitch angle has occurred in the holding part 10 when a portion of the holding part 10 was brought into contact with the product T; and correcting a position of the product T recognized on the basis of first image data captured by a first camera 50, on the basis of a difference between the position of the product T recognized on the basis of the first image data and the acquired position of product T.

Description

Product moving device and control method thereof

[0001] The present invention relates to a product moving device and a control method thereof.

[0002] Patent Document 1 discloses a product moving system including a product moving device that moves products placed on inventory shelves in a store such as a convenience store to a display shelf different from the inventory shelf. This product moving device includes a gripping unit that grips the product, an arm unit that moves the gripping unit to the inventory shelf and the display shelf, a first imaging unit that photographs the display shelf, a second imaging unit that photographs the inventory shelf, and a control unit that controls the operation of the arm unit, gripping unit, and first and second imaging units.

[0003] The control unit of the product moving device identifies replenishment target products, which are products that need to be replenished on the display shelves, based on the display shelf image data generated by the first imaging unit by photographing the display shelves, and identifies the positions of inventory products in the inventory shelf image data that correspond to the replenishment target products, based on the inventory shelf image data generated by the second imaging unit by photographing the inventory shelves. The control unit further causes the gripper to grip the replenishment target products based on the inventory shelf image data, and after the gripper has gripped the replenishment target products, moves the gripper toward the display shelf and places the replenishment target products in product placement positions on the display shelves based on the display shelf image data.

[0004] International Publication No. 2023 / 022214

[0005] In systems such as those described in Patent Document 1, errors can generally occur between the position of a product recognized by a camera equipped on the product moving device and the actual position of the product. Possible causes of such errors include low assembly accuracy of the product moving device, particularly the arm and gripper units, or errors that occur as the product moves repeatedly even if the assembly accuracy is initially sufficient. To correct such errors, for example, an operator must visit the store where the product moving device is installed and perform adjustments to locally optimize the assembly state of the arm and gripper units of the product moving device and the installation state of the camera relative to the product moving device so that the gripper unit of the product moving device can correctly grip products placed on inventory shelves or product shelves.

[0006] One possible method for error calibration is so-called hand-eye calibration, which converts the coordinates of an object detected by a camera from a vision coordinate system to a robot coordinate system. Using this type of hand-eye calibration method makes it possible to perform error calibration remotely, without requiring an operator to visit the site. However, accurate calibration using hand-eye calibration requires high assembly and operational precision, as in industrial robots. Therefore, even with the hand-eye calibration method, accurate error calibration is not always possible in a device configuration such as a product transport device that moves products autonomously.

[0007] Considering the problems with the conventional technology as described above, the inventor came up with a method for enabling the gripping part of a product moving device to correctly grip a product by obtaining product position information through the interaction of the gripping part of the product moving device with the product to be gripped, and then referring to that position information to calibrate the error between the product position recognized by the camera equipped on the product moving device and the actual position of the product.

[0008] An object of the present disclosure is to provide a product moving device or the like that makes it possible to calibrate the error between the position of a product recognized by a camera and the actual position of the product.

[0009] According to one aspect of the present disclosure, there is provided a product moving device for moving products placed on an inventory shelf to a display shelf different from the inventory shelf, the product moving device including: an arm unit with a gripper for gripping the product; a first imaging unit for acquiring first image data including the product placed on the inventory shelf or the display shelf; a sensor for detecting a change in pitch angle of the gripper; and a control unit for controlling the operation of the gripper, the arm unit, the first imaging unit, and the sensor. The control unit is configured to acquire a position of the product based on a position of the gripper when the sensor detects that a change in pitch angle of the gripper has occurred when a part of the gripper is brought into contact with the product, and to correct the position of the product recognized based on the first image data based on a difference between the position of the product recognized based on the first image data and the acquired position of the product.

[0010] Other features and advantages of the present disclosure can be seen from the following description and the accompanying drawings, which are given by way of example and are not exhaustive.

[0011] According to the present disclosure, a product moving device and the like are provided that make it possible to calibrate the error between the position of a product recognized by a camera and the actual position of the product.

[0012] 1 is a plan view schematically showing the arrangement of shelves in a store and a product moving device arranged in the store. FIG. 1(a) is a view of the display shelf as seen from the front side, and FIG. 1(b) is a view of the display shelf as seen from the rear side. FIG. 1 is a side view schematically showing the configuration of the product moving device. FIG. 2 is a perspective view showing the peripheral structure of the tip of the arm part of the product moving device. FIG. 3 is a block diagram showing the configuration of the product moving device. FIG. 4 is an image of the display shelf taken by the first camera (left side) of the product moving device. FIG. 5 is a diagram showing the relative position of the cap part of a product and the grip part when detecting the actual position of the fingers of the grip part relative to the cap part of a PET bottle drink product. FIG. 6 is a flowchart showing the product replenishing operation by the product moving device. FIG. 7 is a flowchart for explaining an example of the operation of the product moving device. FIG. 8 is a diagram showing an example of the operation of searching for a Z-direction reference position where the bottom surface of the fingers contacts the top surface of the cap part of the product. FIG. 9 is a diagram showing an example of the operation of searching for a Y-direction reference position where the tip of the fingers contacts the vicinity of the side edge of the top surface of the cap part of the product. FIG. 10 is a diagram showing an example of the operation of searching for an X-direction reference position where the center of the fingers in the X direction coincides with the center of the cap part of the product in the X direction. 10A and 10B are diagrams illustrating an example of an operation for searching for an X-direction reference position where the center of the finger in the X direction coincides with the center of the cap part of the product in the X direction; and FIG. 10C are diagrams illustrating another example of an operation for searching for an X-direction reference position where the center of the finger in the X direction coincides with the center of the cap part of the product in the X direction.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] <Store layout configuration> First, the layout configuration of the store will be described. Fig. 1 is a plan view showing the layout of shelves in the store and the product moving devices arranged in the store. Fig. 2(a) is a view of the display shelves as seen from the front side, and Fig. 2(b) is a view of the display shelves as seen from the rear side.

[0015] As shown in FIG. 1 , the store is divided into an in-store space SH1 and a backyard space SH2. The in-store space SH1 is a space where customers select and purchase products T. The backyard space SH2 is a space where inventory of products T is stored. A display shelf 410, an inventory shelf 420, and a product moving device 1 are arranged in the store. The product T is, for example, a container having a main body Ta and a cap Tb attached to the upper end of the main body Ta. A neck portion is formed on the upper side of the main body Ta, the outer diameter of which decreases from the outer diameter of the main body Ta to the outer diameter of the cap Tb. The product T can be made of various materials, such as PET, glass, aluminum, steel, and other metals.

[0016] As shown in Figures 2(a) and (b), the display shelf 410 has a plurality of shelves 411 (also referred to as display shelf tiers). A plurality of types of products T are arranged on the shelves 411. For example, the same type of products T are arranged in two or three rows. The products T may also be arranged in only one row. A plurality of partitions 412 are provided on the upper surface of each shelf 411 to separate the products T in each row. Figure 2 shows an example in which the same type of products T are arranged in two rows, and in this example, a partition 412 is provided for every two rows of products T. The arrangement of the partitions 412 is not limited to this, and the partitions 412 can be arranged at intervals of one row or two or more rows.

[0017] The front of the display shelf 410 faces the in-store space SH1, allowing customers to pick up products T from the front side of the display shelf 410. The shelf board 411 is inclined so that the front side of the display shelf 410 is relatively lower than the rear side. As a result, when a customer picks up a product T, the other products T lined up behind that product T slide on the shelf board 411 and move to the front side.

[0018] The rear of the display shelf 410 faces the backyard space SH2, and a store clerk or the product moving device 1 replenishes the products T onto the display shelf 410 from the rear side of the display shelf 410. Although not shown, doors may be provided on the front and rear of the display shelf 410. Although only one display shelf 410 is shown in Fig. 1 for the sake of simplicity, multiple display shelves 410 may be arranged in the store.

[0019] The inventory shelf 420 is arranged in the backyard space SH2 facing the display shelf 410, with the front of the inventory shelf 420 facing the rear of the display shelf 410. Like the display shelf 410, the inventory shelf 420 has multiple shelves (tiers) arranged in the height direction. Products to be replenished, which are products to be replenished on the display shelf 410, are arranged on the shelves of the inventory shelf 420. The products to be replenished may be arranged by a store clerk or by the product moving device 1. <Configuration of the product moving device 1>

[0020] The product moving device 1 will be described below with reference to Fig. 1 and Fig. 3 to Fig. 5. Fig. 3 is a side view showing a schematic configuration of the product moving device 1. Fig. 4 is a perspective view showing the peripheral structure of the tip of the arm portion of the product moving device 1. Fig. 5 is a block diagram showing the configuration of the product moving device 1.

[0021] The product moving device 1 has a gripping unit 10, an arm unit 20, a contact detection sensor 30, first cameras 50R and 50L, a second camera 60, a third camera 70, a horizontal movement mechanism 80, a lifting mechanism 90, and a control device 150. The product moving device 1 is a robot that moves in the space between a display shelf 410 and an inventory shelf 420. The product moving device 1 grips a product T in the inventory shelf 420 with the gripping unit 10, and then moves the gripped product T to the display position of that product on the display shelf 410 (the lane where that product T is displayed).

[0022] As shown in FIG. 4 , the gripping portion 10 has a pair of fingers 11 a, 11 b for gripping an object. The pair of fingers 11 a, 11 b are shaped to be able to grip the vicinity of the cap portion Tb of a container of the product T, such as a PET bottled beverage, and to grip the outer periphery of the product T, such as a canned beverage. The gripping portion 10 may be configured with a pair of fingers 11 a, 11 b, or may have an attraction structure for attracting and holding the object, or a structure for holding the object using adhesive force, magnetic force, or the like. It is preferable that a flexible holding member (not shown) is provided on the gripping surface of each finger 11 a, 11 b to better hold the product T when gripped. The holding member may be formed of, for example, a rubber sheet material, a urethane resin sheet material, or the like.

[0023] The arm unit 20 has a plurality of link members 21, 22, and 23. The plurality of link members 21, 22, and 23 constitute an articulated robot arm. As an example, the articulated robot arm may be a six-axis arm having degrees of freedom in linear directions along the X-axis, Y-axis, and Z-axis, and degrees of freedom around the X-axis, Y-axis, and Z-axis. The articulated robot arm may also have any other mechanism, such as a Cartesian coordinate system robot arm, a polar coordinate system robot arm, a cylindrical coordinate system robot arm, or a SCARA robot arm. One end of the arm unit 20 is fixed to the lifting mechanism 90. A gripper 10 is provided at the tip of the arm unit 20. The operation of the arm unit 20 is controlled by a control device 150.

[0024] The arm unit 20 can move the gripper 10 toward the inventory shelf 420 or toward the display shelf 410 by moving the respective link members 21, 22, and 23. The orientation of the arm unit 20 is not fixed to a specific direction, but for convenience of explanation, the direction in which the respective link members 21, 22, and 23 are extended will be referred to as the extension direction Ay of the arm unit 20 (see FIG. 4). The arm unit 20 moves the gripper 10 forward toward the product T to grip the product T. The extension direction Ay of the arm unit 20 corresponds to the forward direction of the gripper 10 in the gripping operation.

[0025] The contact detection sensor 30 is a sensor that detects that the gripping unit 10 has come into contact with the product T when gripping the product T, and that when placing the product T gripped by the gripping unit 10 on a shelf 411 of the display shelf 410, the product T gripped by the gripping unit 10, the gripping unit 10, or the arm unit 20 has come into contact with an obstacle such as a wall or a support of the display shelf 410. The contact detection sensor 30 may be, for example, a torque sensor, an acceleration sensor, an inertial measurement unit (IMU), a motor input current sensor, or the like.

[0026] The torque sensor may be, for example, a strain gauge that detects torque generated at the axis of each joint of the arm unit 20. The acceleration sensor may be any of various types of acceleration sensors, such as capacitance type or piezo-resistance type, that are installed in the gripping unit 10 or the arm unit 20.

[0027] An inertial measurement unit (IMU) is a device that detects three-dimensional inertial motion (translational and rotational motion in three orthogonal axes) and includes an acceleration sensor that detects translational motion and an angular velocity (gyro) sensor that detects rotational motion. The gyro sensor detects angular velocity, allowing the angle or angular change of an object to be acquired. For example, if a gear or a gear and a toothed belt are used as a drive transmission means for the wrist portion of the gripper 10, redundancy in the movement of the wrist portion of the gripper 10 can occur due to play between the gears or stretching of the toothed belt. Therefore, for example, while placing a product T held by the gripper 10 on a shelf 411 of a display shelf 410, if the arm 20 is further operated with the product T in contact with the shelf 411 and further force is applied to the gripper 10, some displacement occurs in the wrist portion of the gripper 10, causing a change in the attitude (i.e., angle) of the gripper 10. Therefore, by detecting such angle changes that may occur in the gripping unit 10 during the operation of placing the product T on the shelf 411 using an IMU installed in the gripping unit 10, it is possible to detect that the product T has come into contact with the shelf 411.

[0028] Furthermore, in this embodiment, by using an inertial measurement unit (IMU) as the contact detection sensor 30, it is possible to detect the relative height position of the PET bottled beverage to be grasped and the gripping portion 10 by detecting the pitch angle that occurs in the gripping portion 10 when the height of the gripping portion 10 is lowered and the gripping portion 10 is brought into contact with the vicinity of the neck of the PET bottled beverage in the series of actions when the gripping portion 10 grasps the cap portion Tb of the PET bottled beverage of the product T. Then, by adjusting the height of the gripping portion 10 from the state in which the gripping portion 10 is in contact with the vicinity of the neck of the PET bottled beverage, it becomes possible for the gripping portion 10 to grasp the cap portion Tb of the PET bottled beverage at an appropriate height position.

[0029] When a servo motor or the like is used to drive each joint of the arm unit 20, when an external force occurs that causes an angle deviation from the angle at which the gripping posture is maintained, the servo motor operates to reduce the angle deviation to zero in order to maintain the original angle. When the arm unit 20 is further moved with the product T gripped by the gripper 10 in contact with the shelf 411 of the display shelf 410, a current that drives the servo motor against the load is input to the servo motor. Therefore, by detecting the current input to the servo motor for such an operation with a motor input current sensor, it is possible to detect that the product T is in contact with the shelf 411. The motor input current sensor can be configured, for example, as the control unit 151 (see FIG. 5 ), which will be described later.

[0030] The two first cameras 50R, 50L are disposed on the left and right sides of the arm unit 20, respectively. The first camera 50L, attached to the first side surface 23a, which is the left side of the arm unit 20, faces in a first direction A1 along a direction perpendicular to the extension direction Ax of the arm unit 20 (see FIG. 4 ). The first camera 50L is mainly used to photograph the display shelves 410. The first camera 50R, attached to the second side surface 23b, which is the right side of the arm unit 20 and parallel to the first side surface 23a, opposite the first side surface 23a, faces in a second direction A2 opposite to the first direction A1. The first camera 50R is mainly used to photograph the inventory shelves 420. By arranging the first cameras 50R and 50L facing in opposite directions in this manner, the display shelf 410 and the inventory shelf 420 can be photographed simultaneously by the first cameras 50R and 50L, respectively, while keeping the arm section 20 in the same position.

[0031] The performance of the first cameras 50R, 50L may be the same or different, but for simplicity, an example in which both cameras have the same performance will be described below. However, because the purpose and conditions for photographing the display shelves 410 are different from the purpose and conditions for photographing the inventory shelves 420, it goes without saying that cameras with different performance may be used to suit the respective purposes and conditions.

[0032] The first cameras 50R and 50L may include, for example, an imaging element that generates an image (e.g., an RGB image) in which pixels are arranged two-dimensionally, and a depth sensor that is a distance detection device that generates distance data. The depth sensor is not limited to a specific type as long as it can acquire distance data to an object. For example, a stereo lens type or a LiDAR (Light Detection and Ranging) type can be used. The depth sensor may generate, for example, a depth image. In another aspect of the present invention, one or both of the first cameras 50R and 50L may acquire distance data using, for example, an ultrasonic element.

[0033] Note that first camera 50L is pointed in a first direction A1, which means that the imaging direction of the imaging element and depth sensor of first camera 50L is direction A1. Similarly, first camera 50R is pointed in a second direction A2, which means that the imaging direction of the imaging element and depth sensor of first camera 50R is direction A2. Directions A1 and A2 do not necessarily have to be 180° opposite, but may be any direction that allows images of product display shelves 410 and inventory shelves 420 to be captured.

[0034] One or both of the first cameras 50R, 50L may be provided on the grip portion 10. The first cameras 50R, 50L do not necessarily have to be provided on the same member; for example, the first camera 50R may be attached to one of the link members 21 to 23, and the first camera 50L may be attached to another of the link members 21 to 23. However, when the first cameras 50R, 50L are provided on the same member, a common coordinate system is used, which has the advantage of simplifying the calculations for image processing compared to when the cameras 50R, 50L are provided on separate link members.

[0035] The second camera 60 is used to capture an image of the state in which the gripping unit 10 is gripping the product T, and the relative positional relationship between the product T gripped by the gripping unit 10 and the shelf 411 of the display shelf 410. Similar to the first cameras 50R and 50L, the second camera 60 may also have, for example, an imaging element that generates an image in which pixels are arranged two-dimensionally (an RGB image, for example), and a depth sensor that generates distance data.

[0036] As an example, the second camera 60 is installed in a position close to the gripping unit 10, below the link member 23 that is closest to the gripping unit 10 among the link members 21 to 23 of the arm 20, with the imaging direction of its imaging element and depth sensor directed directly below the link member 23 and the gripping unit 10 (the −z direction in FIGS. 3 and 4) or downward and forward (slightly further in the +y direction than the −z direction in FIGS. 3 and 4). This enables the second camera 60 to capture an image of at least the lower portion of the product T gripped by the gripping unit 10 and the shelf board 411 located in front of the gripping unit 10.

[0037] The third camera 70 is a camera that changes direction in response to operation by, for example, a remote operator to capture an image of a predetermined object. The third camera 70 is attached to a part of the lifting mechanism 90, for example. The third camera 70 is capable of horizontal and vertical movement in the space between the display shelf 410 and the inventory shelf 420 in accordance with the operation of the horizontal movement mechanism 80 and the lifting mechanism 90. The part of the lifting mechanism 90 to which the third camera 70 is attached is rotatable about a support 95, and the third camera 70 is configured to rotate left and right about the support 95 in accordance with the rotation of that part, thereby capturing an image of the display shelf 410 or the inventory shelf 420 as needed.

[0038] The third camera 70 may be, for example, a stereo lens type camera. Although not limited thereto, the third camera 70 may have a wider angle of view than the first cameras 50R and 50L and the second camera 60.

[0039] The horizontal movement mechanism 80 has a base plate 81 and a drive mechanism (not shown). The base plate 81 supports the lifting mechanism 90 and allows it to slide along rails (not shown) laid between the display shelves 410 and the inventory shelves 420 in the store. The drive mechanism (not shown) includes a motor, rollers, etc., and operates based on control signals from the control device 150 (see FIG. 5 ) to move the lifting mechanism 90 to a predetermined position along the rails.

[0040] The lifting mechanism 90 has a support column 95, a first lifting mechanism 91, and a second lifting mechanism 92. The support column 95 is fixed onto the base plate 81 and extends in the vertical direction.

[0041] The first lifting mechanism 91 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., and operates based on a control signal from a control device 150 (see FIG. 5). By operating the drive mechanism (not shown), the first lifting mechanism 91 moves up and down in the vertical direction along the support column 95. The upper portion of the first lifting mechanism 91, to which the third camera 70 is attached, is configured to be rotated left and right around the support column 95.

[0042] The second lifting mechanism 92 is held by the first lifting mechanism 91. One end of the arm unit 20 is attached to the second lifting mechanism 92. The second lifting mechanism 92 has a drive mechanism (not shown). The drive mechanism (not shown) includes a motor, a linear guide, etc., and operates based on a control signal from the control device 150 (see FIG. 5). By operating the drive mechanism (not shown), the second lifting mechanism 92 also moves up and down in the vertical direction.

[0043] When grasping a product T that is located at a predetermined height, the lifting mechanism 90 uses the first lifting mechanism 91 to move the arm unit 20 and the gripping unit 10 to a height near where the product T can be grasped, and then uses the second lifting mechanism 92 to fine-tune the height of the arm unit 20 and the gripping unit 10.

[0044] In this embodiment, a first lifting mechanism 91 and a second lifting mechanism 92 are provided as lifting mechanisms, but in other aspects of the present invention, a configuration in which only one lifting mechanism is provided may be used.

[0045] <Configuration of control device 150> As shown in Fig. 5, control device 150 has a control unit 151, a storage unit 160, an input unit 191, an output unit 193, and a communication unit 195. Although control device 150 is depicted as a single element in Fig. 5, control device 150 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0046] The input unit 191 is a device for receiving input from an operator. The input unit 191 may be configured with devices for inputting to a computer, such as a keyboard, a mouse, or a touch panel. The input unit 191 may also have a voice input device, such as a microphone. The input unit 191 may also have a gesture input device that recognizes and identifies the operator's movements through image recognition.

[0047] The output unit 193 is used by the product moving device 1 to output an alert to a store clerk or the like, and is configured, for example, by one or a combination of a speaker, a display, a light-emitting device, and a vibration device. The communication unit 195 has a function of receiving data from an external device and a function of transmitting data to an external device. If the product moving device 1 is configured to be remotely operable, the communication unit 195 receives input from an operator via an operation unit of an external device (not shown), and the control device 150 causes the product moving device 1 to perform a predetermined operation based on the input. Note that communication between the operation unit of the external device and the communication unit 195 may be wired or wireless.

[0048] When the product moving device 1 is configured to be remotely operable, the operation unit 191 may be a device worn by the operator. This device includes a display device (not shown) and an operation device (not shown). The display device may be, for example, a head-mounted display (HMD) having a display visible to the operator. The operation device may include, for example, one or more input sensors capable of detecting the movement of a body part (e.g., a hand or arm) of the operator.

[0049] The storage unit 160 includes temporary or non-temporary storage media such as a read-only memory (ROM), a random access memory (RAM), and a hard disk drive (HDD). The storage unit 160 stores computer programs executed by the control unit 151, trained models (described later), and the like. The computer programs stored in the storage unit 160 include instructions for implementing a method for controlling the product moving device 1 by the control unit 151 (described later with reference to Figures 8 and 9, etc.).

[0050] The storage unit 160 includes an acquired data storage unit 160a and a reference data storage unit 160b. The acquired data storage unit 160a stores, for example, image data captured by the cameras 50R, 50L, 60, and 70. The reference data storage unit 160b stores various data necessary for the operation of the product moving device 1. These various data include, for example, data on the product display shelves 410 and inventory shelves 420 (shape data, position data, lane coordinate data, etc.) and data on the products T (shape data, position data, etc.). In particular, in this embodiment, data on the reference position height of each shelf 411 of the display shelves 410 and data on the height intervals between the shelves 411 are also stored in the reference data storage unit 160b.

[0051] The control unit 151 is configured with, for example, one or more central processing units (CPUs). The control unit 151 executes computer programs stored in the storage unit 160 to function as an operation control unit 152, an imaging control unit 153, and an image data processing unit 155.

[0052] The operation control unit 152 generates control signals for operating the gripping unit 10, the arm unit 20, the horizontal movement mechanism 80, the lifting mechanism 90, and each unit of the control device 150. The operation control unit 152 generates the control signals by referring to input signals from the operation unit 191 and various data stored in the storage unit 160. The control signals may be generated using the processing results of the image data processing unit 155. The operation control unit 152 also transmits and receives data via the communication unit 195, and performs predetermined output via the output unit 193.

[0053] The imaging control unit 153 controls the operation of each of the cameras 50R, 50L, 60, and 70. The imaging timing of each of the cameras 50R, 50L, 60, and 70 may be determined using data stored in advance in the reference data storage unit 160b, for example.

[0054] The image data processing unit 155 performs various information processing using the captured image data and distance data (depth data) captured by each of the cameras 50R, 50L, 60, and 70. As an example, the image data processing unit 155 performs image analysis on the image data captured by the first camera 50R to identify the products lined up on the inventory shelf 420. The image data processing unit 155 has a display eligibility determination unit 155a and a gripping target identification unit 155b.

[0055] The display feasibility determination unit 155a determines, for example, whether or not there is a space Sp (see FIG. 6 ) behind the last product T lined up on the display shelf 410 where an additional product can be placed, based on at least one of the image data captured by the first camera 50L and the distance data. FIG. 6 is an image of the display shelf 410 captured by the first camera 50L of the product moving device 1. If the space Sp exists, it means that the product T needs to be replenished. Therefore, if the space Sp exists, the display feasibility determination unit 155a sends a notification to the operation control unit 151 to notify the operation control unit 152 that the product T should be replenished on the shelf board 411 below the space Sp. When the operation control unit 152 receives this notification, it performs a replenishment operation for the product T.

[0056] The gripping target specifying unit 155b performs at least one of the following processes based on at least one of the image data captured by the first camera 50R and the distance data: determining whether or not a replenishment target product to be gripped is present on the inventory shelf 420; specifying the size or shape of the replenishment target product; and determining a gripping position for the replenishment target product. In the case of a product T having a cap Tb as shown in FIG. 1, the gripping target specifying unit 155b sets the gripping position, for example, near the cap Tb. On the other hand, in the case of a product T such as a canned beverage without a cap, the gripping target specifying unit 155b may set the gripping position to the side portion of the container.

[0057] 7A and 7B are diagrams showing the relative positions of the cap Tb of the product T and the gripping portion 10 when detecting the actual positions of the fingers 11a, 11b (hereinafter, the reference numeral "11") of the gripping portion 10 relative to the cap Tb of the PET bottle beverage product T. Fig. 7A is a side view of the product T and the gripping portion 10 as viewed from the X direction, and Fig. 7B is a top view of the product T and the gripping portion 10 as viewed from the Z direction.

[0058] In this specification and claims, the Z direction, which is the up-down direction (vertical direction) of the product T, is defined as the "first direction," the Y direction, which is a direction approximately perpendicular to the Z direction (first direction) and is the extension direction of the gripping portion 10 (fingers 11), is defined as the "second direction," and the X direction, which is approximately perpendicular to the first and second directions (Z and Y directions), is defined as the "third direction."

[0059] As described above, an error may occur between the position of the product T recognized by the first cameras 50R, 50L provided on the product moving device 1 and the actual position of the product T. Calibration for correcting such an error can be performed by aligning the product T placed on the inventory shelf 420 or the product shelf 410 with the gripper 10 of the product moving device 1. Figure 7 shows a positional relationship (hereinafter referred to as the "reference position") that serves as a reference for aligning the product T with the gripper 10. This reference position includes a Z-direction reference position where the lower surface of the finger 11 of the gripper 10 contacts the upper surface of the cap portion Tb of the product T, a Y-direction reference position where the tip of the finger 11 contacts the vicinity of the side edge of the upper surface of the cap portion Tb of the product T, and an X-direction reference position where the center of the finger 11 in the X direction coincides with the center of the cap portion Tb of the product T in the X direction.

[0060] The product moving device 1 of this embodiment aligns the gripping unit 10 with respect to the product T placed on the inventory shelf 420 or the product shelf 410, obtains the reference positions of the product T in each of the X, Y, and Z directions, and performs calibration to correct the error between the position of the product T recognized by the first cameras 50R and 50L of the product moving device 1 and the actual position of the product T.

[0061] (Product Replenishment Operation by Product Moving Device 1) FIG. 8 is a flowchart showing the product T replenishing operation by the product moving device 1.

[0062] First, in step S11, the rear surface of the display shelf 410 is photographed by the first camera 50L attached to the arm unit 20 of the product moving device 1. The product moving device 1 moves the arm unit 20, horizontal movement mechanism 80, and lifting mechanism 90 so that the first camera 50L can photograph each level of the display shelf 410. Next, the product moving device 1 operates the first camera 50L to capture an image of the rear surface of the display shelf 410 and also acquire distance data to the products T lined up on the display shelf 410.

[0063] Next, in step S12, a determination is made as to whether or not the product can be displayed. The "determination as to whether or not the product can be displayed" is a step in which the display possibility determination unit 155a in the image data processing unit 155 of the product moving device 1 analyzes the captured image of the rear surface of the display shelf and determines which product T can be displayed on which shelf 411 (in other words, which product T needs to be replenished). The product moving device 1 acquires an image of the rear surface of the display shelf 410 as shown in FIG. 6.

[0064] The image data processing unit 155 of the product moving device 1 identifies the lane on the shelf 411 where the product T is to be displayed on the display shelf 410 based on the image data captured by the first camera 50L. Furthermore, the image data processing unit 155 of the product moving device 1 can acquire distance data to the product T (depth data Dep visualized in FIG. 6 ), and therefore can recognize that there is space Sp behind the product T for products whose number of displays has decreased, such as the product T in lanes "7" and "8." The display feasibility determination unit 155a of the product moving device 1 determines whether there is space Sp for the product T based on the distance data to the product T, and determines whether or not more product T can be displayed in lanes "7" and "8." Step S12 determines which product T needs to be replenished on which shelf 411 of the display shelf 410.

[0065] The method for determining whether a product needs to be replenished is not limited to the above method, and various other methods can be used. For example, it may be possible to determine the percentage of the product T that is allocated within a predetermined three-dimensional space, and if this value is equal to or less than a predetermined reference value, it may be determined that the product T needs to be replenished.

[0066] Next, in step S13, the inventory shelf 420 is photographed. The product moving device 1 operates the arm unit 20, the horizontal movement mechanism 80, the lifting mechanism 90, and the first camera 50R to photograph the inventory shelf 420 from the front side. As an example, the product moving device 1 photographs the inventory shelf 420 one shelf at a time to obtain a photographed image showing the inventory status of the products T. The photograph of the inventory shelf 420 does not need to be taken after the photograph of the display shelf 410; the photograph of the inventory shelf 420 may be taken before the photograph of the display shelf 410.

[0067] Next, in step S14, the image data processing unit 155 of the product moving device 1 analyzes the captured image of the inventory shelf 420 acquired in step S13, and identifies the products lined up on the inventory shelf 420. In addition, the grasping target specifying unit 155b of the image data processing unit 155 specifies the grasping position of the product. Through the steps up to this point, the product moving device 1 acquires information indicating which product T needs to be replenished at which position on which shelf 411 of the display shelf 410, as well as information indicating the position on the inventory shelf 420 of the product to be replenished that corresponds to that product T, and the grasping position of that product to be replenished.

[0068] Next, in step S15, the product moving device 1 performs a product replenishment operation (pick-and-place operation) based on the acquired information. Specifically, the operation control unit 152 of the control unit 151 (see FIG. 5 ) of the product moving device 1 operates the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripper 10 toward a predetermined product to be replenished on the inventory shelf 420. The gripper 10 then grips the predetermined gripping position of the product to be replenished (for example, the cap Tb of the product T if the product T is a PET bottle beverage) and lifts the product. The operation control unit 152 then operates the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripped product T to a predetermined placement position on the display shelf 410, and releases the product T from the gripper 10 to place the product T in the predetermined position. Thereafter, the product moving device 1 repeats the same pick-and-place operation to complete the replenishment of the product T.

[0069] The above series of steps are the basic operations performed by the product moving device 1 to automatically replenish the products T from the inventory shelves 420 to the display shelves 410. In the product replenishment operation (pick-and-place operation) in step S15 of this series of steps, if there is an error between the position of the product T recognized by the camera 50 provided in the product moving device 1 and the actual position of the product T, this may cause problems in the gripping operation of the gripping unit 10 to grip the product T.

[0070] For example, if the actual height position (Z direction position) of the gripping portion 10 is deviated from the Z direction reference position described with reference to Fig. 7 and the gripping portion 10 grips the cap portion Tb of the PET bottled beverage, which is the product T, at a height higher than the appropriate gripping position, sufficient gripping force may not be obtained due to reasons such as a reduced contact area between the fingers 11a and 11b of the gripping portion 10 and the cap portion Tb, and the gripped PET bottled beverage may fall off from the gripping portion 10 when moving. Alternatively, if the actual position (Y direction position) of the fingers 11 of the gripping portion 10 in the extension direction is deviated from the Y direction reference position described with reference to Fig. 7 and the gripping portion 10 attempts to grip the cap portion Tb of the PET bottled beverage, which is the product T, at a height shorter than the appropriate gripping position in the extension direction of the fingers 11, the fingers 11 may not reach the appropriate gripping position for the cap portion Tb, and the gripping portion 10 may not be able to grip the product T.

[0071] This embodiment provides a product moving device etc. that enables calibration to be performed without the need for an on-site worker to correct the error between the position of product T recognized by camera 50 and the actual position of the product T.

[0072] <Operation Example> Next, an operation example of the product moving device 1 of this embodiment will be described.

[0073] Fig. 9 is a flowchart for explaining an example of an operation of the product moving device 1 of this embodiment. Fig. 10 is a diagram showing an example of an operation for searching for a Z-direction reference position where the lower surface of the finger 11 of the gripping unit 10 contacts the upper surface of the cap portion Tb of the product T. Fig. 11 is a diagram showing an example of an operation for searching for a Y-direction reference position where the tip of the finger 11 contacts the vicinity of a side edge of the upper surface of the cap portion Tb of the product T. Fig. 12 is a diagram showing an example of an operation for searching for an X-direction reference position where the center of the finger 11 in the X direction coincides with the center of the cap portion Tb of the product T in the X direction.

[0074] First, in step S21, the operation control unit 152 of the control unit 151 (see Figure 5) of the product moving device 1 operates at least one of the gripping unit 10, the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90, and performs an operation to search for the Z-direction reference position of the product T to be aligned, which is placed on the product shelf 410 or the inventory shelf 420, etc.

[0075] 10, the operation of searching for the Z-direction reference position of the product T in step S21 will be described. First, based on the position of the product T recognized by the first camera 50, the operation control unit 152 moves the gripping unit 10 to an initial height position where the fingers 11 of the gripping unit 10 are positioned above the cap portion Tb of the product T (see FIG. 10(a)). At this time, it is preferable that the fingers 11a and 11b of the gripping unit 10 are closed, but at least one of the fingers 11a and 11b may be open enough to contact the cap portion Tb in the following process.

[0076] In step S21, the operation control unit 152 then operates at least one of the gripper 10, the arm 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripper 10 a predetermined distance from the initial height position along the longitudinal direction of the product T toward the cap Tb of the PET bottled beverage product T (see FIG. 10B). More specifically, in this example, the gripper 10 is moved vertically downward (in the minus Z direction in the figure) along the longitudinal direction of the product T from the initial height position. The predetermined distance by which the gripper 10 is moved may be set to a distance, for example, approximately 5 to 20 mm longer than the distance between the cap Tb and the lower surface of the finger 11 of the gripper 10 positioned at the initial height position as described above (this distance may include some error).

[0077] The operation control unit 152 then obtains the pitch angle of the gripping unit 10 after moving the gripping unit 10 a predetermined distance from the contact detection sensor 30, and determines whether the pitch angle of the gripping unit 10 has changed from a predetermined angle (e.g., the angle at which the gripping unit 10 is horizontal).

[0078] On the other hand, if the pitch angle of the gripper 10 changes from the predetermined angle while or after moving the gripper 10 a predetermined distance, it is recognized that the lower surfaces of the fingers 11 of the gripper 10 are in contact with the upper surface of the cap portion Tb of the product T. This is because, as the gripper 10 moves to a predetermined distance while the lower surfaces of the fingers 11 of the gripper 10 are in contact with the upper surface of the cap portion Tb of the product T, the gripper 10 tilts obliquely so that the tips of the fingers 11 point vertically upward. FIG. 10( c) shows a state in which the lower surfaces of the fingers 11 of the gripper 10 are in contact with the upper surface of the cap portion Tb of the product T and the gripper 10 is tilted obliquely. When the operation control unit 152 determines that the pitch angle of the gripper 10 has changed from the predetermined angle, it acquires the "Z-direction reference position (first position)" of the product T based on the height position of the gripper 10 at the time of the determination.

[0079] Next, in step S22, the operation control unit 152 operates at least one of the gripping unit 10, the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to perform an operation to search for the Y-direction reference position of the same product T for which the Z-direction reference position was obtained.

[0080] 11 , the Y-direction reference position search operation for the product T in step S22 will be described. First, based on the position of the product T recognized by the image data processing unit 155 based on the image captured by the first camera 50, the operation control unit 152 moves the gripper 10 to an initial height position above the cap portion Tb of the product T and to a position where the tips of the fingers 11 of the gripper 10 are positioned further rearward in the Y direction than the cap portion Tb (see the gripper 10 in the dotted line portion shown in FIG. 11( a)). At this time, the fingers 11 a and 11 b of the gripper 10 are preferably closed, but at least one of the fingers 11 a and 11 b may be open enough to contact the cap portion Tb in the following steps.

[0081] In step S22, the operation control unit 152 then operates at least one of the gripper 10, the arm 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripper 10 vertically downward (in the minus Z direction in the figure) a predetermined distance from the initial height position. More specifically, in this example, the gripper 10 is moved vertically downward (in the minus Z direction in the figure) a predetermined distance from the initial height position, and the distance is set to be approximately 5 to 20 mm longer than the distance between the lower surface of the finger 11 of the gripper 10 disposed at the initial height position as described above and the cap portion Tb (this distance may include some error) (see the gripper 10 indicated by the solid line in FIG. 11( a)).

[0082] The operation control unit 152 then obtains the pitch angle of the gripping unit 10 after moving the gripping unit 10 a predetermined distance from the contact detection sensor 30, and determines whether the pitch angle of the gripping unit 10 has changed from a predetermined angle (e.g., the angle at which the gripping unit 10 is horizontal).

[0083] If the pitch angle of the gripper 10 remains unchanged from the predetermined angle even after the gripper 10 has been moved a predetermined distance in the negative Z direction as described above, the tip of the finger 11 of the gripper 10 has not yet contacted the cap Tb of the product T. In this case, the system of the product moving device 1 (particularly the operation control unit 152) has not yet determined how far the tip of the finger 11 of the gripper 10 is located in the Y direction relative to the cap Tb of the product T. In this case, the operation control unit 152 returns the gripper 10 in the Z direction to the initial height position (see the gripper 10 in the dotted line portion shown in FIG. 11( a) ) and then moves the gripper 10 a predetermined distance in the Y direction toward the cap Tb of the PET bottle beverage product T (see FIG. 11( b) ). The predetermined distance by which the gripper 10 is moved in the Y direction toward the cap Tb may be, for example, between 3 and 5 mm. The fingers 11 of the gripper 10 are thus moved stepwise in the Y direction towards the cap Tb.

[0084] On the other hand, if the pitch angle of the gripping unit 10 changes from the predetermined angle while or after moving the gripping unit 10 a predetermined distance in the minus Z direction as described above, it is recognized that the tips of the fingers 11 of the gripping unit 10 are in contact with the vicinity of the side edge of the upper surface of the cap portion Tb of the product T. This is because, as the gripping unit 10 moves to a predetermined distance while the lower surfaces of the fingers 11 of the gripping unit 10 are in contact with the upper surface of the cap portion Tb of the product T, the gripping unit 10 tilts obliquely so that the tips of the fingers 11 point vertically upward. Figure 11(c) shows a state in which the lower surfaces of the tip portions of the fingers 11 of the gripping unit 10 are in contact with the side edge of the upper surface of the cap portion Tb of the product T, causing the gripping unit 10 to tilt obliquely.

[0085] In the above-described determination operation, if the operation control unit 152 determines that the pitch angle of the gripping unit 10 has not changed from the predetermined angle, the operation control unit 152 repeats the above-described movement operation of the gripping unit 10 in the minus Z direction and the Y direction and the process of determining whether or not the pitch angle of the gripping unit 10 has changed, until it determines that the pitch angle of the gripping unit 10 has changed from the predetermined angle. On the other hand, if the operation control unit 152 determines that the pitch angle of the gripping unit 10 has changed from the predetermined angle, it acquires the position in the Y direction of the tip of the finger 11 of the gripping unit 10 at the time of the determination as the "Y-direction reference position (second position)" of the product T.

[0086] While the above example shows a case in which the presence or absence of a change in the pitch angle of the gripping unit 10 is determined when the gripping unit 10 is moved in the minus Z direction while the fingers 11 of the gripping unit 10 are gradually brought closer in the Y direction, the opposite operation may be performed such that the gripping unit 10 is initially positioned at a position where the fingers 11 overlap the cap portion Tb of the product T when viewed from the Z direction, and then the gripping unit 10 is gradually moved in the minus Y direction, in which the fingers 11 move away from the cap portion Tb, to determine whether or not the pitch angle of the gripping unit 10 changes when the gripping unit 10 is moved in the minus Z direction. In this case, while the fingers 11 overlap the cap portion Tb, a change in the pitch angle of the gripping unit 10 occurs when the gripping unit 10 is moved in the minus Z direction, but after the fingers 11 move to a position where they do not overlap with the cap portion Tb, no change in the pitch angle of the gripping unit 10 occurs when the gripping unit 10 is moved in the minus Z direction. Therefore, in this case, the position of the tip of the finger 11 of the gripping portion 10 in the Y direction at the last time when a change in the pitch angle of the gripping portion 10 was detected is acquired as the ``Y direction reference position (second position)'' of the product T.

[0087] Next, in step S23, the operation control unit 152 operates at least one of the gripping unit 10, the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to perform an operation to search for the X-direction reference position of the same product T for which the Z-direction and Y-direction reference positions have been obtained.

[0088] 12A and 12B , the operation of searching for the X-direction reference position of the product T in step S23 will be described. In step S23, the operation control unit 152 first moves the gripping unit 10 to a position above the cap portion Tb of the product T and where the fingers 11 of the gripping unit 10 are positioned away from the cap portion Tb in the positive X direction, based on the position of the product T recognized by the image data processing unit 155 on the basis of the image captured by the camera 50 (see the gripping unit 10 in the dotted line portion shown in FIG. 12A(a)(i)). At this time, the fingers 11 a and 11 b of the gripping unit 10 are preferably closed, but at least one of the fingers 11 a and 11 b may be open enough to contact the cap portion Tb in the following steps.

[0089] The operation control unit 152 then operates at least one of the gripper 10, the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripper 10 vertically downward (in the minus Z direction shown in the figure) by a predetermined distance. More specifically, in this example, the gripper 10 is moved vertically downward (in the minus Z direction shown in the figure) by a distance that is, for example, about 5 to 20 mm longer than the distance between the lower surface of the finger 11 of the gripper 10 arranged as described above and the cap portion Tb (this distance may include some error) (see the gripper 10 indicated by the solid line in FIG. 12A(a)(i)).

[0090] The operation control unit 152 then obtains the pitch angle of the gripping unit 10 after moving the gripping unit 10 a predetermined distance from the contact detection sensor, and determines whether the pitch angle of the gripping unit 10 has changed from a predetermined angle (e.g., the angle at which the gripping unit 10 is horizontal).

[0091] If the pitch angle of the gripper 10 remains unchanged from the predetermined angle even after the gripper 10 has been moved a predetermined distance in the negative Z direction as described above, the fingers 11 of the gripper 10 have not yet contacted the cap Tb of the product T. In this case, the system of the product moving device 1 (particularly the operation control unit 152) has not yet determined how far the fingers 11 of the gripper 10 are from the cap Tb of the product T in the X direction. In this case, the operation control unit 152 returns the gripper 10 in the Z direction to the Z direction position (see the gripper 10 in the dotted line portion shown in FIG. 12A(a)(i)), and then moves the gripper 10 a predetermined distance in the direction approaching the cap Tb of the PET bottled beverage product T (the negative X direction) (see FIG. 12A(b)(ii)). The predetermined distance by which the gripper 10 is moved in the direction approaching the cap Tb (the negative X direction) may be, for example, between 3 and 5 mm. The fingers 11 of the gripping part 10 are thus moved stepwise in the minus X direction towards the cap part Tb.

[0092] On the other hand, if the pitch angle of the gripping unit 10 changes from the predetermined angle while or after the gripping unit 10 is moved a predetermined distance in the minus Z direction as described above, it is recognized that the fingers 11 of the gripping unit 10 are in contact with the upper surface of the cap portion Tb of the product T near the side edge in the X direction. This is because the gripping unit 10 moves a predetermined distance while the lower surfaces of the fingers 11 of the gripping unit 10 are in contact with the upper surface of the cap portion Tb of the product T, causing the gripping unit 10 to tilt obliquely so that the tips of the fingers 11 point vertically upward. Figure 12A (c) shows a state in which the lower surfaces of the tips of the fingers 11 of the gripping unit 10 are in contact with the upper surface of the cap portion Tb of the product T, causing the gripping unit 10 to tilt obliquely.

[0093] In the above-described determination operation, if the operation control unit 152 determines that the pitch angle of the gripping unit 10 has not changed from the predetermined angle, the operation control unit 152 repeats the above-described movement operation of the gripping unit 10 in the minus Z direction and the minus X direction and the process of determining whether or not the pitch angle of the gripping unit 10 has changed, until it determines that the pitch angle of the gripping unit 10 has changed from the predetermined angle. On the other hand, if the operation control unit 152 determines that the pitch angle of the gripping unit 10 has changed from the predetermined angle, the position of the gripping unit 10 along the X direction at the time of determination is acquired as the first end position of the product T.

[0094] Next, based on the position of the product T recognized by the image data processing unit 155 based on the image captured by the camera 50, the operation control unit 152 moves the gripping unit 10 to a position above the cap portion Tb of the product T and where the fingers 11 of the gripping unit 10 are positioned away from the cap portion Tb in the minus X direction (see the gripping unit 10 in the dotted line portion shown in FIG. 12B(a)(i)). At this time, it is preferable that the fingers 11a and 11b of the gripping unit 10 are closed, but at least one of the fingers 11a and 11b may be open enough to contact the cap portion Tb in the following steps.

[0095] The operation control unit 152 then operates at least one of the gripping unit 10, the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripping unit 10 vertically downward (in the minus Z direction shown in the figure) by a predetermined distance. More specifically, in this example, the gripping unit 10 is moved vertically downward (in the minus Z direction shown in the figure) by a distance that is, for example, approximately 5 to 20 mm longer than the distance between the lower surface of the finger 11 of the gripping unit 10 arranged as described above and the cap unit Tb (this distance may include some error) (see the gripping unit 10 indicated by the solid line in FIG. 12B(a)(i)).

[0096] The operation control unit 152 then obtains the pitch angle of the gripping unit 10 after moving the gripping unit 10 a predetermined distance from the contact detection sensor 30, and determines whether the pitch angle of the gripping unit 10 has changed from a predetermined angle (e.g., the angle at which the gripping unit 10 is horizontal).

[0097] If the pitch angle of the gripper 10 remains unchanged from the predetermined angle even after the gripper 10 has been moved a predetermined distance in the negative Z direction as described above, the fingers 11 of the gripper 10 have not yet contacted the cap Tb of the product T. In this case, the system of the product moving device 1 (particularly the operation control unit 152) has not yet determined how far the fingers 11 of the gripper 10 are from the cap Tb of the product T in the X direction. In this case, the operation control unit 152 returns the gripper 10 in the Z direction to the Z direction position (see the gripper 10 in the dotted line portion shown in FIG. 12B(a)(i)), and then moves the gripper 10 a predetermined distance in the direction approaching the cap Tb of the PET bottled beverage product T (positive X direction) (see FIG. 12B(b)(ii)). The predetermined distance by which the gripper 10 is moved in the direction approaching the cap Tb (positive X direction) may be, for example, between 3 and 5 mm. The fingers 11 of the gripping part 10 are thus moved stepwise in the positive X direction towards the cap part Tb.

[0098] On the other hand, if the pitch angle of the gripping unit 10 changes from the predetermined angle while or after the gripping unit 10 is moved a predetermined distance in the minus Z direction as described above, the fingers 11 of the gripping unit 10 are recognized as being in contact with the upper surface of the cap portion Tb of the product T near the side edge in the X direction. This is because the gripping unit 10 moves a predetermined distance while the lower surfaces of the fingers 11 of the gripping unit 10 are in contact with the upper surface of the cap portion Tb of the product T, causing the gripping unit 10 to tilt obliquely so that the tips of the fingers 11 point vertically upward. Figure 12B(c) shows a state in which the lower surfaces of the tips of the fingers 11 of the gripping unit 10 are in contact with the upper surface of the cap portion Tb of the product T, causing the gripping unit 10 to tilt obliquely.

[0099] In the above-described determination operation, if the operation control unit 152 determines that the pitch angle of the gripping unit 10 has not changed from the predetermined angle, the operation control unit 152 repeats the above-described movement operation of the gripping unit 10 in the minus Z direction and the plus X direction and the process of determining whether or not the pitch angle of the gripping unit 10 has changed, until it determines that the pitch angle of the gripping unit 10 has changed from the predetermined angle. On the other hand, if the operation control unit 152 determines that the pitch angle of the gripping unit 10 has changed from the predetermined angle, the position of the gripping unit 10 along the X direction at the time of determination is acquired as the second end position of the product T.

[0100] Next, the operation control unit 152 determines the intermediate position between the first end positions in the X direction. This intermediate position may correspond to the center position in the X direction of the cap portion Tb of the product T. The operation control unit 152 acquires this center position as the "X direction reference position" of the product T.

[0101] While the above example shows a case in which the presence or absence of a change in the pitch angle of the gripping portion 10 is determined when the gripping portion 10 is moved in the minus Z direction while the fingers 11 of the gripping portion 10 are gradually brought closer to the minus X direction or the plus X direction, the converse operation may be performed such that the gripping portion 10 is initially positioned at a position where the fingers 11 overlap the cap portion Tb of the product T when viewed from the Z direction, and the presence or absence of a change in the pitch angle of the gripping portion 10 is determined when the gripping portion 10 is moved in the minus Z direction while the fingers 11 are gradually moved in the plus X direction or the minus X direction, which is a direction in which the fingers 11 move away from the cap portion Tb. In this case, while the fingers 11 overlap the cap portion Tb, a change in the pitch angle of the gripping portion 10 occurs when the gripping portion 10 is moved in the minus Z direction. After the fingers 11 move to a position where they do not overlap with the cap portion Tb, the pitch angle of the gripping portion 10 no longer changes when the gripping portion 10 is moved in the minus Z direction. Therefore, in this case, the last position in the positive X direction where a change in the pitch angle of the gripping portion 10 is detected is acquired as the first end position of the product T, and the last position in the negative X direction where a change in the pitch angle of the gripping portion 10 is detected is acquired as the second end position of the product T.

[0102] Next, in step S24, the control unit 151 performs calibration to correct the error between the position of the product T recognized by the image data processing unit 155 based on the image captured by the camera 50 and the actual position of the product T.

[0103] In step S24, the control unit 151 first calculates the differences in the X direction, Y direction, and Z direction between the "reference position" that serves as a reference for aligning the gripping unit 10 with respect to the product T, which is specified based on the X direction reference position, Y direction reference position, and Z direction reference position acquired by the operation control unit 152 in steps S21, S22, and S23 described above, and the position of the product T recognized by the image data processing unit 155 based on the image captured by the first camera 50. These differences in each direction are errors in each direction between the position of the product T recognized by the image data processing unit 155 and the actual position of the product T.

[0104] Next, the control unit 151 corrects the position of the product T in the X, Y, and Z directions recognized by the image data processing unit 155 based on the image captured by the camera 50 by the amount of the error in each direction calculated as described above. This correction performs calibration to correct the error between the position of the product T recognized by the image data processing unit 155 based on the image captured by the camera 50 and the actual position of the product T.

[0105] Thus, according to the product moving device 1 of this embodiment, even if there is an error between the position of the product T recognized by the image data processing unit 155 based on the image taken by the first camera 50 and the actual position of the product T, the reference positions of the actual product T in the up-down direction (Z direction), the front-back direction (Y direction), and the left-right direction (X direction) can be identified by the searching operation of the gripping unit 10, and a process for correcting the error can be performed, thereby making it possible to perform calibration to correct the error autonomously without the help of an on-site worker.

[0106] 9 illustrates an example in which the Z-direction reference position, the Y-direction reference position, and the X-direction reference position are searched for in this order. However, the search order for these reference positions is not limited to this and may be performed in any order. In particular, when the Z-direction reference position is first searched for (step S21), if the Y-direction position or the X-direction position of the gripper 10 is significantly deviated from the original reference position, the finger 11 may not contact the cap portion Tb even when the gripper 10 is moved downward to a height where the finger 11 should contact the top surface of the cap portion Tb of the product T. In such a case, step S21 may be temporarily skipped, and the Y-direction reference position search (step S22) or the X-direction reference position search (step S23) may be performed first, and then the Z-direction reference position search (step S21) may be performed again.

[0107] In addition, although the above example shows a process of acquiring the reference positions of the product T in all of the X, Y, and Z directions and correcting the position error, it is also possible to acquire the reference position in any one of the X, Y, and Z directions and perform a process of correcting the position error only in the direction in which the reference position was acquired. This makes it possible to individually calibrate only the position error in the direction in which an error that interferes with the gripping operation of the gripping unit 10 occurs.

[0108] (Other Operation Examples) In the above-described operation examples, the operation of searching for the X-direction reference position of the product T in step S23 has been described as being performed by the sensor 30 detecting an angle change of the gripping portion 10 that occurs when the finger 11 is brought into contact with the cap portion Tb of the product T. Alternatively, the operation of searching for the X-direction reference position of the product T in step S23 can also be performed based on an image that includes at least a portion of the product T (cap portion Tb) and the finger 11, which is captured by a second camera 60 separate from the first camera 50.

[0109] 13 , another example of the X-direction reference position search operation for the product T in step S23 will be described. The operation control unit 152 first moves the gripper 10 above the cap portion Tb of the product T based on the position of the product T recognized by the first camera 50 (see FIG. 13( a)). FIG. 13( a) is a diagram showing an example of an image of the lower front of the gripper 10 captured by the second camera 60 installed on the arm 20 when the gripper 10 is positioned at that position, and FIG. 13( b) is a plan view of the gripper 10 and the product T viewed from above (from the Z direction) when the gripper 10 is positioned at that position. When the gripper 10 is positioned at the above position, the tips of the fingers 11 of the gripper 10 may be positioned rearward and away from the cap portion Tb in the Y direction as shown in FIG. 13( a), or may be positioned so as to overlap above the cap portion Tb of the product T. At this time, it is preferable that the fingers 11a and 11b of the gripping portion 10 are in a closed state.

[0110] 13 , for example, the image data processing unit 155 in the control unit 151 of the present embodiment generates a center line La of the cap portion Tb that passes through the center of the cap portion Tb in the X direction on the upper surface of the cap portion Tb and extends in the extension direction (Y direction) of the fingers 11 based on the contour shape of the upper surface portion of the cap portion Tb of the product T shown in the captured image, and also generates a center line Lb of the fingers 11 that passes through the center of the fingers 11 of the gripping unit 10 (for example, the boundary between the left and right fingers 11a and 11b in the closed state) and extends in the extension direction (Y direction) of the fingers 11 based on the contour shape of the lower surface portion of the fingers 11 shown in the captured image. These center lines La and Lb may or may not be drawn on the captured image.

[0111] 13 exemplarily illustrates a state in which the center line La of the cap portion Tb is offset by a distance dx in the X direction from the center line Lb of the finger 11. This state may occur, for example, when the X-direction position of the product T captured by the first camera 50 is offset by dx from the X-direction position of the actual product T, causing the finger 11 to be positioned at a distance dx in the negative X direction from the center of the cap portion Tb of the actual product T.

[0112] The center line La of the cap portion Tb can be generated, for example, by using any image recognition technology to identify the most convex parts on both the left and right sides in the X direction of the outline shape of the top surface of the cap portion Tb, and generating a straight line that passes through the middle position between these left and right convex parts in the X direction and extends in the Y direction.Furthermore, the center line Lb of the finger 11 can be generated, for example, by using any image recognition technology to identify the boundary between the left and right fingers 11a and 11b in the closed state, and generating a straight line that passes through this boundary and extends in the Y direction.

[0113] In step S23, the operation control unit 152 then operates at least one of the gripper 10, the arm unit 20, the horizontal movement mechanism 80, and the lifting mechanism 90 to move the gripper 10 in the X direction while maintaining the center line Lb of the fingers 11 extending in the Y direction until the center line Lb of the fingers 11 substantially coincides with the center line La of the cap unit Tb. The operation control unit 152 acquires the position in the X direction of the gripper 10 where the center line Lb of the fingers 11 substantially coincides with the center line La of the cap unit Tb as the "X-direction reference position" of the product T. The other processes in this operation example are similar to those described above with reference to FIG. 9 , and therefore descriptions of those processes will be omitted.

[0114] In the above example, the operation control unit 152 automatically moves the gripping unit 10 to the "X-direction reference position" based on the center lines La and Lb. In contrast, if the product moving device 1 is configured to be remotely operable, the operator may remotely operate the gripping unit 10 to move it in the X direction based on the center lines La and Lb displayed on the display device (not shown) of the operation unit 191 while viewing an image from the camera 60 as shown in FIG. 12( a) .

[0115] Although the present invention has been described above through the embodiments of the invention, the above-described embodiments do not limit the scope of the invention according to the claims. Furthermore, combinations of features described in the embodiments of the present invention may also fall within the technical scope of the present invention. Furthermore, it will be apparent to those skilled in the art that various modifications and improvements can be made to the above-described embodiments.

Claims

1. A product moving device for moving a product placed on an inventory shelf to a display shelf different from the inventory shelf, comprising: an arm unit with a gripping unit for gripping the product; a first imaging unit for acquiring first image data including the product placed on the inventory shelf or the display shelf; a sensor for detecting a change in pitch angle that has occurred in the gripping unit; and a control unit for controlling operation of the gripping unit, the arm unit, the first imaging unit and the sensor, wherein the control unit is configured to perform the following: acquire a position of the product based on the position of the gripping unit when the sensor detects that a change in pitch angle has occurred in the gripping unit when a part of the gripping unit is brought into contact with the product; and correct the position of the product recognized based on the first image data based on the difference between the position of the product recognized based on the first image data and the acquired position of the product.

2. A product moving device as described in claim 1, wherein acquiring the position of the product includes at least one of: acquiring a first position which is the position of the top end of the product in a first direction which is the up-down direction of the product; acquiring a second position which is the position of an end of the product in a second direction which is a direction approximately perpendicular to the first direction and which is an extension direction of the gripping portion; and acquiring a third position which is an intermediate position of the product in a third direction which is approximately perpendicular to the first and second directions.

3. A product moving device as described in claim 2, wherein obtaining the first position includes moving the gripper in the first direction closer to the product until the sensor detects that a change in pitch angle has occurred in the gripper.

4. A product moving device as described in claim 2, wherein obtaining the second position includes: moving the gripping portion a predetermined distance from an initial height position in the first direction, and then moving the gripping portion in a direction opposite to the first direction to the initial height position; and moving the gripping portion a predetermined distance in the second direction so as to approach the product, repeatedly until the sensor detects that a change in pitch angle has occurred in the gripping portion.

5. The acquiring of the third position includes repeatedly executing a process of moving the gripping part from an initial height position in the first direction by a predetermined distance, and then moving the gripping part in a direction opposite to the first direction to the initial height position, and moving the gripping part by a predetermined distance in one direction along the third direction so as to approach the commodity, until the sensor detects that a change in the pitch angle of the gripping part has occurred, thereby acquiring a first end position in the one direction along the third direction; repeatedly executing a process of moving the gripping part from an initial height position in the first direction by a predetermined distance, and then moving the gripping part in a direction opposite to the first direction to the initial height position, and moving the gripping part by a predetermined distance in the other direction opposite to the one direction along the third direction so as to approach the commodity, until the sensor detects that a change in the pitch angle of the gripping part has occurred, thereby acquiring a second end position in the other direction along the third direction; The article moving device of claim 2 , further comprising: determining a center position between the first end position and the second end position in the third direction.

6. A product moving device as described in claim 2, further comprising a second imaging unit that acquires second image data including at least a portion of the product and at least a portion of the gripping part located in the vicinity of the product, wherein the control unit further controls the second imaging unit, and the control unit is configured to: acquire the first position and the second position based on the position of the gripping part when the sensor detects that a change in pitch angle has occurred in the gripping part when a portion of the gripping part is brought into contact with the product; and acquire the third position based on the positional relationship between at least a portion of the product and at least a portion of the gripping part, which is included in the second image data.

7. The product moving device of claim 6, wherein acquiring the third position based on the positional relationship between at least a part of the product included in the second image data and at least a part of the gripping portion is configured to: acquire a central position of at least a part of the product in the third direction based on the second image data; acquire a central position of at least a part of the gripping portion in the third direction based on the second image data; move the gripping portion along the third direction to a position where the central position of at least a part of the gripping portion approximately coincides with the central position of at least a part of the product, and acquire the position in the third direction as the third position.

8. A control method for a product moving device that moves a product placed on an inventory shelf to a display shelf different from the inventory shelf, wherein the product moving device comprises: an arm section with a gripping section for gripping the product; a first imaging section that acquires first image data including the product placed on the inventory shelf or the display shelf; a sensor that detects a change in pitch angle that has occurred in the gripping section; and a control section that controls operation of the gripping section, the arm section, the first imaging section and the sensor, the control method including the following steps executed by the control section: acquiring a position of the product based on the position of the gripping section when the sensor detects that a change in pitch angle has occurred in the gripping section when a part of the gripping section is brought into contact with the product; and correcting the position of the product recognized based on the first image data based on the difference between the position of the product recognized based on the first image data and the acquired position of the product.

9. The control method of claim 8, wherein acquiring the position of the product includes at least one of: acquiring a first position which is a position of an upper end of the product in a first direction which is a vertical direction of the product; acquiring a second position which is a position of an end of the product in a second direction which is a direction substantially perpendicular to the first direction and which is an extension direction of the gripping portion; and acquiring a third position which is an intermediate position of the product in a direction substantially perpendicular to the first and second directions.

10. The control method described in claim 9, wherein the product moving device further comprises a second imaging unit that acquires second image data including at least a portion of the product and at least a portion of the gripping part located in the vicinity of the product, and the control unit further controls the second imaging unit, and the control unit executes the following steps: acquiring the first position and the second position based on the position of the gripping part when the sensor detects that a change in pitch angle has occurred in the gripping part when a portion of the gripping part is brought into contact with the product; and acquiring the third position based on the positional relationship between at least a portion of the product and at least a portion of the gripping part included in the second image data.

11. A program for controlling a product moving device that moves a product placed on an inventory shelf to a display shelf different from the inventory shelf, the product moving device comprising: an arm unit with a gripping unit for gripping the product; a first imaging unit for acquiring first image data including the product placed on the inventory shelf or the display shelf; a sensor for detecting a change in pitch angle that has occurred in the gripping unit; and a control unit for controlling the operation of the gripping unit, the arm unit, the first imaging unit and the sensor, the program causing the control unit to: acquire a position of the product based on the position of the gripping unit when the sensor detects that a change in pitch angle has occurred in the gripping unit when a part of the gripping unit is brought into contact with the product; and correct the position of the product recognized based on the first image data based on the difference between the position of the product recognized based on the first image data and the acquired position of the product.

12. The program described in claim 11, wherein acquiring the position of the product includes at least one of: acquiring a first position which is the position of the top end of the product in a first direction which is the up-down direction of the product; acquiring a second position which is the position of an end of the product in a second direction which is a direction approximately perpendicular to the first direction and which is the extension direction of the gripping portion; and acquiring a third position which is an intermediate position of the product in a direction approximately perpendicular to the first and second directions.

13. The program described in claim 12, wherein the product moving device further comprises a second imaging unit that acquires second image data including at least a portion of the product and at least a portion of the gripping part located in the vicinity of the product, and the control unit further controls the second imaging unit, and the program causes the control unit to: acquire the first position and the second position based on the position of the gripping part when the sensor detects that a change in pitch angle has occurred in the gripping part when a portion of the gripping part is brought into contact with the product, and acquire the third position based on the positional relationship between at least a portion of the product and at least a portion of the gripping part, which is included in the second image data.

Images