Article moving device and method for controlling same

Abstract

One embodiment of the present invention provides an article moving device 100 that comprises: an arm part 110; a camera 70 for capturing a moving image for monitoring the position of another article 10 being conveyed on a belt conveyor 40; and a processor serving as a control part. The processor is configured to: monitor the position of the other article 10 being conveyed on the belt conveyor 40 on the basis of the moving image; cause the article moving device to move an article 10 to a predetermined reference position; determine whether or not the article moving device 100 can move the article 10 from the predetermined reference position to a movement destination position on the belt conveyor 40 without contacting the other article 10 on the belt conveyor 40; and cause the article moving device 100 to execute an operation of moving the article 10 while the article moving device 100 can perform the moving.

Description

Article moving device and control method thereof

[0001] The present invention relates to an article moving device and a control method thereof.

[0002] Patent Document 1 discloses a palletizing robot (article moving device) that picks up items from a pallet and transfers them onto a belt conveyor. In the configuration of Patent Document 1, no items are transported from the upstream side of the belt conveyor, so the palletizing robot can place the items it picks up from the pallet onto the belt conveyor at any timing.

[0003] Japanese Unexamined Patent Publication No. 8-91579

[0004] When the palletizing robot disclosed in Patent Document 1 is set up and arranged to work in cooperation with other robots, workers, etc., the other robots, workers, etc. place packages on the belt conveyor upstream of the transfer area used by the palletizing robot, and the packages are continuously transported to the transfer area on the belt conveyor downstream of that area. In this case, the palletizing robot must place packages on the belt conveyor so as not to interfere with other packages being transported on the belt conveyor, and therefore cannot place packages on the belt conveyor at any arbitrary timing.

[0005] It is an object of one aspect of the present disclosure to provide a means for an article moving device to place an article on a belt conveyor in a manner that does not interfere with other articles being conveyed on the belt conveyor.

[0006] According to one aspect of the present disclosure, there is provided an article moving device for moving an article, the article moving device including: an arm unit having a holder for holding an article; a camera for capturing video images for monitoring the positions of other articles being transported on a belt conveyor on which the article is to be placed; and a control unit for controlling the operation of the holder unit, the arm unit, and the camera. The control unit is configured to monitor the positions of the other articles being transported on the belt conveyor based on the video images acquired from the camera, cause the article moving device to move the article to a predetermined reference position relative to the belt conveyor, determine whether the article moving device can move the article from the predetermined reference position to a predetermined destination position on the belt conveyor without contacting the other articles being transported on the belt conveyor, and, while the article moving device is able to move the article from the predetermined reference position to the destination position on the belt conveyor, cause the article moving device to move the article from the predetermined reference position to the destination position on the belt conveyor.

[0007] Other features and advantages of the present disclosure can be seen from the following description and the accompanying drawings, which are given by way of example and are not exhaustive.

[0008] According to one aspect of the present disclosure, means are provided that allow an article moving device to place an article on a belt conveyor in a manner that does not interfere with other articles being conveyed on the belt conveyor.

[0009] 11 is a schematic plan view showing an item moving device and objects arranged around it in a warehouse. FIG. 12 is a schematic front view showing an item moving device and objects in a warehouse. FIG. 13 is a schematic side view showing the configuration of an item moving device according to the present embodiment. FIG. 14 is a block diagram showing the configuration of an item moving device according to the present embodiment. FIG. 15 is a diagram showing an example of a UI screen for selecting an object type displayed on a display unit. FIG. 16 is a diagram showing how the position, orientation, and dimensions of a model in a virtual world displayed on a display unit are changed in accordance with operation input from a user via an operation unit. FIG. 17 is a flowchart showing an item moving operation by an item moving device. FIG. 18 is a diagram explaining a first operation and a second operation in step S11 shown in FIG. 7. FIG. 19 is a diagram showing an example of a state in which a position or area to which an item is to be moved has been specified. FIG. 19 is a flowchart showing an operation for determining the timing to place an item on a belt conveyor by an item moving device. FIG. 19 is a diagram for explaining the relationship between the range of a safety zone defined on a belt conveyor and the position of an item transported on the belt conveyor. FIG. 19 is a diagram showing a visualization in a virtual space of the relationship between the safety zone shown in FIG. 11 and the position of an item transported on the belt conveyor.

[0010] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0011] First, the layout configuration of a logistics warehouse (hereinafter also simply referred to as a "warehouse") will be described. Fig. 1 is a schematic plan view showing an article moving device in the warehouse and objects arranged around it (items such as products packed in packing boxes, pallets, carts, belt conveyors, etc.). Fig. 2 is a schematic front view showing the article moving device and objects in the warehouse.

[0012] 1 and 2, an article moving device 100 equipped with a robot arm 110, a pallet 20 on which articles 10 such as products packed in packing boxes are placed, a cart 30 for a worker to transport the articles 10, and a belt conveyor 40 for transporting the articles 10 placed on a rotating belt are arranged in a warehouse. The pallet 20, cart 30, and belt conveyor 40 are arranged within a distance range where the robot arm 110 of the article moving device 100 can perform article moving work.

[0013] The article moving device 100 is installed in a fixed state on a platform 50. The platform 50 has two side surfaces 52 and one top surface 54, and a space is formed between the top surface 54 and the floor surface or the like, into which an automated guided vehicle (AGV) 60 (described later) can enter.

[0014] The pallet 20 has two side portions 22 and one top portion 24, and a space is formed between the top portion 24 and the floor surface or the like so that an automatic guided vehicle (AGV) 60 (described later) can enter.

[0015] The cart 30 includes a loading section 32 on which the items 10 are loaded, a plurality of casters 34 installed on the underside of the loading section 32, and a protective fence 36 provided on top of the loading section 32. The cart 30 includes the casters 34, which allow a worker to push the cart 30 around the warehouse. The protective fences 36 are provided on three sides of the loading section 32 and prevent the items 10 loaded on the loading section 32 from collapsing and falling off the cart 30, for example.

[0016] The platform 50 carrying the article moving device 100 and the pallet 20 are transported by an automatic guided vehicle (AGV) 60 (hereinafter referred to as an "AGV"). The AGV 60 can be moved within a warehouse by a hydraulic system. The AGV 60 can move autonomously by sensing the surrounding environment, or can move in response to remote control by a user. The AGV 60 is equipped with a hydraulic system that moves its upper surface 62 up and down. The AGV 60 lowers its upper surface 62 to a lower height and enters under a U-shaped pallet 20 or platform 50. Then, the hydraulic system is activated to raise the upper surface 62, lifting the pallet 20 or platform 50 to a height clear of the floor, and transports the pallet 20 or platform 50 to another location within the warehouse. After moving to the transport position, the AGV 60 lowers its upper surface 62, bringing the pallet 20 or platform 50 into contact with the floor, completing the transport and exiting from underneath. The AGV 60 can also tow a cart 30 to move within the warehouse.

[0017] In this embodiment, a camera 70 is disposed near the belt conveyor 40 to capture moving images of the belt conveyor 40. The camera 70 captures images of the belt conveyor 40 from above so as to detect the relative positions of the articles 10 placed on the belt conveyor 40 and transported along the belt conveyor 40. The camera 70 may have, for example, an imaging element that generates an image in which pixels are arranged two-dimensionally (an RGB image, for example). It is preferable that the camera 70 be installed in a position that does not interfere with the operation of the article moving device 100 or with the work of an operator.

[0018] <Configuration of the article moving device 100> Next, the article moving device 100 according to this embodiment will be described below with reference to Figures 1 to 4. Figure 3 is a side view that schematically shows the configuration of the article moving device 100 according to this embodiment, and Figure 4 is a block diagram that shows the configuration of the article moving device 100 according to this embodiment.

[0019] The article moving device 100 includes an arm unit 110, a base unit 120 that securely supports the arm unit 110, a holding unit 130 provided at the tip of the arm unit 110, a camera 140 provided on the holding unit 130, and a control device 150 that controls the operation of the arm unit 110 and the holding unit 130 and the overall control processing of the article moving device 100, including the various processes described below. The article moving device 100 can function as a cargo handling robot that holds articles 10 and moves them to another position (hereinafter also referred to as a "pick-and-place (PnP) operation"). For example, the article moving device 100 transfers articles 10, such as packaged goods loaded on pallets 20 in a logistics warehouse, onto carts 30 or moves them onto a belt conveyor 40.

[0020] The arm unit 110 has multiple link members 112, 113, and 116. The multiple link members 112, 113, and 116 constitute an articulated robot arm. For example, the articulated robot arm may be a six-axis arm having degrees of freedom in linear directions along the X-axis, Y-axis, and Z-axis, and degrees of freedom around the X-axis, Y-axis, and Z-axis. The articulated robot arm may also have any other mechanism, such as a Cartesian coordinate system robot arm, a polar coordinate system robot arm, a cylindrical coordinate system robot arm, or a SCARA robot arm. The base end of the arm unit 110 is fixed to and placed on the mounting table 50. By moving the link members 112, 113, and 116, the arm unit 110 can move the holding unit 130 within the reachable distance of the arm unit 110.

[0021] The holding unit 130 is, for example, a suction gripper, and is provided with a plurality of suction cups 132 for holding an object, as shown in Fig. 3. The holding unit 130 is provided with suction means (not shown), such as a vacuum pump, and is able to hold the surface of an object with the plurality of suction cups 132 by sucking air from suction ports opened on the inside of each suction cup 132.

[0022] As an example, the camera 140 is installed on a side surface of the holding unit 130 adjacent to the surface on which the suction cup 132 is provided. The camera 140 is used to acquire environmental information about the area around the item moving device 100 below the tip of the arm unit 110. The camera 140 may include, for example, an imaging element that generates an image (e.g., an RGB image) in which pixels are arranged two-dimensionally, and a depth sensor that is a distance detection device that generates distance data. The depth sensor is not limited to a specific type as long as it can acquire distance data to an object. For example, a stereo lens type or a LiDAR (Light Detection and Ranging) type can be used. The depth sensor may, for example, generate a depth image. The camera 140 may also acquire distance data using, for example, an ultrasonic element.

[0023] 4, the control device 150 has a processor 155, a storage unit 160, an operation unit 172, a display unit 174, and an input / output unit 176. Although the control device 150 is depicted as a single element in FIG. 4, the control device 150 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0024] The operation unit 172 is a device for receiving input from a user. The operation unit 172 may be configured with devices for inputting to a computer, such as a keyboard, a mouse, a touch panel, or a remote controller called a VR controller or the like, which is capable of tracking position and posture using infrared rays or the like and is equipped with a trigger button or the like. The operation unit 172 may also have a voice input device such as a microphone. The operation unit 172 may also have a gesture input device that identifies the user's movements through image recognition.

[0025] The display unit 174 is a display device that displays a display screen generated by the processor 155, and may be, for example, a flat display device such as a liquid crystal display or an organic EL display device, or may be a head-mounted display (HMD). The input / output unit 176 is connected to the arm unit 110, the holding unit 130, the camera 140, and the camera 70 of the item moving device 100 by wired or wireless communication, and outputs control signals and inputs acquired information between these components.

[0026] The storage unit 160 includes a temporary or non-temporary storage medium such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a solid state drive (SSD). The storage unit 160 stores a computer program executed by the processor 155. The computer program stored in the storage unit 160 includes instructions for implementing a control method for the article moving device 100 by the processor 155, which will be described later with reference to Figures 7 and 10, etc. The storage unit 160 also at least temporarily stores information received from the cameras 70 and 140 and various data (including intermediate data) generated by processing operations by the processor 155.

[0027] The processor 155 is configured with, for example, one or more CPUs (Central Processing Units). The processor 155 executes a computer program stored in the storage unit 160, thereby mainly managing processing based on inputs made by a user via the operation unit 172, input / output control of the input / output unit 176, display control of the display unit 174, etc. In particular, the processor 155 generates one or more control signals for operating each of the drive units (not shown) of the arm unit 110 and the holding unit 130 and the camera 140, based on user inputs made by the user via the operation unit 172.

[0028] Furthermore, the processor 155 is configured to generate a UI (user interface) screen to be presented to the user and display it on the display unit 174. The UI screen (not shown) includes, for example, a selection button display that provides the user with multiple options. Furthermore, the processor 155 generates images or videos of a virtual world (simulation space) based on real-world images or videos of the surrounding environment of the item movement device 100 captured by the camera 140 of the item movement device 100, and displays the images or videos on the display unit 174. When generating images or videos of the virtual world based on real-world images or videos, the processor 155 establishes a correlation between the real world and the virtual world, for example, by associating a coordinate system of the real world with a coordinate system of the virtual world. Furthermore, images or videos of the real world and images or videos of the virtual world (simulation space) may be displayed simultaneously on the display unit 174.

[0029] The images or videos of the virtual world (simulation space) generated based on real-world images or videos of the surrounding environment of the item moving device 100 also include objects (items 10, pallets 20, carts 30, conveyor belts 40, etc.) that exist in the surrounding environment of the item moving device 100. By establishing a correlation between the real world and the virtual world when the processor 320 generates images or videos of the virtual world based on real-world images or videos, it becomes possible to associate user operations in the virtual world with the operation of the item moving device 100 in the real world, as will be described in detail below.

[0030] Furthermore, the processor 155 of the control device 150 is configured to generate a model corresponding to an object contained in an image or video of the virtual world (simulation space), and to perform processing to assign attribute information of the object to the generated model.

[0031] 5 and 6 are diagrams illustrating the processing operation of the processor 155 of the control device 150 to assign annotation information to an object via a model corresponding to the object.

[0032] When the processor 155 of the control device 150 receives a predetermined operation performed by the user on the operation unit 172 (such as pressing a predetermined button on the controller), the processor 155 displays a UI screen on the display unit 174, which allows the user to select the type of object for which a model is to be generated. FIG. 5 shows an example of a UI screen for selecting an object type, which is displayed on the display unit 174 by the processor 155. The UI screen shown in FIG. 5 presents selectable object types, such as "Box (packed product)," "Conveyor (belt conveyor)," "Pallet (pallet)," and "Cart," and shows a state in which the user is pointing to and selecting "Conveyor (belt conveyor)" on the operation unit 172. The object type selection operation on the UI screen can be performed, for example, by pointing the end of an indication line, which moves in conjunction with the operation unit 172, to the position of an option for the object type desired and pressing a predetermined button on the operation unit 172. By accepting the object type selection operation in this manner, processor 155 subsequently acquires the attributes of the selected object type (in the above example, the attribute is Conveyor (belt conveyor)) as attribute information to be associated with the model to be generated as described below.

[0033] After receiving the selection of the object type via the operation unit 172, the processor 155 then displays a virtual world (simulation space) of the surrounding environment of the article moving device 100 on the display unit 174. At this time, the display unit 174 displays a model for specifying the contour shape, posture, and dimensions of the selected object (in the above example, a conveyor (belt conveyor)) within the same virtual world (simulation space). As an example, the model has a rectangular parallelepiped shape.

[0034] Next, the processor 155 changes the position, dimensions, and orientation of the model within the simulation space displayed on the display unit 174 in accordance with operation input by the user from the operation unit 172. Operation input from the operation unit 172 can be performed, for example, by pointing at the model Mdl and pressing a predetermined button on the controller to drag the model Mdl and move the model Mdl to a desired position and orientation, or by pointing at the model Mdl and pressing a predetermined button on the controller to move any edge or vertex of the model Mdl in a manner similar to changing the dimensions of a so-called bounding box.

[0035] FIG. 6 is a diagram showing how the position, dimensions, and orientation of a model are changed in the simulation space displayed on the display unit 174 in response to an operation input from the operation unit 172 by the user.

[0036] 6(a) is a diagram showing a scene in which the position, dimensions, and orientation of the model are changed in response to operation input from the operation unit 172. FIG. 6(a) shows a state in which the model Mdl displayed in the simulation space is roughly aligned with the position of a scanned image of the conveyor displayed in the same simulation space. From this state, when the position, orientation, and dimensions of each side of the model Mdl are further adjusted in response to operation input from the operation unit 172 by the user, the outline of the model Mdl roughly matches the outer contour of the conveyor in the scanned image, as shown in FIG. 6(b). This completes the generation of the model Mdl.

[0037] When the processor 155 receives a model generation input operation from the user on the operation unit 172 (for example, pressing a predetermined button on the controller), it acquires the position, orientation, and dimensions of each side of the model Mdl specified at that time as the position, orientation, and dimensions of each side of the corresponding object (in this example, a belt conveyor). The data of the model Mdl generated in this manner is stored in the storage unit 160 of the control device 150.

[0038] Through the series of processes described above, the processor 155 assigns annotation information including attribute information and information on the position, orientation, and dimensions (length, width, and height dimensions) to the corresponding object via the model Mdl in the coordinate system of the virtual world (simulation space). Based on the annotation information assigned in this manner, the processor 155 can calculate and recognize the position, orientation, and dimensions of the corresponding object (belt conveyor) in the coordinate system of the real world.

[0039] The processor 155 is also configured to detect objects present in images captured by the camera 140 using any image recognition technology and / or a trained model. As an example, the processor 155 can detect the contour shape of the upper surface of each of the items 10 exposed above based on images captured by the camera 140 from above the items 10 loaded on the pallet 20. The trained model can be generated by performing machine learning using, for example, a neural network composed of multiple layers, each layer including neurons, using image data of various images captured by the camera 140 from above the items 10 loaded on the pallet 20 as described above. Such a neural network may be a deep neural network, such as a convolutional neural network (CNN) with 20 or more layers. Machine learning using such a deep neural network is referred to as deep learning. Alternatively, the trained model described above can be generated using a "Visual Transformer," which is a type of deep neural network based mainly on a self-attention mechanism and is applied to the field of computer vision. The trained model generated in this manner is stored in the storage unit 160.

[0040] <Article Moving Operation by Article Moving Device 100> Next, a series of processes and operations relating to the article moving operation by the article moving device 100 of this embodiment will be described mainly with reference to Fig. 7. Fig. 7 is a flowchart showing the article moving operation by the article moving device 100.

[0041] First, in step S11 , the processor 155 of the article moving device 100 acquires information about the surrounding environment of the article moving device 100 using the camera 140 installed in the holding section 130 .

[0042] The article moving device 100 is moved by the AGV 60, and its location within the warehouse is changed as appropriate. Pallets 20 loaded with articles 10, carts 30 to which the articles 10 are to be moved, belt conveyors 40, and the like are arranged around the moved article moving device 100. The processor 155 causes the article moving device 100 to perform a first operation and a second operation as an operation for acquiring ambient environment information for the article moving device 100 in step S11. Figure 8 is a diagram illustrating the first operation and the second operation in step S11 shown in Figure 7.

[0043] In the first operation, the processor 155 extends each of the link members 112, 113, and 116 of the arm unit 110 of the item moving device 100 linearly so that they protrude from the base unit 120, as shown in Fig. 8(a), and operates the actuators (not shown) of each joint so that the imaging direction of the camera 140 installed in the holding unit 130 faces downward and forward. Thereafter, the processor 155 rotates the entire arm unit 110 by a predetermined rotation angle (up to one revolution) relative to the base unit 120 while capturing an image with the camera 140. According to this first operation, the camera 140, which is located at a higher position, acquires ambient environment information in a first range that includes an area relatively far from the item moving device 100.

[0044] In the subsequent second operation, the processor 155 operates the actuators (not shown) of each joint so that the link members 112, 113, and 116 of the arm unit 110 are bent and the imaging direction of the camera 140 installed in the holding unit 130 faces downward and forward, as shown in Fig. 8(b). Thereafter, the processor 155 rotates the entire arm unit 110 by a predetermined rotation angle (up to one revolution) relative to the base unit 120 while capturing an image with the camera 140. According to this second operation, the camera 140, which is positioned at a lower position, acquires ambient environment information in a second range that includes an area relatively close to the item moving device 100.

[0045] Thus, according to step S11, which includes the first and second operations described above, it is possible to obtain ambient environment information in a relatively wide first range around the item moving device 100, and it is also possible to obtain ambient environment information with higher resolution for objects present in a second range that is relatively close to the item moving device 100.

[0046] Although the above example shows an example in which the operation of acquiring ambient environment information of the article moving device 100 includes the first and second operations, only one of the first and second operations (i.e., only one acquisition operation) may be performed as the operation to acquire ambient environment information. Furthermore, the operation to acquire ambient environment information may be automatically performed under control of the processor 155, or may be performed by manually operating the arm unit 110 and the holding unit 130 through a user input operation via the operation unit 172. In the latter case, by acquiring information by focusing the camera 140 around an object for which ambient environment information is to be acquired, it is possible to acquire information necessary for the operation of the article moving device 100 even with a smaller amount of data than when information about the entire surroundings is automatically acquired.

[0047] Next, in step S12, the processor 155 generates a virtual world (simulation space) that reproduces the environment surrounding the item moving device 100 based on the ambient environment information acquired in step S11, and displays the generated virtual world on the display unit 174. The virtual world displays not only the scenery surrounding the item moving device 100 in the real world (such as the floor of a logistics warehouse), but also each real-world object that exists at least within a range accessible to the item moving device 100. The objects may be represented by two-dimensional or three-dimensional images of the real-world objects acquired by the camera 140, depth maps, point clouds, or the like. Alternatively, the objects may be represented by computer graphics that represent the objects.

[0048] Next, in step S13, the processor 155 assigns annotation information to an object displayed in the virtual world (simulation space) based on selection information (see FIG. 5 ) regarding the type of object to which annotation information is to be assigned, which is input by the user operating the operation unit 172 as described above with reference to FIGS. 5 and 6 , and information regarding the position, orientation, and dimensions of a model determined for the object in the virtual world.

[0049] The annotation information is assigned to the objects in step S13 for each of the various objects (items 10, pallets 20, carts 30, conveyor belts 40, etc.) displayed in the virtual world. In particular, when multiple items 10 are loaded on the pallet 20, annotation information may be assigned individually to each of the items 10. Alternatively, when the multiple items 10 loaded on the pallet 20 are all the same size (length, width, and height), annotation information (information regarding position, orientation, and dimensions) may be assigned to at least one item 10 on the pallet 20. In the latter case, the processor 155 assigns the same annotation information as the annotation information assigned to at least one item 10 to other items 10 detected using any image recognition technology and / or trained model.

[0050] In step S13, the processor 155 assigns annotation information, including attribute information and information regarding the position, orientation, and dimensions, to various objects via the model Mdl in the virtual world coordinate system. By assigning annotation information to the objects in this manner, the processor 155 recognizes that the objects in the virtual space displayed on the display unit 174 are not simply objects occupying a certain volumetric space, but represent objects identified by the assigned annotation information. Based on the assigned annotation information, the processor 155 can calculate and recognize the positions, orientations, and dimensions of various corresponding objects in the real-world coordinate system. In other words, the processor 155 recognizes, based on the annotation information, what types of objects exist around the item moving device 100 in the real world, and in what positions, orientations, and dimensions. The annotation information assigned to each object is at least temporarily stored in the storage unit 160.

[0051] Next, in step S14, the processor 155 determines, for each object recognized based on the annotation information, whether the object is located within a range accessible to the holding unit 130 by the arm unit 110 of the item moving device 100. Information regarding the range accessible to the holding unit 130 by the arm unit 110 of the item moving device 100 is pre-stored in the storage unit 160 as known information. The information regarding the accessible range may be planar information (e.g., a circular or sectorial shape) centered on the item moving device 100, or may be three-dimensional information (e.g., a cylindrical shape, sectorial cubic shape, hemispherical shape, etc.) obtained by adding height information to such planar information. Such information regarding the accessible range is determined based on the dimensions, movable range, etc. of the arm unit 110 and the holding unit 130 of the item moving device 100.

[0052] The processor 155 determines, for each object, whether 1) the entire object is within the accessible range or 2) at least a part of the object is outside the accessible range, based on the information about the accessible range. The determination result is stored at least temporarily in the storage unit 160 in association with the annotation information assigned to each object.

[0053] Next, in step S15, the processor 155 accepts a first input specifying the object (item 10) to be moved by the item moving device 100, and a second input specifying the position or area on the object to which the item 10 is to be moved.

[0054] The first input for specifying an object (article 10) to be moved by the article moving device 100 can be executed, for example, by a user operating the operation unit 172 performing an input operation for specifying a model Mdl corresponding to each article 10 displayed in the virtual space. The model Mdl corresponding to each article 10 can be specified, for example, by pointing at each model Mdl individually and pressing a predetermined button on the controller to confirm the specification. Alternatively, the specification can also be performed by an operation for surrounding multiple models Mdl corresponding to multiple articles 10 with a three-dimensional bounding box.

[0055] FIG. 9 is a diagram showing an example of a state in which a position or area on an object to which an item is to be moved is specified. FIG. 9 shows a state in which the center of a belt conveyor 40 is specified as the position or area on an object to which an item 10 is to be moved. In FIG. 9, a cube-shaped marker P indicating the destination of the item is placed above the center of a model Mdl_1 representing the belt conveyor 40. The marker P is placed at an angle that follows the angle of the model Mdl_1, which means that the item 10 is specified to be transported in the same orientation as the marker P to the position indicated by the marker P on the belt conveyor 40 in the real world. Note that FIG. 9 also shows a model Mdl_2 representing a pallet 20.

[0056] Furthermore, the second input specifying the position or area to which the article 10 is to be moved can be executed, for example, by a user operating the operation unit 172 performing an input operation to specify a model Mdl corresponding to a destination object displayed in the virtual space. More specifically, the specification of the position or area to which the article 10 is to be moved can be performed by specifying a position on the model Mdl corresponding to the belt conveyor 40 when each article 10 is to be moved to a specific position on the belt conveyor 40, or by specifying an area on the model Mdl corresponding to the pallet 20 or cart 30 using a bounding box when each article 10 is to be moved to an area on the pallet 20 or cart 30.

[0057] Next, in step S16, the processor 155 determines, based on the first and second inputs, whether the specified object to be moved (item 10) and each object (pallet 20, cart 30, conveyor belt 40, etc.) related to the position or area on the specified object to be moved are within the accessible range. This determination process can be performed based on the above determination results associated with the annotation information assigned to these specified objects. If the processor 155 determines that the entirety of each of these specified objects is within the accessible range (Y), the processor 155 proceeds to the processing of step S17, which will be described later.

[0058] Otherwise (N), since there is a possibility that the movement operation of the specified item 10 cannot be completed safely, the processor 155 displays a message on the display unit 174 for the object that it has determined is not entirely within the accessible range, such as "The movement operation cannot be performed because ○○ (object) is not within the accessible range. Please move ○○ (object) to a position closer to the robot and run the process from the beginning," and terminates the process.

[0059] Finally, in step S17, the processor 155 performs motion planning for operating the article moving device 100 (particularly the arm unit 110 and the holding unit 130) to move the specified article 10 to the specified position or area based on the user input received in step S15, generates an operation command for operating the article moving device 100, and operates the article moving device 100 based on the operation command to move the specified article 10 to the position or area on the specified object. The processor 155 causes the article moving device 100 to repeatedly execute the pick-and-place operation of the article 10 until all of the specified articles 10 have been moved to the specified positions or areas.

[0060] In the operation of moving the items 10 by the item moving device 100, the operation of picking up the items 10 with the holding unit 130 is controlled by the processor 155, for example, to detect the contour shape of the upper surface of each of the items 10 exposed above based on an image of the multiple loaded items 10 taken from above by the camera 140, and to appropriately align the suction cups 132 of the holding unit 130 with the upper surface of each item 10 based on the detected contour shape of each item 10, and to suction and hold each item 10 with the suction cups 132.

[0061] Furthermore, after the operation of picking up the item 10 with the holder 130, the processor 155 may execute an operation of acquiring the height dimension of the picked-up item 10. For example, an additional camera (not shown) having the same function as the camera 140 may be installed on the base 120 of the item moving device 100, and after the holder 130 picks up the item 10, the processor 155 executes an operation of moving the item 10 once in front of the additional camera and capturing an image of the item 10 held by the holder 130 from the side with the additional camera. The processor 155 then executes a process of applying any image recognition technology to the captured image to determine the height dimension of the item 10, thereby acquiring the height dimension of the item 10.

[0062] On the other hand, the pick-and-place operation of moving the articles 10 to a designated position or area is controlled by the processor 155, for example, to repeatedly move each article 10 to a designated position (for example, the center of the belt conveyor 40 in the above example) and repeatedly perform the gripping and releasing operation by the suction cups 132, or to repeatedly load the articles 10 onto a designated area of ​​the pallet 20 or cart 30. In particular, the latter operation of sequentially loading the articles 10 onto a designated area is controlled by the processor 155 to determine the loading positions and orientations of the articles 10 based on the contour shapes of the top surfaces of the articles 10 to be moved so that the designated area is filled by these contour shapes, and to sequentially load the articles 10 onto the designated area according to the loading positions and orientations.

[0063] As described above, each article 10 is provided with annotation information, which includes information about the height of the article 10. Furthermore, information about the height of each article 10 can be acquired by an additional camera installed in the article moving device 100. Based on this information about the height of the article, when the article 10 held by the holder 130 is moved to a designated position, the processor 155 controls the operation of the arm unit 110 and the holder 130 so that a distance greater than the height of the article 10 is maintained between the holder 130 and the designated position (e.g., the top surface of the pallet 20, cart 30, or belt conveyor 40). This prevents the article 10 held by the holder 130 from being pressed against the pallet 20, cart 30, belt conveyor 40, etc. at the designated position, which could result in damage to the article 10 or the pallet 20, cart 30, belt conveyor 40, etc. at the designated position.

[0064] In particular, when transferring items 10 onto another pallet 20 or cart 30, if annotation information has been individually assigned to each item 10 as described above, the processor 155 takes into account the size (length, width, and height dimensions) of each item 10, performs motion planning to move the items 10 so that part or all of the space on the destination pallet 20 or cart 30 is filled with the volume of the multiple items 10 to be moved, and causes the item moving device 100 to perform the movement operation of the multiple items 10. Alternatively, in cases where the multiple items 10 to be moved are all the same size (length, width, and height dimensions), the processor 155 recognizes that each item 10 at the source has the same assigned size based on the annotation information (particularly information regarding position, posture, and dimensions) assigned to at least one item 10 as described above, and performs motion planning to move the items 10 so that part or all of the space on the destination pallet 20 or cart 30 is filled with the volume of the multiple items 10 to be moved, and causes the item moving device 100 to perform the movement operation of the multiple items 10.

[0065] As described above, according to the article moving device 100 of this embodiment, in a virtual world (simulation space) that reproduces the surrounding environment of the article moving device 100 in the real world, annotation information including attribute information and information regarding the position, attitude, and dimensions of objects (the article 10 to be moved, the pallet 20, cart 30, belt conveyor 40, etc. to be moved) that exist around the article moving device 100 in the real world can be assigned, thereby allowing the article moving device 100 to recognize the attributes, position, attitude, and dimensions of those objects that exist in the real world. Therefore, for example, in an operation in which the article moving device 100 is moved appropriately by the AGV 60 within a logistics warehouse, and objects are placed around the article moving device 100 after the movement to perform an article moving operation, the article moving device 100 can be made to easily and quickly recognize the surrounding environment of the article moving device 100 after the movement.

[0066] Furthermore, the item moving device 100 of this embodiment is configured to, after receiving a user input specifying the item 10 to be moved and a user input specifying the destination position or area, determine whether any objects related to the movement operation of the specified item 10 are present within an accessible range of the item moving device 100, and not execute the item moving operation if any of the objects are not present within the accessible range. This makes it possible to prevent incidents (such as damage to the item 10 or other objects due to the item 10 falling or collapsing, etc.) that may occur if the item moving operation is executed when any of the objects are not present within the accessible range.

[0067] In the above embodiment, when it is determined in the processing of step S16 shown in FIG. 7 that the designated object is not within the accessible range of the item moving device 100 (N), the processing is terminated without executing the operation of moving the item (step S17). However, the operation of moving the item may be executed even when it is determined that the designated object is not within the accessible range of the item moving device 100. As one means for achieving this, when it is determined that the designated object is not within the accessible range of the item moving device 100, the object determined not to be within the accessible range is moved to a position closer to the item moving device 100 by the AGV 60. The movement of the object by the AGV 60 may be performed by a user manually operating the AGV 60 by remote control, or the movement of the AGV 60 may be controlled by the processor 155 of the item moving device 100 while the item moving device 100 and the AGV 60 are communicating with each other via the input / output unit 176 of the item moving device 100.

[0068] <Operation of determining timing to place an article on the belt conveyor 40 by the article moving device 100> Next, a series of processes and operations related to the operation of determining timing to place an article on the belt conveyor 40 by the article moving device 100 of this embodiment will be described mainly with reference to Fig. 10. Fig. 10 is a flowchart showing the operation of determining timing to place an article on the belt conveyor 40 by the article moving device 100.

[0069] First, in step S21, the processor 155 of the item moving device 100 constantly monitors the position information of the item 10 being transported on the belt conveyor 40 based on video image data constantly received from a camera 70 installed near the belt conveyor 40.

[0070] The article moving device 100 may be configured and arranged to cooperate with other article moving devices, workers, etc., in which case articles 10 are placed on the belt conveyor 40 by those other article moving devices, workers, etc. upstream of the destination position P of the article to be moved by the article moving device 100, and those articles 10 are continuously transported to the destination position P downstream of that on the belt conveyor 40. The processor 155 of the article moving device 100 constantly monitors the position information of the articles 10 being transported on the belt conveyor 40 to the destination position P in this manner, in particular the distance between the destination position P of the article on the belt conveyor 40 and one or more articles 10 being transported on the belt conveyor 40 from the upstream side.

[0071] More specifically, the processor 155 of the article moving device 100 uses any image recognition technology to perform image analysis on the video image data constantly received from the camera 70, recognize one or more articles 10 moving on the belt conveyor 40, and track them within the video image. At this time, the processor 155 calculates in real time the distance between the article destination position P on the belt conveyor 40 and the downstream end face or edge of each article 10 being transported on the belt conveyor 40. In this way, the processor 155 constantly monitors the distance between the article destination position P on the belt conveyor 40 and each article 10 being transported from the upstream side on the belt conveyor 40.

[0072] Next, in step S22, the processor 155 of the article moving device 100 operates the article moving device 100 to move the specified article 10 to a predetermined reference position based on the user input received in step S15 of the flowchart shown in Fig. 7. The operation of step S22 is part of the operation described above in relation to step S15 of Fig. 7.

[0073] Here, the "predetermined reference position" is a position through which any article 10 moved from any position passes when being moved to the article destination position on the belt conveyor 40. In other words, the processor 155 of the article moving device 100 performs motion planning so that all articles 10 moved from any position to the article destination position P on the belt conveyor 40 pass through this predetermined reference position. The predetermined reference position is a predetermined position in real space, and may be, for example, a position located a predetermined distance (height) directly above the article destination position P on the belt conveyor 40, but is not limited to this.

[0074] Next, in step S23, the processor 155 determines whether to cause the article moving device 100 to perform the operation of moving the article 10 to the article destination position P on the belt conveyor 40 and placing it thereon.

[0075] In this example, the processor 155 operates the article moving device 100 to move the article 10 at a predetermined operating speed along a predetermined trajectory from a predetermined reference position to the article destination position P on the belt conveyor 40. In other words, the moving distance and moving speed of the article 10 from the predetermined reference position to the article destination position P on the belt conveyor 40 are constant, and therefore the moving time (hereinafter referred to as "T1") is constant. The moving time T1 of the article 10 from the predetermined reference position to the article destination position P on the belt conveyor 40 is calculated in advance by the processor 155 and stored in the memory unit 160.

[0076] Furthermore, the conveying speed of the belt conveyor 40 is constant (e.g., 25 m / min), and the time (hereinafter referred to as "T2") required for the item 10 on the belt conveyor 40 to reach the item destination position P is obtained by dividing the distance between the item destination position P on the belt conveyor 40 and the item 10 being conveyed on the belt conveyor 40 by the conveying speed of the belt conveyor 40. The control device 150 can obtain the conveying speed of the belt conveyor 40 by any method. For example, the control device 150 can obtain the conveying speed information of the belt conveyor 40 by having an operator input the value of the conveying speed of the belt conveyor 40 via the input / output unit 176, or by having the belt conveyor 40 transmit conveying speed information to the control device 150 via wired or wireless communication.

[0077] The values ​​of the times T1 and T2 obtained as described above are compared, and when the value of time T2 is greater than the value of time T1, other articles 10 being transported on the belt conveyor 40 will not arrive at the article destination position P before the article 10 is moved from the predetermined reference position to the article destination position P on the belt conveyor 40 and placed thereon, so it is possible to cause the article moving device 100 to execute the operation of moving the article 10 to the article destination position on the belt conveyor 40 and placing it thereon. On the other hand, when the value of time T2 is the same as or smaller than the value of time T1, other articles 10 being transported on the belt conveyor 40 will arrive at the article destination position P before the article 10 is moved from the predetermined reference position to the article destination position P on the belt conveyor 40 and placed thereon, so if the article moving device 100 were to execute the operation of moving the article 10 to the article destination position P, there is a risk that the articles 10 will interfere with each other, and therefore it is not possible to cause the article moving device 100 to execute such a moving operation.

[0078] Here, the relationship between the range of the safety zone SZ defined on the belt conveyor 40 and the position of the article 10 being transported on the belt conveyor 40 will be described with reference to FIG.

[0079] As described above, the time T1 required for an article 10 to move from a predetermined reference position to the article destination position P on the belt conveyor 40 is constant. Therefore, while another article 10 is positioned at a position where the value of the time T2 required for the article 10 being transported on the belt conveyor 40 to reach the article destination position P is greater than the value of the time T1, the article moving device 100 can perform an operation to place the moving article 10 at the article destination position P on the belt conveyor 40.

[0080] 11(a), the position on the belt conveyor 40 where the value of time T2 is greater than the value of time T1 can be set as an upstream limit position Lu, which is a position away from the item destination position P on the belt conveyor 40 by a distance D obtained by multiplying a predetermined conveying speed of the belt conveyor 40 by the time T2 toward the upstream side in the conveying direction of the belt conveyor 40. As shown in FIG. 11(a), while another item 10 is located upstream of the limit position Lu in the conveying direction of the belt conveyor 40 (to the right in the figure), the position of the other item 10 on the belt conveyor 40 is outside the range of distance D from the item destination position P and satisfies the relationship T1<T2, so the item moving device 100 can perform the operation of placing the item 10 at the item destination position P on the belt conveyor 40.

[0081] On the other hand, as shown in Figure 11 (b), if the other item 10 is located at the limit position Lu or further downstream in the conveying direction of the belt conveyor 40 (on the left side in the figure), the position of the other item 10 on the belt conveyor 40 is within the range of distance D from the destination item position P, and the relationship T1 < T2 is not satisfied (the relationship is T1 = T2 or T1 > T2), so the item moving device 100 cannot perform the operation of placing the item 10 at the destination item position P on the belt conveyor 40. In this case, as shown in Figure 11 (c), the item 10 on the belt conveyor 40 waits to be transported downstream of the downstream limit position Ld, which is a predetermined distance B downstream in the conveying direction from the item destination position P, and if another item 10 that may be located upstream in the conveying direction from the item destination position P at that time is located upstream of the limit position Lu, in other words, if there are no other items 10 within the safety zone SZ that includes the distance D upstream in the conveying direction from the item destination position P and the distance B downstream, the item moving device 100 can perform the operation of placing the item 10 at the item destination position P on the belt conveyor 40.

[0082] 12 is a diagram visualizing in a virtual space the relationship between the safety zone SZ described with reference to FIG. 11 and the position of the article 10 being transported on the belt conveyor 40. In each diagram in FIG. 12, the article 10 is transported on the belt conveyor 40 in the left direction in the drawing.

[0083] 12(a), the item 10 being transported on the belt conveyor 40 is located upstream of the safety zone SZ in the transport direction and outside of that range, so the item moving device 100 is in a "SAFE" status, meaning that the item 10 can be safely placed at the item destination position P on the belt conveyor 40. During this status, the processor 155 can cause the item moving device 100 to execute the operation of placing the item 10 at the item destination position P on the belt conveyor 40. During the "SAFE" status, the safety zone SZ is displayed in blue, for example.

[0084] 12(b), the item 10 being transported on the belt conveyor 40 has moved downstream in the transport direction and is within the range of the safety zone SZ, so the item moving device 100 is in a "NOT SAFE" status, meaning that the item 10 cannot safely be placed at the item destination position P on the belt conveyor 40. During this status, the processor 155 cannot cause the item moving device 100 to execute the operation of placing the item 10 at the item destination position P on the belt conveyor 40. During the "NOT SAFE" status, the safety zone SZ is displayed in red, for example.

[0085] In the state shown in Figure 12(c), the item 10 that was within the range of the safety zone SZ in the state shown in Figure 12(b) has moved further downstream and is now located outside the range of the safety zone SZ downstream, and other items 10 newly transported on the belt conveyor 40 are located upstream of the safety zone SZ in the transport direction, and both items 10 are located outside the range of the safety zone SZ, so the item moving device 100 is in a status "SAFE" state in which it can safely place the item 10 at the item destination position P on the belt conveyor 40. Even during this state, the processor 155 can cause the item moving device 100 to execute the operation of placing the item 10 at the item destination position P on the belt conveyor 40. Figure 12(c) shows the item moving device 100 placing the item 10 at the item destination position P on the belt conveyor 40 while in the status "SAFE".

[0086] Thus, in step S23, the processor 155 determines whether to cause the article moving device 100 to execute the operation of moving and placing the article 10 at the article destination position on the belt conveyor 40, based on whether the position of the article 10 on the belt conveyor 40 is outside the range of the safety zone SZ starting from the article destination position P. If the position of the article 10 on the belt conveyor 40 is outside the range of the safety zone SZ (Y), the processor 155 proceeds to step S24. On the other hand, if the position of the article 10 on the belt conveyor 40 is within the range of the safety zone SZ (N), the processor 155 repeats the determination process of S23 until the articles 10 on the belt conveyor 40 are further transported and their positions are outside the range of the safety zone SZ.

[0087] Next, in step S24, the processor 155 causes the article moving device 100 to move the article 10 to the article destination position P on the belt conveyor 40 and place it there.

[0088] As a result of the operation of step S22, the article moving device 100 has moved the article 10 to the predetermined reference position. In step S24, when the processor 155 commands the article moving device 100 to execute an operation of moving the article 10 to the article destination position P on the belt conveyor 40 and placing the article 10 thereon, the article moving device 100 moves the article 10 from the predetermined reference position to the article destination position P on the belt conveyor 40 along a predetermined trajectory in a predetermined operating time T1 and places the article 10 at position P.

[0089] As described above, if the other item 10 is outside the safety zone SZ on the belt conveyor 40, the item moving device 100 will complete the operation of moving and placing the item 10 from a predetermined reference position to the item destination position P on the belt conveyor 40 before the other item 10 is transported on the belt conveyor 40 and arrives at the item destination position P, so the item moving device 100 can be safely placed on the belt conveyor 40 without the item 10 being moved by the item moving device 100 interfering with the other items 10 being transported on the belt conveyor 40.

[0090] Although the above example shows monitoring the position of the article 10 on the belt conveyor 40 using video image data acquired by the camera 70 installed near the belt conveyor 40, the installation location of the camera 70 that acquires video image data for monitoring the position of the article 10 on the belt conveyor 40 is not limited to this. For example, the camera 70 may be installed on the arm unit 110, base unit 120, or holding unit 130 of the article moving device 100, or may be installed on the belt conveyor 40.

[0091] Although the present invention has been described above through the embodiments of the invention, the above-described embodiments do not limit the scope of the invention according to the claims. Furthermore, combinations of features described in the embodiments of the present invention may also fall within the technical scope of the present invention. Furthermore, it will be apparent to those skilled in the art that various modifications and improvements can be made to the above-described embodiments.

Claims

1. An article moving device for moving an article, comprising: an arm unit with a holding unit for holding the article; a camera for capturing moving images for monitoring the position of other articles being transported on the conveyor belt on which the article is placed; and a control unit for controlling the operation of the holding unit, the arm unit, and the camera, wherein the control unit is configured to: monitor the position of the other articles being transported on the conveyor belt based on the moving images obtained from the camera; move the article by the article moving device to a predetermined reference position relative to the conveyor belt; determine whether the article moving device is capable of moving the article from the predetermined reference position to a predetermined destination position on the conveyor belt without contacting the other articles being transported on the conveyor belt; and cause the article moving device to perform an operation of moving the article from the predetermined reference position to the destination position on the conveyor belt while the article moving device is capable of moving the article from the predetermined reference position to the destination position on the conveyor belt.

2. The article moving device of claim 1, wherein determining whether the article moving device is capable of moving the article from the predetermined reference position to the destination position on the belt conveyor includes: setting a safety zone on the belt conveyor in which the article moving device can place the article on the belt conveyor without contacting the other articles being transported on the belt conveyor; and determining that the article moving device is capable of moving the article from the predetermined reference position to the destination position on the belt conveyor when the other articles being transported on the belt conveyor are outside the range of the safety zone.

3. An article moving device as described in claim 2, wherein the safety zone includes a destination position on the conveyor belt where the article to be moved by the article moving device is placed, and an upstream limit position that defines the boundary of the safety zone upstream of the destination position in the conveyor belt conveying direction, and the distance from the destination position to the upstream limit position is set based on the time it takes the article moving device to move the article from the specified reference position to the destination position on the conveyor belt and the conveying speed of the other articles being conveyed on the conveyor belt.

4. An article moving device as described in claim 3, wherein the safety zone further includes a downstream limit position that defines a boundary of the safety zone downstream in the conveying direction of the belt conveyor from the destination position.

5. The item moving device of claim 1, further comprising an acquisition unit that acquires ambient environment information of the item moving device, and the control unit is configured to further perform the following: cause the acquisition unit to acquire the ambient environment information of the item moving device in the real world; generate a virtual world including objects present around the item moving device in the real world based on the ambient environment information; assign annotation information to the objects in the virtual world including attribute information of the objects and information regarding their position, orientation and dimensions; accept a first input for the objects present in the virtual world that specifies an item to be moved and a second input that specifies a position or area to which the item is to be moved; and cause the holding unit and the arm unit to perform an operation to move the specified item to the specified position or area in the real world based on the first and second inputs.

6. A control method for an article moving device that moves an article, the article moving device comprising: an arm section with a holding section that holds the article; a camera that takes moving images for monitoring the position of other articles being transported on the conveyor belt on which the article is placed; and a control section that controls the operation of the holding section, the arm section, and the camera, the control section including the following operations executed by the control section: monitoring the position of the other articles being transported on the conveyor belt based on the moving images obtained from the camera; moving the article on the article moving device to a predetermined reference position relative to the conveyor belt; determining whether the article moving device is able to move the article from the predetermined reference position to a predetermined destination position on the conveyor belt without contacting the other articles being transported on the conveyor belt; and causing the article moving device to perform an operation of moving the article from the predetermined reference position to the destination position on the conveyor belt while the article moving device is able to move the article from the predetermined reference position to the destination position on the conveyor belt.

7. A computer program executable by a processor, said computer program comprising instructions for carrying out the method according to claim 6.

Images