Article moving apparatus and control method thereof

Abstract

According to one embodiment of the present invention, an article moving device (100) is provided with: an arm section (110) provided with a holding section (130); an acquisition unit (140) that acquires surrounding environment information; and a processor (155) as a control unit. The processor (155) is configured so as to execute the following processes: causing the acquisition unit (140) to acquire the surrounding environment information; generating a virtual space including an object in the real world based on the surrounding environment information; assigning, to the object in the virtual world, annotation information including attribute information of the object and information relating to the position, posture, and size; receiving a first input for specifying an article to be moved and a second input for specifying a position or area of a movement destination of the article for an object existing in the virtual world; and causing the holding portion and the arm portion to perform an operation of moving the specified article to the specified position or region in the real world on the basis of the first input and the second input.

Description

Technical Field

[0001] The invention relates to an article moving device and a control method thereof. Background Art

[0002] In the past, in a logistics warehouse or the like, which is a base station for transporting articles, an article moving device equipped with a robot arm was used to transfer articles such as packaged goods stacked on a pallet to a cart or to move them sequentially to a belt conveyor. In order for the article moving device to perform appropriate operations on articles packaged in packaging boxes, pallets, carts, belt conveyors, etc. (hereinafter collectively referred to as "objects"), it is preferred that the article moving device recognize the attributes, positions, postures, etc. of these objects.

[0003] In Patent Document 1, as a method for assigning attribute information to an object, a method is proposed that includes the following processing: generating a virtual world that reproduces the environment of the real world; acquiring a model corresponding to the object to which the attribute information is assigned in the virtual world; and assigning attribute information including information related to at least a portion of the model to the model.

[0004] In addition, in Patent Document 2, as a method for determining the position and posture of an object, a method is proposed that includes the following processing: generating a virtual world that includes a display of objects in the real world; displaying a model corresponding to the object in the virtual world; overlapping the model with the object in the virtual world; and comparing the object with the model to determine the position and posture of the object.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: International Publication No. 2020 / 218533

[0008] Patent Document 2: International Publication No. 2020 / 235539 Summary of the invention

[0009] Problem that the invention aims to solve

[0010] However, neither Patent Document 1 nor Patent Document 2 provides a scheme for collectively assigning attribute information and information on position and posture to a certain object through a series of operations.

[0011] Furthermore, even if it is possible to obtain information related to the properties, positions, and postures of objects such as articles, pallets, carts, and belt conveyors that exist around an article moving device in a logistics warehouse, etc., through the methods disclosed in Patent Documents 1 and 2, Patent Documents 1 and 2 do not provide a solution for determining whether these objects that exist around an article moving device are configured so that the article moving device can perform article moving operations (such as transferring articles on a pallet to a cart).

[0012] At the work site in a logistics warehouse, etc., the positions of the article moving device and the objects arranged around it are not fixed, and their positions are sometimes appropriately changed according to the moving operation. In the case where the positions of the article moving device and the various objects around it are changed in this way, if the positions of these objects after the change are not arranged at the positions where the article moving device can perform the article moving operation, the article moving device cannot perform the article moving operation.

[0013] One aspect of the present disclosure aims to provide a solution that can easily give an object annotation information related to the attributes, position, posture and size of the object arranged around the object moving device. In addition, another aspect of the present disclosure aims to provide a solution that can determine whether the arrangement position of the object is a arrangement position where the object moving device can perform an object moving operation.

[0014] Solutions for solving problems

[0015] According to one embodiment of the present disclosure, there is provided an article moving device for moving an article, the article moving device comprising: an arm having a holding portion for holding an article; an acquisition portion for acquiring environmental information surrounding the article moving device; and a control portion for controlling the actions of the holding portion, the arm, and the acquisition portion. The control portion is configured to perform the following processing: causing the acquisition portion to acquire environmental information surrounding the article moving device in the real world; generating a virtual world including objects existing around the article moving device in the real world based on the environmental information; giving annotation information including attribute information of the object and information related to position, posture, and size to the object in the virtual world; accepting a first input for specifying an article to be moved and a second input for specifying a position or area of ​​a destination for moving the article for the object existing in the virtual world; and causing the holding portion and the arm to perform an action of moving the specified article to a specified position or area in the real world based on the first input and the second input.

[0016] According to another embodiment of the present disclosure, the control unit is configured to also perform the following processing: determining whether the object specified in the first input and the second input exists within an accessible range of the article moving device, and executing an action of causing the holding unit and the arm unit to move the specified article to a specified position or area when the specified object exists within the accessible range.

[0017] Other features and advantages of the present disclosure can be understood from the following description and the accompanying drawings which are provided by way of example and not by way of exhaustiveness.

[0018] Effects of the Invention

[0019] According to one embodiment of the present disclosure, there is provided a scheme that can easily give an object annotation information related to the attributes, position, posture and size of the object arranged around the object moving device. In addition, according to another embodiment of the present disclosure, there is provided a scheme that can determine whether the arrangement position of the object is a arrangement position where the object moving operation can be performed by the object moving device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic top view showing an article moving device in a warehouse and objects arranged around it.

[0021] Figure 2 This is a schematic front view showing the article moving device and objects in the warehouse.

[0022] Figure 3 It is a side view schematically showing the structure of the article moving device according to the present embodiment.

[0023] Figure 4 This is a block diagram showing the structure of the article moving device according to this embodiment.

[0024] Figure 5 1 is a diagram showing an example of a UI screen for object type selection displayed on the display unit.

[0025] Figure 6 The diagram shows how the position, posture, and size of a model are changed in the virtual world displayed on the display unit according to the operation input from the operation unit by the user.

[0026] Figure 7 This is a flowchart showing the article moving operation of the article moving device.

[0027] Figure 8 It is used to illustrate Figure 7 FIG. 1 is a diagram showing the first action and the second action in step S11.

[0028] Fig. 9This is a diagram showing an example of a state in which a position or area is designated as a destination for moving an article.

[0029] Fig.10 This is a diagram showing a modified example of the article moving device in the present embodiment.

[0030] Fig.11 This is a diagram showing another modified example of the article moving device in the present embodiment. DETAILED DESCRIPTION

[0031] Hereinafter, one embodiment of the present disclosure will be described with reference to the drawings.

[0032] <Logistics warehouse configuration structure>

[0033] First, the configuration structure of a logistics warehouse (hereinafter, also simply referred to as a "warehouse") will be described. Figure 1 This is a schematic plan view schematically showing an article moving device in a warehouse and objects arranged around it (articles such as goods packaged in packing boxes, pallets, carts, belt conveyors, etc.). Figure 2 This is a schematic front view showing the article moving device and objects in the warehouse.

[0034] As an example, Figure 1 and Figure 2 As shown, the warehouse is provided with: an article moving device 100 having a robot arm 110; a pallet 20 for carrying articles 10 such as commodities packed in packaging boxes; a cart 30 for workers to move the articles 10; and a belt conveyor 40 for moving the articles 10 placed on a rotating belt. The pallet 20, the cart 30, and the belt conveyor 40 are arranged within a distance range within which the robot arm 110 of the article moving device 100 can perform an article moving operation.

[0035] The article moving device 100 is installed on the stage 50 in a state of being fixed on the stage 50. The stage 50 has two side surfaces 52 and a top surface 54, and a space into which an automatic conveyor (AGV) 60 described later can enter is formed between the top surface 54 and the floor surface.

[0036] The pallet 20 includes two side surfaces 22 and one upper surface 24 , and a space into which an automatic conveyor vehicle (AGV) 60 described later can enter is formed between the upper surface 24 and the floor surface or the like.

[0037] The cart 30 includes a stacking portion 32 for stacking articles 10, a plurality of casters 34 provided at the lower side of the stacking portion 32, and a protective fence 36 provided on the stacking portion 32. Since the cart 30 includes the casters 34, an operator can push the cart 30 to move it in the warehouse. The protective fence 36 is provided on three sides of the stacking portion 32 to prevent the articles 10 stacked on the stacking portion 32 from collapsing and falling off the cart 30.

[0038] The loading platform 50 and the pallet 20 on which the article moving device 100 is mounted can be moved in the warehouse by an automatic transport device 60 (hereinafter referred to as "AGV (Automatic Guided Vehicle)"). The AGV 60 can sense the surrounding environment and move autonomously, or it can also move according to the user's remote operation. The AGV 60 is provided with a hydraulic system for moving its upper surface portion 62 in the up and down directions. After the AGV 60 enters the "コ"-shaped pallet 20 or the loading platform 50 under the state of lowering the upper surface portion 62 to make the height lower, it operates the hydraulic system to raise the upper surface portion 62 to lift the pallet 20 or the loading platform 50 to a height separated from the floor surface, and transports the pallet 20 or the loading platform 50 to other locations in the warehouse in a loaded state. After moving to the transport position, the AGV 60 lowers the upper surface portion 62 to make the pallet 20 or the loading platform 50 contact the floor surface to end the transport, and exits from under it. In addition, the AGV 60 can also tow the cart 30 to move it within the warehouse.

[0039] <Structure of Article Moving Device 100>

[0040] Next, refer to Figure 1 to Figure 4 The article moving device 100 according to this embodiment will be described below. Figure 3 is a side view schematically showing the structure of the article moving device 100 according to the present embodiment. Figure 4 It is a block diagram showing the structure of the article moving device 100 according to the present embodiment.

[0041] The article moving device 100 includes an arm 110, a base 120 that fixes and supports the arm 110, a holding portion 130 that is provided at the front end of the arm 110, a camera 140 that is provided at the holding portion 130, and a control device 150 that manages the overall control process of the article moving device 100, including the motion control of the arm 110 and the holding portion 130 and each process described later. The article moving device 100 can function as a loading and unloading robot that performs an action of holding an article 10 and moving it to another location (hereinafter also referred to as a "pick and place (PnP) action"). The article moving device 100, for example, transfers an article 10 such as a packaged product stacked on a pallet 20 in a logistics warehouse to a cart 30, or moves the article 10 to a belt conveyor 40.

[0042] The arm 110 has a plurality of link members 112, 113, 116. The plurality of link members 112, 113, 116 constitute a multi-joint robot. As an example, the multi-joint robot can also be a 6-axis arm having degrees of freedom in the linear directions along the X-axis direction, the Y-axis direction, and the Z-axis direction, and having degrees of freedom in each direction around the X-axis, around the Y-axis, and around the Z-axis. In addition, the multi-joint robot can also have any mechanism such as an orthogonal coordinate robot arm, a polar coordinate robot arm, a cylindrical coordinate robot arm, a SCARA (Selective Compliance Assembly Robot Arm: Selective Compliance Assembly Robot Arm) type robot arm. The base end portion of the arm 110 is mounted on the above-mentioned mounting table 50 in a state fixed to the above-mentioned mounting table 50. The arm 110 can move the holding portion 130 within the distance range that the arm 110 can reach by moving each link member 112, 113, 116.

[0043] As an example, the holding portion 130 is a suction gripper, such as Figure 3 As shown, the holding unit 130 includes a plurality of suction cups 132 for holding the article. The holding unit 130 includes a suction unit (not shown) such as a vacuum pump, and can hold the surface of the article by the plurality of suction cups 132 by sucking air from a suction port opened inside each suction cup 132 .

[0044] As an example, the camera 140 is disposed on a side surface of the holding portion 130 adjacent to the surface on which the suction cup 132 is disposed. The camera 140 is used to obtain environmental information about the surroundings of the article moving device 100 in the area below the arm 110 from near the front end thereof. The camera 140 may also have, for example: an imaging element that generates a camera image (an RGB image in one example) in which pixels are arranged in a two-dimensional manner; and a depth sensor that is a distance detection device that generates distance data. The depth sensor is not limited to a specific method as long as it can obtain distance data to the object. For example, a stereo lens method or a LiDAR (Light Detection and Ranging) method can be used. The depth sensor may also generate a depth image, for example. The camera 140 may also obtain distance data using an ultrasonic element, for example.

[0045] <Structure of Control Device 150>

[0046] like Figure 4 As shown, the control device 150 includes a processor 155, a storage unit 160, an operation unit 172, a display unit 174, and an input / output unit 176. Figure 4 In FIG. 1 , the control device 150 is depicted as a single element, but the control device 150 does not necessarily need to be physically one element, and may be composed of a plurality of physically separate elements.

[0047] The operation unit 172 is a device for accepting input from the user. In addition to the keyboard, mouse, and touch panel, the operation unit 172 can also be composed of a tracker that can track the position and posture using infrared rays and has a trigger button, a remote controller called a VR controller, etc., which is used to input to the computer. The operation unit 172 may also have a sound input device such as a microphone. In addition, the operation unit 172 may also have a posture input device that recognizes the user's movement by image recognition.

[0048] The display unit 174 is a display device for displaying a display screen generated by the processor 155, and may be, for example, a flat display device such as a liquid crystal display or an organic EL display device, or may be a head mounted display (HMD). The input / output unit 176 is connected to the arm 110, the holding unit 130, and the camera 140 of the article moving device 100 through wired communication or wireless communication, and outputs control signals and inputs acquired information to and from these components.

[0049] The storage unit 160 includes a temporary or non-temporary storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), or an SSD (Solid State Drive). The storage unit 160 stores computer programs executed by the processor 155. The computer programs stored in the storage unit 160 include reference Figure 7 The storage unit 160 also stores at least temporarily information received from the camera 140 and various data generated by the processing operation of the processor 155 (including intermediate generated data).

[0050] The processor 155 is composed of, for example, one or more CPUs (Central Processing Units). The processor 155 mainly manages processing based on inputs made by the user via the operation unit 172, input / output control of the input / output unit 176, display control of the display unit 174, etc. by executing computer programs stored in the storage unit 160. In particular, the processor 155 generates one or more control signals for operating the arm 110 and each drive unit (not shown) of the holding unit 130 and the camera 140 based on the user inputs input from the user via the operation unit 172.

[0051] Furthermore, the processor 155 is configured to generate a UI (user interface) screen to be presented to the user, and display the UI screen on the display unit 174. The UI screen (not shown) includes, for example, a selection button display that provides the user with a plurality of options. Furthermore, the processor 155 generates an image or a moving image of a virtual world (simulated space) based on an image or a moving image of the real world of the surrounding environment of the article moving device 100 acquired by the camera 140 of the article moving device 100, and displays it on the display unit 174. When the processor 155 generates an image or a moving image of the virtual world based on an image or a moving image of the real world, for example, the coordinate system of the real world corresponds to the coordinate system of the virtual world, thereby establishing a correlation between the real world and the virtual world. Furthermore, the image or the moving image of the real world and the image or the moving image of the virtual world (simulated space) may also be displayed on the display unit 174 at the same time.

[0052] The image or moving image of the virtual world (simulated space) generated based on the image or moving image of the real world surrounding the article moving device 100 also includes objects (articles 10, pallets 20, carts 30, belt conveyors 40, etc.) existing in the surrounding environment of the article moving device 100. By establishing a correlation between the real world and the virtual world when the processor 320 generates the image or moving image of the virtual world based on the image or moving image of the real world, it is possible to associate the operation of the user in the virtual world with the action of the article moving device 100 in the real world, as described in detail below.

[0053] Furthermore, the processor 155 of the control device 150 is configured to perform a process of generating a model corresponding to an object included in an image or a moving image of a virtual world (simulated space) and providing attribute information of the object to the generated model.

[0054] Figure 5 and Figure 6 This is a diagram for explaining a processing operation of the processor 155 of the control device 150 to add annotation information to an object using a model corresponding to the object.

[0055] When receiving a predetermined operation (such as pressing a predetermined button of the controller) performed by the user via the operation unit 172 , the processor 155 of the control device 150 displays a UI screen on the display unit 174 for the user to select the type of object for which a model is to be generated. Figure 5 FIG. 2 shows an example of a UI screen for selecting an object type displayed on the display unit 174 by the processor 155. Figure 5 In the UI screen shown, "Box (packaged product)", "Conveyor (conveyor belt)", "Pallet (pallet)", and "Cart (cart)" are presented as the types of objects that can be selected, and the user indicates "Conveyor (conveyor belt)" through the operation unit 172 to select it. For example, the object type selection operation in the UI screen can be performed by indicating the destination of the indication line that moves in conjunction with the operation unit 172 to the position of the option of the object type to be selected and pressing a specified button of the operation unit 172. By accepting the object type selection operation in this way, the processor 155 obtains the attribute of the selected object type (in the above example, Conveyor (conveyor belt) as an attribute) as attribute information associated with the model generated as described below.

[0056] When the processor 155 receives the selection of the object type by the operation unit 172, it then displays the virtual world (simulation space) of the surrounding environment of the article moving device 100 on the display unit 174. At this time, a model for specifying the outline shape, posture, and size of the selected object (in the above example, the conveyor) is displayed in the same virtual world (simulation space) on the display unit 174. As an example, the model has a rectangular parallelepiped shape.

[0057] Next, the processor 155 changes the position, size, and posture of the model in the simulation space displayed on the display unit 174 according to the operation input by the user from the operation unit 172. The operation input of the operation unit 172 can be performed by, for example, dragging the model Md1 and moving it to a desired position and posture by pointing to the model Md1 and pressing a predetermined button of the controller; and moving any edge or vertex in the model Md1 by pointing to the model Md1 and pressing a predetermined button of the controller according to the point of changing the size of the so-called bounding box.

[0058] Figure 6 The diagram shows how the position, size, and posture of the model are changed in the simulation space displayed on the display unit 174 according to the operation input by the user through the operation unit 172 .

[0059] Figure 6 (a) is a diagram showing a scene in which the position, size, and posture of the model are changed according to the operation input from the operation unit 172. Figure 6 (a) shows a state in which the approximate position of the model Md1 displayed in the simulation space is aligned with the position of the scanned image of the conveyor displayed in the same simulation space. When the position, posture, and size of each side of the model Md1 are further adjusted according to the operation input by the user from the operation unit 172, the state changes from this state to the state shown in FIG. Figure 6 As shown in (b) of FIG. 1 , the contour of the model Md1 is substantially consistent with the outer contour of the conveyor in the scanned image. Thus, the generation of the model Md1 is completed.

[0060] When the processor 155 receives the input operation of model generation performed by the user in the operation unit 172 (for example, pressing a predetermined button of the controller), the position, posture and size of each side of the model Mdl specified at that time point are obtained as the position, posture and size of each side of the corresponding object (in this example, the belt conveyor). The data of the model Mdl generated in this way is stored in the storage unit 160 of the control device 150.

[0061] Through a series of processing as described above, the processor 155 gives annotation information including attribute information and information related to position, posture, and size (vertical, horizontal, and height dimensions) to the corresponding object by means of the model Md1 in the coordinate system of the virtual world (simulated space). The processor 155 can calculate the position, posture, and size of the corresponding object (belt conveyor) in the coordinate system of the real world based on the annotation information thus given and recognize it.

[0062] In addition, the processor 155 is configured to detect an object existing in the image based on the image captured by the camera 140 using any image recognition technology and / or a learned model. As an example, the processor 155 can detect the contour shape of the upper surface of each item 10 exposed upward based on an image obtained by photographing the plurality of items 10 stacked on the tray 20 from above by the camera 140. The learned model can be generated by performing machine learning using image data of various images obtained by photographing the plurality of items 10 stacked on the tray 20 from above by the camera 140 as described above, for example, by a neural network composed of a plurality of layers including neurons in each layer. As such a neural network, for example, a deep neural network such as a convolutional neural network (CNN: Convolutional Neural Network) having 20 or more layers can also be used. Machine learning using such a deep neural network is called deep learning. Alternatively, such a learned model can also be generated using a "Visual Transformer" that applies a Transformer, which is a type of deep neural network mainly based on a self-attention mechanism, to the field of computer vision. The learned model generated in this way is stored in the storage unit 160.

[0063] <Article Moving Operation of Article Moving Device 100>

[0064] Next, the main reference Figure 7 , a series of processes and actions related to the article moving operation of the article moving device 100 according to this embodiment are described. Figure 7 1 is a flowchart showing the article moving operation of the article moving device 100 .

[0065] First, in step S11 , the processor 155 of the article moving device 100 acquires the surrounding environment information of the article moving device 100 through the camera 140 provided in the holding unit 130 .

[0066] The article moving device 100 is moved by the AGV 60 and its position in the warehouse is appropriately changed. A pallet 20 on which articles 10 are stacked, a cart 30, a belt conveyor 40, and the like, which are the destinations for moving the articles 10, are arranged around the moved article moving device 100. The processor 155 causes the article moving device 100 to perform a first action and a second action as an action of acquiring the surrounding environment information of the article moving device 100 in step S11. Figure 8 It is used to illustrate Figure 7 FIG. 1 is a diagram showing the first action and the second action in step S11.

[0067] In the first action, Figure 8 As shown in (a), the processor 155 operates the actuators (not shown) of the joints so that the link members 112, 113, and 116 of the arm 110 of the article moving device 100 extend linearly in a manner protruding from the base 120, and the shooting direction of the camera 140 provided on the holding portion 130 is directed to the lower front side. Then, the processor 155 rotates the entire arm 110 relative to the base 120 by a predetermined rotation angle (maximum one rotation) while performing shooting with the camera 140. According to such a first action, the camera 140 disposed at a higher position acquires the surrounding environment information within a first range including an area relatively far from the article moving device 100.

[0068] In the next second action, if Figure 8 As shown in (b), the processor 155 operates the actuators (not shown) of the joints to bend the link members 112, 113, and 116 of the arm 110 and direct the shooting direction of the camera 140 provided on the holding portion 130 toward the lower front side. Then, the processor 155 rotates the entire arm 110 relative to the base 120 by a predetermined rotation angle (maximum one rotation) while performing the shooting of the camera 140. According to this second action, the camera 140 disposed at a lower position acquires the surrounding environment information within the second range including the area relatively close to the article moving device 100.

[0069] In this way, according to step S11 including the above-mentioned first action and second action, the surrounding environment information within a relatively large first range around the article moving device 100 can be obtained, and the surrounding environment information can be obtained with higher resolution for objects existing in a second range relatively close to the article moving device 100.

[0070] In the above, an example is shown in which the action of acquiring the surrounding environment information of the article moving device 100 includes the first action and the second action. However, as the action of acquiring the surrounding environment information, only one of the first action and the second action may be performed (i.e., only one acquisition action). In addition, the action of acquiring the surrounding environment information may be automatically performed based on the control of the processor 155, or may be performed by manually operating the arm 110 and the holding part 130 through the user input operation via the operation part 172. In the latter case, by focusing the camera 140 toward the surroundings of the object for which the surrounding environment information is to be acquired, the information required for the action of the article moving device 100 can be acquired even if the amount of data is less than the amount of data in the case of automatically acquiring the information of the entire surroundings.

[0071] Next, in step S12, the processor 155 generates a virtual world (simulated space) that reproduces the surrounding environment of the article moving device 100 based on the surrounding environment information acquired in the above step S11, and displays the virtual world on the display unit 174. In addition to displaying the surrounding scenery of the article moving device 100 in the real world (the floor of the logistics warehouse, etc.), the virtual world also displays various objects in the real world that exist at least within the range accessible by the article moving device 100. The object may also be a representation based on a two-dimensional or three-dimensional image, a depth map, or a point cloud of an object in the real world obtained by the camera 140. Alternatively, it may be represented by a computer graphic representing the object.

[0072] Next, in step S13, the processor 155 adds annotation information to the objects displayed in the virtual world (simulated space). Figure 5 and Figure 6 As described above, the selection information of the type of the object to which the annotation information is to be given is input by the user operating the operation unit 172 (see Figure 5 ), as well as information related to the position, posture, and size of the model determined for the object in the virtual world, to give annotation information to the object.

[0073] The annotation information of step S13 is assigned to each object (article 10, tray 20, cart 30, belt conveyor 40, etc.) displayed in the virtual world. In particular, when a plurality of articles 10 are stacked on the tray 20, annotation information may be assigned to each article 10 independently, or when the plurality of articles 10 stacked on the tray 20 are all of the same size (dimensions of each part in the longitudinal direction, transverse direction, and height), annotation information (information related to position, posture, and size) may be assigned to at least one article 10 on the tray 20. In the latter case, the processor 155 assigns the same annotation information as the annotation information assigned to at least one article 10 to other articles 10 detected using any image recognition technology and / or a learned model.

[0074] Through this step S13, the processor 155 gives annotation information including attribute information and information related to position, posture and size to various objects by means of the model Md1 in the coordinate system of the virtual world. By giving annotation information to objects in this way, the processor 155 recognizes that the objects displayed in the virtual space of the display unit 174 are not objects that only occupy a certain volume space, but represent objects determined according to the given annotation information. The processor 155 can calculate the position, posture and size of various corresponding objects in the coordinate system of the real world based on the annotation information given in this way and recognize them. In other words, the processor 155 recognizes which type of object exists around the object moving device 100 in the real world in what position, posture and size based on the annotation information. The annotation information given to each object is at least temporarily stored in the storage unit 160.

[0075] Next, in step S14, the processor 155 determines whether each object identified based on the annotation information exists within the range accessible to the holding unit 130 through the arm 110 of the article moving device 100. Information related to the range accessible to the holding unit 130 through the arm 110 of the article moving device 100 is pre-stored in the storage unit 160 as known information. The information related to the accessible range can be either plane information centered on the article moving device 100 (e.g., a circular shape, a sector shape, etc.), or three-dimensional information obtained by adding height information to such plane information (e.g., a cylindrical shape, a sector-shaped cube shape, a hemispherical shape, etc.). Such information related to the accessible range is determined based on the size, movable range, etc. of the arm 110 and the holding unit 130 of the article moving device 100.

[0076] The processor 155 determines for each object whether 1) the entire object is within the accessible range or 2) at least a portion of the object is outside the accessible range based on the information related to the accessible range. The determination result is at least temporarily stored in the storage unit 160 in a manner associated with the annotation information assigned to each object.

[0077] Next, in step S15 , the processor 155 receives a first input for specifying an object (article 10 ) to be moved by the article moving device 100 , and a second input for specifying a position or area on the object as a destination for moving the article 10 .

[0078] The first input for specifying the object (article 10) to be moved by the article moving device 100 can be performed, for example, by the user operating the operation unit 172 performing an input operation to specify the model Mdl corresponding to each article 10 displayed in the virtual space. The designation of the model Mdl corresponding to each article 10 can be performed, for example, by independently indicating each model Mdl and pressing a predetermined button of the controller. Alternatively, it can also be performed by an operation of surrounding a plurality of models Mdl corresponding to a plurality of articles 10 with a three-dimensional boundary frame.

[0079] Fig. 9 FIG. 1 is a diagram showing an example of a state in which a position or area on an object is designated as a destination for moving an article. Fig. 9 , a state in which the central portion of the belt conveyor 40 is designated as a position or area on the object as a destination for moving the article 10 is shown. Fig. 9 In the example, a cube-shaped mark P indicating the destination of the article is arranged in the center of the model Mdl_1 representing the belt conveyor 40. The mark P is arranged in a state of being inclined along the inclination of the model Mdl_1, which means that the article 10 is transported to the position indicated by the mark P on the belt conveyor 40 in the real world in the same posture as the mark P. In addition, Fig. 9 A model Md1_2 representing the tray 20 is also shown.

[0080] In addition, the second input for specifying the position or area as the destination for moving the article 10 can be performed, for example, by the user operating the operation unit 172 performing an input operation to specify the model Md1 corresponding to the object as the destination for movement displayed in the virtual space. More specifically, the position or area as the destination for movement of the article 10 can be specified by specifying a position on the model Md1 corresponding to the belt conveyor 40 when each article 10 is to be moved to a specific position on the belt conveyor 40, or by specifying an area on the model Md1 corresponding to the pallet 20 or the cart 30 by a bounding box when each article 10 is to be moved to an area on the pallet 20 or the cart 30.

[0081] Next, in step S16, the processor 155 determines whether the object (article 10) designated as the moving object and each object (tray 20, cart 30, belt conveyor 40, etc.) related to the position or area on the object designated as the moving destination exists within the accessible range based on the first input and the second input. This determination process can be performed based on the above-mentioned determination result associated with the annotation information given to these designated objects. When the processor 155 determines that the entirety of each object exists within the accessible range for all these designated objects ("Yes"), it proceeds to the processing of step S17 described later.

[0082] Otherwise ("No"), the moving action of the specified object 10 may not be completed safely. Therefore, regarding the object determined to be not in the accessible range as a whole, the processor 155 causes the display unit 174 to display a message such as "Since ○○ (object) does not exist in the accessible range, the moving action cannot be performed. Please move ○○ (object) to a position close to the robot and start the process from the beginning." and ends the processing.

[0083] Finally, in step S17, the processor 155 performs motion planning for moving the article moving device 100 (particularly the arm 110 and the holding unit 130) in a manner to move the designated article 10 to the designated position or area based on the user input received in step S15, and generates a motion command for moving the article moving device 100. Based on the motion command, the article moving device 100 is moved to the designated position or area on the object. The processor 155 causes the article moving device 100 to repeatedly perform the picking and placing actions of the article 10 until all designated articles 10 are moved to the designated position or area.

[0084] In the movement action of the object moving device 100 for the object 10, as an example, the processor 155 controls the action of picking up the object 10 using the holding unit 130, so that the contour shape of the upper surface of each object 10 exposed upward is detected based on the image obtained by the camera 140 photographing the stacked multiple objects 10 from above, and based on the detected contour shape of each object 10, the suction cup 132 of the holding unit 130 is appropriately positioned to the upper surface of each object 10, and each object 10 is adsorbed and held by the suction cup 132.

[0085] Furthermore, after the holding unit 130 picks up the article 10, the processor 155 may also perform an operation of acquiring the height of the picked-up article 10. For example, an additional camera (not shown) having the same function as the camera 140 is provided on the base unit 120 of the article moving device 100, and the processor 155 performs an operation of the holding unit 130 moving the article 10 once to the front of the additional camera after picking up the article 10, and the additional camera photographing the article 10 held in the holding unit 130 from the side. Then, the processor 155 can acquire the height of the article 10 by performing a process of applying any image recognition technology to the photographed image to obtain the height of the article 10.

[0086] On the other hand, as an example, the processor 155 controls the pick-up and place actions of moving the articles 10 to the designated positions or areas, so that each article 10 is repeatedly moved to the designated position (for example, the central part of the belt conveyor 40 in the above example) and repeatedly releases the grip of the suction cup 132, or repeatedly stacks the articles 10 in sequence on the designated areas of the tray 20 and the cart 30. In particular, the processor 155 controls the actions of sequentially stacking the articles 10 in the designated areas as in the latter, so that the stacking positions and postures of the articles 10 are obtained based on the contour shapes of the upper surfaces of the articles 10 to be moved so as to fill the designated areas with these contour shapes, and the articles 10 are sequentially stacked in the designated areas according to the stacking positions and postures.

[0087] In addition, as described above, annotation information is given to each article 10, and the annotation information particularly includes information on the height dimension of the article 10. In addition, the information on the height dimension of each article 10 can also be acquired by an additional camera provided in the article moving device 100. When the article 10 held by the holding unit 130 is moved to a designated position based on such information on the height dimension of the article, the processor 155 controls the operation of the arm 110 and the holding unit 130 so that a gap larger than the height dimension of the article 10 is maintained between the holding unit 130 and the designated position (the upper surface of the pallet 20, the cart 30, the belt conveyor 40, etc.) . Thus, it is possible to prevent the article 10 held by the holding unit 130 from being pressed against the pallet 20, the cart 30, the belt conveyor 40, etc. at the designated position, thereby preventing the article 10 or the pallet 20, the cart 30, the belt conveyor 40, etc. at the designated position from being damaged.

[0088] In addition, in particular, when the articles 10 are transferred to other pallets 20 or carts 30 and annotation information is independently assigned to each article 10 as described above, the processor 155 takes into account the size of each article 10 (the longitudinal, lateral, and height dimensions), and performs motion planning to move the multiple articles 10 in a manner that fills part or all of the space on the pallet 20 or cart 30 that is the destination of movement with the volume of the multiple articles 10 that are the objects of movement, and executes the movement action of the multiple articles 10 by the article moving device 100. Alternatively, in a case where multiple objects 10 that are to be moved are all of the same size (vertical, horizontal, and height dimensions), the processor 155 recognizes that the objects 10 at the moving source have the same assigned size based on the annotation information (especially, information related to position, posture, and size) assigned to at least one object 10 as described above, and performs motion planning to move the multiple objects 10 in a manner that fills part or all of the space on the pallet 20 or cart 30 that is the moving destination with the volume of the multiple objects 10, thereby executing the movement action of the multiple objects 10 by the object moving device 100.

[0089] As described above, according to the article moving device 100 of the present embodiment, by providing annotation information including attribute information of each object and information related to position, posture, and size to objects (articles 10 as moving objects, pallets 20 as moving destinations, carts 30, belt conveyors 40, etc.) around the article moving device 100 existing in the real world in a virtual world (simulated space) that reproduces the surrounding environment of the article moving device 100 in the real world, the article moving device 100 can recognize the attributes and the position, posture, and size related to these objects existing in the real world. Therefore, for example, in the case where the article moving device 100 is appropriately moved by the AGV 60 in the logistics warehouse and objects are arranged around the moved article moving device 100 to perform an article moving operation, the article moving device 100 can easily and quickly recognize the surrounding environment of the moved article moving device 100.

[0090] Furthermore, the article moving device 100 of the present embodiment is configured to determine whether an object associated with the movement action of the designated article 10 exists within the accessible range of the article moving device 100 after receiving a user input for designating an article 10 to be moved and a user input for designating a location or area as a moving destination, and not to perform the article moving action if any object does not exist within the accessible range. Therefore, it is possible to prevent the occurrence of an incident (damage to the article 10 or other objects due to the falling or collapse of the article 10, etc.) that may occur when the article moving action is performed when any object does not exist within the accessible range.

[0091] [Modifications]

[0092] In the above-mentioned embodiment, it is described that in Figure 7 In the process of step S16 shown in the figure, if the designated object does not exist in the accessible range of the article moving device 100 ("No"), the article moving action is not performed (step S17) and the process is terminated. In contrast, this variation provides a scheme in which the article moving action can be performed even when it is determined that the designated object does not exist in the accessible range of the article moving device 100.

[0093] In this modified example, when it is determined that the designated object does not exist within the accessible range of the article moving device 100, the object determined not to exist within the accessible range is moved to a position closer to the article moving device using the AGV 60. The movement of the object by the AGV 60 can be performed by the user operating the AGV 60 manually through remote operation, or by the processor 155 of the article moving device 100 controlling the movement of the AGV 60 in a state where the article moving device 100 and the AGV 60 communicate with each other via the input / output unit 176 of the article moving device 100.

[0094] (First Modification)

[0095] Fig.10 FIG. 1 is a diagram showing a first modified example of the article moving device in this embodiment. Fig.10 As shown in (a), the article moving device 100 of this modified example has a reference Figure 3 In addition to the structure of the article moving device 100 described above, an indication display unit 180 is provided in the holding unit 130, and the indication display unit 180 projects an indication display for indicating the position of the destination to be moved. The indication display unit 180 can be composed of an optical projector that projects the image of the indication display onto the floor surface on which the article moving device 100 is placed, a laser irradiation device that can radiate while scanning a laser to depict desired graphics and text, and the like. The indication display presented by the indication display unit 180 can also be a graphic of any shape, such as a cross, a circle, an arrow, or the like. Fig.10 In the illustrated example, the instruction display unit 180 is configured to project a cross-shaped instruction display.

[0096] According to the article moving device 100 of this modified example having such an indication display unit 180, the indication display is projected to a place where objects such as a pallet, a cart, and a belt conveyor should be arranged by the indication display unit 180. As an example, the movement of the object by the AGV 60 can be performed as follows: the user operates the AGV 60 by manual remote operation to move the object so that the position of the AGV 60 is aligned with the indication display projected on the floor surface, etc. Alternatively, it can also be performed as follows: in the case where the AGV 60 is equipped with a shooting camera and an image recognition device, the AGV 60 recognizes the indication display projected on the floor surface, etc., and the AGV 60 drives autonomously in a manner that its own position is consistent with the position of the indication display. Alternatively, it can also be that the operator visually observes the indication display projected by the indication display unit 180, and the operator himself moves the object in a manner that the position of the object is aligned with the indication display.

[0097] Fig.10(b) shows an example of the projection action of the indication display by the article moving device 100 of this modification. According to the article moving device 100 of this modification, first, the indication display mk1 is projected by the indication display unit 180 to a place where a certain object (e.g., a pallet) should be arranged, and the pallet is arranged at the position of the indication display using the AGV 60 or the like. Next, the robot arm 110 is caused to perform a telescopic action and / or a rotation action, and the next indication display mk2 is projected by the indication display unit 180 to a place where the next object (e.g., a cart) should be arranged, and the cart is arranged at the position of the indication display using the AGV 60 or the like. In this way, the article moving device 100 of this modification repeatedly performs the action of the indication display unit 180 displaying the indication display at the place where the object should be arranged. In addition, the user can use the input / output unit 176 of the control device 150 to specify the position where the indication display is to be projected by the indication display unit 180, for example, in a virtual world (simulated space) in which the control device 150 reproduces the surrounding environment of the article moving device 100 in the real world.

[0098] After the object is moved closer to the article moving device 100 by the AGV 60, the processor 155 executes the reference operation again from step S11 at the beginning. Figure 7 Thus, the surrounding environment information including the moved object (pallet 20, cart 30, belt conveyor 40, etc.) is reacquired (step S11), the moved object is reproduced in the virtual world (step S12), annotation information is given to the moved object (step S13), at least a determination is made as to whether the moved object exists in the access range (step S14), and at least a designated input is made for the moved object (step S15).

[0099] Then, next, if the result of the determination (step S16) as to whether the designated object exists within the accessible range of the article moving device 100 is that the designated object exists within the accessible range, the movement action of the article 10 is performed (step S17). If it is again determined that the designated object does not exist within the accessible range of the article moving device 100, the AGV 60 re-moves the object and the processing of the above steps S11 to S16 is repeated until it is determined that the designated object exists within the accessible range.

[0100] As described above, according to this modification, when it is determined that the designated object is not within the accessible range of the article moving device 100 , the moving operation of the AGV 60 can be repeated until the object is within the accessible range, and the article moving operation can be finally executed.

[0101] (Second Modification)

[0102] Next, refer to Fig.11The second variation of the article moving device in this embodiment is described below. Fig.11 As shown, in the second modification, instead of the indication display unit 180 provided on the holding unit 130 of the first modification, an indication display unit 200 is provided, and the indication display unit 200 is arranged on the base unit 120 that fixedly supports the arm unit 110 of the article moving device 100. In this example, the base unit 120 is fixed on the mounting table 50.

[0103] The indication display unit 200 comprises: a peripheral fixing portion 210, which is fixed to the base portion 120 in a manner surrounding the periphery of the columnar base portion 120; a peripheral rotating portion 215, which is supported by the peripheral fixing portion 210 in a manner surrounding the outer periphery of the peripheral fixing portion 210 and capable of rotating along the circumferential direction of the peripheral fixing portion 210; and a rotation driving source 220, which rotates the peripheral rotating portion 215.

[0104] A plurality of ball bearings (not shown) are enclosed between the outer circumferential surface of the peripheral fixing portion 210 and the inner circumferential surface of the peripheral rotating portion 215, and the peripheral rotating portion 215 is supported by the peripheral fixing portion 210 in a manner that it can rotate in the circumferential direction of the peripheral fixing portion 210 but cannot move in the axial direction (z direction in the figure) of the peripheral fixing portion 210. In addition, a gear is formed on the outer circumferential surface of the peripheral rotating portion 215.

[0105] The rotational drive source 220 includes a transmission device (gear) having a gear formed on the outer peripheral surface, and a motor that rotates the transmission device. The gear of the transmission device of the rotational drive source 220 meshes with the gear formed on the outer peripheral surface of the peripheral rotating part 215, and the peripheral rotating part 215 can be rotated by rotating the transmission device of the rotational drive source 220. The rotation direction of the peripheral rotating part 215 can be switched by switching the rotation direction of the motor of the rotational drive source 220.

[0106] The indication display unit 200 also has: a first driving part 230, which is arranged on the peripheral rotating part 215; a rod part 240, whose end on the root side is supported by the first driving part 230; a second driving part 250, which is arranged on the end on the front end side of the rod part 240; and an indication display part 260, which is supported by the second driving part 250.

[0107] The first drive unit 230 is composed of, for example, a servo motor, and can rotate the rod 240 supported by the first drive unit 230 with its root side as the rotation center. Fig.11As shown, the rod 240 is rotated in a direction in which the front end is lifted upward from the state in which the front end is facing downward. In addition, the second driving unit 250 is also composed of a servo motor, for example, and can rotate the indication display unit 260. The indication display unit 260 can also be composed of an optical projector that projects an indication display image onto the floor surface on which the article moving device 100 is placed, a laser irradiation device that can radiate while scanning a laser to draw a desired graphic or text, etc.

[0108] As described above, the indication display unit 200 of this modification can change the position and orientation of the indication display unit 260 relative to the base unit 120 by three degrees of freedom, namely, the rotation of the peripheral rotating unit 215 (first degree of freedom), the rotation by the first driving unit 230 (second degree of freedom), and the rotation by the second driving unit 250 (third degree of freedom). Therefore, according to the indication display unit 200 of this modification, by driving each driving unit to appropriately change the position and orientation of the indication display unit 260, it is possible to project the indication display from the indication display unit 260 to the place around the article moving device 100 where objects such as a pallet, a cart, and a belt conveyor should be arranged. The indication display unit 200 of this modification is independent of the arm 110 of the article moving device 100, and therefore, unlike the structure in which the indication display unit 180 is provided on the holding unit 130 at the front end of the arm 110 as in the first modification, there is an advantage that the indication display unit 260 can project the indication display independently of the movement of the arm 110.

[0109] The present invention has been described above by way of the embodiments of the invention, but the above embodiments do not limit the invention to which the claims relate. In addition, a method obtained by combining the features described in the embodiments of the present invention may also be included in the technical scope of the present invention. Furthermore, it is also obvious to those skilled in the art that various changes or improvements may be made to the above embodiments.

Claims

1. An article moving device for moving an article, the article moving device comprising: an arm having a holding portion for holding the article; an acquisition unit, which acquires the surrounding environment information of the object moving device; a control unit that controls the actions of the holding unit, the arm unit, and the acquiring unit, in, The control unit is configured to execute the following processing: causing the acquisition unit to acquire the surrounding environment information of the object moving device in the real world; generating a virtual world including objects in the real world that exist around the item moving device based on the surrounding environment information; giving the object in the virtual world annotation information including attribute information of the object and information related to position, posture and size; receiving a first input designating an item to be moved with respect to the object existing in the virtual world and a second input designating a position or area of ​​a movement destination of the item; as well as Based on the first input and the second input, the holding unit and the arm are caused to execute an action of moving the designated object to the designated position or area in the real world.

2. The article moving device according to claim 1, wherein: Assigning the annotation information to the object includes the following processing: Accepting selection of attribute information of the object as a target for determining position, posture, and size in the virtual world; Displaying a model for determining the position, posture and size of the object in the virtual world, and acquiring information related to the position, posture and size of the model that matches and overlaps with the outline of the object; as well as The received attribute information and the acquired information on the position, posture, and size of the model are added to the object as the annotation information.

3. The article moving device according to claim 2, wherein: Acquiring information on the position, posture, and size of the model superimposed to match the outline of the object includes the following process: accepting input for changing the position, posture, and size of the model to match the outline of the object in the virtual world.

4. The article moving device according to claim 2, wherein: The selection of the attribute information of the object includes the following processing: Displaying a screen for displaying options related to the attribute information of the object in the virtual world; as well as An input of attribute information for selecting the object is received in the virtual world.

5. The article moving device according to claim 1, wherein: The control unit is configured to further execute the following processing: determining whether the object specified in the first input and the second input exists in an accessible range of the article moving device, The holding unit and the arm unit execute an operation of moving the designated object to the designated position or area when the designated object exists within the accessible range.

6. A method for controlling an article moving device, the method being a method for controlling an article moving device for moving an article, The article moving device comprises: an arm having a holding portion for holding the article; an acquisition unit, which acquires the surrounding environment information of the object moving device; a control unit that controls the actions of the holding unit, the arm unit, and the acquiring unit, The control method comprises the following steps performed by the control unit: causing the acquisition unit to acquire the surrounding environment information of the object moving device in the real world; generating a virtual world including objects in the real world that exist around the item moving device based on the surrounding environment information; giving the object in the virtual world annotation information including attribute information of the object and information related to position, posture and size; accepting a first input designating an item to be moved with respect to an object existing in the virtual world and a second input designating a position or area of ​​a movement destination of the item; as well as Based on the first input and the second input, the holding unit and the arm are caused to execute an action of moving the designated object to the designated position or area in the real world.

7. A computer program executable by a processor, the computer program comprising instructions for implementing the method according to claim 6.

Images