Handling robot system, information processing device, processing method and program

Abstract

To obtain a handling robot system, information processing device, processing method, and program that can grasp the relative positional relationship between a container and a manipulator. [Solution] The information processing device of the handling robot system of the embodiment comprises a position acquisition unit and a determination unit. The position acquisition unit acquires the position of the container based on the captured image. The determination unit determines, based on the position of the container, whether the manipulator is positioned in a position where it can perform the item retrieval operation, in relation to the relative positional relationship between the manipulator and the container.

Description

Technical Field

[0001] Embodiments of the present invention relate to a handling robot system, an information processing device, a processing method, and a program.

Background Art

[0002] In workplaces such as manufacturing factories and logistics warehouses, there is an operation of sorting items such as parts and products from storage boxes to the requester. For example, the operation of identifying and sorting a huge variety of items from storage boxes (which are containers) is performed manually. Due to the shortage of workers, automation of the sorting operation is demanded. However, it is difficult to have the manipulator (robot) take on all of it, and a work mode in which a person or a manipulator alternately undertakes the sorting operation according to the working hours and work load has been considered. The sorting operation is performed in a dedicated place where storage boxes are supplied. The manipulator is mounted on a trolley so that it can move, and the operator transports and installs the trolley at the sorting work place, enabling the manipulator to perform the sorting operation. Even if the manipulator is transported each time, it is important that the manipulator is installed at a predetermined installation position in order to perform the sorting operation correctly.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Summary of the Invention

Problems to be Solved by the Invention

[0004] In this type of technology, understanding the relative positional relationship between the container and the manipulator is necessary for proper sorting.

[0005] Therefore, one of the objectives of the embodiments of the present invention is to obtain a handling robot system, an information processing device, a processing method, and a program that can grasp the relative positional relationship between a container and a manipulator. [Means for solving the problem]

[0006] The handling robot system of the embodiment comprises a trolley, a manipulator, an imaging device, and an information processing device. The manipulator is mounted on the trolley and is capable of removing an item from a container that has been sent to a predetermined position. The imaging device is mounted on the trolley and is capable of imaging the container at the predetermined position. The information processing device comprises a position acquisition unit and a determination unit. The position acquisition unit acquires the position of the container based on the image captured by the imaging device. The determination unit determines, based on the position of the container acquired by the position acquisition unit, whether the manipulator is positioned in a position where the item removal operation can be performed in relation to the manipulator and the container. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a diagram showing an example configuration of a handling robot system according to the first embodiment, and depicts the handling robot system in a state where it is deployed at the work site. [Figure 2] Figure 2 is a block diagram showing a part of the handling robot system of the first embodiment. [Figure 3] Figure 3 is a block diagram showing an example of the functional configuration of the information processing device according to the first embodiment. [Figure 4] Figure 4 is a diagram illustrating an example of the processing performed by the position acquisition unit of the first embodiment. [Figure 5]Figure 5 is a diagram illustrating another example of the processing performed by the position acquisition unit of the first embodiment. [Figure 6] Figure 6 is a diagram illustrating an example of a screen displayed on the display device of the first embodiment. [Figure 7] Figure 7 is a diagram illustrating an example of a screen displayed on the display device of the first embodiment. [Figure 8] Figure 8 is a diagram illustrating an example of the process performed by the setting unit of the first embodiment. [Figure 9] Figure 9 is a diagram illustrating another example of the processing performed by the setting unit of the first embodiment. [Figure 10] Figure 10 is a flowchart showing an example of a process performed by the information processing device of the first embodiment. [Figure 11] Figure 11 is a diagram showing an example configuration of a handling robot system according to a second embodiment, and depicts the handling robot system in a state where it is deployed at a work site. [Figure 12] Figure 12 is a diagram illustrating an example of processing performed by the information processing device of the second embodiment. [Figure 13] Figure 13 is a flowchart showing an example of a process performed by the information processing device of the second embodiment. [Figure 14] Figure 14 is a plan view showing a container according to the third embodiment. [Figure 15] Figure 15 is a diagram illustrating an example of the processing performed by the position acquisition unit of the third embodiment. [Figure 16] Figure 16 is a diagram illustrating another example of the processing performed by the position acquisition unit of the third embodiment. [Figure 17] Figure 17 is a diagram illustrating an example of the processing performed by the setting unit of the fourth embodiment. [Figure 18] Figure 18 is a diagram illustrating another example of the processing performed by the setting unit of the fourth embodiment. [Figure 19] Figure 19 is a diagram illustrating an example of processing performed by the processing execution unit of the fifth embodiment. [Figure 20] Figure 20 is a diagram showing a configuration example of a handling robot system according to the sixth embodiment, and is a diagram of the state where the handling robot system is arranged at the work site. [Figure 21] Figure 21 is a diagram showing another configuration example of a handling robot system according to the sixth embodiment, and is a diagram of the state where the handling robot system is arranged at the work site.

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments will be described with reference to the drawings. In this specification, the components according to the embodiments and the description of the components may be described in a plurality of expressions. The components and the description thereof are examples and are not limited by the expressions in this specification. The components may be specified by different names from those in this specification. Also, the components may be described by expressions different from those in this specification.

[0009] <First Embodiment> Figure 1 is a diagram showing a configuration example of a handling robot system according to the first embodiment, and is a diagram of the state where the handling robot system is arranged at the work site.

[0010] As shown in FIG. 1, the handling robot system 1 can perform an article taking-out operation of taking out an article 3 in a container 2 sent to a predetermined position F outside the container 2 at a work site such as a manufacturing factory or a logistics warehouse. The predetermined position is, for example, a position on a workbench surface. The container 2 is conveyed to a predetermined position by a conveying device such as a belt conveyor, for example. The container 2 is, for example, a box. Specifically, the container 2 has a quadrangular shape in plan view and is open upward. Note that the container 2 may be a polygon other than a quadrangle in plan view (that is, a triangular prism shape in three dimensions, etc.), or may be circular in plan view (for example, a cylindrical shape in three dimensions, etc.). The article 3 is, for example, a part or a commodity.

[0011] Figure 2 is a block diagram showing a part of the handling robot system of the first embodiment. As shown in Figures 1 and 2, the handling robot system 1 comprises a trolley 11 (Figure 1), a manipulator 12, an imaging device 13, an illumination device 14, a display device 15, a speaker 16 (Figure 2), an input device 17 (Figure 2), and an information processing device 20. The manipulator 12, imaging device 13, illumination device 14, display device 15, speaker 16 (Figure 2), input device 17 (Figure 2), and information processing device 20 are mounted on the trolley 11. The trolley 11, manipulator 12, imaging device 13, illumination device 14, input device 17, and information processing device 20 constitute the trolley device 10. Note that the information processing device 20 does not necessarily have to be mounted on the trolley 11. In this case, the information processing device 20 should be in a state of communication connection with each component of the handling robot system 1, such as the manipulator 12, imaging device 13, and lighting device 14, via wired communication or wireless communication.

[0012] In addition to the above, the handling robot system 1 is equipped with various sensors, a power supply unit for operating various drive units, a cylinder for storing compressed air, a compressor, a vacuum pump, a controller and external interfaces such as a UI (User Interface), and safety mechanisms such as a light curtain and a collision detector.

[0013] As shown in Figure 1, the trolley 11 is moved by the power of worker H.

[0014] The manipulator 12 is capable of removing items 3 from a container 2 that has been sent to a predetermined position F. The manipulator 12 has an arm portion 12a and a picking tool portion 12b.

[0015] The arm section 12a is a multi-joint robot driven by multiple servo motors. The arm section 12a is composed of a combination of, for example, a 6-axis vertical articulated robot, a multi-axis vertical articulated robot, a SCARA robot, and a linear motion robot.

[0016] The picking tool section 12b is capable of gripping the article 3 by suction, jamming, pinching, and multi-finger mechanisms. The picking tool section 12b is also referred to as the handling tool section. When gripping by suction, the picking tool section 12b can do so by electromagnetic suction.

[0017] The imaging device 13 is configured as an area camera (area sensor) capable of outputting a color or black and white two-dimensional image (image) by imaging a subject. The imaging device 13 can image the container 2 at a predetermined position F. The imaging device 13 transmits the image (image data) obtained by imaging to the information processing device 20.

[0018] The illumination device 14 illuminates the imaging area of ​​the imaging device 13.

[0019] The display device 15 is, for example, a liquid crystal or an organic EL display. The display device 15 displays various types of information.

[0020] Speaker 16 is capable of outputting sound.

[0021] The input device 17 shown in Figure 2 includes a keyboard, mouse, etc. The input device 17 may also be a touch panel (including non-contact types) superimposed on the display device 15.

[0022] As shown in Figure 2, the information processing device 20 comprises a control device 21, a communication device 22, and a storage device 23.

[0023] The control device 21 includes a processor 25, a storage unit 26, and a robot controller 27. The processor 25 is, for example, a CPU or GPU. The storage unit 26 includes ROM, RAM, etc. The processor 25 performs various controls and calculations on the information processing device 20 by, for example, executing a program stored in the ROM of the storage unit 26. The robot controller 27 controls the manipulator 12 in response to commands from the processor 25. That is, the processor 25 controls the manipulator 12 via the robot controller 27.

[0024] The communication device 22 is an element that realizes communication functions for communicating with external devices. In the information processing device 20 of this embodiment, the communication device 22 establishes communication with a management device (not shown), etc.

[0025] The storage device 23 is composed of, for example, an HDD (Hard Disk Drive). The storage device 23 stores various kinds of information and data.

[0026] Figure 3 is a block diagram showing an example of the functional configuration of an information processing apparatus according to the first embodiment. As shown in Figure 3, the control device 21 has as functional units a position acquisition unit 21a, a determination unit 21b, a processing execution unit 21c, a setting unit 21d, a planning unit 21e, and a control unit 21f.

[0027] The program executed by the information processing device 20 in this embodiment is provided pre-installed in the ROM of the storage unit 26.

[0028] The program executed by the information processing device 20 of this embodiment may be configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk), or USB memory.

[0029] Furthermore, the program executed by the information processing device 20 of this embodiment may be configured to be stored on a computer, such as an information processing device, connected to a network such as the Internet, and provided by being downloaded via the network. Alternatively, the program executed by the information processing device 20 of this embodiment may be provided or distributed via a network such as the Internet.

[0030] The program executed by the information processing device 20 of this embodiment has a modular configuration that includes the above-mentioned parts (position acquisition unit 21a, determination unit 21b, processing execution unit 21c, setting unit 21d, planning unit 21e, and control unit 21f). In actual hardware, the processor 25 reads the program from the ROM and executes it, loading the above-mentioned parts onto the main memory such as the RAM of the storage unit 26, and generating the position acquisition unit 21a, determination unit 21b, processing execution unit 21c, setting unit 21d, planning unit 21e, and control unit 21f on the main memory such as the RAM.

[0031] The position acquisition unit 21a acquires the position of the container 2 based on the captured image obtained by the imaging device 13. For example, the position acquisition unit 21a acquires the position of the container 2's frame 2a (Figure 1) as the position of the container 2. The position acquisition unit 21a acquires the position of the container 2 by performing image processing on the captured image. Here, the position recognized by the information processing device 20, including the position acquisition unit 21a, is the coordinate position (X, Y) in a two-dimensional coordinate system set for the manipulator 12. This coordinate position in the two-dimensional coordinate system is position information obtained from an captured image taken from above and viewed in plan. The reason for acquiring position information in plan view (hereinafter also referred to as the position in the planar direction) in this way is that there is variation in the position in the planar direction when the container 2 is transported and positioned. Note that the container 2 is transported according to the design layout information and flows along on a transport conveyor, etc., so the height position of the container 2 as seen from the floor surface when it is transported is at a predetermined height (almost exactly as designed in the layout). Therefore, in this embodiment, the planar position of the container 2 is obtained in order to adjust for any misalignment in the planar direction.

[0032] Figure 4 is a diagram illustrating an example of the processing performed by the position acquisition unit of the first embodiment. Figure 5 is a diagram illustrating another example of the processing performed by the position acquisition unit of the first embodiment. The position acquisition unit 21a performs, for example, the processing shown in Figures 4 and 5.

[0033] As shown in Figure 4, the position acquisition unit 21a acquires an image G from the imaging device 13 (Figure 4(a)). The image G includes a container image 2G, which is an image of the container 2. Next, the position acquisition unit 21a extracts (recognizes) a frame image 2aG, which is an image of the frame 2a of the container 2 in the image G, and acquires (calculates) the position of the frame image 2aG (Figure 4(b)). Next, the position acquisition unit 21a acquires (calculates) the position of the center C1 of the container 2 from the position of the frame image 2aG, and determines whether the position of the center C1 of the container 2 is located within a predetermined range R1. Figure 4(c) shows the case where the position of the center C1 of the container 2 is located within the predetermined range R1, and Figure 4(d) shows the case where the position of the center C1 of the container 2 is located outside the predetermined range R1. As an example, the position acquisition unit 21a determines the center C1 of the container 2 by using the container image 2G to find a matching position using a template matching method or the like, based on the center position (X, Y) of the container 2 on the captured image G taken from above. Since the height of the container 2 is assumed to be the height specified in the design, it can be considered known, and therefore it is unnecessary to determine the center of gravity (X, Y, Z) of the container 2.

[0034] As shown in Figure 5, the position acquisition unit 21a acquires an image G from the imaging device 13 (Figure 5(a)). The image G includes a container image 2G, which is an image of the container 2. Next, the position acquisition unit 21a extracts (recognizes) a frame image 2aG, which is an image of the frame 2a of the container 2 in the image G, and acquires (calculates) the position of the frame image 2aG (Figure 5(b)). Next, the position acquisition unit 21a acquires (calculates) the positions of the four corner images 2a1G, which are images of the four corners of the container 2, from the position of the frame image 2aG, and determines whether the position of the four corner images 2a1G is located within a predetermined range R1. Figure 5(c) shows the case where the position of the four corner images 2a1G is located within the predetermined range R1, and Figure 5(d) shows the case where the position of the four corner images 2a1G is located outside the predetermined range R1. Here, since the position acquisition unit 21a acquires the position of the four corners image 2a1G, that is, the positions of the four corners of the container 2, even if there is an item 3 inside the container 2, the four corners of the container 2 are not hidden by the item 3, which has the advantage of being able to recognize the positions of the four corners of the container 2. Note that the part of the container image 2G from which the position acquisition unit 21a acquires the position may be used differently depending on the situation and state of the container 2, such as the presence or absence of the item 3 inside the container 2.

[0035] The determination unit 21b shown in Figure 3 determines, based on the position of the container 2 acquired by the position acquisition unit 21a, whether the relative positional relationship between the manipulator 12 and the container 2 is a predetermined relative positional relationship that allows the manipulator 12 to perform the item retrieval operation, that is, whether the manipulator 12 is positioned at a predetermined working position that enables the operation. For example, the determination unit 21b determines whether a predetermined part of the container 2 (e.g., the center or four corners of the container 2) is within a preset range. Specifically, the determination unit 21b determines whether a predetermined part of the container image 2G (e.g., the center C1 or the four corner images 2a1G) is within a preset range R1. If the determination unit 21b determines that a predetermined part of the container image 2G (e.g., the center C1 or the four corner images 2a1G) is within the preset range R1, it determines that the manipulator 12 is positioned at the predetermined working position. If the determination unit 21b determines that a predetermined part of the container image 2G (for example, the center C1 or the corner images 2a1G) is not within a preset range R1, it determines that the manipulator 12 is not located at the specified work position.

[0036] If the determination unit 21b determines that the manipulator 12 is not in the specified working position, the processing execution unit 21c will have the worker H perform a positioning process between the container 2 and the manipulator 12. If the determination unit 21b determines that the manipulator 12 is in the specified working position, the processing execution unit 21c will not perform the positioning process.

[0037] Here, the alignment process is performed so that the object, such as container 2, and the manipulator 12 are in a proper relative positional relationship. Specifically, the alignment process calculates the direction and amount of movement of the trolley 11 required for the manipulator 12 to reach a specified work position relative to the object, such as container 2, which has been sent to a predetermined position F (i.e., the direction and amount of movement, respectively, are the movement directions and amounts that eliminate so-called positional misalignment), and communicates the calculated direction and amount of movement of the trolley 11 to the worker H. The specified work position is, for example, a position in the relative positional relationship between the manipulator 12 and container 2 where the manipulator 12 can perform the item retrieval operation.

[0038] The processing execution unit 21c calculates the direction of movement (direction of movement) and the amount of movement (amount of movement) of the trolley 11 necessary for the manipulator 12 to reach the specified working position during the alignment process. The specified working position is the relative position of the manipulator 12 to the container 2 when a predetermined part of the container image 2G (for example, the center C1 or the four corner images 2a1G) is located within a preset range R1. The alignment process can be performed in the manner described above using the screen displayed on the display device 15 shown in Figure 6, which will be described later.

[0039] Figure 6 is a diagram illustrating an example of a screen displayed on the display device of the first embodiment. Figure 6 shows a state where the positional misalignment between the container 2 and the trolley 11 shown in Figure 1 is large. Figure 7 is a diagram illustrating an example of a screen displayed on the display device of the first embodiment. Figure 7 shows a state where the positional misalignment is smaller than in the case of Figure 6. In each of Figures 6 and 7, the center C1 of the container 2 is displayed within the container image 2G of the container 2. As shown in Figures 6 and 7, for example, the processing execution unit 21c shown in Figure 3 displays an arrow K on the display device 15 that represents the direction and amount of movement of the trolley 11 for the manipulator 12 to reach a specified work position. The direction of the arrow K indicates the direction of movement of the trolley 11 for the manipulator 12 to reach a specified work position. The size of the arrow K indicates the amount of movement of the trolley 11 for the manipulator 12 to reach a specified work position. The size of the arrow K is set to correspond to the amount of movement of the trolley 11 for the manipulator 12 to reach a specified work position. The larger the arrow K, the greater the amount of movement the trolley 11 needs to make for the manipulator 12 to reach the designated work position. Note that for arrow K, which is proportional to the amount of movement (deviation) of the trolley 11 needed for the manipulator 12 to reach the designated work position, a threshold may be set for the maximum size of arrow K displayed in proportion to the amount of movement (deviation). If the size of arrow K relative to the amount of movement (deviation) is greater than or equal to this threshold, the size of arrow K may be kept at its maximum size. The direction and amount of movement of the trolley 11 needed for the manipulator 12 to reach the designated work position are also referred to as the direction in which the container 2 is installed and the deviation from the position in which the container 2 is installed, respectively. By manually moving the trolley 11 according to the size and direction of arrow K displayed on the display device 15, the operator H can move the manipulator 12 to the designated work position and position a predetermined part of the container image 2G (for example, the center C1 or the four corner images 2a1G) within a preset range R1.

[0040] As another example, in the alignment process, the processing execution unit 21c outputs the direction and amount of movement of the trolley 11 necessary for the manipulator 12 to reach a specified work position as audio to the speaker 16.

[0041] The setting unit 21d shown in Figure 3 sets the predetermined work position. For example, the predetermined work position is set in advance by the setting unit 21d before the item retrieval operation. The setting unit 21d sets the predetermined work position using a three-dimensional model used for the operation of the manipulator 12, which includes, for example, a predetermined position F, the container 2 sent to the predetermined position F, and position information of each part of the trolley device 10. The three-dimensional model recognizes in virtual space the ideal design arrangement in which each piece of equipment should be placed in the ideal design arrangement. The operator performs a "positioning operation" to align the actual equipment using the methods of the first to fourth embodiments, including this embodiment, so that it is in this arrangement.

[0042] Figure 8 is a diagram illustrating an example of the processing performed by the setting unit of the first embodiment. Figure 9 is a diagram illustrating another example of the processing performed by the setting unit of the first embodiment. The setting unit 21d performs, for example, the processing shown in Figures 8 and 9.

[0043] As shown in Figure 8, the position acquisition unit 21a shown in Figure 3 generates a simulation image GA from the position of the center C1 of the container 2 in the 3D model, the position of the imaging device 13 in the 3D model, and the field of view of the imaging device 13 (Figure 8(a)). The field of view of the imaging device 13 indicates the range of real space being imaged by the imaging device 13. In this case, it is the imaging range (2D) determined by the relative distance between the imaging device 13 and the container 2 in the 3D model. Next, the position acquisition unit 21a determines a predetermined range R1 based on the container image 2G in the simulation image GA and determines a specified working position (Figure 8(b)).

[0044] As shown in Figure 9, the position acquisition unit 21a shown in Figure 3 generates a simulation image GA from the positions of the four corners of the container of the 3D model (each corner image 2a1G), the position of the imaging device 13 of the 3D model, and the field of view of the imaging device 13 (Figure 9(a)). Next, the position acquisition unit 21a determines a predetermined range R1 based on the container image 2G in the simulation image GA and determines a specified working position (Figure 9(b)).

[0045] The planning unit 21e generates control information, including the trajectory of the manipulator 12 as it grasps the article 3 and moves it out of the container 2, using a known method, and transmits the control information to the control unit 21f.

[0046] The control unit 21f controls the manipulator 12 based on the control information. Specifically, until the gripping operation is successful, the control unit 21f generates angle command values ​​for each joint of the manipulator 12 in a time series based on the control information received sequentially from the planning unit 32, and controls the manipulator 12 to grip the item 3. Alternatively, the position and orientation of the end effector of the manipulator 12 may be controlled to grip the item 3.

[0047] Next, an example of processing performed by the information processing device 20 will be described with reference to Figure 10. Figure 10 is a flowchart showing an example of processing performed by the information processing device of the first embodiment.

[0048] As shown in Figure 10, the position acquisition unit 21a acquires an image G from the imaging device 13 (S101) and recognizes the position of the container 2 (S102).

[0049] Next, the determination unit 21b determines, based on the position of the container 2 acquired by the position acquisition unit 21a, whether the manipulator 12 is positioned at a specified working position where the relative positional relationship between the manipulator 12 and the container 2 is a specified relative positional relationship that allows the manipulator 12 to perform the item retrieval operation (S103).

[0050] Next, if the determination unit 21b determines that the manipulator 12 is not in the specified work position, the processing execution unit 21c calculates the direction and amount of movement (i.e., the amount of displacement) of the trolley 11 necessary for the manipulator 12 to reach the specified work position (S104). Next, the processing execution unit 21c notifies the calculated direction and amount of movement (i.e., the amount of displacement) of the trolley 11 by outputting it through the display device 15 and speaker 16 (S105). For example, by manually moving the trolley 11 according to the size and direction of the arrow K displayed on the display device 15, the operator can move the manipulator 12 to the specified work position and position a predetermined part of the container image 2G (for example, the center C1 or the four corner images 2a1G) within a preset range R1.

[0051] If the item retrieval operation is not completed (S106: No), the information processing device 20 repeats the process from S101 onwards, and if the item retrieval operation is completed, it terminates the above process (S106: Yes).

[0052] As described above, the handling robot system 1 of this embodiment comprises a trolley 11, a manipulator 12, an imaging device 13, and an information processing device 20. The manipulator 12 is mounted on the trolley 11 and is capable of removing items 3 from a container 2 that has been sent to a predetermined position F. The imaging device 13 is mounted on the trolley 11 and is capable of imaging the container 2 at the predetermined position F. The information processing device 20 comprises a position acquisition unit 21a, a determination unit 21b, and a processing execution unit 21c. The position acquisition unit 21a acquires the position of the container 2 based on the image G obtained by the imaging device 13. The determination unit 21b determines, based on the position of the container 2 acquired by the position acquisition unit 21a, whether the manipulator 12 is positioned in a position where the item removal operation can be performed (a predetermined working position) in relation to the manipulator 12 and the container 2. If the determination unit 21b determines that the manipulator 12 is not in the specified working position, the processing execution unit 21c performs alignment processing to align the container 2 and the manipulator 12.

[0053] With this configuration, the determination unit 21b determines, based on the position of the container 2 acquired by the position acquisition unit 21a, whether the manipulator 12 is positioned in a position where the item can be retrieved, in relation to the relative position of the manipulator 12. This makes it relatively easy to grasp the relative position of the container 2 and the manipulator 12. Furthermore, although the position of the container 2 may change, this configuration allows for quick positioning even when the position of the container 2 changes. In addition, this configuration makes it easier to perform calibration (i.e., position correction or alignment) between the equipment and the manipulator 12, etc., when changing the layout of equipment and facilities at the work site. Therefore, it becomes easier to change the layout as required.

[0054] In conventional technology, it is difficult to install the trolleys in the exact same position. Also, the boxes are transported from the storage shelves to the sorting area via a conveyor. The layout of each part is designed on the premise that workers will sort the items 3 from the boxes, and there is no equipment to accurately position the boxes, so the supply positions of the boxes vary. Therefore, it is necessary to understand the precise positional relationships. If a method for mechanically positioning the trolleys is used, it is necessary to permanently install dedicated positioning equipment in the work area where the trolleys will be installed. However, permanently installing dedicated positioning equipment separate from the trolley system in a place where workers and manipulators work in shifts would interfere with the workers' work. In contrast, in this embodiment, the trolley system 10 can be positioned without installing dedicated positioning equipment separate from the trolley system 10, thus minimizing interference with the workers' work.

[0055] Furthermore, the position acquisition unit 21a acquires the position of the frame 2a of the container 2 as the position of the container 2.

[0056] With this configuration, the position of container 2 can be obtained relatively easily.

[0057] The designated working position is predetermined.

[0058] With this configuration, the worker does not need to set a predetermined work position during the item retrieval process.

[0059] Furthermore, the handling robot system 1 is equipped with a display device 15. In the positioning process, the processing execution unit 21c causes the display device 15 to display an arrow K representing the direction and amount of movement of the trolley 11 necessary for the manipulator 12 to reach a predetermined work position.

[0060] With this configuration, the operator can easily understand the direction and amount of movement of the trolley 11 required for the manipulator 12 to reach the specified working position.

[0061] Furthermore, the handling robot system 1 is equipped with a speaker 16. During the positioning process, the processing execution unit 21c outputs the direction and amount of movement of the trolley 11 necessary for the manipulator 12 to reach a predetermined work position to the speaker 16 as audio.

[0062] With this configuration, the operator can easily understand the direction and amount of movement of the trolley 11 required for the manipulator 12 to reach the specified working position.

[0063] Furthermore, the processing method of this embodiment is a processing method performed by an information processing device 20 provided in the handling robot system 1, and includes the steps of: a position acquisition unit 21a acquiring the position of the container 2 based on an image captured by the imaging device 13; a determination unit 21b determining whether the manipulator 12 is positioned in a position where it can perform the item retrieval work, based on the position of the container 2 acquired by the position acquisition unit 21a, in relation to the relative position of the manipulator 12 and the container 2; and a processing execution unit 21c performing a positioning process for aligning the container 2 and the manipulator 12 when the determination unit 21b determines that the manipulator 12 is not positioned in a predetermined work position.

[0064] With this configuration, the determination unit 21b determines, based on the position of the container 2 acquired by the position acquisition unit 21a, whether the manipulator 12 is positioned in a position where it can perform the item removal operation in relation to the container 2. Therefore, the relative positional relationship between the container 2 and the manipulator 12 can be grasped relatively easily.

[0065] Furthermore, the program of this embodiment causes the computer of the information processing device 20 to function as: a position acquisition unit 21a that acquires the position of the container 2 based on the captured image G obtained by imaging by the imaging device 13; a determination unit 21b that determines whether the manipulator 12 is in a specified position where the item retrieval work can be performed, in relation to the relative position of the manipulator 12 and the container 2, based on the position of the container 2 acquired by the position acquisition unit 21a; and a processing execution unit 21c that performs a position alignment process for aligning the container 2 and the manipulator 12 if the determination unit 21b determines that the manipulator 12 is not in the specified work position.

[0066] With this configuration, the determination unit 21b determines, based on the position of the container 2 acquired by the position acquisition unit 21a, whether the manipulator 12 and the container 2 are in a predetermined relative positional relationship that allows the manipulator 12 to perform the item removal operation. In this way, the relative positional relationship between the container 2 and the manipulator 12 can be grasped relatively easily.

[0067] Next, a modified example will be described. In this modified example, the position acquisition unit 21a acquires the position of the container 2 using a trained model that has been trained to output the position of the container 2 from the captured image G.

[0068] In subsequent embodiments, the requirements of the first embodiment will generally be used. Reference numerals and other numerals will remain the same. Requirements that differ from those of the first embodiment will be described.

[0069] <Second Embodiment> Figure 11 is a diagram showing an example configuration of a handling robot system according to the second embodiment, illustrating the handling robot system in a state where it is deployed at a work site. Figure 12 is a diagram illustrating an example of processing performed by the information processing device of the second embodiment. Figure 13 is a flowchart illustrating an example of processing performed by the information processing device of the second embodiment.

[0070] As shown in Figure 11, this embodiment differs from the first embodiment in that the handling robot system 1 is equipped with a floodlight 18.

[0071] The floodlight 18 is mounted on the trolley 11 and constitutes the trolley device 10. The floodlight 18 is capable of projecting a beam of light in a predetermined shape onto the container 2 at a predetermined position F from above. The predetermined shape is, for example, a cross shape, but is not limited to this.

[0072] The position acquisition unit 21a acquires the position of the container 2 based on the image of light rays in the captured image G obtained by the imaging device 13. For example, if a cross-shaped optical axis is projected from above onto the workbench surface including the empty container 2 as the image of light rays in the captured image G, the endpoints of the cross-shaped optical axis will shift depending on the position of the empty container 2 and the imaging device 13 that is taking the image. By comparing the position of the endpoint on the captured image G taken at the taught position with a predetermined range R1 (reference range frame), the positional displacement between the container 2 and the manipulator 12 on the X,Y coordinate plane can be calculated.

[0073] Figure 12 shows a ray image 100R, which is an image of the light rays projected onto the container 2 from above by the floodlight 18. The position acquisition unit 21a extracts (recognizes) an end image 100aR, which is an image of the end of the light ray shift, from the captured image G, which includes the light ray shift caused by the height difference of the subject (the upper surface of the wall of the container 2 or the bottom surface of the container 2). Figure 12 shows a ray image 100R acquired by the imaging device 13 when it captures a state in which the light rays projected onto the container 2 from above by the floodlight 18 illuminate the area including the container 2. The position acquisition unit 21a extracts (recognizes) the end image 100aR of the light ray shift caused by the height difference of the projected light rays, which is captured in the captured image G. That is, the position acquisition unit 21a extracts (recognizes) the end image 100aR. The position of the end image 100aR is an example of the position of the container 2.

[0074] The determination unit 21b determines whether the position of the end image 100aR, which is the position of the container 2 acquired by the position acquisition unit 21a, is within a preset range R1. If the determination unit 21b determines that the position of the end image 100aR is within the preset range R1, it determines that the manipulator 12 is located at the specified work position. If the determination unit 21b determines that the position of the end image 100aR is not within the preset range R1, it determines that the manipulator 12 is not located at the specified work position.

[0075] Next, an example of the processing performed by the information processing device 20 will be explained with reference to Figure 13. The position acquisition unit 21a acquires the captured image G (S201). The position acquisition unit 21a converts the captured image G into a grayscale image (S202). The position acquisition unit 21a binarizes the grayscale captured image G (S203). The position acquisition unit 21a inverts the black and white of the binarized captured image G (S204). The position acquisition unit 21a performs a Hough transform on the black and white inverted captured image G to detect vertical lines (S205). The position acquisition unit 21a calculates the slope of the detected vertical lines (S206). The position acquisition unit 21a searches for (extracts) the longest vertical line (S207). The position acquisition unit 21a performs a Hough transform on the black and white inverted captured image G to detect horizontal lines (S208). The position acquisition unit 21a calculates the slope of the detected horizontal line (S209). The position acquisition unit 21a searches for (extracts) the longest horizontal line (S210).

[0076] If rotational correction is required for the detected straight line (S211:Yes), the position acquisition unit 21a rotates the straight line to eliminate the inclination (S212) and proceeds to S213. If rotational correction is not required for the detected straight line (S211:No), the position acquisition unit 21a proceeds to S213 without rotating the straight line.

[0077] The position acquisition unit 21a extracts the position of the vertical endpoint of the vertical line (S213) and the position of the vertical endpoint of the horizontal line (S214). The processing execution unit 21c reads the endpoint positions (reference positions) of the vertical and horizontal lines at the time of teaching (S215), compares the read endpoint positions with the vertical and horizontal endpoint positions extracted in S213 and S214 (S216), and calculates the direction of movement and amount of movement (i.e., the amount of displacement) of the trolley 11 (S217). The processing execution unit 21c notifies the calculated direction of movement and amount of movement (i.e., the amount of displacement) of the trolley 11 by outputting through the display device 15 and speaker 16 (S218). For example, by manually moving the trolley 11 according to the size and direction of the arrow K displayed on the display device 15, the operator can move the manipulator 12 to a predetermined working position and position a predetermined part of the container image 2G (for example, the center C1 or the four corner images 2a1G) within a preset range R1.

[0078] If the item retrieval operation is not completed (S219: No), the information processing device 20 repeats the process from S201 onwards, and if the item retrieval operation is completed, it terminates the above process (S219: Yes).

[0079] As described above, the handling robot system 1 is mounted on a trolley 11 and includes a light projector 18 that projects a light ray of a predetermined shape onto a container 2 at a predetermined position F. The position acquisition unit 21a acquires the position of the container 2 based on the light ray image 100R in the captured image G obtained by imaging by the imaging device 13.

[0080] With this configuration, the position of container 2 can be obtained relatively easily.

[0081] <Third Embodiment> Figure 14 is a plan view showing a container of the third embodiment. Figure 15 is a diagram illustrating an example of the processing performed by the position acquisition unit of the third embodiment. Figure 16 is a diagram illustrating another example of the processing performed by the position acquisition unit of the third embodiment.

[0082] As shown in Figure 14, this embodiment differs from the first embodiment in that the container 2 (the frame 2a of the container 2) has a mark 2b. The mark 2b is provided, for example, on the bottom surface that constitutes the inner surface of the container 2, at a position that includes the center of the bottom surface, but it may also be provided at a different location.

[0083] Furthermore, the position acquisition unit 21a acquires the position of mark 2b as the position of container 2. For example, the position acquisition unit 21a performs the processing shown in Figures 15 and 16.

[0084] As shown in Figure 15, the position acquisition unit 21a acquires an image G from the imaging device 13 (Figure 15(a)). The image G includes a mark image 2bG, which is an image of the mark 2b of the container 2. Next, the position acquisition unit 21a extracts (recognizes) the mark image 2bG from the image G and acquires (calculates) the position of the mark image 2bG (Figure 15(b)). Next, the position acquisition unit 21a acquires (calculates) the position of the center C2 of the mark 2b from the position of the mark image 2bG and determines whether the position of the center C2 of the mark 2b is located within a predetermined range R1. Figure 15(c) shows the case where the position of the center C2 of the mark 2b is located within the predetermined range R1, and Figure 15(d) shows the case where the position of the center C2 of the mark 2b is located outside the predetermined range R1.

[0085] As shown in Figure 16, the position acquisition unit 21a acquires an image G from the imaging device 13 (Figure 16(a)). The image G includes a mark image 2bG, which is an image of the mark 2b of the container 2. Next, the position acquisition unit 21a extracts (recognizes) the mark image 2bG from the image G and acquires (calculates) the position of the mark image 2bG (Figure 16(b)). Next, the position acquisition unit 21a acquires (calculates) the position of each of the four corner images 2b1G of the rectangular area surrounding the mark image 2bG, and determines whether each of the positions of the four corner images 2b1G is located within a predetermined range R1. Figure 16(c) shows the case where each of the positions of the four corner images 2b1G is located within the predetermined range R1, and Figure 16(d) shows the case where some of the positions of the four corner images 2b1G are located outside the predetermined range R1.

[0086] The determination unit 21b determines, based on the position of the container 2 acquired by the position acquisition unit 21a, whether the manipulator 12 is positioned at a predetermined working position where the relative positional relationship between the manipulator 12 and the container 2 is a predetermined relative positional relationship that allows the manipulator 12 to perform the item retrieval operation. For example, the determination unit 21b determines whether a predetermined part of the container 2 (for example, the center of mark 2b or the four corners of the rectangle surrounding mark 2b) is within a preset range. Specifically, the determination unit 21b determines whether a predetermined part of the container image 2G (for example, the center C2 or the four corner images 2b1G) is within a preset range R1. If the determination unit 21b determines that a predetermined part of the container image 2G (for example, all of the aforementioned center C2 or four corner images 2b1G) is within the preset range R1, it determines that the manipulator 12 is positioned at the predetermined working position. If the determination unit 21b determines that a predetermined part of the container image 2G (for example, the center C2 or some of the corner images 2a1G) is not within the preset range R1, it determines that the manipulator 12 is not located at the predetermined work position.

[0087] As described above, in this embodiment, the container 2 has a mark 2b. The position acquisition unit 21a acquires the position of the container 2 as the position of the mark 2b (the position of the center of the mark 2b and the positions of the four corners of the rectangle surrounding the mark 2b).

[0088] With this configuration, the position of container 2 can be obtained relatively easily. Based on the position of container 2 obtained by the position acquisition unit 21a, the determination unit 21b determines whether the manipulator 12 and container 2 are in a specified relative positional relationship that allows the manipulator 12 to perform the item removal operation. Thus, the relative positional relationship between container 2 and manipulator 12 can be determined relatively easily.

[0089] <Fourth Embodiment> Figure 17 is a diagram illustrating an example of the processing performed by the setting unit of the fourth embodiment. Figure 18 is a diagram illustrating another example of the processing performed by the setting unit of the fourth embodiment.

[0090] In this embodiment, the processing performed by the setting unit 21d differs from that of the first embodiment. The setting unit 21d, for example, receives input information for setting a predetermined work position from the input device 17 operated by a worker, and sets the predetermined work position based on the received information.

[0091] As shown in Figure 17, the position acquisition unit 21a acquires an image G from the imaging device 13 (Figure 17(a)). The image G includes a container image 2G, which is an image of the container 2. Next, the position acquisition unit 21a extracts (recognizes) the frame image 2aG, which is an image of the frame 2a of the container 2 in the image G, and acquires (calculates) the position of the frame image 2aG and the position of the center C1 (Figure 17(b)). Next, the setting unit 21d displays the image G on the display device 15 and receives input information (coordinate information) from the input device 17 for setting a predetermined working position, and sets the predetermined working position based on the received information (Figure 17(c)). Here, the predetermined range R1 in Figure 17(c) is set by the input coordinate information. This predetermined range R1 is the range in which it can be determined from the image G that the center position of the container 2 is at the predetermined position.

[0092] As shown in Figure 18, the position acquisition unit 21a acquires an image G from the imaging device 13 (Figure 18(a)). The image G includes a container image 2G, which is an image of the container 2. Next, the position acquisition unit 21a extracts (recognizes) the frame image 2aG, which is an image of the frame 2a of the container 2 in the image G, and acquires (calculates) the position of the frame image 2aG and the positions of the four corner images 2a1G (Figure 18(b)). Next, the setting unit 21d displays the image G on the display device 15 and receives input information (coordinate information) from the input device 17 for setting a predetermined working position, and sets the predetermined working position based on the received information (Figure 18(c)). Here, the predetermined range R1 in Figure 18(c) is set by the input coordinate information. This predetermined range R1 is the range in which it can be determined from the image G that the outer shape of the container 2 is in the predetermined position.

[0093] As described above, the setting unit 21d of this embodiment receives input information for setting a predetermined work position and sets the predetermined work position based on the received information.

[0094] With this configuration, the prescribed working position can be set on-site, allowing for the setting of a prescribed working position tailored to the specific site. Furthermore, if the relative positional relationship between the container 2 and the manipulator 12 is the same in the same work area, there is no need to prepare positional determination criteria for the container 2's position for each actual work area, eliminating the need for individual prior teaching of the container 2's position.

[0095] <Fifth Embodiment> Figure 19 is a diagram illustrating an example of processing performed by the processing execution unit of the fifth embodiment.

[0096] In this embodiment, the processing performed by the processing execution unit 21c differs from that of the first embodiment. In the alignment process, the processing execution unit 21c corrects the position of the container 2 on a three-dimensional model used for the operation of the manipulator 12, which includes position information of a predetermined position F, the container 2 sent to the predetermined position F, and the trolley 11, based on the position of the container 2 acquired by the position acquisition unit 21a. Specifically, the processing execution unit 21c corrects the position of the container model 2M so that the manipulator 12 is positioned at a predetermined work position. Figure 19(a) shows the captured image G. Figure 19(b) shows the container model 2M as a model of the container 2 on the three-dimensional model. The dashed line of the container model 2M is before correction, and the solid line of the container model 2M is after correction. In this way, in this embodiment, the positional deviation of the manipulator 12 in real space is corrected on the three-dimensional model. This eliminates the need for the worker to move the manipulator 12 (trolley 11) in real space. Such corrections on the 3D model can also be performed after the basic positioning of the container 2 and the manipulator 12 has been completed, as in the embodiment described above. That is, after the manipulator 12 has been positioned relative to the container 2 so that it is in a predetermined working position, this correction can be applied when another container 2 is sent and a shift occurs in their relative positioning. In this embodiment, if the position of the manipulator 12 is shifted from the predetermined working position in the relative positioning of the newly sent container 2 and the manipulator 12, the manipulator 12 can be operated on the 3D model to adjust the relative positioning between the container 2 and the manipulator 12 to eliminate the shift.

[0097] As described above, in the alignment process, the processing execution unit 21c of this embodiment corrects the position of the container 2 in the 3D model used for the operation of the manipulator 12, which includes the position information of a predetermined position F, the container 2 sent to the predetermined position F, and the trolley 11, based on the position of the container 2 acquired by the position acquisition unit 21a.

[0098] With this configuration, the relative position between the container 2 and the manipulator 12 can be corrected.

[0099] <Sixth Embodiment> Figure 20 is a diagram showing an example configuration of the handling robot system of the sixth embodiment, illustrating the handling robot system in a state where it is deployed at the work site. Figure 21 is a diagram showing another example configuration of the handling robot system of the sixth embodiment, illustrating the handling robot system in a state where it is deployed at the work site.

[0100] The examples shown in Figures 20 and 21 illustrate the application of the above-described embodiment to a work site that has the function of transporting containers 2 stored in an automated warehouse to a product dispensing port Fa via an automated transport path consisting of a transport conveyor and a rotating stage, and then transporting them back to their original storage location. The product dispensing port Fa is an example of a predetermined location.

[0101] Specifically, Figure 20 shows an example where the method of the third embodiment is applied. That is, Figure 20 shows an example where a container 2 with a mark 2b at a position including the center of the bottom surface is transported by a transport means to a product dispensing port Fa via an automatic transport path, the direction of movement of the trolley 11 required for the manipulator 12 to reach a predetermined work position is calculated, and the calculated direction of movement and amount of movement (i.e., amount of deviation) of the trolley 11 is communicated to the operator. In this example, for example, when the trolley 11 is positioned in advance at the product dispensing port Fa, the mark 2b of the container 2 transported by the transport means is imaged, and the position of the mark 2b in the imaged image G is saved. This makes it possible to align the trolley 11 at other locations with product dispensing ports Fa with similar mechanical configurations by positioning the trolley 11 so that the same imaged image G as the saved image G is captured when positioning the trolley 11 at other locations with product dispensing ports Fa with similar mechanical configurations.

[0102] Furthermore, the display device 50 in Figure 20 displays various information related to the work, for example. Figure 21 is an example of a configuration in which a light source 18 is provided, that is, an example in which the method of the second embodiment is applied.

[0103] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0104] 1…Handling robot system 2…Container 3...Goods 11... Trolley 12... Manipulator 13…Imaging device 15...Display device 16…Speakers 20…Information Processing Devices 21a...Position acquisition unit 21b...judgment section 21c... Processing execution unit 21d...Settings section F... Designated position Fa...Product dispensing slot (designated location) G...imaging

Claims

1. A trolley and A manipulator mounted on the trolley and capable of removing items from a container that has been sent to a predetermined position, and An imaging device mounted on the trolley and capable of imaging the container at the predetermined position, Information processing equipment and Equipped with, The aforementioned information processing device is A position acquisition unit acquires the position of the container based on the captured image obtained by the imaging device, A determination unit determines, based on the position of the container obtained by the position acquisition unit, whether the manipulator is positioned in a position where the item removal operation can be performed, in relation to the relative positional relationship between the manipulator and the container. Equipped with, Handling robot system.

2. The position acquisition unit acquires the position of the container frame as the position of the container. The handling robot system according to claim 1.

3. The trolley is equipped with a floodlight that projects a light beam of a specified shape onto the container at the predetermined position, The position acquisition unit acquires the position of the container based on the image of the light rays in the captured image obtained by the imaging device. The handling robot system according to claim 1.

4. The aforementioned container has a mark, The position acquisition unit acquires the position of the mark as the position of the container. The handling robot system according to claim 1.

5. The position in which the manipulator can perform the item removal operation is predetermined. The handling robot system according to claim 1.

6. The information processing device includes a setting unit that receives input of information for setting the position in which the manipulator can perform the item retrieval operation, and sets the position based on the received information. The handling robot system according to claim 1.

7. The system includes a processing unit that calculates the direction and amount of movement of the trolley required for the manipulator to reach the position where the item can be retrieved, based on the relative positional relationship between the manipulator and the container. The handling robot system according to claim 1.

8. Equipped with a display device, The processing unit causes the display device to show arrows on the display device that represent the direction and amount of movement of the trolley required for the manipulator to reach the position where the item removal operation can be performed, in relation to the relative positional relationship between the manipulator and the container. The handling robot system according to claim 7.

9. Equipped with speakers, The processing unit outputs to the speaker, in terms of the relative positional relationship between the manipulator and the container, the direction and amount of movement of the trolley required for the manipulator to reach the position where the item removal operation can be performed. The handling robot system according to claim 7.

10. The system includes a processing execution unit that corrects the position of the container in a three-dimensional model, which includes the position information of the predetermined position, the container sent to the predetermined position, and the trolley, based on the position of the container acquired by the position acquisition unit. The handling robot system according to claim 1.

11. The position acquisition unit performs image processing on the captured image to acquire the position of the container. The handling robot system according to claim 1.

12. The position acquisition unit acquires the position of the container using a trained model that has been trained to output the position of the container from the captured image. The handling robot system according to claim 1.

13. A trolley and A manipulator mounted on the trolley and capable of removing items from a container that has been sent to a predetermined position, and An imaging device mounted on the trolley and capable of imaging the container at the predetermined position, An information processing device that can be used in a handling robot system equipped with, A position acquisition unit acquires the position of the container based on the captured image obtained by the imaging device, A determination unit determines, based on the position of the container obtained by the position acquisition unit, whether the manipulator is positioned in a position where the item removal operation can be performed, in relation to the relative positional relationship between the manipulator and the container. Equipped with, Information processing device.

14. The system includes a processing execution unit that calculates, based on the position of the container acquired by the position acquisition unit, the direction and amount of movement of the trolley required for the manipulator to reach a position where the item can be retrieved, in relation to the relative positional relationship between the manipulator and the container. The information processing apparatus according to claim 13.

15. A trolley and A manipulator mounted on the trolley and capable of removing items from a container that has been sent to a predetermined position, and An imaging device mounted on the trolley and capable of imaging the container at the predetermined position, A processing method performed by an information processing device provided in a handling robot system equipped with the following: The position acquisition unit acquires the position of the container based on the captured image obtained by the imaging device, The determination unit determines, based on the position of the container obtained by the position acquisition unit, whether the manipulator is positioned in a position where the item removal operation can be performed, in relation to the relative position of the manipulator and the container. A processing method that includes this.

16. The processing unit calculates the direction and amount of movement of the trolley required for the manipulator to reach the position where the item can be retrieved, based on the relative positional relationship between the manipulator and the container. The processing method according to claim 15.

17. A trolley and A manipulator mounted on the trolley and capable of removing items from a container that has been sent to a predetermined position, and An imaging device mounted on the trolley and capable of imaging the container at the predetermined position, The computer of the information processing device installed in a handling robot system equipped with A position acquisition unit acquires the position of the container based on the captured image obtained by the imaging device, A determination unit determines, based on the position of the container obtained by the position acquisition unit, whether the manipulator is positioned in a position where the item removal operation can be performed, in relation to the relative positional relationship between the manipulator and the container. A program designed to function as such.

18. The aforementioned computer, A processing execution unit calculates the direction and amount of movement of the trolley required for the manipulator to reach the position where the item retrieval operation can be performed, in relation to the relative positional relationship between the manipulator and the container. The program according to claim 17 for functioning as such.

Images