Control device, control method, and recording medium

Abstract

A control device according to the present invention comprises an acquisition unit, a display control unit, an identification unit, and a control unit. The acquisition unit acquires a first image generated by a first imaging device and including, in the imaging range, a first region in which an article is positioned, and a second image generated by a second camera and including, in the imaging range, a second region to which the article will be moved. The display control unit causes the first image and the second image to be displayed on a display. The identification unit acquires first designation information designating a position of the article in the first image displayed on the display to identify a movement start position of the article using the first designation information, and acquires second designation information designating a movement end position of the article in the second image displayed on the display to identify the movement end position of the article using the second designation information. The control unit uses the identified movement start position and the movement end position to perform, as a moving means, processing necessary for moving the article from the first region to the second region.

Description

Control device, control method, and recording medium

[0001] The present invention relates to a control device, a control method, and a recording medium.

[0002] In recent years, devices with arms have been used to move objects. For example, a robot system described in Patent Document 1 picks up an object from a pickup area including a start position and moves it to a drop area including a target work position. In this robot system, an image of the pickup area is used to detect the posture and start position of the object. In addition, this robot system processes the image of the drop area to determine the available space between the objects.

[0003] Japanese Patent Application Laid-Open No. 2021-3801

[0004] When an article is moved using a moving means for moving an article, such as an arm, it is sometimes preferable to enable a person to easily specify the position of the article and the position to which the article is to be moved.

[0005] The control device of the present disclosure comprises: an acquisition means for acquiring a first image generated by a first imaging device whose imaging range includes a first area in which an item is located, and a second image generated by a second camera whose imaging range includes a second area to which the item is to be moved; a display control means for displaying the first image and the second image on a display; an identification means for acquiring first designation information that designates the position of the item in the first image displayed on the display, and using the first designation information to identify a start position of movement of the item, and acquiring second designation information that designates an end position of movement of the item in the second image displayed on the display, and using the second designation information to identify an end position of movement of the item; and a control means for performing processing necessary to cause a moving means to move the item from the first area to the second area using the identified start position and end position of movement.

[0006] The control method disclosed herein includes a computer: acquiring a first image generated by a first imaging device whose imaging range includes a first area in which an item is located; and a second image generated by a second camera whose imaging range includes a second area to which the item is to be moved; displaying the first image and the second image on a display; acquiring first designation information that specifies the position of the item in the first image displayed on the display; using the first designation information to identify a start position of the item's movement; acquiring second designation information that specifies an end position of the item's movement in the second image displayed on the display; using the second designation information to identify an end position of the item's movement; and using the identified start position and end position of movement, performing the processing necessary to cause a moving means to move the item from the first area to the second area.

[0007] The recording medium of the present disclosure records a program that causes a computer to have: an acquisition means that acquires a first image generated by a first imaging device whose imaging range includes a first area in which an item is located, and a second image generated by a second camera whose imaging range includes a second area to which the item is to be moved; a display control means that displays the first image and the second image on a display; an identification means that acquires first designation information that designates the position of the item in the first image displayed on the display, and uses the first designation information to identify a start position of movement of the item, and acquires second designation information that designates an end position of movement of the item in the second image displayed on the display, and uses the second designation information to identify an end position of movement of the item; and a control means that uses the identified start position and end position of movement to cause a moving means to perform processing necessary to move the item from the first area to the second area.

[0008] According to the present disclosure, when an item is moved using a moving means for moving an item, a person can easily specify the position of the item and the position to which the item is to be moved.

[0009] 1 is a diagram showing an example of a usage environment of a control device according to the present disclosure; FIG. 1 is a diagram showing an example of a functional configuration of a control device; FIG. 1 is a diagram showing an example of a screen that the control device causes to be displayed on a display; FIG. 1 is a diagram showing an example of a screen that the control device causes to be displayed on a display; FIG. 2 is a diagram showing an example of a hardware configuration of a control device; FIG. 2 is a flowchart showing an example of a process performed by a control device; FIG. 3 is a diagram showing an example of a screen that the control device causes to be displayed on a display; FIG. 4 is a diagram showing an example of a screen that the control device causes to be displayed on a display; FIG. 5 is a diagram showing an example of a screen that the control device causes to be displayed on a display; FIG. 6 is a diagram showing an example of a screen that the control device causes to be displayed on a display; FIG. 7 is a diagram showing an example of a screen that the control device causes to be displayed on a display; FIG. 8 is a diagram showing an example of a screen that the control device causes to be displayed on a display;

[0010] Hereinafter, in this disclosure, the drawings relate to one or more embodiments. In addition, in all drawings, similar components are given similar reference numerals and descriptions thereof will be omitted as appropriate.

[0011] As shown in FIG. 1, the control device 10 is a device that controls a moving device 40 that moves an article, and is used together with the moving device 40, a first imaging device 20, a second imaging device 30, a display 50, and an input device 55.

[0012] The moving device 40 is used, for example, in at least one of a factory, a warehouse, and a logistics center, and has a mechanism for moving an article. The moving device 40 is, for example, a handler robot, and has an arm for moving an article. The moving device 40 may be self-propelled. The moving device 40 moves an article located in a first area to a second area, for example, using the arm. There are various combinations of the first area and the second area. For example, the first area and the second area are both shelves, conveyors, vehicles, carts, baskets, or transport containers. The first area and the second area may be the same type or different types.

[0013] The first imaging device 20 and the second imaging device 30 generate images at a predetermined frame rate. This frame rate is, for example, but not limited to, 2 frames / second or more, preferably 10 frames / second or more. In the following description, the image generated by the first imaging device 20 will be referred to as a first image, and the image generated by the second imaging device 30 will be referred to as a second image.

[0014] Preferably, the first and second image capture devices 20 and 30 are depth cameras that generate depth information for objects captured in the images. The depth information indicates the distance from the image capture device to the object. Examples of methods for generating depth information include positioning using a laser or analyzing images generated by a stereo camera.

[0015] Furthermore, the first imaging device 20 and the second imaging device 30 may each have a tilt sensor, which can identify the rotation angle of the first image and the rotation angle of the second image relative to the horizontal plane.

[0016] The first imaging device 20 includes the first region in its imaging range. Therefore, the first image generated before the moving device 40 moves the item includes the item. On the other hand, the second imaging device 30 includes the second region in its imaging range. Therefore, the second image generated before the moving device 40 moves the item shows a location in the second region where the item can be placed. Furthermore, the second image generated after the moving device 40 moves the item includes the item.

[0017] The display 50 and the input device 55 are used, for example, by an administrator of the moving device 40. The control device 10 causes the display 50 to display a first image and a second image before the moving device 40 moves the item. The control device 10 may cause these two images to be displayed simultaneously, or may cause the display 50 to display the first image and then the second image.

[0018] When the manager of the moving device 40 sees the first image displayed on the display 50, he or she operates the input device 55 to input first designation information that designates the position of an item in the first image displayed on the display 50 to the control device 10. For example, the input device 55 generates the first position designation information by moving a pointer displayed on the display 50 with a mouse or the like and placing the pointer over an item included in the first image.

[0019] Furthermore, when the manager views the second image displayed on the display 50, he operates the input device 55 to input second designation information that designates the end position of the movement of the item in the second image displayed on the display 50 to the control device 10. For example, the input device 55 generates the second position designation information by moving a pointer displayed on the display 50 with a mouse or the like and superimposing the pointer on a desired position on the second image.

[0020] If the display 50 is a touch panel, the display 50 may also serve as the input device 55. In this case, the administrator may input at least one of the first designation information and the second designation information by touching a predetermined position on the display 50.

[0021] The control device 10 then controls the moving device 40 using the first designation information and the second designation information to move the article from the first area to the second area.

[0022] The control device 10 includes, for example, an acquisition unit 110, a display control unit 120, an identification unit 130, and a control unit 140, as shown in FIG.

[0023] The acquisition unit 110 acquires the first image and the second image. Specifically, the acquisition unit 110 repeatedly acquires these images in real time.

[0024] The display control unit 120 displays the first image and the second image on the display 50 shown in Fig. 1. The display control unit 120 may cause the first image and the second image to be displayed on the display 50 simultaneously, or may cause the first image to be displayed on the display 50 and then cause the second image to be displayed on the display 50.

[0025] If the tilt of the first imaging device 20 can be identified, the display control unit 120 of the control device 10, for example, may use the detection result of the tilt sensor to correct the first image to an image that would be obtained if the first imaging device 20 were horizontal, and display the corrected first image on the display 50. Furthermore, if the second imaging device 30 has a tilt sensor, the display control unit 120 of the control device 10, for example, may use the detection result of the tilt sensor to correct the second image to an image that would be obtained if the second imaging device 30 were horizontal, and display the corrected second image on the display 50.

[0026] The identification unit 130 acquires first designation information and second designation information. The first designation information indicates the position of the item 60 in the first image displayed on the display 50, and the second designation information indicates the end position of the movement of the item 60 in the second image displayed on the display. In other words, the first designation information indicates the location in the first image where the item 60 appears, and the second designation information indicates the location in the second image to which the item 60 should move.

[0027] Furthermore, the identification unit 130 identifies the movement start position of the article 60 using the first designation information, and identifies the movement end position of the article 60 using the second designation information. For example, the identification unit 130 identifies the movement start position and movement end position in a coordinate system that is predefined in the space in which the article 60, the moving device 40, the first imaging device 20, and the second imaging device 30 are arranged, i.e., an absolute coordinate system. In this case, information indicating the positions of the moving device 40, the first imaging device 20, and the second imaging device 30 in the absolute coordinate system is stored in advance in the control device 10. The absolute coordinate system may, for example, have an origin at a specific point in the above-mentioned space, and the z-axis, y-axis, and z-axis may be defined based on the origin, or may be a world coordinate system based on so-called latitude and longitude information.

[0028] The identification unit 130 identifies the movement start position and movement end position in the absolute coordinate system, for example, as follows. First, the identification unit 130 acquires position information in the absolute coordinate system of the first imaging device 20 at the time the first image is generated. The identification unit 130 also acquires position information in the absolute coordinate system of the second imaging device 30 at the time the second image is generated. At this time, the identification unit 130 acquires information indicating the orientations and angles of view of the first imaging device 20 and the second imaging device 30 in the absolute coordinate system. The identification unit 130 then uses this information to generate information indicating which position in the absolute coordinate system each pixel of the first image corresponds to, i.e., position information in the local coordinates applied to the first image, and also generates information indicating which position in the absolute coordinate system each pixel of the second image corresponds to, i.e., position information in the local coordinates applied to the second image. The identification unit 130 then converts the generated position information in the local coordinates according to a predetermined rule to identify the movement start position and movement end position in the absolute coordinate system. In this case, the identification unit 130 may use the depth information generated by the first imaging device 20 and the second imaging device 30 .

[0029] The control unit 140 uses the identified movement start position and movement end position to perform the processing required to cause the movement device 40 to move the item 60 from the first area to the second area.

[0030] As an example, the control unit 140 transmits the movement start position and movement end position in the absolute coordinate system to the movement device 40. The movement device 40 then moves the arm to the movement start position, holds the item 60 on the arm, moves the arm in that state to the movement end position, and then releases the item 60 from the arm, thereby moving the item 60 from the first area to the second area.

[0031] As another example, the control unit 140 directly controls the moving device 40. In this case, the control unit 140 moves the arm of the moving device 40 to a movement start position, causes the arm to hold the item 60, moves the arm in that state to a movement end position, and then releases the item 60 from the arm, thereby moving the item 60 from the first area to the second area.

[0032] In either example, the control unit 140 uses angle information of the motors attached to each joint of the arm of the moving device 40 to place the item 60 in the second area so that the item 60 is oriented in a predetermined direction.

[0033] As described above, the display control unit 120 causes the display 50 to display the first image and the second image. In the examples shown in Figures 3 and 4, the display control unit 120 causes the display 50 to display the first image, and then causes the display 50 to display the second image.

[0034] 3, when the first image is displayed on the display 50, the administrator of the mobile device 40, i.e., the user of the control device 10, performs an input to the input device 55 so that, for example, the cursor is positioned over the item 60. Then, with the cursor positioned over the item 60, the administrator performs a predetermined operation on the input device 55 to input first designation information.

[0035] When the first designation information is input, the display control unit 120 of the control device 10 displays the second image as shown in Fig. 4. The user of the control device 10 performs an input to the input device 55, for example, so that the cursor overlaps the position in the second area where the item 60 should be placed. Then, with the cursor overlapping the position, the user performs a predetermined operation on the input device 55 to input the second designation information.

[0036] The control device 10 has, as a hardware configuration, for example, as shown in FIG. 5, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0037] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0038] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0039] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0040] The storage device 1040 is an auxiliary storage device realized by removable media such as a hard disk drive (HDD), a solid state drive (SSD), or a memory card, or a read-only memory (ROM), and has a recording medium. The recording medium of the storage device 1040 stores program modules that realize each function of the control device 10 (e.g., the acquisition unit 110, the display control unit 120, the identification unit 130, and the control unit 140). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module.

[0041] The input / output interface 1050 is an interface for connecting the control device 10 to various input / output devices. For example, the control device 10 may communicate with at least one of the first imaging device 20, the second imaging device 30, the mobile device 40, the display 50, and the input device 55 via the input / output interface 1050.

[0042] The network interface 1060 is an interface for connecting the control device 10 to a network. This network is, for example, a local area network (LAN) or a wide area network (WAN). The network interface 1060 may connect to the network wirelessly or by wire. The control device 10 may communicate with at least one of the first imaging device 20, the second imaging device 30, the mobile device 40, the display 50, and the input device 55 via the network interface 1060.

[0043] The control device 10 operates, for example, as shown in FIG. 6 . First, when moving the item 60 from the first area to the second area, the acquisition unit 110 of the control device 10 acquires a first image and a second image (step S10). Here, the first imaging device 20 may always repeatedly generate and transmit the first image to the acquisition unit 110. Also, the second imaging device 30 may always repeatedly generate and transmit the second image to the acquisition unit 110. Also, the first imaging device 20 may transmit the first image to the acquisition unit 110 only when moving the item 60 from the first area to the second area. Similarly, the second imaging device 30 may transmit the second image to the acquisition unit 110 only when moving the item 60 from the first area to the second area.

[0044] Next, the display control unit 120 causes the display 50 to display the first image and the second image. The identification unit 130 also acquires first and second designation information (step S20). The identification unit 130 then uses the first designation information to identify a start position for the item 60, and uses the second designation information to identify an end position for the item 60 (step S30). The control unit 140 then performs processing required to cause the moving device 40 to move the item 60 from the first area to the second area (step S40).

[0045] In this way, when the control device 10 is used, the user of the control device 10, for example, the manager of the moving device 40, can easily specify the position of the article 60 before it is moved and the position of the article 60 to which it is moved.

[0046] It is also conceivable that the moving device 40 automatically specifies the pre-movement position and the destination position of the item 60. However, if this is done, when at least one of the shape and size of the item 60 is indefinite, or when the state of the destination is indefinite, the pre-movement position of the item 60 may be erroneously detected, or the stability of the item 60 may decrease after it is moved. In contrast, when the control device 10 is used, the pre-movement position and the destination position of the item 60 are specified by the manager of the moving device 40. Therefore, the above-mentioned problems are unlikely to occur.

[0047] The identification unit 130 of the control device 10 may generate a display item 70 having a size corresponding to the size of the item 60 in the first image, as shown in Fig. 7. The size of the display item 70 in the first image is, for example, 80% to 120%, preferably 90% to 110%, of the size of the item 60 in the first image, but is not limited to these. An example of the display item 70 is a rectangular frame surrounding the item 60. However, the display item 70 may also be an icon or a mark.

[0048] For example, the identification unit 130 may detect the item 60, its size, and its tilt by processing the first image using a machine learning model. Then, for example, if the first imaging device 20 is attached to the tip of an arm of the moving device 40, the identification unit 130 may set the orientation of a rectangular frame as the display item 70 using angle information of motors attached to each joint of the arm.

[0049] The identification unit 130 may then display the display item 70 superimposed on the second image as shown in FIG. 8 , move the display item 70 in accordance with input from the user of the control device 10, and determine the position of the display item 70 after the movement as the movement end position. The movement end position may be, for example, the center of gravity of the display item 70. In this manner, the user can easily visually recognize the movement end position. In the example shown in this figure and other examples, the display item 70 is moved by, for example, dragging and dropping.

[0050] As shown in Fig. 9, the identification unit 130 may include, in the display item, a partial image that indicates a part of the first image in which the item 60 is shown. For example, in the example shown in Fig. 9, the display item includes a rectangular partial image that is obtained by cutting out the area in which the item 60 is shown from the first image, and a rectangular frame that indicates the edge of the image. In this way, it is easier for the user to visually recognize the movement end position.

[0051] 10 , the identification unit 130 may set the size of the display item 70 on the second image to a size that corresponds to the ratio of the size of the second region to the size of the item 60 in real space. For example, the identification unit 130 sets the ratio of the size of the display item 70 on the second image to the size of the second region in the second image to the size of the second region to the size of the item 60 in real space. In this way, the user can easily imagine how the item 60 will be placed in the second region after being moved.

[0052] The identification unit 130 determines the size of the display item 70 on the second image, for example, as follows: First, the control device 10 stores in advance the ratio of the size of the object in the second image to the size of the object in the first image when the same object is photographed by the first imaging device 20 and the second imaging device 30. The control device 10 then determines the size of the display item 70 on the second image by enlarging or reducing the display item 70 in the first image according to this ratio.

[0053] Furthermore, if the first imaging device 20 and the second imaging device 30 are depth cameras, the identification unit 130 may use depth information generated by the first imaging device 20 and the second imaging device 30 in this process. For example, the identification unit 130 identifies an object that overlaps with the display item 70 among the objects shown in the second image. The identification unit 130 then determines the size of the display item 70 on the second image using a comparison result between the depth information of the object and the depth information of the item 60 in the first image. For example, if the depth of the identified object is greater than the depth of the item 60 in the first image, the identification unit 130 makes the display item 70 on the second image smaller than the display item 70 on the first image. On the other hand, if the depth of the identified object is smaller than the depth of the item 60 in the first image, the identification unit 130 makes the display item 70 on the second image larger than the display item 70 on the first image.

[0054] However, the specification unit 130 may use other methods to determine the size of the display item 70 on the second image.

[0055] As shown in FIG. 11 , the specification unit 130 may rotate the display item 70 on the second image from the display item 70 on the first image in accordance with input from a user. For example, the specification unit 130 may freely rotate the display item 70 in accordance with input from an input device such as a mouse. In this case, the display item 70 may be freely rotatable only in a specific direction, or may be freely rotatable in all directions. Furthermore, the specification unit 130 may make the display item 70 rotatable from the time the display item 70 is on the first image, or may make the display item 70 rotatable after the display item 70 is displayed on the second image.

[0056] The identification unit 130 then acquires rotation information capable of identifying the orientation of the rotated display item 70, and identifies the orientation of the item 60 in the second area using the rotation information. The rotation information may indicate, for example, the amount of rotation of the item 60 on each of the x-axis, y-axis, and z-axis. The control unit 140 then controls, for example, the arm of the movement device 40, to position the item 60 at the movement end position in the second area so that the orientation of the item 60 is the orientation acquired by the identification unit 130.

[0057] In this way, the user can specify the orientation of the moved item 60 while imagining how the moved item 60 will be placed in the second area.

[0058] 12 , the display control unit 120 may display, on the display 50, an input item 52 for inputting an input for rotating the orientation of the display item 70. The input item 52 may be an input field for inputting the amount of rotation as a numerical value, or may be an input button for rotating the display item 70 by a predetermined angle.

[0059] 13, the display control unit 120 may simultaneously display the first image and the second image on the display 50. In this way, the user can specify the movement end position by moving the display item 70 on the first image onto the second image, as shown in FIG. 14, which makes it easy to specify the movement end position.

[0060] In this case, the control device 10 may perform, for example, the following process: As a premise, the first imaging device 20 and the second imaging device 30 are both depth cameras, and generate color information (e.g., red pixel value, green pixel value, and blue pixel value) and depth information for each pixel.

[0061] First, the identification unit 130 processes the first image to determine the center coordinates (x １ , y １ ), and the size and tilt θ of the item 60 in two-dimensional space. The coordinates, size, and tilt used here are indicated using coordinates in the first image, i.e., pixel positions. The size of the item 60 is determined by, for example, the width w of a rectangular frame (an example of a display item 70) surrounding the item 60, as shown in FIG. １ and height h １ The inclination of the article 60 is indicated by, for example, the angle θ that one side of the rectangular frame makes with respect to the horizontal plane. The identification unit 130 stores this information in, for example, the storage unit 150.

[0062] Next, the specifying unit 130 moves the display item 70 in accordance with the input from the input device 55. This movement is performed by, for example, drag and drop. The specifying unit 130 calculates the center coordinates (x ２ , y ２ ), as well as the size and tilt θ of the article 60 in two-dimensional space. The coordinates, size, and tilt used here are indicated using coordinates, i.e., pixel positions, in the second image. The identification unit 130 stores this information in the storage unit 150, for example.

[0063] Then, the identification unit 130 calculates the center coordinate (x １ , y １ ) depth information z １ and obtain the center coordinates (x ２ , y ２ ) depth information z ２ Then, the specification unit 130 obtains the size of the display item 70 in the second image, for example, the width w of the rectangular frame (an example of the display item 70) surrounding the item 60, by using the following calculation formula, for example: ２ and height h２ The size of the item 60 in the display item 70 is also changed accordingly. ２ , H ２ ) = (W １ xz ２ / z １ , H １ xz ２ / z １ )

[0064] If the article 60 is a rectangular parallelepiped, the display item 70 in this method indicates the bottom surface of the article 60. However, the identification unit 130 may change the display item 70 to indicate the top surface of the article 60. In this case, the identification unit 130 determines the height D of the article 60 using the depth information included in the first image. １ In the above formula, z ２ instead of z ２ ´=(z ２ -D １ ) may be used to determine the position and size of the display item 70 in the second image.

[0065] In addition, if at least one of the first imaging device 20 and the second imaging device 30 is at an angle to the vertical direction of the ground, the identification unit 130 may perform a perspective projection transformation according to the inclination of the imaging device, and make the frame included in the displayed item 70 a trapezoid.

[0066] Furthermore, if the determination unit 130 wants to draw an area more accurately, taking into account lens distortions of the first imaging device 20 and the second imaging device 30, the determination unit 130 may determine the position and size of the display item 70 in the second image using the coordinates and depth information of the four corners of the item 60 or display item 70 instead of the center coordinates.

[0067] 15, the identification unit 130 may display the display item 70 on the second image at the same time as displaying the display item 70 on the first image. In this case, the display item 70 on the second image may be as described using FIG. 10 or may be rotatable as described using FIG. 11.

[0068] As shown in FIG. 16 , the article 60 may be transported to the first area by a conveyor. In this case, the first area is, for example, the end of the conveyor. The conveyor may be a pipe conveyor or a belt conveyor. In this case, an example of the second area is a cart or a basket cart. The moving device 40 then moves the article 60 that has been transported to the first area by the conveyor to the cart or basket cart.

[0069] When the moving device 40 has an arm, the first imaging device 20 is attached to the arm and the imaging range includes the tip of the arm, i.e., the part holding the item 60. The image generated by the first imaging device 20 at the timing when the tip of the arm faces the first area becomes the first image.

[0070] The second imaging device 30 is attached to a cart or a cage cart, and the imaging range includes the area of ​​the cart or cage cart where the article 60 is placed.

[0071] The control device 10 pre-stores the positions and orientations in the absolute coordinate system of the first area, the moving device 40, and the cart or basket, and also pre-stores information indicating which part of the cart or basket the second imaging device 30 is attached to.

[0072] The identification unit 130, for example, of the control device 10 identifies the position and orientation of the first image capture device 20 in the absolute coordinate system using control information for the arm of the moving device 40, for example, information indicating in what direction and to what extent the tip of the arm extends from the main body of the moving device 40 and how much it has rotated. Furthermore, the identification unit 130, for example, of the control device 10 identifies the position and orientation of the second image capture device 30 in the absolute coordinate system using information indicating the position and orientation of the cart or cage cart in the absolute coordinate system and which part of the cart or cage cart the second image capture device 30 is attached to.

[0073] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0074] In addition, although the flowcharts used in the above description show a number of steps (processes) in a sequential order, the order of the steps in each embodiment is not limited to the order shown in the flowcharts. In each embodiment, the order of the steps shown in the diagrams can be changed as long as it does not cause any problems in terms of the content.

[0075] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: 1. A control device comprising: an acquisition means for acquiring a first image generated by a first imaging means whose imaging range includes a first area where an item is located, and a second image generated by a second imaging means whose imaging range includes a second area to which the item is to be moved; a display control means for displaying the first image and the second image on a display; an identification means for acquiring first designation information designating the position of the item in the first image displayed on the display, and specifying a start position of movement of the item using the first designation information, and acquiring second designation information designating an end position of movement of the item in the second image displayed on the display, and specifying a end position of movement of the item using the second designation information; and a control means for performing processing required to cause a moving means to move the item from the first area to the second area using the specified start position and end position of movement. 2. In the control device described above in 1, the display control means causes the first image and the second image to be displayed simultaneously on the display. 3. 3. The control device according to claim 1 or 2, wherein the specifying means generates a display item having a size corresponding to the size of the item in the first image, displays the display item on the second image in accordance with input from a user, and determines the position of the display item after the movement as the movement end position. 4. The control device according to claim 3, wherein the specifying means sets the size of the display item on the second image to a size corresponding to the ratio of the size of the second area to the size of the item in real space. 5. The control device according to claim 3 or 4, wherein the display item includes a partial image that is part of the first image and indicates an area in which the item is shown.6. The control device according to any one of claims 3 to 5, wherein the identification means rotates the display item on the second image from the display item on the first image in accordance with input from the user, acquires rotation information capable of identifying the orientation of the displayed item after rotation, and identifies the orientation of the item in the second area using the rotation information, and the control means controls the movement means to position the item at the movement end position in the second area so that the item is in the orientation of the item acquired by the identification means. 7. The control device according to any one of claims 1 to 6, wherein the identification means identifies the movement start position and the movement end position in an absolute coordinate system predefined in a space in which the item, the movement means, the first imaging means, and the second imaging means are disposed. 8. The control device according to any one of claims 1 to 7, wherein the item is transported to the first area by a conveyor, the second area is a cart or a basket, and the movement means has an arm. 9. The control device according to claim 8, wherein the first imaging means is attached to the arm. 10. 11. A control method comprising: a computer acquiring a first image generated by a first imaging device whose imaging range includes a first area in which an article is located, and a second image generated by a second camera whose imaging range includes a second area to which the article is to be moved, displaying the first image and the second image on a display, acquiring first designation information that designates the position of the article in the first image displayed on the display, specifying a start position of movement of the article using the first designation information, and acquiring second designation information that designates an end position of movement of the article in the second image displayed on the display, specifying a end position of movement of the article using the second designation information, and performing processing required to cause a moving means to move the article from the first area to the second area using the specified start position and end position of movement.12. A recording medium having recorded thereon a program that causes a computer to have: an acquisition means that acquires a first image generated by a first imaging device whose imaging range includes a first area in which an article is located, and a second image generated by a second camera whose imaging range includes a second area to which the article is to be moved; a display control means that displays the first image and the second image on a display; an identification means that acquires first designation information that designates the position of the article in the first image displayed on the display, and uses the first designation information to identify a start position of movement of the article, and acquires second designation information that designates an end position of movement of the article in the second image displayed on the display, and uses the second designation information to identify an end position of movement of the article; and a control means that performs processing necessary to cause a moving means to move the article from the first area to the second area using the identified start position and end position of movement.

[0076] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 10, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 11 and 12 in the same dependent relationship as Supplementary Notes 2 to 10. Furthermore, not limited to Supplementary Notes 1, 11, and 12, some or all of the configurations described as Supplements may be made dependent on various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0077] REFERENCE SIGNS LIST 10 control device 20 first imaging device 30 second imaging device 40 moving device 50 display 55 input device 60 item 70 display item 80 trolley or basket cart 90 conveyor 110 acquisition unit 120 display control unit 130 identification unit 140 control unit 150 storage unit

Claims

1. A control device comprising: an acquisition means for acquiring a first image generated by a first imaging means whose imaging range includes a first area in which an item is located, and a second image generated by a second imaging means whose imaging range includes a second area to which the item is to be moved; a display control means for displaying the first image and the second image on a display; an identification means for acquiring first designation information which designates the position of the item in the first image displayed on the display, and using the first designation information to identify a start position of the item's movement, and acquiring second designation information which designates an end position of the item's movement in the second image displayed on the display, and using the second designation information to identify an end position of the item's movement; and a control means for performing the processing necessary to cause a moving means to move the item from the first area to the second area using the identified start position and end position of movement.

2. A control device according to claim 1, wherein the display control means causes the first image and the second image to be displayed simultaneously on the display.

3. A control device according to claim 1 or 2, wherein the identification means generates a display item having a size corresponding to the size of the item in the first image, displays the display item on the second image in accordance with input from the user, and determines the position of the display item after the movement as the movement end position.

4. A control device according to claim 3, wherein the specifying means sets the size of the displayed item on the second image to a size corresponding to the ratio of the size of the second area to the size of the item in real space.

5. A control device according to claim 3 or 4, wherein the display item includes a partial image that is a part of the first image and indicates an area in which the item is shown.

6. A control device according to any one of claims 3 to 5, wherein the identification means rotates the display item on the second image from the display item on the first image in accordance with input from the user, obtains rotation information capable of identifying the orientation of the display item after rotation, and identifies the orientation of the item in the second area using the rotation information, and the control means controls the movement means to position the item at the end of movement position in the second area so that it is oriented in the orientation of the item obtained by the identification means.

7. A control device according to any one of claims 1 to 6, wherein the specifying means specifies the movement start position and the movement end position in an absolute coordinate system that is predefined in a space in which the article, the moving means, the first imaging means, and the second imaging means are arranged.

8. A control device according to any one of claims 1 to 7, wherein the article is transported to the first area by a conveyor, the second area is a cart or a cage cart, and the moving means has an arm.

9. A control device according to claim 8, wherein the first imaging means is attached to the arm.

10. A control device according to claim 8 or 9, wherein the second imaging means is attached to the carriage or cart.

11. A control method in which a computer acquires a first image generated by a first imaging device whose imaging range includes a first area in which an item is located, and a second image generated by a second camera whose imaging range includes a second area to which the item is to be moved, displays the first image and the second image on a display, acquires first designation information that specifies the position of the item in the first image displayed on the display, identifies a start position for the item to move using the first designation information, acquires second designation information that specifies an end position for the item to move using the second designation information, and identifies an end position for the item to move using the second designation information, and performs processing required for a moving means to move the item from the first area to the second area using the identified start position and end position for the item.

12. A recording medium having recorded thereon a program that causes a computer to have: an acquisition means for acquiring a first image generated by a first imaging device whose imaging range includes a first area in which an item is located, and a second image generated by a second camera whose imaging range includes a second area to which the item is to be moved; a display control means for displaying the first image and the second image on a display; an identification means for acquiring first designation information that designates the position of the item in the first image displayed on the display, and using the first designation information to identify a start position of the item's movement, and acquiring second designation information that designates an end position of the item's movement in the second image displayed on the display, and using the second designation information to identify an end position of the item's movement; and a control means for performing processing necessary to cause a moving means to move the item from the first area to the second area using the identified start position and end position of movement.

Images