Packing support device, packing support method, and packing support system

Abstract

To provide box packing assistance means with which it is possible to efficiently arrange objects of various shapes and sizes in an open space such as a box.SOLUTION: Provided is a box packing assistance device including: an image acquisition unit for acquiring a target object image that indicates a target object to be boxed and a box image that indicates a box; an object analysis unit for analyzing the target object image, extracting the contour shape of the target object, and generating a contour shape image that indicates the contour shape of the target object; a box analysis unit for analyzing the box image, discriminating an occupied region of the box that is occupied by other objects and a non-occupied region where an object can be arranged, and generating an arrangement state image that indicates the occupied region and non-occupied region of the box; and an arrangement configuration determination unit for determining, on the basis of the contour shape image and the arrangement state image, an arrangement configuration that indicates the position and angle recommended for arrangement of the target object in the non-occupied region of the box, and outputting arrangement configuration information that indicates the arrangement configuration.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a packing support device, a packing support method, and a packing support system. [Background technology]

[0002] When transporting goods, a plurality of goods may be packed into a container such as a cardboard box in order to improve the convenience of transport. Currently, the work of packing goods into boxes is often performed manually. However, the work of packing goods into boxes must be performed quickly and carefully, and when the goods are packed into boxes in a hurry by hand, the goods may be arranged in a random order or may be arranged in a state where they are easily damaged during transport.

[0003] To avoid the above problems, it is very laborious to continue the work of carefully and quickly packing items into boxes for a long time, and it is difficult to secure the manpower to do the work. Therefore, in recent years, there has been an increasing demand for means to automate the boxing work using robots, etc., and several proposals have been made. For example, Non-Patent Document 1 discloses the following. "Is it possible to learn a robotic assembly policy that can generalize to new objects? We consider this idea in the context of a kit assembly task. Traditional methods rely heavily on object pose estimation and therefore often struggle to generalize to new objects without a 3D CAD model or task-specific training data. In this paper, we propose to formulate the kit assembly task as a shape matching problem. The goal is to learn, from visual input, shape descriptors that establish geometric correspondences between object surfaces and target placement locations on that object. This formulation allows the model to gain a broad understanding of how shapes and surfaces come together for assembly, and to generalize to new objects and kits. To obtain training data for the model, we present a self-supervised data collection pipeline that obtains ground truth indicating object-placement correspondences by disassembling complete kits. The resulting real-world system, Form2Fit, learns effective pick-and-place strategies for assembling objects into a variety of kits. It achieves an average success rate of 90% under a variety of initial conditions (e.g., various object and kit poses), and achieves success rates of over 86% on multiple kits, as well as on completely new objects and kits." [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly, Kevin Zakka, Andy Zeng, Johnny Lee, Shuran Song, IEEE International Conference on Robotics and Automation (ICRA) 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] Non-Patent Document 1 proposes a deep learning model for arranging an object of a specific shape at a specific placement position by learning to match the object of a specific shape with a specific placement position. According to Non-Patent Document 1, it is possible to automate the assembly of cardboard products, furniture, and the like.

[0006] However, in the method described in Non-Patent Document 1, objects are arranged based on a previously learned correspondence between objects of a specific shape and specific arrangement positions, and it is not assumed that objects of various shapes and sizes can be efficiently arranged in an open space such as a box, etc. For this reason, it is difficult to apply this method to, for example, logistics that requires packing many types of items, and a method capable of automatically packing any object is desired.

[0007] Therefore, an object of the present disclosure is to provide a packing assistance means that can efficiently arrange objects of various shapes and sizes in an open space such as a box. [Means for solving the problem]

[0008] In order to solve the above problems, one representative packing support device disclosed herein is a packing support device that determines a recommended arrangement configuration for packing an object of an arbitrary shape into a box, and includes an image acquisition unit that acquires a target object image showing a target object to be packed into a box and a box image showing the box, an object analysis unit that analyzes the target object image, extracts a contour shape of the target object, and generates a contour shape image showing the contour shape of the target object, a box analysis unit that analyzes the box image, distinguishes between occupied areas of the box that are occupied by other objects and non-occupied areas in which an object can be placed, and generates an arrangement state image showing the occupied areas and the non-occupied areas in the box, and an arrangement configuration determination unit that determines an arrangement configuration indicating a position and angle recommended for packing the target object in the non-occupied area of ​​the box based on the contour shape image and the arrangement state image, and outputs arrangement configuration information indicating the arrangement configuration. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a packing assistance means that can efficiently arrange objects of various shapes and sizes in an open space such as a box. Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a computer system for implementing an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of a packing support system according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a diagram illustrating an example of a data flow in the packing support device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of an arrangement state image and a contour shape image according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a diagram illustrating an example of a flow of a packing support method according to an embodiment of the present disclosure. [Figure 6]FIG. 6 is a diagram illustrating an example of a flow of a training process of a deep reinforcement neural network of the arrangement configuration determination unit according to an embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates an example arrangement for leaving the largest rectangular area of ​​an unoccupied area of ​​a box, according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating an example of a collision determination process according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating an example of predictive placement according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The examples are illustrative for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings. When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts, or when there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0012] In the embodiments, a process performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., CPU, GPU), and performs the process defined by the program while using a storage resource (e.g., memory) and an interface device (e.g., communication port), etc. Therefore, the subject of the process performed by executing the program may be the processor. Similarly, the subject of the process performed by executing the program may be a controller, device, system, computer, or node having a processor. The subject of the process performed by executing the program may be a calculation unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device), etc.

[0013] The program may be installed in the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource that stores the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in the embodiment, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0014] First, referring to Fig. 1, a computer system 100 for implementing an embodiment of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 100 include one or more processors 102, memory 104, a terminal interface 112, a storage interface 113, an I / O (input / output) device interface 114, and a network interface 115. These components may be interconnected via a memory bus 106, an I / O bus 108, a bus interface unit 109, and an I / O bus interface unit 110.

[0015] Computer system 100 may include one or more general purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to as processors 102. In some embodiments, computer system 100 may include multiple processors, while in other embodiments, computer system 100 may be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include an on-board cache.

[0016] In one embodiment, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 104 may store all or a portion of the programs, modules, and data structures that implement the functions described herein. For example, memory 104 may store a boxing assistance application 150. In one embodiment, boxing assistance application 150 may include instructions or descriptions that execute on processor 102 the functions described below.

[0017] In some embodiments, the boxing assistance application 150 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, the boxing assistance application 150 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with the bus interface unit 109, the processor 102, or other hardware of the computer system 100.

[0018] Computer system 100 may include a bus interface unit 109 that provides communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may couple to an I / O bus 108 for transferring data to and from various I / O units. I / O bus interface unit 110 may communicate via I / O bus 108 with multiple I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs).

[0019] Display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 126. Computer system 100 may also include one or more sensors or other devices configured to collect data and provide the data to processor 102.

[0020] For example, computer system 100 may include biometric sensors to collect heart rate data, stress level data, etc., environmental sensors to collect humidity data, temperature data, pressure data, etc., and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 124 may be connected to a display device 126, such as a standalone display screen, a television, a tablet, or a handheld device.

[0021] The I / O interface unit provides the ability to communicate with various storage or I / O devices. For example, the terminal interface unit 112 may be attached to user I / O devices 116, such as user output devices, such as a video display, a television with speakers, and user input devices, such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions to the user I / O devices 116 and the computer system 100, and receive output data from the computer system 100, by manipulating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer, for example, via the user I / O devices 116.

[0022] Storage interface 113 allows attachment of one or more disk drives or direct access storage device 117 (usually a magnetic disk drive storage device, but may be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, storage device 117 may be implemented as any secondary storage device. Contents of memory 104 may be stored in storage device 117 and retrieved from storage device 117 as needed. I / O device interface 114 may provide an interface to other I / O devices such as printers, fax machines, etc. Network interface 115 may provide a communications path to allow computer system 100 and other devices to communicate with each other. This communications path may be, for example, network 130.

[0023] In one embodiment, computer system 100 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer, etc. In other embodiments, computer system 100 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

[0024] Next, a configuration of a packing support system according to an embodiment of the present disclosure will be described with reference to FIG.

[0025] Fig. 2 is a diagram illustrating an example of a configuration of a packing support system 200 according to an embodiment of the present disclosure. As illustrated in Fig. 2, the packing support system 200 mainly includes a packing support device 230 that determines a recommended arrangement configuration when arranging a target object 220 in a box 210, a communication network 240, and a robot 250. The packing support device 230 and the robot 250 are connected to each other via the communication network 240. The communication network 240 here may include, for example, the Internet, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), etc.

[0026] The box 210 is a box for containing one or more target objects 220 for transportation or the like. The box 210 here is a container made of metal, wood, plywood, recycled wood, fiberboard, plastic, or other suitable material, and is configured with perfect rectangular or polygonal surfaces. The size, area, volume, shape, etc. of the box 210 are not particularly limited, and may be appropriately selected depending on the type, number, etc. of the target objects 220 to be packed. As an example, the box 210 may be a cardboard box measuring 23 cm x 32 cm x 15 cm.

[0027] The target object 220 is an object contained in the box 210. The target object 220 here may be any object having any size and shape. In FIG. 2, a case where the target object 220 is a book is shown as an example, but in reality, the target object 220 may be any object, such as clothes, a water bottle, a smartphone, or pet food. Furthermore, when multiple target objects 220 are placed in the box 210, each target object 220 may be different.

[0028] The packing assistance device 230 is a device that determines a recommended arrangement configuration when packing the target object 220 into the box 210.

[0029] 2, the packing support device 230 includes functional units, namely, an image acquisition unit 232, an object analysis unit 234, a box analysis unit 236, and a layout configuration determination unit 238. Each of these functional units may be realized, for example, as a software module in the packing support application 150 executed by the computer system 100 shown in FIG. 1, or may be realized as a dedicated hardware device.

[0030] The image acquisition unit 232 is a functional unit that acquires a target object image showing the target object 220 to be packed into the box 210 and a box image showing the box 210. The image acquisition unit 232 here may be an imaging device such as an RGB camera or a depth camera. In addition, in an embodiment, the image acquisition unit 232 may acquire a target object image and a box image captured by another imaging device such as a smartphone from the other imaging device. The target object image may be, for example, an overhead image showing the overall shape of the target object 220 as viewed from above. The box image may be an overhead image showing the entire bottom surface of the box 210 as viewed from above.

[0031] The object analysis unit 234 is a functional unit that analyzes the target object image, extracts the contour shape of the target object 220, and generates a contour shape image showing the contour shape of the target object 220. The object analysis unit 234 may be a deep neural network for object detection, a deep neural network for image segmentation, a depth camera, or the like. By using a deep neural network for object detection, a deep neural network for image segmentation, or a depth camera as the object analyzer 234, the contour shape of objects of any shape and size can be determined.

[0032] The box analysis unit 236 is a functional unit that analyzes the box image, distinguishes between occupied areas in the box 210 that are occupied by other objects and unoccupied areas in which objects can be placed, and generates an arrangement state image showing the occupied and unoccupied areas in the box. The box analysis unit 236 may be, for example, a deep neural network for image division, a depth camera, or the like. A deep neural network for object detection, a deep neural network for image segmentation, or a depth camera can be used as the box analyzer 236 to determine occupied and unoccupied areas in any box.

[0033] The arrangement configuration determination unit 238 is a functional unit that determines an arrangement configuration indicating a position and angle recommended for arranging the target object 220 in the unoccupied area of ​​the box 210 based on the contour shape image generated by the object analysis unit 234 and the arrangement state image generated by the box analysis unit 236, and outputs arrangement configuration information indicating the determined arrangement configuration. The "recommended arrangement configuration" here means an arrangement position and arrangement angle (direction) for leaving the largest usable area in the box 210 when the target object 220 is arranged in the box 210. In this way, according to this embodiment, it is possible to improve the space utilization efficiency of the box and arrange more target objects 220 in the box 210. The arrangement configuration determination unit 238 may be, for example, a deep neural network.

[0034] The robot 250 is a robot that performs a boxing operation based on the arrangement configuration information output to the arrangement configuration determining unit 238. For example, the robot 250 may arrange the target object 220 in the box 210 according to a recommended arrangement configuration specified in the arrangement configuration information output to the arrangement configuration determining unit 238. The robot 250 here is desirably a robot for light work, such as a single-arm robot or a dual-arm robot, suitable for boxing operation, but is not particularly limited in the present disclosure, and may be a robot of any structure as long as it is capable of boxing.

[0035] According to the above-described packing support system 200, it is possible to provide a packing support means capable of efficiently arranging objects of various shapes and sizes in an open space such as a box.

[0036] Next, a data flow in the packing support device according to an embodiment of the present disclosure will be described with reference to FIG.

[0037] FIG. 3 is a diagram showing an example of a data flow in the packing support device 230 according to an embodiment of the present disclosure.

[0038] First, the image acquisition unit 232 acquires a target object image 325 showing a target object to be packed into a box, and a box image 315 showing the box. As described above, the target object image 325 may be, for example, an overhead image showing the overall shape of the target object as seen from above. This makes it possible to determine the shape and occupied area of ​​the target object, as will be described later. Also, the box image 315 may be an overhead image showing the entire bottom surface as seen from above the box. This makes it possible to determine occupied areas of the box that are already occupied by other objects and unoccupied areas where objects can be placed. Here, the image acquisition unit 232 may be a camera that captures the target object and the box. In an embodiment, the image acquisition unit 232 may input the target object image 325 and the box image 315 from another device (e.g., a smartphone, etc.). After acquiring the target object image 325 and the box image 315, the image acquisition unit 232 transfers the target object image 325 to the object analysis unit 234 and transfers the box image 315 to the box analysis unit 236.

[0039] The box analysis unit 236 analyzes the box image 315, distinguishes between occupied areas of the box that are occupied by other objects and unoccupied areas in which objects can be placed, and generates an arrangement state image showing the occupied areas and unoccupied areas of the box. Here, the box analysis unit 236 may generate an arrangement state image showing the occupied areas and unoccupied areas of the box by analyzing the box image 315 using a deep neural network for image division. This arrangement state image may be, for example, a binary matrix that represents unoccupied areas as a first value (e.g., "0") and occupied areas as a second value (e.g., "1"). In addition, the method of training the deep neural network for image segmentation to generate this placement state image is not particularly limited, and any existing training method may be used as long as it can distinguish between areas where objects are placed and areas where objects are not placed.

[0040] The object analysis unit 234 analyzes the target object image 325, extracts the contour shape of the target object, and generates a contour shape image showing the contour shape of the target object. Here, the object analysis unit 234 may extract the contour shape of the target object by analyzing the target object image 325 using a deep neural network for object detection, and then generate a contour shape image showing the contour shape of the target object. This contour shape image may be a binary matrix that represents the background region as a first value (e.g., "0") and represents the region belonging to the target object as a second value (e.g., "1"), similar to the arrangement state image. In addition, the method of training the deep neural network for object detection to generate this contour shape image is not particularly limited, and any existing training method may be used as long as it can determine the shape and occupied area of ​​the target object.

[0041] The placement configuration determination unit 238 determines a placement configuration indicating a position and angle recommended for placing the target object in the unoccupied area of ​​the box based on the contour shape image generated by the object analysis unit 234 and the placement state image generated by the box analysis unit 236, and outputs placement configuration information 345 indicating the determined placement configuration. This placement configuration information 345 may include, for example, the coordinates and angle of the position at which the target object is to be placed. In addition, this placement configuration information 345 may be displayed on a predetermined display, for example, or may be transferred to a robot (for example, the robot 250 shown in FIG. 2) that performs the boxing work. Here, the layout configuration determination unit 238 may use a deep reinforcement neural network trained to determine a layout configuration in which a rectangular area (e.g., the largest inscribed rectangular area) that satisfies a predetermined area standard remains when a target object is placed in an unoccupied area of ​​a box, as described below. This deep reinforcement neural network may be, for example, a so-called DQN (Deep Q Network).

[0042] According to the above-described packing support device 230, it is possible to provide a packing support means capable of efficiently arranging objects of various shapes and sizes in an open space such as a box.

[0043] Next, an arrangement state image and a contour shape image according to an embodiment of the present disclosure will be described with reference to FIG.

[0044] 4 is a diagram showing an example of an arrangement state image 410 and a contour shape image 420 according to an embodiment of the present disclosure. As described above, the arrangement state image 410 and the contour shape image 420 may be expressed by an N×M binary matrix. For example, as shown in FIG. 4, the placement state image 410 may represent unoccupied areas 412 where an object can be placed as a first value such as "0" (white), and occupied areas 414 that are occupied by other objects as a second value such as "1" (black). Similarly, the contour shape image 420 may be a binary matrix that represents background regions 422 as a first value such as "0" (white) and regions 424 belonging to the target object as a second value such as "1" (black).

[0045] In this way, by expressing the arrangement state image 410 and the contour shape image 420 as a binary matrix, visual information irrelevant to the determination of the arrangement configuration is omitted from the original box image and target object image, reducing the file size and facilitating collision determination and arrangement configuration determination by the arrangement configuration determination unit.

[0046] Next, a packing support method according to an embodiment of the present disclosure will be described with reference to FIG.

[0047] Fig. 5 is a diagram showing an example of the flow of a packing support method 500 according to an embodiment of the present disclosure. The packing support method 500 shown in Fig. 5 is a process for generating a recommended arrangement configuration when packing a target object into a box, and is executed by each functional unit of the above-mentioned packing support device (for example, the packing support device 230 shown in Figs. 2 and 3). When executing the packing support method 500, each neural network used (for example, an object detection neural network that generates a contour extraction image or an arrangement state image, an image segmentation neural network, and a deep reinforcement neural network for determining the arrangement configuration) has already been trained. The training process of the deep reinforcement neural network for determining the arrangement configuration will be described later with reference to FIG. 6, for example.

[0048] First, in step S502, an image acquisition unit (for example, image acquisition unit 232 shown in FIG. 2 and FIG. 3) of the packing support device acquires a target object image. As described above, the target object image may be, for example, an overhead image showing the overall shape of the target object as seen from above. Here, when the image acquisition unit is a photographing device such as a camera, the image acquisition unit may acquire the target object image by photographing the target object. Also, in an embodiment, the image acquisition unit may acquire a target object image photographed by another photographing device from the other photographing device. For example, the image acquisition unit may access a storage unit (hard disk drive, cloud-type storage) in which the target object image is saved, and acquire the target object image, or may send a notification to the user to prompt the user to input the target object image.

[0049] Next, in step S504, the image acquisition unit acquires a box image. As described above, the box image 315 may be an overhead image showing the entire bottom surface of the box as seen from above. Here, when the image acquisition unit is a photographing device such as a camera, the image acquisition unit may acquire a target object image by photographing the box. Also, in an embodiment, the image acquisition unit may acquire a box image photographed by another photographing device from the other photographing device. For example, the image acquisition unit may access a storage unit (hard disk drive, cloud-type storage) in which the box image is saved, and acquire the box image, or may send a notification to the user to prompt the user to input the box image.

[0050] Next, in step S506, an object analysis unit (for example, object analysis unit 234 shown in Figs. 2 and 3) analyzes the target object image acquired in step S502 using a deep neural network for object detection or a deep neural network for image segmentation to determine the contour shape of the target object and generate a contour shape image. The contour shape image here may be, for example, a binary matrix in which the background region is represented as a first value such as "0" and the region belonging to the target object is represented as a second value such as "1" as described with reference to Fig. 4.

[0051] Next, in step S508, a box analysis unit (for example, box analysis unit 236 shown in Figs. 2 and 3) analyzes the box image acquired in step S504 using a deep neural network for image division to determine occupied and unoccupied areas in the box and generate an arrangement state image. The arrangement state image here may be, for example, as described with reference to Fig. 4, a binary matrix in which an unoccupied area where an object can be placed is represented as a first value such as "0" and an occupied area 414 occupied by another object is represented as a second value such as "1".

[0052] Next, in step S510, a placement configuration determination unit (for example, placement configuration determination unit 238 shown in FIG. 2 and FIG. 3) determines a placement configuration indicating a recommended position and angle for placing the target object in the unoccupied area of ​​the box, based on the contour shape image generated in step S506 and the placement state image generated in step S508. This recommended placement configuration means a placement position and placement angle (orientation) for leaving the largest usable area in the box when an object is placed in the box. In addition, this placement configuration information may include, for example, the coordinates and angle of the position at which the target object is placed. Here, the arrangement configuration determination unit may determine the arrangement configuration using a deep reinforcement neural network that is trained to determine, as the arrangement configuration, the position and angle at which a rectangular area that satisfies a predetermined area criterion remains in the unoccupied area of ​​the box when the target object is placed. The training process of this deep reinforcement neural network will be described with reference to FIGS.

[0053] Next, in step S512, the placement configuration determination unit outputs the placement configuration information determined in step S510. The placement configuration information may be displayed on a predetermined display, for example, or may be transferred to a robot that performs the packing work (for example, the robot 250 shown in FIG. 2).

[0054] According to the above-described packing support method 500, it is possible to provide a packing support means capable of efficiently arranging objects of various shapes and sizes in an open space such as a box.

[0055] Next, a training process of the deep reinforcement neural network of the layout configuration determination unit according to the embodiment of the present disclosure will be described with reference to FIG.

[0056] 6 is a diagram showing an example of a flow of a training process 600 of a deep reinforcement neural network used in an arrangement configuration determination unit according to an embodiment of the present disclosure. The training process 600 shown in FIG. 6 is a process for training a deep reinforcement neural network to generate an optimal arrangement configuration, and is performed on an untrained deep reinforcement neural network.

[0057] First, in step S602, an untrained deep reinforcement neural network inputs a training outline shape image and an arrangement state image. The training outline shape image and arrangement state image here are substantially similar to the outline shape image and arrangement state image used in the above-described box packing support method 500, but show a different target object (e.g., a second target object) or a different box (e.g., a second box) from the outline shape image and arrangement state image. These contour images and placement images may be generated by, for example, analyzing target object images and box images acquired by photographing a set of various target objects and boxes using the object analyzer and box analyzer described above. In another embodiment, the training contour images and placement images may be generated by a user or machine learning techniques.

[0058] Next, in step S604, the untrained deep reinforcement neural network randomly generates an arrangement for arranging the target object shown in the training outline shape image in the box shown in the training arrangement state image, based on the training outline shape image and the arrangement state image acquired in step S602. More specifically, the untrained deep reinforcement neural network arranges the region of the target object shown in the training outline shape image in the training arrangement state image at a random position and angle.

[0059] Next, in step S606, the untrained deep reinforcement neural network performs collision determination for the random arrangement generated in step S604. This collision determination is a process for determining whether or not the target object is placed in the occupied area of ​​the box in the random arrangement generated in step S604 (i.e., whether or not the target object collides with another object already placed in the box). If it is determined that the target object and the occupied area of ​​the box overlap (i.e., a collision exists), the process returns to step S604, and the untrained deep reinforcement neural network randomly generates a new arrangement. If it is determined that the target object and the occupied area of ​​the box do not overlap (i.e., no collision exists), the process proceeds to step S608. Note that if it is determined that a collision exists, the process may return to step S602. An example of the collision determination will be described with reference to FIG.

[0060] Next, if there is no collision between the target object and the occupied area of ​​the box in the random configuration generated in step S604, in step S608, the untrained deep reinforcement neural network calculates a reward for the deep reinforcement neural network and adjusts the network parameters of the deep reinforcement neural network based on the reward. More specifically, when the target object is placed in the box according to the placement configuration generated in step S604, the untrained deep reinforcement neural network calculates the area of ​​the largest rectangular area remaining in the unoccupied area of ​​the box, and adjusts the network parameters of the deep reinforcement neural network using the calculated area of ​​the largest rectangular area as a reward for the deep reinforcement neural network. Here, the untrained deep reinforcement neural network may adjust the network parameters using a so-called backpropagation method.

[0061] Here, the deep reinforcement neural network may identify the unoccupied area of ​​the box using, for example, OpenCV or a neural network for image segmentation, and then calculate the area of ​​the largest rectangular region in the unoccupied area. In this way, the deep reinforcement neural network can learn to generate configurations that leave larger rectangular areas in the box with high accuracy by repeating this training process, since the deep reinforcement neural network can earn higher rewards by generating configurations that leave larger rectangular areas in the box. As a result, a trained deep reinforcement neural network is constructed that can generate configurations that leave the most usable area in the box, making it possible to efficiently place objects of various shapes and sizes in an open space such as a box. An example of an arrangement in which the largest rectangular area in the unoccupied area of ​​the box is left will be described with reference to FIG.

[0062] Next, in step S610, it is determined whether the placement determination accuracy of the deep reinforced neural network achieves a predetermined accuracy standard. Here, when a test outline shape image and a placement state image are input to the deep reinforced neural network, the placement determination accuracy of the deep reinforced neural network may be verified by evaluating the placement configuration output from the deep reinforced neural network (whether there is a collision between the target object and the occupied area of ​​the box, and whether the maximum rectangular area remains in the unoccupied area). If the placement determination accuracy of the deep reinforcement neural network meets the predetermined accuracy standard, the process proceeds to step S612. On the other hand, if the placement determination accuracy of the deep reinforcement neural network does not meet the predetermined accuracy standard, the process returns to step S602, and the process from step S602 onwards is repeated using another training contour shape image and placement state image.

[0063] Next, if the placement determination accuracy of the deep reinforcement neural network achieves a predetermined accuracy standard, in step S612, the training process 600 ends, the current network parameters of the deep reinforcement neural network are determined, and a trained deep reinforcement neural network is constructed. This trained deep reinforcement neural network is used to determine a placement configuration, for example, in the packing support method 500 described with reference to FIG. The predetermined accuracy here may be set by the user, for example, or may be set by statistical analysis, etc. In step S610, the above-mentioned determination may not be performed, and for example, the number of repetitions may be counted, and when the number reaches a predetermined number, the process may proceed to step S612. Alternatively, other determination methods may be used.

[0064] The training process 600 described above allows for the construction of a trained deep reinforcement neural network capable of generating configurations that leave the most usable area in the box. Although the above describes an example in which the deep reinforcement neural network is trained to generate an arrangement that leaves a larger rectangular area in the unoccupied area of ​​the box, the present disclosure is not limited to this. For example, the deep reinforcement neural network may be trained to generate an arrangement that minimizes the distance between multiple objects. In addition, for the sake of convenience, the above describes an example in which collision detection and reward calculation are performed using a deep reinforcement neural network, but the present disclosure is not limited to this, and collision detection and reward calculation may be performed by the above-mentioned configuration detection unit or other functional units.

[0065] Next, with reference to FIG. 7, an arrangement for leaving the largest rectangular area in the unoccupied area of ​​a box according to an embodiment of the present disclosure will be described.

[0066] As described above, during the training phase of the deep reinforcement neural network that serves as the layout determination unit 238 of the present disclosure, the deep reinforcement neural network is trained to generate a layout that leaves the most usable area in the box. Figure 7 illustrates an example of a layout that leaves the largest rectangular area in the unoccupied area of ​​the box, according to an embodiment of the present disclosure. In this disclosure, the "largest rectangular region" refers to the rectangular region that has the most area in the remaining unoccupied region of the box when the target object is placed in the unoccupied region of the box. As described above, the area of ​​this largest rectangular region is used as a reward for the deep reinforcement neural network, so that the deep reinforcement neural network is trained to generate a placement configuration that leaves the most available area.

[0067] FIG. 7 shows an arrangement state image 710 showing occupied areas 712 and unoccupied areas 714 in a box, a contour shape image 720 showing the contour shape of a target object 725, a first arrangement configuration 730, and a second arrangement configuration 740.

[0068] 7, in the first arrangement 730, the target object 725 is arranged so as not to collide with the occupied area 712 of the box, but the arrangement is incomplete, so the largest rectangular area 732 remaining in the unoccupied area 714 of the box has a relatively small area. Therefore, the space utilization efficiency of the box is low, and the arrangement of other objects is limited. Thus, if the deep reinforcement neural network generates an arrangement like the first arrangement 730, the area of ​​the largest reward rectangular region will be low.

[0069] On the other hand, in the second arrangement 740 shown in FIG. 7, the target object 725 has a relatively large area in the largest rectangular region 742 remaining in the unoccupied area 714 of the box, so that the space of the box is more efficiently utilized than in the first arrangement 730, making it easier to place other objects therein. Therefore, if the deep reinforcement neural network generates an arrangement like the second arrangement 740, the area of ​​the largest reward rectangular region will be high.

[0070] In this way, a deep reinforcement neural network is trained to generate configurations that leave the most usable area in the box.

[0071] Next, the collision determination process according to the embodiment of the present disclosure will be described with reference to FIG.

[0072] 8 is a diagram showing an example of a collision determination process according to an embodiment of the present disclosure. As described above, in the training stage of the deep reinforcement neural network that is the arrangement configuration determination unit 238 of the present disclosure, collision determination is performed based on a training box image and a training target object image. This collision determination is a process for determining whether or not the target object is placed in the occupied area of ​​the box (i.e., whether or not the target object will collide with another object already placed in the box).

[0073] An example of collision determination will be described with reference to Fig. 8. First, as described above, in the training stage, the deep reinforcement neural network randomly arranges the target object regions 815 shown in the contour shape image in the arrangement state image 810. After the target object regions 815 are arranged randomly in the arrangement state image 810, the deep reinforcement neural network determines whether the target object regions 815 and the box occupation region 820 overlap. Here, the deep reinforcement neural network may determine whether the target object region 815 and the box occupation region 820 overlap by performing a LOGIC AND operation or the like on the target object region 815 and the box occupation region 820. If the result of the LOGIC AND operation is "1", the deep reinforcement neural network determines the region where the result of the LOGIC AND operation is "1" as a collision region 825. If a collision is determined, the deep reinforcement neural network is given a reward of 0 (or another low value) and stored for use in adjusting the network parameters later. Next, returning to step S604, the deep reinforcement neural network randomly generates a new arrangement configuration and repeats the above process.

[0074] According to this collision detection process, the deep reinforcement neural network is trained to avoid configurations that result in collisions between the target object's area 815 and the box's occupied area 820, and to generate configurations that result in a rectangular area that satisfies a predetermined area criterion.

[0075] Next, predictive placement according to an embodiment of the present disclosure will be described with reference to FIG.

[0076] Generally, in consumer trading markets, multiple products that are highly related are sometimes ordered together. In this case, a logistics center that handles the transportation of products may pack multiple products ordered by a consumer in the same box and transport them. As an example, a product called "toothbrush" may be ordered together with a related product called "toothpaste," and the products may be packed and transported in the same box. Therefore, in one embodiment of the present disclosure, when a specific target object (first target object) is placed in a box, the placement configuration determination unit may predict another target object (second target object) that is highly likely to be transported together with the target object, and determine a placement configuration that takes into account the predicted second target object.

[0077] 9 is a diagram illustrating an example of predictive arrangement according to an embodiment of the present disclosure. As illustrated in FIG. 9, when a first target object 925 is arranged in a box 910, the arrangement configuration determination unit may predict a second target object that is highly likely to be packed in the box 910 together with the first target object 925. Here, the arrangement configuration determination unit may analyze, for example, arrangement history information indicating the history of objects previously arranged in a box, or sales information in a consumer trading market, calculate the probability that each of a plurality of candidate objects (for example, products that have been purchased together with the first object in the past) will be packed in the box 910 together with the first target object 925, and predict an object that satisfies a predetermined probability criterion (for example, the highest probability, a probability of 75% or more, etc.) as the second object. As an example, the configuration determination unit may predict candidate object 934 as the second object if it calculates a probability of candidate object 932 being packed into box 910 together with first target object 925 of 45%, 31% for candidate object 934, and 77% for candidate object 936.

[0078] After predicting the second target object, the arrangement configuration determination unit may generate an arrangement configuration for the first target object 925 such that a sufficient area remains for arranging the predicted second object. In this way, by predicting other objects that are likely to be arranged in the box together with a specific object and determining an arrangement configuration that takes the other objects into consideration, it is possible to further improve the space utilization efficiency of the box. In the above, an example has been described in which a second object to be packed into a box together with a first target object is calculated based on placement history information indicating the history of objects that have been placed in a box in the past and sales information from a consumer trading market. However, the present disclosure is not limited to this, and any information may be used as long as it is an index indicating the probability that a second object will be packed into a box together with a first target object.

[0079] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]

[0080] 210 boxes 220 Target Object 230 Packing Support Device 232 Image Acquisition Unit 234 Object Analysis Department 236 Box Analysis Department 238 Layout configuration determination section 240 Communication Network 250 Robot

Claims

1. A box packing assist device that determines a recommended arrangement for packing an object of any shape into a box, comprising: an image acquisition unit that acquires a target object image showing a target object to be packed into a box and a box image showing the box; an object analyzer that analyzes the target object image, extracts a contour shape of the target object, and generates a contour shape image showing the contour shape of the target object; a box analysis unit that analyzes the box image, distinguishes between occupied areas of the box that are occupied by other objects and unoccupied areas in which objects can be placed, and generates an arrangement state image showing the occupied areas and the unoccupied areas of the box; a placement configuration determining unit that determines a placement configuration indicating a position and angle recommended for placing the target object in the unoccupied area of ​​the box based on the contour shape image and the placement state image, and outputs placement configuration information indicating the placement configuration; The arrangement configuration determination unit is When placing the target object in the unoccupied area of ​​the box, determining, from among the plurality of candidate objects, a third target object whose probability of being placed in the box together with the target object satisfies a predetermined probability criterion, using placement history information indicating a frequency with which the plurality of candidate objects are placed in the box together with the target object; determining a configuration for the target objects that leaves a sufficient area for placing the third target object in the unoccupied area of ​​the box; A packing support device characterized by the above.

2. The arrangement configuration determination unit is Includes deep reinforcing neural networks, The deep reinforcement neural network comprises: A training contour shape image showing a contour shape of a second target object and a training arrangement state image showing an occupied area and an unoccupied area in the second box are input; randomly generating a second arrangement configuration for arranging the second target object in the second box based on the training contour shape image and the training arrangement state image; determining whether the second target object overlaps the occupied area of ​​the second box in the second arrangement; When it is determined that the second target object does not overlap the occupied region of the second box in the second arrangement configuration, training the deep reinforcement neural network by adjusting network parameters of the deep reinforcement neural network using as a reward the area of ​​a maximum rectangular region that is left when the second target object is arranged in the unoccupied region of the second box.

2. The packing assist device according to claim 1, wherein:

3. The object analysis unit is Selected from deep neural networks for object detection, depth cameras, The box analysis unit includes: Deep neural networks for image segmentation,selected from depth cameras, 2. The packing assist device according to claim 1, wherein:

4. The arrangement state image is a binary matrix that represents the unoccupied region as a first value and the occupied region as a second value; 2. The packing assist device according to claim 1, wherein:

5. A method for assisting packing in a box, comprising the steps of: determining a recommended arrangement for packing an object of any shape into a box, the method comprising: acquiring a first object image showing a first object to be packed into a first box and a first box image showing the first box; analyzing the first object image, extracting a contour shape of the first object, and generating a training contour shape image showing the contour shape of the first object; analyzing the first box image to distinguish between occupied areas of the first box that are occupied by other objects and unoccupied areas in which objects can be placed; generating a training layout image showing the occupied and unoccupied areas in the first box; inputting the training contour shape image and the training layout state image into a deep reinforcement neural network; randomly generating a first arrangement configuration for arranging the first target object in the first box based on the training outline shape image and the training arrangement state image using the deep reinforcement neural network; determining whether the first target object overlaps the occupied area of ​​the first box in the first arrangement; training the deep reinforcement neural network by adjusting network parameters of the deep reinforcement neural network using, as a reward, an area of ​​a maximum rectangular area remaining when the first target object is placed in the unoccupied area of ​​the first box, when it is determined that the first target object does not overlap the occupied area of ​​the first box in the first arrangement configuration, thereby constructing a trained deep reinforcement neural network; acquiring a second object image showing a second object to be packed into a second box and a second box image showing the second box; analyzing the second object image, extracting a contour shape of the second object, and generating a second contour shape image showing the contour shape of the second object; analyzing the second box image to distinguish between occupied areas of the second box that are occupied by other objects and unoccupied areas in which objects can be placed; generating a second configuration image showing the occupied and unoccupied areas of the second box; determining a second arrangement configuration indicating a recommended position and angle for placing the second target object in the unoccupied area of ​​the second box based on the second outline shape image and the second arrangement state image; and outputting arrangement configuration information indicating the second arrangement configuration, The step of determining the second arrangement configuration includes: When the second target object is placed in the unoccupied area of ​​the second box, determining, from among the plurality of candidate objects, a third object whose probability of being placed in the second box together with the second object satisfies a predetermined probability criterion, using placement history information indicating a frequency with which the plurality of candidate objects are placed in the second box together with the second object; determining an arrangement for the second target object that leaves a sufficient area for placing the third target object in the unoccupied area of ​​the second box; A packing support method comprising:

6. A box packing support system that determines a recommended arrangement for packing an object of any shape into a box, comprising: A box packing assistance device that generates a layout configuration for packing a target object into a box; a robot that performs a packing operation according to the arrangement; The boxing support device includes: an image acquisition unit that acquires a target object image showing the target object to be packed into the box and a box image showing the box; an object analyzer that analyzes the target object image, extracts a contour shape of the target object, and generates a contour shape image showing the contour shape of the target object; a box analysis unit that analyzes the box image, distinguishes between occupied areas of the box that are occupied by other objects and unoccupied areas in which objects can be placed, and generates an arrangement state image showing the occupied areas and the unoccupied areas of the box; a placement configuration determining unit that determines a placement configuration indicating a recommended position and angle for placing the target object in the unoccupied area of ​​the box based on the contour shape image and the placement state image, and outputs placement configuration information indicating the placement configuration to the robot; The arrangement configuration determination unit is When placing the target object in the unoccupied area of ​​the box, determining, from among the plurality of candidate objects, a third target object whose probability of being placed in the box together with the target object satisfies a predetermined probability criterion, using placement history information indicating a frequency with which the plurality of candidate objects are placed in the box together with the target object; determining a configuration for the target objects that leaves a sufficient area for placing the third target object in the unoccupied area of ​​the box; The robot comprises: placing the target object in the box according to the placement configuration information; A packing support system.

Images