Storehouse model generation device, and storehouse model generation method

Abstract

To efficiently generate a storehouse model in which an arrangement state of articles in a storehouse is reproduced by a 3D model even when a variety of types of articles are arranged in a wide storehouse by efficiently acquiring a 3D model of the articles supposed to be installed in the storehouse.SOLUTION: In a storehouse model generation device 5: point group data of objective installation articles is acquired; an installation article model being a 3D model of the installation article is generated and registered to a database on the basis of the point group data of the installation articles; an installation article detection engine is constructed by learning while using the point group data of the installation articles; installation articles are detected from the point group data in the storehouse by using the installation article detection engine; installation article information regarding an actual arrangement state of the installation articles in the storehouse is acquired; and a storehouse model in which a layout is changeable is generated by arranging the installation article model in the three-dimensional space representing inside of the storehouse on the basis of the installation article information so as to correspond to an actual installation state of installation articles in the storehouse.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a warehouse model generation device and a warehouse model generation method that generate a warehouse model that reproduces the arrangement of objects in a warehouse as a 3D model based on point cloud data acquired by 3D measurement of the warehouse. [Background technology]

[0002] Reviewing the layout of a warehouse can improve the efficiency of warehouse operations. In this case, using a warehouse model that reproduces the layout of objects such as shelves in the warehouse as a 3D model can help to efficiently review the warehouse layout.

[0003] A known technology related to 3D models targeting such warehouses involves performing 3D sensing of the target location to obtain 3D information (point cloud data) of the target location, detecting objects placed in the target location from the 3D information of the target location, and placing 3D models of the detected objects in the 3D space of the target location so as to correspond to the actual placement of the objects in the target location (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7226534 Summary of the Invention [Problem to be solved by the invention]

[0005] According to conventional technology, 3D models of objects expected to be placed in a warehouse are created in advance and registered in a database, so that 3D models of objects detected from point cloud data of the target location can be obtained from the database.

[0006] However, creating a 3D model of an object requires manual work using a CG editing application while referring to a blueprint of the object, and creating 3D models of the required types of objects and registering them in a database requires a great deal of effort. In particular, when a large warehouse is filled with a variety of types of objects arranged in a variety of states, the number of types of objects to be registered in the database increases, creating a problem of a significant increase in the amount of work.

[0007] Therefore, the main object of the present invention is to provide a warehouse model generation device and a warehouse model generation method that can efficiently obtain 3D models of objects that are expected to be installed in a warehouse, and can efficiently generate a warehouse model that reproduces the arrangement of objects in the warehouse in 3D models, even when various types of objects are arranged in various states within a large warehouse. [Means for solving the problem]

[0008] The warehouse model generation device of the present invention is a warehouse model generation device that uses a processor to execute a process of generating a warehouse model that reproduces the arrangement of installed objects within a warehouse as a 3D model based on point cloud data acquired by 3D measurement of the warehouse. The processor acquires point cloud data of the target installed object, generates an installed object model that is a 3D model of the installed object based on the point cloud data of the installed object, stores the installed object model in a memory unit, constructs an installed object detection engine by learning using the point cloud data of the installed object, detects installed objects from the point cloud data within the warehouse using the installed object detection engine, acquires installed object information regarding the actual arrangement of the installed objects within the warehouse, and places the installed object model in a 3D space representing the interior of the warehouse based on the installed object information so as to correspond to the actual arrangement of the installed objects within the warehouse, thereby generating the warehouse model whose layout can be changed.

[0009] In addition, the warehouse model generation method of the present invention is a warehouse model generation method that causes a processor to execute a process of generating a warehouse model that reproduces the arrangement of installed objects within a warehouse as a 3D model based on point cloud data obtained by 3D measurement of the warehouse, and is configured to acquire point cloud data of the target installation, generate an installation model that is a 3D model of the installation based on the point cloud data of the installation, store the installation model in a memory unit, construct an installation detection engine by learning using the point cloud data of the installation, detect installations from the point cloud data within the warehouse using the installation detection engine, obtain installation information regarding the actual arrangement of the installations within the warehouse, and place the installation model in a 3D space representing the interior of the warehouse based on the installation information so as to correspond to the actual installation status of the installations within the warehouse, thereby generating a warehouse model whose layout can be changed. [Effects of the Invention]

[0010] According to the present invention, a 3D model of an installed object (installed object model) can be obtained by converting point cloud data of the installed object into a 3D model. This makes it possible to efficiently obtain 3D models of the installed object and efficiently generate a warehouse model even when various types of installed objects are installed in various states in a large warehouse. [Brief explanation of the drawings]

[0011] [Figure 1] Overall configuration diagram of a warehouse model generation system according to this embodiment [Figure 2] FIG. 1 is an explanatory diagram illustrating an overview of an installation model generation process performed by an installation model generation device. [Figure 3] An explanatory diagram showing an overview of the installation point cloud correction process performed by the warehouse model generation device. [Figure 4] A block diagram showing the schematic configuration of an installation model generation device, a warehouse point cloud generation device, and a warehouse model generation device. [Figure 5] A block diagram showing an overview of the processes performed by the installation model generation device, the warehouse point cloud generation device, and the warehouse model generation device. [Figure 6] A flowchart showing the steps of a process performed when constructing an installation detection engine in a warehouse model generation device. [Figure 7] FIG. 1 is a flowchart showing the steps of a process performed when a warehouse model is generated in a warehouse model generation device. [Figure 8] FIG. 10 is an explanatory diagram showing a first example of a warehouse model creation screen displayed on a display device of the warehouse model generation device; [Figure 9] FIG. 10 is an explanatory diagram showing a first example of a warehouse model creation screen displayed on a display device of the warehouse model generation device; [Figure 10] FIG. 10 is an explanatory diagram showing a first example of a warehouse model editing screen displayed on a display device of the warehouse model generating device; [Figure 11] FIG. 10 is an explanatory diagram showing a first example of a warehouse model editing screen displayed on a display device of the warehouse model generating device; [Figure 12] FIG. 10 is an explanatory diagram showing a second example of a warehouse model creation screen displayed on a display device of the warehouse model generation device; [Figure 13] FIG. 10 is an explanatory diagram showing a second example of a warehouse model editing screen displayed on a display device of the warehouse model generation device; [Figure 14] FIG. 10 is an explanatory diagram showing a second example of a warehouse model editing screen displayed on a display device of the warehouse model generation device; [Figure 15] A block diagram showing an overview of the processing performed by a warehouse point cloud generation device and a warehouse model generation device according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0012] The first invention made to solve the above problem is a warehouse model generation device that uses a processor to execute a process to generate a warehouse model that reproduces the arrangement of installations within a warehouse as a 3D model based on point cloud data acquired by 3D measurement of the warehouse. The processor acquires point cloud data of the target installation, generates an installation model that is a 3D model of the installation based on the point cloud data of the installation, stores the installation model in a memory unit, constructs an installation detection engine by learning using the point cloud data of the installation, uses the installation detection engine to detect installations from the point cloud data within the warehouse and acquires installation information regarding the actual arrangement of the installations within the warehouse, and based on the installation information, places the installation model in a 3D space representing the interior of the warehouse so as to correspond to the actual installation status of the installations within the warehouse, thereby generating the warehouse model whose layout can be changed.

[0013] According to this, by converting point cloud data of the installed objects into a 3D model, a 3D model of the installed object (installed object model) can be obtained. This makes it possible to efficiently obtain 3D models of the installed objects and efficiently generate a warehouse model even when various types of installed objects are installed in various states in a large warehouse. Note that the installed object information may include attribute information regarding the type and size of the detected installed object, and placement information regarding the orientation and position of the installed object.

[0014] In a second aspect of the present invention, the processor acquires point cloud data of an object installed in a warehouse, the point cloud data being generated by three-dimensional measurement of the object.

[0015] This reduces the effort required for 3D measurement of the installation object compared to when the target installation object is installed separately in a location other than the warehouse and 3D measurement is performed.

[0016] In a third aspect of the present invention, the processor converts a three-dimensional model of an installation into point cloud data to acquire the point cloud data of the installation.

[0017] According to this, if a 3D model of an installed object already exists, point cloud data of the installed object can be generated from that 3D model and used as learning data for building an installed object detection engine, thereby eliminating the need to measure the installed object in 3D. The 3D model of the installed object can also be used to generate a warehouse model.

[0018] In a fourth aspect of the present invention, the processor generates a three-dimensional model that is a basic unit of a target installation as the installation model.

[0019] According to this, when installation items are arranged in a stacked state and side by side in a warehouse, the installation item models that are the basic units of the installation items can be repeatedly arranged in the warehouse model, thereby reproducing the arrangement of the installation items in the warehouse in a 3D model.

[0020] In a fifth aspect of the present invention, the processor generates a mesh model as the installation object model.

[0021] This makes it possible to generate a warehouse model that is easy to view and edit.

[0022] In addition, a sixth invention is configured such that the processor displays an editing screen including an editing menu related to the warehouse model on a display device, and changes the placement status of the installation model in the warehouse model in accordance with user operations on the editing screen.

[0023] This allows the user to easily review the layout of the warehouse. In this case, it is preferable that the editing screen has an editing menu that includes options for deleting, duplicating, and moving the installation model.

[0024] In addition, a seventh invention is a warehouse model generation method that causes a processor to execute a process of generating a warehouse model that reproduces the arrangement of installed objects within a warehouse as a 3D model based on point cloud data obtained by 3D measurement of the warehouse, the method comprising: obtaining point cloud data of a target installed object; generating an installed object model that is a 3D model of the installed object based on the point cloud data of the installed object; storing the installed object model in a memory unit; constructing an installed object detection engine by learning using the point cloud data of the installed object; detecting installed objects from the point cloud data within the warehouse using the installed object detection engine; obtaining installed object information regarding the actual arrangement of the installed objects within the warehouse; and arranging the installed object model in a 3D space representing the interior of the warehouse based on the installed object information so as to correspond to the actual installation status of the installed objects within the warehouse, thereby generating a warehouse model whose layout can be changed.

[0025] According to this, as in the first invention, by converting point cloud data of the installed objects into a 3D model, a 3D model of the installed objects (installed object model) can be obtained. As a result, the 3D models of the installed objects can be obtained efficiently, and a warehouse model can be efficiently generated even when various types of installed objects are installed in various states in a large warehouse.

[0026] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0027] FIG. 1 is a diagram showing the overall configuration of a warehouse model generation system according to this embodiment.

[0028] The warehouse model generation system generates a warehouse model that reproduces the layout of installed objects in a warehouse as a 3D model (three-dimensional model) based on point cloud data acquired by 3D sensing (three-dimensional measurement) within the warehouse. The warehouse model generation system includes a first LiDAR sensor 1, a second LiDAR sensor 2, an installed object model generation device 3, a warehouse point cloud generation device 4, and a warehouse model generation device 5.

[0029] The first LiDAR sensor 1 is used for 3D sensing of installed objects to detect the target installed objects. The second LiDAR sensor 2 is used for 3D sensing of the inside of a warehouse to detect the target objects inside the warehouse. The first LiDAR sensor 1 is configured to be smaller and easier to carry than the second LiDAR sensor 2.

[0030] The installation model generation device 3 can be configured as a PC. The installation model generation device 3 generates and outputs point cloud data of the installation by 3D sensing of the installation using the first LiDAR sensor 1. The installation model generation device 3 also converts the point cloud data of the installation into a 3D model and outputs the installation model (3D model of the installation).

[0031] The warehouse point cloud generation device 4 can be configured as a PC. The warehouse point cloud generation device 4 generates and outputs point cloud data of the warehouse interior by 3D sensing of the interior of the warehouse using the second LiDAR sensor 2.

[0032] The warehouse model generation device 5 can be configured as a PC. The warehouse model generation device 5 uses an installation detection engine to detect installations in the warehouse from point cloud data of the warehouse, and based on the detection results, arranges installation models in a three-dimensional space representing the interior of the warehouse, thereby generating a warehouse model that reproduces the arrangement of the installations in the warehouse as a three-dimensional model. The installation detection engine is constructed by machine learning using the point cloud data of the installations generated by the installation model generation device 3 as training data.

[0033] The generated warehouse model should preferably include 3D models (installation models) of all the objects installed in the warehouse that are subject to layout review. Therefore, the target installations are not limited to fixtures such as shelves for storing items such as cardboard boxes.

[0034] Furthermore, as long as the data represents 3D space, the data used for machine learning (learning and inference) does not have to be point cloud data.

[0035] In addition, in this embodiment, LiDAR sensors 1 and 2 are used as three-dimensional sensors capable of acquiring depth information, but a depth camera (stereo camera) that detects infrared light to capture an image of a subject, or a laser scanner that measures the distance to the subject by irradiating the subject with a laser, may be used instead of the LiDAR sensors 1 and 2. Furthermore, the LiDAR sensors 1 and 2 may be configured as separate sensors, or may be configured as a shared sensor that performs 3D sensing of the installed objects and the warehouse interior.

[0036] Next, a description will be given of the installation model generation process performed by the installation model generation device 3. Fig. 2 is an explanatory diagram showing an overview of the installation model generation process.

[0037] An installation model (3D model of the installation) is generated in the installation model generation device 3. First, as shown in Fig. 2(A), point cloud data of the installation is generated by 3D sensing using the first LiDAR sensor 1. Next, as shown in Fig. 2(B), the point cloud data of the installation is converted into a 3D model to generate the installation model.

[0038] In this case, 3D sensing may be performed on an object actually placed in the warehouse, or the object may be installed separately in a location other than the warehouse and 3D sensing may be performed.

[0039] In particular, when 3D sensing is performed on installations actually placed in a warehouse, the generated point cloud data includes point clouds other than the target installation. Therefore, it is preferable to perform an operation to edit the generated point cloud data so that an appropriate installation model is generated when the point cloud data is converted into a 3D model. For example, the point cloud of the target installation may be extracted from the generated point cloud data by a user's visual operation. Also, unnecessary point clouds may be deleted from the generated point cloud data.

[0040] Next, we will explain the outline of the installation point cloud adjustment process performed by the warehouse model generation device 5. Fig. 3 is an explanatory diagram showing the outline of the installation point cloud adjustment process.

[0041] The actual installations in the warehouse may contain stored items such as cardboard boxes. In this case, when 3D sensing is performed on the installations, the point cloud of the stored items is included in the point cloud of the installations that is generated.

[0042] Therefore, in this embodiment, the warehouse model generation device 5 detects and removes point clouds representing other objects from the point cloud of installed objects, and also performs processing to repair missing parts caused by partial removal of the point cloud (installed object point cloud correction processing). This improves the accuracy of detecting installed objects in the warehouse from point cloud data within the warehouse using the installed object detection engine.

[0043] In the example shown in Figure 3(A), a cardboard box (a stored item) is placed on a shelf (an installed object). In this case, an object recognition engine built using machine learning such as deep learning is first used to detect the cardboard box from the point cloud of the shelf. Next, as shown in Figure 3(B), the point cloud of the detected cardboard box is removed from the point cloud of the shelf. Next, as shown in Figure 3(C), the missing part resulting from the partial removal of the point cloud is complemented and repaired by referring to information from the surrounding point clouds.

[0044] Incidentally, there are cases where it is desirable to reproduce the current state of a warehouse as a 3D model. In this case, to detect stored items (such as cardboard boxes) along with installed items, it is preferable to use an object detection engine built by machine learning using point cloud data of the installed items as learning data. It is also preferable to register the 3D model of the stored items together with the 3D model of the installed items (installed item model) in a database. As a result, the stored items are recognized and the point cloud of the stored items is replaced with the 3D model of the stored items, and a warehouse model including the 3D model of the stored items is generated together with the 3D model of the installed items. In this warehouse model, the stored items contained in the installed items are also reproduced together with the installed items.

[0045] In this embodiment, the warehouse model generation device 5 performs an installation point cloud correction process on the point cloud data of the installations acquired from the installation model generation device 3, but the installation model generation device 3 may also perform an installation point cloud correction process on the point cloud data of the installations generated in the installation point cloud generation process.

[0046] Next, we will explain the schematic configurations of the installation object model generating device 3, the warehouse point cloud generating device 4, and the warehouse model generating device 5. Fig. 4 is a block diagram showing the schematic configurations of the installation object model generating device 3, the warehouse point cloud generating device 4, and the warehouse model generating device 5. Fig. 5 is a block diagram showing an overview of the processes performed by the installation object model generating device 3, the warehouse point cloud generating device 4, and the warehouse model generating device 5.

[0047] As shown in FIG. 4, the installation object model generation device 3 includes an input / output unit 11, a communication unit 12, a display device 13, an input device 14, a storage unit 15, and a processor 16.

[0048] The input / output unit 11 inputs and outputs data to and from the first LiDAR sensor 1.

[0049] The communication unit 12 communicates with the warehouse model generation device 5.

[0050] The display device 13 displays a screen on which the user can instruct the processor 16 to execute various processes and on which the results of the processes are displayed. The input device 14 is configured by a mouse, a touch panel, or the like, and detects user operations.

[0051] The storage unit 15 stores programs executed by the processor 16 and the like.

[0052] The processor 16 performs various processes by executing programs stored in the storage unit 15. In this embodiment, as shown in Fig. 5, the processor 16 performs an installation object point cloud generation process P11, an installation object model generation process P12, and the like.

[0053] In the installed object point cloud generation process P11, the processor 16 generates point cloud data of the installed object based on the detection result of the first LiDAR sensor 1.

[0054] In the installation model generation process P12, the processor 16 converts the point cloud data of the installation into a 3D model to generate an installation model (3D model of the installation). Specifically, the point cloud data of the installation is converted into a mesh model in, for example, a USD (Universal Scene Description) format.

[0055] The installation model generating device 3 may execute a process of shaping the installation model in response to a user's editing operation on the installation model generated from the point cloud data. This allows the user to edit the installation model to an appropriate state as a basic unit when reviewing the warehouse layout.

[0056] As shown in FIG. 4, the warehouse point cloud generation device 4 includes an input / output unit 21, a communication unit 22, a display device 23, an input device 24, a storage unit 25, and a processor 26.

[0057] The input / output unit 21 inputs and outputs data to and from the second LiDAR sensor 2.

[0058] The communication unit 22 communicates with the warehouse model generation device 5.

[0059] The display device 23 displays a screen on which the user can instruct the processor 26 to execute various processes and on which the processing results are displayed. The input device 24 is configured by a mouse, a touch panel, or the like, and detects user operations.

[0060] The storage unit 25 stores programs executed by the processor 26 and the like.

[0061] The processor 26 performs various processes by executing programs stored in the storage unit 25. In this embodiment, as shown in Fig. 5, the processor 26 performs a warehouse inside point cloud generation process P21 and the like.

[0062] In the warehouse interior point cloud generation process P21, the processor 26 generates point cloud data within the warehouse based on the detection result of the second LiDAR sensor 2.

[0063] As shown in FIG. 4, the warehouse model generation device 5 includes a communication unit 31, a display device 32 (display device), an input device 33, a storage unit 34, and a processor 35.

[0064] The communication unit 31 communicates with the installation model generation device 3 and the warehouse point cloud generation device 4.

[0065] The display device 32 displays a screen on which the user can instruct the processor 35 to execute various processes and on which the processing results are displayed. The input device 33 is configured by a mouse, a touch panel, or the like, and detects user operations.

[0066] The storage unit 34 stores programs and the like to be executed by the processor 35. The storage unit 34 also stores registration information of the installation model database, that is, an installation model (3D model of the installation) for each installation that is expected to be installed in the warehouse.

[0067] The processor 35 performs various processes by executing programs stored in the storage unit 34. In this embodiment, as shown in Fig. 5, the processor 35 performs an installed object point cloud modification process P31, an installed object detection engine construction process P32, an in-warehouse point cloud modification process P33, an installed object detection process P34, an installed object model placement process P35, and a warehouse model editing process P36.

[0068] In the installation point cloud correction process P31, the processor 35 detects and removes unnecessary point clouds from the point cloud data of the installation, and repairs missing parts caused by the partial removal of the point cloud. Specifically, the processor 35 uses an object recognition engine built using machine learning such as deep learning to detect other objects from the point cloud data of the installation. Next, the processor 35 removes the point cloud of the detected other object from the point cloud of the installation. Next, the processor 35 complements and repairs the missing parts caused by the partial removal of the point cloud by referring to information on the surrounding point clouds.

[0069] In the installation object detection engine construction process P32 (learning process), the processor 35 constructs an installation object detection engine that detects installation objects in the warehouse from the point cloud data in the warehouse by machine learning such as deep learning. At this time, machine learning is performed using the point cloud data of the installation objects as learning data.

[0070] In the warehouse point cloud correction process P33, the processor 35 detects and removes unnecessary point clouds from the point cloud data inside the warehouse, and repairs missing parts caused by removing partial point clouds. The contents of the warehouse point cloud correction process are the same as those of the installed object point cloud correction process P31.

[0071] In the installation object detection process P34, the processor 35 uses an installation object detection engine to detect installation objects located within the warehouse from the point cloud data within the warehouse and obtains installation object information relating to the actual placement of the installation objects within the warehouse as detection results. At this time, the processor 35 recognizes the type and size of the detected installation object. The processor 35 also sets a bounding box for the detected installation object and obtains the position and orientation of the detected installation object based on the bounding box. The installation object information includes attribute information relating to the type and size of the detected installation object and placement information relating to the orientation and position of the installation object.

[0072] In the installation model placement process P35, the processor 35 selects an installation model corresponding to the detected installation from the installation model database based on the installation information, and places the installation model in the three-dimensional space representing the inside of the warehouse so as to correspond to the actual installation status of the installation in the warehouse. In this way, a warehouse model (3D model of the inside of the warehouse) whose layout can be changed is generated.

[0073] In the warehouse model editing process P36, the processor 35 changes the arrangement of the installation models in the warehouse model generated in the installation model arrangement process P35 in response to a user operation. At this time, the processor 35 displays a warehouse model editing screen 71 (see FIGS. 10 and 11) on the display device 32, and accepts editing operations (deletion, duplication, and movement) by the user on the installation models included in the warehouse model.

[0074] Next, a description will be given of the procedure of the process carried out when constructing an installed object detection engine in the warehouse model generation device 5. Fig. 6 is a flowchart showing the procedure of the process carried out when constructing an installed object detection engine.

[0075] First, the processor 35 acquires point cloud data of the installation received from the installation model generation device 3 (ST101).

[0076] Next, processor 35 determines whether to perform a process of removing point clouds representing objects other than the installed object from the point cloud data of the installed object (installed object point cloud modification process) (ST102). This determination may be made based on the setting information or on a response to a query made to the user.

[0077] Here, if an object other than the installed object is to be removed (Yes in ST102), processor 35 detects and removes a point cloud representing an object other than the installed object from the point cloud of the installed object, and repairs the missing part caused by the partial removal of the point cloud (installed object point cloud correction processing) (ST103).

[0078] Next, the processor 35 constructs an installation object detection engine that detects installation objects in the warehouse from the point cloud data in the warehouse by machine learning using the point cloud data of the installation objects as learning data (installation object detection engine construction process) (ST104).

[0079] Next, a description will be given of the procedure of the process carried out when generating a warehouse model in the warehouse model generating device 5. Fig. 7 is a flow diagram showing the procedure of the process carried out when generating a warehouse model.

[0080] First, the processor 35 displays the warehouse model creation screen 51 (see Figures 8 and 9) on the display device 32, and in response to user operation, acquires the point cloud data within the warehouse received from the warehouse point cloud generation device 4 and stored in the memory unit 34 (ST201).

[0081] Next, a loop is entered in which installations within the warehouse are detected from the point cloud data within the warehouse, and an installation model corresponding to that installation is placed in the 3D space representing the interior of the warehouse, and this process is repeated for each installation (ST202 to ST206).

[0082] First, the processor 35 uses an installation object detection engine to detect installation objects in the warehouse from the point cloud data in the warehouse, and acquires installation object information relating to the actual placement of the installation objects in the warehouse (installation object detection process) (ST203). The installation object information includes attribute information relating to the type and size of the detected installation object, and placement information relating to the position and orientation of the installation object.

[0083] Before the installation object detection process, unnecessary point clouds are detected and removed from the point cloud within the warehouse, and a process (warehouse point cloud correction process) is performed to repair missing parts caused by the partial removal of the point cloud.

[0084] Next, processor 35 displays bounding boxes set for the detected installations (shelves, etc.) as detection results of the installation detection process on warehouse model creation screen 51 (see FIG. 12) superimposed on the point cloud image, and also displays installation model candidates corresponding to the detected installations (ST204), which allows the user to confirm whether the installation detection process has been executed appropriately.

[0085] Next, based on the installation information, processor 35 selects an installation model corresponding to the detected installation from the installation model database, and places the installation model in the 3D space representing the warehouse so as to correspond to the actual installation status of the installation within the warehouse (installation model placement process) (ST205).

[0086] The processes of ST203 to ST205 are repeated the same number of times as the number of installations detected from the point cloud data in the warehouse. When the processes of ST203 to ST205 have been performed for all the detected installations, a warehouse model is generated that includes installation object models corresponding to all the detected installations.

[0087] Next, processor 35 determines whether the user has instructed editing to change the layout of the warehouse model (ST207). If the user has instructed editing to change the layout (Yes in ST207), processor 35 displays warehouse model editing screen 71 (see FIGS. 10 and 11) on display device 32 and accepts editing operations such as deleting, duplicating, and moving the installation models included in the warehouse model (warehouse model editing process) (ST208). This allows the user to easily review the layout within the warehouse.

[0088] Next, a description will be given of a first example of the warehouse model creation screen 51 displayed on the display device 32 of the warehouse model generating device 5. Figures 8 and 9 are explanatory diagrams showing the first example of the warehouse model creation screen 51.

[0089] The warehouse model creation screen 51 is provided with an operation section 52, an image display section 53, and an installation model display section .

[0090] As shown in FIG. 8(A), the operation unit 52 is provided with a "Load point cloud" button 55. When the user operates the "Load point cloud" button 55, the processor 35 reads out the point cloud data of the warehouse interior stored in the storage unit 34. At this time, as shown in FIG. 8(B), the image display unit 53 displays a point cloud image 58, which is an image of the point cloud data of the warehouse interior read out from the storage unit 34 based on a specified viewpoint. Note that the operator can change the viewpoint to his or her preference by performing a predetermined operation on the point cloud image 58 on the image display unit 53.

[0091] Furthermore, the operation unit 52 is provided with a "model conversion" button 56. When the user operates the "model conversion" button 56, the processor 35 converts the point cloud data of the interior of the warehouse into a 3D model to generate a warehouse model (3D model of the interior of the warehouse). That is, installed objects in the warehouse are detected from the point cloud data of the interior of the warehouse (installed object detection process), and based on the detection results, installed object models are arranged in the 3D space representing the interior of the warehouse so as to correspond to the actual installation status of the installed objects in the warehouse (installed object model arrangement process), thereby generating the warehouse model.

[0092] At this time, as shown in Fig. 9, a model image 61, which is an image of the generated warehouse model based on a specified viewpoint, is displayed on the image display unit 53. Also, a model image 62, which is an image of a candidate installation model (3D model of the installation) corresponding to the installation detected in the installation detection process based on a specified viewpoint, is displayed on the installation model display unit 54. This allows the user to check whether the detection result of the installation detection process is appropriate. Note that the example shown in Fig. 9 shows a case where one type of installation is detected, and if multiple types of installations are detected, candidate installation models for each type of installation are displayed.

[0093] The operation unit 52 is also provided with a "change layout" button 57. When the user operates the "change layout" button 57, the screen transitions to a warehouse model editing screen 71 (see FIG. 10).

[0094] Next, a description will be given of a first example of the warehouse model editing screen 71 displayed on the display device 32 of the warehouse model generating device 5. Figures 10 and 11 are explanatory diagrams showing the first example of the warehouse model editing screen 71.

[0095] On the warehouse model editing screen 71, the user can select an installation model (3D model of an installation) included in the warehouse model using the input device 33, and thereby delete the installation model, duplicate the installation model, or move the installation model to any position. This allows the user to consider the layout of the installations in the warehouse.

[0096] On the warehouse model editing screen 71, the operation unit 52 is provided with a "Cut" button 72.

[0097] The example shown in FIG. 10 is a case where an installation model is to be deleted. In this case, first, as shown in FIG. 10(A), the user performs an operation to select the installation model to be deleted. At this time, the installation model changes to a selected state on the image display unit 53, and an image 73 indicating the selected state of the installation model is displayed. Note that by repeating the operation to select the installation model, multiple installation models can be selected. Next, as shown in FIG. 10(B), the user operates the "Cut" button 72. As a result, the selected installation model is deleted from the warehouse model.

[0098] The example shown in Fig. 11 is a case where an installation model is moved. In this case, first, as shown in Fig. 11(A), the user performs an operation to select the installation model to be moved. Next, as shown in Fig. 11(B), the user performs an operation to move the installation model (drag-and-drop operation). As a result, the selected installation model is moved to the required position.

[0099] Next, a description will be given of a second example of the warehouse model creation screen 51 displayed on the display device 32 of the warehouse model generating device 5. Fig. 12 is an explanatory diagram showing the second example of the warehouse model creation screen 51.

[0100] The warehouse model creation screen 51 in this example is displayed when the user operates the "model conversion" button 56 on the warehouse model creation screen 51 (see Figure 8 (B)) and a process of detecting objects installed in the warehouse from point cloud data within the warehouse (installation object detection process) is executed.

[0101] On the warehouse model creation screen 51 according to this example, an image 81 representing a bounding box set for an installation (such as a shelf) detected in the installation detection process is superimposed on the point cloud image 58. Also, on the warehouse model creation screen 51 according to this example, similar to the warehouse model creation screen 51 (see FIG. 9), a model image 62 representing a candidate installation model (3D model of the installation) corresponding to the installation detected in the installation detection process is displayed on the installation model display unit 54. This allows the user to confirm whether the installation detection process has been executed appropriately.

[0102] Next, a description will be given of a second example of the warehouse model editing screen 71 displayed on the display device 32 of the warehouse model generating device 5. Figures 13 and 14 are explanatory diagrams showing the second example of the warehouse model editing screen 71.

[0103] In the examples shown in Figures 10 and 11, the layout of the warehouse is changed by changing the placement of the installation models included in the warehouse model, but there are also cases where the placement of basic unit installation models stacked in multiple layers is changed.

[0104] The example shown in Figure 13 is a case where an installation model that is stacked in multiple layers is to be deleted. In this case, first, as shown in Figure 13(A), the user performs an operation to select the installation model to be deleted. Next, as shown in Figure 13(B), the user operates the "Cut" button 72. This causes the selected installation model to be deleted from the warehouse model.

[0105] The example shown in Fig. 14 is a case where installation models that are stacked in multiple layers are moved. In this case, first, as shown in Fig. 14(A), the user performs an operation to select the installation model to be moved. Next, as shown in Fig. 14(B), the user performs an operation to move the installation model (drag-and-drop operation). As a result, the selected installation model is moved to the required position.

[0106] 13 and 14, in addition to the "Cut" button 72, the operation unit 52 of the warehouse model editing screen 71 is provided with a "Copy" button 91 and a "Paste" button 92. When the user selects an installation model and then operates the "Copy" button 91 and the "Paste" button 92 in that order, the selected installation model is duplicated at the cursor position.

[0107] 13 and 14, it is possible to edit the installation model in a state where it is stacked in multiple layers, and such editing is possible by generating the installation model (3D model of the installation) in a state appropriate as a basic unit when reviewing the warehouse layout when generating the installation model in the installation model generation device 3. On the other hand, as in the example shown in Fig. 13 and 14, when the warehouse layout is reviewed while the installations remain stacked, the stacked installations become the basic unit, and therefore an installation model corresponding to the stacked state may be generated.

[0108] (Variation) Next, a modified example will be described. Note that points not particularly mentioned here are the same as those in the above embodiment. Fig. 15 is a block diagram showing an outline of the processing performed by the warehouse point cloud generation device 4 and warehouse model generation device 5 according to the modified example.

[0109] In some cases, ready-made installation models (3D models of installations) are available. For example, ready-made installation models may be provided by the installation manufacturer. In this modification, the ready-made installation models are registered in an installation model database and used to generate a warehouse model. Furthermore, point cloud data of the installations is generated from the ready-made installation models and used as learning data for building an installation detection engine.

[0110] The warehouse point cloud generation device 4 is similar to the embodiment (see FIG. 5), and generates point cloud data inside the warehouse by 3D sensing using the second LiDAR sensor 2.

[0111] In the warehouse model generation device 5, similarly to the embodiment (see FIG. 5), the processor 35 performs an installed object detection engine construction process P32, an in-warehouse point cloud modification process P33, an installed object detection process P34, an installed object model placement process P35, and a warehouse model editing process P36. Furthermore, in this modification, the processor 35 performs an installed object point cloud generation process P41.

[0112] In the installation point cloud generation process P41, the processor 35 acquires an installation model (a 3D model of an installation), converts the installation model into a point cloud, and generates point cloud data of the installation.

[0113] In some cases, ready-made installation models are available for only some of the multiple types of installations expected to be installed in a warehouse. In this case, for installations for which ready-made installation models are not available, installation model generation device 3 in the embodiment (see FIG. 5) generates installation models.

[0114] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0115] The warehouse model generation device and warehouse model generation method of the present invention have the effect of efficiently acquiring 3D models of objects expected to be installed in a warehouse, and efficiently generating a warehouse model that reproduces the arrangement of objects in the warehouse in a 3D model, even when a large warehouse has a wide variety of types of objects arranged in a wide variety of states.They are useful as a warehouse model generation device and warehouse model generation method that generate a warehouse model that reproduces the arrangement of objects in a warehouse in a 3D model based on point cloud data acquired by 3D measurement of the warehouse. [Explanation of symbols]

[0116] 1: First LiDAR sensor 2: Second LiDAR sensor 3: Installation model generation device 4: Warehouse point cloud generation device 5: Warehouse model generator 16: Processor 26: Processor 32: Display device 33: Input device 34: Storage part 35: Processor 51: Warehouse model creation screen 52:Operation unit 53: Image display unit 54: Installation model display section 58: Point cloud image 61: Model image 62: Model image 71: Warehouse model editing screen

Claims

1. A warehouse model generation device that uses a processor to execute a process of generating a warehouse model that reproduces the layout of objects in a warehouse as a three-dimensional model based on point cloud data acquired by three-dimensional measurement of the warehouse, The processor: Obtain point cloud data of the target installation, generating an installation model, which is a three-dimensional model of the installation, based on the point cloud data of the installation, and storing the installation model in a storage unit; constructing an installation object detection engine by learning using point cloud data of the installation object; Using the installation object detection engine, detect installation objects from point cloud data in the warehouse, and acquire installation object information relating to the actual arrangement status of the installation objects in the warehouse; A warehouse model generation device characterized by generating a warehouse model whose layout can be changed by placing the installation model in a three-dimensional space representing the interior of the warehouse based on the installation information so as to correspond to the actual installation status of the installations within the warehouse.

2. The processor: The warehouse model generation device according to claim 1, wherein point cloud data of an object installed in a warehouse is acquired by three-dimensional measurement of the object.

3. The processor: The warehouse model generation device according to claim 1, wherein the point cloud data of the installation is acquired by converting a three-dimensional model of the installation into point cloud data.

4. The processor:

2. The warehouse model generation device according to claim 1, wherein a three-dimensional model that is a basic unit of a target installation is generated as the installation model.

5. The processor: The warehouse model generation device according to claim 1, wherein a mesh model is generated as the installation model.

6. The processor: displaying an edit screen including an edit menu related to the warehouse model on a display device; The warehouse model generation device according to claim 1 , wherein the arrangement of the installation model in the warehouse model is changed in response to a user operation on the editing screen.

7. A warehouse model generation method that causes a processor to execute a process of generating a warehouse model that reproduces the arrangement of installed objects in a warehouse as a three-dimensional model based on point cloud data acquired by three-dimensional measurement of the warehouse, Obtain point cloud data of the target installation, generating an installation model, which is a three-dimensional model of the installation, based on the point cloud data of the installation, and storing the installation model in a storage unit; constructing an installation object detection engine by learning using point cloud data of the installation object; Using the installation object detection engine, detect installation objects from point cloud data in the warehouse, and acquire installation object information relating to the actual arrangement status of the installation objects in the warehouse; A warehouse model generation method characterized by generating a warehouse model whose layout can be changed by placing the installation model in a three-dimensional space representing the interior of the warehouse based on the installation information so as to correspond to the actual installation status of the installations within the warehouse.

Images