Real equipment virtualization device

Abstract

To enable reducing effort while suppressing placement discrepancies between actual equipment and a 3D model.SOLUTION: A real equipment virtualization device is provided with: an environment point cloud acquisition unit 152 that acquires an environmental 3D point cloud obtained by observing an equipment group with a sensor 20; an equipment recognition unit 102 that recognizes the equipment contained in the equipment group based on the observation results of the equipment group from the sensor 20; a point cloud extraction unit 154 that extracts 3D point clouds from equipment 3D models recognized by the equipment recognition unit 102 from the equipment 3D models included in a virtual layout; a matching processing unit 155 that matches the environmental 3D point cloud acquired by the environment point cloud acquisition unit 152 with the model 3D point cloud extracted from the 3D models by the point cloud extraction unit 154; and a model position correction unit 156 that corrects positions of the 3D models in the virtual layout based on the matching performed by the matching processing unit 155 by adjusting the discrepancy between the actual equipment placement in the equipment group and the 3D model placement in the virtual layout.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a real facility virtualization device that virtualizes a real facility. [Background technology]

[0002] Labor shortages in production sites due to a decline in the working-age population have become a social issue, creating a demand for automated systems that use robots to work in place of humans. However, it takes a great deal of effort to position robots in real space so that they can perform the desired operations while avoiding interference with surrounding structures. Therefore, when introducing robots into production sites, it is desirable to reduce the effort required for system integration. In response to this, for example, Patent Document 1 discloses a technology for constructing a virtual robot system in which a virtual three-dimensional model of a robot and a structure surrounding the robot is arranged in a virtual space, and teaching the robot a movement path. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-140958 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, even with the technology disclosed in Patent Document 1, if there is a discrepancy between the layout of a virtual three-dimensional model (hereinafter referred to as a 3D model) in a virtual space and the layout of the actual equipment, which is the actual equipment, there is a risk of requiring a lot of work to redo the teaching while operating the actual equipment to check whether the movement is actually performed as instructed. With the technology disclosed in Patent Document 1, teaching (i.e., teaching) is performed in a virtual space, which makes teaching easy, but it requires a lot of work to align the layout of the actual equipment with the layout of the 3D model so that the teaching results can be used. The work of aligning the layout of the actual equipment with the 3D model includes the work of moving the heavy actual equipment and the work of adjusting the position, layout, tilt, etc. of the actual equipment with high precision.

[0005] One object of this disclosure is to provide a real facility virtualization device that makes it possible to reduce deviations in the layout between the real facility and a 3D model while reducing the amount of work required. [Means for solving the problem]

[0006] The above object is achieved by the combination of features recited in the independent claims, and the subclaims define further advantageous embodiments of the disclosure. The reference numerals in parentheses in the claims correspond to specific means described in the following embodiment as one aspect, and do not limit the technical scope of the present disclosure.

[0007] In order to achieve the above object, the real equipment virtualization device of the present disclosure includes an environment point cloud acquisition unit (152) that acquires an environment three-dimensional point cloud, which is a three-dimensional point cloud obtained by observing a group of equipment in which multiple pieces of equipment are actually arranged using a sensor (20); an equipment recognition unit (102) that recognizes equipment included in the group of equipment based on the observation results of the group of equipment by the sensor; a point cloud extraction unit (154) that extracts a three-dimensional point cloud from a three-dimensional model of equipment recognized by the equipment recognition unit, among three-dimensional models of equipment included in a virtual layout, which is an arrangement of the group of equipment that has been generated in advance in a virtual space; a matching processing unit (155) that matches the environment three-dimensional point cloud acquired by the environment point cloud acquisition unit with a model three-dimensional point cloud, which is a three-dimensional point cloud extracted from the three-dimensional model by the point cloud extraction unit; and a model position correction unit (156) that corrects the position of the three-dimensional model in the virtual layout by the amount of deviation of the arrangement of the three-dimensional model in the virtual layout relative to the arrangement of the real equipment, which is the actual equipment in the group of equipment, based on the matching in the matching processing unit.

[0008] According to the above configuration, an environment 3D point cloud obtained by observing a facility group where multiple pieces of equipment are actually arranged using sensors is matched with a model 3D point cloud extracted from a 3D model of the equipment included in a virtual layout, which is a layout of the facility group pre-generated in a virtual space. Matching the 3D point clouds makes it easier to align the 3D model with the actual equipment in the facility group. Based on this matching, the position of the 3D model in the virtual layout is corrected by the amount of misalignment of the 3D model in the virtual layout relative to the layout of the actual equipment. This makes it easier to reduce the misalignment between the actual equipment and the 3D model. Furthermore, correcting the position of the 3D model in the virtual layout makes it easier to reduce the misalignment between the actual equipment and the 3D model than by adjusting the layout of the actual equipment. This also makes it easier to reduce the misalignment between the actual equipment and the 3D model. In addition, because a 3D point cloud is extracted from the 3D model of the equipment recognized by the equipment recognition unit based on the environment 3D point cloud acquired by the environment point cloud acquisition unit, it becomes easier to match the environment 3D point cloud to be matched with the model 3D point cloud. This also makes it easier to reduce the deviation in the placement of the actual equipment and the 3D model. As a result, it becomes possible to reduce the deviation in the placement of the actual equipment and the 3D model while reducing the amount of work required. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a CPS. [Figure 2] FIG. 1 is a diagram illustrating an example of a portion of a facility group. [Figure 3] FIG. 10 is a sequence diagram for explaining an example of an operation plan and an operation flow of a group of facilities using a CPS. [Figure 4] 10A and 10B are diagrams for explaining an example of cutting out a three-dimensional point cloud from a three-dimensional point cloud constructed by a point cloud constructor. [Figure 5] 1A and 1B are diagrams showing a schematic representation of a 3D point cloud of a 3D model before and after extraction of an observable region. [Figure 6]This is a diagram for explaining the change in the inclination of the vertical axis of the 3D model indicated by the model 3D point cloud when aligning the environmental 3D point cloud and the model 3D point cloud. [Figure 7] This is a flowchart showing an example of the flow of CPS-related processing in CPS. [Figure 8] This is a flowchart showing an example of the flow of correction processing in the actual equipment virtualization device. [Figure 9] This is a flowchart showing an example of the flow of point cloud extraction processing in the actual equipment virtualization device. [Figure 10] This is a flowchart showing an example of the flow of matching processing in the actual equipment virtualization device. [Figure 11] This is a diagram for explaining the effect of improving the matching accuracy according to this embodiment. [Figure 12] This is a diagram for explaining the effect of improving the accuracy of position correction of the 3D model according to the configuration of this embodiment. [Figure 13] This is a diagram for explaining the effect of improving the accuracy of position correction of the 3D model according to the configuration of this embodiment. [Figure 14] This is a diagram for explaining the effect of reducing the adjustment man-hours according to the configuration of this embodiment.

Embodiments for Carrying Out the Invention

[0010] While referring to the drawings, a plurality of embodiments for disclosure will be described. For the sake of convenience in explanation, among the plurality of embodiments, parts having the same functions as those shown in the drawings used in the previous explanations may be denoted by the same reference numerals, and the explanations thereof may be omitted. For the parts denoted by the same reference numerals, the explanations in other embodiments can be referred to.

[0011] (Embodiment 1) <Schematic Configuration of CPS1> A first embodiment of the present disclosure will be described below with reference to the drawings. A CPS (Cyber ​​Physical System) 1 shown in FIG. 1 recreates the layout of a group of facilities in a real space in a virtual space and performs operation planning for the group of facilities in the virtual space. The group of facilities is assumed to be an arrangement of multiple facilities. The group of facilities may be a group of facilities in the operating environment of a robot. The group of facilities may be, for example, a robot and its peripheral structures. While the group of facilities does not necessarily have to include a robot, the following description will be given assuming that the group of facilities includes a robot. Examples of facilities included in the group of facilities include a robot cell and a parts shooter. A robot cell is a robot organized into a cell. A robot cell may be, for example, a combination of a robot and a cart. The parts shooter moves parts to the next process. By arranging the robot cells and the parts shooters in the group of facilities, parts can be supplied from the parts shooter to the robot cell, and parts obtained by the work of the robot cell can be handed over to the parts shooter.

[0012] Here, an example of an equipment group and equipment will be described using Figure 2. Figure 2 is a diagram showing an example of a portion of an equipment group. RC in Figure 2 shows an example of a robot cell. PS in Figure 2 shows an example of a part shooter. The equipment refers to the robot cell RC and part shooter PS in Figure 2. An equipment group is an arrangement of multiple pieces of equipment such as the robot cell RC and part shooter PS.

[0013] As shown in FIG. 1, the CPS 1 includes a real facility virtualization device 10, a sensor 20, and an operation planning unit 30. The sensor 20 observes a group of actually installed facilities. Examples of the sensor 20 include an imaging device and a search wave sensor. The imaging device captures an image of a predetermined range starting from the imaging device itself. The search wave sensor transmits search waves within a predetermined range starting from the sensor itself. The search wave sensor obtains, as sensing information, a scan result based on a received signal obtained when the search wave is reflected by an object and receives a reflected wave. The search wave sensor observes the distance and shape of the object by transmitting and receiving the search wave. Examples of the search wave sensor include Lidar (Light Detection and Ranging / Laser Imaging Detection and Ranging). The search wave sensor may be millimeter-wave radar, sonar, etc. The sensor 20 may be an imaging device or a search wave sensor. The sensor 20 may be a combination of an imaging device and a search wave sensor. For example, Lidar may be used as the search wave sensor.

[0014] The real facility virtualization device 10 is mainly composed of a computer equipped with, for example, a processor, volatile memory, nonvolatile memory, I / O, and buses connecting these. The real facility virtualization device 10 executes a control program stored in the nonvolatile memory to perform processing related to reproducing the layout of a group of facilities in real space in a virtual space. Details of the real facility virtualization device 10 will be described later. The motion planning unit 30 teaches positions to robots in the virtual space (hereinafter referred to as "teaching"). The motion planning unit 30 also plans the motion of the group of facilities. In the motion planning of the group of facilities, motions are planned that enable the robots included in the group of facilities to perform the desired tasks while preventing interference with surrounding structures. This motion may be performed along six axes: x, y, and z axes that are perpendicular to each other, and roll, pitch, and yaw, which are the rotation angles of each of these axes. The motion planning unit 30 plans the motion based on the layout of the group of facilities (hereinafter referred to as "virtual layout") that has been generated in advance in the virtual space. When the virtual layout is corrected by the real facility virtualization device 10, the operation planning unit 30 performs an operation plan based on this corrected virtual layout.

[0015] Here, an example of an operation plan and operation flow for a group of equipment using the CPS 1 will be described using the sequence diagram in FIG. 3. First, at t1, the operation planner 30 performs an operation plan in the virtual space based on the virtual layout. At t2, actual equipment (hereinafter referred to as real equipment) is manually arranged so that it is arranged according to the virtual layout. With this manual arrangement of real equipment, it is difficult to make the group of equipment operate according to the operation plan with a single adjustment. This is because it is difficult to accurately match the arrangement of the real equipment to the virtual layout due to factors such as the inclination of the floor on which the equipment is installed and the weight of the equipment. At t2, it is sufficient to manually arrange the real equipment so that it roughly follows the virtual layout.

[0016] At t3, the real equipment virtualization device 10 corrects the virtual layout based on the observation results of the real equipment by the sensor 20. In other words, it corrects the arrangement of the equipment group in the virtual space. This correction should be made to the virtual layout by the amount of deviation of the virtual layout from the arrangement of the real equipment in the equipment group. At t4, the motion planning unit 30 modifies the motion plan in the virtual space in response to the correction of the virtual layout made at t3.

[0017] At t5, the equipment in the real space is operated in accordance with the motion plan corrected at t4. In CPS1, the virtual layout is corrected to eliminate any discrepancies with the actual equipment layout, making it easier to reduce the discrepancy between the virtual layout and the actual equipment layout than by adjusting the actual equipment layout. Furthermore, by reducing the discrepancy between the virtual layout and the actual equipment layout, it becomes possible for the equipment in the real space to perform operations in accordance with the motion plan in the virtual space. Therefore, it becomes possible to perform operations in the real space that are in accordance with the motion plan in the virtual space with less effort, without having to repeatedly adjust the layout of the actual equipment.

[0018] <Schematic configuration of real facility virtualization device 10> Next, the schematic configuration of the real equipment virtualization device 10 will be described with reference to Fig. 1. As shown in Fig. 1, the real equipment virtualization device 10 includes, as functional blocks, a point cloud construction unit 101, an equipment recognition unit 102, a layout database (hereinafter referred to as DB) 103, a model DB 104, and an equipment identification unit 105. Note that some or all of the functions executed by the real equipment virtualization device 10 may be configured as hardware using one or more ICs or the like. Furthermore, some or all of the functional blocks included in the real equipment virtualization device 10 may be realized by a combination of software execution by a processor and hardware components.

[0019] The point cloud construction unit 101 constructs a 3D point cloud from the observation results of the sensor 20. If the sensor 20 is an imaging device, the 3D point cloud may be constructed from the captured images, for example, by SfM (Structure From Motion). In this case, the imaging device may capture multiple images of the target equipment group by changing the imaging direction. If the sensor 20 is a survey wave sensor, the 3D point cloud may be constructed from the point cloud obtained by positioning. The point cloud construction unit 101 may construct a 3D point cloud covering the entire range observed by the sensor 20. In other words, the 3D point cloud constructed by the point cloud construction unit 101 may include 3D point clouds for multiple pieces of equipment included in the equipment group.

[0020] The equipment recognition unit 102 recognizes the equipment included in the equipment group based on the observation results of the equipment group by the sensor 20. If the sensor 20 is an imaging device, the equipment included in the equipment group can be recognized from the captured image of the equipment group using image recognition technology. If the sensor 20 is a survey wave sensor, the equipment included in the equipment group can be recognized using machine learning from the 3D point cloud constructed by the point cloud construction unit 101, for example. In this case, a learner that has learned the correspondence between the equipment and the 3D point cloud by machine learning can be used.

[0021] The layout DB 103 stores information about the virtual layout. In the virtual layout, three-dimensional models of the equipment included in the equipment group are arranged in a virtual space. This three-dimensional model represents the equipment using three-dimensional coordinates in the virtual space. The three-dimensional model may be, for example, a three-dimensional model in 3D CAD. It is preferable that the actual equipment group be arranged in advance with reference to this virtual layout. This is because it is possible to minimize any discrepancy between the actual equipment group arrangement and the virtual layout when adjusting it later. As mentioned above, it is difficult to accurately match the actual equipment group arrangement with the virtual layout due to factors such as the inclination of the floor on which the equipment is installed and the weight of the equipment. Note that the actual equipment group arrangement may also be arranged with reference to an arrangement of the equipment group drawn on paper rather than the virtual layout. The layout DB 103 may be provided outside the real equipment virtualization device 10. For example, it may be provided on a server connectable to the real equipment virtualization device 10 via a network.

[0022] The model DB 104 stores 3D models of each piece of equipment included in the equipment group, which are generated in advance in a virtual space. For example, the model DB 104 may store 3D models for each type of equipment, such as a robot cell or a parts shooter. The model DB 104 may be provided outside the real equipment virtualization device 10. For example, the model DB 104 may be provided in a server that can be connected to the real equipment virtualization device 10 via a network.

[0023] The equipment identification unit 105 corrects the layout of the equipment group reproduced in the virtual space so that it matches the layout of the actual equipment group. As shown in Fig. 1, the equipment identification unit 105 includes, as sub-functional blocks, a range determination unit 151, an environment point cloud acquisition unit 152, a model selection unit 153, a point cloud extraction unit 154, a matching processing unit 155, and a model position correction unit 156.

[0024] The range determination unit 151 determines the range of the 3D point cloud to be used for matching in the matching processing unit 155, from the 3D point cloud constructed by the point cloud construction unit 101. The range determination unit 151 may determine the range of the 3D point cloud so that matching, described below, can be performed for each target facility among the multiple facilities included in the facility group. The range determination unit 151 may determine different 3D point cloud ranges in order so that matching, described below, can be performed for each target facility. The target facilities may be one at a time or multiple at a time, but the following description will be given assuming one at a time. The range determination unit 151 may determine the range by narrowing down the 3D point cloud constructed by the point cloud construction unit 101 to the point cloud surrounding the actual facility, based on the virtual layout stored in the layout DB 103. The range determination unit 151 may determine the range by narrowing the 3D point cloud constructed by the point cloud construction unit 101 to the point cloud surrounding the target real facility based on the image recognition result of the image captured by the imaging device in the equipment recognition unit 102. Note that, to further improve the accuracy of determining the range by narrowing it to the point cloud surrounding the real facility, the virtual layout and the image recognition result may be used in combination. Note that it is assumed that the coordinate system in which the 3D point cloud is constructed by the point cloud construction unit 101 corresponds to the coordinate system of the virtual layout. This may be achieved by performing observation with the sensor 20 so as to include a reference object such as a marker that serves as a reference for a common position of each coordinate system. Alternatively, this may be achieved by fixing the observation position of the sensor 20. In this case, the observation direction may be changed by rotating the sensor 20, for example.

[0025] The environment point cloud acquisition unit 152 extracts and acquires a 3D point cloud of the range determined by the range determination unit 151 from the 3D point cloud constructed by the point cloud construction unit 101. In other words, the environment point cloud acquisition unit 152 extracts an environment 3D point cloud by narrowing down the point cloud to the periphery of the actual facility. This makes matching easier than when the entire 3D point cloud constructed by the point cloud construction unit 101 is used as the target of matching, which will be described later. When the range determination unit 151 sequentially determines ranges for different facilities, the environment point cloud acquisition unit 152 may be configured to sequentially acquire 3D point clouds of different ranges according to the sequentially determined ranges. The 3D point cloud acquired by the environment point cloud acquisition unit 152 is a 3D point cloud obtained by observing the actual facility. Hereinafter, the 3D point cloud acquired by the environment point cloud acquisition unit 152 will be referred to as the environment 3D point cloud. In this embodiment, the environment point cloud acquisition unit 152 is configured to extract an environment three-dimensional point cloud by focusing on the point cloud around the actual facility, but this is not necessarily limited to this. For example, the environment point cloud acquisition unit 152 may be configured to acquire the three-dimensional point cloud constructed by the point cloud construction unit 101.

[0026] Here, an example of cutting out a 3D point cloud of the range determined by the range determination unit 151 from the 3D point cloud constructed by the point cloud construction unit 101 in the environment point cloud acquisition unit 152 will be described with reference to Fig. 4. OPC in Fig. 4 indicates the 3D point cloud constructed by the point cloud construction unit 101. COR in Fig. 4 indicates the range determined by the range determination unit 151. As shown in Fig. 4, the environment point cloud acquisition unit 152 cuts out a 3D point cloud of the range determined by the range determination unit 151 from the 3D point cloud constructed by the point cloud construction unit 101.

[0027] The model selection unit 153 selects a 3D model to be used for matching in the matching processing unit 155 from among the 3D models stored in the model DB 104. The model selection unit 153 may select a 3D model of the equipment that was the target when the range was determined by the range determination unit 151. When the ranges for different equipment are determined sequentially by the range determination unit 151, the model selection unit 153 may be configured to sequentially select 3D models of different equipment according to the sequentially determined ranges.

[0028] The point cloud extraction unit 154 converts the 3D model selected by the model selection unit 153 into a point cloud. The point cloud extraction unit 154 may convert the 3D model into a point cloud by converting the surfaces of the 3D model into a point cloud with equal intervals. The intervals between the point clouds may be set arbitrarily. When 3D models of different equipment are sequentially selected by the model selection unit 153, the point cloud extraction unit 154 may be configured to sequentially convert the sequentially selected 3D models into a point cloud. In other words, the point cloud extraction unit 154 extracts a 3D point cloud from the 3D model of the equipment recognized by the equipment recognition unit 102, among the 3D models of the equipment included in the virtual layout.

[0029] The point cloud extraction unit 154 preferably extracts a portion of the 3D point cloud obtained from the 3D model of the equipment recognized by the equipment recognition unit 102. This extraction of the 3D point cloud may be performed by extracting a portion of the 3D point cloud obtained by converting the 3D model into a point cloud. This extraction of the 3D point cloud may be performed by converting only a portion of the 3D model into a point cloud. For example, this may be performed by converting only a portion of the surfaces of the 3D model into a point cloud. With the above configuration, it is possible to reduce matching errors and variations between matches in the matching processing unit 155. More details are provided below.

[0030] Because the environmental 3D point cloud is a 3D point cloud obtained by observing the actual equipment, no point cloud is obtained for parts of the actual equipment that cannot be observed by the sensor 20. On the other hand, because the 3D point cloud of the 3D model is a model, it may also include point clouds for the interior, bottom, etc. of the equipment that cannot be observed by the sensor 20. Therefore, if an attempt is made to directly match the environmental 3D point cloud and the 3D point cloud of the 3D model for the same equipment, too many point clouds will not match, resulting in large matching errors and variations between matches. In other words, point clouds in areas of the equipment that cannot be observed by the sensor 20 will become noise. In contrast, with the above configuration, by extracting a portion of the 3D point cloud obtained from the 3D model, it is possible to reduce the noise point clouds that cannot match the environmental 3D point cloud. In other words, it is possible to remove noise point clouds that cause incorrect matching. Therefore, it is possible to reduce matching errors and variations between matches in the matching processing unit 155.

[0031] The point cloud extraction unit 154 may estimate an observable area of ​​the actual equipment observed by the sensor 20. The observable area is an area observable by the sensor 20. It is preferable to extract a 3D point cloud of an area corresponding to the observable area in the 3D model from the 3D point cloud obtained from the 3D model of the equipment recognized by the equipment recognition unit 102. Here, extraction of a 3D point cloud of an area corresponding to the observable area will be described with reference to FIG. 5. FIG. 5 is a diagram schematically showing 3D point clouds of a 3D model before and after extraction of the observable area. For ease of understanding, FIG. 5 depicts each surface of the 3D model as a surface rather than a point cloud. PE in FIG. 5 is a diagram schematically showing a 3D point cloud before extraction of the observable area. AE in FIG. 5 is a diagram schematically showing a 3D point cloud after extraction of the observable area. In FIG. 5, the foreground side of the 3D model is the side adjacent to other equipment. The point cloud extraction unit 154 may estimate, for example, the observable area as an area of ​​the equipment, excluding the interior that is not exposed to the outside and the lower part that is estimated to be in contact with the floor. For example, as shown in FIG. 5, the upper and side parts may be estimated as the observable area. Alternatively, based on the information of the virtual layout stored in the layout DB 103, the observable area may be estimated as an area excluding the side parts that are in contact with other equipment and are spaced less than a threshold, as shown in FIG. 5. The threshold here may be set to a narrow enough distance that observation by the sensor 20 is estimated to be blocked, and may be set to any value.

[0032] The above configuration makes it easier to extract the outer shell portion observable by the sensor 20 from the 3D point cloud obtained from the 3D model. This makes it possible to more reliably reduce point clouds that cannot match the environmental 3D point cloud. This makes it more reliably possible to suppress matching errors and variations between matches in the matching processing unit 155.

[0033] The point cloud extraction unit 154 may estimate an observable area of ​​the real equipment observed by the sensor 20 based on the relative positions of the sensor 20 and the real equipment. It is more preferable to extract, from the estimation result, a 3D point cloud of an area corresponding to the observable area in the 3D model of the equipment recognized by the equipment recognition unit 102, from the 3D point cloud obtained from the 3D model. The observable area may be estimated as follows: The point cloud extraction unit 154 may estimate the surface of the real equipment facing the sensor 20 as the observable area. For example, the observable area may be an area excluding areas of the real equipment that are shaded relative to the sensing range of the sensor 20. In this case, areas behind, below, and shaded sides of the real equipment as viewed from the sensor 20 are excluded from the observable area.

[0034] The above configuration makes it easier to extract with higher accuracy the outer shell portion observable by the sensor 20 from the 3D point cloud obtained from the 3D model. This makes it possible to more reliably reduce point clouds that cannot match the environmental 3D point cloud. This further ensures that matching errors and variations between matches in the matching processing unit 155 are reduced.

[0035] The matching processing unit 155 matches the environment 3D point cloud acquired by the environment point cloud acquisition unit 152 with the 3D point cloud extracted from the 3D model by the point cloud extraction unit 154 (hereinafter referred to as the model 3D point cloud). The matching processing unit 155 preferably performs multiple attempts to align the environment 3D point cloud with the model 3D point cloud, and matches the environment 3D point cloud with the model 3D point cloud at the position where the most point clouds match among the attempts. This makes it less likely that an incorrectly matched result will be adopted compared to when multiple attempts are not performed. This further improves the accuracy of matching.

[0036] The matching processing unit 155 preferably matches the environment 3D point cloud and the model 3D point cloud by performing registration between the environment 3D point cloud and the model 3D point cloud in two stages: a coarser registration stage than the other, followed by a finer registration stage than the other. This allows the coarser registration to determine an initial value for the finer registration. The coarser registration stage enables faster coarse registration. The finer registration stage enables a more accurate solution to be found within the roughly registered range. Therefore, the above configuration makes it possible to improve both the speed and accuracy of matching. The coarser registration stage may be performed using, for example, RANSAC (Random Sample Consensus). The finer registration stage may be performed using, for example, ICP (Iterative Closest Point). The following description will be given using an example in which the coarser registration is performed using RANSAC and the finer registration is performed using ICP.

[0037] When aligning the environment 3D point cloud with the model 3D point cloud, if the change in the tilt of the vertical axis of the 3D model represented by the model 3D point cloud exceeds a specified value, the matching processing unit 155 preferably redoes the alignment between the environment 3D point cloud and the model 3D point cloud. The specified value may be a large enough value that it is possible to estimate that the alignment is incorrect, and may be any value that can be set. Here, using FIG. 6, the change in the tilt of the vertical axis of the 3D model represented by the model 3D point cloud when aligning the environment 3D point cloud with the model 3D point cloud will be described. The TDM in FIG. 6 represents the 3D model represented by the model 3D point cloud. In FIG. 6, x, y, and z represent the original x-axis, y-axis, and z-axis of the 3D model represented by the model 3D point cloud before aligning the environment 3D point cloud with the model 3D point cloud. The x-axis is the horizontal axis in the coordinate system. The horizontal direction may also be referred to as the left-right direction. The y-axis is the vertical axis in the coordinate system. The vertical direction can be rephrased as the depth direction. The z axis is the vertical axis in the coordinate system. The vertical direction can be rephrased as the height direction. m ,y m ,z m represent the x-axis, y-axis, and z-axis of the 3D model represented by the model 3D point cloud after the alignment of the environment 3D point cloud and the model 3D point cloud. If the alignment of the environment 3D point cloud and the model 3D point cloud is incorrect, as shown in Figure 6, there will be a large change in the tilt of the vertical axis of the 3D model represented by the model 3D point cloud.

[0038] Due to gravity, the vertical axes of the environment 3D point cloud and the model 3D point cloud do not deviate significantly because the equipment is installed on the floor. However, if the tilt of the vertical axis of the 3D model indicated by the model 3D point cloud changes significantly when aligned with the environment 3D point cloud, there is a high possibility that the alignment is incorrect. In response, the above configuration eliminates alignments that are clearly incorrect and reduces mismatching. In particular, if a large error occurs during the rough alignment stage, many unnecessary processes will be required. Therefore, redoing the alignment eliminates waste and reduces matching errors.

[0039] The model position correction unit 156 corrects the position of the 3D model in the virtual layout by the amount of deviation of the arrangement of the 3D model in the virtual layout relative to the arrangement of the actual equipment, based on the matching performed by the matching processing unit 155. The model position correction unit 156 only needs to correct the position of the 3D model in the virtual layout by the amount of deviation of the x-axis, y-axis, and z-axis coordinates that occurred when the model 3D point cloud was matched to the environment 3D point cloud. The deviation of the x-axis, y-axis, and z-axis coordinates can be calculated by converting the environment 3D point cloud and the model 3D point cloud into coordinates on the same coordinate system, based on the correspondence between the coordinate system of the environment 3D point cloud and the coordinate system of the virtual layout.

[0040] According to the above configuration, since the 3D point clouds are matched, it is possible to more easily align the actual equipment and the 3D model in the equipment group. Then, based on this matching, the position of the 3D model in the virtual layout is corrected by the amount of deviation of the placement of the 3D model in the virtual layout relative to the placement of the actual equipment. This makes it possible to more easily reduce the deviation in placement between the actual equipment and the 3D model. Furthermore, since the position of the 3D model in the virtual layout is corrected, it is possible to more easily reduce the deviation in placement between the actual equipment and the 3D model than by adjusting the placement of the actual equipment. This also makes it possible to more easily reduce the deviation in placement between the actual equipment and the 3D model. In addition, since the 3D point cloud is extracted from the 3D model of the equipment recognized by the equipment recognition unit 102 based on the environment 3D point cloud acquired by the environment point cloud acquisition unit 152, it is possible to more easily match the environment 3D point cloud to be matched with the model 3D point cloud. This also makes it possible to more easily reduce the deviation in placement between the actual equipment and the 3D model. As a result, it is possible to reduce the discrepancy in the placement of the actual equipment and the 3D model while reducing the amount of work required. Reducing the discrepancy in the placement of the actual equipment and the 3D model increases the accuracy of the placement of the equipment reproduced in the virtual space. As the accuracy of the placement of the equipment reproduced in the virtual space increases, it becomes possible to achieve with high reproducibility the actions in accordance with the action plan in the virtual space in the equipment in real space. As a result, it becomes possible to achieve with high reproducibility the actions in accordance with the action plan in the virtual space in the equipment in real space while reducing the amount of work required.

[0041] Here, an example of processing related to operations from correcting the layout of a 3D model based on the results of measuring an actual equipment group by the CPS 1 to planning operations (hereinafter referred to as CPS-related processing) will be described using the flowchart in Fig. 7. The flowchart in Fig. 7 may be started, for example, when a button operation for starting the CPS-related processing is received. Here, an example will be described in which the real equipment virtualization device 10 has already acquired observation results of the equipment group by the sensor 20.

[0042] First, in step S1, the real equipment virtualization device 10 performs a correction process, and then proceeds to step S2. In the correction process, the arrangement of the 3D model of the equipment in the virtual layout is corrected based on the results of measuring the actual equipment group. An example of the flow of the correction process will be described later.

[0043] In step S2, the motion planning unit 30 teaches the position to the robot in the virtual space based on the arrangement of the 3D model of the equipment corrected in step S1. In step S3, the motion planning unit 30 performs motion planning in the virtual space and ends the CPS-related processing.

[0044] Next, an example of the above-mentioned correction process will be described using the flowchart in Fig. 8. In the flowchart in Fig. 8, an example will be described in which a 3D point cloud is extracted by extracting a part of the 3D point cloud obtained by converting a 3D model into a point cloud.

[0045] In step S11, the point cloud constructor 101 constructs a 3D point cloud from the observation results of the sensor 20. In step S12, the range determiner 151 determines the range of the 3D point cloud to be used for matching from the 3D point cloud constructed in S11. Then, the environment point cloud acquirer 152 cuts out and acquires the 3D point cloud of the range determined by the range determiner 151 from the 3D point cloud constructed in S11. Note that the processing of S12 may be performed after the processing of step S13 or step S14 described below.

[0046] In step S13, the model selection unit 153 selects a 3D model of the facility that was the target when the range was determined in S12 from the 3D models stored in the model DB 104. Then, the point cloud extraction unit 154 converts the 3D model selected by the model selection unit 153 into a point cloud.

[0047] In step S14, the point cloud extraction unit 154 performs a point cloud extraction process, and the process proceeds to step S 15. Here, an example of the flow of the point cloud extraction process will be described with reference to the flowchart in FIG.

[0048] First, in step S141, the point cloud extraction unit 154 calculates the relative position between the sensor 20 and the actual equipment. If the position of the sensor 20 is fixed, the relative position between the sensor 20 and the actual equipment can be calculated from the coordinates of the fixed position of the sensor 20 and the positioning results of the sensor 20. If the sensor 20 is an imaging device and a 3D point cloud is constructed by SfM, the relative position between the sensor 20 and the actual equipment can be calculated from the imaging positions estimated from multiple captured images and the positioning results. In this case, the relative position between the sensor 20 and the actual equipment can be calculated for each of the multiple imaging positions.

[0049] In step S142, the point cloud extraction unit 154 estimates the observable area of ​​the real equipment based on the relative position between the sensor 20 and the real equipment calculated in S141. Then, from the estimation result, a 3D point cloud of an area in the 3D model corresponding to the observable area is extracted from the 3D point cloud obtained by converting the 3D model into a point cloud in S13. In step S143, the point cloud extraction unit 154 adds the points that have been newly extracted as areas corresponding to the observable area to the already extracted points. The already extracted points may be temporarily stored in the memory of the real equipment virtualization device 10.

[0050] In step S143, if the processing from S141 to S143 has been repeated a specified number of times (YES in S143), the process proceeds to step S15. On the other hand, if the number of times the processing from S141 to S143 has been repeated has not reached the specified number of times (NO in S143), the process returns to S141 and the processing is repeated. The specified number of times may be set arbitrarily.

[0051] 8, in step S15, the matching processing unit 155 performs the matching process, and the process proceeds to step S 16. Here, an example of the flow of the matching process will be described with reference to the flowchart in FIG.

[0052] First, in step S151, the matching processing unit 155 performs rough registration between the environment 3D point cloud acquired in S12 and the model 3D point cloud extracted in S14. In this embodiment, the registration is performed using RANSAC. In step S152, if the change in tilt of the vertical axis of the 3D model indicated by the model 3D point cloud during the registration in S151 is less than the specified value (YES in S152), the process proceeds to step S153. On the other hand, if the change in this tilt is equal to or greater than the specified value (NO in S152), the process returns to step S151 and performs the registration again.

[0053] In step S153, the matching processing unit 155 performs fine registration between the environment 3D point cloud acquired in S12 and the model 3D point cloud extracted in S14. In this embodiment, the registration is performed using ICP. In step S154, if the change in tilt of the vertical axis of the 3D model indicated by the model 3D point cloud during the registration in S153 is less than the specified value (YES in S154), the environment 3D point cloud and the model 3D point cloud are matched at the position where registration was performed in S153, and the process proceeds to step S155. On the other hand, if the change in tilt is equal to or greater than the specified value (NO in S154), the process returns to step S153 and the registration is performed again.

[0054] In step S155, if the number of times matching has been achieved reaches N (YES in S155), the process proceeds to step S156. On the other hand, if the number of times matching has been achieved does not reach N (NO in S155), the process returns to S151 and repeats. N may be any number that can be set.

[0055] In step S156, the matching processing unit 155 adopts the best matching result among the N matchings as the matching result, and proceeds to S 16. For example, the matching result with the highest match rate may be adopted.

[0056] Returning to FIG. 8, in step S16, the model position correction unit 156 corrects the position of the 3D model in the virtual layout by the amount of deviation of the placement of the 3D model in the virtual layout relative to the placement of the actual equipment, based on the matching result in S15, and proceeds to S2.

[0057] In this embodiment, as described above, the matching accuracy is improved by extracting a 3D point cloud of an area corresponding to the observable area as a model 3D point cloud. Here, the effect of improving the matching accuracy according to this embodiment will be described using FIG. 11 . In FIG. 11 , a method of extracting a 3D point cloud of an area corresponding to the observable area as a model 3D point cloud is referred to as the proposed method. On the other hand, in FIG. 11 , a method of using the 3D point cloud obtained by converting the entire 3D model into a point cloud directly for matching is referred to as the existing method. In FIG. 11 , box plots are used to represent the variation in matching positions for the proposed method and the existing method. Note that the variation for the proposed method is shown for 15 matching attempts. The variation for the existing method is shown for 5 matching attempts. The vertical axis in FIG. 11 is in millimeters. As shown in FIG. 11 , the matching variation for the existing method is on the order of several tens of centimeters, whereas the matching variation for the proposed method is consistently kept to 0.3 mm or less.

[0058] Furthermore, according to this embodiment, it is possible to minimize the error between the position of equipment in an actual facility group and the position of a 3D model of that equipment in a virtual layout. This error will be simply referred to as a position error hereinafter. Here, the effect of improving the accuracy of 3D model position correction using the configuration of this embodiment will be described using FIGS. 12 and 13. FIG. 12 is a diagram illustrating the results of evaluation in a laboratory. The example in FIG. 12 shows the results of position error for a small-scale facility group consisting of a small number of pieces of equipment in a laboratory. In FIG. 12, the method of correcting the position of a 3D model in a virtual layout using the configuration of this embodiment is referred to as the proposed method. On the other hand, in FIG. 12, an existing method, such as manually adjusting actual equipment to fit the virtual layout, is referred to as the existing method.

[0059] As shown in Figure 12, the position error in a laboratory environment with the existing method was 0.87 mm in the x direction, 5.89 mm in the y direction, and -22.99 mm in the z direction. On the other hand, with the proposed method, the error was 0.14 mm in the x direction, 0.17 mm in the y direction, and 0.21 mm in the z direction. As shown in Figure 12, the position error in a laboratory environment was smaller with the proposed method than with the existing method. With the proposed method, the position error in a laboratory environment was kept to less than 1 mm in all directions: x, y, and z.

[0060] Figure 13 is a diagram for explaining the evaluation results in an actual factory. The example in Figure 13 shows the results of position error for a group of equipment actually installed in an actual factory. A and B in Figure 13 represent the distances of line segments connecting a point on one piece of equipment to a different point on another piece of equipment in the group of equipment. As shown in Figure 13, the position error in the actual factory environment is suppressed to 6.5 mm for line segment A and 2.0 mm for line segment B. With the proposed method, the position error is suppressed to less than 1 mm in all directions, x, y, and z. With the proposed method, the position error in the actual factory environment is suppressed to less than 10 mm.

[0061] Furthermore, according to this embodiment, it is possible to reduce the man-hours required from the adjustment to reduce the position error to the completion of an operation plan that can achieve the desired task. This man-hour is hereinafter referred to as the adjustment man-hour. Here, using FIG. 14 , the effect of reducing the adjustment man-hours using the configuration of this embodiment will be described. The example in FIG. 14 shows the adjustment man-hours for a group of equipment actually installed in a real factory. In FIG. 14 , a method using the configuration of this embodiment is referred to as the proposed method. On the other hand, in FIG. 14 , an existing method, such as manually adjusting or teaching the actual equipment to conform to the virtual layout, is referred to as the existing method. As shown in FIG. 14 , the adjustment man-hours required using the existing method are 24.4 hours. In other words, with the existing method, it takes more than three days from the adjustment to reduce the position error to the completion of an operation plan that can achieve the desired task. In contrast, with the proposed method, the adjustment man-hours are a total of 7.6 hours, consisting of 6 hours for the adjustment to reduce the position error and 1.6 hours for the completion of an operation plan that can achieve the desired task. Furthermore, in the proposed method, the real facility virtualization device 10 automatically performs adjustments to reduce positional errors, and the motion planning unit 30 automatically completes the motion plan that will achieve the desired work. This makes it easy to proceed with the work even at night, reducing the amount of manual work required.

[0062] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also within the technical scope of the present disclosure. The control unit and method described in the present disclosure may be implemented by a special-purpose computer comprising a processor programmed to execute one or more functions embodied in a computer program. Alternatively, the apparatus and method described in the present disclosure may be implemented by a special-purpose hardware logic circuit. Alternatively, the apparatus and method described in the present disclosure may be implemented by one or more special-purpose computers configured by combining a processor executing a computer program with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible recording medium. [Explanation of symbols]

[0063] 1 CPS, 10 motion planning unit, 20 sensor, 102 equipment recognition unit, 152 environmental point cloud acquisition unit, 154 point cloud extraction unit, 155 matching processing unit, 156 model position correction unit

Claims

1. an environment point cloud acquisition unit (152) that acquires an environment three-dimensional point cloud, which is a three-dimensional point cloud obtained by observing a facility group in which a plurality of facilities are actually arranged with a sensor (20); an equipment recognition unit (102) that recognizes equipment included in the equipment group based on the observation results of the equipment group by the sensor; a point cloud extraction unit (154) that extracts a three-dimensional point cloud from the three-dimensional model of the equipment recognized by the equipment recognition unit, among three-dimensional models of the equipment included in a virtual layout that is an arrangement of the equipment group generated in advance in a virtual space; a matching processing unit (155) that matches the environment three-dimensional point cloud acquired by the environment point cloud acquisition unit with a model three-dimensional point cloud, which is a three-dimensional point cloud extracted from the three-dimensional model by the point cloud extraction unit; and a model position correction unit (156) that corrects the position of the three-dimensional model in the virtual layout by an amount corresponding to a deviation of the arrangement of the three-dimensional model in the virtual layout from the arrangement of real equipment, which is actual equipment in the group of equipment, based on the matching in the matching processing unit.

2. 2. The real facility virtualization device according to claim 1, The environmental point cloud acquisition unit is a real equipment virtualization device that acquires, as the environmental three-dimensional point cloud, at least one of the three-dimensional point cloud obtained from an image of the equipment group captured by an imaging device serving as the sensor, and the three-dimensional point cloud obtained by observing the equipment group by an exploration wave sensor serving as the sensor, which observes the distance and shape of an object by sending and receiving exploration waves.

3. 2. The real facility virtualization device according to claim 1, The point cloud extraction unit is a real facility virtualization device that extracts a portion of the three-dimensional point cloud obtained from the three-dimensional model of the facility recognized by the facility recognition unit.

4. 4. The real facility virtualization device according to claim 3, The point cloud extraction unit is a real equipment virtualization device that estimates an observable area, which is an area that can be observed by the sensor, of the real equipment observed by the sensor, and extracts a 3D point cloud of an area in the 3D model of the equipment recognized by the equipment recognition unit that corresponds to the observable area from the 3D point cloud obtained from the 3D model of the equipment.

5. 5. The real facility virtualization device according to claim 4, The point cloud extraction unit estimates an observable area, which is an area that can be observed by the sensor, of the real equipment observed by the sensor based on the relative position between the sensor and the real equipment, and extracts a 3D point cloud of an area in the 3D model of the equipment recognized by the equipment recognition unit that corresponds to the observable area from the 3D point cloud obtained from the 3D model.

6. 6. The real facility virtualization device according to claim 5, the environment point cloud acquisition unit extracts the environment three-dimensional point cloud by narrowing down the point cloud to a point cloud around the actual facility based on the virtual layout; A real facility virtualization device that matches the environment three-dimensional point cloud acquired by the environment point cloud acquisition unit with a model three-dimensional point cloud, which is a three-dimensional point cloud extracted from the three-dimensional model by the point cloud extraction unit.

7. 6. The real facility virtualization device according to claim 5, The environment point cloud acquisition unit acquires, as the environment three-dimensional point cloud, the three-dimensional point cloud obtained from an image of the group of facilities captured by an imaging device serving as the sensor, and extracts the environment three-dimensional point cloud by narrowing it down to a point cloud around the actual facilities based on at least one of the virtual layout and an image recognition result for the image captured by the imaging device; A real facility virtualization device that matches the environment three-dimensional point cloud acquired by the environment point cloud acquisition unit with a model three-dimensional point cloud, which is a three-dimensional point cloud extracted from the three-dimensional model by the point cloud extraction unit.

8. 4. The real facility virtualization device according to claim 3, The matching processing unit attempts to align the environment 3D point cloud with the model 3D point cloud multiple times, and matches the environment 3D point cloud with the model 3D point cloud at the position where the most point clouds match among the attempts.

9. 4. The real facility virtualization device according to claim 3, The matching processing unit matches the environment 3D point cloud with the model 3D point cloud by performing alignment between the environment 3D point cloud and the model 3D point cloud in two stages: a coarser alignment stage compared to the other, followed by a finer alignment stage compared to the other. This is a real facility virtualization device.

10. 10. The real facility virtualization device according to claim 9, The matching processing unit is a real facility virtualization device that redoes the alignment of the environment 3D point cloud and the model 3D point cloud if the change in the tilt of the vertical axis of the 3D model indicated by the model 3D point cloud when aligning the environment 3D point cloud and the model 3D point cloud exceeds a specified value.

Images