Information processing method, program, and information processing system

Abstract

To output data obtained by scanning a real space as easy-to-edit data.SOLUTION: An information processing method includes acquiring scan data obtained by scanning an object in a real space and representing the object by a plurality of meshes (S1), calculating a normal line of each of the plurality of meshes (S2), detecting a plurality of continuous regions from the plurality of meshes (S4) with two or more continuous meshes as one continuous region, and determining a continuous region having a normal line in the same direction among the plurality of continuous regions as one plane (S3). A plurality of planes are generated from the plurality of meshes (S5, S6), and the plurality of planes are output as form data indicating the three dimensional form of the object (S8).SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, a program, and an information processing system for processing data obtained by scanning a real space. [Background technology]

[0002] Non-Patent Document 1 discloses a technology that generates 3DCG (3 Dimensional Computer Graphics) of an object in real space when the object is scanned with a smartphone. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Scaniverse(URL:https: / / scaniverse.com / ) Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to provide an information processing method and the like that can output data obtained by scanning a real space as data that is easy to edit. [Means for solving the problem]

[0005] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which acquires scan data representing an object in real space using a plurality of meshes, calculates normals for each of the plurality of meshes, detects a plurality of continuous regions from the plurality of meshes, treats two or more consecutive meshes as one continuous region, and generates a plurality of planes from the plurality of meshes by treating a continuous region among the plurality of continuous regions that has normals in the same direction as one plane, and outputs the plurality of planes as shape data representing the three-dimensional shape of the object.

[0006] Furthermore, a program according to one aspect of the present disclosure is a program that causes a computer to execute the information processing method described above.

[0007] Moreover, an information processing system according to one aspect of the present disclosure includes an acquisition unit that acquires scan data representing an object in real space using a plurality of meshes, the scan data being obtained by scanning the object; a normal calculation unit that calculates the normals of each of the plurality of meshes; a continuous region detection unit that detects a plurality of continuous regions from the plurality of meshes, with two or more consecutive meshes considered as one continuous region; a plane generation unit that generates a plurality of planes from the plurality of meshes, with a continuous region among the plurality of continuous regions having normals in the same direction considered as one plane; and an output unit that outputs the plurality of planes as shape data indicating the three-dimensional shape of the object.

[0008] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0009] According to the present disclosure, there is an advantage that data obtained by scanning a real space can be output as data that is easy to edit. [Brief explanation of the drawings]

[0010] [Figure 1] Figure 1 is an explanatory diagram of the issues involved in generating 3DCG by scanning an object in real space. [Figure 2] FIG. 2 is an explanatory diagram of another problem that arises when generating 3DCG by scanning an object in real space. [Figure 3] FIG. 3 is a block diagram showing an overall configuration including an information processing system according to an embodiment. [Figure 4] FIG. 4 is an explanatory diagram of a method for calculating the normals of each mesh. [Figure 5] FIG. 5 is an explanatory diagram of a specific example of the calculated normal line. [Figure 6] FIG. 6 is an explanatory diagram of a method for detecting a continuous region. [Figure 7] FIG. 7 is an explanatory diagram of a method for generating a plane. [Figure 8] FIG. 8 is a flowchart illustrating an example of the operation of the information processing system according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of shape data output by the information processing system according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating another example of shape data output by the information processing system according to the embodiment. [Figure 11] FIG. 11 is an explanatory diagram of the feature amount of a texture image. [Figure 12] FIG. 12 is an explanatory diagram of another method for generating a plane. [Figure 13] FIG. 13 is an explanatory diagram of a method for dividing a plane. [Figure 14] FIG. 14 is an explanatory diagram of a method for dividing a plane. DETAILED DESCRIPTION OF THE INVENTION

[0011] In recent years, a technology has become known in which a 3DCG representation of an object is generated by scanning the object in real space with an information processing terminal such as a smartphone, as disclosed in, for example, Non-Patent Document 1. However, such a technology has the following problems.

[0012] Figure 1 is an explanatory diagram of the challenges involved in generating 3DCG by scanning a real-world object. The example shown in Figure 1 represents a 3DCG generated by scanning a conference room, a real-world object, with an information processing device such as a smartphone. As shown in Figure 1, the 3DCG generated using the above technology is composed of numerous fine meshes M1, each of which does not have attribute information indicating whether it belongs to one of the object's components, such as a floor or a wall. Therefore, when editing the texture of the 3DCG, for example, it is not possible to edit the texture collectively for each component, such as a floor or a wall. Instead, the texture must be edited for each mesh M1, making data editing difficult. Here, "texture" refers to texture or material information that records light reflection information.

[0013] FIG. 2 is an explanatory diagram of another issue that arises when generating 3DCG by scanning an object in real space. The example shown in FIG. 2, like that shown in FIG. 1, represents 3DCG generated by scanning a conference room, an object in real space, with an information processing device such as a smartphone. As shown in FIG. 2, the 3DCG generated using the above technology suffers from distortions such as unevenness on the wall due to measurement errors, as shown in the area surrounded by dashed line A1, and defects at the joint between the floor and wall due to occlusion, as shown in the area surrounded by dashed line A2. Therefore, when editing the 3DCG, editing is required to correct the distortion or defects, which again presents the issue of making it difficult to edit the data.

[0014] In view of the above, an object of the present disclosure is to provide an information processing method and the like that can output data obtained by scanning a real space as data that is easy to edit.

[0015] More specifically, an information processing method according to a first aspect of the present disclosure is an information processing method executed by a computer, which acquires scan data representing an object in real space by scanning the object, calculating normals of each of the plurality of meshes, treating two or more consecutive meshes as one consecutive region, detecting a plurality of consecutive regions from the plurality of meshes, treating a continuous region among the plurality of consecutive regions that has normals in the same direction as one plane, generating a plurality of planes from the plurality of meshes, and outputting the plurality of planes as shape data indicating the three-dimensional shape of the object.

[0016] This allows for generating a plane P1 for each attribute indicating that it belongs to one of the constituent elements of the object, such as a floor or a wall, from a plurality of meshes representing the object, and outputting it as shape data including a plurality of planes indicating the three-dimensional shape of the object. Therefore, this has the advantage that data obtained by scanning real space (scan data) can be output as data (shape data) that is easy to edit.

[0017] Also, for example, in the information processing method according to the second aspect of the present disclosure, in the first aspect, a plurality of blocks in which a predetermined thickness in the direction of the normal is added to each of a plurality of planes are output as shape data.

[0018] This has the advantage that the user does not need to add a specified thickness to each of multiple planes, and the data obtained by scanning real space (scan data) can be output as data (shape data) that is easier to edit.

[0019] Also, for example, in an information processing method according to a third aspect of the present disclosure, in the first or second aspect, the normals of each of the multiple meshes are classified into one of the following directions: upward, downward, leftward, rightward, forward, and backward.

[0020] This has the advantage that each mesh is classified into one of six classes based on the direction of its normal: upward, downward, leftward, rightward, forward, and backward, making it easier to classify each mesh into each component of the object.

[0021] Also, for example, in the information processing method according to the fourth aspect of the present disclosure, in the third aspect, the scan data is rotated so that one of the sides of the object is parallel to one of the X-axis, Y-axis, and Z-axis in the Cartesian coordinate system, and then the normals of each of the multiple meshes are calculated.

[0022] This has the advantage that it becomes easier to classify each mesh into one of six classes based on the direction of its normal: upward, downward, leftward, rightward, forward, and backward.

[0023] Also, for example, in an information processing method according to a fifth aspect of the present disclosure, in any one of the first to fourth aspects, two or more adjacent meshes among a plurality of meshes are treated as one continuous area and assigned the same identifier, thereby detecting multiple continuous areas.

[0024] This has the advantage that two or more adjacent meshes are assigned an identifier as one continuous area, making it easy to detect multiple continuous areas and to distinguish between multiple continuous areas based on the identifier.

[0025] Also, for example, in the information processing method according to the sixth aspect of the present disclosure, in the fifth aspect, if two or more meshes are adjacent to each other and the texture images referenced by the two or more meshes are similar, the two or more meshes are assigned an identifier as a single continuous area.

[0026] This has the advantage that multiple continuous areas are detected so that the texture images referenced by two or more meshes belonging to a continuous area are similar, making it more likely that two or more meshes belonging to a continuous area will belong to the same component of the object.

[0027] Also, for example, in an information processing method according to a seventh aspect of the present disclosure, in any one of the first to fourth aspects, a trained model trained by machine learning to receive an image including at least a portion of a plurality of meshes as input and output attributes of the image is used to detect multiple continuous regions from the plurality of meshes, with two or more meshes having the same attributes output by the trained model being considered as one continuous region.

[0028] This has the advantage that two or more meshes belonging to a continuous region are likely to belong to the same component of the object, since multiple continuous regions are detected so that two or more continuous meshes have the same attribute.

[0029] Also, for example, in an information processing method according to an eighth aspect of the present disclosure, in any one of the first to seventh aspects, a virtual plane is generated by performing principal component analysis on the coordinates of each vertex of each mesh included in a continuous region having normals in the same direction, and one plane is generated by projecting the coordinates of each vertex of each mesh onto the generated virtual plane.

[0030] This has the advantage that it is easy to generate a plane that approximates each mesh included in the continuous region.

[0031] Also, for example, in an information processing method according to a ninth aspect of the present disclosure, in any one of the first to seventh aspects, a single plane is generated using a representative value of the normal direction of each mesh included in a continuous area having normals in the same direction.

[0032] This has the advantage that it is easy to generate a plane that approximates each mesh included in the continuous region.

[0033] Also, for example, in an information processing method according to a tenth aspect of the present disclosure, in any one of the first to ninth aspects, if one plane can be divided into multiple rectangles, each of the multiple divided rectangles is considered to be a plane.

[0034] This has the advantage that it is easier to output shape data that accurately represents the three-dimensional shape of the object, compared to when multiple rectangles are collectively treated as one plane.

[0035] Furthermore, for example, a program according to an eleventh aspect of the present disclosure causes a computer to execute the information processing method according to any one of the first to tenth aspects.

[0036] This has the advantage that it is possible to achieve the same effect as the above-mentioned information processing method.

[0037] Also, for example, an information processing system according to a twelfth aspect of the present disclosure includes an acquisition unit, a normal calculation unit, a continuous area detection unit, a plane generation unit, and an output unit. The acquisition unit acquires scan data representing the object in multiple meshes obtained by scanning the object in real space. The normal calculation unit calculates the normal of each of the multiple meshes. The continuous area detection unit detects multiple continuous areas from the multiple meshes, with two or more continuous meshes considered as one continuous area. The plane generation unit generates multiple planes from the multiple meshes, with continuous areas having normals oriented in the same direction considered as one plane. The output unit outputs the multiple planes as shape data indicating the three-dimensional shape of the object.

[0038] This has the advantage that it is possible to achieve the same effect as the above-mentioned information processing method.

[0039] Furthermore, these comprehensive or specific aspects may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0040] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, or step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components. Note that each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0041] (Embodiment) [1. Configuration] First, the overall configuration including an information processing system according to an embodiment will be described. Fig. 3 is a block diagram showing the overall configuration including an information processing system according to an embodiment. As shown in Fig. 3, an information processing system 1 according to an embodiment is a 3DCG creation device 1 that acquires scan data obtained by scanning an object in real space with an imaging device 2, generates shape data (here, a 3DCG model) indicating the three-dimensional shape of the object based on the scan data, and outputs the generated shape data to a 3DCG editing device 3. In this embodiment, the object is a structure that can be approximated by a rectangular parallelepiped, such as a conference room, an office, or an ordinary house.

[0042] In the embodiment, the 3DCG creation device 1, the photographing device 2, and the 3DCG editing device 3 are independent devices, but one or more of these devices may be realized as a single device. For example, the 3DCG creation device 1, the photographing device 2, and the 3DCG editing device 3 may be realized as a single portable information processing terminal such as a smartphone.

[0043] The image capturing device 2 is a portable information processing terminal such as a smartphone or tablet terminal, and has a camera function for capturing images of an object. The image capturing device 2 is realized by starting an application on such an information processing terminal that is capable of performing, for example, a three-dimensional scan that measures the shape of an object in three dimensions and acquires point cloud data. Note that the image capturing device 2 may also be, for example, a dedicated information processing device capable of performing three-dimensional scanning. The image capturing device 2 includes an image capturing unit 21 and a display unit 22.

[0044] The photographing unit 21 has an image sensor and photographs an object in real space in response to a user's operation, thereby performing a three-dimensional scan of the object. In this embodiment, the photographing unit 21 three-dimensionally scans a conference room as the object.

[0045] The display unit 22 is, for example, a liquid crystal display or an organic EL display built into the imaging device 2, and displays point cloud data of the object obtained by three-dimensional scanning by the imaging unit 21. In the embodiment, the display unit 22 displays a 3DCG model in which the object is represented by a plurality of meshes.

[0046] The 3DCG editing device 3 is an information processing terminal such as a desktop or laptop personal computer, and is realized by running a game engine such as Unreal Engine (registered trademark). The 3DCG editing device 3 may be a portable information processing terminal such as a smartphone or tablet terminal, or may be an information processing terminal dedicated to editing 3DCG. The 3DCG editing device 3 includes an editing unit 31 and a display unit 32.

[0047] The editing unit 31 edits the data (3DCG model) generated by the 3DCG creation device 1 in response to user operations. For example, the editing unit 31 edits the texture of the generated 3DCG model or modifies the shape of the 3DCG model. Here, the user edits the 3DCG model using an input device such as a keyboard or a mouse while viewing the 3DCG model displayed on the display unit 32.

[0048] The display unit 32 is, for example, a liquid crystal display or an organic EL display, and displays the 3DCG model generated by the 3DCG creation device 1. The display unit 32 may be a display built into the 3DCG editing device 3, or may be an external display. The display unit 32 may also be, for example, a touch panel display, and may also function as an input device.

[0049] The 3DCG creation device 1 is realized by an information processing terminal such as a desktop or laptop personal computer. The 3DCG creation device 1 may be a portable information processing terminal such as a smartphone or tablet terminal, or may be a dedicated information processing terminal for generating shape data.

[0050] The 3DCG creation device 1 includes, for example, a processor and a memory, and programs stored in the memory are executed by the processor to perform various functions of the 3DCG creation device 1. In the embodiment, the 3DCG creation device 1 includes an acquisition unit 11, a normal calculation unit 12, a continuous area detection unit 13, a plane generation unit 14, and an output unit 15.

[0051] The acquisition unit 11 acquires scan data that is obtained by scanning an object in real space and represents the object with a plurality of meshes M1 (see FIG. 4). In the embodiment, the acquisition unit 11 acquires scan data of the object that is obtained by performing a three-dimensional scan with the imaging device 2, for example, by communicating with the imaging device 2 via wired or wireless communication.

[0052] The normal calculation unit 12 calculates the normal N1 (see FIG. 4) of each of the multiple meshes M1 representing the object. A method for calculating the normal N1 of each mesh M1 will be described below with reference to FIG. 4. FIG. 4 is an explanatory diagram of the method for calculating the normal N1 of each mesh M1. The mesh M1 shown in FIG. 4 is an arbitrary mesh selected from the multiple meshes M1 representing the object. Here, the coordinates of the three vertices of the mesh M1 are "V0," "V1," and "V2." The normal N1 of the mesh M1 is calculated by finding the cross product of the vector VV1 (= V1 - V0) and the vector VV2 (= V2 - V1). In this embodiment, the normal N1 is normalized so that its length is 1.0.

[0053] In this embodiment, the normal calculation unit 12 classifies the normal N1 of each of the multiple meshes M1 into one of the upward, downward, leftward, rightward, forward, and backward directions. Here, when the coordinates of each vertex of each mesh M1 are expressed in a Cartesian coordinate system consisting of the X-axis, Y-axis, and Z-axis, the upward direction is the positive Y-axis direction, the downward direction is the negative Y-axis direction, the leftward direction is the negative X-axis direction, the rightward direction is the positive X-axis direction, the forward direction is the positive Z-axis direction, and the backward direction is the negative Z-axis direction.

[0054] FIG. 5 is an explanatory diagram of a specific example of the calculated normal N1. The example shown in FIG. 5 represents scan data in which the target object is a rectangular parallelepiped conference room. Furthermore, FIG. 5 illustrates only the normal N11 of an arbitrary mesh M1 representing the right wall, among the normals N1 of each mesh M1. In the example shown in FIG. 5, the normal N1 of each mesh M1 representing the floor is a normal in the upward direction (positive Y-axis direction), and the normal N1 of each mesh M1 representing the ceiling is a normal in the downward direction (negative Y-axis direction). Furthermore, the normal N1 of each mesh M1 representing the left wall is a normal in the right direction (positive X-axis direction), and the normal N1 of each mesh M1 representing the right wall is a normal in the left direction (negative X-axis direction). Furthermore, the normal N1 of each mesh M1 representing the front wall is a normal in the backward direction (negative Z-axis direction), and the normal N1 of each mesh M1 representing the back wall is a normal in the forward direction (positive Z-axis direction).

[0055] In the example shown in Fig. 5, the normal N11 of each mesh representing the right wall of the conference room, which is the target object, is directed to the left (negative X-axis direction). Here, when the normal N11 is expressed as a normal vector (nx, ny, nz), the normal N11 is expressed as nx<0, |ny|<0.1, |nz|<0.1. Note that "nx" represents the X-axis component of the normal vector, "ny" represents the Y-axis component of the normal vector, and "nz" represents the Z-axis component of the normal vector.

[0056] In this embodiment, the normal calculation unit 12 performs preprocessing on the scan data so that the normal N1 is parallel to one of the X-axis, Y-axis, and Z-axis, and then calculates the normal N1 of each mesh M1. Specifically, the normal calculation unit 12 rotates the scan data so that one side of the object represented by the scan data is parallel to one of the X-axis, Y-axis, and Z-axis in a Cartesian coordinate system, and then calculates the normal N1 of each of the multiple meshes M1. In other words, the process of rotating the scan data corresponds to preprocessing.

[0057] For example, suppose that one side of the floor in the scan data of a conference room, which is the target object, is tilted at 45 degrees with respect to the X axis. In this case, the normal calculation unit 12 rotates the scan data by 45 degrees so that the side is parallel to the X axis, and then calculates the normal N1 of each mesh M1. As a result, the calculated normal N1 of each mesh M1 is all parallel to one of the X axis, Y axis, and Z axis.

[0058] In addition, if any one of the sides of the object represented by the scan data is parallel to any one of the X-axis, Y-axis, and Z-axis from the beginning, the normal calculation unit 12 does not need to perform preprocessing.

[0059] The continuous region detection unit 13 detects a plurality of continuous regions R1 from the plurality of meshes M1 representing the object, with two or more continuous meshes M1 being treated as one continuous region R1 (see FIG. 6). In the embodiment, the continuous region detection unit 13 detects the plurality of continuous regions R1 by treating two or more meshes M1 that are adjacent to each other among the plurality of meshes M1 as one continuous region R1 and assigning an identifier (for example, a group number) to the two or more meshes M1. This detection method will be described below with reference to FIG. 6.

[0060] FIG. 6 is an explanatory diagram of a method for detecting a continuous region R1. In the example shown in FIG. 6, four meshes M11, M12, M13, and M14 are illustrated as the multiple meshes M1. The meshes M11 and M12 are adjacent to each other. The meshes M13 and M14 are adjacent to each other. The meshes M13 and M14 are spaced apart from the meshes M11 and M12. In the example shown in FIG. 6, the continuous region detection unit 13 groups the two adjacent meshes M11 and M12 into one continuous region R11 and assigns the identifier "1" to these meshes M11 and M12. The continuous region detection unit 13 groups the two adjacent meshes M13 and M14 into one continuous region R12 and assigns the identifier "2" to these meshes M13 and M14.

[0061] As described above, the continuous region detection unit 13 divides all meshes M1 representing the object into multiple continuous regions R1. In this embodiment, the multiple meshes M1 representing the object are classified into multiple mesh groups based on the direction of the normal N1. For example, two or more meshes M1 among the multiple meshes M1 that have a normal in the upward direction (positive Y-axis direction) are classified to belong to the same mesh group. Then, the continuous region detection unit 13 executes a process of detecting a continuous region R1 for each mesh group. Therefore, for example, if a mesh M1 representing a floor and a mesh M1 representing a wall are adjacent to each other, these meshes M1 will be detected as different continuous regions R1.

[0062] The plane generation unit 14 generates a plurality of planes P1 (see FIG. 7) from a plurality of meshes M1 by treating, as one plane, a continuous region R1 having a normal N1 in the same direction among the plurality of continuous regions R1 detected by the continuous region detection unit 13. In this embodiment, the plane generation unit 14 generates a plane P1 for each continuous region R1 using Principle Component Analysis (PCA), thereby generating a plurality of planes P1 from a plurality of meshes M1. This generation method will be described below with reference to FIG. 7.

[0063] FIG. 7 is an explanatory diagram of a method for generating a plane P1. In FIG. 7, the coordinates of each vertex of each mesh M1 included in an arbitrary continuous region R1 are represented by circles. In FIG. 7, the coordinates of each vertex of each mesh M1 are projected onto a virtual plane P0. The plane generation unit 14 generates a virtual plane P0 including each vertex of each mesh M1 by performing principal component analysis on the coordinates of each vertex of each mesh M1 included in the continuous region R1. Then, the plane generation unit 14 generates one plane P1 by projecting the coordinates of each vertex of each mesh M1 onto the generated virtual plane P0. Here, the plane generation unit 14 calculates a rectangle that surrounds all the vertices of each mesh M1 projected onto the virtual plane P0, and generates this rectangle as one plane P1.

[0064] For example, each mesh M1 representing the left wall of a conference room, which is the target object, is detected as one continuous region R1, and the normal N1 of each mesh M1 is all in the right direction (positive direction of the X-axis). Therefore, the plane generation unit 14 generates the continuous region R1 as one plane P1 representing the left wall of the conference room. Similarly, the plane generation unit 14 generates one plane P1 representing the right wall of the conference room, one plane P1 representing the front wall, one plane P1 representing the back wall, one plane P1 representing the ceiling, and one plane P1 representing the floor. In other words, the plane generation unit 14 generates six planes P1 that make up the conference room.

[0065] Although no mention is made here of the furniture such as desks and chairs placed in the conference room, each mesh M1 that constitutes this furniture is also detected as an independent continuous region R1 and generated as an independent plane P1.

[0066] The output unit 15 outputs the multiple planes P1 generated by the plane generation unit 14 as shape data indicating the three-dimensional shape of the object. In the embodiment, the output unit 15 transmits the shape data to the 3DCG editing device 3, for example, by communicating with the 3DCG editing device 3 via wired or wireless communication. In the embodiment, the output unit 15 outputs, as shape data, multiple blocks B1 (see FIG. 9) in which a predetermined thickness t1 (see FIG. 9) in the direction of the normal N1 is added to each of the multiple planes P1. The output unit 15 may determine an initial value stored in memory in advance as the predetermined thickness t1, or may determine a value appropriately set by the user as the predetermined thickness t1.

[0067] The output unit 15 may output, as shape data, not the plurality of blocks B1 but the plurality of planes P1 to which the predetermined thickness t1 has not been added.

[0068] [2. Operation] The operation of the information processing system 1 (3DCG creation device 1) according to the embodiment, that is, the information processing method, will be described below. Fig. 8 is a flowchart showing an example of the operation of the information processing system 1 according to the embodiment.

[0069] First, when the operation of the information processing system 1 starts, the acquisition unit 11 acquires scan data obtained by scanning an object in real space with the imaging device 2 (S1). Next, the normal calculation unit 12 calculates the normal N1 of each of the multiple meshes M1 included in the scan data acquired by the acquisition unit 11 (S2). Then, the normal calculation unit 12 classifies the normal N1 of each mesh M1 based on the direction of the calculated normal N1 (S3). In this embodiment, the normal calculation unit 12 classifies the normal N1 of each of the multiple meshes M1 into one of the upward direction, downward direction, leftward direction, rightward direction, forward direction, and backward direction.

[0070] Next, the continuous region detection unit 13 detects multiple continuous regions R1 from the multiple meshes M1, with two or more continuous meshes M1 being considered as one continuous region R1 (S4). Then, the continuous region detection unit 13 extracts mesh groups having normals N1 facing the same direction, based on the direction of the normals N1 and the detected continuous regions R1 (S5). In this embodiment, the multiple meshes M1 are classified into multiple mesh groups based on the direction of the normals N1 by the normal calculation unit 12. Then, the continuous region detection unit 13 executes a process of detecting a continuous region R1 for each mesh group. Therefore, by executing step S4, step S5 has already been executed.

[0071] Next, the plane generation unit 14 generates a plane P1 for each detected continuous region R1, in other words, for each extracted mesh group (S6). In this embodiment, the plane generation unit 14 generates a virtual plane P0 by performing principal component analysis on the coordinates of each vertex of each mesh M1 included in the continuous region R1, and generates the plane P1 by calculating a rectangle that encloses the coordinates of each vertex of each mesh M1 projected onto the generated virtual plane P0. If the process of generating planes P1 has not been performed for all continuous regions R1 (S7: No), the plane generation unit 14 performs the process of generating planes P1 for the remaining continuous regions R1 (S6).

[0072] Then, when the process of generating planes P1 for all continuous regions R1 has been executed (S7: Yes), the output unit 15 outputs the multiple planes P1 as shape data indicating the three-dimensional shape of the object (S8). In this embodiment, the output unit 15 outputs multiple blocks B1, each of which is formed by adding a predetermined thickness t1 in the direction of the normal N1 to each of the multiple planes P1, as shape data.

[0073] [3. Advantages] The advantages of the information processing system 1 (information processing method) according to the embodiment will be described below. As described above, the information processing system 1 according to the embodiment can generate, from a plurality of meshes M1 representing an object, a plane P1 for each attribute indicating that the plane P1 belongs to one of the constituent elements of the object, such as a floor or a wall, and output the plane P1 as shape data including a plurality of planes P1 indicating the three-dimensional shape of the object. Therefore, when editing the texture of the shape data using the 3DCG editing device 3, for example, the user can collectively edit the texture in units of planes P1, that is, in units of the constituent elements of the object, such as a wall or a floor.

[0074] FIG. 9 is a diagram showing an example of shape data output by the information processing system 1 according to the embodiment. FIG. 9 shows an example of shape data output by the information processing system 1 when the object is a conference room. In FIG. 9, the three-dimensional shape of the object is composed of a plurality of blocks B1, each of which has a predetermined thickness t1 added to a plurality of planes P1. As shown in FIG. 9, the left wall of the conference room is represented by block B11, and the floor is represented by block B12. Therefore, using the 3DCG editing device 3, a user can collectively edit the texture of the left wall by editing block B11, or collectively edit the texture of the floor by editing block B12.

[0075] Furthermore, since the information processing system 1 according to the embodiment generates smooth planes P1 from the meshes M1 representing the object, the planes P1 are free from defects due to occlusion and distortion due to measurement errors, eliminating the need for the user to correct these defects and distortions using the 3DCG editing device 3.

[0076] Fig. 10 is a diagram showing another example of shape data output by the information processing system 1 according to the embodiment. Similar to Fig. 9, Fig. 10 shows an example of shape data output by the information processing system 1 when the object is a conference room. Note that in Fig. 10, similar to Fig. 9, the three-dimensional shape of the object is formed by a plurality of blocks B1, each of which has a predetermined thickness t1 added to a plurality of planes P1. As shown in Fig. 10, the block B11 representing the left wall does not have distortion due to measurement error, and the joint between the block B11 representing the left wall and the block B12 representing the floor does not have any defects due to occlusion.

[0077] In this way, the information processing system 1 according to the embodiment has an advantage in that it can output data (scan data) obtained by scanning a real space as data (shape data) that is easy to edit.

[0078] (Other embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0079] In the above embodiment, the continuous region detection unit 13 assigns an identifier to two or more adjacent meshes M1 as a single continuous region R1, but this is not limited to this. For example, if two or more meshes M1 are adjacent to each other and the texture images referenced by the two or more meshes M1 are similar, the continuous region detection unit 13 may assign an identifier to the two or more meshes M1 as a single continuous region R1. In other words, even if the two or more meshes M1 are adjacent to each other, if the texture images referenced by the two or more meshes M1 are dissimilar, the continuous region detection unit 13 assigns identifiers to the two or more meshes M1 as different continuous regions R1.

[0080] Here, the scan data includes multiple texture images referenced by multiple meshes M1, respectively. Therefore, it is possible to determine whether two or more meshes M1 are continuous based on whether the texture images are similar. For example, if one of two adjacent meshes M1 references a texture image of a floor and the other mesh M1 references a texture image of a desk, it is possible to determine that these meshes M1 belong to different continuous regions R1.

[0081] Whether two texture images are similar or not can be determined by, for example, using Bag-of-Visual Words to represent each of the two texture images with multidimensional features, and calculating the distance between the two texture images based on the multidimensional features of each of the two texture images. In other words, if the distance between the texture images exceeds a threshold, it can be determined that the two texture images are dissimilar, and if the distance between the texture images is below the threshold, it can be determined that the two texture images are similar.

[0082] FIG. 11 is an explanatory diagram of feature amounts of a texture image. In the example shown in FIG. 11, an image of a piano viewed from above is used as a texture image, and feature amounts of the image are expressed in histograms using Bag-of-Visual Words. In the example shown in FIG. 11, the image is expressed by histograms of seven feature amounts a1, a2, a3, a4, a5, a6, and a7. Note that the images of the feature amounts a1 to a7 are visual representations of the feature amounts for easier understanding. In reality, the feature amounts a1 to a7 are expressed by local feature amounts such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Features).

[0083] Alternatively, the continuous region detection unit 13 may use, for example, a trained model for semantic segmentation to detect multiple continuous regions R1 from multiple meshes M1, with two or more meshes M1 having the same attribute output by the trained model being treated as one continuous region R1. In this case, the trained model is trained by machine learning to receive an input image including at least a portion of the multiple meshes M1 and output the attributes of the image. For example, the input image to the trained model may be a planar image obtained by capturing the three-dimensional shape of an object represented by scan data with a virtual camera in a virtual space. In this case, the plane generation unit 14 does not generate the plane P1 by calculating a rectangle from the virtual plane P0, but may generate a plane P1 that approximates the shape of the continuous region R1.

[0084] In the above embodiment, the plane generation unit 14 generates one plane P1 corresponding to the continuous region R1 using principal component analysis, but this is not limiting. For example, the plane generation unit 14 may generate one plane P1 using a representative value (e.g., average, mode, or median) of the orientation of the normals N1 of the meshes M1 included in the continuous region R1 that have normals N1 oriented in the same direction.

[0085] Fig. 12 is an explanatory diagram of another method for generating a plane P1. As shown in Fig. 12, the general form of the plane P1 is expressed by the equation ax+by+cz+d=0. In this equation, (a, b, c) represent the normal vector of the plane P1, and (x, y, z) represent the coordinates on the plane P1. Here, the distance L from an arbitrary point (x1, y1, z1) to the plane P1 is expressed by the following equation.

[0086]

number

[0087] Therefore, the plane generation unit 14 calculates a representative value of the direction of the normal N1 of each mesh M1 included in the continuous region R1, thereby calculating an estimated value of the normal vector (a, b, c) of the plane P1. Then, the plane generation unit 14 substitutes the estimated value of the normal vector into (a, b, c) in the above formula, and then substitutes the coordinates of each vertex of each mesh M1 into (x1, y1, z1) in the above formula to sequentially calculate the distance L, and calculates "d" that minimizes the distance L. In this way, the plane generation unit 14 can generate one plane P1 corresponding to the continuous region R1.

[0088] In the above embodiment, the plane generation unit 14 calculates a rectangle that surrounds all the vertices of each mesh M1 projected onto the virtual plane P0 and generates the rectangle as one plane P1, but this is not limiting. For example, if the plane generation unit 14 can divide one plane P1 into multiple rectangles, it may also set each of the multiple divided rectangles as a plane P1.

[0089] Fig. 13 is an explanatory diagram of a method for dividing a plane P1. In the example shown in Fig. 13, the group of meshes corresponding to the continuous region R1 has parts where no vertices of the mesh M1 exist, such as the hatched areas. In such cases, if the areas are included in one plane P1, it may not be possible to accurately represent the shape corresponding to the continuous region R1.

[0090] FIG. 14 is an explanatory diagram of a method for dividing a plane P1. FIG. 14(a) shows a plane P11 that simply represents a group of meshes corresponding to the continuous region R1 shown in FIG. 13. Here, as shown in FIG. 14(b), if a circumscribing rectangle C1 that surrounds the plane P11 is generated as a single plane P1, the plane P1 will not accurately represent the shape of the plane P11. Therefore, as shown in FIG. 14(c), if the plane P11 can be divided into two rectangles C11 and C12, generating each of the rectangles C11 and C12 as a plane P1 makes it easier to accurately represent the shape of the plane P11.

[0091] In the above embodiment, the output unit 15 outputs the shape data without adding attribute information indicating whether the components of the object, such as the floor or the wall, belong to that class. However, this is not limited to this. For example, if material information is assigned to each mesh M1, the output unit 15 may output shape data in which attributes are added to each plane P1 by referencing the material information. Also, for example, the output unit 15 may estimate the attributes of each plane P1 based on the relative positional relationship between the planes P1, and output shape data in which the estimated attributes are added to each plane P1.

[0092] Furthermore, for example, the information processing system described in the above embodiments may be realized as a single device that has all of the components, or may be realized by allocating each function to multiple devices and having these multiple devices work together.

[0093] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0094] In the above embodiments, each component may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU (Central Processing Unit) or a processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0095] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0096] Furthermore, the general or specific aspects of the present disclosure may be realized as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0097] For example, the present disclosure may be realized as an information processing method executed by a computer, or as a program for causing a computer to execute the information processing method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0098] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0099] The present disclosure is useful in editing data obtained by scanning a real space. [Explanation of symbols]

[0100] 1. 3DCG creation device (information processing system) 11 Acquisition Department 12 Normal calculation section 13 Continuous area detection unit 14 Plane generator 15 Output section 2. Imaging equipment 21 Photography Department 22 Display section 3 3DCG editing equipment 31 Editorial Department 32 Display section a1, a2, a3, a4, a5, a6, a7 features A1, A2 dashed lines B1, B11, B12 blocks C1 circumscribed rectangle C11, C12 rectangle M1, M11, M12, M13, M14 mesh N1, N11 normal P0 Virtual plane P1, P11 plane R1, R11, R12 continuous area t1 thickness

Claims

1. 1. A computer-implemented information processing method, comprising: acquiring scan data representing an object in real space using a plurality of meshes; Calculating normals for each of the plurality of meshes; Two or more consecutive meshes are regarded as one consecutive region, and a plurality of consecutive regions are detected from the plurality of meshes; generating a plurality of planes from the plurality of meshes by treating, as one plane, a continuous region among the plurality of continuous regions having the normals in the same direction; outputting the plurality of planes as shape data indicating the three-dimensional shape of the object; Information processing methods.

2. a plurality of blocks each having a predetermined thickness in the direction of the normal to each of the plurality of planes are output as the shape data; The information processing method according to claim 1 .

3. classifying the normal of each of the plurality of meshes into any one of an upward direction, a downward direction, a leftward direction, a rightward direction, a forward direction, and a backward direction; 3. The information processing method according to claim 1 or 2.

4. rotate the scan data so that any one side of the object is parallel to any one of the X-axis, Y-axis, and Z-axis in a Cartesian coordinate system, and then calculate the normal of each of the plurality of meshes; The information processing method according to claim 3 .

5. two or more meshes adjacent to each other among the plurality of meshes are identified as one continuous region and assigned an identifier to detect the plurality of continuous regions.

3. The information processing method according to claim 1 or 2.

6. If the two or more meshes are adjacent to each other and the texture images referenced by the two or more meshes are similar, the two or more meshes are treated as one continuous region and the identifier is assigned to the two or more meshes. The information processing method according to claim 5 .

7. using a trained model trained by machine learning to receive an image including at least a portion of the plurality of meshes as input and output attributes of the image, two or more meshes having the same attribute output by the trained model are considered as one continuous region, and the plurality of continuous regions are detected from the plurality of meshes; 3. The information processing method according to claim 1 or 2.

8. generating a virtual plane by performing principal component analysis on the coordinates of each vertex of each mesh included in the continuous region having the normals in the same direction; generating the one plane by projecting the coordinates of each vertex of each mesh onto the generated virtual plane; 3. The information processing method according to claim 1 or 2.

9. generating the one plane using a representative value of the direction of the normal of each mesh included in the continuous region having the normal in the same direction; 3. The information processing method according to claim 1 or 2.

10. When the plane can be divided into a plurality of rectangles, each of the divided rectangles is defined as a plane.

3. The information processing method according to claim 1 or 2.

11. On the computer, Executing the information processing method according to claim 1 or 2, program.

12. an acquisition unit that acquires scan data representing an object in real space using a plurality of meshes; a normal calculation unit that calculates normals of each of the plurality of meshes; a continuous region detection unit that detects a plurality of continuous regions from the plurality of meshes, with two or more continuous meshes being one continuous region; a plane generation unit that generates a plurality of planes from the plurality of meshes by treating, as one plane, a continuous region among the plurality of continuous regions having the normals in the same direction; an output unit that outputs the plurality of planes as shape data indicating a three-dimensional shape of the object, Information processing system.

Images