Texture image generation device, texture image generation method, and texture image generation program

Abstract

To provide a device, method and program for improving the work efficiency of a radio wave propagation simulation by easily grasping a simply shaped object.SOLUTION: In a texture image generation system 10, a texture image generation device includes a clustering part for clustering point group data including three-dimensional coordinates and image data on the three-dimensional coordinates into point group data belonging to a plane object and point group data belonging to a non-plane object, a shaping part for shaping the three-dimensional coordinates included in the point group data belonging to the plane object into a plane object and shaping the three-dimensional coordinates included in the point group data belonging to the non-plane object into a non-plane object, a texture image generation part for generating a texture image of the plane object on the basis of the image data included in the point group data belonging to the plane object, and generating a texture image of the non-plane object on the basis of the image data included in the point group data belonging to the non-plane object, and a display part for displaying the texture image of the plane object and the texture image of the non-plane object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a texture image generating device, a texture image generating method, and a texture image generating program. [Background technology]

[0002] Conventionally, when introducing wireless systems such as wireless LAN (Local Area Network) systems and local 5G (5th Generation) systems, simulations may be performed using three-dimensional simulation models constructed in a virtual space.

[0003] In wireless system simulations, it is sometimes necessary to simulate in a virtual space how radio waves transmitted from a wireless station are reflected, transmitted, and diffracted by structures such as walls and furniture within the target area. This type of simulation is called radio wave propagation simulation.

[0004] In radio wave propagation simulation, it is sometimes required to accurately represent the shape, size, and material of structures in the target area in a virtual space. This makes it possible to improve the accuracy of radio wave propagation simulation to a certain level.

[0005] Methods for constructing a model used in radio wave propagation simulation include, for example, the following.

[0006] That is, there is a technology for detecting planes using a "region growing method" that clusters point cloud data by expanding the region from the center and adding points with similar characteristics to the region, and then using a "RANSAC (Random Sample Consensus)" method that repeatedly extracts random samples and performs fitting using the least squares method on the clustered point cloud data to eliminate outliers (noise) (for example, Non-Patent Document 1 below).

[0007] This technology makes it possible to construct simplified shaped planar models in a virtual space. Furthermore, the simplified shapes make it possible to reduce the amount of data in the virtual space and the amount of calculations required when simplifying radio wave propagation.

[0008] On the other hand, there are techniques for generating texture images by pasting images onto objects in a virtual space. For example, the following techniques are available.

[0009] That is, there is an image processing device that selects the viewpoint with the highest resolution among multiple camera viewpoints for one object, selects multiple polygons from that viewpoint, and generates a texture map using the pixel values ​​from that viewpoint (for example, Patent Document 1 below).

[0010] There is also a three-dimensional model generation device that reads in multiple images from different viewpoints, matches feature points between the images by matching using image features, stores the three-dimensional coordinates and normal vectors of each feature point as three-dimensional point cloud data, and determines the color of pixels in occluded parts based on the average color of pixels other than the occluded parts of a target polygon, thereby generating a three-dimensional model (for example, Patent Document 2 below). [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] “Proposal of a method for creating a planar model using point cloud data of indoor environments and evaluation of radio wave propagation using the ray tracing method”, Wataru Okamura, Yuki Matsuyama, Yukiko Yoshiki, Zhihang Chen, Kentaro Saito, Junichi Takada, IEICE Technical Report, vol. 118, no. 451, AP2018-173, pp. 19-24, February 2019. [Patent documents]

[0012] [Patent Document 1] Patent Publication No. 2021-64334 [Patent Document 2] Patent No. 5997664 Summary of the Invention [Problem to be solved by the invention]

[0013] However, an operator who performs a radio wave propagation simulation (or a creator who creates a virtual space for performing a radio wave propagation simulation) may not be able to confirm the original type of object in the virtual space that has been simplified in shape by the technology of Non-Patent Document 1. If the operator cannot confirm the type of object, it takes a long time to identify the material of the object in the virtual space. As a result, the work efficiency of the radio wave propagation simulation may drop below a certain level.

[0014] Patent Document 1 describes generating a texture image for a target polygon using image data from a viewpoint with the highest resolution. Patent Document 2 describes determining a color for an occluded portion of a target polygon. However, neither of these patent documents discloses anything about confirming what kind of object the target polygon itself originally is. Thus, Patent Document 1 and Patent Document 2 do not disclose any solution for the case where an operator cannot confirm what kind of object the target polygon is.

[0015] Therefore, an object of the present disclosure is to provide a texture image generating device, a texture image generating method, and a texture image generating program that are capable of easily grasping an object with a simplified shape, and that improve the work efficiency of radio wave propagation simulation. [Means for solving the problem]

[0016] A texture image generating device according to one aspect includes a clustering unit that clusters point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object. The texture image generating device also includes a shaping unit that shapes the three-dimensional coordinates included in the point cloud data belonging to the planar object into a planar object and shapes the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into a non-planar object. The texture image generating device further includes a texture image generating unit that generates a texture image of the planar object based on image data included in the point cloud data belonging to the planar object and generates a texture image of the non-planar object based on image data included in the point cloud data belonging to the non-planar object. The texture image generating device also includes a display unit that displays the texture image of the planar object and the texture image of the non-planar object.

[0017] A texture image generating method according to one aspect is a texture image generating method in a texture image generating device. The texture image generating method includes a step of clustering point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object. The texture image generating method also includes a step of shaping the three-dimensional coordinates included in the point cloud data belonging to the planar object into a planar object, and shaping the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into a non-planar object. The texture image generating method further includes a step of generating a texture image of the planar object based on image data included in the point cloud data belonging to the planar object, and generating a texture image of the non-planar object based on image data included in the point cloud data belonging to the non-planar object. The texture image generating method further includes a step of displaying the texture image of the planar object and the texture image of the non-planar object on a display unit.

[0018] Further, a texture image generating program according to one embodiment is a texture image generating program executed by a computer of a texture image generating device. The texture image generating program causes a computer to execute a process of clustering point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object. The texture image generating program also causes a computer to execute a process of shaping three-dimensional coordinates included in the point cloud data belonging to the planar object into a planar object, and shaping three-dimensional coordinates included in the point cloud data belonging to the non-planar object into a non-planar object. The texture image generating program also causes a computer to execute a process of generating a texture image of a planar object based on image data included in the point cloud data belonging to the planar object, and generating a texture image of a non-planar object based on image data included in the point cloud data belonging to the non-planar object. The texture image generating program also causes a computer to execute a process of displaying the texture image of the planar object and the texture image of the non-planar object on a display unit. Effect of the Invention

[0019] According to the present disclosure, it is possible to provide a texture image generating device, a texture image generating method, and a texture image generating program that are capable of easily grasping an object with a simplified shape. Also, according to the present disclosure, it is possible to provide a texture image generating device, a texture image generating method, and a texture image generating program that improve the work efficiency of radio wave propagation simulation. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a texture image generating system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of point cloud data according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a virtual space according to the first embodiment. [Figure 4]FIG. 4 is a diagram illustrating an example of operation according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the operation of the texture image generation process according to the first embodiment. [Figure 6] 6(A) and 6(B) are diagrams for explaining an example of generation of a texture image according to the first embodiment. [Figure 7] FIG. 7 is a diagram for explaining an example of generation of a texture image according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of generation of a texture image according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the operation of the texture image generation process according to the first embodiment. [Figure 10] 10(A) and 10(B) are diagrams for explaining an example of generation of a texture image according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a texture image according to the first embodiment. [Figure 12] 12A and 12B are diagrams illustrating an example of point cloud data according to the first embodiment. [Figure 13] 13(A) and 13(B) are diagrams illustrating examples of texture images according to the first embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a resolution setting UI according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] [First embodiment] The texture image generating device according to the first embodiment will be described below with reference to the drawings. In the description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0022] (Example of the configuration of a texture image generating device) First, an example of the configuration of a texture image generating device according to the first embodiment will be described.

[0023] FIG. 1 is a diagram illustrating an example of the configuration of a texture image generating system 10 including a texture image generating device 100. As shown in FIG.

[0024] The texture image generating device 100 is capable of generating a three-dimensional simulation model in a virtual space based on point cloud data. The texture image generating device 100 is capable of displaying a simulation model with an image pasted on the simulation model. An image pasted on a simulation model may be referred to as a "texture image". Alternatively, a simulation model with an image pasted on it may be referred to as a "texture image". The texture image generating device 100 is capable of generating a texture image based on point cloud data. The texture image generating device 100 is capable of displaying the generated texture image on the display unit 120.

[0025] As shown in FIG. 1, the texture image generating system 10 includes an RGBD (Red, Green, Blue, and Depth) camera 200 and a texture image generating device 100.

[0026] The RGBD camera 200 captures an image of the real space that is the target of the radio wave propagation simulation model, and acquires image data from the captured image. The image data may be represented in an RGB (Red, Green, Blue) color space, or an HSV (Hue, Saturation, Value) color space. The image data may be referred to as color information. The RGBD camera 200 acquires three-dimensional coordinates for the real space that is the target of the radio wave propagation simulation model. The RGBD camera 200 may be rotatable, and may acquire image data and three-dimensional coordinates of the target real space by scanning its surroundings.

[0027] As shown in FIG. 1, the RGBD camera 200 has an image data acquisition unit 210 and a three-dimensional information acquisition unit 220.

[0028] The image data acquisition unit 210 acquires image data from a captured image, and transmits the acquired image data to the texture image generating device 100. The image data acquisition unit 210 may acquire image data of a plurality of image frames (i.e., image data of a moving image).

[0029] On the other hand, the three-dimensional information acquisition unit 220 acquires three-dimensional coordinates (for example, (x, y, z)) corresponding to the real space targeted by the radio wave propagation simulation model, and transmits the acquired three-dimensional coordinates to the texture image generating device 100. The three-dimensional information acquisition unit 220 may be a combination of a light source unit using LiDAR (Light Detection And Ranging) technology and a camera (or a light sensor). In this case, a laser light from the light source unit may be irradiated onto an object, and the reflected light may be received by the camera, and three-dimensional coordinates including depth information may be acquired from the time difference between the irradiated light and the reflected light. The three-dimensional information acquisition unit 220 may also be a stereo camera including two cameras. The stereo camera is, for example, a camera that acquires three-dimensional coordinates including depth information based on the parallax with respect to an object. In this way, the three-dimensional information acquisition unit 220 may only need to acquire three-dimensional coordinates by a known method. The three-dimensional information acquisition unit 220 may transmit image data of a depth image (or a depth image) obtained by imaging the acquired three-dimensional coordinates to the texture image generating device 100.

[0030] The image data acquisition section 210 and the three-dimensional information acquisition section 220 make it possible to simultaneously acquire image data and three-dimensional coordinates for the same object in real space.

[0031] As shown in FIG. 1, the texture image generating device 100 includes a control unit 110, a display unit 120, and a storage unit .

[0032] The control unit 110 controls each unit (the display unit 120 and the storage unit 130) of the texture image generating device 100. The control unit 110 links the image data output from the RGBD camera 200 with three-dimensional coordinates to acquire point cloud data. That is, the point cloud data includes three-dimensional coordinates and image data on the three-dimensional coordinates. For example, the point cloud data can be expressed as RGB data on three-dimensional coordinates (x, y, z). The control unit 110 generates a texture image based on the point cloud data. The control unit 110 may store the point cloud data and the texture image in the storage unit 130. The control unit 110 can also cause the display unit 120 to display a three-dimensional simulation model to which the texture image is attached.

[0033] As shown in FIG. 1, the control unit 110 includes a point cloud data acquisition unit 111, a clustering unit 112, a shaping unit 113, a texture image generation unit 114, and a parameter setting unit 115.

[0034] The point cloud data acquisition unit 111 acquires image data from the image data acquisition unit 210, and acquires three-dimensional coordinates from the three-dimensional information acquisition unit 220. The point cloud data acquisition unit 111 acquires point cloud data by linking the acquired image data and the three-dimensional coordinates. The point cloud data acquisition unit 111 may store the point cloud data in the storage unit 130. The point cloud data acquisition unit 111 outputs the point cloud data to the clustering unit 112.

[0035] The clustering unit 112 clusters the point cloud data. Specifically, the clustering unit 112 clusters the point cloud data belonging to a planar object and the point cloud data belonging to a non-planar object. The number of planar objects may be one or more. The number of non-planar objects may be one or more. Here, the planar object may be a planar object (for example, a ceiling, a floor, a wall, a desk, etc.) reflected in an image captured by the RGBD camera 200. The planar object may be a simulation model configured in a virtual space in association with a planar object reflected in an image captured by the RGBD camera 200. On the other hand, the non-planar object may be a non-planar (non-planar) object reflected in an image captured by the RGBD camera 200, or may be a simulation model configured in a virtual space in association with a non-planar object reflected in an image captured by the RGBD camera 200. Here, an example of a clustering method according to the first embodiment will be described.

[0036] (Clustering method) First, the clustering unit 112 clusters the point cloud data belonging to the planar object. The clustering unit 112 may cluster the point cloud data belonging to the planar object using the method described in Non-Patent Document 1. The method described in Non-Patent Document 1 is a clustering method based on plane detection using the RANSAC method. Here, the clustering method of the point cloud data belonging to the planar object described in Non-Patent Document 1 will be described.

[0037] That is, the clustering unit 112 first performs a filter process to thin out the point cloud data. Specifically, the clustering unit 112 arranges a three-dimensional cubic lattice of a predetermined size in the point cloud data, regards a center of gravity existing in the cubic lattice as an approximate point of the point cloud data existing in the cubic lattice, and replaces the point cloud data with the center of gravity. Next, the clustering unit 112 performs a segmentation process (or clustering process) on the point cloud data after the filter process. Specifically, the clustering unit 112 calculates a normal vector of a region centered on a certain point (point cloud data), compares it with the normal vector of an adjacent region, and expands the planar region by regarding the adjacent region as the same region as the region if the angle between the two normal vectors is equal to or less than a threshold. Such a method is called a "region growing method." The clustering unit 112 may treat the point cloud data clustered by the "region growing method" as point cloud data belonging to a planar object. The "region growing method" is just one example, and the clustering unit 112 may use other known methods to cluster point cloud data belonging to a planar object.

[0038] Secondly, the clustering unit 112 clusters the point cloud data belonging to the non-planar object. The clustering unit 112 may perform clustering using DBSCAN (Density-based spatial clustering of applications with noise) for non-planar objects. For example, the clustering unit 112 may perform clustering using DBSCAN for point cloud data that could not be clustered in the clustering process using the "region growing method" according to the above-mentioned Non-Patent Document 1. DBSCAN is a density-based clustering method that determines, for example, reachable points (point cloud data) within a predetermined distance (or a predetermined radius) from a core point (point cloud data) as the same cluster, and grows the cluster. The clustering unit 112 may cluster the point cloud data belonging to the non-planar object using a known clustering method other than DBSCAN.

[0039] The clustering method has been described above.

[0040] The shaping unit 113 uses each clustered point cloud data to shape a planar object and a non-planar object. Specifically, the shaping unit 113 shapes three-dimensional coordinates included in the point cloud data belonging to a planar object into a planar object. The shaping unit 113 also shapes three-dimensional coordinates included in the point cloud data belonging to a non-planar object into a non-planar object. The shaping in the shaping unit 113 may be referred to as "simple shaping" below. The simple shaping will be described below.

[0041] (Simple shape) First, the shaping unit 113 shapes the three-dimensional coordinates of the point cloud data clustered as point cloud data belonging to a planar object into a planar object. For example, the shaping unit 113 may shape the point cloud data into a planar object using the RANSAC method described in Non-Patent Document 1. The RANSAC method is, for example, a method of learning model parameters by excluding the influence of outliers (or noise values) from data including outliers. Here, the model is a planar model represented by ax+by+cz+d=0. The clustering unit 112 performs model detection of each planar object, that is, plane detection, by obtaining parameters a, b, c, and d for each planar object using the RANSAC method. For example, a certain point cloud data clustered as belonging to a planar object is shaped into a planar object represented by a1x+b1y+c1z+d1=0, and another point cloud data clustered as belonging to a planar object is shaped into a planar object represented by a2x+b2y+c2z+d2=0. The shaping unit 113 may classify the detected planar object model into three categories, "horizontal plane", "vertical plane", and "ceiling or floor", based on the parameters a, b, c, and d, and construct a planar object model based on each of the classified categories. The shaping unit 113 may construct a part of a simulation model in a virtual space (ceiling, floor, wall surface, etc.) based on the planar object model.

[0042] Secondly, the shaping unit 113 shapes the three-dimensional coordinates of the point cloud data clustered as point cloud data belonging to a non-planar object into a non-planar object. For example, the shaping unit 113 may shape the three-dimensional coordinates into a non-planar object using a convex hull. The convex hull is a method of constructing the smallest convex polygon (or convex polyhedron) that contains all of the given three-dimensional coordinates. The shaping unit 113 may shape the three-dimensional coordinates into a non-planar object using, for example, an "α-shape" (or alpha shape) other than the convex hull. The α-shape is a method of creating an object (i.e., a non-planar object) using the edge connecting the two points when a generalized disk with a certain radius 1 / α exists for two points of the point cloud data, the point cloud data is not included inside the disk, and the boundary of the disk includes the two points of the point cloud data. A convex hull is obtained when α=∞, and point cloud data is obtained when α=0. α-shape is also a generalization of the convex hull.

[0043] The above describes the simplified shape technique.

[0044] FIG. 2 is a diagram showing an example of point cloud data according to the first embodiment. FIG. 3 is a diagram showing an example of a simplified simulation model according to the first embodiment. FIG. 3 shows a simulation model in a virtual space generated based on the point cloud data of FIG. 2. In the example shown in FIG. 3, colors are displayed for each planar object and each non-planar object. For example, when a simplified shape is generated into a planar object, the shaping unit 113 sets a representative value for image data of a plurality of point cloud data belonging to the planar object, and causes the display unit 120 to display the representative value of the image data as color information of the planar object.

[0045] However, for some of the objects shown in Figure 3, it is not clear exactly what the original object represents.

[0046] Therefore, in the first embodiment, a texture image for a planar object and a texture image for a non-planar object are generated in the texture image generating unit 114. That is, the texture image generating unit 114 generates a texture image of the planar object based on image data included in the point cloud data belonging to the planar object. Also, the texture image generating unit 114 generates a texture image of the non-planar object based on image data included in the point cloud data belonging to the non-planar object.

[0047] In this way, in the first embodiment, the texture image of the planar object and the texture image of the non-planar object are generated separately, so that the operator who operates the texture image generating device 100 can separately grasp what the planar object and the non-planar object were originally like. Note that the texture image generating unit 114 may be divided into two blocks, a block that generates the texture image of the planar object and a block that generates the texture image of the non-planar object. A method of generating a texture image for each object will be described in the operation example.

[0048] Returning to Fig. 1, the parameter setting unit 115 sets parameters for generating a texture image. For example, the parameter setting unit 115 sets the resolution of a texture image for a planar object. Fig. 14 is a diagram showing an example of a setting screen for setting the resolution of a texture image for a planar object. Details will be described later. The parameter setting unit 115 can display the parameter setting screen on the display unit 120.

[0049] Returning to Fig. 1, the display unit 120 displays various data or information under the control of the control unit 110. The display unit 120 displays a simulation model to which a texture image is attached, which is generated by the control unit 110. The display unit 120 is also capable of displaying a setting screen for parameter setting.

[0050] The storage unit 130 stores various data or information under the control of the control unit 110. The storage unit 130 may store point cloud data. The storage unit 130 may also store three-dimensional coordinates representing a simplified planar object and three-dimensional coordinates representing a simplified non-planar object. The storage unit 130 may also store texture image data.

[0051] (Operation example according to the first embodiment) Next, an operation example according to the first embodiment will be described.

[0052] 4 is a diagram illustrating an example of operation according to the first embodiment. The example of operation illustrated in FIG.

[0053] As shown in FIG. 4, in step S10, control unit 110 starts the process.

[0054] In step S11, the control unit 110 acquires point cloud data. Specifically, the point cloud data acquisition unit 111 associates the image data and three-dimensional coordinates output from the RGBD camera 200 and acquires them as point cloud data.

[0055] In step S12, the control unit 110 clusters the point cloud data. Specifically, the clustering unit 112 clusters the point cloud data acquired by the point cloud data acquisition unit 111 into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object. For example, the clustering unit 112 may use the "region growing method" described in Non-Patent Document 1 to cluster the point cloud data acquired by the point cloud data acquisition unit 111 into point cloud data belonging to a planar object. The clustering unit 112 may perform clustering of the point cloud data belonging to a non-planar object by performing clustering by DBSCAN on the point cloud data that could not be clustered as point cloud data belonging to a planar object.

[0056] In step S13, the control unit 110 performs the processes of steps S14 and S15 for each of the clustered objects.

[0057] That is, in step S14, the shaping unit 113 shapes (i.e., simplifies) the three-dimensional coordinates included in the point cloud data belonging to a planar object into a planar object. Also, the shaping unit 113 shapes (i.e., simplifies) the three-dimensional coordinates included in the point cloud data belonging to a non-planar object into a non-planar object. Specifically, the shaping unit 113 may use the RANSAC method described in Non-Patent Document 1 to shape the three-dimensional coordinates included in the point cloud data belonging to a planar object into a planar object. Also, the shaping unit 113 may use an α-shape to shape the three-dimensional coordinates included in the point cloud data belonging to a non-planar object into a non-planar object.

[0058] In step S15, the control unit 110 generates a texture image for the simplified planar object. Also, the control unit 110 generates a texture image for the simplified non-planar object. Below, the generation process of the texture image for the planar object and the generation process of the texture image for the non-planar object will be described separately.

[0059] (Texture image generation processing for planar objects) Fig. 5 is a diagram showing an example of the operation of texture image generation processing according to the first embodiment. Fig. 5 shows an example of the operation of texture image generation processing for a planar object. Before explaining Fig. 5, an overview of the processing performed in the texture image generation processing will be explained using Fig. 6(A) and Fig. 6(B).

[0060] Figures 6(A) and 6(B) are diagrams showing an example of generation of a texture image according to the first embodiment. The planar object shown in Figure 6(A) represents a planar object obtained by simplifying the shape of the planar object from three-dimensional coordinates of point cloud data clustered as point cloud data belonging to the planar object. Original point cloud data exists around the planar object.

[0061] In the first embodiment, a texture image is generated by projecting original point cloud data onto a simplified planar object. Specifically, a position on the planar object where a perpendicular line is drawn from the three-dimensional coordinates included in the point cloud data belonging to the planar object to the planar object is set as image data of the point cloud data, thereby forming a texture image on the planar object. FIG. 6(B) shows an example in which image data of the point cloud data is projected onto a planar object. This allows the texture image generating device 100 to generate a texture image by directly using image data included in the original point cloud data, for example, and generate a texture image according to the color included in the point cloud data.

[0062] The operation of generating a texture image will be described with reference back to Fig. 5. The operation example shown in Fig. 5 is mainly performed by the texture image generating unit 114.

[0063] In step S160, the texture image generating unit 114 starts the process.

[0064] In step S161, the texture image generating unit 114 determines the resolution of the texture image. FIG. 7 shows a view of a simplified planar object viewed from above. FIG. 7 shows a view of the surface of the planar object shown in FIG. 6(A) or FIG. 6(B) viewed from above. FIG. 7 also shows the coordinate system of the texture image formed on the surface of the planar object (i.e., the texture image coordinate system). Finally, the texture image is generated in the texture image coordinate system. In the texture image coordinate system, the vertical direction is represented by u and the horizontal direction is represented by v. In step S161, the texture image generating unit 114 determines the resolution of the texture image, that is, the number of pixels in the vertical direction u and the number of pixels in the horizontal direction v. In the first embodiment, the texture image generating unit 114 determines the resolution of the texture image according to the size of the object in the real space. For example, when the size of the object is "3m" vertically and "4m" horizontally, and 1 cm per pixel, the vertical direction u is "300 pixels" and the horizontal direction v is "400 pixels". In this case, in FIG. 7, the size of the rectangle on the surface of the planar object is 300 pixels in the vertical direction u and 400 pixels in the horizontal direction v. For example, if the size of the object is 0.5 m in the vertical direction and 1 m in the horizontal direction, and if one pixel is 1 cm, the size of the rectangle on the surface of the planar object can be 50 pixels in the vertical direction u and 100 pixels in the horizontal direction v in FIG. 7. When the resolution of the texture image is fixed, the texture image of an object that has a larger area than other objects, such as a ceiling or floor, may have a relatively lower resolution than the other objects, and the texture image may be displayed coarsely. Therefore, by displaying the texture image at a resolution according to the size of the object, it is possible to avoid the situation where the texture image is displayed coarsely and display the texture image at a resolution according to the size of the object.

[0065] It is assumed that the size of the object is known in advance by the operator who operates the texture image generating device 100. The operator can input the size of the object to the texture image generating device 100 using a setting screen displayed on the display unit 120. This enables the texture image generating unit 114 to display the texture image of the planar object on the display unit 120 at a resolution according to the size of the object.

[0066] Also, although a rectangle has been used as an example of the size of an object, the object may be a polygon other than a rectangle, such as a quadrangle, etc. For example, if the planar object is a triangle, the texture image generating unit 114 may process the base and height as the size, and if the planar object is a polygon with pentagons or more sides, the texture image generating unit 114 may process the maximum size in the vertical and horizontal directions as the size.

[0067] Returning to FIG. 5, in step S162, the texture image generating unit 114 performs the processes of steps S163 to S165 on each piece of point cloud data belonging to the planar object.

[0068] That is, in step S163, the texture image generating unit 114 obtains the coordinates of the foot of a perpendicular line to the planar object. Specifically, the texture image generating unit 114 obtains the coordinates of a position on the planar object where a perpendicular line is dropped from the three-dimensional coordinates included in the point cloud data to the planar object. The position H(x, y, z) on the planar object can be obtained, for example, by the following formula.

[0069]

number

[0070]

number

[0071] In step S164, the texture image generating unit 114 determines which pixel in the texture image the foot of the perpendicular line corresponds to. The position H(x, y, z) on the planar object determined by Equation 1 and Equation 2 represents a position in the xyz coordinate system. Therefore, in step S164, the texture image generating unit 114 performs a process of converting the position H on the planar object from the xyz coordinate system to the texture image coordinate system (e.g., FIG. 7). The coordinate system may be converted from the xyz coordinate system (three-dimensional) to the texture image coordinate system H(u, v) (two-dimensional) by using parallel projection that discards the z coordinate of the position H(x, y, z) on the planar object determined by Equation 1 and Equation 2. Alternatively, the coordinate system may be converted from the xyz coordinate system (three-dimensional) to the texture image coordinate system (two-dimensional) by using perspective projection with an arbitrary point on the z axis as the viewpoint or perspective projection with respect to an arbitrary viewpoint. The texture image generating unit 114 may convert the coordinate system using a known determinant used in perspective projection. As shown in Fig. 7, the coordinate system is converted to obtain the coordinates H(U,V) of the foot of the perpendicular line. Note that the viewpoint position of the texture image coordinate system may be in the upward direction relative to the planar object.

[0072] 5, in step S165, the texture image generating unit 114 colors the coordinates of the foot of the perpendicular line with image data (i.e., color information) of the point cloud data. Specifically, the texture image generating unit 114 sets the coordinates of the foot of the perpendicular line relative to the three-dimensional coordinates of the point cloud data as image data included in the point cloud data. In other words, the image data included in the point cloud data becomes image data of the coordinates of the foot of the perpendicular line.

[0073] There may be a plurality of three-dimensional coordinates of point cloud data corresponding to the coordinates of the foot of the perpendicular line (hereinafter, sometimes referred to as H(U,V)). For example, in FIG. 7, there may be a plurality of point cloud data with different heights as viewed from the planar object (or as viewed from the texture image coordinate system) for the coordinates H(U,V) of the foot of the perpendicular line. In such a case, the texture image generating unit 114 may set image data of any one of the plurality of point cloud data as image data at H(U,V), or may set image data of the point cloud data that is closest to the planar object as image data at H(U,V). Alternatively, the texture image generating unit 114 may set the average value (or median value) of the image data of the plurality of point cloud data as image data at H(U,V).

[0074] In addition, there may be cases where the three-dimensional coordinates of the point cloud data corresponding to the coordinate H(U,V) of the foot of the perpendicular line do not exist in the texture image coordinate system. In such cases, the texture image generating unit 114 may set the color of the pixel to blank (black or white) or may determine the color based on the image data of the surrounding pixels (i.e., pixels associated with the coordinates near the coordinate H(U,V) of the foot of the perpendicular line).

[0075] In step S166, the texture image generating unit 114 repeats steps S163 to S165 until texture images are generated for all point cloud data belonging to the planar object. Then, when the texture image generating unit 114 has generated texture images for all point cloud data belonging to the planar object, the texture image generating unit 114 ends the generation process of the texture image for the planar object (step S167).

[0076] FIG. 8 is a diagram showing an example of a texture image according to the first embodiment. In FIG. 8, the upper left shows point cloud data, and two texture images corresponding to the point cloud data included in the white square among the point cloud data are displayed on the display unit 120. As shown in FIG. 8, in the texture image formed for a planar object such as a window or a desk, each pixel is colored with a color based on the image data of the point cloud data belonging to the planar object, and the texture image is displayed with a resolution according to the actual size of the planar object (the window or the desk in FIG. 8). Therefore, an operator who operates the texture image generating device 100 can easily grasp what kind of object the planar object is by checking the texture image displayed on the display unit 120.

[0077] (Texture image generation processing for non-surface objects) Next, the process of generating a texture image for a non-planar object will be described.

[0078] 9 is a diagram illustrating an example of the operation of the texture image generation process for a non-planar object according to the first embodiment. The example of the operation shown in FIG.

[0079] As shown in FIG. 9, in step S170, the texture image generating unit 114 starts the process.

[0080] In step S171, the texture image generating unit 114 determines the resolution of the texture image for the non-planar object. In the first embodiment, the texture image for the non-planar object is obtained by coloring each surface of the simplified non-planar object with a single color. Therefore, the texture image generating unit 114 sets the resolution of the texture image for the non-planar object to "1 pixel" in the vertical direction and "1 pixel" in the horizontal direction. In other words, the resolution for the entire non-planar object is expressed as "1 pixel" in the vertical direction x "the number of surfaces that constitute the non-planar object" in the horizontal direction. Note that a non-planar object simplified by a method such as α-shape is composed of multiple surfaces including vertices. For example, a non-planar object includes multiple triangular surfaces having at least three vertices. In the first embodiment, such surfaces are colored with a specific color to generate a texture image for the non-planar object.

[0081] In step S172, the texture image generating unit 114 performs the processes in steps S173 to S177 for each surface included in the non-planar object.

[0082] In step S173, the texture image generating unit 114 performs the process of step S174 for each vertex that constitutes the face.

[0083] That is, in step S174, the texture image generating unit 114 obtains the colors of the vertices.

[0084] Fig. 10(A) shows an example of a non-planar object according to the first embodiment, and Fig. 10(B) shows an example of a texture image for the non-planar object according to the first embodiment. In the example shown in Fig. 10(A), a surface is composed of three vertices. Therefore, the texture image generating unit 114 obtains image data included in the point cloud data corresponding to each vertex from the point cloud data.

[0085] In step S175, the texture image generating unit 114 repeats step S174 until it has acquired image data of all vertices included in the face. When the texture image generating unit 114 has acquired image data of all vertices included in the face, it performs the process of step S176.

[0086] In step S176, the texture image generating unit 114 determines the color of the face based on the image data of each vertex. For example, the texture image generating unit 114 may determine the average value of the image data of each vertex (three points in the example of FIG. 10(A)) as the color of the face, or may determine the median value of the image data of each vertex as the color of the face. Furthermore, the texture image generating unit 114 may determine the color of the face using a value obtained by converting the image data into the HSV color space, or may determine the color of the face using a value obtained by converting the image data into the RGB color space.

[0087] In step S177, the texture image generating unit 114 colors each pixel included in the surface with the surface color determined in step S176. For example, the texture image generating unit 114 colors the surface by setting all pixels included in the surface to image data represented in the surface color determined in step S176. In this way, the texture image generating unit 114 generates a texture image of each surface included in the non-planar object based on the image data of the point cloud data included in each vertex.

[0088] FIG. 10(B) shows an example of a texture image for the point cloud data for FIG. 10(A). As shown in FIG. 10(B), the texture image generating unit 114 may assign identification information (hereinafter, sometimes referred to as "surface ID") to each surface, and generate a texture image in a format corresponding to each identification information. For example, the color of a surface with a surface ID of "0" corresponds to image data at pixel (0, 0) of the texture image, and the color of a surface with a surface ID of "1" corresponds to image data at pixel (0, 1) of the texture image, and so on. That is, the texture image generating unit 114 corresponds to image data at pixel (0, i) of the texture image for a color with a surface ID of "i" (i is an integer equal to or greater than 0). For example, the texture image generating unit 114 stores the texture image shown in FIG. 10(B) in the storage unit 130, and colors the non-planar object based on the texture image, thereby displaying the non-planar object with each surface colored as shown in FIG. 10(A) on the display unit 120.

[0089] Fig. 11 is a diagram showing an example of point cloud data and a texture image according to the first embodiment. For a non-planar object (a chair in Fig. 11) included in a white-lined square in the point cloud data, each surface included in the non-planar object is colored with a color shown in the texture image. For a non-planar object, image data of the point cloud data corresponding to vertices included in each surface is used as a texture image. Therefore, an operator who sees a non-planar object displayed on the display unit 120 can easily grasp what kind of object the non-planar object is.

[0090] In step S178, when the processes from step S173 to step S177 are completed for each surface after the simplified shape, the texture image generating unit 114 ends the texture image generation process for the non-planar object in step S179.

[0091] Returning to FIG. 4, in step S18, when the processes of steps S14 and S15 are completed for each object, the control unit 110 ends the series of processes in step S19.

[0092] (Example of final result) Fig. 12(A) and Fig. 12(B) are diagrams showing examples of point cloud data according to the first embodiment. Of these, Fig. 12(A) shows the point cloud data itself, and Fig. 12(B) shows an example of point cloud data obtained by enlarging the area shown by the white square in the point cloud data shown in Fig. 12(A). In Fig. 12(B), the part shown by the square shows the area of ​​a planar object (a desk in Fig. 12(B)). Also, in Fig. 12(B), the part shown by the dotted square shows the area of ​​a non-planar object (a chair in Fig. 12(B)).

[0093] On the other hand, Fig. 13(A) shows an example of an image after performing a shape simplification process (step S14 in Fig. 4) on the point cloud data shown in Fig. 12(A) and performing a texture image generation process (step S15 in Fig. 4). Also, Fig. 13(B) shows an example of an image obtained by enlarging the white square area shown in Fig. 13(A).

[0094] As shown in Fig. 13(B), the desk, which is a planar object, is displayed with a resolution according to the size of the planar object, and the color of the desk's surface is roughly reproduced. The chair, which is a non-planar object, is colored by taking the average of the colors of the vertices of each face, and although it is somewhat dark, it is clear that the color of the chair is roughly reproduced. When an operator sees the images of the planar object and non-planar object reproduced in this way, the operator can easily distinguish between the desk and the chair.

[0095] (Parameter settings) It is conceivable that by changing the parameters of the above operations, the final result may be improved. For example,

[0096] (1) Instead of using all point cloud data belonging to a planar object, a texture image is generated using point cloud data whose distance from the planar object is within a threshold value x, and point cloud data whose distance from the planar object exceeds the threshold value x is not used to generate the texture image. This is expected to increase the accuracy of coloring the planar object, since, for example, point cloud data that does not actually belong to the planar object is not used. For example, the parameter setting unit 115 may display a setting screen on the display unit 120 where the threshold value x can be input, so that the operator can input the threshold value x.

[0097] (2) The resolution of the texture image for the planar object may be set by a setting screen. For example, the display unit 120 may display a screen so that it can be set whether the resolution of the texture image for the planar object is fixed or variable according to the actual size of the planar object. Also, for example, when the resolution of the planar object is variable, the display unit 120 may display a screen so that it can be set as to how many mm per pixel of the texture image. FIG. 14 is a diagram showing an example of a resolution setting screen. In FIG. 14, the resolution can be set when the resolution is fixed. After the texture image generation process is completed, the setting may be changed using the setting screen shown in FIG. 14, and the final result according to the changed setting may be displayed on the display unit 120.

[0098] (3) When the texture image generating device 100 generates a texture image for a planar object, if there is one position on the planar object for a plurality of point cloud data, the color to be colored at that position may be set by a setting screen. For example, an arbitrary color may be set, or the average or median value of the image data of the plurality of point cloud data may be set. Alternatively, it may be possible to set which color space (e.g., RGB color space, HSV color space, etc.) to be used. The parameter setting unit 115 may display a setting screen on the display unit 120, and the setting may be possible via the setting screen.

[0099] (4) When generating a texture image for a planar object using the texture image generating device 100, if there is no point cloud data corresponding to a pixel on the planar object, it may be possible to set on a setting screen whether to make the pixel black, white, or transparent.

[0100] What is common to the above (1) to (4) is that after the texture image generation process is completed and the final result is displayed, the settings can be changed using the settings screen (also called retry or tuning), and the final result can be displayed according to the changed settings.

[0101] [Other embodiments] In the first embodiment, a radio wave propagation simulation has been described as a simulation target, but the present invention is not limited thereto. As an example of the simulation, the simulation may be used for any of acoustic design, sound insulation and soundproofing design, and air conditioning design in a facility such as a studio or an acoustic hall. Even for such targets, the texture image generating device 100 can generate a texture image for the object that is the target of the simulation model, as in the first embodiment.

[0102] In the first embodiment, an example has been described in which image data of a real space and three-dimensional coordinates corresponding to the image data are acquired by the RGBD camera 200 having the image data acquisition unit 210 and the three-dimensional information acquisition unit 220, but the present invention is not limited to this. For example, image data of a real space may be acquired by the RGBD camera 200, and three-dimensional coordinates corresponding to the image data may be acquired by a depth camera such as a LiDAR.

[0103] Also, a program may be provided that causes a computer to execute each process performed by the control unit 110 described in the first embodiment. The program may be recorded in a computer-readable medium. Using the computer-readable medium, it is possible to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Also, circuits that execute each process performed by the control unit 110 may be integrated, and at least a part of the texture image generating device 100 may be configured as a semiconductor integrated circuit (chip set, SoC).

[0104] Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope of the gist. In addition, it is also possible to combine all or part of each embodiment, each operation, each process, and each step within the scope of not contradicting each other.

[0105] (Additional Note) (Appendix 1) a clustering unit that clusters point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object; a shaping unit that shapes the three-dimensional coordinates included in the point cloud data belonging to the planar object into the planar object, and shapes the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into the non-planar object; a texture image generating unit that generates a texture image of the planar object based on the image data included in the point cloud data belonging to the planar object, and generates a texture image of the non-planar object based on the image data included in the point cloud data belonging to the non-planar object; a display unit that displays the texture image of the planar object and the texture image of the non-planar object.

[0106] (Appendix 2) The texture image generating unit generates a texture image of the planar object by projecting the image data included in the point cloud data belonging to the planar object onto the planar object. 2. The texture image generating device according to claim 1.

[0107] (Appendix 3) The texture image generating unit generates a texture image of the planar object by taking a position on the planar object, where a perpendicular line is dropped from the three-dimensional coordinates included in the point cloud data belonging to the planar object toward the planar object, as image data of the point cloud data. 3. The texture image generating device according to claim 1 or 2.

[0108] (Appendix 4) The display unit displays a texture image of the planar object at a resolution according to a size of the planar object in real space. 4. A texture image generating device according to claim 1,

[0109] (Appendix 5) The display unit displays a resolution setting screen for setting the resolution. 5. A texture image generating device according to claim 1,

[0110] (Appendix 6) the non-planar object is composed of a plurality of faces including vertices; The texture image generating unit generates the texture image of each surface included in the non-planar object based on image data of the point cloud data included in the vertices. 6. A texture image generating device according to claim 1,

[0111] (Appendix 7) A texture image generating method in a texture image generating device, comprising: A step of clustering point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object; shaping the three-dimensional coordinates included in the point cloud data belonging to the planar object into the planar object, and shaping the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into the non-planar object; generating the texture image of the planar object based on the image data included in the point cloud data belonging to the planar object, and generating the texture image of the non-planar object based on the image data included in the point cloud data belonging to the non-planar object; and displaying the texture image of the planar object and the texture image of the non-planar object on a display unit. Texture image generation method.

[0112] (Appendix 8) A texture image generating program executed on a computer of a texture image generating device, A process of clustering point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to planar objects and point cloud data belonging to non-planar objects; A process of shaping the three-dimensional coordinates included in the point cloud data belonging to the planar object into the planar object, and shaping the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into the non-planar object; generating the texture image of the planar object based on the image data included in the point cloud data belonging to the planar object, and generating the texture image of the non-planar object based on the image data included in the point cloud data belonging to the non-planar object; and displaying a texture image of the planar object and a texture image of the non-planar object on a display unit. Texture image generation program. [Explanation of symbols]

[0113] 100: Texture image generating device 110: Control unit 111: Point cloud data acquisition unit 112: Clustering unit 113: Shape generation unit 114: Texture image generation unit 115: Parameter setting section 120: Display section 130: Memory unit 200: RGBD camera

Claims

1. a clustering unit that clusters point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object; a shaping unit that shapes the three-dimensional coordinates included in the point cloud data belonging to the planar object into the planar object, and shapes the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into the non-planar object; a texture image generating unit that generates a texture image of the planar object based on the image data included in the point cloud data belonging to the planar object, and generates a texture image of the non-planar object based on the image data included in the point cloud data belonging to the non-planar object; a display unit that displays the texture image of the planar object and the texture image of the non-planar object. Texture image generator.

2. The texture image generating unit generates a texture image of the planar object by projecting the image data included in the point cloud data belonging to the planar object onto the planar object.

2. The texture image generating device according to claim 1.

3. The texture image generating unit generates a texture image of the planar object by taking a position on the planar object, where a perpendicular line is dropped from the three-dimensional coordinates included in the point cloud data belonging to the planar object toward the planar object, as image data of the point cloud data.

3. The texture image generating device according to claim 2.

4. The display unit displays a texture image of the planar object at a resolution according to a size of the planar object in real space.

2. The texture image generating device according to claim 1.

5. The display unit displays a resolution setting screen for setting the resolution.

5. The texture image generating device according to claim 4.

6. the non-planar object is composed of a plurality of faces including vertices; The texture image generating unit generates the texture image of each surface included in the non-planar object based on image data of the point cloud data included in the vertices.

2. The texture image generating device according to claim 1.

7. A texture image generating method in a texture image generating device, comprising: A step of clustering point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to a planar object and point cloud data belonging to a non-planar object; shaping the three-dimensional coordinates included in the point cloud data belonging to the planar object into the planar object, and shaping the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into the non-planar object; generating the texture image of the planar object based on the image data included in the point cloud data belonging to the planar object, and generating the texture image of the non-planar object based on the image data included in the point cloud data belonging to the non-planar object; and displaying the texture image of the planar object and the texture image of the non-planar object on a display unit. Texture image generation method.

8. A texture image generating program executed on a computer of a texture image generating device, A process of clustering point cloud data including three-dimensional coordinates and image data on the three-dimensional coordinates into point cloud data belonging to planar objects and point cloud data belonging to non-planar objects; A process of shaping the three-dimensional coordinates included in the point cloud data belonging to the planar object into the planar object, and shaping the three-dimensional coordinates included in the point cloud data belonging to the non-planar object into the non-planar object; generating the texture image of the planar object based on the image data included in the point cloud data belonging to the planar object, and generating the texture image of the non-planar object based on the image data included in the point cloud data belonging to the non-planar object; and displaying a texture image of the planar object and a texture image of the non-planar object on a display unit. Texture image generation program.

Images