Three-dimensional point cloud densification device, three-dimensional point cloud densification method, and program

Abstract

A three-dimensional point cloud densification device (100) comprises: an acquisition unit (110) that acquires a plurality of three-dimensional points each indicating a three-dimensional position of at least one object, and auxiliary information for creating a polygon; and a densification unit (130) that specifies, on the basis of the auxiliary information, a polygon having at least three of the plurality of three-dimensional points as vertices, and that generates a new three-dimensional point in the specified polygon.

Description

[Technical Field]

[0001] The present disclosure relates to a three-dimensional point cloud densification device, a three-dimensional point cloud densification method, and a program. [Background technology]

[0002] In recent years, there has been a technology that measures the three-dimensional shape of a subject (object) such as a person or a building using multi-viewpoint images obtained by photographing the object from multiple viewpoints, and generates a three-dimensional model of the object consisting of multiple three-dimensional points (three-dimensional point cloud) (see, for example, Patent Document 1).

[0003] The technology disclosed in Patent Document 1 geometrically estimates the camera parameters (camera position, orientation, focal length in pixels, lens distortion, etc.) when each image constituting a multi-view image was captured, and then generates a three-dimensional model of the object by projecting each pixel on each image into three-dimensional space using the estimated camera parameters. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-056142 Summary of the Invention [Problem to be solved by the invention]

[0005] The three-dimensional points are generated based on, for example, characteristic positions (feature points) of objects included in a plurality of images.

[0006] Here, if the number of feature points is small, the number of points in the 3D point cloud that constitutes the object will be small. In this case, new 3D points are added to the 3D point cloud as 3D points that constitute the 3D model. This generates a 3D point cloud with a high density of points, thereby generating a 3D model that more closely resembles the subject in appearance.

[0007] In this way, when a new 3D point is added to a 3D point group that constitutes a 3D model, it is necessary to add the 3D point at an appropriate position.

[0008] The present disclosure provides a 3D point cloud densification device and the like that can generate 3D points at appropriate positions. [Means for solving the problem]

[0009] A three-dimensional point cloud densification device according to one embodiment of the present disclosure includes an acquisition unit that acquires a plurality of three-dimensional points, each of which indicates the three-dimensional position of at least one object, and auxiliary information for creating a polygon, and a densification unit that, based on the auxiliary information, identifies a polygon having at least three of the plurality of three-dimensional points as vertices, and generates a new three-dimensional point within the polygon.

[0010] Furthermore, a three-dimensional point cloud densification method according to one embodiment of the present disclosure includes an acquisition step of acquiring a plurality of three-dimensional points, each of which indicates the three-dimensional position of at least one object, and auxiliary information for creating a polygon; an identification step of identifying a polygon having at least three of the plurality of three-dimensional points as vertices based on the auxiliary information; and a densification step of generating new three-dimensional points within the polygon.

[0011] The present disclosure may be realized as a program that causes a computer to execute the steps included in the above-described 3D point cloud densification method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network, such as the Internet. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to provide a three-dimensional point cloud densification device and the like that can generate three-dimensional points at appropriate positions. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram for explaining a three-dimensional point cloud generation system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of a three-dimensional point cloud generation system according to an embodiment. [Figure 3] FIG. 3 is a diagram for explaining a dense 3D point cloud generated by a 3D point cloud generation system according to an embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of a three-dimensional point cloud densification device according to an embodiment. [Figure 5] FIG. 5 is a sequence diagram showing the processing procedures of each unit included in the three-dimensional point cloud densification device according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the acquisition process of the three-dimensional point cloud densification device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of an image displayed by a UI unit included in the three-dimensional point cloud densification device according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of an image when a user operates a UI unit included in the three-dimensional point cloud densification device according to the embodiment. [Figure 9] FIG. 9 is a diagram showing another example of an image when the UI unit included in the three-dimensional point cloud densification device according to the embodiment is operated by a user. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for generating auxiliary information executed by a three-dimensional point cloud densification device according to an embodiment. [Figure 11] FIG. 11 is a diagram for explaining another example of the method for generating auxiliary information executed by the three-dimensional point cloud densification device according to the embodiment. [Figure 12] FIG. 12 is a diagram for explaining a method of capturing images using a camera provided in the three-dimensional point cloud generation system according to the embodiment. [Figure 13] FIG. 13 is a diagram for explaining the densification process executed by the three-dimensional point cloud densification device according to the embodiment. [Figure 14] FIG. 14 is a flowchart illustrating a processing procedure of the three-dimensional point cloud densification device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Summary of the Disclosure) A three-dimensional point cloud densification device according to one embodiment of the present disclosure includes an acquisition unit that acquires a plurality of three-dimensional points, each of which indicates the three-dimensional position of at least one object, and auxiliary information for creating a polygon, and a densification unit that, based on the auxiliary information, identifies a polygon having at least three of the plurality of three-dimensional points as vertices, and generates a new three-dimensional point within the polygon.

[0015] This allows the densification unit to identify a polygon having at least three appropriate 3D points as vertices based on the auxiliary information, and therefore the densification unit can generate new 3D points at appropriate positions by adding (generating) 3D points within the identified polygon.

[0016] Also, for example, the acquisition unit acquires the auxiliary information via an operation device for receiving instructions from a user, and the instructions specify the shape of the polygon.

[0017] According to this, the densification unit acquires a designation of the polygon shape from the user and specifies the polygon, so that new three-dimensional points can be generated at positions desired by the user.

[0018] Also, for example, the three-dimensional point cloud densification device according to the present disclosure further includes a generation unit that generates the auxiliary information based on image data obtained by photographing the at least one object.

[0019] Also, for example, the generation unit generates type information indicating the types of multiple objects included in the image data as the auxiliary information, and assigns the type information corresponding to each of the multiple three-dimensional points to each of the multiple three-dimensional points, and the densification unit identifies the polygon having vertices made up of at least three three-dimensional points of the same object type.

[0020] This allows the densification unit to generate new 3D points so as to complement between 3D points that belong to the same type of object.

[0021] Also, for example, the generation unit generates edges of objects included in the image data as the auxiliary information, and the densification unit identifies the polygons that do not intersect with the edges.

[0022] This allows the densification unit to suppress the generation of new three-dimensional points between different objects at positions where no object exists.

[0023] Also, for example, the acquisition unit acquires camera information indicating the position and attitude of the camera at the time of shooting that generated the multiple image data obtained by photographing the at least one object as the auxiliary information, and the densification unit identifies the polygon based on the camera information.

[0024] Also, for example, the densification unit identifies the polygon having vertices of at least three of the multiple three-dimensional points located below a predetermined position based on the up and down direction in a virtual three-dimensional space representing the multiple three-dimensional points, which is determined based on the camera information.

[0025] According to this, the densification unit can, for example, determine the type of object by considering multiple three-dimensional points located below among the three-dimensional points included in the three-dimensional point cloud as the ground such as a road, and identify polygons whose vertices are three-dimensional points of the same object type.

[0026] Also, for example, the densification unit determines the height of the camera in the up-down direction based on the camera information, and determines the predetermined position below the determined height.

[0027] This allows the densification unit to more appropriately identify 3D points whose object type is ground based on the height of the camera.

[0028] Also, for example, the densification unit identifies the polygon based on the angle between the normal direction of the polygon having at least three vertices of the plurality of three-dimensional points located below the specified position and the up-down direction.

[0029] For example, the normal direction of a polygon having vertices that are three-dimensional points constituting the ground, a road, etc., tends to be the same direction as the up-down direction. In other words, if the angle between the normal direction of a polygon having vertices that are at least three three-dimensional points constituting the ground, a road, etc. and the up-down direction is not parallel but too large, there is a high possibility that at least one of the at least three three-dimensional points is noise rather than a three-dimensional point constituting the ground, a road, etc. Therefore, with this configuration, for example, the densification unit can prevent new three-dimensional points from being generated within a polygon having vertices that are three-dimensional points that are unlikely to be three-dimensional points constituting the ground, a road, etc.

[0030] Also, for example, the auxiliary information is information indicating a group to which each of the multiple three-dimensional points belongs, and the densification unit identifies the polygon having at least three vertices of the three-dimensional points belonging to the same group based on the auxiliary information.

[0031] According to this, the densification unit can identify polygons whose vertices are three-dimensional points that are likely to be the same or that constitute the same objects, such as three-dimensional points of the same object type, three-dimensional points that can be connected by a line without crossing the edges of objects contained in the image data, or three-dimensional points that are located below a predetermined position in a virtual three-dimensional space (virtual space) that represents the three-dimensional points, by identifying polygons based on auxiliary information that indicates three-dimensional points that are likely to be the same or that constitute the same objects as three-dimensional points belonging to the same group.

[0032] Furthermore, a three-dimensional point cloud densification method according to one embodiment of the present disclosure includes an acquisition step of acquiring a plurality of three-dimensional points, each of which indicates the three-dimensional position of at least one object, and auxiliary information for creating a polygon; an identification step of identifying a polygon having at least three of the plurality of three-dimensional points as vertices based on the auxiliary information; and a densification step of generating new three-dimensional points within the polygon.

[0033] This provides the same effect as the three-dimensional point cloud densification device.

[0034] The present disclosure may be realized as a program that causes a computer to execute the steps included in the above-described three-dimensional point cloud densification method.

[0035] Hereinafter, embodiments of a three-dimensional point cloud densification device and the like according to the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, arrangement and connection of components, steps, order of steps, and the like shown in each of the following embodiments are merely examples and are not intended to limit the present disclosure.

[0036] Furthermore, among the components in the following embodiments, components that are not recited in the independent claims showing implementation forms according to one aspect of the present disclosure are described as optional components. Implementation forms of the present disclosure are not limited to the current independent claims, and may also be expressed by other independent claims.

[0037] Furthermore, each drawing is a schematic diagram and is not necessarily an exact illustration. In addition, in each drawing, substantially the same components are denoted by the same reference numerals, and duplicated explanations may be omitted or simplified.

[0038] (Embodiment) [3D point cloud generation system] First, the configuration of a three-dimensional point cloud generation system 400 including a three-dimensional point cloud densification device 100 according to an embodiment will be described.

[0039] Fig. 1 is a diagram for explaining a three-dimensional point cloud generation system 400 according to an embodiment. Fig. 2 is a block diagram showing the configuration of the three-dimensional point cloud generation system 400 according to an embodiment.

[0040] The three-dimensional point cloud generation system 400 is a system that generates a three-dimensional point cloud (more specifically, point cloud data indicating a three-dimensional point cloud) consisting of a plurality of three-dimensional points that constitute a three-dimensional model of an object included in a plurality of images based on a photographing unit 300, which is a photographing device (camera) that photographs a specified area, and a plurality of images (more specifically, image data) generated by the photographing unit 300, and generates a dense three-dimensional point cloud (more specifically, data indicating a dense three-dimensional point cloud) by generating and adding new three-dimensional points to the generated three-dimensional point cloud.

[0041] Note that the term "object" refers to, for example, at least one object in a space (a predetermined area). Alternatively, for example, the term "object" refers to, for example, at least one subject when generated by photography. Alternatively, for example, the term "object" refers to, for example, at least one measurement target when generated by distance measurement.

[0042] Furthermore, a three-dimensional model is an object (e.g., objects 710, 711, and 712 shown in FIG. 1) included in a plurality of images in which the actual object (e.g., objects 700, 701, and 702 shown in FIG. 1) is photographed, and the object is reconstructed in a virtual three-dimensional space, that is, the object is expressed (generated). A three-dimensional model is a collection of a plurality of three-dimensional points (a three-dimensional point cloud).

[0043] Furthermore, the predetermined area is an area that includes, as objects, stationary objects, moving objects such as people, or both. In other words, the predetermined area is, for example, an area that includes, as subjects, at least one of stationary objects and moving objects. Examples of a predetermined area that includes both stationary objects and moving objects include a sports game such as basketball, or a city street with people or cars. Furthermore, the predetermined area may be not only a specific object that serves as a subject, but also a scene that includes scenery, etc. Hereinafter, a predetermined area that includes not only a specific object but also a scenery, etc. will also be simply referred to as an object.

[0044] Point cloud data is information including data on each of a plurality of three-dimensional points. A three-dimensional point is, for example, information indicating the three-dimensional position (three-dimensional coordinates) of at least one object. A three-dimensional point includes, for example, ternary information consisting of an X component, a Y component, and a Z component of a three-dimensional space consisting of mutually orthogonal X, Y, and Z axes (i.e., position information indicating coordinates). In addition to position information, a three-dimensional point may also include so-called attribute information such as color information indicating the color of the three-dimensional point and texture information representing the surface shape of the three-dimensional point and its surroundings.

[0045] The three-dimensional point cloud generation system 400 includes an imaging unit 300, a three-dimensional point cloud generation device 200, and a three-dimensional point cloud densification device 100.

[0046] The photographing unit 300 is a photographing device for photographing an object. The photographing unit 300 is composed of, for example, one or more cameras 301, and outputs a plurality of images generated by photographing the object to the 3D point cloud generating device 200.

[0047] The photographing unit 300 may include a plurality of cameras 301 or a single camera 301. For example, a user may move a single camera 301 and cause the camera 301 to photograph images so as to generate a plurality of images with different viewpoints (in other words, multi-viewpoint video data). In other words, the plurality of images output by the photographing unit 300 to the 3D point cloud generating device 200 may be images photographed (generated) by cameras 301 of the photographing unit 300, each of which has a different position or orientation from the others.

[0048] Furthermore, the camera 301 may be a camera that generates a two-dimensional image, or may be a camera equipped with a three-dimensional measurement sensor that generates a three-dimensional model. In this embodiment, the photographing unit 300 has the camera 301 that generates a two-dimensional image.

[0049] The photographing unit 300 is configured to be connectable, via wire or wireless, to the three-dimensional point cloud generating device 200 itself or to a hub (not shown) such as a communication device or server so that the generated image can be output to the three-dimensional point cloud generating device 200.

[0050] The images generated by the imaging unit 300 may be output in real time, or may be recorded once in an external storage device such as a memory or a cloud server, and then output from the external storage device.

[0051] Furthermore, when the imaging unit 300 has multiple cameras 301, the multiple cameras 301 may be fixed cameras 301 such that at least one of the position and the attitude is different from each other. The attitude of the camera 301 indicates at least one of the imaging direction of the camera 301 and the tilt of the camera 301.

[0052] There are no particular limitations on the type of camera 301. Camera 301 may be, for example, a mobile camera such as a video camera, a smartphone, or a wearable camera, a mobile camera such as a drone with a shooting function, or a fixed camera such as a surveillance camera.

[0053] The three-dimensional point cloud generating device 200 is a device that generates a three-dimensional point cloud based on a plurality of images output from the imaging unit 300.

[0054] The three-dimensional point cloud generating device 200 estimates (calculates) camera information (also called camera parameters) indicating the position, orientation, etc. when the photographing unit 300 generated the image, based on a plurality of images, for example.

[0055] Here, the camera information refers to parameters that indicate the characteristics of the camera 301, such as internal parameters including the focal length and image center of the camera 301, external parameters that indicate the position (more specifically, the three-dimensional position) and attitude of the camera 301 at the time of shooting, and parameters such as the distance between the camera 301 and the object. Furthermore, if the camera 301 takes multiple images while moving, the camera information may include information that indicates the trajectory of the camera 301 that generated the multiple images at the time of shooting.

[0056] Note that there is no particular limitation on the method by which the 3D point cloud generation device 200 estimates the position and orientation of the camera 301. The 3D point cloud generation device 200 estimates the position and orientation when the camera 301 generated an image using, for example, Visual-SLAM (Simultaneous Localization and Mapping) technology, Structure-from-Motion (SfM) technology, or the like. The 3D point cloud generation device 200 may include a LiDAR in the imaging unit 300, and estimate camera information based on information obtained from the LiDAR.

[0057] Furthermore, the camera information may be output from the imaging unit 300 to the 3D point cloud generation device 200, or may be stored in advance in a memory (not shown) included in the 3D point cloud generation device 200. In this case, the 3D point cloud generation device 200 does not need to estimate the camera information.

[0058] The three-dimensional point cloud generating device 200 generates a three-dimensional model (three-dimensional point cloud) in a three-dimensional space (virtual space) based on camera information and a plurality of images.

[0059] The three-dimensional point cloud generation device 200 is realized by, for example, a computer system including a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. In this embodiment, the three-dimensional point cloud generation device 200 is realized as a computer 500 together with the three-dimensional point cloud densification device 100.

[0060] The computer 500 is a device (system) that includes a processor, a memory in which a program executed by the processor is stored, a communication interface for communicating with the imaging unit 300, etc., an operation device 510 that accepts input from the user, and a display device 520 that displays images, three-dimensional point clouds, etc.

[0061] The communication interface may be a wireless communication interface having an antenna and a wireless communication circuit, or may be a wired communication interface having a connector to be connected to a communication line.

[0062] The operation device 510 is a user interface operated by a user, and is realized by a keyboard, a mouse, a touch panel, and the like.

[0063] The display device 520 is a display for displaying images, three-dimensional point clouds, and the like.

[0064] The three-dimensional point cloud generating device 200 outputs the generated three-dimensional point cloud and camera information to the three-dimensional point cloud densification device 100. The three-dimensional point cloud generating device 200 may also output a plurality of images used in generating the three-dimensional point cloud to the three-dimensional point cloud densification device 100.

[0065] The imaging unit 300 and the 3D point cloud generating device 200 may be integrated into one computer having an imaging function.

[0066] Furthermore, when the three-dimensional point cloud generating device 200 and the three-dimensional point cloud densification device 100 are realized by different computers, etc., the three-dimensional point cloud generating device 200 and the three-dimensional point cloud densification device 100 may be connected to each other so that they can communicate with each other via a network, or may be configured to allow data exchange offline using an HDD (Hard Disk Drive) or the like.

[0067] Furthermore, the 3D point cloud densification device 100 may calculate the camera information or may store the camera information in advance. In this case, the 3D point cloud generation device 200 does not need to output the camera information to the 3D point cloud densification device 100.

[0068] The 3D point cloud densification device 100 is a device that generates a dense 3D point cloud by generating new 3D points in the 3D point cloud generated by the 3D point cloud generation device 200.

[0069] The three-dimensional point cloud densification device 100 is realized, for example, by a computer system including a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. In this embodiment, the three-dimensional point cloud densification device 100 is realized as a computer 500 together with the three-dimensional point cloud generation device 200.

[0070] The three-dimensional point cloud densification device 100 and the three-dimensional point cloud generation device 200 may be realized by a single processor or by different processors. Furthermore, the programs executed by the three-dimensional point cloud densification device 100 and the three-dimensional point cloud generation device 200 may be stored in a single memory or may be stored in different memories.

[0071] FIG. 3 is a diagram for explaining a dense 3D point cloud generated by a 3D point cloud generation system 400 according to an embodiment.

[0072] First, as shown in FIG. 3( a ), the 3D point cloud generating device 200 generates a 3D point cloud based on a plurality of images generated by the imaging unit 300 .

[0073] Next, as shown in FIG. 3(b), the 3D point cloud densification device 100 meshes the 3D point cloud by connecting the 3D points included in the 3D point cloud with straight lines (also called mesh lines).

[0074] Note that meshing refers to creating polygons with multiple 3D points as vertices by connecting the 3D points with mesh lines. A mesh is formed by multiple polygons.

[0075] In the example shown in Figure 3(b), five polygons, 720, 721, 722, 723, and 724, are created, each with three three-dimensional points as vertices. Here, the three-dimensional point cloud densification device 100 connects adjacent three-dimensional points with mesh lines so that the mesh lines do not come into contact (intersect) with each other. Of course, the three-dimensional point cloud densification device 100 does not need to generate mesh lines to connect three-dimensional points, as long as it can identify areas (polygons) with each three-dimensional point as a vertex.

[0076] Next, as shown in (c) of Figure 3, the 3D point cloud densification device 100 generates a dense 3D point cloud by generating new 3D points (additional 3D points) in the area surrounded by the mesh lines, i.e., within the polygon.

[0077] Note that "inside the polygon" means not only the area inside the area surrounded by the mesh lines, but also the area on the mesh lines.

[0078] Here, if the positions connecting the three-dimensional points are not appropriate, that is, if a polygon with at least three three-dimensional points as vertices is not created properly, there is a risk that a three-dimensional point cloud will be added to a position where no object exists, resulting in a three-dimensional model that differs from the shape of the actual object.

[0079] Therefore, the 3D point cloud densification device 100 generates a new 3D point within a polygon having at least three vertices from among the plurality of 3D points, based on auxiliary information for creating the polygon.

[0080] [Configuration of 3D point cloud densification device] Next, the configuration of the three-dimensional point cloud densification device 100 will be described in detail.

[0081] FIG. 4 is a block diagram showing the configuration of a three-dimensional point cloud densification device 100 according to an embodiment.

[0082] The three-dimensional point cloud densification device 100 includes an acquisition unit 110, an assignment unit (generation unit) 120, a densification unit 130, a UI (User Interface) unit 140, a control unit 150, and a storage unit 160.

[0083] The acquisition unit 110 is a processing unit that acquires data output from the three-dimensional point cloud generation device 200, data stored in the storage unit 160, etc. The acquisition unit 110 acquires data from the three-dimensional point cloud generation device 200, the storage unit 160, and an external device such as a server (not shown), for example, via a communication interface or communication line (not shown) provided in the three-dimensional point cloud densification device 100 for performing wired or wireless communication.

[0084] The acquisition unit 110 acquires a plurality of three-dimensional points each indicating the three-dimensional position of at least one object, and auxiliary information for creating (identifying) a polygon.

[0085] Information for creating a polygon having at least three 3D points as vertices includes primary information and auxiliary information other than the primary information. The primary information is well-known information used to create a polygon. The primary information relates to, for example, the distance between two 3D points. If the distance between two 3D points is less than a predetermined threshold, the two 3D points are used as vertices, and mesh lines are defined by connecting the two 3D points. The auxiliary information is used to select at least one of the 3D points and the mesh lines connecting the two 3D points in order to create an appropriate polygon.

[0086] For example, the mesh lines connect 3D points that belong to the same group, but do not connect 3D points that belong to different groups. In this way, for example, the auxiliary information is information that indicates the group to which each of the multiple 3D points belongs.

[0087] In addition, the auxiliary information may be, for example, information indicating the type of object constituted by the three-dimensional points, information indicating a group arbitrarily set by the user, and information indicating the objects contained in the image used to generate the three-dimensional point cloud.

[0088] The edge here may not only refer to a line that defines the perimeter of an object included in an image, but also to a ridge line (a boundary between two different surfaces) within an object in an image. An edge may also be a line (a connection between three-dimensional points) that defines a three-dimensional model in a three-dimensional space on a computer (i.e., a virtual three-dimensional space that represents multiple three-dimensional points). Furthermore, an edge may be a projection of a line that defines the three-dimensional model onto a two-dimensional plane (e.g., an image).

[0089] The above-mentioned camera information may be used as auxiliary information.

[0090] Alternatively, the auxiliary information may be a list of three-dimensional points that may be connected by mesh lines.

[0091] For example, the acquisition unit 110 acquires auxiliary information via an operation device 510 for receiving instructions from a user. The instruction is, for example, an instruction to specify the shape of a polygon. Specifically, the instruction to specify three-dimensional points to be connected by mesh lines and / or an instruction to specify the positions of the mesh lines to be connected.

[0092] The acquisition unit 110 is communicably connected to, for example, an operation device 510, and acquires auxiliary information from a user via the operation device 510.

[0093] Furthermore, for example, the acquisition unit 110 acquires, as auxiliary information, camera information indicating the position (i.e., trajectory) and attitude of the camera 301 that generated the plurality of image data obtained by photographing at least one object. The acquisition unit 110 may acquire the camera information as auxiliary information.

[0094] Furthermore, for example, the acquisition unit 110 acquires a plurality of image data used when the three-dimensional point cloud is generated.

[0095] The assigning unit 120 is a processing unit that generates auxiliary information. For example, the assigning unit 120 is a processing unit that assigns auxiliary information corresponding to each of the three-dimensional points included in the three-dimensional point cloud. "Assigning" means, for example, adding (linking) auxiliary information to data of the three-dimensional points.

[0096] For example, the assigning unit 120 generates auxiliary information based on image data obtained by photographing at least one object. The image data is, for example, an image used when the 3D point cloud generation device 200 generates a 3D point cloud. Note that the image data may be an image not used when the 3D point cloud generation device 200 generates a 3D point cloud.

[0097] Specifically, the assignment unit 120 generates type information indicating the types of multiple objects contained in the image data acquired by the acquisition unit 110 as auxiliary information, and assigns type information corresponding to each of the multiple three-dimensional points to each of the multiple three-dimensional points.

[0098] Alternatively, for example, the adding unit 120 generates the edge of an object included in the image data acquired by the acquiring unit 110 as auxiliary information.

[0099] The densification unit 130 is a processing unit that generates a dense 3D point cloud (more specifically, data indicating a dense 3D point cloud) by generating and adding new 3D points (more specifically, data indicating 3D points) to the 3D point cloud. The densification unit 130 identifies (creates) a polygon having at least three of the multiple 3D points as vertices based on the auxiliary information, and generates new 3D points within the identified polygon.

[0100] Note that the new 3D point may be generated, for example, by interpolation between at least two 3D points, or by feature point matching between at least two images (for example, images generated by capturing images by the camera 301). However, the method for generating the new 3D point is not limited to these methods, and any well-known method may be used.

[0101] For example, the densification unit 130 identifies a polygon having at least three vertices among the multiple three-dimensional points based on auxiliary information specifying the shape of the polygon obtained by the obtaining unit 110 from the user via the operation device 510.

[0102] Note that specifying a polygon means specifying the position of a polygonal area having at least three 3D points as vertices to determine the position where a new 3D point is to be generated. As with mesh lines, it is not necessary for a polygon to be created (i.e., a mesh line to be generated), as long as the polygon is specified.

[0103] For example, the densification unit 130 identifies polygons having at least three vertices among the 3D points belonging to the same group based on the auxiliary information, and generates new 3D points within the identified polygons.

[0104] The densification unit 130 may identify a polygon by connecting three-dimensional points with mesh lines based on auxiliary information, and generate new three-dimensional points within the identified polygon. Alternatively, the densification unit 130 may identify a polygon by first connecting three-dimensional points with mesh lines, and then, based on the auxiliary information, delete some of the mesh lines connecting the three-dimensional points (i.e., treat them as not being connected by mesh lines), and generate new three-dimensional points within the polygon formed by the remaining mesh lines. In other words, the densification unit 130 may identify a polygon by determining three-dimensional points that will become the vertices of the polygon based on auxiliary information, or may first identify a polygon and then re-identify the polygon by reducing the three-dimensional points that will become the vertices based on auxiliary information to generate additional three-dimensional points.

[0105] For example, the densification unit 130 identifies a polygon whose vertices are at least three 3D points of the same object type, and generates a new 3D point within the identified polygon. Note that the densification unit 130 (or the assigning unit 120) may assign type information indicating the same type as the three 3D points to the newly generated 3D point.

[0106] Alternatively, for example, the densification unit 130 identifies polygons that do not intersect with edges and generates new 3D points within the identified polygons. For example, the densification unit 130 identifies polygons that can be formed so as not to intersect with edges and generates new 3D points within the identified polygons.

[0107] Alternatively, for example, the densification unit 130 identifies a polygon having at least three vertices among the plurality of three-dimensional points based on the camera information, and generates a new three-dimensional point within the identified polygon. Specifically, the densification unit 130 identifies a polygon having at least three vertices among the plurality of three-dimensional points located below a predetermined position based on the up-down direction in the virtual three-dimensional space representing the plurality of three-dimensional points (i.e., in the virtual space on the computer), which is determined based on the camera information, and generates a new three-dimensional point within the identified polygon.

[0108] The predetermined position may be arbitrarily determined in advance. As described above, the camera information includes, for example, information indicating the position of the camera 301 at the time of shooting. For example, the densification unit 130 determines the height of the camera 301 in the up-down direction in the virtual three-dimensional space based on the camera information, and then determines the predetermined position based on the determined height, more specifically, below the determined height. Alternatively, the densification unit 130 may set the position (height) of the camera 301 indicated by the camera information as the predetermined height.

[0109] The densification unit 130 may determine three-dimensional points to be the vertices of the polygon based on auxiliary information and camera information indicating at least one of the trajectory of the camera 301 during shooting and the attitude of the camera 301.

[0110] The camera information may include information indicating the trajectory (movement trajectory) of the camera 301 during shooting, and the densification unit 130 may identify a polygon having at least three of the multiple three-dimensional points as vertices based on the information indicating the trajectory of the camera 301 included in the camera information. For example, since the trajectory of the camera 301 is a space that the camera 301 can pass through, it is highly likely that no object exists on the trajectory of the camera 301. Therefore, the densification unit 130 may not generate or may exclude mesh lines that cross the trajectory of the camera.

[0111] The UI unit 140 is a user interface for receiving input from a user and displaying images to the user, and is realized by, for example, an operation device 510, a display device 520, and the like.

[0112] The control unit 150 is a processing unit for controlling the operation timing of each processing unit included in the three-dimensional point cloud densification device 100, the UI unit 140, and the like.

[0113] The storage unit 160 is a storage device that stores data such as a 3D point cloud, a dense 3D point cloud, auxiliary information, a plurality of images, camera information, and the like, as well as control programs executed by each processing unit included in the 3D point cloud densification device 100. The storage unit 160 is realized by, for example, an HDD, a flash memory, or the like.

[0114] The storage unit 160 includes, for example, a video storage unit 161, a camera information storage unit 162, a three-dimensional point cloud storage unit 163, and a dense three-dimensional point cloud storage unit 164.

[0115] The image storage unit 161 stores a plurality of images (a plurality of image data) used to generate the three-dimensional point cloud.

[0116] The camera information storage unit 162 stores camera information.

[0117] The three-dimensional point cloud storage unit 163 stores a three-dimensional point cloud (three-dimensional point cloud data).

[0118] The dense 3D point cloud storage unit 164 stores a dense 3D point cloud (dense 3D point cloud data). Note that the dense 3D point cloud storage unit 164 may store only information on 3D points added to the 3D point cloud stored in the 3D point cloud storage unit 163.

[0119] The video memory unit 161, the camera information memory unit 162, the three-dimensional point cloud memory unit 163, and the dense three-dimensional point cloud memory unit 164 may be realized by a single storage device, or may be realized by different storage devices.

[0120] [Processing procedure of the three-dimensional densification device] Next, the processing procedure for generating a dense 3D point cloud executed by the 3D point cloud densification device 100 will be described in detail.

[0121] <Summary> FIG. 5 is a sequence diagram showing the processing procedures of each unit (more specifically, the UI unit 140, the acquisition unit 110, the assignment unit 120, and the densification unit 130) included in the three-dimensional point cloud densification device 100 according to the embodiment.

[0122] First, the acquiring unit 110 acquires information for the densification unit 130 to generate a dense 3D point cloud (S101). For example, the acquiring unit 110 acquires the 3D point cloud, auxiliary information, and the like from the storage unit 160.

[0123] The acquisition unit 110 may start step S101 when, for example, an instruction to start execution of processing for generating a dense 3D point cloud (densification processing) is acquired from the user via the UI unit 140.

[0124] Next, the UI unit 140 displays an image for receiving instructions from the user (S102). Here, for example, the UI unit 140 prompts the user to select whether or not to manually add auxiliary information to the three-dimensional point cloud.

[0125] Next, for example, the UI unit 140 determines whether or not input of auxiliary information has been received from the user (S103). This determination may be performed by any processing unit included in the 3D point cloud densification device 100, and may be performed by, for example, the acquisition unit 110 or the control unit 150.

[0126] When it is determined that the UI unit 140 has not received input of auxiliary information from the user (No in S103), the assigning unit 120 generates auxiliary information corresponding to each of the three-dimensional points (S104).

[0127] If it is determined that the UI unit 140 has accepted input of auxiliary information from the user (Yes in S103), the acquisition unit 110 acquires the auxiliary information from the UI unit 140, and if the assignment unit 120 has generated the auxiliary information (S104), the acquisition unit 110 acquires the auxiliary information from the assignment unit 120 (S105).

[0128] Next, the densification unit 130 generates new three-dimensional points based on the three-dimensional point cloud and the auxiliary information acquired by the acquisition unit 110 (S106).

[0129] <Acquisition process> FIG. 6 is a flowchart showing the acquisition process of the 3D point cloud densification device 100 according to the embodiment.

[0130] First, the acquisition unit 110 acquires a three-dimensional point cloud from the three-dimensional point cloud generating device 200, and stores the acquired three-dimensional point cloud in the three-dimensional point cloud storage unit 163 (S201).

[0131] Next, the acquisition unit 110 acquires, from the 3D point cloud generation device 200, images captured in the same space as the 3D point cloud acquired in step S201, and stores the acquired images in the video storage unit 161 (S202). The images captured in the same space are, for example, multiple images used to generate the 3D point cloud. Alternatively, examples of the images captured in the same space include, when a 3D point cloud is generated based on an image, such as in SfM, the images, and, when a 3D point cloud is generated using laser scanning, images captured by a camera 301 whose positional relationship with a laser device that performed the laser scanning to assign attribute information indicating color to each 3D point of the 3D point cloud are known.

[0132] Next, the acquisition unit 110 acquires, from the 3D point cloud generation device 200, camera information indicating the three-dimensional position and orientation of the camera 301 when the image acquired in step S202 was captured, and stores the acquired camera information in the camera information storage unit 162 (S203). For example, when a 3D point cloud is generated based on an image, such as in SfM, the camera information includes the three-dimensional position and orientation of the camera 301 when the image was captured. Alternatively, when a 3D point cloud is generated using laser scanning, the camera information includes information indicating the three-dimensional position and orientation of the camera 301, whose positional relationship with the laser device that performed the laser scan is known. The camera information may also include information indicating the three-dimensional position of the laser device.

[0133] The acquiring unit 110 may acquire any image as long as the image is an image of the same space as the three-dimensional point cloud. For example, the acquiring unit 110 may acquire an image that is not used by the three-dimensional point cloud generating device 200 to generate the three-dimensional point cloud, and camera information indicating the position and orientation of the camera that generated the image using a global positioning system (GPS) or a gyro sensor, etc.

[0134] Furthermore, the acquisition unit 110 does not need to acquire the image and camera information if they are not used as auxiliary information for the three-dimensional point cloud.

[0135] This allows the processing performed by the three-dimensional point cloud densification device 100 to be reduced, and also allows the amount of information stored in the storage unit 160 to be reduced.

[0136] <Display example> Fig. 7 is a diagram showing an example of an image displayed by the UI unit 140 included in the three-dimensional point cloud densification device 100 according to the embodiment. Fig. 7 also shows a display device included in the UI unit 140 that displays an image.

[0137] In step S102 shown in FIG. 5, the UI unit 140 displays, for example, a point cloud image 600, a superimposed image 610, an image including an adjustment unit 620, and an image including a selection unit 630, as shown in FIG.

[0138] The point cloud image 600 is an image that includes a three-dimensional point cloud.

[0139] The superimposed image 610 is an image in which a three-dimensional point cloud is superimposed on the image (two-dimensional image) used to generate the three-dimensional point cloud.

[0140] The image including the adjustment unit 620 is an image for receiving input from a user and changing the orientation of the display of the three-dimensional point cloud. For example, the UI unit 140 changes the orientation of the three-dimensional point cloud of the image displayed as the point cloud image 600 in accordance with the input received by the adjustment unit 620. This allows the user to check the three-dimensional point cloud from various angles by operating the adjustment unit 620.

[0141] The image including the selection unit 630 is an image for accepting input from the user and determining whether to generate auxiliary information automatically (for example, by the attachment unit 120) or to determine auxiliary information by the user. For example, if the user selects "automatic" (No in step S103 shown in FIG. 5), the attachment unit 120 performs processing for generating auxiliary information (step S104 shown in FIG. 5). On the other hand, for example, if the user selects "manual" (No in step S103 shown in FIG. 5), the UI unit 140 switches the image displayed to an image for accepting input of auxiliary information.

[0142] Fig. 8 is a diagram showing an example of an image when the UI unit 140 included in the 3D point cloud densification device 100 according to the embodiment is operated by a user. Specifically, (a) of Fig. 8 is a diagram showing an example of an image displayed by the UI unit 140 when the user selects "manual" using the selection unit 630. (b) of Fig. 8 is a diagram showing an example of an image showing a state in which the user is inputting auxiliary information by operating the UI unit 140. (c) of Fig. 8 is a diagram showing another example of an image showing a state in which the user is inputting auxiliary information by operating the UI unit 140.

[0143] When the UI unit 140 receives an input of "manual" from the user via the selection unit 630, it displays an image showing a three-dimensional point cloud, as shown in FIG. 8(a).

[0144] In this case, as shown in (b) of Figure 8, the user operates the operation device provided in the UI unit 140 to move the cursor 640 to select a three-dimensional point, and inputs type information indicating the type of object, etc., as auxiliary information at the selected three-dimensional point.

[0145] 8, three-dimensional point clouds to which auxiliary information has not been input (assigned) are indicated by white circles, and three-dimensional points to which auxiliary information has been input (assigned) are indicated by hatching. In this way, while viewing the three-dimensional point cloud displayed on the UI unit 140, the user assigns auxiliary information to each three-dimensional point, such as a "first type" where the object type is a "tree," a "second type" where the object type is a "house," etc.

[0146] As shown in FIG. 8(c), the UI unit 140 may perform processing so that the user can select a group of three-dimensional points at once by moving a cursor 640 along an operation trajectory 650 by dragging the cursor 640.

[0147] Fig. 9 is a diagram showing another example of an image when the UI unit 140 included in the 3D point cloud densification device 100 according to the embodiment is operated by a user. (a) of Fig. 9 is a diagram corresponding to (a) of Fig. 8, (b) of Fig. 9 is a diagram corresponding to (b) of Fig. 8, and (c) of Fig. 9 is a diagram corresponding to (c) of Fig. 8.

[0148] In the example shown in FIG. 9, unlike FIG. 8, the UI unit 140 displays a three-dimensional point group superimposed on an image (two-dimensional image).

[0149] This makes it easier for the user to appropriately select the type of object made up of three-dimensional points.

[0150] In addition, in Figure 7, the selection unit 630 shows buttons for "automatic" and "manual" (the part selected by the user), but the selection unit 630 only needs to be able to select whether the user inputs the auxiliary information or has the addition unit 120 generate it.

[0151] Furthermore, the selection unit 630 may have an option such as "semi-automatic" in addition to "automatic" and "manual." For example, if the user selects "semi-automatic," the attachment unit 120 generates auxiliary information. Next, the UI unit 140 displays an image including the auxiliary information generated by the attachment unit 120. Furthermore, if the user wants to change the auxiliary information generated by the attachment unit 120, the user operates the UI unit 140 to input the changed auxiliary information.

[0152] This allows the user to start inputting auxiliary information from a state where the auxiliary information has been roughly correctly added by the adding unit 120. This reduces the amount of work required for the user to input auxiliary information to each 3D point as intended.

[0153] <Auxiliary information generation> Fig. 10 is a diagram for explaining an example of a method for generating auxiliary information executed by the three-dimensional point cloud densification device 100 according to the embodiment. Specifically, (a) of Fig. 10 is a schematic diagram showing a state in which the camera 301 is capturing an image as viewed from above, (b) of Fig. 10 is a diagram showing an example of an image generated by the camera 301 in (a) of Fig. 10, and (c) of Fig. 10 is a diagram showing an example in which the attachment unit 120 performs image recognition (object recognition) on the image shown in (b) of Fig. 10.

[0154] In the example shown in FIG. 10, the attachment unit 120 recognizes (identifies) the type of object included in the image by performing image recognition.

[0155] If the position of the camera 301 that generated the image used to generate the densified 3D point cloud and the position of the actual object corresponding to the 3D point cloud are known, the relationship between the 3D points and the pixels contained in the image is uniquely determined.

[0156] Therefore, by grouping three-dimensional points according to which three-dimensional points constitute which object contained in the image, it is possible to prevent mesh lines from being generated that straddle objects.

[0157] This allows the densification unit 130 to perform densification processing on the 3D point cloud with high precision, that is, to add 3D points at appropriate positions.

[0158] 10, the assigning unit 120 identifies the type of object constituted by each pixel included in the image by performing image recognition. Furthermore, the assigning unit 120 generates and assigns type information indicating the type of object constituted by each 3D point as auxiliary information for each 3D point based on the relationship between the 3D points and the pixels included in the image. The densification unit 130 identifies polygons whose vertices are 3D points to which the same type information is assigned.

[0159] The method for recognizing the type of object is not particularly limited. For example, the assignment unit 120 may recognize the type of object configured by each pixel through image recognition, such as semantic segmentation. Alternatively, the assignment unit 120 may treat different individuals of the same type of object as different types, such as through instance segmentation. For example, when there are two objects identified as "trees" in an image, the assignment unit 120 may determine that one "tree" and the other "tree" are different objects.

[0160] Fig. 11 is a diagram for explaining another example of the method for generating auxiliary information executed by the three-dimensional point cloud densification device 100 according to the embodiment. Specifically, (a) of Fig. 11 is a schematic diagram showing the state of the camera 301 capturing an image as viewed from above, (b) of Fig. 11 is a diagram showing an example of the image generated by the camera 301 in (a) of Fig. 11, and (c) of Fig. 11 is a diagram showing an example of image recognition performed by the attachment unit 120 on the image shown in (b) of Fig. 11.

[0161] In the example shown in FIG. 11 , the assigning unit 120 extracts (generates) auxiliary information indicating the edges of objects formed by each pixel included in the image by performing the image recognition described above. For example, the assigning unit 120 extracts the edges of objects included in the image using the Canny algorithm or the like. In this case, the assigning unit 120 may assign the same information (connection information) to three-dimensional points that can be connected by mesh lines so as not to cross edges (i.e., not to intersect at a certain point). For example, the assigning unit 120 associates unique ID (identifier) ​​information and connection information for each three-dimensional point, and stores the information in the storage unit 160.

[0162] For example, the densification unit 130 identifies polygons whose vertices are three-dimensional points having the same connectivity information. This allows the densification unit 130 to connect three-dimensional points with mesh lines that do not cross the edges extracted by the assignment unit 120 (i.e., do not intersect at a certain point). In other words, the densification unit 130 can mesh the three-dimensional point cloud by limiting the locations where mesh lines are connected based on the edges.

[0163] This makes it possible to prevent mesh lines from crossing at positions that are likely to form the outline of an object made up of a 3D point cloud, and also to prevent mesh lines from connecting different objects.

[0164] In addition, when a stereo camera or an RGB-D camera is used as the camera 301, the densification unit 130 may mesh the three-dimensional point cloud based on the depth information obtained from the image and the edge information generated by the assignment unit 120.

[0165] This allows three-dimensional points to be added at appropriate positions based on depth information even when the three-dimensional point cloud obtained from the three-dimensional point cloud generating device 200 is sparse (in other words, the number of three-dimensional points contained in the three-dimensional point cloud is small) and there are not enough three-dimensional points around the contour of the object to properly mesh it.

[0166] In addition, the densification unit 130 may determine the number of three-dimensional points included in the three-dimensional point cloud shown in the point cloud data acquired by the acquisition unit 110, and perform densification processing if the number is less than a predetermined number, or not perform densification processing if the number is greater than or equal to the predetermined number.

[0167] FIG. 12 is a diagram for explaining a method of capturing images using the camera 301 provided in the 3D point cloud generation system 400 according to the embodiment.

[0168] The densification unit 130 may determine the group to which each 3D point belongs based on camera information indicating at least one of the position and orientation of the camera 301.

[0169] For example, camera 301 often takes pictures while maintaining the same height from the ground. Also, when taking pictures with camera 301, the ground is generally located below (below) the image.

[0170] Therefore, for example, it is assumed that the user takes an image while moving the camera 301 horizontally or parallel to the ground. Also, for example, it is assumed that the user takes an image with the camera 301 so that the ground is positioned below.

[0171] Under these conditions, when the position of the camera 301 that generated the image used to generate the densified three-dimensional point cloud and the position of the actual object corresponding to the three-dimensional point cloud are known, the assignment unit 120 can identify the position of the ground located below the camera 301 (for example, in the direction of the ground shown in Figure 12) based on trajectory information indicating the trajectory of the camera 301, and set the object type of the three-dimensional point located at the identified position to "ground".

[0172] For example, the assigning unit 120 estimates the position of a plane through which the camera 301 passes, based on the camera trajectory indicated in the camera information. The assigning unit 120 also estimates that the estimated plane is a plane parallel to the ground. This makes it possible to identify a position in the 3D point cloud corresponding to the ground based on information indicating the orientation of the ground from the camera 301, that is, information indicating the up-down direction, and information indicating the height from the camera 301 to the ground, and therefore to assign auxiliary information of "ground" to the 3D point located at that position.

[0173] In this way, for example, the assigning unit 120 may estimate the up-down direction (direction of gravity) in the virtual three-dimensional space in which the three-dimensional point cloud is placed based on an estimated plane that is parallel to the ground, and generate information indicating the estimated up-down direction as auxiliary information.

[0174] This allows the assignment unit 120 to estimate or remove the type of object of a plurality of three-dimensional points that form a plane based on the up-down direction.

[0175] For example, the densification unit 130 identifies a polygon based on the angle between the normal direction of the polygon, which has at least three vertices among the multiple three-dimensional points located below a predetermined position, and the vertical direction. Specifically, for example, the densification unit 130 determines whether the angle between the normal direction of the polygon, which has at least three vertices among the multiple three-dimensional points located below the predetermined position, and the vertical direction is equal to or greater than a predetermined angle, and if it determines that the angle is equal to or greater than the predetermined angle, does not generate a new three-dimensional point in the polygon, which has at least three vertices among the multiple three-dimensional points located below the predetermined position.

[0176] Alternatively, for example, the densification unit 130 removes 3D points that form a plane whose normal direction is at a predetermined angle or more with respect to the vertical direction estimated by the assignment unit 120. Specifically, the densification unit 130 determines whether the angle between the normal direction of a polygon having at least three of the multiple 3D points located below a predetermined position and the vertical direction is at least a predetermined angle. If it determines that the angle is at least a predetermined angle, the densification unit 130 does not generate a new 3D point in the polygon having at least three of the multiple 3D points located below the predetermined position as vertices, but removes at least one of the at least three of the multiple 3D points located below the predetermined position. All of the at least three 3D points may be removed, or 3D points to be removed may be determined based on other conditions, such as whether the 3D points are not vertices of other polygons among the multiple polygons.

[0177] This is because 3D points that form a plane whose normal direction is at a predetermined angle or more with respect to the up-down direction estimated by the assignment unit 120 are likely to be noise that spans different objects. As a result, even if the densification is performed with low accuracy, that is, even if a 3D point is added at an inappropriate position, the densification unit 130 can appropriately remove the 3D point that was added at an inappropriate position.

[0178] When the densification unit 130 removes a three-dimensional point, the data of the three-dimensional point may be deleted from the storage unit 160, or the data of the three-dimensional point may not be deleted from the storage unit 160 by not displaying the three-dimensional point on the display device 520, for example.

[0179] The predetermined angle may be arbitrarily determined in advance and is not particularly limited.

[0180] Furthermore, for example, the assignment unit 120 can estimate that the type of object is a wall for three-dimensional points that form a plane whose normal direction is orthogonal to the up-down direction.

[0181] Furthermore, as described above, the ceiling of a building or the like is generally located at the top of an image captured by the camera 301. Therefore, similar to the method of estimating the object type as "ground," when the position of the camera 301 that generated the image used to generate the densified 3D point cloud and the position of the actual object corresponding to the 3D point cloud are known, the assignment unit 120 can identify the position of the ceiling located above the camera 301 based on trajectory information indicating the trajectory of the camera 301, and can assign the object type of the 3D point located at the identified position to "ceiling."

[0182] <Densification processing> Fig. 13 is a diagram for explaining the densification process executed by the 3D point cloud densification device 100 according to the embodiment. Specifically, Fig. 13(a) is a diagram showing the 3D point cloud and auxiliary information acquired by the acquisition unit 110. Fig. 13(b) is a diagram showing a state in which the densification unit 130 has connected mesh lines to the 3D point cloud shown in Fig. 13(a). Fig. 13(c) is a diagram showing a dense 3D point cloud generated by the densification unit 130 performing the densification process on the 3D point cloud shown in Fig. 13(a), i.e., by generating new 3D points for the 3D point cloud.

[0183] Note that FIG. 13(a) shows an example in which the assigning unit 120 assigns type information indicating the type of object to each 3D point as auxiliary information.

[0184] For example, the acquisition unit 110 acquires a three-dimensional point cloud stored in the three-dimensional point cloud storage unit 163. Next, the assignment unit 120 identifies the type of object constituted by each pixel included in the image, for example, by performing image recognition on the image stored in the video storage unit 161. Next, type information indicating the type of object constituted by each pixel is assigned as auxiliary information to the three-dimensional point corresponding to each pixel. As a result, as shown in (a) of FIG. 13, each three-dimensional point is assigned type information such as, for example, "tree," "house," or "road."

[0185] The three-dimensional point may be assigned type information indicating multiple types, such as "road" and "house."

[0186] Next, the densification unit 130 identifies polygons whose vertices are three-dimensional points to which type information indicating the same type has been assigned, as shown in (b) of Fig. 13. For example, the densification unit 130 identifies polygons by meshing three-dimensional points that have at least one of the same type indicated in the type information together with a mesh line.

[0187] Next, the densification unit 130 generates new 3D points within the polygon, as shown in FIG. 13(c), to generate a dense 3D point cloud.

[0188] In this way, the densification unit 130 can, for example, prevent the generation of mesh lines connecting three-dimensional points that constitute different objects such as roads, trees, or houses by connecting only three-dimensional points of the same type with mesh lines.

[0189] The densification unit 130 performs meshing by projecting three-dimensional points onto the image, for example. An example of a meshing method is Delaunay triangulation.

[0190] In addition, when there are results of instance segmentation by the assignment unit 120, the densification unit 130 may perform meshing under constraints such as, for example, only being able to connect three-dimensional points assigned with type information indicating the same type with mesh lines within the same area as three-dimensional points assigned with type information indicating the same type (i.e., an area surrounded by three-dimensional points).

[0191] This prevents mesh lines from being generated that connect three-dimensional points when the types indicated by the type information are the same and three-dimensional points that constitute different objects are adjacent to each other.

[0192] In addition, the densification unit 130 may project (backproject) the dense three-dimensional point cloud to generate a dense depth image, generate a three-dimensional point cloud based on the generated dense depth image, and perform densification on the generated three-dimensional point cloud.

[0193] Furthermore, the densification unit 130 may perform meshing so as to connect three-dimensional points that are close to each other in three-dimensional space, rather than projecting three-dimensional points onto an image and determining positions at which mesh lines are connected.

[0194] The densification unit 130 may also calculate the normal of a mesh line (more specifically, a polygon) and filter the mesh line using the calculated normal. For example, when a polygon is identified whose vertices are three-dimensional points that constitute the ground, a road, or the like, the normal direction of the polygon is likely to be the same as the up-down direction (the direction of gravity). In other words, if the normal direction of a polygon whose vertices are at least three three-dimensional points that constitute the ground, a road, or the like is not parallel to the up-down direction, it is highly likely that at least one of the at least three three-dimensional points has been assigned incorrect type information. Therefore, the densification unit 130 may determine whether the angle between the normal and the up-down direction is equal to or smaller than a predetermined angle, and based on the determination result, decide whether to delete the mesh line.

[0195] The predetermined angle may be determined based on the distance from the lens center of the camera 301 to the polygon having the normal line. For example, the predetermined angle may be set to be larger as the distance becomes smaller.

[0196] <Summary> FIG. 14 is a flowchart showing the processing procedure of the three-dimensional point cloud densification device 100 according to the embodiment.

[0197] First, the acquisition unit 110 acquires a plurality of three-dimensional points, each of which indicates the three-dimensional position of at least one object, and auxiliary information for creating a polygon (S301). The acquisition unit 110 may acquire point cloud data including the plurality of three-dimensional points from the three-dimensional point cloud generating device 200, or may acquire it from the three-dimensional point cloud storage unit 163. Furthermore, the acquisition unit 110 may acquire the auxiliary information from the user via the UI unit 140, or may acquire the auxiliary information generated by the assignment unit 120 from the assignment unit 120.

[0198] Next, the densification unit 130 identifies polygons having at least three vertices among the plurality of 3D points based on the auxiliary information (S302). For example, the densification unit 130 determines a group to which each of the 3D points belongs based on the auxiliary information. For example, when the auxiliary information is type information, the densification unit 130 determines (classifies) 3D points to which type information including at least one of the same type is assigned into the same group. Next, the densification unit 130 identifies polygons having vertices that are 3D points belonging to the same group.

[0199] Next, the densification unit 130 generates new three-dimensional points within the identified polygon (S303).

[0200] [Effects, etc.] As described above, the three-dimensional point cloud densification device 100 according to the embodiment includes an acquisition unit 110 that acquires a plurality of three-dimensional points, each of which indicates the three-dimensional position of at least one object, and auxiliary information for creating a polygon, and a densification unit 130 that, based on the auxiliary information, identifies a polygon having at least three of the plurality of three-dimensional points as vertices, and generates a new three-dimensional point within the polygon.

[0201] This allows the densification unit 130 to identify a polygon having at least three appropriate 3D points as vertices based on the auxiliary information, and therefore the densification unit 130 can generate a new 3D point at an appropriate position by adding a 3D point to the identified polygon.

[0202] Furthermore, for example, the acquisition unit 110 acquires auxiliary information via an operation device (for example, the UI unit 140) for receiving instructions from a user. For example, the instruction is an instruction to specify the shape of a polygon.

[0203] According to this, the densification unit 130 acquires a polygon shape specification from the user and specifies the polygon, so that new 3D points can be generated at positions desired by the user.

[0204] Moreover, for example, the three-dimensional point cloud densification device 100 further includes an adding unit 120 that generates auxiliary information based on image data obtained by photographing at least one object.

[0205] Furthermore, for example, the assigning unit 120 generates type information indicating the types of a plurality of objects included in the image data as auxiliary information, and assigns the type information corresponding to each of the plurality of 3D points to each of the plurality of 3D points. In this case, for example, the densification unit 130 identifies a polygon having vertices made up of at least three 3D points of the same object type.

[0206] This allows the densification unit 130 to generate new 3D points so as to complement between 3D points that belong to the same type of object.

[0207] Furthermore, for example, the adding unit 120 generates the edges of objects included in the image data as auxiliary information. In this case, for example, the densifying unit 130 identifies polygons that do not intersect with the edges.

[0208] This allows the densification unit 130 to suppress the generation of new 3D points between different objects at positions where no object exists.

[0209] Furthermore, for example, the acquiring unit 110 acquires, as auxiliary information, camera information indicating the position of the camera 301 that generated the plurality of image data obtained by capturing an image of at least one object and the attitude of the camera 301 at the time of capturing the image. In this case, for example, the densification unit 130 identifies polygons based on the camera information.

[0210] Also, for example, the densification unit 130 identifies a polygon having at least three vertices of a plurality of three-dimensional points located below a predetermined position based on the up and down direction in a virtual three-dimensional space representing a plurality of three-dimensional points, which is determined based on camera information.

[0211] According to this, the densification unit 130 can, for example, determine the type of object by considering multiple three-dimensional points located below among the three-dimensional points included in the three-dimensional point cloud as the ground such as a road, and identify polygons whose vertices are three-dimensional points of the same object type.

[0212] Furthermore, for example, the densification unit 130 determines the height of the camera 301 in the up-down direction in the virtual three-dimensional space based on the camera information, and determines a predetermined position based on the determined height.

[0213] This allows the densification unit 130 to more appropriately identify 3D points whose object type is the ground, based on the height of the camera 301.

[0214] Furthermore, for example, the densification unit 130 identifies a polygon based on the angle formed between the normal direction of the polygon and the up-down direction, the polygon having at least three vertices among a plurality of three-dimensional points located below a predetermined position.

[0215] For example, the normal direction of a polygon having vertices that are three-dimensional points constituting the ground, a road, etc., tends to be the same direction as the up-down direction. In other words, if the angle between the normal direction of a polygon having vertices that are at least three three-dimensional points constituting the ground, a road, etc. and the up-down direction is not parallel but too large, there is a high possibility that at least one of the at least three three-dimensional points is noise rather than a three-dimensional point constituting the ground, a road, etc. Therefore, with this configuration, for example, the densification unit can prevent new three-dimensional points from being generated within a polygon having vertices that are three-dimensional points that are unlikely to be three-dimensional points constituting the ground, a road, etc.

[0216] Furthermore, for example, the auxiliary information is information indicating a group to which each of the multiple 3D points belongs. In this case, for example, the densification unit 130 identifies a polygon having at least three vertices of the 3D points belonging to the same group based on the auxiliary information.

[0217] According to this, the densification unit 130 can identify polygons whose vertices are three-dimensional points that are likely to be the same or that constitute the same objects, such as three-dimensional points of the same object type, three-dimensional points that can be connected by a line without crossing the edges of objects contained in the image data, or three-dimensional points that are located below a predetermined position in the virtual three-dimensional space that represents the three-dimensional points, by identifying polygons based on auxiliary information that indicates three-dimensional points that are likely to be the same or that constitute the same objects as three-dimensional points that belong to the same group.

[0218] In addition, a three-dimensional point cloud densification method according to one embodiment of the present disclosure includes an acquisition step (S301) of acquiring a plurality of three-dimensional points, each of which indicates the three-dimensional position of at least one object, and auxiliary information for creating a polygon, an identification step (S302) of identifying a polygon having at least three of the plurality of three-dimensional points as vertices based on the auxiliary information, and a densification step (S303) of generating new three-dimensional points within the polygon.

[0219] This provides the same effect as the three-dimensional point cloud densification device.

[0220] The present disclosure may be realized as a program that causes a computer to execute the steps included in the above-described 3D point cloud densification method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network, such as the Internet.

[0221] (Other embodiments) Although the three-dimensional point cloud densification device and the like according to the present disclosure have been described based on the above-described embodiment, the present disclosure is not limited to the above-described embodiment.

[0222] For example, the 3D point cloud generating device 200 may use the dense 3D point cloud generated by the 3D point cloud densifying device 100 to generate a 3D point cloud again.

[0223] Furthermore, for example, if the 3D point cloud densification device 100 automatically performs densification processing when a 3D point cloud is input without receiving an instruction from a user and outputs the densification processing, the UI unit 140 may not be provided.

[0224] Furthermore, for example, in the above embodiment, each of the components of the processing unit, such as the control unit 150, included in the 3D point cloud densification device 100 may be configured with one or more electronic circuits. The one or more electronic circuits may be general-purpose circuits or dedicated circuits. The one or more electronic circuits may include, for example, a semiconductor device, an integrated circuit (IC), or a large-scale integration (LSI). The IC or LSI may be integrated on a single chip or on multiple chips. Although the IC or LSI is referred to here as an IC or LSI, the name may vary depending on the degree of integration, and may be called a system LSI, a very large-scale integration (VLSI), or an ultra-large-scale integration (ULSI). A field-programmable gate array (FPGA), which is programmed after the LSI is manufactured, can also be used for the same purpose.

[0225] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, or a computer program. Alternatively, the general or specific aspects may be realized as a computer-readable non-transitory recording medium such as an optical disk, a hard disk drive (HDD), or a semiconductor memory on which the computer program is stored. Alternatively, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0226] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, and forms realized by arbitrarily combining the components and functions of the embodiments within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0227] The present disclosure is applicable to a system that can appropriately increase the number of points in three-dimensional point cloud data of an object or the like generated from an image or the like, for example. [Explanation of symbols]

[0228] 100 3D point cloud densification device 110 Acquisition Department 120 Assignment section (generation section) 130 High density section 140 UI section 150 control section 160 Storage section 161 Video storage unit 162 Camera information storage unit 163 3D point cloud storage 164 Dense 3D point cloud storage 200 3D point cloud generator 300 Filming Department 301 Camera 400 3D point cloud generation system 500 computers 510 Operating device 520 Display device 600 point cloud images 610 Superimposed Images 620 Adjustment section 630 Selection Section 640 Cursor 650 Operation trajectory 700, 701, 702, 710, 711, 712 objects 720, 721, 722, 723, 724 polygons

Claims

1. an acquisition unit that acquires a plurality of three-dimensional points, each of which indicates a three-dimensional position of at least one object, and auxiliary information for creating a polygon; identifying the polygon having at least three vertices of the plurality of three-dimensional points based on the auxiliary information; a densification unit that generates new three-dimensional points within the polygon; The auxiliary information is information for selecting at least one of the plurality of three-dimensional points and a mesh line connecting any of the plurality of three-dimensional points. Three-dimensional point cloud densification device.

2. the acquisition unit acquires the auxiliary information via an operation device for receiving instructions from a user; The instruction specifies the shape of the polygon. The three-dimensional point cloud densification device according to claim 1 .

3. The image processing device further includes a generating unit that generates the auxiliary information based on image data obtained by photographing the at least one object. The three-dimensional point cloud densification device according to claim 1 .

4. the generation unit generates type information indicating types of a plurality of objects included in the image data as the auxiliary information, and assigns the type information corresponding to each of the plurality of three-dimensional points to each of the plurality of three-dimensional points; The densification unit identifies the polygon having vertices of at least three three-dimensional points of the same type of object. The three-dimensional point cloud densification device according to claim 3 .

5. the generation unit generates an edge of an object included in the image data as the auxiliary information; The densification unit identifies the polygons that do not intersect with the edge. The three-dimensional point cloud densification device according to claim 3 .

6. the acquisition unit acquires, as the auxiliary information, camera information indicating a position and an attitude of a camera at the time of capturing an image of the at least one object, the camera having generated a plurality of image data sets obtained by capturing an image of the at least one object; The densification unit identifies the polygon based on the camera information. The three-dimensional point cloud densification device according to claim 1 .

7. The densification unit identifies the polygon having vertices that are at least three of the three-dimensional points located below a predetermined position, based on the up-down direction in the virtual three-dimensional space representing the three-dimensional points, which is determined based on the camera information. The three-dimensional point cloud densification device according to claim 6 .

8. The densification unit determines a height of the camera in the up-down direction based on the camera information, and determines the predetermined position below the determined height. The three-dimensional point cloud densification device according to claim 7 .

9. The densification unit identifies the polygon based on an angle formed between the vertical direction and a normal direction of the polygon having vertices that are at least three of the plurality of three-dimensional points located below the predetermined position. The three-dimensional point cloud densification device according to claim 7 or 8.

10. the auxiliary information is information indicating a group to which each of the plurality of 3D points belongs, The densification unit identifies the polygons having vertices that are at least three of the three-dimensional points that belong to the same group based on the auxiliary information. The three-dimensional point cloud densification device according to claim 1 .

11. an acquisition step of acquiring a plurality of 3D points each indicating a 3D position of at least one object and auxiliary information for creating a polygon; a specifying step of specifying the polygon having at least three vertices of the plurality of 3D points based on the auxiliary information; a densification step of generating new 3D points within the polygons; The auxiliary information is information for selecting at least one of the plurality of three-dimensional points and a mesh line connecting any of the plurality of three-dimensional points. Three-dimensional point cloud densification method.

12. A method for causing a computer to execute the three-dimensional point cloud densification method according to claim 11. program.

Images