Three-dimensional point cloud densification device, three-dimensional point cloud densification method, and program

Abstract

Generate a three-dimensional point at the appropriate location. [Solution] The three-dimensional point cloud densification device acquires a plurality of three-dimensional points indicating the three-dimensional position of at least one object in a virtual three-dimensional space, and camera information indicating the position and orientation of the camera when at least one object was photographed in real three-dimensional space to generate the plurality of three-dimensional points (S301). Based on the camera information, it determines the vertical direction in the virtual three-dimensional space and identifies a polygon having at least three three-dimensional points as vertices from the plurality of three-dimensional points (S302). Only if (i) the angle between the vertical direction and the normal direction of the polygon is smaller than a first threshold, and (ii) the polygon is located below a second threshold in the virtual three-dimensional space, it generates at least one new three-dimensional point in the polygon (S303).

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional point cloud densification device, a three-dimensional point cloud densification method, and a program.

Background Art

[0002] In recent years, there has been a technique for measuring the three-dimensional shape of an object by using multi-viewpoint images obtained by photographing an object (such as a person or a building) from multiple viewpoints, and generating a three-dimensional model of the object consisting of a plurality of three-dimensional points (a three-dimensional point cloud) (see, for example, Patent Document 1).

[0003] In the technique disclosed in Patent Document 1, after geometrically estimating the camera parameters (the position, orientation, pixel unit focal length, lens distortion, etc.) when each image constituting the multi-viewpoint image is taken, a three-dimensional model of the object is generated by projecting each pixel on each image onto the three-dimensional space using the estimated camera parameters.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The three-dimensional points are generated, for example, based on the positions (feature points) that are the features of the object included in a plurality of images.

[0006] Here, when the number of feature points is small, the number of points in the three-dimensional point cloud that constitutes the object decreases. In this case, new three-dimensional points are added to the three-dimensional point cloud as the three-dimensional points that constitute the three-dimensional model. As a result, a three-dimensional point cloud with dense points is generated, and a three-dimensional model that more closely resembles the appearance of the object is generated.

[0007] Thus, when adding new three-dimensional points to a three-dimensional point cloud that constitutes a three-dimensional model, it is necessary to add the three-dimensional points at appropriate locations.

[0008] This disclosure provides a three-dimensional point cloud densification device, etc., that can generate three-dimensional points at appropriate locations. [Means for solving the problem]

[0009] A three-dimensional point cloud densification device according to one aspect of the present disclosure comprises at least one memory and a circuit connected to the at least one memory, the circuit each acquiring a plurality of three-dimensional points indicating the three-dimensional position of at least one object in a virtual three-dimensional space and camera information indicating the position and orientation of a camera when the at least one object was photographed in real three-dimensional space to generate the plurality of three-dimensional points, determining the vertical direction in the virtual three-dimensional space based on the camera information, identifying a polygon having at least three three-dimensional points as vertices from the plurality of three-dimensional points, and generating at least one new three-dimensional point in the polygon only if (i) the angle between the vertical direction and the normal direction of the polygon is smaller than a first threshold, and (ii) the polygon is located below a second threshold in the virtual three-dimensional space.

[0010] Furthermore, a three-dimensional point cloud densification method according to one aspect of the present disclosure acquires a plurality of three-dimensional points indicating the three-dimensional position of at least one object in a virtual three-dimensional space, and camera information indicating the position and orientation of a camera when the at least one object was photographed in real three-dimensional space in order to generate the plurality of three-dimensional points, determines the vertical direction in the virtual three-dimensional space based on the camera information, identifies a polygon having at least three three-dimensional points as vertices from the plurality of three-dimensional points, and generates at least one new three-dimensional point in the polygon only if (i) the angle between the vertical direction and the normal direction of the polygon is smaller than a first threshold, and (ii) the polygon is located below a second threshold in the virtual three-dimensional space.

[0011] Furthermore, this disclosure may be implemented as a program that causes a computer to execute the steps included in the above-described three-dimensional point cloud densification method. Alternatively, this disclosure may be implemented as a non-temporary recording medium such as a CD-ROM readable by the computer on which the program is recorded. Furthermore, this disclosure may be implemented as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network such as the Internet. [Effects of the Invention]

[0012] According to this disclosure, it is possible to provide a three-dimensional point cloud densification device that can generate three-dimensional points at appropriate locations. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a diagram illustrating a three-dimensional point cloud generation system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing the configuration of a three-dimensional point cloud generation system according to an embodiment. [Figure 3] Figure 3 is a diagram illustrating the dense three-dimensional point cloud generated by the three-dimensional point cloud generation system according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of the three-dimensional point group densification device according to the embodiment. [Figure 5] FIG. 5 is a sequence diagram showing the processing procedures of each part included in the three-dimensional point group densification device according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the acquisition process of the three-dimensional point group densification device according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an example of an image displayed by the UI unit included in the three-dimensional point group densification device according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of an image when the UI unit included in the three-dimensional point group densification device is operated by a user. [Figure 9] FIG. 9 is a diagram showing another example of an image when the UI unit included in the three-dimensional point group densification device is operated by a user. [Figure 10] FIG. 10 is a diagram for explaining an example of a method for generating auxiliary information executed by the three-dimensional point group densification device according to the embodiment. [Figure 11] FIG. 11 is a diagram for explaining another example of a method for generating auxiliary information executed by the three-dimensional point group densification device according to the embodiment. [Figure 12] FIG. 12 is a diagram for explaining a photographing method by a camera included in the three-dimensional point group generation system according to the embodiment. [Figure 13] FIG. 13 is a diagram for explaining the densification process executed by the three-dimensional point group densification device according to the embodiment. [Figure 14] FIG. 14 is a flowchart showing the processing procedure of the three-dimensional point group densification device according to the embodiment.

Embodiments for Carrying Out the Invention

[0014] (Summary of the Present Disclosure) A three-dimensional point cloud densification device according to one aspect of the present disclosure includes an acquisition unit that acquires a plurality of three-dimensional points each indicating the three-dimensional position of at least one object, and auxiliary information for creating a polygon, and a densification unit that identifies a polygon having at least three of the plurality of three-dimensional points as vertices based on the auxiliary information and generates a new three-dimensional point within the polygon.

[0015] According to this, the densification unit can identify a polygon having at least three appropriate three-dimensional points as vertices based on the auxiliary information. Therefore, the densification unit can generate a new three-dimensional point at an appropriate position by adding (generating) a three-dimensional point within the identified polygon.

[0016] Also, for example, the acquisition unit acquires the auxiliary information via an operating device for receiving an instruction from a user, and the instruction specifies the shape of the polygon.

[0017] According to this, since the densification unit acquires the specification of the polygon shape from the user to identify the polygon, a new three-dimensional point can be generated at a position desired by the user.

[0018] Also, for example, the three-dimensional point cloud densification device according to the present disclosure further includes a generation unit that generates the auxiliary information based on image data obtained by photographing the at least one object.

[0019] Also, for example, the generation unit generates type information indicating the types of a plurality of objects included in the image data as the auxiliary information, assigns the type information corresponding to each of the plurality of three-dimensional points to each of the plurality of three-dimensional points, and the densification unit identifies the polygon having at least three three-dimensional points with the same object type as vertices.

[0020] According to this, the densification unit can generate a new three-dimensional point so as to complement between three-dimensional points having the same type of object to be configured.

[0021] Furthermore, for example, the generation unit generates the edges of objects included in the image data as auxiliary information, and the densification unit identifies the polygons that do not intersect with the edges.

[0022] According to this, the densification unit can suppress the generation of new three-dimensional points between different objects, at locations where no objects exist.

[0023] Furthermore, for example, the acquisition unit acquires camera information as auxiliary information, which indicates the position and orientation of the camera at the time of shooting, and the densification unit identifies the polygon based on the camera information.

[0024] Furthermore, for example, the densification unit identifies the polygon whose vertices are at least three of the plurality of three-dimensional points located below a predetermined position, based on the vertical direction in a virtual three-dimensional space representing the plurality of three-dimensional points, which is determined based on the camera information.

[0025] According to this, the densification unit can, for example, determine the type of object by identifying multiple three-dimensional points located below it among the three-dimensional points included in the three-dimensional point cloud as ground such as a road, and then identify polygons whose vertices are three-dimensional points of the same object type.

[0026] Furthermore, for example, the high-density unit determines the height of the camera in the vertical direction based on the camera information, and determines the predetermined position below the determined height.

[0027] According to this, the densification unit can more appropriately identify three-dimensional points where the object type is the ground, based on the camera's height.

[0028] Furthermore, for example, the densification unit identifies the polygon based on the angle between the normal direction of the polygon whose vertices are at least three of the plurality of three-dimensional points located below the predetermined position and the vertical direction.

[0029] For example, the normal direction of a polygon whose vertices are three-dimensional points that constitute the ground or a road is likely to be the same direction as the vertical direction. In other words, if the angle between the normal direction of a polygon whose vertices are at least three three-dimensional points that constitute the ground or a road is not parallel and is too large, then it is highly likely that at least one of those at least three three-dimensional points is noise rather than a three-dimensional point that constitutes the ground or a road. Therefore, with such a configuration, for example, the densification unit can prevent the generation of new three-dimensional points within a polygon whose vertices are at three-dimensional points that are highly unlikely to be three-dimensional points that constitute the ground or a road.

[0030] Furthermore, for example, the auxiliary information is information indicating the group to which each of the plurality of three-dimensional points belongs, and the densification unit identifies the polygon whose vertices are at least three of the three-dimensional points belonging to the same group, based on the auxiliary information.

[0031] According to this, the densification unit can identify polygons based on auxiliary information that indicates three-dimensional points that are likely to be composed of the same object or are of the same type, such as three-dimensional points of the same type, three-dimensional points that can be connected by lines without crossing the edges of objects included in the image data, or three-dimensional points located below a predetermined position in a virtual three-dimensional space (virtual space) that represents three-dimensional points, as three-dimensional points belonging to the same group.

[0032] Furthermore, a method for densifying a three-dimensional point cloud according to one aspect of the present disclosure includes an acquisition step of acquiring a plurality of three-dimensional points, each indicating the three-dimensional position of at least one object, and auxiliary information for creating a polygon; an identification step of identifying a polygon whose vertices are at least three of the plurality of three-dimensional points, based on the auxiliary information; and a densification step of generating new three-dimensional points within the polygon.

[0033] According to this, it produces the same effect as a three-dimensional point cloud densification device.

[0034] Furthermore, this disclosure may be implemented as a program that causes a computer to execute the steps included in the above-described three-dimensional point cloud densification method.

[0035] The embodiments of the three-dimensional point cloud densification apparatus and the like described below will be explained in detail with reference to the drawings. Each embodiment described below is merely one specific example of the disclosure. Therefore, the numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in each embodiment below are examples only and are not intended to limit the scope of the disclosure.

[0036] Furthermore, among the components in the following embodiments, those components not described in the independent claims representing an implementation of one aspect of this disclosure will be described as optional components. The implementations of this disclosure are not limited to the current independent claims and may also be expressed by other independent claims.

[0037] Furthermore, each figure is a schematic diagram and not necessarily a strictly accurate representation. Also, in each figure, substantially identical components are given the same reference numerals, and redundant explanations may be omitted or simplified.

[0038] (Embodiment) [Three-dimensional point cloud generation system] First, the configuration of the three-dimensional point cloud generation system 400, including the three-dimensional point cloud densification device 100 according to the embodiment, will be described.

[0039] Figure 1 is a diagram illustrating a three-dimensional point cloud generation system 400 according to an embodiment. Figure 2 is a block diagram showing the configuration of the three-dimensional point cloud generation system 400 according to an embodiment.

[0040] The three-dimensional point cloud generation system 400 is a system that includes an imaging unit 300, which is an imaging device (camera) that images a predetermined area, and a system that generates a three-dimensional point cloud (more specifically, point cloud data representing a three-dimensional point cloud) consisting of multiple three-dimensional points that constitute a three-dimensional model of an object contained in multiple images, based on multiple images (more specifically, image data) generated by the imaging unit 300, and generates a dense three-dimensional point cloud (more specifically, data representing a dense three-dimensional point cloud) by generating and adding new three-dimensional points to the generated three-dimensional point cloud.

[0041] Furthermore, an object refers, for example, to at least one object within a given space (a predetermined area). Alternatively, for example, an object, when generated by photography, refers to at least one subject. Alternatively, for example, an object, when generated by distance measurement, refers to at least one object being measured.

[0042] Furthermore, when an object (for example, objects 700, 701, and 702 shown in Figure 1) is photographed in real life, the object (for example, objects 710, 711, and 712 shown in Figure 1) included in those multiple images is reconstructed in a virtual three-dimensional space, that is, represented (generated), and this object is called a three-dimensional model. A three-dimensional model is a collection of multiple three-dimensional points (a three-dimensional point cloud).

[0043] Furthermore, the designated area is an area that includes stationary objects, moving objects such as people, or both as objects. In other words, the designated area is an area that includes, for example, at least one of stationary objects and moving animals as subjects. Examples of a designated area that includes both stationary objects and animals include a sports match such as basketball, or a street with people or cars. Moreover, the designated area may not be limited to specific objects that are subjects, but may also include a scene such as a landscape. Hereafter, the designated area that includes not only specific objects but also landscapes will simply be referred to as an object.

[0044] Point cloud data is information that includes data for each of multiple three-dimensional points. A three-dimensional point is, for example, information that indicates the three-dimensional position (three-dimensional coordinate) of at least one object. A three-dimensional point includes, for example, ternary information consisting of the X, Y, and Z components of a three-dimensional space consisting of mutually orthogonal X, Y, and Z axes (i.e., position information indicating coordinates). In addition to position information, a three-dimensional point may also include so-called attribute information such as color information indicating the color of the three-dimensional point, and texture information representing the surface shape of the three-dimensional point and its surroundings.

[0045] The three-dimensional point cloud generation system 400 comprises an imaging unit 300, a three-dimensional point cloud generation device 200, and a three-dimensional point cloud densification device 100.

[0046] The imaging unit 300 is an imaging device for photographing objects. The imaging unit 300 consists of, for example, one or more cameras 301, and outputs multiple images generated by photographing objects to the three-dimensional point cloud generator 200.

[0047] The imaging unit 300 may be equipped with multiple cameras 301 or with a single camera 301. For example, the user may move a single camera 301 and have the camera 301 capture multiple images (in other words, multi-view video data) with different viewpoints. In other words, the multiple images that the imaging unit 300 outputs to the three-dimensional point cloud generator 200 are images captured (generated) by cameras 301 of the imaging unit 300, at least one of which is different in position and orientation.

[0048] Furthermore, the camera 301 may be a camera that generates two-dimensional images, or a camera equipped with a three-dimensional measurement sensor that generates a three-dimensional model. In this embodiment, the imaging unit 300 has a camera 301 that generates two-dimensional images.

[0049] The imaging unit 300 is configured to be connectable via wired or wireless connection to the three-dimensional point cloud generator 200 itself, or to a hub (not shown) such as a communication device or server, so that the generated images can be output to the three-dimensional point cloud generator 200.

[0050] The images generated by the imaging unit 300 may be output in real time, or the images may be recorded once in memory or an external storage device such as a cloud server, and then output from that external storage device.

[0051] Furthermore, if the imaging unit 300 has multiple cameras 301, each of the multiple cameras 301 may be a fixed camera 301 such that at least one of its position and orientation differs from the others. The orientation of the camera 301 refers to at least one of the shooting direction of the camera 301 and the tilt of the camera 301.

[0052] The type of camera 301 is not particularly limited. Camera 301 may be, for example, a mobile camera such as a video camera, smartphone, or wearable camera, a mobile camera such as a drone with a shooting function, or a fixed camera such as a surveillance camera.

[0053] The three-dimensional point cloud generator 200 is a device that generates a three-dimensional point cloud based on multiple images output from the imaging unit 300.

[0054] The three-dimensional point cloud generator 200 estimates (calculates) camera information (also called camera parameters) indicating the position and orientation of the image when the imaging unit 300 generated the image, based on multiple images.

[0055] Here, camera information refers to parameters that indicate the characteristics of camera 301, including internal parameters such as the focal length and image center of camera 301, external parameters indicating the position (more specifically, three-dimensional position) and orientation of camera 301 at the time of shooting, and parameters such as the distance between camera 301 and the object. Furthermore, if camera 301 is moved and multiple images are taken with camera 301, the camera information may also include information indicating the trajectory of camera 301 at the time of shooting, which generated the multiple images.

[0056] The method by which the three-dimensional point cloud generator 200 estimates the position and orientation of the camera 301 is not particularly limited. The three-dimensional point cloud generator 200 estimates the position and orientation of the camera 301 when it generates the image using, for example, Visual-SLAM (Simultaneous Localization and Mapping) technology, Structure-from-Motion (SfM) technology, etc. The three-dimensional point cloud generator 200 may include a LiDAR in the imaging unit 300 and estimate camera information based on information obtained from the LiDAR.

[0057] Furthermore, camera information may be output from the imaging unit 300 to the three-dimensional point cloud generator 200, or it may be pre-stored in a memory (not shown) provided by the three-dimensional point cloud generator 200. In this case, the three-dimensional point cloud generator 200 does not need to estimate the camera information.

[0058] The three-dimensional point cloud generator 200 generates a three-dimensional model (three-dimensional point cloud) in a three-dimensional space (virtual space) based on camera information and multiple images.

[0059] The three-dimensional point cloud generator 200 is implemented by a computer system comprising, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. In this embodiment, the three-dimensional point cloud generator 200 is implemented together with the three-dimensional point cloud density enhancement device 100 as a computer 500.

[0060] The computer 500 is a device (system) that includes a processor, a memory in which a program executed by the processor is stored, a communication interface for communicating with the imaging unit 300, etc., an operating device 510 for receiving input from the user, a display device 520 for displaying images and three-dimensional point clouds, etc.

[0061] The communication interface may be a wireless communication interface comprising an antenna and a wireless communication circuit, or it may be a wired communication interface comprising a connector connected to a communication line.

[0062] The operating device 510 is a user interface operated by the user, and is implemented using a keyboard, mouse, touch panel, or the like.

[0063] The display device 520 is a display for displaying images and three-dimensional point clouds, etc.

[0064] The three-dimensional point cloud generator 200 outputs the generated three-dimensional point cloud and camera information to the three-dimensional point cloud densification device 100. The three-dimensional point cloud generator 200 may also output multiple images used in generating the three-dimensional point cloud to the three-dimensional point cloud densification device 100.

[0065] The imaging unit 300 and the three-dimensional point cloud generation device 200 may be integrated as a single computer with imaging capabilities.

[0066] Furthermore, if the three-dimensional point cloud generation device 200 and the three-dimensional point cloud densification device 100 are implemented using different computers or the like, the three-dimensional point cloud generation device 200 and the three-dimensional point cloud densification device 100 may be connected to each other via a network for communication, or they may be configured to allow offline data exchange using an HDD (Hard Disk Drive) or the like.

[0067] Furthermore, the three-dimensional point cloud densification device 100 may calculate camera information, or it may store camera information in advance. In this case, the three-dimensional point cloud generation device 200 does not need to output camera information to the three-dimensional point cloud densification device 100.

[0068] The three-dimensional point cloud densification device 100 is a device that generates a dense three-dimensional point cloud by generating new three-dimensional points in the three-dimensional point cloud generated by the three-dimensional point cloud generation device 200.

[0069] The three-dimensional point cloud densification device 100 is implemented by a computer system comprising, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. In this embodiment, the three-dimensional point cloud densification device 100 is implemented as a computer 500 together with the three-dimensional point cloud generation device 200.

[0070] The three-dimensional point cloud densification device 100 and the three-dimensional point cloud generation device 200 may be implemented by a single processor, or they may be implemented by different processors. Furthermore, the programs executed by the three-dimensional point cloud densification device 100 and the three-dimensional point cloud generation device 200 may be stored in a single memory, or they may be stored in different memories.

[0071] Figure 3 is a diagram illustrating the dense three-dimensional point cloud generated by the three-dimensional point cloud generation system 400 according to the embodiment.

[0072] First, as shown in Figure 3(a), the three-dimensional point cloud generator 200 generates a three-dimensional point cloud based on multiple images generated by the imaging unit 300.

[0073] Next, as shown in Figure 3(b), the three-dimensional point cloud densification device 100 meshes the point cloud by connecting multiple three-dimensional points included in the three-dimensional point cloud with straight lines (also called mesh lines).

[0074] Meshing refers to the process of creating polygons whose vertices are multiple three-dimensional points by connecting three-dimensional points with mesh lines. A mesh is formed from multiple polygons.

[0075] In the example shown in Figure 3(b), five polygons, polygons 720, 721, 722, 723, and 724, are created, each with three three-dimensional points as its vertices. Here, the three-dimensional point cloud densification device 100 connects adjacent three-dimensional points with mesh lines so that the mesh lines do not touch (intersect). Of course, the three-dimensional point cloud densification device 100 does not need to generate mesh lines to connect the three-dimensional points; it is sufficient if it can identify the regions (polygons) with each three-dimensional point as its vertex.

[0076] Next, as shown in Figure 3(c), the three-dimensional point cloud densification device 100 generates a dense three-dimensional point cloud by generating new three-dimensional points (additional three-dimensional points) in the region enclosed by the mesh lines, that is, within the polygon.

[0077] Note that "within a polygon" means not only the area enclosed by the mesh lines, but also the area on the mesh lines themselves.

[0078] If the positions connecting the three-dimensional points are not appropriate, that is, if polygons with at least three three-dimensional points as vertices are not properly created, there is a risk that a three-dimensional model different from the actual shape of the object may be generated, such as by adding a cloud of three-dimensional points to locations where the object does not exist.

[0079] Therefore, the three-dimensional point cloud densification device 100 generates new three-dimensional points within a polygon whose vertices are at least three of the multiple three-dimensional points, based on auxiliary information for creating polygons.

[0080] [Configuration of a 3D point cloud densification system] Next, we will explain the details of the configuration of the three-dimensional point cloud densification device 100.

[0081] Figure 4 is a block diagram showing the configuration of a three-dimensional point cloud densification device 100 according to an embodiment.

[0082] The three-dimensional point cloud densification device 100 comprises an acquisition unit 110, an assignment unit (generation unit) 120, a densification unit 130, a UI (User Interface) unit 140, a control unit 150, and a storage unit 160.

[0083] The acquisition unit 110 is a processing unit that acquires data output from the three-dimensional point cloud generator 200 and data stored in the storage unit 160. The acquisition unit 110 acquires data from the three-dimensional point cloud generator 200, the storage unit 160, and external devices such as a server (not shown) via a communication interface or communication line (not shown) for wired or wireless communication provided by the three-dimensional point cloud densification device 100.

[0084] The acquisition unit 110 acquires a plurality of three-dimensional points, each indicating the three-dimensional position of at least one object, and auxiliary information for creating (identifying) a polygon.

[0085] The information required to create a polygon with at least three three-dimensional points as vertices includes primary information and auxiliary information other than the primary information. The primary information is well-known information used to create the polygon. For example, the primary information relates to the distance between two three-dimensional points. If the distance between two three-dimensional points is less than a predetermined threshold, the mesh line is defined by connecting the two three-dimensional points, with those two points as vertices. The auxiliary information is used to select at least one of the three-dimensional points and the mesh line connecting the two three-dimensional points in order to create an appropriate polygon.

[0086] Mesh lines connect, for example, three-dimensional points belonging to the same group, but not three-dimensional points belonging to different groups. Thus, auxiliary information, for instance, indicates the group to which each of multiple three-dimensional points belongs.

[0087] Furthermore, auxiliary information includes, for example, information indicating the type of object composed of the three-dimensional points, information indicating groups arbitrarily set by the user, and information indicating objects included in the image used when generating the three-dimensional point cloud.

[0088] Here, "edge" refers not only to lines that define the outer perimeter of an object in an image, but also to edges within an object in an image (boundaries between two different faces). Furthermore, an edge may also refer to a line that defines a three-dimensional model in a three-dimensional space on a computer (i.e., a virtual three-dimensional space representing multiple three-dimensional points) (a connection of three-dimensional points). Moreover, an edge may be a projection of the line defining the three-dimensional model onto a two-dimensional plane (for example, an image).

[0089] Furthermore, the camera information described above may be used as supplementary information.

[0090] Alternatively, the auxiliary information may be a list of three-dimensional points that may be connected by mesh lines.

[0091] For example, the acquisition unit 110 acquires auxiliary information via an operating device 510 for receiving instructions from the user. Such instructions are, for example, instructions to specify the shape of a polygon. Specifically, instructions to specify the shape of a polygon are instructions to specify three-dimensional points to be connected by mesh lines, and / or instructions to specify the positions of the mesh lines to be connected.

[0092] The acquisition unit 110 is, for example, connected to the operating device 510 in a communicative manner, and acquires auxiliary information from the user via the operating device 510.

[0093] Furthermore, for example, the acquisition unit 110 acquires camera information as auxiliary information, indicating the position (i.e., trajectory) and orientation of the camera 301 at the time of shooting, which generated multiple image data obtained by photographing at least one object. The acquisition unit 110 may also acquire camera information as auxiliary information.

[0094] Furthermore, for example, the acquisition unit 110 acquires multiple image data used when the three-dimensional point cloud was generated.

[0095] The assignment unit 120 is a processing unit that generates auxiliary information. The assignment unit 120 is a processing unit that, for example, assigns auxiliary information corresponding to each three-dimensional point included in the three-dimensional point cloud. Assigning means, for example, adding (associating) auxiliary information with the data of the three-dimensional point.

[0096] For example, the information provision unit 120 generates auxiliary information based on image data obtained by photographing at least one object. This image data is, for example, an image used by the three-dimensional point cloud generator 200 when generating a three-dimensional point cloud. However, this image data may also be an image not used by the three-dimensional point cloud generator 200 when generating the three-dimensional point cloud.

[0097] Specifically, the assignment unit 120 generates type information as auxiliary information indicating the types of multiple objects included in the image data acquired by the acquisition unit 110, and assigns the type information corresponding to each of the multiple three-dimensional points to each of the multiple three-dimensional points.

[0098] Alternatively, for example, the assignment unit 120 generates the edges of objects included in the image data acquired by the acquisition unit 110 as auxiliary information.

[0099] The densification unit 130 is a processing unit that generates a dense three-dimensional point cloud (more specifically, data representing a dense three-dimensional point cloud) by generating and adding new three-dimensional points (more specifically, data representing three-dimensional points) to the three-dimensional point cloud. Based on auxiliary information, the densification unit 130 identifies (creates) a polygon whose vertices are at least three of the multiple three-dimensional points, and generates new three-dimensional points within the identified polygon.

[0100] The new three-dimensional points may be generated, for example, by interpolation between at least two three-dimensional points, or by feature point matching between at least two images (for example, images generated by being captured by camera 301). However, the method for generating the new three-dimensional points is not limited to these methods, and any well-known method may be used.

[0101] For example, the densification unit 130 identifies a polygon whose vertices are at least three of a plurality of three-dimensional points, based on auxiliary information that specifies the shape of the polygon, which is acquired by the acquisition unit 110 from the user via the operating device 510.

[0102] Identifying a polygon means identifying the location of a polygonal region whose vertices are at least three three-dimensional points, which is necessary to determine the location where new three-dimensional points will be generated. Also, similar to mesh lines, it is not necessary for polygons to be created (i.e., for mesh lines to be generated); it is sufficient for polygons to be identified.

[0103] For example, the densification unit 130 identifies a polygon whose vertices are at least three of the three-dimensional points belonging to the same group, based on the auxiliary information, and generates new three-dimensional points within the identified polygon.

[0104] The densification unit 130 may identify polygons by connecting three-dimensional points with mesh lines based on auxiliary information, and generate new three-dimensional points within the identified polygons. Alternatively, the densification unit 130 may identify polygons by first connecting three-dimensional points with mesh lines, and then, based on auxiliary information, partially delete the mesh lines connecting the three-dimensional points (i.e., treat them as if they were not connected by mesh lines), and generate new three-dimensional points within the polygons formed by the remaining mesh lines. In other words, the densification unit 130 may identify polygons by determining the three-dimensional points that will become the vertices of the polygons based on auxiliary information, or it may identify polygons once and then, based on auxiliary information, reduce the number of three-dimensional points that will become vertices and re-identify polygons from which additional three-dimensional points will be generated.

[0105] For example, the densification unit 130 identifies a polygon whose vertices are at least three three-dimensional points of the same object type, and generates new three-dimensional points within the identified polygon. The densification unit 130 (or the assignment unit 120) may also assign type information to the newly generated three-dimensional points that indicates the same type as the three three-dimensional points.

[0106] Alternatively, for example, the densification unit 130 identifies polygons that do not intersect with edges and generates new three-dimensional points within the identified polygons.

[0107] Alternatively, for example, the densification unit 130 identifies a polygon whose vertices are at least three of the multiple three-dimensional points based on camera information, and generates a new three-dimensional point within the identified polygon. Specifically, the densification unit 130 identifies a polygon whose vertices are at least three of the multiple three-dimensional points located below a predetermined position, based on the vertical direction in the virtual three-dimensional space representing the multiple three-dimensional points (i.e., in the virtual space on the computer) determined based on camera information, and generates a new three-dimensional point within the identified polygon.

[0108] The predetermined position may be arbitrarily determined in advance. As described above, the camera information includes, for example, information indicating the position of camera 301 at the time of shooting. For example, the densification unit 130 determines the height of camera 301 in the vertical direction in a virtual three-dimensional space based on the camera information, and more specifically, determines a predetermined position below the determined height based on the determined height. Alternatively, the densification unit 130 may use the position (height) of camera 301 indicated by the camera information as the predetermined height.

[0109] The densification unit 130 may determine the three-dimensional points to be polygon vertices based on auxiliary information and camera information indicating at least one of the trajectory of the camera 301 during shooting and the orientation of the camera 301.

[0110] Furthermore, the camera information includes information indicating the trajectory (movement trajectory) of camera 301 during shooting, and the densification unit 130 may identify polygons whose vertices are at least three of a plurality of three-dimensional points based on the information indicating the trajectory of camera 301 included in the camera information. For example, since the trajectory of camera 301 is the space that camera 301 can pass through, it is highly likely that no objects exist on the trajectory of camera 301. Therefore, the densification unit 130 may not generate or may exclude mesh lines that cross the camera's trajectory.

[0111] The UI unit 140 is a user interface for receiving input from the user and displaying images to the user. The UI unit 140 is implemented by, for example, an operating device 510, a display device 520, etc.

[0112] The control unit 150 is a processing unit for controlling the operation timing of each processing unit, UI unit 140, etc., that are included in the three-dimensional point cloud densification device 100.

[0113] The memory unit 160 is a storage device that stores data such as three-dimensional point clouds, dense three-dimensional point clouds, auxiliary information, multiple images, camera information, and control programs executed by each processing unit of the three-dimensional point cloud densification device 100. The memory unit 160 is implemented by, for example, an HDD or flash memory.

[0114] The memory unit 160 includes, for example, an image memory unit 161, a camera information memory unit 162, a three-dimensional point cloud memory unit 163, and a dense three-dimensional point cloud memory unit 164.

[0115] The image storage unit 161 stores multiple images (multiple image data) used to generate the three-dimensional point cloud.

[0116] The camera information storage unit 162 stores camera information.

[0117] The three-dimensional point cloud storage unit 163 stores the three-dimensional point cloud (three-dimensional point cloud data).

[0118] The dense three-dimensional point cloud storage unit 164 stores a dense three-dimensional point cloud (dense three-dimensional point cloud data). The dense three-dimensional point cloud storage unit 164 may also store only information about three-dimensional points added to the three-dimensional point cloud stored in the three-dimensional point cloud storage unit 163.

[0119] The image memory unit 161, the camera information memory unit 162, the three-dimensional point cloud memory unit 163, and the dense three-dimensional point cloud memory unit 164 may be implemented by a single memory device, or they may be implemented by different memory devices.

[0120] [Processing Procedure for Three-Dimensional Densification Equipment] Next, we will explain in detail the processing procedure for generating a dense three-dimensional point cloud performed by the three-dimensional point cloud densification device 100.

[0121] <Overview> Figure 5 is a sequence diagram showing the processing procedures of each part (more specifically, the UI unit 140, acquisition unit 110, assignment unit 120, and densification unit 130) of the three-dimensional point cloud densification apparatus 100 according to the embodiment.

[0122] First, the acquisition unit 110 acquires information for the densification unit 130 to generate a dense three-dimensional point cloud (S101). For example, the acquisition unit 110 acquires the three-dimensional point cloud and auxiliary information from the storage unit 160.

[0123] The acquisition unit 110 may, for example, start step S101 when it receives an instruction from the user via the UI unit 140 to start the process of generating a dense three-dimensional point cloud (densification process).

[0124] Next, the UI unit 140 displays an image for receiving instructions from the user (S102). Here, for example, the user is given the option to manually add auxiliary information to the three-dimensional point cloud.

[0125] Next, for example, the UI unit 140 determines whether or not it has received input of auxiliary information from the user (S103). This determination can be performed by any processing unit of the three-dimensional point cloud densification device 100, for example, the acquisition unit 110 or the control unit 150.

[0126] If the UI unit 140 determines that it has not received input of auxiliary information from the user (No in S103), the assignment unit 120 generates auxiliary information corresponding to each three-dimensional point (S104).

[0127] If the acquisition unit 110 determines that the UI unit 140 has received input of auxiliary information from the user (Yes in S103), it acquires the auxiliary information from the UI unit 140, and if the assignment unit 120 generates the auxiliary information (S104), it acquires the auxiliary information from the assignment unit 120 (S105).

[0128] Next, the densification unit 130 generates new three-dimensional points based on the three-dimensional point cloud and the auxiliary information acquired by the acquisition unit 110 (S106).

[0129] <Retrieval process> Figure 6 is a flowchart showing the acquisition process of the three-dimensional point cloud densification device 100 according to the embodiment.

[0130] First, the acquisition unit 110 acquires a three-dimensional point cloud from the three-dimensional point cloud generator 200 and stores the acquired three-dimensional point cloud in the three-dimensional point cloud storage unit 163 (S201).

[0131] Next, the acquisition unit 110 acquires images taken in the same space as the three-dimensional point cloud acquired in step S201 from the three-dimensional point cloud generation device 200, and stores the acquired images in the image storage unit 161 (S202). Images taken in the same space are, for example, multiple images used to generate the three-dimensional point cloud. Alternatively, examples of images taken in the same space include the image used when the three-dimensional point cloud was generated based on an image, such as SfM, or images taken by a camera 301 whose positional relationship with the laser device that performed the laser scan is known in order to assign attribute information indicating color to each three-dimensional point of the three-dimensional point cloud when the three-dimensional point cloud was generated using a laser scan.

[0132] Next, the acquisition unit 110 acquires camera information from the three-dimensional point cloud generator 200, indicating the three-dimensional position and orientation of the camera 301 when the image acquired in step S202 was taken, and stores the acquired camera information in the camera information storage unit 162 (S203). The camera information includes, for example, the three-dimensional position and orientation of the camera 301 when the image was taken, if the three-dimensional point cloud was generated based on an image, such as SfM. Alternatively, if the three-dimensional point cloud was generated using laser scanning, the camera information includes information indicating the three-dimensional position and orientation of the camera 301, whose positional relationship with the laser device that performed the laser scan is known. The camera information may also include information indicating the three-dimensional position of the laser device.

[0133] The acquisition unit 110 may acquire any image as long as it captures the same space as the three-dimensional point cloud. For example, the acquisition unit 110 may acquire an image that the three-dimensional point cloud generator 200 does not use to generate the three-dimensional point cloud, as well as camera information indicating the position and orientation of the camera that generated the image, such as a GPS (Global Positioning System) or gyro sensor.

[0134] Furthermore, the acquisition unit 110 does not need to acquire image and camera information if it is not to be used as auxiliary information for the three-dimensional point cloud.

[0135] According to this, the processing performed by the three-dimensional point cloud densification device 100 can be reduced, and the amount of information stored in the storage unit 160 can also be reduced.

[0136] <Example Display> Figure 7 shows an example of an image displayed by the UI unit 140 of the three-dimensional point cloud densification device 100 according to the embodiment. Figure 7 also shows the display device for displaying the image, which is provided by the UI unit 140.

[0137] In step S102 shown in Figure 5, the UI unit 140 displays, for example, a point cloud image 600, an overlay image 610, an image including the adjustment unit 620, and an image including the selection unit 630, as shown in Figure 7.

[0138] Point cloud image 600 is an image containing a three-dimensional point cloud.

[0139] The superimposed image 610 is an image in which the three-dimensional point cloud is superimposed on the image (two-dimensional image) used to generate the three-dimensional point cloud.

[0140] The image including the adjustment unit 620 is an image for receiving user input and changing the orientation in which the three-dimensional point cloud is displayed. The UI unit 140, for example, changes the orientation of the three-dimensional point cloud in the image displayed as the point cloud image 600 in response to the input received by the adjustment unit 620. This allows the user to view the three-dimensional point cloud from various angles by operating the adjustment unit 620.

[0141] The image including the selection unit 630 is an image used to receive input from the user and determine whether to automatically generate auxiliary information (for example, the assignment unit 120) or for the user to decide on the auxiliary information. For example, if the user selects "automatic" (No in step S103 shown in Figure 5), the assignment unit 120 performs the process to generate auxiliary information (step S104 shown in Figure 5). On the other hand, if the user selects "manual" (No in step S103 shown in Figure 5), the UI unit 140 switches the image displayed for receiving input of auxiliary information.

[0142] Figure 8 shows an example of an image when the UI unit 140 of the three-dimensional point cloud densification device 100 according to the embodiment is operated by a user. Specifically, Figure 8(a) shows an example of an image displayed by the UI unit 140 when the user selects "Manual" in the selection unit 630. Figure 8(b) shows an example of an image showing the user inputting auxiliary information by operating the UI unit 140. Figure 8(c) shows another example of an image showing the user inputting auxiliary information by operating the UI unit 140.

[0143] When the UI unit 140 receives a "manual" input from the user via the selection unit 630, it displays an image showing a three-dimensional point cloud, as shown in Figure 8(a).

[0144] In this case, as shown in Figure 8(b), the user moves the cursor 640 to select a three-dimensional point by operating the operating device provided in the UI unit 140, and inputs type information indicating the object type, etc., as auxiliary information to the selected three-dimensional point.

[0145] For example, in the example shown in Figure 8, three-dimensional point clouds without input (assigned) auxiliary information are shown as white circles, and three-dimensional points with input (assigned) auxiliary information are shown with hatching. In this way, the user looks at the three-dimensional point cloud displayed on the UI unit 140 and assigns auxiliary information to each three-dimensional point, for example, "Type 1" where the object type is "tree", "Type 2" where the object type is "house", etc.

[0146] Furthermore, as shown in Figure 8(c), the UI unit 140 may process the data so that the user can select the three-dimensional point cloud all at once by dragging the cursor 640 as indicated by the operation trajectory 650.

[0147] Figure 9 shows another example of an image when the UI unit 140 of the three-dimensional point cloud densification device 100 according to the embodiment is operated by a user. Figure 9(a) corresponds to Figure 8(a), Figure 9(b) corresponds to Figure 8(b), and Figure 9(c) corresponds to Figure 8(c).

[0148] In the example shown in Figure 9, unlike in Figure 8, the UI unit 140 displays a three-dimensional point cloud superimposed on an image (two-dimensional image).

[0149] This makes it easier for users to appropriately select the type of object that is composed of three-dimensional points.

[0150] In Figure 7, the selection unit 630 shows buttons labeled "Automatic" and "Manual" (the part selected by the user), but the selection unit 630 only needs to be able to select whether the user inputs the auxiliary information or has the information generation unit 120 generate it.

[0151] Furthermore, the selection unit 630 may also have options other than "automatic" and "manual," such as "semi-automatic." For example, if the user selects "semi-automatic," the assignment unit 120 generates auxiliary information. Next, the UI unit 140 displays an image containing the auxiliary information generated by the assignment unit 120. Furthermore, if the user wishes to change the auxiliary information generated by the assignment unit 120, they input the changed auxiliary information by operating the UI unit 140.

[0152] This allows the user to begin inputting auxiliary information from a state where the auxiliary information has been roughly and correctly assigned by the assignment unit 120. Therefore, the amount of work required for the user to input auxiliary information to each three-dimensional point as intended can be reduced.

[0153] <Auxiliary information generation> Figure 10 is a diagram illustrating an example of a method for generating auxiliary information performed by the three-dimensional point cloud densification device 100 according to the embodiment. Specifically, Figure 10(a) is a schematic diagram showing the camera 301 taking a picture as viewed from above, Figure 10(b) is a diagram showing an example of an image generated by the camera 301 in Figure 10(a), and Figure 10(c) is a diagram showing an example of the addition unit 120 performing image recognition (object recognition) on the image shown in Figure 10(b).

[0154] In the example shown in Figure 10, the assignment unit 120 recognizes (identifies) the type of object contained in the image by performing image recognition.

[0155] When the position of the camera 301 that generated the image used to create the high-density three-dimensional point cloud, and the position of the actual object corresponding to the three-dimensional point cloud, are known, the relationship between the three-dimensional point and the pixel contained in the image is uniquely determined.

[0156] Therefore, by grouping three-dimensional points according to which object they constitute within the image, it is possible to suppress the generation of mesh lines that span across objects.

[0157] As a result, the densification unit 130 can perform densification processing on the three-dimensional point cloud with high accuracy, that is, to add three-dimensional points at appropriate locations.

[0158] In the example shown in Figure 10, the assignment unit 120 identifies the type of object that each pixel in the image constitutes by performing image recognition. Furthermore, the assignment unit 120 generates and assigns type information, indicating the type of object that each three-dimensional point constitutes, as auxiliary information for each three-dimensional point, based on the relationship between the three-dimensional point and the pixels contained in the image. The densification unit 130 identifies polygons whose vertices are three-dimensional points to which the same type information has been assigned.

[0159] The method for recognizing the type of object is not particularly limited. The assignment unit 120 may, for example, recognize the type of object that each pixel constitutes through image recognition, such as semantic segmentation. Alternatively, the assignment unit 120 may, for example, treat objects of the same type as different types if they are individual objects, such as in instance segmentation. For example, if there are two objects in an image that are identified as "trees," the assignment unit 120 may consider one "tree" and the other "tree" to be different objects.

[0160] Figure 11 is a diagram illustrating another example of a method for generating auxiliary information performed by the three-dimensional point cloud densification device 100 according to the embodiment. Specifically, Figure 11(a) is a schematic diagram showing the camera 301 taking a picture as viewed from above, Figure 11(b) is a diagram showing an example of an image generated by the camera 301 in Figure 11(a), and Figure 11(c) is a diagram showing an example of the application unit 120 performing image recognition on the image shown in Figure 11(b).

[0161] In the example shown in Figure 11, the assignment unit 120 extracts (generates) auxiliary information indicating the edges of objects composed of each pixel in the image by performing the image recognition described above. For example, the assignment unit 120 extracts the edges of objects in the image using the Canny method or the like. In this case, the assignment unit 120 may assign the same information (connection information) to three-dimensional points that can be connected by mesh lines without crossing edges (i.e., without intersecting at a certain point). For example, the assignment unit 120 associates unique ID (Identifier) ​​information and connection information for each three-dimensional point and stores them in the storage unit 160.

[0162] The densification unit 130 identifies polygons whose vertices are three-dimensional points that have the same connection information. This allows the densification unit 130 to connect three-dimensional points with mesh lines without crossing edges extracted by the assignment unit 120 (i.e., without intersecting at a certain point). In other words, the densification unit 130 can mesh the three-dimensional point cloud by limiting the locations where mesh lines are connected based on edges.

[0163] This method can suppress the intersection of mesh lines at locations that are likely to form the contours of objects composed of three-dimensional point clouds. It can also suppress the connection of mesh lines that span across different objects.

[0164] Furthermore, if a stereo camera or an RGB-D camera is used as the camera 301, the high-density unit 130 may mesh the three-dimensional point cloud based on the depth information obtained from the image and the edge information generated by the addition unit 120.

[0165] According to this, even when the three-dimensional point cloud acquired from the three-dimensional point cloud generator 200 is sparse (in other words, the number of three-dimensional points included in the three-dimensional point cloud is small), and there are few three-dimensional points around the contour of an object, making proper meshing impossible, three-dimensional points can be added to the appropriate positions based on depth information.

[0166] The densification unit 130 may determine the number of three-dimensional points included in the three-dimensional point cloud shown in the point cloud data acquired by the acquisition unit 110, and perform densification processing if the number is less than a predetermined number, and not perform densification processing if the number is greater than or equal to a predetermined number.

[0167] Figure 12 is a diagram illustrating the method of capturing images using the camera 301 provided in the three-dimensional point cloud generation system 400 according to the embodiment.

[0168] The densification unit 130 may determine the group to which each three-dimensional point belongs based on camera information indicating at least one of the position and orientation of the camera 301.

[0169] For example, camera 301 often maintains the same height from the ground while taking pictures. Also, when taking pictures with camera 301, the ground is generally located at the bottom (below) of the image.

[0170] For example, suppose the user takes a picture while moving camera 301 horizontally or parallel to the ground. Alternatively, suppose the user takes a picture with camera 301 so that the ground is positioned below them.

[0171] Under these conditions, if the position of the camera 301 that generated the image used to generate the high-density three-dimensional point cloud and the position of the actual object corresponding to the three-dimensional point cloud are known, the assigning unit 120 can identify the position of the ground located below the camera 301 (for example, in the direction of the ground as shown in Figure 12) based on the trajectory information indicating the trajectory of the camera 301, and set the object type of the three-dimensional point located at the identified position to "ground".

[0172] For example, the assignment unit 120 estimates the position of the plane that camera 301 passes through based on the camera trajectory shown in the camera information. The assignment unit 120 also estimates the estimated plane to be a plane parallel to the ground. Based on this, the position corresponding to the ground in the three-dimensional point cloud can be identified based on information indicating the orientation of the ground from camera 301, that is, information indicating the vertical direction, and information indicating the height from camera 301 to the ground. As a result, auxiliary information such as "ground" can be assigned to the three-dimensional point located at that position.

[0173] Thus, for example, the assignment unit 120 may estimate the vertical direction (direction of gravity) in the virtual three-dimensional space where the three-dimensional point cloud is located, based on the estimated plane which is a plane parallel to the ground, and generate information indicating the estimated vertical direction as auxiliary information.

[0174] According to this, the assignment unit 120 can estimate or remove the type of object of multiple three-dimensional points that constitute a plane based on the vertical direction.

[0175] For example, the densification unit 130 identifies a polygon based on the angle between the normal direction of the polygon whose vertices are at least three of a plurality of three-dimensional points located below a predetermined position and the vertical direction. Specifically, for example, the densification unit 130 determines whether the angle between the normal direction of the polygon whose vertices are at least three of a plurality of three-dimensional points located below a predetermined position and the vertical direction is greater than or equal to a predetermined angle. If it determines that the angle is greater than or equal to a predetermined angle, it does not generate any new three-dimensional points within the polygon whose vertices are at least three of a plurality of three-dimensional points located below a predetermined position.

[0176] Alternatively, for example, the densification unit 130 removes three-dimensional points that constitute a plane in which the normal direction is at or above a predetermined angle with respect to the vertical direction estimated by the assignment unit 120. Specifically, the densification unit 130 determines whether the angle between the normal direction of a polygon whose vertices are at least three of the multiple three-dimensional points located below a predetermined position and the vertical direction is at or above a predetermined angle. If it determines that the angle is at or above a predetermined angle, it removes at least one of the at least three of the multiple three-dimensional points located below a predetermined position without generating any new three-dimensional points within the polygon whose vertices are at least three of the multiple three-dimensional points located below a predetermined position. All three of these three three-dimensional points may be removed, or the three-dimensional points to be removed may be determined and removed based on other conditions, such as three-dimensional points that are not vertices of other polygons in the multiple polygons.

[0177] This is because three-dimensional points that constitute a plane whose normal direction is at a predetermined angle or greater with respect to the vertical direction estimated by the assignment unit 120 are highly likely to be noise that spans between different objects. As a result, the densification unit 130 can appropriately remove three-dimensional points added in inappropriate locations, even when the densification is done inaccurately, that is, when three-dimensional points are added in inappropriate locations.

[0178] When the densification unit 130 removes a three-dimensional point, it may delete the data of the three-dimensional point from the storage unit 160, or it may choose not to delete the data of the three-dimensional point from the storage unit 160, for example, by preventing the three-dimensional point from being displayed on the display device 520.

[0179] Furthermore, the predetermined angle may be arbitrarily set in advance and is not particularly limited.

[0180] Furthermore, for example, the assignment unit 120 can estimate the type of object to be a wall for three-dimensional points that constitute a plane in which the normal direction is orthogonal to the vertical direction.

[0181] Furthermore, as mentioned above, when the camera 301 takes a picture, the ceiling of a building or the like is generally located above the image. Therefore, similar to the method for estimating the object type as "ground," if the position of the camera 301 that generated the image used to generate the high-density three-dimensional point cloud and the position of the actual object corresponding to the three-dimensional point cloud are known, the assignment unit 120 can identify the position of the ceiling located above the camera 301 based on the trajectory information showing the trajectory of the camera 301, and set the object type of the three-dimensional point located at the identified position to "ceiling."

[0182] <High-density processing> Figure 13 is a diagram illustrating the densification process performed by the three-dimensional point cloud densification device 100 according to the embodiment. Specifically, Figure 13(a) shows the three-dimensional point cloud and auxiliary information acquired by the acquisition unit 110. Figure 13(b) shows the state in which the densification unit 130 has connected mesh lines to the three-dimensional point cloud shown in Figure 13(a). Figure 13(c) shows the dense three-dimensional point cloud generated by the densification unit 130 performing densification processing on the three-dimensional point cloud shown in Figure 13(a), that is, by generating new three-dimensional points from the three-dimensional point cloud.

[0183] Figure 13(a) shows an example in which the assignment unit 120 assigns type information indicating the type of object to each three-dimensional point as auxiliary information.

[0184] For example, the acquisition unit 110 acquires the three-dimensional point cloud stored in the three-dimensional point cloud storage unit 163. Next, the assignment unit 120 performs image recognition on an image stored in, for example, the image storage unit 161, to identify the type of object that each pixel in the image represents. Next, type information indicating the type of object that each pixel represents is assigned as auxiliary information to the three-dimensional point corresponding to each pixel. As a result, as shown in Figure 13(a), each three-dimensional point is assigned type information such as "tree," "house," or "road."

[0185] Furthermore, three-dimensional points may be assigned category information indicating multiple categories, such as "road" and "house."

[0186] Next, the densification unit 130 identifies polygons whose vertices are three-dimensional points that have been assigned type information indicating the same type, as shown in Figure 13(b). For example, the densification unit 130 identifies polygons by meshing them so that at least one three-dimensional point that has the same type as indicated in the type information is connected by mesh lines.

[0187] Next, the densification unit 130 generates a dense three-dimensional point cloud by generating new three-dimensional points within the polygon, as shown in Figure 13(c).

[0188] In this way, the densification unit 130 can suppress the generation of mesh lines connecting three-dimensional points that make up different objects such as roads, trees, or houses, by, for example, connecting only three-dimensional points of the same type with mesh lines.

[0189] The densification unit 130 performs meshing by projecting three-dimensional points onto the image, for example. Examples of meshing methods include Delaunay triangulation.

[0190] Furthermore, if there are results from instance segmentation by the assignment unit 120, the high-density unit 130 may perform meshing under constraints such as only being able to connect three-dimensional points that have been assigned the same type information with mesh lines within the same region (i.e., the region enclosed by three-dimensional points) as three-dimensional points that have been assigned the same type information.

[0191] According to this, when three-dimensional points constituting different objects, but of the same type as indicated by the type information, are adjacent to each other, the generation of mesh lines connecting those three-dimensional points is suppressed.

[0192] Alternatively, the densification unit 130 may project (backproject) the dense three-dimensional point cloud to generate a dense depth image, generate a three-dimensional point cloud based on the generated dense depth image, and perform densification on the generated three-dimensional point cloud.

[0193] Furthermore, the densification unit 130 may not determine the positions of the mesh lines by projecting three-dimensional points onto the image, but rather may mesh by connecting nearby three-dimensional points in three-dimensional space.

[0194] Furthermore, the densification unit 130 may calculate the normals of the mesh lines (more specifically, polygons) and filter the mesh lines using the calculated normals. For example, when a polygon whose vertices are three-dimensional points that constitute the ground or a road is identified, the normal direction of the polygon is likely to be the same as the vertical direction (gravity direction). In other words, if the normal direction of a polygon whose vertices are at least three three-dimensional points that constitute the ground or a road is not parallel to the vertical direction, there is a high possibility that incorrect type information has been assigned to at least one of those at least three three-dimensional points. Therefore, the densification unit 130 may determine whether the angle between the normal and the vertical direction is less than or equal to a predetermined angle, and based on the determination result, decide whether or not to delete the mesh line.

[0195] The predetermined angle may be determined based on the distance from the center of the camera 301's lens to the polygon having the normal vector. For example, the predetermined angle may be set to be larger the smaller the distance.

[0196] <Summary> Figure 14 is a flowchart showing the processing procedure of the three-dimensional point cloud densification apparatus 100 according to the embodiment.

[0197] First, the acquisition unit 110 acquires a plurality of three-dimensional points, each indicating the three-dimensional position of at least one object, and auxiliary information for creating a polygon (S301). The acquisition unit 110 may acquire point cloud data containing the plurality of three-dimensional points from the three-dimensional point cloud generator 200 or from the three-dimensional point cloud storage unit 163. In addition, the acquisition unit 110 may acquire auxiliary information from the user via the UI unit 140 or acquire auxiliary information generated by the assignment unit 120 from the assignment unit 120.

[0198] Next, the densification unit 130 identifies a polygon whose vertices are at least three of the multiple three-dimensional points, based on the auxiliary information (S302). For example, the densification unit 130 determines the group to which each of the three-dimensional points belongs, based on the auxiliary information. For example, if the auxiliary information is type information, the densification unit 130 determines (classifies) three-dimensional points that have type information containing at least one of the same type into the same group. Next, the densification unit 130 identifies a polygon whose vertices are three-dimensional points belonging to the same group.

[0199] Next, the densification unit 130 generates new three-dimensional points within the identified polygon (S303).

[0200] [Effects, etc.] As described above, the three-dimensional point cloud densification apparatus 100 according to the embodiment includes an acquisition unit 110 that acquires a plurality of three-dimensional points, each indicating the three-dimensional position of at least one object, and auxiliary information for creating a polygon, and a densification unit 130 that identifies a polygon whose vertices are at least three of the plurality of three-dimensional points based on the auxiliary information, and generates new three-dimensional points within the polygon.

[0201] According to this, the densification unit 130 can identify a polygon whose vertices are at least three appropriate three-dimensional points based on the auxiliary information. Therefore, the densification unit 130 can generate new three-dimensional points at appropriate locations by adding three-dimensional points to the identified polygon.

[0202] Furthermore, for example, the acquisition unit 110 acquires auxiliary information via an operating device (e.g., the UI unit 140) for receiving instructions from the user. For example, such instructions may specify the shape of a polygon.

[0203] According to this, the densification unit 130 obtains the polygon shape specification from the user and identifies the polygon, so it can generate a new three-dimensional point at the location desired by the user.

[0204] Furthermore, for example, the three-dimensional point cloud densification device 100 further includes an information generation unit 120 that generates auxiliary information based on image data obtained by photographing at least one object.

[0205] Furthermore, for example, the assignment unit 120 generates type information indicating the types of multiple objects included in the image data as auxiliary information, and assigns the type information corresponding to each of the multiple three-dimensional points to each of the multiple three-dimensional points. In this case, for example, the densification unit 130 identifies a polygon whose vertices are at least three three-dimensional points of the same object type.

[0206] According to this, the densification unit 130 can generate new three-dimensional points to complement three-dimensional points of the same type of constituent object.

[0207] Furthermore, for example, the assignment unit 120 generates the edges of objects included in the image data as auxiliary information. In this case, for example, the densification unit 130 identifies polygons that do not intersect with the edges.

[0208] According to this, the densification unit 130 can suppress the generation of new three-dimensional points between different objects at locations where no objects exist.

[0209] Furthermore, for example, the acquisition unit 110 acquires camera information as auxiliary information, which indicates the position of the camera 301 at the time of shooting and the orientation of the camera 301, which generated multiple image data obtained by photographing at least one object. In this case, for example, the densification unit 130 identifies polygons based on the camera information.

[0210] Furthermore, for example, the densification unit 130 identifies a polygon whose vertices are at least three of the multiple three-dimensional points located below a predetermined position, based on the vertical direction in a virtual three-dimensional space representing multiple three-dimensional points determined based on camera information.

[0211] According to this, the densification unit 130 can, for example, determine the type of object by identifying multiple three-dimensional points located below it among the three-dimensional points included in the three-dimensional point cloud as ground such as a road, and then identify polygons whose vertices are three-dimensional points of the same object type.

[0212] Furthermore, for example, the high-density unit 130 determines the height of the camera 301 in the vertical direction in a virtual three-dimensional space based on the camera information, and determines a predetermined position based on the determined height.

[0213] According to this, the densification unit 130 can more appropriately identify a three-dimensional point where the object type is the ground, based on the height of the camera 301.

[0214] Furthermore, for example, the densification unit 130 identifies polygons based on the angle between the normal direction of a polygon whose vertices are at least three of a plurality of three-dimensional points located below a predetermined position, and the vertical direction.

[0215] For example, the normal direction of a polygon whose vertices are three-dimensional points that constitute the ground or a road is likely to be the same direction as the vertical direction. In other words, if the angle between the normal direction of a polygon whose vertices are at least three three-dimensional points that constitute the ground or a road is not parallel and is too large, then it is highly likely that at least one of those at least three three-dimensional points is noise rather than a three-dimensional point that constitutes the ground or a road. Therefore, with such a configuration, for example, the densification unit can prevent the generation of new three-dimensional points within a polygon whose vertices are at three-dimensional points that are highly unlikely to be three-dimensional points that constitute the ground or a road.

[0216] Furthermore, for example, the auxiliary information is information indicating the group to which each of the multiple three-dimensional points belongs. In this case, for example, the densification unit 130 identifies a polygon whose vertices are at least three of the three-dimensional points belonging to the same group, based on the auxiliary information.

[0217] According to this, the densification unit 130 can identify polygons based on auxiliary information that indicates three-dimensional points that are likely to be composed of the same object or are likely to be the same object as three-dimensional points belonging to the same group, such as three-dimensional points of the same object type, three-dimensional points that can be connected by lines without crossing the edges of objects included in the image data, or three-dimensional points located below a predetermined position in the virtual three-dimensional space that represents the three-dimensional points.

[0218] Furthermore, a method for densifying a three-dimensional point cloud according to one aspect of the present disclosure includes an acquisition step (S301) of acquiring a plurality of three-dimensional points, each indicating the three-dimensional position of at least one object, and auxiliary information for creating a polygon; an identification step (S302) of identifying a polygon whose vertices are at least three of the plurality of three-dimensional points based on the auxiliary information; and a densification step (S303) of generating new three-dimensional points within the polygon.

[0219] According to this, it produces the same effect as a three-dimensional point cloud densification device.

[0220] Furthermore, this disclosure may be implemented as a program that causes a computer to execute the steps included in the above-described three-dimensional point cloud densification method. Alternatively, this disclosure may be implemented as a non-temporary recording medium such as a CD-ROM readable by the computer on which the program is recorded. Furthermore, this disclosure may be implemented as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network such as the Internet.

[0221] (Other embodiments) The three-dimensional point cloud densification apparatus, etc. related to this disclosure has been described above based on the above embodiments, but this disclosure is not limited to the above embodiments.

[0222] For example, the three-dimensional point cloud generator 200 may use the dense three-dimensional point cloud generated by the three-dimensional point cloud densification device 100 to generate a three-dimensional point cloud again.

[0223] Furthermore, for example, if the three-dimensional point cloud densification device 100 automatically performs densification processing and outputs a three-dimensional point cloud when it is input without receiving instructions from the user, it does not need to have a UI unit 140.

[0224] Furthermore, in the above embodiment, for example, the components of the processing unit, such as the control unit 150, of the three-dimensional point cloud densification device 100 may each consist of one or more electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit. The one or more electronic circuits may include, for example, a semiconductor device, an IC (Integrated Circuit), or an LSI (Large Scale Integration). The IC or LSI may be integrated on a single chip or on multiple chips. Here, we refer to them as IC or LSI, but the name may change depending on the degree of integration, and they may be called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). Also, FPGAs (Field Programmable Gate Arrays) that are programmed after the LSI is manufactured can be used for the same purpose.

[0225] Furthermore, the general or specific embodiments of this disclosure may be implemented as a system, apparatus, method, integrated circuit, or computer program. Alternatively, they may be implemented as a computer-readable non-temporary recording medium such as an optical disk, HDD (Hard Disk Drive), or semiconductor memory on which the computer program is stored. They may also be implemented as any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0226] Furthermore, this disclosure also includes forms that can be obtained by applying various modifications to each embodiment that a person skilled in the art can conceive, as well as forms that can be realized by arbitrarily combining the components and functions of the embodiments without departing from the spirit of this disclosure. [Industrial applicability]

[0227] This disclosure can be applied, for example, to a system that can appropriately increase the number of points in three-dimensional point cloud data of an object generated from an image or the like. [Explanation of Symbols]

[0228] 100 3D point cloud densification device 110 Acquisition Department 120 Addition unit (generation unit) 130 High density section 140 UI section 150 Control Unit 160 Storage section 161 Video Memory Unit 162 Camera Information Storage Unit 163 3D point cloud storage 164 Dense 3D point cloud storage 200 3D point cloud generator 300 Photography Department 301 Camera 400 Three-Dimensional Point Cloud Generation Systems 500 Computers 510 Operating device 520 Display device 600-point image cloud 610 Superimposed Images 620 Adjustment section 630 Selection Section 640 Cursors 650 Operation trajectory Objects 700, 701, 702, 710, 711, and 712 720, 721, 722, 723, 724 Polygons

Claims

1. At least one memory, The circuit comprises at least one memory and connected to it, The aforementioned circuit is Each acquires a plurality of three-dimensional points indicating the three-dimensional position of at least one object in a virtual three-dimensional space, and camera information indicating the position and orientation of the camera when it photographed the at least one object in real three-dimensional space in order to generate the plurality of three-dimensional points. Based on the camera information, the vertical direction in the virtual three-dimensional space is determined. Identify a polygon having at least three of the aforementioned three-dimensional points as vertices, (i) The angle between the vertical direction and the normal direction of the polygon is smaller than a first threshold, and (ii) The polygon is located below a second threshold in the virtual three-dimensional space. Only then, generate at least one new three-dimensional point within the polygon. Three-dimensional point cloud densification device.

2. Each acquires a plurality of three-dimensional points indicating the three-dimensional position of at least one object in a virtual three-dimensional space, and camera information indicating the position and orientation of the camera when it photographed the at least one object in real three-dimensional space in order to generate the plurality of three-dimensional points. Based on the camera information, the vertical direction in the virtual three-dimensional space is determined. Identify a polygon having at least three of the aforementioned three-dimensional points as vertices, (i) The angle between the vertical direction and the normal direction of the polygon is smaller than a first threshold, and (ii) The polygon is located below a second threshold in the virtual three-dimensional space. Only then, generate at least one new three-dimensional point within the polygon. Three-dimensional point cloud densification method.

3. To cause a computer to execute the three-dimensional point cloud densification method described in claim 2. program.

Images