Image synthesis device, image synthesis method, and program

Abstract

To enable an image synthesizer to select an appropriate image for each mesh. [Solution] The video synthesis device 16 comprises a data synthesis unit 26 that extracts portions of video data VD corresponding to each mesh to generate textures, and an overlap removal unit 34 that selects the texture with the highest appropriateness from multiple textures that satisfy specified proximity conditions in three-dimensional space and outputs textured data GTD, where the appropriateness is an index that increases as the number of pixels in the projected mesh or the effective resolution within the area of ​​the projected mesh increases when the mesh corresponding to the texture is re-projected onto the video data VD from which the texture was extracted.

Description

[Technical Field]

[0001] The disclosed technology relates to an image synthesizing device, an image synthesizing method, and a program. [Background technology]

[0002] As background technology in this technical field, the abstract of Patent Document 1 below states that "a three-dimensional model generation method includes generating point cloud data of a three-dimensional model indicating the three-dimensional shape of a subject based on a plurality of images arranged in order of camera position closest to the subject, generating mesh data from the point cloud data, generating a plurality of texture candidate images to be pasted onto a plurality of polygons constituting the mesh data based on an image having a pixel area corresponding to the mesh data that is closest to the plurality of polygons constituting the mesh data, and pasting, as texture, images of areas on the plurality of texture candidate images corresponding to the positions of each of the plurality of polygons onto each of the plurality of polygons." Furthermore, the following Patent Documents 2 to 4 also describe techniques for synthesizing images captured by multiple cameras, etc. The contents of these documents are incorporated herein by reference. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2024 / 090293 [Patent Document 2] International Publication No. 2018 / 030206 [Patent Document 3] Japanese Patent Application Publication No. 2018-113683 [Patent Document 4] International Publication No. 2015 / 194501 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, in Patent Document 1, a texture candidate image is generated based on "an image in which the distance between the image and the multiple polygons constituting the mesh data is the shortest." However, this "image in which the distance is the shortest" is not necessarily appropriate. For example, if the optical axis of the camera that captured the "image in which the distance is the shortest" intersects the normal line of the mesh at a large angle, it is possible that a sufficient number of pixels will not be assigned to the mesh. The disclosed technology has been made in consideration of the above-mentioned circumstances, and aims to provide an image synthesis device, an image synthesis method, and a program that can select an appropriate image for each mesh. [Means for solving the problem]

[0005] In order to solve the above problem, an image synthesis device, which is one aspect of the disclosed technology, comprises a data input unit that receives multiple lines of image data obtained by photographing an object with multiple camera units; a mesh generation unit that acquires meshed data that represents the shape of the object as a collection of meshes, which are polygons arranged in three-dimensional space; a data synthesis unit that cuts out portions of the image data that correspond to each of the meshes to generate textures; and an overlap removal unit that selects the texture with the highest appropriateness from multiple textures that satisfy specified proximity conditions in three-dimensional space and outputs the textured data, wherein the appropriateness is an index that increases as the number of pixels in the projected mesh or the effective resolution within the area of ​​the projected mesh increases when the mesh corresponding to the texture is reprojected onto the image data from which the texture was cut out. [Effects of the Invention]

[0006] The disclosed technology allows for the selection of an appropriate image for each mesh. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram of a video synthesis system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram of a system-specific data generating unit. [Figure 3] FIG. 1 is a block diagram of a computer. [Figure 4] 4 is a flowchart of an image synthesis routine in the first embodiment. [Figure 5] 10 is a flowchart of an image synthesis routine in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] [First embodiment] <Explanation of terms> First, the meanings of terms used in the following description will be explained. Coordinate point: A point that has coordinates (x-axis, y-axis, and z-axis values) located on the surface of an object in three-dimensional space. Point cloud data: Data that represents the shape of an object using a set of coordinate points. Mesh: A polygon (e.g., a triangle) formed by connecting multiple coordinate points contained in point cloud data. Meshed data: Data that represents the shape of an object in three-dimensional space using multiple meshes. ·Video data: Two-dimensional RGB format video or still image data of a subject. Texture: A part of the video data is cut out to fit the shape of a mesh projected onto the video data, and the cut-out part is then enlarged, reduced, or otherwise transformed to fit the three-dimensional shape of the mesh. Textured data: Data that uses multiple textures to represent the shape and color of an object in three-dimensional space.

[0009] <Configuration of the first embodiment> FIG. 1 is a block diagram of an image synthesis system 1 according to a first embodiment of the disclosed technique. 1, image compositing system 1 includes n systems (n is plural) of camera units 12-1 to 12-n, n systems of three-dimensional measuring devices 14-1 to 14-n, and image compositing device 16 (computer). Camera units 12-1 to 12-n each output n systems of image data VD-1 to VD-n obtained by capturing an image of object 10. Object 10 is, for example, a building, and camera units 12-1 to 12-n are arranged at various locations around object 10. In this embodiment, image data VD-1 to VD-n are moving images, but may also be still images.

[0010] In the following description, multiple components, physical quantities, information, etc. that have the same or similar functions or meanings may be expressed by adding "-" and alphanumeric characters to the same reference numeral, such as "camera units 12-1, 12-n." ​​However, when it is not necessary to distinguish between these multiple components, they may be expressed without "-" and alphanumeric characters, such as "camera unit 12."

[0011] The camera unit 12 is a so-called PTZ camera, and various parameters such as pan (horizontal rotation), tilt (vertical rotation), zoom (enlargement / reduction), and focus can be preset. Furthermore, by continuously changing these parameters, it is possible to capture images of a wide area or with high resolution. Furthermore, while changing these parameters in a small range, super-resolution processing can be performed in cooperation with the noise removal unit 22 described below to synthesize high-definition images.

[0012] Furthermore, the three-dimensional measuring devices 14-1 to 14-n are, for example, LiDAR (Light Detection and Ranging) devices, and are respectively arranged near the camera units 12-1 to 12-n. The three-dimensional measuring devices 14-1 to 14-n each measure the three-dimensional shape of the object 10 and output n systems of point cloud data PD-1 to PD-n representing the results. The image synthesizing device 16 generates output textured data GTD (textured data), which is textured data, based on the image data VD-1 to VD-n and the point cloud data PD-1 to PD-n.

[0013] Image composition device 16 includes n system-specific data generation units 20-1 to 20-n, an image integration unit 30, and a positional relationship calculation unit 40. Image integration unit 30 includes a texture integration unit 32, an overlap removal unit 34 (overlap removal process, overlap removal means), and a data recording unit 36. Positional relationship calculation unit 40 calculates the positional relationship between object 10, each camera unit 12, and each three-dimensional measurement device 14.

[0014] FIG. 2 is a block diagram of the system-specific data generating unit 20-k (where 1≦k≦n). The system-specific data generating unit 20-k generates system-specific textured data TD-k based on the video data VD-k and the point cloud data PD-k. For this reason, the system-specific data generation unit 20-k includes a data input unit 21-k (data input process, data input means), a noise removal unit 22-k, a mesh generation unit 24-k (mesh generation process, mesh generation means), a data synthesis unit 26-k (data synthesis process, data synthesis means), and a data output unit 28-k.

[0015] The data input unit 21-k receives the video data VD-k from the camera unit 12-k and the point cloud data PD-k from the three-dimensional measurement device 14-k. The noise removal unit 22-k performs noise removal processing on the point cloud data PD-k and performs distortion correction on the video data VD-k. Here, in addition to distortion correction, the video data VD-k may be subjected to the above-mentioned super-resolution processing, noise removal, edge removal, etc. to increase the definition of the video data VD-k.

[0016] Here, the noise removal process for the point cloud data PD-k will be described. The noise removal unit 22-k converts the point cloud data PD-k into a depth image. In the acquired depth image, the shading changes significantly in the portions corresponding to the edges of the object 10. When such edges exist, noise that causes slight changes in shading occurs around the edges. Therefore, the noise removal unit 22 removes coordinate points corresponding to noise around such edges from the point cloud data PD-k.

[0017] The mesh generation unit 24-k generates meshes, which are polygons with vertices at the coordinate points, by connecting the coordinate points of the point cloud data PD-k, and obtains meshed data MD-k, which is a collection of meshes. For example, the Poisson method, the Alpha Shape method, or the Marching Cube method can be used to generate this mesh. Furthermore, the mesh generation unit 24-k removes from the meshed data MD-k meshes formed in areas of the point cloud data PD-k where the density of coordinate points is almost zero, i.e., areas where the density of coordinate points is lower than a predetermined density. This is because coordinate points present in areas with almost no coordinate points are considered to be noise.

[0018] Furthermore, the mesh generation unit 24-k performs a weight reduction process on the meshed data MD-k if the user desires. When weight reduction is desired, the user registers the shape of a portion of interest (a portion of particular interest) of the shape of the object 10 in advance in the image synthesis device 16. The mesh generation unit 24-k leaves the mesh of this portion of interest as is, and reduces the number of meshes for the meshes other than the portion of interest by, for example, merging multiple meshes. This reduces the total number of meshes in the meshed data MD-k, thereby reducing the processing load on the image synthesis device 16.

[0019] The data synthesis unit 26-k first generates textures by extracting portions of the video data VD-k that correspond to each mesh. The data synthesis unit 26-k then aggregates these textures to generate textured data TD-k. The data output unit 28-k outputs the textured data TD-k.

[0020] Returning to Fig. 1, the system-specific data generation units 20-1 to 20-n supply the textured data TD-1 to TD-n to the image integration unit 30. In the image integration unit 30, the texture integration unit 32 integrates the textured data TD-1 to TD-n within the same virtual three-dimensional space to generate integrated textured data UTD.

[0021] If there are multiple textures included in the unified textured data UTD that satisfy a predetermined proximity condition, the overlap remover 34 selects only the texture with the highest "suitability" and deletes the other textures. Note that the "proximity condition" refers to the condition that the positions and shapes of the textures are similar enough to be considered "overlapping."

[0022] Here, we will explain the "appropriateness" of a texture. When a mesh corresponding to a certain texture is reprojected onto the video data VD from which the texture was extracted, the number of pixels in the projected mesh or the effective resolution within the area of ​​the projected mesh is defined as X. The "appropriateness" is an index that increases as X increases. For example, when the area of ​​the texture in three-dimensional space is defined as Y, the "appropriateness" can be defined as an index of "X / Y."

[0023] The data recording unit 36 ​​records the unified textured data UTD, from which the texture overlap has been removed by the overlap removal unit 34, as output textured data GTD.

[0024] 3 is a block diagram of the computer 980. The image synthesizer 16 shown in FIG. 1 includes one or more computers 980 shown in FIG. 3, a computer 980 includes a CPU 981, a storage unit 982, a communication port 983, an input / output port 984, and a media port 985. The storage unit 982 includes a RAM 982a, a ROM 982b, and an SSD (Solid State Drive) 982c. The communication port 983 is connected to a communication circuit 986. The input / output port 984 is connected to an input / output device 987. The media port 985 reads and writes data from a recording medium 988.

[0025] The ROM 982b stores an IPL (Initial Program Loader) executed by the CPU, etc. The SSD 982c stores application programs, various data, etc. The CPU 981 executes application programs, etc. loaded from the SSD 982c to the RAM 982a, thereby realizing various functions. The interior of the video synthesizer 16 shown in FIG. 1 is primarily shown as a block diagram of functions realized by application programs, etc.

[0026] <Operation of the First Embodiment> Next, the operation of the first embodiment will be described. Fig. 4 is a flowchart of an image synthesis routine in the first embodiment. This routine is executed in the image synthesis device 16. In Fig. 4, the processes of steps S2 to S12 are executed in parallel in the n-system-specific data generation units 20-1 to 20-n. Furthermore, in each system-specific data generation unit 20-k, the processes of step S2 and steps S4 and S6 are executed in parallel.

[0027] First, when the process proceeds to step S2, the noise removal unit 22-k (where 1≦k≦n) performs distortion correction on the video data VD-k. In parallel with step S2, the processes of steps S4 and S6 are executed. In step S4, the noise removal unit 22-k performs noise removal processing on the point cloud data PD-k. Next, when the process proceeds to step S6, the mesh generation unit 24 generates meshed data MD-k based on the point cloud data PD-k after the noise removal processing.

[0028] Next, when the process proceeds to step S8, the data synthesis unit 26-k generates textured data TD-k for each system. That is, the data synthesis unit 26 generates textures corresponding to each mesh by cutting out the video data VD-k according to the shape of each mesh included in the meshed data MD-k. Then, the generated textures are synthesized to generate the textured data TD-k.

[0029] Next, when the process proceeds to step S10, the data synthesis unit 26-k calculates the distance from the camera unit 12-k to each texture included in the textured data TD-k. Note that the calculated distance is also equal to the distance from the camera unit 12-k to each mesh included in the meshed data MD-k. Next, when the process proceeds to step S12, the data synthesis unit 26-k calculates the above-mentioned "suitability" for each texture included in the textured data TD-k.

[0030] When the processes of steps S2 to S12 described above are completed in all of the system-specific data generation units 20-1 to 20-n, the process proceeds to step S20. Here, the texture integration unit 32 integrates the textured data TD-1 to TD-n in the same virtual three-dimensional space to generate integrated textured data UTD.

[0031] The following steps S22 and S24 are repeated by selecting each texture until processing is completed for all textures belonging to the unified textured data UTD. First, in step S22, the overlap remover 34 determines whether or not there is another texture that satisfies the above-mentioned neighborhood condition for the selected texture. If the determination here is "No," processing for the currently selected texture ends.

[0032] If the determination in step S22 is "Yes," the process proceeds to step S24, where the overlap remover 34 deletes from the unified textured data UTD all textures other than the currently selected texture and the textures that satisfy the neighborhood conditions, except for the texture with the highest appropriateness.

[0033] By repeating the above steps S22 and S24 for all textures, the unified textured data UTD becomes data that does not contain overlapping textures. Next, when the process proceeds to step S26, the overlap remover 34 records the unified textured data UTD, from which the overlapping textures have been removed, as output textured data GTD in the data recorder 36. This completes the processing of this routine.

[0034] [Second embodiment] <Configuration of the second embodiment> Next, a video synthesis system according to a second embodiment of the disclosed technology will be described. Although the configuration of the second embodiment is not shown, it is the same as that of the first embodiment (see FIG. 1) except that it does not include n-system three-dimensional measuring devices 14-1 to 14-n. In the second embodiment, mesh generation unit 24-k (where 1≦k≦n) of each system estimates the three-dimensional shape of object 10 based on n-system video data VD-1 to VD-n, and generates meshed data MD-k based on the estimation results.

[0035] <Operation of the second embodiment> Next, the operation of the second embodiment will be described. Fig. 5 is a flowchart of an image synthesis routine in the second embodiment. This routine is also executed in the image synthesis device 16. In Fig. 5, the processes of steps S2 to S12 are executed in parallel in n system-specific data generation units 20-1 to 20-n.

[0036] First, when the process proceeds to step S2, the noise removal unit 22-k (where 1≦k≦n) performs distortion correction on the video data VD-k. Next, when the process proceeds to step S5, the mesh generation unit 24-k estimates the three-dimensional shape of the object 10 based on the video data VD, and generates meshed data MD based on the estimation result.

[0037] Next, when the process proceeds to step S8, the data synthesis unit 26-k generates textured data TD-k for each system. The process from step S10 onwards is the same as that in the first embodiment (see FIG. 4), and therefore a description thereof will be omitted.

[0038] As described above, when high accuracy is not required for the three-dimensional shape of object 10, it is possible to omit three-dimensional measuring device 14 and estimate the three-dimensional shape of object 10 based on video data VD-1 to VD-n. Furthermore, by omitting three-dimensional measuring device 14, it is possible to reduce the cost of the video composition system.

[0039] [Variations] The disclosed technology is not limited to the above-described embodiments and various modifications are possible. The above-described embodiments are provided as examples to facilitate understanding of the disclosed technology and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete part of the configuration of each embodiment, add other configurations, or replace other configurations. Furthermore, the control lines and information lines shown in the figures are those considered necessary for explanation, and do not necessarily represent all control lines and information lines necessary in the product. In reality, it is possible to consider that almost all configurations are interconnected. Possible modifications of the above-described embodiments include, for example, the following:

[0040] (1) Since the hardware of the image synthesizing device 16 in the above embodiment can be realized by a general computer, the processes corresponding to the above-mentioned block diagrams and flowcharts, as well as programs for executing the various processes described above, may be stored on a storage medium (a computer-readable storage medium on which a program is recorded) or distributed via a transmission path.

[0041] (2) In the above embodiment, the processes corresponding to the block diagrams and flowcharts, as well as the various other processes described above, are described as software processes using programs. However, some or all of these processes may be replaced with hardware processes using an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), etc.

[0042] (3) The various processes executed in the above embodiment may be executed by a server computer via a network (not shown), and the various data stored in the above embodiment may also be stored in the server computer.

[0043] [Effects of the embodiment] As described above, according to the embodiment, the video composition device 16 includes an overlap remover 34 that selects the texture with the highest appropriateness from among multiple textures that satisfy predetermined neighborhood conditions in three-dimensional space and outputs the textured data (GTD). The appropriateness is an index that increases as the number of pixels in the projected mesh or the effective resolution within the projected mesh area increases when the mesh corresponding to the texture is reprojected onto the video data VD from which the texture was extracted. This allows the textured data (GTD) to be generated using the texture with the highest appropriateness, and an appropriate image to be selected for each mesh.

[0044] Furthermore, as in the first embodiment, it is more preferable that the data input unit 21 further acquires point cloud data PD, which is a set of coordinate points, from multiple three-dimensional measurement devices 14, and that the mesh generation unit 24 acquires meshed data MD based on the point cloud data PD. Furthermore, it is more preferable that the appropriateness is an index that is X / Y when a mesh corresponding to a texture is reprojected onto the video data VD from which the texture was extracted, where X is the number of pixels in the projected mesh or the effective resolution within the region of the projected mesh, and Y is the area of ​​the texture in three-dimensional space. This allows the three-dimensional measurement device 14 to acquire accurate point cloud data PD, and more appropriate textured data (GTD) can be generated.

[0045] Furthermore, it is more preferable that the image composition device 16 further includes a noise removal unit 22 that performs noise removal processing on the point cloud data PD. This makes it possible to generate more appropriate textured data (GTD) using the point cloud data PD from which noise has been removed.

[0046] Furthermore, it is more preferable that the mesh generation unit 24 further has a function of removing meshes formed in areas where the density of coordinate points is lower than a predetermined density from the meshed data MD. This makes it possible to remove meshes that are considered unnecessary from the meshed data MD, and to generate more appropriate textured data (GTD).

[0047] Furthermore, it is more preferable that the noise removal unit 22 converts the point cloud data PD into a depth image and performs noise removal processing on the point cloud data PD based on the depth image. This makes it possible to properly remove coordinate points that are considered to be noise, and to generate more appropriate textured data (GTD).

[0048] Furthermore, it is more preferable that the mesh generating unit 24 performs mesh lightening processing to reduce the mesh density in a partial area of ​​the meshed data MD. This reduces the total number of meshes and the processing load.

[0049] Furthermore, it is more preferable that the mesh lightening process is a process of reducing the mesh density of the meshed data MD in the areas other than the areas corresponding to the pre-registered shape of interest, thereby reducing the total number of meshes while maintaining a high-density mesh in the areas corresponding to the shape of interest. [Explanation of symbols]

[0050] 10 Object 12 Camera Section 14 Three-dimensional measuring device 16 Image synthesis device (computer) 21 Data input section (data input process, data input means) 22 Noise removal section 24 Mesh generation unit (mesh generation process, mesh generation means) 26 Data synthesis unit (data synthesis process, data synthesis means) 34 overlap removal unit (overlap removal process, overlap removal means) MD meshed data PD point cloud data VD video data GTD Output Textured Data (Textured Data)

Claims

1. a data input unit that receives a plurality of lines of video data obtained by photographing an object with a plurality of camera units; a mesh generation unit that acquires meshed data that represents the shape of the object as a collection of meshes that are polygons arranged in a three-dimensional space; a data synthesis unit that extracts portions of the video data corresponding to the meshes and generates textures; an overlap removal unit that selects the texture with the highest appropriateness from among the plurality of textures that satisfy a predetermined neighborhood condition in a three-dimensional space and outputs textured data; The appropriateness is an index that increases as the number of pixels in the projected mesh or the effective resolution in the area of ​​the projected mesh increases when the mesh corresponding to the texture is reprojected onto the video data from which the texture was extracted. A video synthesis device characterized by:

2. the data input unit further acquires point cloud data, which is a set of coordinate points, from a plurality of three-dimensional measurement devices; the mesh generation unit acquires the meshed data based on the point cloud data, The appropriateness is an index that is X / Y when the number of pixels in the projected mesh or the effective resolution in the area of ​​the projected mesh when the mesh corresponding to the texture is reprojected onto the video data from which the texture was extracted is X and the area of ​​the texture in the three-dimensional space is Y.

2. The video synthesizing device according to claim 1.

3. The method further includes a noise removal unit that performs noise removal processing on the point cloud data.

3. The video synthesizing device according to claim 2.

4. The mesh generating unit further has a function of removing the mesh formed in an area where the density of the coordinate points is lower than a predetermined density from the meshed data.

4. The video synthesizing device according to claim 3.

5. The noise removal unit converts the point cloud data into a depth image and performs the noise removal process on the point cloud data based on the depth image.

4. The video synthesizing device according to claim 3.

6. The mesh generation unit performs a mesh lightening process for reducing the density of the mesh in a partial area of ​​the meshed data.

3. The video synthesizing device according to claim 2.

7. The mesh lightening process is a process of reducing the mesh density for a portion of the meshed data other than a portion corresponding to a pre-registered shape of interest.

7. The video synthesizing device according to claim 6.

8. The computer a data input step of receiving a plurality of lines of video data obtained by photographing an object with a plurality of camera units; a mesh generation process for acquiring meshed data that represents the shape of the object as a set of meshes, which are polygons arranged in a three-dimensional space; a data synthesis step of extracting portions of the image data corresponding to each of the meshes and generating textures; an overlap removal step of selecting the texture with the highest appropriateness from among the plurality of textures that satisfy a predetermined neighborhood condition in the three-dimensional space and outputting the textured data; The appropriateness is an index that increases as the number of pixels in the projected mesh or the effective resolution in the area of ​​the projected mesh increases when the mesh corresponding to the texture is reprojected onto the video data from which the texture was extracted. A video synthesis method comprising:

9. Computer, a data input means for receiving a plurality of lines of video data obtained by photographing an object with a plurality of camera units; a mesh generating means for acquiring meshed data that represents the shape of the object as a set of meshes, which are polygons arranged in a three-dimensional space; data synthesis means for extracting portions of the video data corresponding to the meshes and generating textures; a program for causing the program to function as overlap removal means for selecting a texture having the highest appropriateness from among a plurality of textures that satisfy a predetermined neighborhood condition in a three-dimensional space, and outputting textured data, the program comprising: The appropriateness is an index that increases as the number of pixels in the projected mesh or the effective resolution in the area of ​​the projected mesh increases when the mesh corresponding to the texture is reprojected onto the video data from which the texture was extracted. A program characterized by:

Images