Image processing device, image processing method and program

Abstract

To process three-dimensional data independently, object by object, even when three-dimensional shapes of a plurality of objects are represented with one piece of three-dimensional data.SOLUTION: An image processing device 100 is configured to: acquire three-dimensional shape sequence data including time-series three-dimensional shape data showing three-dimensional shapes of a plurality of objects; separate data on a combined shape, obtained by combining three-dimensional shapes of two or more objects among the plurality of objects into one three-dimensional shape, into object shape data as three-dimensional shape data corresponding to two or more objects when the data on the combined shape is included in the time-series three-dimensional shape data; generate object shape data corresponding to the plurality of respective objects; and generate, object by object, object shape sequence data as three-dimensional shape sequence data including the time-series object shape data by the objects.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a registration technique for time-varying three-dimensional shape data indicating the shape of an object. [Background technology]

[0002] There is a technique for generating textured 3D shape data based on the 3D shape of an object estimated using data of a plurality of captured images obtained by capturing images using a plurality of imaging devices, and compressing the generated textured 3D shape data. For example, as a method for efficiently compressing 3D shape data, there is a method for tracking a time-series 3D shape corresponding to an object as a pre-processing step before compression. Tracking a 3D shape involves determining reference 3D shape data, and performing non-rigid registration so that the 3D shape corresponding to the 3D shape data matches a 3D shape adjacent in time (hereinafter referred to as an "adjacent 3D shape").

[0003] By aligning the time series of three-dimensional shapes in this way, the changes in the three-dimensional shapes of the object over time are aligned, and the topology of the three-dimensional shapes is made common between adjacent three-dimensional shapes. Furthermore, by making the topology of the three-dimensional shapes common, it becomes possible to express the movement of the object by the movement of each vertex that constitutes the three-dimensional shape. Furthermore, by expressing the three-dimensional shape data using information indicating the movement of the object, i.e., the movement of the vertices, and further compressing the information, the amount of data of the three-dimensional shape can be efficiently reduced.

[0004] However, in a scene where many objects exist at the same time, it is difficult to align all the generated 3D shapes at once in terms of the processing time required for calculations. One method to avoid this is to assign identification information to each 3D shape to identify the object, and align the 3D shapes for each object. However, if the 3D shapes corresponding to multiple objects close to each other are combined and estimated as a single 3D shape, it is not possible to align the 3D shapes corresponding to each object separately.

[0005] Patent Document 1 discloses a technique for aligning each polygon mesh by combining polygon meshes corresponding to a plurality of objects as a polygon mesh corresponding to one object group. According to the technique disclosed in Patent Document 1, even if there are a plurality of objects close to each other, the topology of the polygon meshes in time series is made common. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-136943 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the technology disclosed in Patent Document 1, when multiple adjacent objects are represented by one polygon mesh, the multiple objects are treated as one piece of data. Therefore, in the technology disclosed in Patent Document 1, when multiple objects are close to each other, it is difficult for the user to process the 3D shape data for each object independently. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the image processing device disclosed herein has a shape acquisition means for acquiring 3D shape sequence data including time-series 3D shape data indicating the 3D shapes of a plurality of objects, a separation means for, when data of a combined shape in which the 3D shapes of two or more of the plurality of objects are combined into a single 3D shape is included in the time-series 3D shape data, separating the combined shape data into object shape data which is 3D shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects, and a sequence generation means for generating, for each object, object shape sequence data which is 3D shape sequence data including the time-series object shape data for each object. Effect of the Invention

[0009] According to the present disclosure, even when the three-dimensional shapes of multiple objects are represented by a single piece of three-dimensional shape data, processing of the three-dimensional shape data can be easily performed independently for each object. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 3] 5 is a flowchart showing an example of a processing flow in the image processing device according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining an example of a change in data in processing by the image processing device according to the first embodiment. [Diagram 5] 5 is a flowchart showing an example of a flow of a mesh sequence data generation process in a sequence generation unit according to the first embodiment. [Figure 6]5 is a diagram for explaining a modification path setting process in a sequence generation unit according to the first embodiment. FIG. [Figure 7] 5 is a diagram for explaining a modification path setting process in a sequence generation unit according to the first embodiment. FIG. [Figure 8] FIG. 4 is a diagram showing an example of the data structure of textured mesh sequence data output by a texture adding unit and the data structure of compressed data output by a compression unit according to the first embodiment. [Figure 9] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. [Figure 10] 10 is a flowchart showing an example of a processing flow in an image processing device according to a second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of an editing UI according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. Note that the same components will be described with the same reference numerals.

[0012] [Embodiment 1] <Hardware configuration of image processing device> FIG. 1 is a block diagram showing an example of a hardware configuration of an image processing device 100 according to the first embodiment. The image processing device 100 is configured by an information processing device such as a personal computer, and includes a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, an operation unit 105, a communication I / F 106, and a bus 107. The CPU 101 realizes each function of the image processing device 100 by controlling the entire image processing device 100 using computer programs and data stored in the ROM 102 or the auxiliary storage device 104. Note that the image processing device 100 may include at least one of one or more dedicated hardware and a GPU (Graphics Processing Unit), and at least a part of the processing by the CPU 101 may be performed by the GPU or the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0013] The ROM 102 stores programs that do not require modification. The RAM 103 operates as a work area for the CPU 101, and temporarily stores programs and data supplied from the auxiliary storage device 104, and data supplied from the outside via the communication I / F 107. The auxiliary storage device 104 is configured with a hard disk drive or the like, and stores various data such as input image data. The operation unit 105 is configured with a keyboard, a mouse, or the like, and outputs various instructions to the CPU 101 in response to an operation by a user. The CPU 101 operates as an operation control unit that controls the operation unit 105. The communication I / F 106 is used for communication with an external device of the image processing device 100. For example, when the image processing device 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 106. When the image processing device 100 has a function of wireless communication with an external device, the communication I / F 106 is provided with an antenna. A bus 107 connects each unit that the image processing apparatus 100 has as a hardware configuration and transmits information.

[0014] <Functional configuration of image processing device> FIG. 2 is a block diagram showing an example of a functional configuration of the image processing device 100 according to the first embodiment. The image processing device 100 has, as its functional configuration, an image acquisition unit 201, a shape acquisition unit 202, an identification assignment unit 203, a sequence generation unit 204, a texture assignment unit 205, and a compression unit 206. Each unit of the image processing device 100 as its functional configuration is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, etc., developing it in the RAM 103, and executing it. The image acquisition unit 201 acquires data (hereinafter referred to as "multiple-viewpoint image data") of a plurality of captured images (hereinafter referred to as "multiple-viewpoint images") obtained by imaging using a plurality of imaging devices. Here, each of the plurality of captured image data constituting the multi-viewpoint image data is moving image data composed of a plurality of frame data, and the frame data is obtained by synchronous imaging using a plurality of imaging devices.

[0015] The shape acquisition unit 202 estimates a three-dimensional shape of an object based on the multi-view image data acquired by the image acquisition unit 201, and acquires three-dimensional shape data indicating the three-dimensional shape. Here, the three-dimensional shape data acquired by the shape acquisition unit 202 is three-dimensional shape sequence data composed of time-series three-dimensional shape data based on frame data obtained by synchronous imaging by a plurality of imaging devices in each captured image data. In the following description, it is assumed that a plurality of objects exist in the imaging target areas of the plurality of imaging devices. The identification assignment unit 203 assigns an object number corresponding to each object as identification information to the three-dimensional shape sequence data acquired by the shape acquisition unit 202. The sequence generation unit 204 generates mesh sequence data, which is time-series polygon mesh data independent for each object, based on the three-dimensional shape sequence data acquired by the shape acquisition unit 202 and the identification information assigned by the identification assignment unit 203. Hereinafter, the polygon mesh will be referred to as a mesh, and the mesh data will be referred to as mesh data.

[0016] The texture adding unit 205 adds texture to the time-series meshes indicated by the mesh sequence data based on the mesh sequence data generated by the sequence generation unit 204 and the multi-view image data acquired by the image acquisition unit 201. Specifically, the texture adding unit 205 adds texture to the time-series meshes indicated by the mesh sequence data by generating meshes and texture image data with UV coordinates based on the mesh sequence data and the multi-view image data. Hereinafter, the mesh sequence data to which the texture is added is referred to as textured mesh sequence data. That is, the texture adding unit 205 generates textured mesh sequence data based on the mesh sequence data and the multi-view image data. The compression unit 206 compresses the textured mesh sequence data generated by the texture adding unit 205 and outputs the compressed textured mesh sequence data as compressed data.

[0017] <Operation of image processing device> FIG. 3 is a flowchart showing an example of a processing flow in the image processing device 100 according to the first embodiment. In the following description, "S" at the beginning of each reference symbol indicates a step (process). The processing of each step shown in the flowchart in FIG. 3 is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, etc., expanding it in the RAM 103, and executing it. FIG. 4 is a diagram for explaining an example of data changes in the processing from S302 to S305 shown in FIG. 3 in the image processing device 100 according to the first embodiment. In FIG. 4, the horizontal axis indicates time t, the vertical axis indicates position x, and each of the multiple open circles indicates the three-dimensional shape of an object estimated by the shape acquisition unit 202.

[0018] First, in S301, the image acquisition unit 201 acquires multi-view image data, i.e., multiple video image data obtained by synchronous imaging using multiple imaging devices. Next, in S302, the shape acquisition unit 202 acquires 3D shape data for each object. Specifically, the multi-view image data acquired in S301 is used to estimate the 3D shape of each object, and 3D shape data indicating the 3D shape is acquired. More specifically, the shape acquisition unit 202 acquires 3D shape sequence data by generating 3D shape data for each object corresponding to the imaging time for each imaging time using multiple frame data obtained by synchronous imaging using multiple imaging devices.

[0019] In this embodiment, the shape acquisition unit 202 extracts an area corresponding to an object in each frame constituting each captured image, estimates the three-dimensional shape of the object by a volume intersection method based on the area, and acquires three-dimensional shape data. The method of estimating the three-dimensional shape of the object is not limited to the volume intersection method. For example, the estimation method includes various methods such as a method of estimating a surface point group of the object by multi-view stereo, or a method of estimating a signed distance field around the object by NeuS (Neural implicit surface). In addition, the shape acquisition unit 202 may acquire three-dimensional shape data from an external source without estimating the three-dimensional shape of the object. FIG. 4(a) shows an example of a time-series three-dimensional shape indicated by the three-dimensional shape sequence data acquired by the shape acquisition unit 202. Shapes 401 and 402 each represent a three-dimensional shape of one object different from each other, and shape 403 represents a three-dimensional shape in a state where two objects are in contact with each other.

[0020] After S302, in S303, the identification unit 203 analyzes the 3D shape sequence data acquired in S302 to assign an object number that uniquely corresponds to each object as identification information to the 3D shape data at each time point. As a method for assigning identification information to the 3D shape data, a known method such as the method disclosed in Japanese Patent Application No. 2021-134886 may be used. FIG. 4(b) shows an example of identification information (object number) that the identification unit 203 assigns to the time-series 3D shape shown in FIG. 4(a).

[0021] As described above, shapes 401 and 402 each represent the three-dimensional shape of one different object, and shape 403 represents the three-dimensional shape in a state where the two objects are in contact. Therefore, the three-dimensional shape data corresponding to shapes 401 and 402 are assigned object numbers corresponding to one object, such as shapes 411 and 412. Furthermore, the three-dimensional shape data corresponding to shape 403 is assigned object numbers corresponding to two objects, such as shape 413. Hereinafter, three-dimensional shape data assigned one object number will be referred to as single shape information, and three-dimensional shape data assigned multiple object numbers will be referred to as combined shape information.

[0022] After S303, in S304, the sequence generation unit 204 generates mesh sequence data for each object having a common topology. Details of the processing by the sequence generation unit 204 will be described later. Fig. 4(c) shows an example of mesh sequence data generated by the sequence generation unit 204. As shown in Fig. 4(c), a shape 413 is represented by independent shapes 421 and 421 for each object in the mesh sequence data generated by the sequence generation unit 204.

[0023] Next, in S305, the texture adding unit 205 adds texture to the mesh sequence data for each object generated in S304, to generate textured mesh sequence data for each object. For example, the texture adding unit 205 UV-develops the surface of each mesh included in the mesh sequence onto a two-dimensional texture image, and generates textured mesh sequence data by determining the color of each pixel of the texture image with reference to the captured image. At this time, the same UV coordinates are given to meshes with a common topology. Also, the size of the texture image added to each of the multiple meshes corresponding to the same object is the same.

[0024] Next, in S306, the compression unit 206 compresses the textured mesh sequence data for each object generated in S305 to generate compressed data. The generated compressed data is output to and stored in, for example, the auxiliary storage device 104. The output destination of the compressed data is not limited to the auxiliary storage device 104, and the compressed data may be output to an external device via the communication I / F 106. Fig. 4(d) shows an example of textured mesh sequence data for each object generated by the texture adding unit 205. The compression unit 206 compresses a series of textured mesh data for each object in the time direction, which is enclosed by a rectangle drawn by a solid line in Fig. 4(d).

[0025] For example, the texture image data in each textured mesh data is compressed based on a video compression standard such as H.264. However, the compression method of the texture image data is not limited to this, and the texture image data may be compressed using other codecs such as HEVC or VP9. Furthermore, the mesh data in each textured mesh data is compressed using a Draco library or the like. Here, in the compression of mesh data, only one UV coordinate and topology information may be stored in a range where the topology is common, and only information indicating the coordinates of the vertices may be stored as each mesh data. In order to achieve more efficient compression, the compression unit 206 may compress the difference in information between vertices of the same index between meshes in a time series.

[0026] However, the method of compressing mesh data is not limited to this, and it is not always necessary to utilize the correlation between meshes in time series in compressing mesh data. In addition, mesh data may be compressed using other codecs such as MPEG-3DGC. In addition, the compression unit 206 may convert each textured mesh data into colored point cloud data and compress the converted colored point cloud data using V-PCC (Video-based point cloud compression) standardized by MPEG (Moving Picture Expert Group). After S306, the image processing device 100 ends the process of the flowchart shown in FIG. 3.

[0027] The mesh sequence data generation process in the sequence generation unit 204 will be described in detail with reference to Fig. 5 to Fig. 7. Fig. 5 is a flowchart showing an example of the flow of the mesh sequence data generation process in the sequence generation unit 204 according to the first embodiment, and is a flowchart showing an example of a detailed processing flow of the process of S304 shown in Fig. 3. In the mesh sequence data generation process in S304, the sequence generation unit 204 sets a transformation path for separating a three-dimensional shape including three-dimensional shapes of two or more objects into three-dimensional shapes for each object, based on the identification information.

[0028] 6 and 7 are diagrams for explaining the deformation path setting process in the sequence generation unit 204 according to the first embodiment. In the following, the description will be given assuming that the time-series three-dimensional shapes to which object numbers are assigned are regarded as a graph structure. If the three-dimensional shape of each object is treated as a node (hereinafter referred to as a "shape node") and shape nodes to which the same object number is assigned in adjacent frames are connected by edges, a graph structure such as that shown in FIG. 6(a) is obtained. The sequence generation unit 204 sets the deformation path by analyzing the graph structure shown in FIG. 6(a).

[0029] First, in S501, the sequence generation unit 204 identifies a shape node (hereinafter referred to as a "join node") corresponding to combined shape information to which multiple object numbers have been assigned. In FIG. 6, the combined node is represented by a shaded ellipse. As a result of the identification process in S501, a graph shown as an example in FIG. 6(a) is obtained. Next, in S502, the sequence generation unit 204 extracts a shape node corresponding to single shape information from among shape nodes that have been assigned the same object number as the combined node at a time adjacent to the time of the combined node identified in S501. Hereinafter, a shape node corresponding to single shape information is referred to as a "single node". In FIG. 6(b), the extracted single node is represented by enclosing it in a rectangle drawn with solid lines.

[0030] Next, in S503, the sequence generation unit 204 sets a deformation path from the simplex node to the combined node, if the combined node identified in S501 can be separated into two or more simplex nodes by transforming the simplex nodes in the adjacent frames extracted in S502. For example, in the graph shown as an example in FIG. 6(b), combined node 601 can be separated into two simplex nodes by transforming simplex nodes 602 and 603. In contrast, for combined node 604, there are no two or more simplex nodes that can be separated by transformation. FIG. 6(c) shows, with arrows, an example of a deformation path set by the sequence generation unit 204 in the process of S503 in the graph shown in FIG. 6(b).

[0031] Next, in S504, the sequence generating unit 204 separates the combined node for which the transformation path was set in S503 into a single node for each object. The object number of the object corresponding to each single node, which is the object number assigned to the combined node, is assigned to each of the separated single nodes. In addition, a transformation path from the corresponding single node at an adjacent time is set to each of the separated single nodes. Fig. 6(d) shows an example of a state after the combined node shown in Fig. 6(c) is separated and replaced with a plurality of single nodes.

[0032] Next, in S505, the sequence generation unit 204 judges whether all the shape nodes have become single nodes. If it is judged in S505 that at least some of the shape nodes have not become single nodes, the sequence generation unit 204 repeatedly executes the processes from S502 to S505 until it is judged in S505 that all the shape nodes have become single nodes. For example, the shape nodes in the initial state, which are in the state of the graph shown in FIG. 7(a), become in the state of the graph shown in FIG. 7(b) as a result of the first processing from S502 to S505. Furthermore, by repeating the processing from S502 to S505, the state of the graph shown in FIG. 7(b) changes in sequence to the states of the graphs shown in FIGS. 7(c) and (d).

[0033] If it is determined in S505 that all shape nodes have become simplex nodes, the sequence generation unit 204 executes the process of S506. For example, in the state of the graph shown in Fig. 7(d), all shape nodes have become simplex nodes. In this state, the sequence generation unit 204 executes the process of S506. In the following description, the shape node that is the starting point of transformation is referred to as a key node. A key node can be identified by extracting a simplex node that has no transformation path leading to it and has a transformation path leading from it.

[0034] In S506, the sequence generation unit 204 sets a transformation path for the shape node for which no transformation path was set in the process of S503. Specifically, the sequence generation unit 204 selects a key node with a time closest to the time of the shape node for which no transformation path was set, and sets a transformation path from the selected key node to that shape node. As a result of the setting process in S506, a transformation path from any of the key nodes is set for all shape nodes, as shown in Fig. 7(e).

[0035] Next, in S507, the sequence generation unit 204 meshes the three-dimensional shape data corresponding to the key nodes in the series of transformation paths to generate mesh (hereinafter referred to as "key mesh") data corresponding to the key nodes. For example, the sequence generation unit 204 generates the key mesh data by converting the three-dimensional shape data corresponding to the key nodes into mesh data using the marching cubes method.

[0036] Next, in S508, the sequence generation unit 204 generates mesh sequence data for each object. Specifically, the sequence generation unit 204 performs non-rigid registration to match the key mesh to a three-dimensional shape at another time based on the key mesh data generated in S507 and a series of deformation paths from the key node corresponding to the key mesh data. By performing such non-rigid registration, the sequence generation unit 204 can generate mesh sequence data independent of each object. For example, the sequence generation unit 204 performs key mesh registration by regarding three-dimensional shape data represented by voxels as a signed distance field. The method of key mesh registration by the sequence generation unit 204 is not limited to this, and for example, the sequence generation unit 204 may convert the three-dimensional shape data into a surface point group to perform key mesh registration. Also, for example, the sequence generation unit 204 may mesh the three-dimensional shape data in the same way as the key mesh to perform key mesh registration, or may perform key mesh registration using other data representation.

[0037] After the process of S508, the sequence generation unit 204 ends the process of the flowchart shown in FIG. 5, i.e., the process of S304. As a result of the process of S304, mesh sequence data that is independent for each object and has a common topology can be obtained. After the process of S304, the texture assignment unit 205 generates and assigns texture image data for each mesh sequence data having a common topology as shown in FIG. 7(f). In addition, the compression unit 206 compresses the textured mesh sequence data for each mesh sequence data having a common topology. In the following, the unit of texture image data generation and textured mesh sequence data compression, which is divided by a rectangle drawn by a solid line in FIG. 7(f), will be referred to as GOM (Group of Models) for explanation.

[0038] FIG. 8 is a diagram showing an example of the data structure of textured mesh sequence data output by the texture adding unit 205 according to the first embodiment, and the data structure of compressed data output by the compression unit 206. FIG. 8(a) shows an example of the data structure of textured mesh sequence data output by the texture adding unit 205. The textured mesh sequence data shown as an example in FIG. 8(a) is obtained by converting mesh data with UV coordinates into a file in the ply format and a texture image (texture) into a file in the png format. FIG. 8(b) shows an example of the data structure of compressed data output by the compression unit 206. The compressed data has a structure that allows the compressed data to be transmitted individually for each object, as in the data structure shown as an example in FIG. 8(b). According to such a data structure of compressed data, the compressed data can be transmitted individually for each object, and the transmitted compressed data can be decoded for each object at the transmission destination.

[0039] According to the image processing device 100 configured as above, it is possible to generate textured mesh sequence data independent for each object. As a result, the three-dimensional shape data becomes easy for the user to handle, and the user can easily apply independent processing to the three-dimensional shape data for each object, such as editing the three-dimensional shape data for each object. As a result, according to the image processing device 100, by compressing the textured mesh sequence data for each object, it is possible to efficiently compress the textured mesh sequence data, i.e., improve the compression efficiency.

[0040] [Embodiment 2] The image processing device according to the first embodiment generates independent textured mesh sequence data for each object, and compresses the generated textured mesh sequence data for each object individually. In the second embodiment, a form will be described in which colored point cloud (hereinafter referred to as "colored point cloud") data is separated for each object, and the colored point cloud data for each object after separation is edited and compressed. Note that the second embodiment will be described mainly with respect to the differences from the first embodiment, and the description of the similarities will be omitted.

[0041] 9 is a block diagram showing an example of a functional configuration of an image processing device 100 according to the second embodiment (hereinafter simply referred to as "image processing device 100"). The image processing device 100 has, as its functional configuration, a shape acquisition unit 902, an identification assignment unit 903, a sequence generation unit 904, a sequence modification unit 910, and a compression unit 906. The hardware configuration of the image processing device 100 is similar to that of the image processing device 100 according to the first embodiment, and therefore a description thereof will be omitted. Each unit that the image processing device 100 has as its functional configuration is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, etc., expanding the program in the RAM 103, and executing the program.

[0042] The shape acquisition unit 902 acquires colored point cloud sequence data composed of time-series colored point cloud data. The identification assignment unit 903 identifies a point cloud corresponding to each object for each point cloud data based on information indicating the position and color of each point in each colored point cloud data constituting the colored point cloud sequence data acquired by the shape acquisition unit 902. The identification assignment unit 903 also assigns identification information to the point cloud data corresponding to each identified object. The method of assigning the identification information may be the same as that of the first embodiment. That is, the identification assignment unit 903 assigns, for example, an object number corresponding to the object as identification information to the point cloud data corresponding to each object.

[0043] The sequence generation unit 904 generates independent time-series colored point cloud data for each object based on information indicating the position and color of each point in the time-series colored point cloud data acquired by the shape acquisition unit 902 and the identification information assigned by the identification assignment unit 903. That is, the sequence generation unit 904 generates colored point cloud sequence data composed of time-series colored point cloud data for each object for each object. The sequence modification unit 910 modifies the colored point cloud sequence data for each object according to an editing operation by a user, and outputs the modified colored point cloud sequence data to the compression unit 906. The compression unit 906 compresses the colored point cloud sequence data for each object to generate compressed data for each object. The compressed data generated by the compression unit 906 is output to the auxiliary storage device 104 or the like.

[0044] Fig. 10 is a flowchart showing an example of a processing flow in the image processing device 100 according to the second embodiment. Note that the processing of each step shown in the flowchart in Fig. 10 is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, etc., developing it in the RAM 103, and executing it. First, in S1001, the shape acquisition unit 902 acquires colored point cloud sequence data. Next, in S1002, the identification assignment unit 903 identifies a point cloud corresponding to each object in each of the time-series colored point cloud data constituting the colored point cloud sequence data acquired in S1001, that is, a colored point cloud corresponding to each object. Next, in S1003, the identification assignment unit 903 assigns identification information to the point cloud data corresponding to each object identified in S1002.

[0045] Next, in S1004, the sequence generating unit 904 generates colored point cloud sequence data for each object based on the colored point cloud sequence data acquired in S1001 and the identification information assigned in S1003. Specifically, the sequence generating unit 904 selects colored point cloud data of an arbitrary object, and aligns the colored point cloud of the selected object with the colored point cloud of the alignment target. Here, the alignment target point cloud is, for example, a colored point cloud corresponding to the object at the time closest to the time of the colored point cloud of the selected object. For example, the sequence generating unit 904 aligns the colored point cloud of the selected object so that the color and position of each point in the colored point cloud of the selected object are the same as the color and position of each point in the colored point cloud of the alignment target after alignment. By performing such alignment, the sequence generating unit 904 can generate colored point cloud sequence data independent of each object.

[0046] Each point in the point cloud data acquired in S1001 is accompanied by color information. Therefore, the sequence generation unit 904 can align the point clouds more accurately than when color information is not used. For aligning colored point clouds, reference 1 (Marcelo Saval-Calvo and 5 others, “3D non-rigid registration using color: Color Coherent Point Drift”, [online], February 5, 2018, arXiv, [searched August 23, 2023], Internet) is used.<https: / / arxiv.org / pdf / 1802.01516.pdf> ) may be used.

[0047] After S1004, in S1005, the sequence modification unit 910 modifies the colored point cloud sequence data for each object generated in S1004 in accordance with the user's editing operation to generate modified colored point cloud sequence data. The generated modified colored point cloud sequence data is output to the compression unit 906. Specific editing operations will be described later with reference to FIG. 11. Next, in S1006, the compression unit 906 compresses the modified colored point cloud sequence data for each object generated in S1005 to generate compressed data for each object. The compressed data generated by the compression unit 906 is output to the auxiliary storage device 104 or the like. After S1006, the image processing device 100 ends the processing of the flowchart shown in FIG. 10.

[0048] For example, the modified colored point cloud sequence data for each object may be compressed using G-PCC (Geometry-based Point Cloud Compression) standardized by MPEG. The method of compressing the modified colored point cloud sequence data is not limited to the method using G-PCC, and other methods such as V-PCC may be used for compressing the modified colored point cloud sequence data. In addition, the compression unit 906 may compress the modified colored point cloud sequence data using a method using three-dimensional time correlation as follows. Specifically, for example, first, the compression unit 906 compresses point cloud data corresponding to a key node in the modified colored point cloud sequence data using the Draco library. Next, the compression unit 906 compresses the difference in information indicating coordinates and colors in the point cloud data in the GOM including the key node.

[0049] <Editing operations by the user> 11 is a diagram showing an example of an editing UI (user interface) 1100 for receiving an editing operation of colored point cloud sequence data by a user according to the second embodiment. The editing UI has areas 1101 and 1102. Area 1101 is a screen area in which a colored point cloud for each object in a scene at an arbitrary time point is drawn. Area 1102 is a screen area in which a point cloud sequence managed for each object is drawn using a timeline. A user can have a colored point cloud at a time point desired by the user drawn in area 1101 by operating a seek bar on the timeline.

[0050] A shape 1103 represents three-dimensional shape data erroneously generated when estimating the shape of an object with an object number of 2. For example, a user can delete unnecessary three-dimensional shape data such as the shape 1103 by selecting it in the editing UI 1100. Specifically, for example, a user selects the shape 1103 in the area 1101 by an input operation using a mouse or the like, and then presses a delete button or the like not shown in FIG. 11 to instruct the image processing device 100 to delete the three-dimensional shape data corresponding to the shape 1103. Note that the operation of selecting and deleting the shape 1103 can also be performed on the timeline drawn in the area 1102. In this case, for example, a user can delete the three-dimensional shape data corresponding to the shape 1103 by selecting a bar with an object number of 2 in the timeline and then pressing a delete button not shown in FIG. 11.

[0051] In addition, the user can set, in the editing UI 1100, attribute information of an object corresponding to a three-dimensional shape drawn in the editing UI 1100. Specifically, in the editing UI 1100, the user selects a three-dimensional shape or an object number corresponding to a desired object, and sets a category, a group, a name, or the like of the object. The category is information representing a type of an object, such as a person or an object. The group is information representing an affiliation of an object, and is, for example, information that can further classify a type of an object set in a category. For example, when the subject of the image capture is a team sport such as soccer, the name of a team to which the object belongs may be set in the group. The name is information that allows a person to identify the object, such as a unique name of the object. For example, when the category of the object is a person, the name is set to the name, and the like of the object. Also, when the subject of the image capture is a team sport such as soccer, the name may be set to the uniform number of a player who is the object.

[0052] According to the image processing device 100 configured as above, it is possible to generate independent point cloud sequence data for each object from the point cloud sequence data. As a result, according to the image processing device 100, it is possible to efficiently compress the point cloud sequence data by compressing the point cloud sequence data for each object. Furthermore, according to the image processing device 100 configured as above, it is possible to edit a volumetric video represented by the point cloud sequence data for each object, and compress and output the edited point cloud sequence data.

[0053] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0054] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0055] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0056] <Configuration 1> A shape acquisition means for acquiring three-dimensional shape sequence data including time-series three-dimensional shape data representing three-dimensional shapes of a plurality of objects; a separation means for, when the time-series three-dimensional shape data includes data of a combined shape in which three-dimensional shapes of two or more objects among the plurality of objects are combined into a single three-dimensional shape, separating the combined shape data into object shape data which is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; a sequence generating means for generating object shape sequence data for each object, the object shape sequence data being three-dimensional shape sequence data including the object shape data in time series for each object; a compression means for compressing the object shape sequence data for each object; 13. An image processing device comprising:

[0057] <Configuration 2> the separating means separates the combined shape data into the object shape data corresponding to each of the two or more objects by transforming data of a single shape in which the three-dimensional shapes of the two or more objects are not combined with each other in the three-dimensional shape data that is adjacent in time series to the three-dimensional shape data including the combined shape data; 2. The image processing device according to claim 1,

[0058] <Configuration 3> the three-dimensional shape sequence data is colored point cloud sequence data including time-series colored point cloud data corresponding to surfaces of the plurality of objects as the time-series three-dimensional shape data; the separating means separates the colored point cloud data into colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating a position of each point in the time-series colored point cloud data; the sequence generation means generates, for each object, colored object point cloud sequence data including the colored object point cloud data in time series corresponding to each of the plurality of objects, as the three-dimensional shape sequence data, based on the colored object point cloud data having the same number of points; 3. The image processing device according to configuration 1 or 2,

[0059] <Configuration 4> the separating means separates the colored point cloud data into the colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating a position of each point and information indicating a color of each point in the time-series colored point cloud data; 4. The image processing device according to configuration 3,

[0060] <Component 5> the three-dimensional shape sequence data is textured mesh sequence data including, as the time-series three-dimensional shape data, time-series textured mesh data indicating surface shapes of the plurality of objects; the separating means separates the textured mesh data into object mesh data with textures corresponding to the plurality of objects by transforming the textured mesh data based on information indicating positions of each vertex in the time-series textured mesh data; the sequence generation means generates, for each object, textured object mesh sequence data including the textured object mesh data in time series corresponding to each of the plurality of objects, as the three-dimensional shape sequence data, based on the textured object mesh data having a matching topology; 3. The image processing device according to configuration 1 or 2,

[0061] <Component 6> The method further includes an image acquisition unit for acquiring captured image data obtained by capturing images of the plurality of objects, the shape acquisition means estimates the three-dimensional shape sequence data using the acquired captured image data, thereby acquiring the three-dimensional shape sequence data; 6. The image processing device according to any one of configurations 1 to 5,

[0062] <Component 7> the shape acquisition means assigns color information to the three-dimensional shape sequence data by using the captured image data; 7. The image processing device according to configuration 6,

[0063] <Component 8> further comprising an identification unit for providing identification information corresponding to each of the plurality of objects to each of the time-series three-dimensional shape data included in the three-dimensional shape sequence data; 8. The image processing device according to any one of configurations 1 to 7,

[0064] <Component 9> the sequence generating means generates the object shape sequence data for each object based on the assigned identification information; 9. The image processing device according to configuration 8,

[0065] <Component 10> the separating means separates the combined shape data into the object shape data corresponding to each of the two or more objects based on the assigned identification information; 10. The image processing device according to configuration 8 or 9,

[0066] <Component 11> the separating means comprises: a transformation path setting means for setting, from the time-series three-dimensional shape data, origin shape data that serves as a origin for transforming the three-dimensional shape data, and a transformation path for transforming the three-dimensional shape data, based on the assigned identification information; and a transformation means for transforming the three-dimensional shape data in time-series order according to the transformation path from the origin three-dimensional shape data, based on the origin shape data and the transformation path; 11. The image processing device according to any one of configurations 8 to 10,

[0067] <Component 12> said separating means, when the identification information assigned to the combined shape data includes the identification information assigned to single shape data in which the three-dimensional shapes of the two or more objects adjacent in time series are not combined with each other, sets the deformation path from the single shape data to the combined shape data; 12. The image processing device according to claim 11,

[0068] <Component 13> a compression means for compressing the object shape sequence data for each object, 13. The image processing device according to claim 1, further comprising:

[0069] <Component 14> said compressing means compresses said object shape sequence data for each range in which origin object shape data serving as a origin for transforming said object shape data is common; 14. The image processing device according to claim 13,

[0070] <Component 15> a reception means for receiving an editing operation of the object shape sequence data; a sequence modification means for modifying the object shape sequence data based on the editing operation to generate modified object shape sequence data; Further comprising: said compression means compresses, with respect to the object shape sequence data from which the changed object shape sequence data has been generated, the changed object shape sequence data in place of the object shape sequence data; 15. The image processing device according to configuration 13 or 14,

[0071] <Method> a shape acquiring step of acquiring three-dimensional shape sequence data including time-series three-dimensional shape data indicating three-dimensional shapes of a plurality of objects; a separation step of, when the time-series three-dimensional shape data includes data of a combined shape in which three-dimensional shapes of two or more objects among the plurality of objects are combined into one three-dimensional shape, separating the combined shape data into object shape data which is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; a sequence generating step of generating, for each object, object shape sequence data, which is three-dimensional shape sequence data including the object shape data in time series for each object; 13. An image processing method comprising:

[0072] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0073] 100 Image processing device 202 Shape acquisition section 204 Sequence Generation Unit 206 Compression section

Claims

1. a shape acquisition means for acquiring three-dimensional shape sequence data including time-series three-dimensional shape data representing three-dimensional shapes of a plurality of objects; a separation means for, when the time-series three-dimensional shape data includes data of a combined shape in which three-dimensional shapes of two or more objects among the plurality of objects are combined into a single three-dimensional shape, separating the combined shape data into object shape data which is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; a sequence generating means for generating object shape sequence data for each object, the object shape sequence data being three-dimensional shape sequence data including the object shape data in time series for each object; 13. An image processing device comprising:

2. the separating means separates the combined shape data into the object shape data corresponding to each of the two or more objects by transforming data of a single shape in which the three-dimensional shapes of the two or more objects are not combined with each other in the three-dimensional shape data that is adjacent in time series to the three-dimensional shape data including the combined shape data; 2. The image processing device according to claim 1,

3. the three-dimensional shape sequence data is colored point cloud sequence data including time-series colored point cloud data corresponding to surfaces of the plurality of objects as the time-series three-dimensional shape data; the separating means separates the colored point cloud data into colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating a position of each point in the time-series colored point cloud data; the sequence generation means generates, for each object, colored object point cloud sequence data including the colored object point cloud data in time series corresponding to each of the plurality of objects, as the three-dimensional shape sequence data, based on the colored object point cloud data having the same number of points; 2. The image processing device according to claim 1,

4. the separating means separates the colored point cloud data into the colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating a position of each point and information indicating a color of each point in the time-series colored point cloud data; 4. The image processing device according to claim 3,

5. the three-dimensional shape sequence data is textured mesh sequence data including, as the time-series three-dimensional shape data, time-series textured mesh data indicating surface shapes of the plurality of objects; the separating means separates the textured mesh data into object mesh data with textures corresponding to the plurality of objects by transforming the textured mesh data based on information indicating positions of each vertex in the time series textured mesh data; the sequence generation means generates, for each object, textured object mesh sequence data including the textured object mesh data in time series corresponding to each of the plurality of objects, as the three-dimensional shape sequence data, based on the textured object mesh data having a matching topology; 2. The image processing device according to claim 1,

6. The method further includes an image acquisition unit for acquiring captured image data obtained by capturing images of the plurality of objects, the shape acquisition means estimates the three-dimensional shape sequence data using the acquired captured image data, thereby acquiring the three-dimensional shape sequence data; 2. The image processing device according to claim 1,

7. the shape acquisition means assigns color information to the three-dimensional shape sequence data by using the captured image data; 7. The image processing device according to claim 6,

8. an identification unit that assigns identification information corresponding to each of the plurality of objects to each of the time-series three-dimensional shape data included in the three-dimensional shape sequence data; 2. The image processing device according to claim 1,

9. the sequence generating means generates the object shape sequence data for each object based on the assigned identification information; The image processing device according to claim 8 .

10. the separating means separates the combined shape data into the object shape data corresponding to each of the two or more objects based on the assigned identification information; The image processing device according to claim 8 .

11. the separating means comprises: a transformation path setting means for setting starting point shape data serving as a starting point for transforming the three-dimensional shape data from the time-series three-dimensional shape data and a transformation path for transforming the three-dimensional shape data, based on the assigned identification information; and a transformation means for transforming the three-dimensional shape data in time-series order according to the transformation path from the starting point three-dimensional shape data, based on the starting point shape data and the transformation path. The image processing device according to claim 8 .

12. the separating means, when the identification information assigned to the combined shape data includes the identification information assigned to single shape data in which the three-dimensional shapes of the two or more objects adjacent in time series are not combined with each other, sets the deformation path from the single shape data to the combined shape data; The image processing device according to claim 11 .

13. a compression means for compressing the object shape sequence data for each object, 2. The image processing device according to claim 1, further comprising:

14. said compressing means compresses said object shape sequence data for each range in which origin object shape data serving as a origin for transforming said object shape data is common; The image processing device according to claim 13 .

15. a reception means for receiving an editing operation of the object shape sequence data; a sequence modification means for modifying the object shape sequence data based on the editing operation to generate modified object shape sequence data; and said compression means compresses, with respect to the object shape sequence data from which the changed object shape sequence data has been generated, the changed object shape sequence data in place of the object shape sequence data; The image processing device according to claim 13 .

16. a shape acquiring step of acquiring three-dimensional shape sequence data including time-series three-dimensional shape data indicating three-dimensional shapes of a plurality of objects; a separation step of, when the time-series three-dimensional shape data includes data of a combined shape in which three-dimensional shapes of two or more objects among the plurality of objects are combined into one three-dimensional shape, separating the combined shape data into object shape data which is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; a sequence generating step of generating, for each object, object shape sequence data, which is three-dimensional shape sequence data including the object shape data in time series for each object; 13. An image processing method comprising:

17. A program for causing a computer to function as the image processing device according to any one of claims 1 to 15.

Images