Image processing device, image processing method, and program

Abstract

An image processing device (100): acquires three-dimensional shape sequence data including chronological three-dimensional shape data indicating the three-dimensional shape of a plurality of objects; when data for a combined shape in which the three-dimensional shapes of two or more objects among the plurality of objects are combined into one three-dimensional shape is included in the chronological three-dimensional shape data, separates the data for the combined shape into object shape data that is the three-dimensional shape data corresponding to each of the two or more objects, and generates object shape data corresponding to each of the plurality of objects; and generates, for each object, object shape sequence data that is three-dimensional shape sequence data including chronological object shape data for each object.

Description

Image processing device, image processing method, and program

[0001] The present disclosure relates to a registration technique for time-varying three-dimensional shape data that indicates the shape of an object.

[0002] There is a technology that generates textured 3D shape data based on the 3D shape of an object estimated using data from multiple captured images obtained by capturing images using multiple image capture devices, and compresses the generated textured 3D shape data. For example, one method for efficiently compressing 3D shape data is to track a time-series 3D shape corresponding to the object as a preprocessing step before compression. Tracking the 3D shape involves determining reference 3D shape data and performing non-rigid registration of the 3D shape corresponding to the reference 3D shape data so that it matches a 3D shape that is adjacent in time (hereinafter referred to as an "adjacent 3D shape").

[0003] By aligning the time-series three-dimensional shapes in this way, the three-dimensional shape of the object changes over time, and the topology of the three-dimensional shape is made common between adjacent three-dimensional shapes. Furthermore, by sharing the topology of the three-dimensional shapes, it becomes possible to represent the movement of the object by moving each vertex that constitutes the three-dimensional shape. Furthermore, by expressing the three-dimensional shape data using information indicating the movement of the object, i.e., the movement of the vertices, and further compressing this information, the amount of data in the three-dimensional shape data can be efficiently reduced.

[0004] However, in a scene where many objects exist simultaneously, it is difficult to align all generated 3D shapes at once in terms of the processing time required for calculations. One method to avoid this problem is to assign identification information to each 3D shape to identify the object and align the 3D shapes for each object. However, if the 3D shapes corresponding to multiple objects that are close to each other are combined and estimated as a single 3D shape, it is not possible to align the 3D shapes corresponding to each object separately.

[0005] Patent Literature 1 discloses a technique for aligning each polygon mesh by combining polygon meshes corresponding to a plurality of objects, treating the resulting polygon mesh as a polygon mesh corresponding to one object group. According to the technique disclosed in Patent Literature 1, even when there are a plurality of objects close to each other, the topology of the polygon meshes in time series is made common.

[0006] Japanese Patent Application Laid-Open No. 2020-136943

[0007] However, with the technology disclosed in Patent Document 1, when multiple adjacent objects are represented by a single polygon mesh, the multiple objects are treated as a single piece of data. Therefore, with the technology disclosed in Patent Document 1, when multiple objects are close to each other, it is difficult for the user to process the three-dimensional shape data for each object independently.

[0008] In order to solve the above-mentioned problems, the image processing device according to the present disclosure has a shape acquisition means for acquiring three-dimensional shape sequence data including time-series three-dimensional shape data indicating the three-dimensional shapes of a plurality of objects; a separation means for, when the time-series three-dimensional shape data includes data of a combined shape in which the three-dimensional shapes of two or more of the plurality of objects are combined into a single three-dimensional shape, separating the combined shape data into object shape data that is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; and a sequence generation means for generating, for each object, object shape sequence data that is three-dimensional shape sequence data including the time-series object shape data for each object.

[0009] According to the present disclosure, even when the three-dimensional shapes of multiple objects are represented by a single piece of three-dimensional shape data, it is possible to easily process the three-dimensional shape data independently for each object.

[0010] Further features of the present disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings.

[0011] 1 is a block diagram showing an example of the hardware configuration of an image processing device according to a first embodiment. FIG. 2 is a block diagram showing an example of the functional configuration of the image processing device according to the first embodiment. FIG. 3 is a flowchart showing an example of a processing flow in the image processing device according to the first embodiment. FIG. 4 is a diagram for explaining an example of a change in data in the processing of the image processing device according to the first embodiment. FIG. 5 is a flowchart showing an example of a flow of mesh sequence data generation processing in a sequence generation unit according to the first embodiment. FIG. 6 is a diagram for explaining a deformation path setting process in the sequence generation unit according to the first embodiment. FIG. 7 is a diagram for explaining a deformation path setting process in the sequence generation unit according to the first embodiment. FIG. 8 is a diagram showing an example of a data structure of textured mesh sequence data output by a texture adding unit and a data structure of compressed data output by a compression unit according to the first embodiment. FIG. 9 is a block diagram showing an example of the functional configuration of an image processing device according to a second embodiment. FIG. 10 is a flowchart showing an example of a processing flow in the image processing device according to the second embodiment. FIG. 11 is a diagram showing an example of an editing UI according to the second embodiment.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. Note that the same components will be described with the same reference numerals.

[0013] [First Embodiment] <Hardware Configuration of Image Processing Device> FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing device 100 according to the first embodiment. The image processing device 100 is configured as an information processing device such as a personal computer, and includes a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, an operation unit 105, a communication I / F 106, and a bus 107. The CPU 101 controls the entire image processing device 100 using computer programs and data stored in the ROM 102, the auxiliary storage device 104, or the like, thereby realizing each function of the image processing device 100. Note that in the image processing device 100, the CPU 101 may include at least one or more dedicated hardware and a graphics processing unit (GPU), and at least a portion of the processing by the CPU 101 may be performed by the GPU or dedicated hardware. Examples of dedicated hardware include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a digital signal processor (DSP).

[0014] The ROM 102 stores programs and the like that do not require modification. The RAM 103 operates as a work area for the CPU 101 and temporarily stores programs and data supplied from the auxiliary storage device 104, as well as data supplied from the outside via the communication I / F 107. The auxiliary storage device 104 is configured with a hard disk drive or the like and stores various data such as input image data. The operation unit 105 is configured with a keyboard, a mouse, or the like and outputs various instructions to the CPU 101 in response to user operations. The CPU 101 operates as an operation control unit that controls the operation unit 105. The communication I / F 106 is used for communication with devices external to the image processing device 100. For example, if the image processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 106. If the image processing device 100 has a function for wireless communication with external devices, the communication I / F 106 is equipped with an antenna. The bus 107 connects the various components that the image processing apparatus 100 has as its hardware configuration and transmits information.

[0015] <Functional Configuration of Image Processing Device> FIG. 2 is a block diagram showing an example of the functional configuration of the image processing device 100 according to the first embodiment. The image processing device 100 has, as its functional configuration, an image acquisition unit 201, a shape acquisition unit 202, an identification assignment unit 203, a sequence generation unit 204, a texture assignment unit 205, and a compression unit 206. Each unit included in the functional configuration of the image processing device 100 is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, etc., expanding the program into the RAM 103, and executing the program. The image acquisition unit 201 acquires data (hereinafter referred to as "multi-viewpoint image data") of multiple captured images (hereinafter referred to as "multi-viewpoint images") obtained by capturing images using multiple image capture devices. Here, each of the multiple captured image data constituting the multi-viewpoint image data is moving image data composed of multiple frame data, and the frame data is obtained by synchronized capturing by the multiple image capture devices.

[0016] The shape acquisition unit 202 estimates the three-dimensional shape of an object based on the multi-viewpoint image data acquired by the image acquisition unit 201 and acquires three-dimensional shape data representing the three-dimensional shape. Here, the three-dimensional shape data acquired by the shape acquisition unit 202 is three-dimensional shape sequence data composed of time-series three-dimensional shape data based on frame data obtained by synchronized image capture by multiple image capture devices in each captured image data. In the following description, it is assumed that multiple objects exist in the image capture target areas of the multiple image capture devices. The identification unit 203 assigns object numbers corresponding to each object as identification information to the three-dimensional shape sequence data acquired by the shape acquisition unit 202. The sequence generation unit 204 generates mesh sequence data, which is time-series polygon mesh data independent for each object, based on the three-dimensional shape sequence data acquired by the shape acquisition unit 202 and the identification information assigned by the identification unit 203. Hereinafter, polygon meshes will be referred to as meshes, and mesh data will be referred to as mesh data.

[0017] The texture assigning unit 205 assigns textures to the time-series meshes indicated by the mesh sequence data based on the mesh sequence data generated by the sequence generation unit 204 and the multi-view image data acquired by the image acquisition unit 201. Specifically, the texture assigning unit 205 assigns textures to the time-series meshes indicated by the mesh sequence data by generating mesh and texture image data with UV coordinates based on the mesh sequence data and the multi-view image data. Hereinafter, the mesh sequence data to which textures have been assigned is referred to as textured mesh sequence data. In other words, the texture assigning unit 205 generates textured mesh sequence data based on the mesh sequence data and the multi-view image data. The compression unit 206 compresses the textured mesh sequence data generated by the texture assigning unit 205 and outputs the compressed textured mesh sequence data as compressed data.

[0018] <Operation of Image Processing Device> Fig. 3 is a flowchart showing an example of a processing flow in the image processing device 100 according to the first embodiment. Note that in the following description, "S" at the beginning of a reference symbol indicates a step (process). The processing of each step shown in the flowchart in Fig. 3 is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, etc., expanding it in the RAM 103, and executing it. Fig. 4 is a diagram for explaining an example of data changes in the processing from S302 to S305 shown in Fig. 3 in the image processing device 100 according to the first embodiment. Note that in Fig. 4, the horizontal axis represents time t, the vertical axis represents position x, and each of the multiple open circles represents the three-dimensional shape of an object estimated by the shape acquisition unit 202.

[0019] First, in S301, the image acquisition unit 201 acquires multi-viewpoint image data, i.e., multiple video image data obtained by synchronized image capture using multiple image capture devices. Next, in S302, the shape acquisition unit 202 acquires 3D shape data for each object. Specifically, the multi-viewpoint image data acquired in S301 is used to estimate the 3D shape of each object, and 3D shape data indicating the 3D shape is acquired. More specifically, the shape acquisition unit 202 acquires 3D shape sequence data by generating 3D shape data for each object corresponding to the image capture time using multiple frame data obtained by synchronized image capture using multiple image capture devices.

[0020] In this embodiment, the shape acquisition unit 202 extracts an area corresponding to an object in each frame constituting each captured image, estimates the three-dimensional shape of the object based on the extracted area using a volume intersection method, and acquires three-dimensional shape data. The method for estimating the three-dimensional shape of an object is not limited to the volume intersection method. For example, various estimation methods are available, such as a method of estimating a surface point cloud of an object using multi-view stereo or a method of estimating a signed distance field around an object using NeuS (Neural Implicit Surface). Alternatively, the shape acquisition unit 202 may acquire three-dimensional shape data from an external source without estimating the three-dimensional shape of the object. FIG. 4A illustrates an example of a time-series three-dimensional shape represented by three-dimensional shape sequence data acquired by the shape acquisition unit 202. Shapes 401 and 402 each represent the three-dimensional shapes of two different objects, and shape 403 represents the three-dimensional shape of two objects in contact with each other.

[0021] After S302, in S303, the identification unit 203 analyzes the 3D shape sequence data acquired in S302 and assigns an object number that uniquely corresponds to each object as identification information to the 3D shape data at each time point. A known method for assigning identification information to 3D shape data, such as the method disclosed in Japanese Patent Application No. 2021-134886, may be used. Figure 4(b) shows an example of identification information (object number) that the identification unit 203 assigns to the time-series 3D shapes shown in Figure 4(a).

[0022] As described above, shapes 401 and 402 each represent the three-dimensional shape of a single object, and shape 403 represents the three-dimensional shape of two objects in contact. Therefore, the three-dimensional shape data corresponding to shapes 401 and 402 are assigned object numbers corresponding to one object, such as shapes 411 and 412. Furthermore, the three-dimensional shape data corresponding to shape 403 is assigned object numbers corresponding to each of the two objects, such as shape 413. Hereinafter, three-dimensional shape data assigned one object number will be referred to as single shape information, and three-dimensional shape data assigned multiple object numbers will be referred to as combined shape information.

[0023] After S303, in S304, the sequence generation unit 204 generates mesh sequence data for each object that has a common topology. Details of the processing by the sequence generation unit 204 will be described later. Fig. 4C shows an example of mesh sequence data generated by the sequence generation unit 204. As shown in Fig. 4C, the shape 413 is represented by independent shapes 421 and 421 for each object in the mesh sequence data generated by the sequence generation unit 204.

[0024] Next, in S305, the texture assigning unit 205 assigns texture to the mesh sequence data for each object generated in S304, thereby generating textured mesh sequence data for each object. For example, the texture assigning unit 205 generates textured mesh sequence data by UV mapping the surface of each mesh included in the mesh sequence onto a two-dimensional texture image and determining the color of each pixel in the texture image with reference to the captured image. At this time, the same UV coordinates are assigned to meshes with a common topology. Furthermore, the size of the texture image assigned to each of multiple meshes corresponding to the same object is the same.

[0025] Next, in S306, the compression unit 206 compresses the textured mesh sequence data for each object generated in S305 to generate compressed data. The generated compressed data is output to and stored in, for example, the auxiliary storage device 104. The output destination of the compressed data is not limited to the auxiliary storage device 104, and the compressed data may be output to an external device via the communication I / F 106. Figure 4(d) shows an example of textured mesh sequence data for each object generated by the texture assigning unit 205. The compression unit 206 compresses a series of textured mesh data for each object in the time direction, which is enclosed by a rectangle drawn with a solid line in Figure 4(d).

[0026] For example, texture image data in each textured mesh data is compressed based on a video compression standard such as H.264. However, the compression method for texture image data is not limited to this, and texture image data may be compressed using other codecs such as HEVC or VP9. Furthermore, mesh data in each textured mesh data is compressed using a Draco library or the like. Here, in compressing mesh data, it is sufficient to store only one UV coordinate and topology information in a range with a common topology, and only information indicating the coordinates of the vertices is required to be stored as each mesh data. To achieve more efficient compression, the compression unit 206 may compress the difference in information between vertices with the same index between meshes in a time series.

[0027] However, the method for compressing mesh data is not limited to this, and it is not necessary to utilize the correlation between meshes in time series when compressing mesh data. Furthermore, mesh data may be compressed using other codecs such as MPEG-3DGC. Furthermore, the compression unit 206 may convert each textured mesh data into colored point cloud data and compress the converted colored point cloud data using V-PCC (Video-based point cloud compression), standardized by the Moving Picture Expert Group (MPEG). After S306, the image processing device 100 ends the processing of the flowchart shown in FIG. 3.

[0028] 5 to 7, the mesh sequence data generation process in the sequence generation unit 204 will be described in detail. Fig. 5 is a flowchart showing an example of the flow of the mesh sequence data generation process in the sequence generation unit 204 according to the first embodiment, and is a flowchart showing an example of a detailed processing flow of the process of S304 shown in Fig. 3. In the mesh sequence data generation process in S304, the sequence generation unit 204 sets a transformation path for separating a three-dimensional shape including three-dimensional shapes of two or more objects into three-dimensional shapes for each object, based on the identification information.

[0029] 6 and 7 are diagrams for explaining the deformation path setting process in the sequence generation unit 204 according to the first embodiment. Hereinafter, the description will be given assuming that the time-series three-dimensional shapes to which object numbers are assigned are regarded as a graph structure. When the three-dimensional shape of each object is treated as a node (hereinafter referred to as a "shape node") and shape nodes to which the same object number is assigned in adjacent frames are connected by edges, a graph structure such as that shown in FIG. 6A is obtained. The sequence generation unit 204 sets the deformation path by analyzing the graph structure shown in FIG. 6A.

[0030] First, in S501, the sequence generation unit 204 identifies a shape node (hereinafter referred to as a "join node") corresponding to combined shape information to which multiple object numbers are assigned. In FIG. 6, the combined node is represented by a shaded oval. As a result of the identification process in S501, the graph shown as an example in FIG. 6(a) is obtained. Next, in S502, the sequence generation unit 204 extracts a shape node corresponding to single shape information from among shape nodes that are assigned the same object number as the combined node at a time adjacent to the time of the combined node identified in S501. Hereinafter, a shape node corresponding to single shape information will be referred to as a "simple node." In FIG. 6(b), the extracted simple node is represented by a rectangle enclosed by solid lines.

[0031] Next, in S503, if the join node identified in S501 can be separated into two or more simple nodes by transforming the simple nodes in the adjacent frames extracted in S502, the sequence generation unit 204 sets a transformation path from the simple node to the join node. For example, in the graph shown as an example in Figure 6(b), join node 601 can be separated into two simple nodes by transforming simple nodes 602 and 603. In contrast, for join node 604, there are no two or more simple nodes that can be separated by transformation. Figure 6(c) shows, using arrows, an example of a transformation path set by the sequence generation unit 204 in the process of S503 for the graph shown in Figure 6(b).

[0032] Next, in S504, the sequence generation unit 204 separates the combined node for which the transformation path was set in S503 into simplex nodes for each object. Each of the separated simplex nodes is assigned the object number of the object corresponding to the combined node, which is the object number assigned to the combined node. Furthermore, each of the separated simplex nodes is assigned a transformation path from the corresponding simplex node at an adjacent time. Figure 6(d) shows an example of the state after the combined node shown in Figure 6(c) has been separated and replaced with multiple simplex nodes.

[0033] Next, in S505, the sequence generation unit 204 determines whether all shape nodes have become simplex nodes. If it is determined in S505 that at least some of the shape nodes have not become simplex nodes, the sequence generation unit 204 repeatedly executes the processes from S502 to S505 until it is determined in S505 that all shape nodes have become simplex nodes. For example, a shape node that was initially in the graph state shown in FIG. 7A changes to the graph state shown in FIG. 7B as a result of the first process from S502 to S505. Furthermore, by repeating the processes from S502 to S505, the graph state shown in FIG. 7B changes sequentially to the graph states shown in FIGS. 7C and 7D.

[0034] If it is determined in S505 that all shape nodes have become simplex nodes, the sequence generation unit 204 executes the process of S506. For example, in the graph state shown in FIG. 7D, all shape nodes have become simplex nodes. When this state is reached, the sequence generation unit 204 executes the process of S506. In the following description, the shape node that is the starting point of transformation will be referred to as a key node. A key node can be identified by extracting a simplex node that has no transformation path leading to it and has a transformation path leading from it.

[0035] In S506, the sequence generation unit 204 sets a transformation path for the shape node for which no transformation path was set in the processing of S503. Specifically, the sequence generation unit 204 selects a key node with a time closest to the time of the shape node for which no transformation path was set, and sets a transformation path from the selected key node to that shape node. As a result of the setting processing in S506, a transformation path from one of the key nodes is set for all shape nodes, as shown in FIG. 7( e).

[0036] Next, in S507, the sequence generation unit 204 generates mesh data corresponding to the key nodes (hereinafter referred to as "key mesh") by meshing the three-dimensional shape data corresponding to the key nodes in the series of deformation paths. For example, the sequence generation unit 204 generates the key mesh data by converting the three-dimensional shape data corresponding to the key nodes into mesh data using the marching cubes method.

[0037] Next, in S508, the sequence generation unit 204 generates mesh sequence data for each object. Specifically, the sequence generation unit 204 performs non-rigid registration to align the key mesh with a 3D shape at another time based on the key mesh data generated in S507 and a series of deformation paths from the key nodes corresponding to the key mesh data. By performing such non-rigid registration, the sequence generation unit 204 can generate independent mesh sequence data for each object. For example, the sequence generation unit 204 may align the key mesh by treating the 3D shape data represented by voxels as a signed distance field. The method of aligning the key mesh by the sequence generation unit 204 is not limited to this. For example, the sequence generation unit 204 may convert the 3D shape data into a surface point cloud and then align the key mesh. Furthermore, for example, the sequence generation unit 204 may mesh the 3D shape data in the same way as the key mesh and then align the key mesh, or may align the key mesh using other data representations.

[0038] After the process of S508, the sequence generation unit 204 ends the process of the flowchart shown in FIG. 5, i.e., the process of S304. As a result of the process of S304, mesh sequence data that is independent for each object and intermittently has a common topology can be obtained. After the process of S304, the texture assignment unit 205 generates and assigns texture image data for each piece of mesh sequence data that has a common topology, as shown in FIG. 7(f). Furthermore, the compression unit 206 compresses textured mesh sequence data for each piece of mesh sequence data that has a common topology. In the following description, the unit of texture image data generation and textured mesh sequence data compression, delimited by the solid-line rectangle in FIG. 7(f), will be referred to as a GOM (Group of Models).

[0039] 8A and 8B are diagrams showing an example of the data structure of textured mesh sequence data output by the texture assigning unit 205 according to the first embodiment and the data structure of compressed data output by the compression unit 206. FIG. 8A shows an example of the data structure of textured mesh sequence data output by the texture assigning unit 205. The textured mesh sequence data shown as an example in FIG. 8A is obtained by filing mesh data with UV coordinates in the ply format and a texture image (texture) in the png format. FIG. 8B shows an example of the data structure of compressed data output by the compression unit 206. The compressed data has a structure that enables the compressed data to be transmitted individually for each object, as shown as an example in FIG. 8B. This data structure of compressed data allows the compressed data to be transmitted individually for each object, and the transmitted compressed data to be decoded for each object at the transmission destination.

[0040] The image processing device 100 configured as described above can generate independent textured mesh sequence data for each object. This makes the three-dimensional shape data easier for the user to handle, making it easier for the user to apply independent processing to the three-dimensional shape data for each object, such as editing the three-dimensional shape data for each object. As a result, the image processing device 100 can compress the textured mesh sequence data for each object, thereby efficiently compressing the textured mesh sequence data, i.e., improving compression efficiency.

[0041] [Embodiment 2] The image processing device according to embodiment 1 generates independent textured mesh sequence data for each object and compresses the generated textured mesh sequence data for each object separately. In embodiment 2, an embodiment will be described in which colored point cloud (hereinafter referred to as "colored point cloud") data is separated for each object, and the colored point cloud data for each separated object is edited and compressed. Note that embodiment 2 will be described mainly in terms of differences from embodiment 1, and a description of similarities will be omitted.

[0042] 9 is a block diagram showing an example of the functional configuration of an image processing device 100 according to the second embodiment (hereinafter simply referred to as "image processing device 100"). The image processing device 100 has, as its functional configuration, a shape acquisition unit 902, an identification assignment unit 903, a sequence generation unit 904, a sequence modification unit 910, and a compression unit 906. The hardware configuration of the image processing device 100 is similar to that of the image processing device 100 according to the first embodiment, and therefore a description thereof will be omitted. Each unit that the image processing device 100 has as its functional configuration is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, etc., expanding the program into the RAM 103, and executing the program.

[0043] The shape acquisition unit 902 acquires colored point cloud sequence data composed of time-series colored point cloud data. The identification assignment unit 903 identifies a point cloud corresponding to each object for each piece of point cloud data based on information indicating the position and color of each point in each piece of colored point cloud data constituting the colored point cloud sequence data acquired by the shape acquisition unit 902. The identification assignment unit 903 also assigns identification information to the point cloud data corresponding to each identified object. The method of assigning the identification information may be the same as that of the first embodiment. That is, the identification assignment unit 903 assigns, for example, an object number corresponding to the object as identification information to the point cloud data corresponding to each object.

[0044] The sequence generation unit 904 generates independent time-series colored point cloud data for each object based on information indicating the position and color of each point in the time-series colored point cloud data acquired by the shape acquisition unit 902 and the identification information assigned by the identification assignment unit 903. That is, the sequence generation unit 904 generates colored point cloud sequence data for each object, consisting of time-series colored point cloud data for each object. The sequence modification unit 910 modifies the colored point cloud sequence data for each object in accordance with a user's editing operation, and outputs the modified colored point cloud sequence data to the compression unit 906. The compression unit 906 compresses the colored point cloud sequence data for each object to generate compressed data for each object. The compressed data generated by the compression unit 906 is output to the auxiliary storage device 104, etc.

[0045] 10 is a flowchart showing an example of a processing flow in the image processing apparatus 100 according to the second embodiment. Note that the processing of each step shown in the flowchart in FIG. 10 is realized by the CPU 101 reading a predetermined program from the ROM 102 or the auxiliary storage device 104, loading it into the RAM 103, and executing it. First, in S1001, the shape acquisition unit 902 acquires colored point cloud sequence data. Next, in S1002, the identification unit 903 identifies a point cloud corresponding to each object in each piece of time-series colored point cloud data constituting the colored point cloud sequence data acquired in S1001, i.e., a colored point cloud corresponding to each object. Next, in S1003, the identification unit 903 assigns identification information to the point cloud data corresponding to each object identified in S1002.

[0046] Next, in S1004, the sequence generation unit 904 generates colored point cloud sequence data for each object based on the colored point cloud sequence data acquired in S1001 and the identification information assigned in S1003. Specifically, the sequence generation unit 904 selects colored point cloud data for an arbitrary object and aligns the colored point cloud of the selected object with the colored point cloud to be aligned. Here, the alignment target point cloud is, for example, the colored point cloud corresponding to the object at the time closest to the time of the colored point cloud of the selected object. For example, the sequence generation unit 904 aligns the colored point cloud of the selected object so that the color and position of each point in the colored point cloud of the selected object are the same as the color and position of each point in the colored point cloud to be aligned after alignment. By performing such alignment, the sequence generation unit 904 can generate independent colored point cloud sequence data for each object.

[0047] Note that color information is attached to each point in the point cloud data acquired in S1001. Therefore, the sequence generation unit 904 can align the point clouds more accurately than if color information were not used. Note that the alignment of colored point clouds can be performed using the method disclosed in Reference 1 (Marcelo Saval-Calvo and five others, "3D non-rigid registration using color: Color Coherent Point Drift," [online], February 5, 2018, arXiv, [searched August 23, 2023], Internet: <https: / / arxiv.org / pdf / 1802.01516.pdf>).

[0048] After S1004, in S1005, the sequence modification unit 910 modifies the colored point cloud sequence data for each object generated in S1004 in accordance with the user's editing operation, thereby generating modified colored point cloud sequence data. The generated modified colored point cloud sequence data is output to the compression unit 906. Specific editing operations will be described later with reference to FIG. 11 . Next, in S1006, the compression unit 906 compresses the modified colored point cloud sequence data for each object generated in S1005, thereby generating compressed data for each object. The compressed data generated by the compression unit 906 is output to the auxiliary storage device 104, etc. After S1006, the image processing apparatus 100 ends the processing of the flowchart shown in FIG. 10 .

[0049] The modified colored point cloud sequence data for each object may be compressed using, for example, G-PCC (Geometry-based Point Cloud Compression), which is standardized by MPEG. The method for compressing the modified colored point cloud sequence data is not limited to using G-PCC, and other methods, such as V-PCC, may also be used to compress the modified colored point cloud sequence data. The compression unit 906 may also compress the modified colored point cloud sequence data using a method that utilizes three-dimensional time correlation, such as the following. Specifically, for example, the compression unit 906 first compresses point cloud data corresponding to key nodes in the modified colored point cloud sequence data using the Draco library. Next, the compression unit 906 compresses differences in information indicating coordinates and colors in the point cloud data within the GOM that includes the key nodes.

[0050] <Editing Operation by User> Fig. 11 is a diagram showing an example of an editing UI (user interface) 1100 for accepting editing operations of colored point cloud sequence data by a user according to the second embodiment. The editing UI has areas 1101 and 1102. Area 1101 is a screen area in which a colored point cloud for each object in a scene at a given time point is drawn. Area 1102 is a screen area in which a point cloud sequence managed for each object is drawn using a timeline. The user can draw a colored point cloud at a time point desired by the user in area 1101 by operating a seek bar on the timeline.

[0051] Shape 1103 represents 3D shape data that was erroneously generated when estimating the shape of the object with object number 2. For example, a user can delete unnecessary 3D shape data such as shape 1103 by selecting it in the editing UI 1100. Specifically, for example, the user selects shape 1103 in area 1101 by inputting using a mouse or the like, and then presses a delete button or the like (not shown in FIG. 11 ) to instruct the image processing apparatus 100 to delete the 3D shape data corresponding to shape 1103. Note that the selection and deletion of shape 1103 can also be performed on the timeline drawn in area 1102. In this case, for example, the user can delete the 3D shape data corresponding to shape 1103 by selecting the bar with object number 2 on the timeline and then pressing a delete button (not shown in FIG. 11 ).

[0052] Furthermore, in the editing UI 1100, the user can set attribute information of an object corresponding to a three-dimensional shape drawn in the editing UI 1100. Specifically, in the editing UI 1100, the user selects a three-dimensional shape or an object number corresponding to a desired object and sets a category, group, name, or the like of the object. A category is information representing the type of object, such as a person or an object. A group is information representing the affiliation of an object, such as information that can further classify the type of object set in a category. For example, if the subject of imaging is a team sport such as soccer, the group may be set to the name of the team to which the object belongs. A name is information that allows a person to identify the object, such as the unique name of the object. For example, if the category of the object is a person, the name may be set to the name of the object. Furthermore, if the subject of imaging is a team sport such as soccer, the name may be set to the uniform number of the player who is the object.

[0053] The image processing device 100 configured as described above can generate independent point cloud sequence data for each object from the point cloud sequence data. As a result, the image processing device 100 can efficiently compress the point cloud sequence data by compressing the point cloud sequence data for each object. Furthermore, the image processing device 100 configured as described above can edit a volumetric video represented by the point cloud sequence data for each object, and compress and output the edited point cloud sequence data.

[0054] The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0055] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0056] [Configuration of the Present Disclosure] The present disclosure includes the following configuration, method, and program.

[0057] <Configuration 1> An image processing device comprising: a shape acquisition means for acquiring three-dimensional shape sequence data including time-series three-dimensional shape data indicating the three-dimensional shapes of a plurality of objects; a separation means for, when the time-series three-dimensional shape data includes data of a combined shape in which the three-dimensional shapes of two or more of the plurality of objects are combined into a single three-dimensional shape, separating the combined shape data into object shape data that is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; a sequence generation means for generating, for each object, object shape sequence data that is three-dimensional shape sequence data including the time-series object shape data for each object; and a compression means for compressing the object shape sequence data for each object.

[0058] <Configuration 2> The image processing device according to Configuration 1, wherein the separating means separates the combined shape data into the object shape data corresponding to each of the two or more objects by transforming data of a single shape in which the three-dimensional shapes of the two or more objects are not combined with each other in three-dimensional shape data that is adjacent in time series to the three-dimensional shape data including the combined shape data.

[0059] <Configuration 3> The image processing device according to Configuration 1 or 2, characterized in that the three-dimensional shape sequence data is colored point cloud sequence data including, as the time-series three-dimensional shape data, time-series colored point cloud data corresponding to surfaces of the plurality of objects, the separating means separates the colored point cloud data into colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating a position of each point in the time-series colored point cloud data, and the sequence generating means generates, for each object, colored object point cloud sequence data including the time-series colored object point cloud data corresponding to each of the plurality of objects as the three-dimensional shape sequence data, based on the colored object point cloud data having the same number of points.

[0060] <Configuration 4> The image processing device according to Configuration 3, wherein the separating means separates the colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating a position of each point and information indicating a color of each point in the time-series colored point cloud data.

[0061] <Configuration 5> The image processing device according to configuration 1 or 2, wherein the three-dimensional shape sequence data is textured mesh sequence data including, as the time-series three-dimensional shape data, time-series textured mesh data indicating surface shapes of the plurality of objects, the separating means separates the textured mesh data into textured object mesh data corresponding to each of the plurality of objects by transforming the textured mesh data based on information indicating a position of each vertex in the time-series textured mesh data, and the sequence generating means generates, for each object, textured object mesh sequence data including the time-series textured object mesh data corresponding to each of the plurality of objects as the three-dimensional shape sequence data, based on the textured object mesh data having the same topology.

[0062] <Configuration 6> The image processing device according to any one of Configurations 1 to 5, further comprising: an image acquisition means for acquiring captured image data obtained by capturing images of the plurality of objects; and the shape acquisition means for estimating the three-dimensional shape sequence data using the acquired captured image data, thereby acquiring the three-dimensional shape sequence data.

[0063] <Configuration 7> The image processing device according to Configuration 6, wherein the shape acquisition means uses the captured image data to add color information to the three-dimensional shape sequence data.

[0064] <Configuration 8> The image processing device according to any one of Configurations 1 to 7, further comprising an identification assigning means for assigning identification information corresponding to each of the plurality of objects to each of the time-series three-dimensional shape data included in the three-dimensional shape sequence data.

[0065] <Configuration 9> The image processing device according to Configuration 8, wherein the sequence generating means generates the object shape sequence data for each object based on the assigned identification information.

[0066] <Configuration 10> The image processing device according to Configuration 8 or 9, wherein the separating means separates the combined shape data into the object shape data corresponding to each of the two or more objects based on the assigned identification information.

[0067] <Configuration 11> The image processing device according to any one of configurations 8 to 10, wherein the separating means comprises: a transformation path setting means for setting starting point shape data that serves as a starting point for transforming the three-dimensional shape data from the time-series three-dimensional shape data based on the assigned identification information, and a transformation path for transforming the three-dimensional shape data; and a transformation means for transforming the three-dimensional shape data in time-series order from the starting point three-dimensional shape data according to the transformation path based on the starting point shape data and the transformation path.

[0068] <Configuration 12> The image processing device according to Configuration 11, wherein the separating means sets the deformation path from the single shape data to the combined shape data when the identification information assigned to the combined shape data includes the identification information assigned to single shape data in which the three-dimensional shapes of the two or more objects that are adjacent in time series are not combined with each other.

[0069] <Configuration 13> The image processing apparatus according to any one of claims 1 to 12, further comprising a compression unit that compresses the object shape sequence data for each object.

[0070] <Configuration 14> The image processing device according to configuration 13, wherein the compression means compresses the object shape sequence data for each range in which starting object shape data serving as a starting point for transforming the object shape data is common.

[0071] <Configuration 15> The image processing device according to configuration 13 or 14, further comprising: a receiving means for receiving an editing operation for the object shape sequence data; and a sequence modifying means for modifying the object shape sequence data based on the editing operation to generate modified object shape sequence data, wherein the compressing means compresses the modified object shape sequence data in place of the object shape sequence data for which the modified object shape sequence data has been generated.

[0072] <Method> An image processing method comprising: a shape acquisition step of acquiring three-dimensional shape sequence data including time-series three-dimensional shape data indicating the three-dimensional shapes of a plurality of objects; a separation step of, when the time-series three-dimensional shape data includes combined shape data in which the three-dimensional shapes of two or more of the plurality of objects are combined into a single three-dimensional shape, separating the combined shape data into object shape data that is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; and a sequence generation step of generating, for each object, object shape sequence data that is three-dimensional shape sequence data including the time-series object shape data for each object.

[0073] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 15.

[0074] The present disclosure is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present disclosure. Therefore, the following claims are appended to apprise the public of the scope of the present disclosure.

[0075] This application claims priority based on Japanese Patent Application No. 2023-150880, filed on September 19, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: a shape acquisition means for acquiring three-dimensional shape sequence data including time-series three-dimensional shape data indicating the three-dimensional shapes of a plurality of objects; a separation means for, when the time-series three-dimensional shape data includes data of a combined shape in which the three-dimensional shapes of two or more of the plurality of objects are combined into a single three-dimensional shape, separating the combined shape data into object shape data that is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; and a sequence generation means for generating, for each object, object shape sequence data that is three-dimensional shape sequence data including the time-series object shape data for each object.

2. The image processing device according to claim 1, characterized in that the separation means separates the combined shape data into the object shape data corresponding to each of the two or more objects by transforming single shape data in three-dimensional shape data that is adjacent in time series to the three-dimensional shape data including the combined shape data, in which the three-dimensional shapes of the two or more objects are not combined with each other.

3. The image processing device according to claim 1, wherein the three-dimensional shape sequence data is colored point cloud sequence data including time-series colored point cloud data corresponding to the surfaces of the plurality of objects as the time-series three-dimensional shape data, the separation means separates the colored point cloud data into colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating the position of each point in the time-series colored point cloud data, and the sequence generation means generates colored object point cloud sequence data including the time-series colored object point cloud data corresponding to each of the plurality of objects as the three-dimensional shape sequence data for each object based on the colored object point cloud data having the same number of points.

4. The image processing device of claim 3, wherein the separation means separates the colored object point cloud data corresponding to each of the plurality of objects by transforming the colored point cloud data based on information indicating the position of each point and information indicating the color of each point in the time series colored point cloud data.

5. The image processing device of claim 1, wherein the three-dimensional shape sequence data is textured mesh sequence data including time-series textured mesh data indicating surface shapes of the multiple objects as the time-series three-dimensional shape data, the separation means separates the textured mesh data into textured object mesh data corresponding to each of the multiple objects by transforming the textured mesh data based on information indicating the position of each vertex in the time-series textured mesh data, and the sequence generation means generates textured object mesh sequence data for each object as the three-dimensional shape sequence data, including the time-series textured object mesh data corresponding to each of the multiple objects, based on the textured object mesh data that has a matching topology.

6. The image processing device according to claim 1, further comprising an image acquisition means for acquiring captured image data obtained by imaging the plurality of objects, wherein the shape acquisition means acquires the three-dimensional shape sequence data by estimating the three-dimensional shape sequence data using the acquired captured image data.

7. The image processing device according to claim 6, wherein said shape acquisition means uses said captured image data to add color information to said three-dimensional shape sequence data.

8. The image processing device according to claim 1, further comprising an identification providing means for providing identification information corresponding to each of the plurality of objects to each of the time-series three-dimensional shape data included in the three-dimensional shape sequence data.

9. The image processing apparatus according to claim 8, wherein said sequence generating means generates said object shape sequence data for each object based on said assigned identification information.

10. The image processing device according to claim 8, wherein said separating means separates said combined shape data into said object shape data corresponding to each of said two or more objects based on said assigned identification information.

11. The image processing device of claim 8, wherein the separation means comprises: a transformation path setting means for setting starting point shape data that serves as a starting point for transforming three-dimensional shape data from among the time-series three-dimensional shape data based on the assigned identification information, and a transformation path for transforming the three-dimensional shape data; and a transformation means for transforming the three-dimensional shape data in chronological order according to the transformation path from the starting point three-dimensional shape data based on the starting point shape data and the transformation path.

12. The image processing device according to claim 11, characterized in that the separation means sets the deformation path from the single shape data to the combined shape data when the identification information assigned to the combined shape data includes identification information assigned to single shape data in which the three-dimensional shapes of the two or more objects that are adjacent in time series and are not combined with each other.

13. The image processing apparatus according to claim 1, further comprising a compression means for compressing the object shape sequence data for each object.

14. The image processing device according to claim 13, wherein said compression means compresses said object shape sequence data for each range in which starting object shape data serving as a starting point for transforming said object shape data is common.

15. An image processing device as described in claim 13, further comprising: a receiving means for receiving an editing operation of the object shape sequence data; and a sequence modification means for modifying the object shape sequence data based on the editing operation to generate modified object shape sequence data, wherein, for the object shape sequence data from which the modified object shape sequence data has been generated, the compression means compresses the modified object shape sequence data in place of the object shape sequence data.

16. An image processing method comprising: a shape acquisition step of acquiring three-dimensional shape sequence data including time-series three-dimensional shape data indicating the three-dimensional shapes of a plurality of objects; a separation step of, when the time-series three-dimensional shape data includes combined shape data in which the three-dimensional shapes of two or more of the plurality of objects are combined into a single three-dimensional shape, separating the combined shape data into object shape data that is three-dimensional shape data corresponding to each of the two or more objects, and generating the object shape data corresponding to each of the plurality of objects; and a sequence generation step of generating, for each object, object shape sequence data that is three-dimensional shape sequence data including the time-series object shape data for each object.

17. A program for causing a computer to function as the image processing device according to any one of claims 1 to 15.

Images