Image processing apparatus, image processing method, and program

Abstract

To reduce a shape difference between tracks in tracking processing in a 3D model.SOLUTION: Tracking processing is performed on time-series shape data composed of a frame group that includes a 3D model representing a three-dimensional shape of an object, such that there occurs an overlapping section between adjacent tracks. Then, based on results of the tracking processing, tracked time-series shape data without the overlapping section between the adjacent tracks, using any location within the overlapping section as a track boundary, is output.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to a technique for tracking three-dimensional shape data of an object.

Background Art

[0002] In recent years, in the field of video production, volumetric video technology that reconstructs data representing the three-dimensional shape of an object (subject) (generally referred to as a "3D model") in a three-dimensional space and adds CG effects to visualize it from a free viewpoint has been becoming common. Here, as preprocessing for data compression when transferring the generated 3D model data to a volumetric video generation device or the like, tracking processing of the 3D model is performed. This tracking processing is a technique for associating each component of the three-dimensional shape represented by the 3D model between frames in a series of frame groups constituting a video.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, in the case of tracking processing for a 3D model in mesh format, the common topology is maintained within the tracking range (the frame interval in which the vertices of the polygons constituting the mesh are being tracked. Hereinafter referred to as "track"). Therefore, by moving the vertex positions of the polygons constituting the mesh, it is possible to realize a smooth shape change representation in each frame within the track. However, at the part where the track switches, the common topology is not maintained. Also, even if other tracks are tracking the same correct shape as the own track, the shapes do not completely match, and a shape difference occurs between the tracks. In particular, when performing a video expression that depends on the shape, such as rendering with relighting processing, the shape change at the track switch appears as a sharp change in the video, giving the viewer a sense of discomfort. Figures 17(a) to (d) are diagrams showing an example where the video jitters at the track switch part, and show each state when time progresses from the frame of (a) to the frame of (d). In this example, the track switches between (c) and (d) ((the frames of (a) and (b) belong to the same track, and the frames of (c) and (d) belong to the same track).). It can be seen that the shape change of the back part of the person is particularly large before and after the track switch.

[0005] The present disclosure has been made to solve the above-described problems, and an object thereof is to reduce the shape difference between tracks in the tracking process of a 3D model.

Means for Solving the Problems

[0006] The image processing apparatus according to the present disclosure includes: an acquisition unit that acquires time-series shape data including a frame group including a 3D model representing a three-dimensional shape of an object; a tracking unit that performs tracking processing on the time-series shape data so that an overlapping section occurs between adjacent tracks; and an output unit that outputs tracked time-series shape data without an overlapping section between adjacent tracks, with any position within the overlapping section as a track boundary, based on a result of the tracking processing, where the track indicates a frame section in which components of the 3D model are tracked between frames and a common topology is maintained.

Advantages of the Invention

[0007] According to the present disclosure, it is possible to reduce a shape difference between tracks in tracking processing of a 3D model.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Mode for Carrying Out the Invention

[0009] Hereinafter, the mode for carrying out the present embodiment will be described with reference to the drawings and the like. Note that the following embodiments do not limit the technology of the present disclosure, and not all the configurations described in the following embodiments are essential means for solving the problems.

[0010] (Definition of Terms) In this specification, "object" means a three-dimensional object such as a person. Also, "point" means an element for representing the three-dimensional shape of an object, which is indicated by a single coordinate in three-dimensional space, and "point cloud format" refers to the data format of a 3D model that represents the surface position of an object as a set of one or more points. Also, "polygon" means a polygonal surface with three or more points as vertices, and "mesh format" refers to the data format of a 3D model that represents the surface shape of an object as a set of polygons. Also, the data of a frame group including a plurality of 3D models showing the three-dimensional shapes of a plurality of objects in each of a plurality of consecutive frames, which is the object of the tracking process, will be referred to as "continuous shape data" or "time-series shape data". Among the continuous shape data, those in which the 3D model is represented in the mesh format will be referred to as "continuous mesh data", and those represented in the point cloud format will be referred to as "continuous point cloud data".

[0011] [Embodiment 1] In this embodiment, first, tracking processing is performed on continuous mesh data so that overlapping sections occur between tracks, and a track boundary is determined based on the similarity of the tracked meshes in both overlapping tracks. This mode will be described below.

[0012] <Logical Configuration of Image Processing Apparatus> FIG. 1 is a functional block diagram showing the logical configuration (software configuration) of the image processing apparatus according to this embodiment. The image processing apparatus 100 includes a data acquisition unit 101, a tracking unit 102, a boundary determination unit 103, a shape generation unit 104, and a data output unit 105. Each functional unit will be described below.

[0013] The data acquisition unit 101 acquires continuous mesh data to be subjected to tracking processing. In this case, when each frame constituting the continuous mesh data includes a plurality of 3D models in mesh format, it is assumed to include identification information (object ID) capable of identifying an object corresponding to each 3D model.

[0014] The tracking unit 102 performs tracking processing for each object on the continuous mesh data acquired by the data acquisition unit 101. At this time, first, tracking is performed so that each track overlaps with a part of another adjacent track, that is, an overlapping section occurs between adjacent tracks, and then the track boundary is determined. In the case of a 3D model in mesh format, a method (see Patent Document 1) is adopted in which the vertices of the polygons, which are the components of the mesh of the frame (hereinafter referred to as the key frame) at which tracking starts, are tracked between frames, and the vertex indices are shared between frames. By sharing the index information of the mesh between frames, the continuous mesh data can be expressed in the form of "key frame + difference".

[0015] The boundary determination unit 103 determines the boundaries between two adjacent tracks based on the similarity of the tracked meshes in the overlapping intervals generated by the above tracking process. As a result, one track to which each frame within the overlapping interval should belong is determined, and one of the tracks is assigned to each frame constituting the input continuous mesh data. Hereinafter, in this specification, the track in the state where the overlapping interval immediately after the tracking process occurs is referred to as a "provisional track", and the track in the state where one track is assigned to each frame based on the determined boundary is referred to as an "output track".

[0016] The shape generation unit 104 selects, from the tracked meshes obtained by the tracking process, the output meshes associated with each frame belonging to the output track. Thereby, the tracked continuous mesh data is generated.

[0017] The data output unit 105 outputs the tracked continuous mesh data generated by the shape generation unit 104.

[0018] <Hardware Configuration of Image Processing Apparatus> FIG. 2 is a block diagram showing the hardware configuration of the image processing apparatus according to the present embodiment. As hardware included in a general computer, the image processing apparatus 100 has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication unit 207, and a bus 208, as shown as an example in FIG. 2.

[0019] The CPU 201 realizes each functional unit included in the image processing apparatus 100 by using programs or data stored in the ROM 202 or the RAM 203. Note that the image processing apparatus 100 may have one or more dedicated hardware different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC, an FPGA, and a DSP (Digital Signal Processor). The ROM 202 stores programs and the like that do not require modification. The RAM 203 temporarily stores programs or data supplied from the auxiliary storage device 204, or data and the like supplied from the outside via the communication unit 207. The auxiliary storage device 204 is configured by, for example, a hard disk drive, and stores various data such as image data or audio data.

[0020] The display unit 205 is configured by, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like for the user to operate the image processing apparatus 100 or view necessary information. The operation unit 206 is configured by, for example, a keyboard, a mouse, or a touch panel, and receives operations by the user and inputs various instructions to the CPU 201. The CPU 201 also operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206. The communication unit 207 is used for communication with a device outside the image processing apparatus 100. For example, when the image processing apparatus 100 is wired-connected to an external device, a communication cable is connected to the communication unit 207. When the image processing apparatus 100 has a function of wireless communication with an external device, the communication unit 207 includes an antenna. The bus 208 connects each unit included in the image processing apparatus 100 and transmits information.

[0021] In the present embodiment, the display unit 205 and the operation unit 206 are described as existing inside the image processing apparatus 100, but at least one of the display unit 205 and the operation unit 206 may exist outside the image processing apparatus 100 as another device.

[0022] <Operation of the image processing apparatus 100> Figure 3 is a flowchart showing the flow of operations of the image processing apparatus 100 according to the present embodiment. Figure 4 is a diagram for explaining the process of tracking processing in the present embodiment, and schematically shows a state in which an overlapping section occurs between two adjacent tracks. Hereinafter, with reference to FIGS. 3 and 4, the operations of the image processing apparatus 100 according to the present embodiment will be described. In the following description, the symbol "S" means step.

[0023] In S301, the data acquisition unit 101 acquires continuous mesh data from an external PC or the like. This continuous mesh data is a set of data in which mesh-form 3D models representing the three-dimensional shape of the object, which are obtained by measuring the object over time, are arranged in time series for each frame. "Input mesh" in FIG. 4 indicates continuous mesh data for 12 frames acquired as a processing target.

[0024] In S302, the tracking unit 102 performs tracking processing on the continuous mesh data acquired in S301 for each object such that an overlapping section occurs between adjacent tracks. FIG. 5 is a flowchart showing details of the tracking processing. Here, it will be described in detail with reference to the flow of FIG. 5.

[0025] ≪Details of Tracking Processing≫ In S501, a key frame that serves as a reference when performing tracking on the frame group constituting the continuous mesh data is set. The key frames are set at a predetermined interval with respect to the frame group constituting the continuous mesh data. Alternatively, a specific frame determined based on a predetermined evaluation value such as the amount of movement or surface area of the mesh associated with each frame may be set as the key frame. The mesh associated with the frame set as the key frame will be referred to as "key mesh" for convenience. In the example of FIG. 4 described above, the 3rd, 8th, and 11th frames out of all 12 frames are set as key frames, respectively, and the meshes associated with these three frames are "key meshes".

[0026] In S502, the key mesh associated with the key frame of interest among the key frames set in S501 is set as the base mesh, and each of the meshes associated with the frames before and after the key frame of interest is set as the target mesh. Here, the reason for the frames before and after is that in this embodiment, tracking processing is performed in both the time series direction and the reverse time series direction with respect to the key frame as a reference. The processing after S503 is executed in parallel or in order for each of the time series direction and the reverse time series direction.

[0027] In S503, a process of deforming the base mesh is performed so that the difference from the surface structure of the target mesh becomes small. Here, as a specific method of deforming the base mesh toward the target mesh, for example, the ICP (Iterative Closest Point) method can be used. Thereby, a tracked mesh in which the shape of the base mesh is deformed so that the error from the shape of the mesh associated with the frames before and after the key frame of interest is reduced while maintaining the topology of the key mesh is obtained.

[0028] In S504, it is determined whether the next adjacent frame is a key frame or a frame end, and the next process is allocated. Here, the "adjacent frame" will be explained. In the example of FIG. 4 described above, assume that the key frame of interest is the third frame. In the case of tracking in the time series direction, since the fourth frame advanced in time is processed in S502 as the "frame before and after", each frame after the fifth frame becomes the "adjacent frame" in this case. On the other hand, in the case of tracking in the reverse time series direction, since the second frame that has returned in time is processed in S502 as the "frame before and after", the first frame becomes the "adjacent frame" in this case. When the determined adjacent frame is a key frame or a frame end, the process of S506 is executed next, and when it is not a key frame or a frame end, the process of S505 is executed next. Note that the "frame end" means the first frame or the last frame in the group of frames constituting the continuous mesh data.

[0029] In S505, the tracked mesh obtained in S503 is set as the base mesh, and further, the mesh associated with the next adjacent frame is set as the target mesh. When the base mesh and the target mesh are updated in this way, the process returns to S503 and the same process is repeated.

[0030] In S506, it is determined whether or not the above-described process has been completed for all the key frames set in S501. As a result of the determination, if there is an unprocessed key frame, the process returns to S502 to determine the next key frame of interest and the process continues. On the other hand, if the process has been completed for all the key frames, the process exits and returns to the process of the flowchart in FIG. 3.

[0031] The above is the content of the tracking process. Thus, for example, in the example of FIG. 4, it is possible to acquire tracked continuous mesh data composed of tentative tracks 1 to 3. In the example shown in FIG. 4, there is an overlap in the section of the 4th to 7th frames between tentative track 1 and tentative track 2, and an overlap in the section of the 9th and 10th frames between tentative track 2 and tentative track 3. That is, in the tentative tracked continuous mesh data, each frame in the overlapping section belongs to two tracks. Note that in S502, instead of the key mesh associated with the key frame, a simplified mesh obtained by simplifying the key mesh may be set as the target mesh. Thereby, the process can be speeded up. Return to the description of the flowchart in FIG. 3.

[0032] In S303, the boundary determination unit 103 performs a process of determining a track boundary for obtaining an output track without an overlapping section between adjacent tracks based on the shape difference between the tentative tracks with respect to the tentative tracked continuous mesh data obtained in S302. FIG. 6 is a flowchart showing details of the track boundary determination process according to the present embodiment. Here, a detailed description will be given with reference to the flow of FIG. 6.

[0033] ≪Details of Track Boundary Determination Process≫ In S601, for the overlapping intervals of interest among the overlapping intervals included in the provisional tracked continuous mesh data to be processed, the similarity of the tracked meshes between the provisional tracks is calculated for each frame. In the example of FIG. 4, assume that the overlapping interval of interest is the 4th to 7th frames. In this case, the similarity between the tracked mesh associated with provisional track 1 and the tracked mesh associated with provisional track 2 is calculated for each of the 4th to 7th frames. Here, the similarity may be an evaluation value such as the Hausdorff distance based on the shape represented by the mesh, or an evaluation value such as the cosine similarity based on the normal of the mesh.

[0034] In S602, based on the mesh similarity calculated for each frame for the overlapping interval of interest, frame pairs with high mesh similarity are identified. This frame pair consists of two adjacent frames. For example, a combination of the frame with the highest mesh similarity among each frame included in the overlapping interval and the frame with the higher mesh similarity among the two adjacent frames of the said frame is identified as the frame pair with high mesh similarity. Alternatively, a combination in which the sum of the mesh similarities in two adjacent frames among each frame included in the overlapping interval is maximized may be identified as the frame pair with high mesh similarity. FIG. 7 is a diagram for explaining the state of the track boundary determination process when the overlapping interval of interest is the 4th to 7th frames. The central graph in FIG. 7 shows the mesh similarity calculated for each of the 4th to 7th frames. In this example, the combination of the 5th frame and the 6th frame is identified as the frame pair with high mesh similarity.

[0035] In S603, using the track boundary between the frame pairs identified in S602, a process is performed to eliminate the overlapping state in the two provisional tracks related to the overlapping section of interest. As a result, the two provisional tracks related to the overlapping section of interest are each changed to output tracks without an overlapping state. In the example of FIG. 7 described above, since the boundary between the 5th frame and the 6th frame is the track boundary, it is shown that up to the 5th frame becomes "Output Track 1" and after the 6th frame becomes "Output Track 2".

[0036] In S604, it is determined whether or not the above-described process has been completed for all overlapping sections included in the provisional tracked continuous mesh data. As a result of the determination, if there is an unprocessed overlapping section, the process returns to S601 to determine the next overlapping section and the same process continues. On the other hand, if the process has been completed for all overlapping sections, this process is exited and the process returns to the flowchart in FIG. 3.

[0037] The above is the content of the track boundary determination process. As a result, each frame included in the tracked continuous mesh data is changed to a data format in which it belongs to one track. However, at this stage, only the track boundary is determined and the tracked mesh associated with each frame is not determined, so in subsequent S304, the tracked mesh is associated with each frame. Return to the description of the flowchart in FIG. 3.

[0038] In S304, the shape generation unit 104 performs a process of selecting a tracked mesh that becomes a mesh corresponding to each frame constituting the output track (hereinafter referred to as "output mesh"). Specifically, for each frame included in the overlapping section, a process of selecting the tracked mesh in the provisional track that has been adopted for the output track as the output mesh is performed. FIG. 8 is a diagram for explaining a state in which a tracked mesh as an output mesh is selected in the example of FIG. 7 described above. For the 4th and 5th frames of the output track 1, the tracked meshes associated with the 4th and 5th frames in the provisional track 1 are respectively selected as the output meshes. Also, for the 6th and 7th frames of the output track 2, the tracked meshes associated with the 6th and 7th frames in the provisional track 2 are respectively selected as the output meshes. Note that for key frames, the already associated key meshes are maintained as the output meshes as they are.

[0039] In S305, the data output unit 105 outputs the tracked continuous mesh data in which the output mesh is associated with each frame of the output track obtained in the processing so far, together with the track information indicating the configuration of the output track. The above is the flow of operations in the image processing apparatus 100 according to the present embodiment.

[0040] <Modification Example> In the tracking process (S302) of this embodiment, the tracking is performed with one provisional track from one key frame to the next key frame, but it is not limited to this. By tracking in two directions, forward and backward in time series, an overlapping section may be generated between adjacent tracks. That is, the tracking may be performed in two directions by a number of frames exceeding half of the interval between two consecutive key frames. For example, if the key frame interval is 5 frames, tracking may be performed for 3 or more frames, and if it is 6 frames, tracking may be performed for 4 or more frames. Also, the tracking process may be tracking in any one direction. When performing tracking in one direction, the tracking may be performed up to several frames ahead beyond the next key frame. That is, in S504 of the flow of FIG. 5 described above, it may be determined as Yes when the next adjacent frame is "several frames ahead or the frame end of the next key frame". Thereby, an overlapping section can be generated between adjacent tracks even in one-directional tracking.

[0041] As described above, according to this embodiment, the tracking process is performed so that an overlapping section occurs between adjacent tracks for the input continuous shape data, and the boundary of the output track without an overlapping section is determined based on the similarity of the tracked shapes between the provisional tracks. Thereby, the change in the object shape at the time of track switching is suppressed, and the rendering video can be smoothly transitioned between frames.

[0042] [Embodiment 2] In the method of Embodiment 1, when the similarity of the tracked mesh is low throughout the overlapping section, there remains a problem that the shape difference of the output mesh does not sufficiently decrease before and after the frame at which the track switches, and a sharp change in the video cannot be sufficiently suppressed. Therefore, an aspect of correcting the tracked mesh selected as the output mesh so that the shape difference of the output mesh sufficiently decreases before and after the frame at which the track switches will be described as Embodiment 2. Note that descriptions of the contents common to Embodiment 1, such as the logical configuration and hardware configuration of the image processing apparatus, will be omitted, and the following description will focus on the differences.

[0043] FIG. 9 is a flowchart showing the flow of operations of the image processing apparatus 100 according to the present embodiment. Hereinafter, with reference to FIG. 9, the operations of the image processing apparatus 100 according to the present embodiment will be described. In the following description, the symbol "S" means step.

[0044] Since S901 to S904 respectively correspond to S301 to S304 in the flowchart of FIG. 3 of Embodiment 1, detailed description thereof will be omitted. By the processing so far, each frame included in the provisional tracked continuous mesh data has been changed to a data format belonging to one track. In the subsequent steps, processing for correcting the tracked mesh selected as the output mesh for each frame is performed in order to make the mesh shape transition more smoothly. FIG. 10 is a diagram for explaining how the processing after S905 is applied when the continuous mesh data for 12 frames shown in FIG. 4 is the processing target and the overlapping section of interest is the 4th to 7th frames.

[0045] In S905, the shape generation unit 104 calculates a signed distance field (SDF) for the tracked mesh in each provisional track. For example, a known method disclosed in Patent Document 2 or the like can be used for calculating this SDF. As an example, FIG. 11 shows an SDF 1103 calculated from the tracked mesh 1101 of the provisional track 1 and an SDF 1104 calculated from the tracked mesh 1102 of the provisional track 2 in FIG. 10. Here, for the sake of convenience, the figure shows the tracked mesh and the SDF in two dimensions, but the actual tracked mesh and SDF have three-dimensional information.

[0046] In S906, the shape generation unit 104 calculates an intermediate SDF (hereinafter referred to as "intermediate SDF") for each frame included in the overlapping interval based on the SDF calculated in S905. In the example of FIG. 10 described above, in each of the 4th to 7th frames, the intermediate SDF between the SDF 1103 of the tracked mesh 1101 of the tentative track 1 and the SDF 1104 of the tracked mesh 1102 of the tentative track 2 is calculated by taking the weighted average of both SDFs. FIG. 12 is a diagram for explaining the weights used when calculating the intermediate SDF for each frame in the overlapping interval between the tentative track 1 and the tentative track 2 in FIG. 10. Now, when the weight for the tentative track 1 is w1 and the weight for the tentative track 2 is w2, w1 and w2 are set to values corresponding to the number of frames from each frame to the adjacent key frame as shown in FIG. 12, respectively. For example, in the example of FIG. 10, the 3rd and 8th frames are key frames, and the interval between them is 5 frames. For the 4th frame, the tentative track 1 is 4 frames away from the adjacent key frame, and the tentative track 2 is 1 frame away from the adjacent key frame. Therefore, the weights at this time are determined as w1 = 4 / 5 and weight w2 = 1 / 5, respectively. In this way, by linearly changing the weights according to the number of frames to the adjacent key frame, the mesh shape can be smoothly transitioned between tracks. Thus, as shown in FIG. 11, the intermediate SDF 1105 is obtained by taking the weighted average by applying the weight w1 = 4 / 5 to the SDF 1103 and the weight w2 = 1 / 5 to the SDF 1104. Here, as an example, w1 and w2 are determined with reference to the number of frames from the key frame, but the determination method is not limited to this, and it is only necessary that w1 + w2 = 1.0 is maintained. For example, w1 = 0.5 and w2 = 0.5 may be used for all frames, or it may be determined based on a function having an inflection point such as a sigmoid function.

[0047] In S907, the shape generation unit 104 corrects the tracked mesh selected as the output mesh for each frame based on the intermediate SDF calculated for each frame in S906. As the content of the correction, for example, as shown in Fig. 13(a), there is a method of correcting the position and normal of the polygon vertices (arrows attached to the triangular polygons) while maintaining the mesh topology. In the case of this method, first, the polygon vertices are moved to the points where the SDF value becomes 0 based on the local value of the intermediate SDF, and the normal is changed based on the local slope of the SDF of the moved coordinates. In the example of Fig. 11, a correction example is shown in which the vertex coordinates (white circles) and normals (arrows) of the polygons constituting the tracked mesh 1102 of the tentative track 2 are changed based on the calculated intermediate SDF 1105. In this way, the tracked mesh selected as the output mesh in S904 is corrected. Note that, as in the first embodiment, the key mesh already associated with the key frame is maintained as the output mesh as it is.

[0048] In S908, the data output unit 105 outputs the tracked continuous mesh data in which the corrected tracked mesh is associated as the output mesh for each frame of each output track obtained by the processing so far, together with the track information. In the example of Fig. 10, the key mesh is associated as the output mesh for the third frame belonging to output track 1, and the corrected meshes of the tracked meshes of the tentative track 1 are associated as the output meshes for the fourth and fifth frames, respectively. And for the sixth and seventh frames belonging to output track 2, the corrected meshes of the tracked meshes of the tentative track 2 are associated as the output meshes, and for the eighth frame, the key mesh is associated as the output mesh. The above is the flow of the operation in the image processing apparatus 100 according to the present embodiment.

[0049] <Modification Example 1> In this embodiment, the track boundary is determined based on the mesh similarity between provisional tracks in the same manner as in Embodiment 1, and then the SDF of the tracked mesh is calculated, but it is not limited to this. For example, the SDF of each tracked mesh may be calculated prior to the determination of the track boundary, and the track boundary may be determined based on the similarity of the obtained SDFs.

[0050] <Modification Example 2> In this embodiment, in the correction of the tracked mesh, a correction example of changing the vertex coordinates and normal vectors of the polygon has been described, but it is not limited to this. For example, as shown in FIG. 13(b), by correcting only the normal vector, it is possible to suppress a sharp change in luminance due to relighting. Further, as shown in FIG. 13(c), the correction of changing only the vertex coordinates is an effective correction method when using mesh data in a format that does not have normal vector information (that is, a format composed only of the vertex coordinate information and topology information of the polygon).

[0051] As described above, according to this embodiment, by correcting the tracked mesh selected as the output mesh, it is possible to further reduce the shape difference of the meshes between the output tracks.

[0052] [Embodiment 3] In recent years, colored point clouds have been increasingly used as 3D models of objects. Also in the tracking of continuous point cloud data, as in the case of continuous mesh data, a sharp movement in the rendered video due to the shape difference between tracks causes a sense of discomfort. Therefore, Embodiment 3 will describe an aspect of obtaining tracking results equivalent to those of Embodiments 1 and 2 for continuous point cloud data. Note that descriptions of the contents common to Embodiments 1 and 2, such as the logical configuration and hardware configuration of the image processing apparatus, will be omitted, and the following will focus on the differences.

[0053] FIG. 14 is a flowchart showing the flow of operations of the image processing apparatus 100 according to the present embodiment. Hereinafter, with reference to FIG. 14, the operations of the image processing apparatus 100 according to the present embodiment will be described. In the following description, the symbol "S" means step.

[0054] S1401 to S1402 respectively correspond to S301 and S302 in the flowchart of FIG. 3 in Embodiment 1. Basically, it suffices to read "mesh" as "point cloud", so detailed description thereof will be omitted. By the processing up to this point, provisional tracked continuous point cloud data has been obtained. In the subsequent steps, determination of track boundaries for more smoothly transitioning the tracked point cloud shape between tracks and processing for correcting the tracked point cloud selected as the output point cloud in each frame are performed. FIG. 15 is a diagram for explaining how the processing after S1403 is applied when the overlapping section of interest is the 4th to 7th frames when the 12-frame continuous mesh data shown in FIG. 4 is replaced with 12-frame continuous point cloud data.

[0055] In S1403, the boundary determination unit 103 performs a process of determining a track boundary for the provisional tracked continuous point group data obtained in S1402 in order to obtain an output track without an overlapping section. Specifically, first, the amount of shape change between frames of the input point group in the overlapping section is obtained. Then, the frames between which the obtained amount of shape change is the largest are determined as the track boundary. By switching the track between the frames with a large amount of shape change in this way, the shape difference between the tracks can be hidden in the movement of a larger object. Conversely, it is also possible to determine the track boundary so that the track is switched between the frames with the smallest amount of shape change. A small amount of shape change means that tracking failure is less likely to occur, leading to a reduction in the shape difference between the tracks. In this way, the data format is changed so that each frame included in the provisional tracked continuous point group data belongs to one track. However, at this stage, since the tracked point group associated with each frame has not been determined, the tracked point group is first associated with each frame in the next S1404.

[0056] In S1404, the shape generation unit 104 performs a process of selecting a tracked point group that becomes a point group (hereinafter referred to as "output point group") corresponding to each frame constituting the output track. Specifically, for each frame included in the overlapping section, the tracked point group in the provisional track adopted for the output track is selected as the output point group. In the example of FIG. 15, for the 4th and 5th frames of output track 1, the tracked point groups associated with the 4th and 5th frames in provisional track 1 are selected as the output point groups, respectively. Also, for the 6th and 7th frames of output track 2, the tracked point groups associated with the 6th and 7th frames in provisional track 2 are selected as the output point groups, respectively. For key frames, the already associated key point groups are maintained as the output point groups as they are.

[0057] In S1405, the shape generation unit 104 performs a nearest neighbor search from each point (corresponding to the vertex of the mesh) of the tracked point cloud selected as the output point cloud in the output track to the tracked point cloud of the other tentative track for each frame. As a result, the coordinate information and normal information of the nearest neighbor point in the tracked point cloud of the other tentative track corresponding to each point of the tracked point cloud of the output track can be obtained. As an example, FIG. 16 shows the result 1603 of performing a nearest neighbor search from each point of the tracked point cloud 1601 of the tentative track 1 related to the same frame to the tracked point cloud 1602 of the tentative track 2. For the sake of convenience, similar to FIG. 11 described above, the figure shows the tracked point cloud and the result of the nearest neighbor search in two dimensions, but the actual tracked point cloud and the result of the nearest neighbor search have three-dimensional information.

[0058] In S1406, the shape generation unit 104 corrects the tracked point cloud selected as the output point cloud by weighted interpolation based on the result of the nearest neighbor search obtained in S1405. Specifically, for each frame, a weight w f is used to obtain the three-dimensional information of the intermediate point by interpolation between each point of the selected tracked point cloud and the corresponding nearest neighbor point. Here, the weight w f is desirably changed linearly according to the number of frames to the adjacent key frame, similar to the weight when obtaining the intermediate SDF in Embodiment 2. As a result, a point (interpolation point) with intermediate coordinates and normal, which can smoothly transition the shapes of the two tracks, can be obtained, and a corrected point cloud 1606 as shown in FIG. 16 can be obtained.

[0059] In S1407, the data output unit 105 outputs, for each frame of each output track, the tracked continuous point cloud data in which the corrected tracked point cloud is associated as the output point cloud, together with the track information, obtained by the processing so far. In the example of FIG. 15, for the third frame belonging to output track 1, the key point cloud is associated as the output point cloud, and for the fourth and fifth frames, the corrected point clouds of the tracked point cloud of tentative track 1 are respectively associated as the output point clouds. And for the sixth and seventh frames belonging to output track 2, the corrected point clouds of the tracked point cloud of tentative track 2 are associated as the output point clouds, and for the eighth frame, the key point cloud is associated as the output point cloud.

[0060] The above is the flow of operations in the image processing apparatus 100 according to the present embodiment. In the above-described S1406, an example of interpolating the coordinates and normal information of each point constituting the point cloud was described. However, when information other than these is given to each point, those pieces of information may be interpolated. For example, when the input continuous point cloud data is colored point cloud data having color information for each point, information such as color and transparency such as RGB values and α values given to each point may be interpolated. Or, if information on the size and deformation amount (distortion amount) for each point is given as information applied at the time of rendering, the information on the size and deformation amount may be interpolated. Also, in the present embodiment, an example of correcting the output point cloud by interpolation processing based on the result of nearest neighbor point search has been described in the case where the data format of the 3D model is the point cloud format, but it is similarly applicable in the case of the mesh format. That is, instead of the SDF described above, the mesh shape can also be corrected by interpolation processing with the nearest neighbor surface or nearest neighbor point.

[0061] As described above, according to the present embodiment, by correcting the tracked point cloud selected as the output point cloud, it is possible to reduce the shape difference of the point cloud between the output tracks more than in Embodiment 2.

[0062] (Other Embodiments) The present disclosure can also be implemented by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be implemented by a circuit (for example, ASIC) that realizes one or more functions.

[0063] Furthermore, the present disclosure includes the following configurations and methods.

[0064] [Configuration 1] Acquisition means for acquiring time-series shape data composed of a group of frames including a 3D model representing the three-dimensional shape of an object, Tracking means for performing tracking processing on the time-series shape data so that an overlapping section occurs between adjacent tracks, Output means for outputting tracked time-series shape data without an overlapping section between adjacent tracks, with any position within the overlapping section as a track boundary, based on the result of the tracking processing, having wherein the track indicates a frame section in which components of the 3D model are tracked between frames and a common topology is maintained, An image processing apparatus characterized by the above.

[0065] [Configuration 2] The tracking means performs tracking processing in two directions, the time-series direction and the reverse time-series direction, based on a plurality of key frames set for the group of frames, An image processing apparatus according to Configuration 1, characterized by the above.

[0066] [Configuration 3] The tracking means performs tracking processing in the two directions by a number of frames exceeding half of the interval between adjacent key frames. An image processing apparatus according to Configuration 2, characterized by the above.

[0067] [Configuration 4] The tracking means performs tracking processing in the two directions with one track from a certain key frame to the next key frame, and the image processing apparatus according to Configuration 3, characterized in that.

[0068] [Configuration 5] The tracking means Performs tracking processing in either the time series direction or the reverse time series direction based on a plurality of key frames set for the frame group, And the image processing apparatus according to Configuration 1, characterized in that.

[0069] [Configuration 6] The tracking means performs the one-directional tracking processing with one track up to several frames ahead beyond the adjacent key frame, and the image processing apparatus according to Configuration 5, characterized in that.

[0070] [Configuration 7] The image processing apparatus according to any one of Configurations 1 to 6, further comprising determination means for determining the track boundary based on the difference in the shapes represented by the tracked 3D models of the respective adjacent tracks in each frame included in the overlapping section.

[0071] [Configuration 8] The determination means Calculates the similarity of the shapes represented by the tracked 3D models of the respective adjacent tracks in units of frames included in the overlapping section, Identifies the frame pairs with the highest calculated shape similarity, And determines the track boundary between the identified frame pairs, And the image processing apparatus according to Configuration 7, characterized in that.

[0072] [Configuration 9] The determination means specifies, as the frame pair, a combination of the frame with the highest similarity in shape among each of the frames included in the overlapping section and the frame with the higher similarity in shape among the two frames adjacent to the frame, according to the image processing apparatus described in Configuration 8.

[0073] [Configuration 10] The determination means specifies, as the frame pair, a combination in which the sum of the similarities in shape between two adjacent frames among each of the frames included in the overlapping section is maximized, according to the image processing apparatus described in Configuration 8.

[0074] [Configuration 11] The similarity in shape is a value evaluated using the Hausdorff distance based on the shape represented by the tracked 3D model, according to the image processing apparatus described in any one of Configurations 7 to 10.

[0075] [Configuration 12] The similarity in shape is a value evaluated using the cosine similarity based on the normal of the tracked 3D model, according to the image processing apparatus described in any one of Configurations 7 to 10.

[0076] [Configuration 13] The determination means further includes a determination means for obtaining the signed distance fields of the tracked 3D models of the adjacent tracks in each of the frames included in the overlapping section and determining the track boundary based on the similarity of the obtained signed distance fields, according to the image processing apparatus described in Configuration 7.

[0077] [Configuration 14] The determination means obtains the amount of shape change between the frames of the 3D models associated with each of the frames of the time-series shape data in the overlapping section, and determines the frames with the most amount of shape change as the track boundary. This is the image processing apparatus described in Configuration 7.

[0078] [Configuration 15] The determination means in the overlapping section, obtains the amount of shape change between frames of the 3D model associated with each frame of the time-series shape data, and determines the frames with the least amount of shape change as the track boundary. The image processing apparatus according to Configuration 7, characterized in that.

[0079] [Configuration 16] The image processing apparatus according to any one of Configurations 7 to 15, further comprising generation means for generating a tracked 3D model corresponding to each frame of each track constituting the tracked time-series shape data based on the track boundary determined by the determination means.

[0080] [Configuration 17] The generation means generates a tracked 3D model corresponding to each frame of each track constituting the tracked time-series shape data by selecting one of the tracked 3D models of the adjacent tracks based on the track boundary determined by the determination means. The image processing apparatus according to Configuration 16, characterized in that.

[0081] [Configuration 18] The generation means generates a tracked 3D model corresponding to each frame of each track constituting the tracked time-series shape data by correcting the selected tracked 3D model among the tracked 3D models of the adjacent tracks based on the track boundary determined by the determination means. The image processing apparatus according to Configuration 16, characterized in that.

[0082] [Configuration 19] In the correction, information assigned to each vertex of the components of the selected tracked 3D model is corrected. The image processing apparatus according to Configuration 18, characterized in that.

[0083] [Configuration 20] In the correction, at least one of the coordinates or the normal of the component of the selected tracked 3D model is corrected as information given to the vertex of the component, and the image processing apparatus according to Configuration 19 is characterized in that.

[0084] [Configuration 21] In the correction, at least one of the color or the transparency of the component of the selected tracked 3D model is corrected as information given to the vertex of the component, and the image processing apparatus according to Configuration 19 is characterized in that.

[0085] [Configuration 22] In the correction, at least one of the size or the amount of deformation for each component applied at the time of rendering is corrected as information given to the vertex of the component of the selected tracked 3D model, and the image processing apparatus according to Configuration 19 is characterized in that.

[0086] [Configuration 23] The generation means calculates a signed distance field for each of the tracked 3D models of the adjacent tracks, calculates an intermediate signed distance field for each frame included in the overlapping section based on the calculated signed distance field, performs the correction based on the intermediate signed distance field calculated for each frame, and the image processing apparatus according to Configuration 19 is characterized in that.

[0087] [Configuration 24] The intermediate signed distance field is calculated by taking a weighted average of two SDFs calculated for each of the adjacent tracks, and the image processing apparatus according to Configuration 23 is characterized in that.

[0088] [Configuration 25] The image processing apparatus according to configuration 24, wherein the weight in the weighted average linearly changes according to the number of frames to an adjacent key frame.

[0089] [Configuration 26] The generation means searches for the nearest surface or nearest point from the components of the untracked 3D model to the components of the tracked 3D model of the selected side, and performs the correction by weighted interpolation using the result of the search. The image processing apparatus according to configuration 19, wherein:

[0090] [Configuration 27] The image processing apparatus according to configuration 26, wherein the weight in the weighted interpolation linearly changes according to the number of frames to an adjacent key frame.

[0091] [Configuration 28] The image processing apparatus according to any one of configurations 1 to 27, wherein the time-series shape data is mesh-form data using polygons as components of the 3D model.

[0092] [Configuration 29] The image processing apparatus according to any one of configurations 1 to 27, wherein the time-series shape data is point-cloud-form data using points as components of the 3D model.

[0093] [Method 1] An acquisition step of acquiring time-series shape data composed of a frame group including a 3D model representing the three-dimensional shape of an object, A tracking step of performing tracking processing on the time-series shape data so that an overlapping section occurs between adjacent tracks, An output step of outputting tracked time-series shape data without an overlapping section between adjacent tracks, with any position within the overlapping section as a track boundary, based on the result of the tracking processing. comprising The track indicates a frame section in which components of the 3D model are tracked between frames and a common topology is maintained. An image processing method characterized by the above.

[0094] [Configuration 30] A program for causing a computer to function as the image processing apparatus according to any one of Configurations 1 to 29.

Claims

1. An acquisition means for acquiring time-series shape data composed of a group of frames including a 3D model representing the three-dimensional shape of an object; A tracking means for performing tracking processing on the time-series shape data so that an overlapping section occurs between adjacent tracks; An output means for outputting tracked time-series shape data without an overlapping section between adjacent tracks, with any position within the overlapping section as a track boundary, based on the result of the tracking processing; comprising; The track indicates a frame section in which components of the 3D model are tracked between frames and a common topology is maintained; An image processing apparatus characterized by the above.

2. The tracking means; Performs tracking processing in two directions, the time-series direction and the reverse time-series direction, based on a plurality of key frames set for the group of frames; The image processing apparatus according to claim 1, characterized by the above.

3. The tracking means performs tracking processing in the two directions with the number of frames exceeding half of the interval between adjacent key frames. The image processing apparatus according to claim 2, characterized by the above.

4. The tracking means performs tracking processing in the two directions with one track being from a certain key frame to the next key frame. The image processing apparatus according to claim 3, characterized by the above.

5. The tracking means; Performs tracking processing in either one direction, the time-series direction or the reverse time-series direction, based on a plurality of key frames set for the group of frames; The image processing apparatus according to claim 1, characterized by the above.

6. The tracking means performs tracking processing in the one direction with one track being several frames ahead beyond the adjacent key frame. The image processing apparatus according to claim 5, characterized by the above.

7. The image processing apparatus according to claim 1, further comprising a determination means for determining the track boundary based on the difference in the shapes represented by the tracked 3D models of each of the adjacent tracks in each frame included in the overlapping section.

8. The determination means; Calculates the similarity of the shapes represented by the tracked 3D models of each of the adjacent tracks in units of frames included in the overlapping section; Identifies the frame pairs with the highest calculated shape similarity; Determine the track boundary between the specified frame pairs. The image processing apparatus according to claim 7, characterized in that.

9. The determination means specifies, as the frame pair, a combination of the frame having the highest similarity in shape among the frames included in the overlapping section and the frame having the higher similarity in shape among the two frames adjacent to the frame. The image processing apparatus according to claim 8, characterized in that.

10. The determination means specifies, as the frame pair, a combination in which the sum of the similarities in shape between two adjacent frames among the frames included in the overlapping section is the largest. The image processing apparatus according to claim 8, characterized in that.

11. The similarity in shape is a value evaluated using the Hausdorff distance based on the shape represented by the tracked 3D model. The image processing apparatus according to claim 7, characterized in that.

12. The similarity in shape is a value evaluated using the cosine similarity based on the normal of the tracked 3D model. The image processing apparatus according to claim 7, characterized in that.

13. The determination means further includes determination means for obtaining the signed distance fields of the tracked 3D models of the respective adjacent tracks in each frame included in the overlapping section and determining the track boundary based on the similarity of the obtained signed distance fields. The image processing apparatus according to claim 7, characterized in that.

14. The determination means is In the overlapping section, obtain the amount of shape change between the frames of the 3D models associated with each frame of the time-series shape data. Determine the frame pair with the largest amount of shape change as the track boundary. The image processing apparatus according to claim 7, characterized in that.

15. The determination means is In the overlapping section, obtain the amount of shape change between the frames of the 3D models associated with each frame of the time-series shape data. Determine the frame pair with the smallest amount of shape change as the track boundary. The image processing apparatus according to claim 7, characterized in that.

16. The image processing apparatus according to claim 7, further comprising generation means for generating a tracked 3D model corresponding to each frame of each track constituting the tracked time-series shape data based on the track boundary determined by the determination means.

17. The generation means generates a tracked 3D model corresponding to each frame of each track constituting the tracked time-series shape data by selecting one of the tracked 3D models of the adjacent tracks based on the track boundary determined by the determination means. The image processing apparatus according to claim 16, characterized in that.

18. The generation means generates a tracked 3D model corresponding to each frame of each track constituting the tracked time-series shape data by correcting the selected tracked 3D model among the tracked 3D models of the adjacent tracks based on the track boundary determined by the determination means. The image processing apparatus according to claim 16, characterized in that.

19. In the correction, the information assigned to each vertex of the components of the selected tracked 3D model is corrected. The image processing apparatus according to claim 18, characterized in that.

20. In the correction, at least one of the coordinates or the normal of the component is corrected as the information assigned to the vertex of the component of the selected tracked 3D model. The image processing apparatus according to claim 19, characterized in that.

21. In the correction, at least one of the color or the transparency of the component is corrected as the information assigned to the vertex of the component of the selected tracked 3D model. The image processing apparatus according to claim 19, characterized in that.

22. In the correction, at least one of the size or the deformation amount of each component applied during rendering is corrected as the information assigned to the vertex of the component of the selected tracked 3D model. The image processing apparatus according to claim 19, characterized in that.

23. The generation means calculates a signed distance field for each of the tracked 3D models of the adjacent tracks, Based on the calculated signed distance field, an intermediate signed distance field is calculated for each frame included in the overlapping section, the correction is performed based on the intermediate signed distance field calculated for each frame, The image processing apparatus according to claim 19, characterized in that.

24. The intermediate signed distance field is calculated by taking a weighted average of two SDFs calculated for each of the adjacent tracks, The image processing apparatus according to claim 23, characterized in that.

25. The weight in the weighted average linearly changes according to the number of frames to the adjacent key frame, The image processing apparatus according to claim 24, characterized in that.

26. The generation means, search for the nearest neighbor surface or nearest neighbor point from the components of the unselected tracked 3D model to the components of the selected tracked 3D model, perform the correction by weighted interpolation using the result of the search, The image processing apparatus according to claim 19, characterized in that.

27. The weight in the weighted interpolation linearly changes according to the number of frames to the adjacent key frame, The image processing apparatus according to claim 26, characterized in that.

28. The time-series shape data is mesh-form data using polygons as components of the 3D model, The image processing apparatus according to claim 1, characterized in that.

29. The time-series shape data is point-cloud-form data using points as components of the 3D model, The image processing apparatus according to claim 1, characterized in that.

30. an acquisition step of acquiring time-series shape data composed of a frame group including a 3D model representing the three-dimensional shape of an object; a tracking step of performing tracking processing on the time-series shape data so that an overlapping section occurs between adjacent tracks; an output step of outputting tracked time-series shape data without an overlapping section between adjacent tracks, with any position within the overlapping section as a track boundary, based on the result of the tracking processing; having, The track indicates a frame section in which the components of the 3D model are tracked between frames and a common topology is maintained, An image processing method, characterized in that.

31. A program for causing a computer to execute the image processing method according to claim 30.

Images