Image processing device, image processing method, and program

Abstract

To improve the efficiency of video compression in the time direction.SOLUTION: A bone model that shows the connection relationship between the parts of a 3D model is obtained. The bone model is divided into parts. A projection plane for projecting the 3D model is set for each divided part on the basis of the divided bone model. A 2D image onto which the 3D model is projected is generated on the basis of each projection plane set for each divided part. The 2D image is encoded for each divided part.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] A virtual viewpoint video generation system is a technology that can create an image seen from a virtual viewpoint specified by the user from images captured by an imaging system using multiple cameras and play it back as a virtual viewpoint video. In recent years, attention has been focused on a technology that compresses and distributes the 3D model data of the subject generated by the virtual viewpoint video generation system, allowing the user to display the virtual viewpoint video on their own device, such as a PC, HMD (head-mounted display), or tablet.

[0003] In the technology disclosed in Patent Document 1, parameters related to multiple point cloud models are converted, and a hexahedral projection surface is set for the multiple point cloud models. Next, a two-dimensional planar image projected from the set projection surface is generated, and the generated two-dimensional planar image is encoded to generate coded data of the two-dimensional planar image and a bitstream including information related to the parameter conversion. By decoding the coded two-dimensional image, it becomes possible to display a virtual viewpoint video on a user's terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 012968 [Patent Document 2] Japanese Patent Publication No. 2020-29777 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology described in Patent Document 1, a hexahedron that covers each subject is basically set as the projection surface, but if part of the subject is hidden, the shape cannot be encoded correctly. Furthermore, in the technology described in Patent Document 1, when motion compensation encoding of a two-dimensional image is performed, if the subject moves quickly, the difference in the time direction during video compression increases, reducing the overall compression rate. This reduction in compression rate increases the amount of data transmitted, which may hinder smooth distribution.

[0006] An object of the present invention is to improve the efficiency of video compression in the time direction. [Means for solving the problem]

[0007] To achieve the object of the present invention, for example, an image processing device according to one embodiment comprises the following arrangement: an acquisition means for acquiring a bone model that indicates the connection relationships between parts of a three-dimensional model, a first division means for dividing the bone model into parts, a setting means for setting, for each divided part, a projection plane onto which the three-dimensional model is to be projected based on the divided bone model, a generation means for generating a two-dimensional image onto which the three-dimensional model is projected based on each of the projection planes set for each divided part, and an encoding means for encoding the two-dimensional image for each divided part. [Effects of the Invention]

[0008] To improve the efficiency of video compression in the time direction. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Figure 3] FIG. 2 is a diagram for explaining part information and part connection information of a three-dimensional model. [Figure 4] A diagram to explain the relationship between the 3D model and the bone model. [Figure 5]FIG. 10 is a diagram for explaining the division of a bone model into parts. [Figure 6] FIG. 10 is a diagram illustrating division of a three-dimensional model. [Figure 7] 10A to 10C are diagrams for explaining a process of setting a front direction in a divided point cloud model. [Figure 8] 10A and 10B are diagrams for explaining a process of setting a projection plane for each divided part. [Figure 9] 10A and 10B are diagrams for explaining a two-dimensional image generation process on a projection surface. [Figure 10] FIG. 4 is a diagram showing an example of a format of transmission data. [Figure 11] FIG. 10 is a diagram showing an example of a header structure of a sequence. [Figure 12] FIG. 2 is a diagram showing an example of the configuration of a frame header. [Figure 13] FIG. 10 is a diagram showing an example of the header structure of data for each divided part. [Figure 14] FIG. 2 is a diagram showing an example of the structure of coded data of a projected two-dimensional image. [Figure 15] FIG. 10 is a diagram showing an image of connections when combining divided point cloud models. [Figure 16] 10 is a flowchart showing an example of image processing by an image processing device that performs encoding. [Figure 17] 10 is a flowchart showing an example of image processing by an image processing device that performs decoding. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] 1 is a schematic diagram showing an example of an image processing system 101 including an image processing device according to this embodiment. The image processing system 101 according to this embodiment includes an image processing device 102, an image processing device 103 connected to the image processing device 102, and an image generation device 107.

[0012] The image processing device 102 according to this embodiment acquires a bone model that indicates the connection relationship between parts of a three-dimensional (3D) model, and divides the acquired bone model into parts. Next, the image processing device 102 sets a projection plane for projecting the 3D model for each divided part, and generates a two-dimensional image onto which the 3D model is projected based on each of the projection planes. Furthermore, the image processing device 102 encodes the two-dimensional image for each divided part.

[0013] In FIG. 1, multiple cameras (imaging devices) 106 capture multiple captured images of an imaging target 105, and a 3D point cloud model of the imaging target is generated based on these captured images. In this embodiment, an example is described in which a 3D point cloud model of a human being generated in this manner is used as the 3D model to be processed, but this is not particularly limited as long as a bone model of the 3D model can be acquired. Herein, this 3D point cloud model is generated based on images captured by the cameras 106, but the 3D point cloud model of the imaging target 105 may be acquired in advance from an external device (not shown). Hereinafter, when simply referred to as a "point cloud model," this refers to the 3D point cloud model of the imaging target 105 to be processed, acquired in this manner.

[0014] The image processing device 102 divides the bone model, sets a projection plane for projecting a point cloud model for each divided region, and generates a 2D image onto which the point cloud model is projected based on each of the projection planes. The image processing device 102 then encodes the 2D image for each divided region. Meanwhile, the image processing device 103 decodes the 2D image encoded by the image processing device 102. In this embodiment, the image generation device 107 regenerates the point cloud model based on the decoding result by the image processing device 103 and presents it to the user as a virtual viewpoint video. The image generation device 107 accepts user operations via a virtual viewpoint operation unit and generates a virtual viewpoint video based on the user input. Note that, although the following description is given assuming that the image processing device 103 performs the decoding process, the image processing device 102 may also be capable of performing a similar decoding process.

[0015] 2 is a block diagram showing an example of the functional configuration of the image processing system 101 of this embodiment. The image processing device 102 includes a model acquisition unit 201, a bone generation unit 202, a bone division unit 203, a division model generation unit 204, a projection plane direction setting unit 205, a projection plane setting unit 206, a two-dimensional (2D) image generation unit 207, a 2D image encoding unit 208, and a transmission unit 209.

[0016] The model acquisition unit 201 acquires a 3D model (point cloud model) to be processed. As described above, the model acquisition unit 201 can generate a point cloud model based on captured images captured by multiple cameras 106. The process of generating a point cloud model based on captured images can be performed using any known technology. For example, the model acquisition unit 201 separates a foreground region and a background region from the captured image using a commonly used known foreground / background separation technology. Next, the model acquisition unit 201 generates a foreground silhouette image by taking a logical product of the foreground region and the 3D point cloud model generation space projection region. Here, the 3D point cloud model generation space projection region is, for example, a predefined 3D point cloud model generation space reprojected onto the angle of view of each imaging device, and is set in advance via a UI screen (not shown). The generated foreground silhouette image may be an image with the same resolution as the captured image, or may be one or more images obtained by cutting out regions corresponding to one or more foreground regions with circumscribing rectangles. The model acquisition unit 201 can generate a three-dimensional point cloud model by the volume intersection method using the generated foreground silhouette image and preset camera parameters of the camera 106.

[0017] The bone generation unit 202 generates a bone model of the imaging target 105 from the point cloud model acquired by the model acquisition unit 201. The bone model can be generated by a known method, and the method is not particularly limited. For example, the model generation unit 202 may estimate the bone model of the point cloud model by obtaining a posture model described in Patent Document 2.

[0018] The bone generation unit 202 stores in advance a three-dimensional standard shape model that mimics a standard human shape and its posture model. A general model used in CG can be used as the standard shape model and posture model, and a general file format such as FBX can also be used for the standard shape model and posture model. The three-dimensional standard shape model is expressed as a three-dimensional point cloud model that includes information on the IDs (vertex IDs) of the vertices that make up triangular or quadrangular faces and the coordinates of those vertices (vertex coordinates). Meanwhile, the bone model according to this embodiment is composed of part information that represents each part of the human body, such as the head, neck, arms, or ankles, and part connection information that represents the connection relationships between the parts.

[0019] FIG. 3 is a diagram showing an example of part information and part connection information. As shown in FIG. 3(a), part information according to this embodiment is composed of a 3D model ID, part codes, and joint point coordinates. The 3D model ID is a unique ID assigned to each bone model. The part code is a number assigned to each part of the bone model, and is predetermined here as shown in FIG. 3(b). Part connection information according to this embodiment is composed of the 3D model ID, part codes, and connection part codes that indicate the part codes of the parts to which each part is connected. By associating the parts of the standard shape model with the parts of the bone model, it is possible to change the posture of the bone model according to the posture of the standard shape model.

[0020] FIG. 4 is a diagram illustrating the relationship between a 3D model (standard shape model) 401 and a bone model 402. The black circles in the bone model 402 represent joints, and the white circles represent junctions and end portions. The bone model 402 is composed of bones connecting the joints, junctions, and end portions. In this example, the white circles representing the end portions of the head are positioned as the center of gravity of the head. In the posture estimation process, the standard shape model corresponding to the bone model is deformed so that the images projected onto each camera image match the foreground areas of each camera, and the posture at which the two images most closely match is determined as the bone model of the subject. Note that the bone model estimation method is not limited to this, and any method for generating bones for a 3D model can be used. For example, the bone generation unit 202 may estimate the two-dimensional posture of the subject based on the two-dimensional images obtained from each camera, and then estimate the bone model based on the relationship between the cameras.

[0021] The bone splitting unit 203 splits the bone model generated by the bone generation unit 202 into parts. Specifically, the bone splitting unit 203 splits the bone model into parts (here, by part) using part information indicating each part of the bone model set by the bone generation unit 202 and part connection information indicating the connection relationships between the parts. FIG. 5 shows an image of the splitting of the bone model into parts. FIG. 5(a) shows a bone model of an arm, and FIG. 5(b) shows a bone model obtained by splitting the bone model shown in FIG. 5(a) into parts. In this embodiment, the bone splitting unit 203 splits the bone model based on the coordinates of the joint points shown in FIG. 3(a). In splitting the bone model, the joints connecting the parts may be included in both adjacent split parts as shown in FIG. 5(b), or may be included in only one part as shown in FIG. 5(c). In addition, a normal plane may be set at the joint point relative to the line connecting the joint points, and the part included between these normal planes may be treated as one part after division (for example, the torso part may be treated as one part by combining multiple joints and connecting parts).

[0022] The model generation unit 204 divides the point cloud model acquired by the model acquisition unit 201 into parts using the bone model divided into parts generated by the bone division unit 203. Specifically, the model generation unit 204 divides the 3D point cloud model corresponding to the bone model in accordance with the division of the bone model by the bone division unit 203. FIG. 6 shows an image of the division of the 3D model. FIG. 6(a) shows a bone model of an arm corresponding to FIG. 5(a), and FIG. 6(b) shows a bone model obtained by dividing the bone model shown in FIG. 6(a) into parts corresponding to FIG. 5(b). The point cloud model is divided from the point cloud model shown in FIG. 6(a) into the point cloud model shown in FIG. 6(b) using a plane normal to a line connecting joint points as a cutting plane. In the example of FIG. 6, the right upper arm of the bone model 402 shown in FIG. 6(a) is divided into bone models 601a, 601b, and 601c after division in FIG. 6(b). Based on these bone models 601a, 601b, and 601c, the model generation unit 204 divides the right upper arm of the point cloud model 401 into point cloud models 602a, 602b, and 602c, and acquires each of these as a divided point cloud model (hereinafter, sometimes simply referred to as a divided point cloud model). Hereinafter, some parts of the bone models and point cloud models divided in this way may be referred to as "divided parts."

[0023] The projection plane direction setting unit 205 defines the direction of the coordinate axes for setting the projection plane for each divided region (each divided point cloud model) generated by the model generation unit 204. By defining the direction of the coordinate axes for each region and defining the front direction at the time of projection, it becomes possible to always obtain a projected 2D image of that region from the same direction. Here, for a certain region, the front direction is set as one of the coordinate axes of the spatial coordinate system (hereinafter, the positive x-axis direction). The method of defining the direction of the coordinate axes and the front direction may differ for each region, or may be consistent for all regions. Below, with reference to FIG. 7, a specific example of the process of defining the direction of the coordinate axes and the front direction by the projection plane direction setting unit 205 will be described.

[0024] FIG. 7 is a diagram illustrating an example of the direction definition for each divided point cloud model. For example, for the head, the projection plane direction setting unit 205 determines the direction of the front of the face as the forward direction (positive x-axis direction) based on the shape of the point cloud model as shown in FIG. 7(a), and defines the line connecting the neck joint point and the head endpoint on the bone model as the z-axis. Here, the projection plane direction setting unit 205 defines the head endpoint side on the bone model as the positive z-axis direction, the line normal to the x-axis and normal to the z-axis as the y-axis, and defines the right side of the forward direction as the positive y-axis direction. For the upper arm, the projection plane direction setting unit 205 defines the line connecting the shoulder joint point and the elbow joint point on the bone model as the z-axis, as shown in FIG. 7(b), and defines the elbow joint point side on the bone model as the positive z-axis direction. Next, the projection plane direction setting unit 205 defines the line perpendicular to the z-axis and parallel to the wrist as the x-axis, and defines the wrist side on the bone model as the positive x-axis direction (forward direction). The projection surface direction setting unit 205 defines the y-axis as a line that is normal to both the x-axis and the z-axis, and defines the right side of the forward direction as the positive direction of the y-axis. For the right and left hands, the palm may be defined as the forward direction, or the upper arm may be defined as the joint point and the direction facing the chest may be defined as the forward direction. In this manner, in this embodiment, direction is defined using a line connecting the joint points and the endpoints, and for the hands and feet, the forward direction is determined using the z-axis as a line connecting the wrist joint point and the endpoint of the middle finger.

[0025] In this way, the definition of the coordinate axes for setting the projection plane according to this embodiment is different for each part. Furthermore, the projection plane direction setting unit 205 may perform the above-described direction definition based on the bone model alone, or may perform the direction definition using both the bone model and the point cloud model. Furthermore, the method of direction definition is not limited to the above-described example, and the user may be able to arbitrarily set the direction for each part.

[0026] The projection plane setting unit 206 sets a projection plane for projecting the point cloud model for each divided body part based on the divided bone model. Here, the projection plane setting unit 206 sets multiple projection planes for each body part using the divided point cloud model generated by the model generation unit 204 and the coordinate axes for each body part set by the projection plane direction setting unit 205. Fig. 8 shows an image diagram of the projection plane setting process. Fig. 8(a) shows a 3D point cloud model of the head in which the directions of the coordinate axes are defined by the projection plane direction setting unit 205, and using these coordinate axes, projection planes (here, six for one body part) that surround the 3D point cloud model are set as shown in Fig. 8(b). Here, six projection planes are generated so that each face of a rectangular parallelepiped having a surface perpendicular to the coordinate axis (front direction) serves as a projection plane.

[0027] An example of the projection plane setting process shown in FIG. 8(b) will be specifically described below. The projection plane setting unit 206 first determines a projection range for each part. Here, the projection plane setting unit 206 sets the projection range so as to cover the division point cloud model using the division point cloud model generated by the model generation unit 204. Next, the projection plane setting unit 206 sets a rectangular parallelepiped that includes the set projection range. Each face of the rectangular parallelepiped is set so as to be parallel to one of the xy plane, yz plane, or zx plane using the coordinate axes for each part set by the projection plane direction setting unit 205. Each face of the set rectangular parallelepiped becomes the projection plane for that part.

[0028] The 2D image generation unit 207 generates a two-dimensional image (projected two-dimensional image) by projecting the point cloud model based on each of the multiple projection planes set by the projection plane setting unit 206. In this embodiment, the 2D image generation unit 207 generates a projected two-dimensional image by projecting the surface of the point cloud model onto the projection plane. FIG. 9 shows an image diagram of the image generation process. FIG. 9(a) is a diagram showing a three-dimensional point cloud model of the head for which the projection planes have been set by the projection plane setting unit 206. Projection plane X+ is the projection plane located in the positive direction of two projection planes parallel to the xy plane, and projection plane X- is the projection plane located in the negative direction of the two. Similarly, projection plane Y+ is the projection plane located in the positive direction of two projection planes parallel to the xy plane, and projection plane X- is the projection plane located in the negative direction of the two, and the same applies to Z+ and Z-. Specifically, the projection plane setting unit 206 generates a projected two-dimensional image by projecting the divided point cloud model onto a two-dimensional plane defined as the projection plane. In this embodiment, for ease of explanation, six projection planes are set for one part to form a hexahedron of projection planes, but the number and directions of the projection planes are not limited to this. For example, twelve projection planes may be set for one part, and the projection planes may form a regular dodecahedron surrounding the point cloud model.

[0029] The 2D image encoding unit 208 encodes the projected 2D image for each division point cloud model generated by the 2D image generation unit 207. The 2D image encoding unit 208 can encode the projected 2D image by arbitrarily adopting a commonly used image encoding method. For example, the 2D image encoding unit 208 may encode the projected 2D image by an encoding method for 2D planar images such as JPEG (Joint Photographic Experts Group) or AVC (Advanced Video Coding).

[0030] When encoding video, if a part of the subject is hidden (for example, if the face is covered by the hand during dancing) or if the subject is moving at high speed, the difference in the time direction of the video tends to increase, resulting in a decrease in the compression rate. On the other hand, by encoding the projected 2D image for each segmented point cloud model as described above, it becomes unnecessary to consider the difference in the time direction that is unnecessary for the entire subject, making it possible to further improve the compression rate. Furthermore, by setting the front direction for each part, it is possible to suppress changes in shape over time, and the encoding efficiency can be improved by motion compensation encoding.

[0031] The transmitting unit 209 transmits the 2D image encoded by the 2D image encoding unit 208 and conversion information required for decoding to an external device (here, the image processing device 103 that performs decoding processing). FIG. 10 shows an example of the format of data transmitted to the image processing device 103. Hereinafter, the entire data transmitted from the image processing device 102 to the image processing device 103 will be referred to as a sequence. A sequence begins with a Sequence Header. The Sequence Header contains information about the point cloud model to be encoded. Here, the point cloud model is encoded frame by frame, and the sequence contains an item, Frame Data, for managing information on such frame by frame. The Frame Data begins with a Frame Header containing information about the frame, and the Frame Header contains Part Data that has been divided into parts and encoded. The Part Data contains a Parts Header containing information about the parts and information required for decoding. The Part Data also contains a Projection Image Header that indicates the content of each projected 2D image, and the Projection Image Header is followed by the encoded data, Projection Image Data, of the actual image.

[0032] FIG. 11 shows an example of the structure of a Sequence Header. In the example of FIG. 11, the Sequence Header includes Sequence Start_Code, Model_ID, Model_location, Start_Frame_Time, Frame_rate, number_of_frames, user_data_frag, and user_data, in this order. Sequence_Start_Code indicates the beginning of the Sequence Header. Model_ID is information for identifying the encoded 3D point cloud model. Model_location is information indicating the location of the encoded point cloud model in virtual space. Start_Frame_Time indicates the date and time or time of the first frame of the point cloud model. There are no particular limitations on how the date and time are expressed; for example, Greenwich Mean Time may be used to express the date and time or hour, minute, and second, with a time code written at the end. The same applies to the following date and time expressions. Frame_rate indicates the frame rate of the video, and number_of_frame indicates the number of frames included in the sequence. user_data_flag is flag information that indicates, with 0 or 1, whether meta information is included as user_data, and user_data is assigned when user_data_flag is 1. user_data is user data generated in units that can be collectively managed, such as the event, venue, or date and time at which the 3D point cloud model was generated. Note that the data stored in the Sequence Header is not limited to these, and may include additional information, or some of it may be omitted.

[0033] Next, we will explain the details of Frame Data. Frame Data includes a Frame Header that represents a frame, followed by Part Data, which is data for each divided part of the point cloud model. FIG. 12 shows an example of the configuration of a Frame Header. In the example of FIG. 12, the Frame Header includes Frame_Start_Code, Frame_Data_Time, number_of_Part_Data, and size_of Part_Data[1..p], in this order. Frame_Start_Code indicates the start of the Frame Header. Frame_Date_Time indicates the date and time of the first frame of the point cloud model. number_of_Part_Data indicates the number p of part data included in the frame. size_of Part_Data is an item that stores the data size (e.g., in bytes) of the Part Data for each part in encoded order. This data structure makes it possible to access the Part Data for each part. Furthermore, the size of the Frame Data can be calculated by adding up the sizes of all the Part Data, making it possible to access the next Frame Data.

[0034] Next, we will explain the details of Part Data. Part Data includes a Part Header that represents a frame, followed by Part Data, which is data for each divided part of a point cloud model. Figure 13 shows an example of the configuration of a Part Header. In the example of Figure 13, the Part Header includes Part_Start_Code, Part_ID, number_of_Parts, Part_code[1..c], Part_location, Connection_1 st _Point_location, Connection_1 st _Parts_ID, Connection_2 nd _Point_location, Connection_2 ndIt contains _Parts_ID, Frontal_vector, number_of_Projection_Images, and size_of_Projection_Image[1..pr] in this order. Part_Start_Code indicates the start of the Part Header. Part_ID contains an ID that identifies the data of the divided parts within the frame. Since one Part Data can contain multiple parts, the number c of parts included in the division point cloud model is stored in number_of_Parts. Part_code stores the code of the part included in the division part, for example, the code shown in Figure 3(b). Part_location stores the position of the division point cloud model, for example, the position of the center of gravity, in 3D coordinates. Connection_1 st _Point_location stores the coordinates of the joint point between the divided part and the adjacent part, for example, joint point 1 in Figure 3(a), and the ID of the adjacent part that joins at that joint point is Connection_1. st _Part_ID. On the other hand, the coordinates of the joint point between the divided part and the adjacent part, for example, joint point 2 in Figure 3(a), are stored in Connection_2. nd _Point_location is stored, and the ID of the adjacent part that joins at that junction point is Connection_2 nd The ID of the adjacent part to be joined may be omitted. Frontal_vector contains information indicating the front direction of the part. There are no particular limitations on the method of describing Frontal_vector, and it may be expressed, for example, in polar coordinates indicating the front direction from the position of the division point cloud model indicated in Part_location. number_of_Projection_Images contains the number pr of encoded projection 2D images, and size_of_Projection_Image describes the size of the encoded data in the order in which it is encoded and stored.

[0035] Finally, we will explain the coded data of a projection 2D image. Hereinafter, when simply referred to as "coded data," it refers to data obtained by coding a projection 2D image. The coded data consists of a Projection Image Header item that contains information necessary for decoding and reconstructing point cloud data of the body part, and coded data of the actual image, Projection Image Data.

[0036] FIG. 14 shows an example of the configuration of a Projection Image Header. In the example of FIG. 14, the Projection Image Header includes Projection_Image_Start_Code, Projection_Image_ID, Projection_Image_Coding_method, Visual_camera_Location, and Visual_camera_Rotation_Quaternion, in this order. Projection_Image_Start_Code indicates the beginning of the Projection Image Header. Projection_Image_ID includes an ID that identifies the projected 2D image data. Projection_Image_Coding_method includes information indicating the coding method for the projected 2D image. The description method in Projection_Image_Coding_method is not particularly limited, and may be, for example, the ISO number of the coding method, the name of the coding method, or a list number corresponding to the coding method. Virtual_camera_Location includes information indicating the position of the projection point (generally a virtual camera in a virtual viewpoint video) that generates the projected 2D image. Here, the position of the projection point is expressed in three-dimensional coordinates. The position of the projection point according to this embodiment may be expressed in the world coordinate system, or may be expressed as a relative position from the position of the corresponding divided part. Virtual_camera_Rotation_Quaternion represents the orientation of the virtual camera, and here, the orientation of the virtual camera is represented by three values ​​of quaternion.

[0037] In addition, if the encoding method for the projected 2D image is a motion compensation encoding method such as MPEG or multi-view encoding, encoding can be achieved by including in the Projection Image Header the date and time or time information of the frame indicating the referenced Part Data, an identification ID of the division part, or an ID that identifies the projected 2D image.

[0038] Next, we will explain the decoding process executed by the image processing device 103. As described above, the image processing device 103 decodes the two-dimensional image encoded by the image processing device 102, and includes a receiving unit 210, a decoding unit 211, a model regeneration unit 212, a model synthesis unit 213, and an output unit 214.

[0039] The receiving unit 210 receives a plurality of coded 2D images and transformation information transmitted from the transmitting unit 209 of the image processing device 102. The decoding unit 211 encodes the 2D images for each divided portion. Here, the decoding unit 211 receives the Sequence data received by the receiving unit 210 and decodes each header to obtain information necessary for decoding and synthesizing the 2D images. Next, the decoding unit 211 decodes the coded data (Projection Image Data) of the projected 2D images using the decoding method specified by Projection_Image_Coding_method. The decoding unit 211 decodes all projected 2D images indicated by number_of_Projection Images.

[0040] The model regeneration unit 212 generates (regenerates) a division model (division point cloud model) by dividing the 3D model of the imaging target 105 into parts, using the projected 2D images decoded by the decoding unit 211. Specifically, the model regeneration unit 212 generates a point cloud model by projecting the projected 2D images for each divided part into a 3D space based on the descriptions of Virtual_camera_Location and Virtual_camera_Rotation Quaternion. The method of the point cloud model generation process by the model regeneration unit 212 is not particularly limited, and may use, for example, the volume intersection method described above.

[0041] The model synthesis unit 213 synthesizes the divided point cloud models generated by the model regeneration unit 212 into one point cloud model. Specifically, the model synthesis unit 213 determines how to connect the divided point cloud models and synthesizes them using information such as the position of the part, the joint position, and the front direction described in the Part Header received by the receiving unit 210. FIG. 15 shows an image diagram of connections when synthesizing divided point cloud models. For example, the model synthesis unit 213 can generate one point cloud model by acquiring the coordinates of the joint points included in each divided point cloud model as shown in FIG. 15 from Part_location and arranging the divided point cloud models based on these coordinates. Here, the model synthesis unit 213 sets the front direction to the direction described in Frontal_vector and synthesizes Connection_1. st Connection_2 from the first connection point of the part included in _Point_location nd The point cloud model is regenerated by arranging each divided point cloud model at the second joining position included in _Point_location. Note that the process of regenerating the point cloud model is not limited to this, and for example, the point cloud model may be generated using information on the bone model. For example, the bone model information can be transmitted from the image processing device 102 to the image processing device 103 by entering all the data shown in FIG. 3(a) in user_data shown in FIG. 11.

[0042] The output unit 214 outputs the 3D point cloud model generated by the model synthesis unit 213 to the image generation device 107. The image generation device 107 generates a virtual viewpoint image using the 3D point cloud model in a virtual viewpoint image generation unit. This generation process can be realized by arranging bone models so as to correspond to the part code Part_code[1..c] of each part. The image generation process and the generated images can be performed using known techniques used for image generation using 3D models, so detailed explanations will be omitted here.

[0043] 16 is a flowchart showing an example of a series of processes in the image processing system 101 of this embodiment, from acquiring a point cloud model from a captured image to transmitting an encoded two-dimensional image to the image processing device 103. A detailed description will be given below with reference to the flowchart in FIG.

[0044] In S1600, the image processing system 101 synchronizes and controls the multiple cameras 106 to start capturing images of the object 105 within the imaging range 104. In S1601, the model acquisition unit 201 generates a foreground silhouette from the image captured in the frame to be processed, and then acquires a point cloud model. By default, the frame to be processed is the first frame after imaging has started.

[0045] In S1602, the bone generation unit 202 determines whether a bone model can be generated from the point cloud model generated in S1601. If a bone model can be generated, the process proceeds to S1603; if not, the process proceeds to S1604. The bone generation unit 202 according to this embodiment determines that a bone model can be generated if bone models of all parts of the imaging target 105 can be acquired based on the captured image acquired in S1600. An example of when a bone model cannot be generated is when there are multiple imaging targets 105 within the imaging range 104 and these imaging targets are crowded together, so that part of the body of the imaging target is hidden and a bone model cannot be acquired.

[0046] In S1603, the bone generation unit 202 generates a bone model from the point cloud model generated in S1601, and the process proceeds to S1605. In S1604, the projection plane setting unit 206 sets a projection plane for the entire point cloud model as a single processing target, and the process proceeds to S1611.

[0047] In S1605, the bone division unit 203 divides the generated bone model into parts based on the joint points and end points, and generates a bone model for each divided part. In S1606, the model generation unit 204 generates a point cloud model for each divided part based on the bone model for each divided part generated in S1603.

[0048] In S1607, the image processing device 102 acquires, for each point cloud model for each divided part generated in S1606, information necessary to be written in the Part Header, i.e., information necessary for reconstructing the part, such as the part's identification ID within the frame, the part's code, and information on the joining position.

[0049] In S1608, the projection plane direction setting unit 205 sets the coordinate axes for setting the projection plane (setting the coordinates of the virtual camera) for each point cloud model for each divided region generated in S1606. In S1609, the projection plane direction setting unit 205 determines the front direction for each point cloud model divided between frames, and saves the coordinate settings set in S1608. Here, if the coordinate settings have already been saved, the saved coordinate settings are reflected.

[0050] In S1610, the projection plane setting unit 206 sets a projection plane for each divided point cloud model using information on the coordinates and orientations of the multiple virtual cameras for each determined divided point cloud model.

[0051] In S1611, the 2D image generation unit 207 generates a projected 2D image for each division point cloud model based on the projection plane set in S1610.

[0052] In S1612, the 2D image encoding unit 208 encodes the projected 2D image generated in S1611 for each division point group model.

[0053] In S1613, the transmission unit 209 transmits the information necessary for reconstructing the region and the projected 2D image generated in S1607 and S1612 to the image processing device 103 that performs decoding processing. In S1614, the image processing device 102 determines whether to continue imaging. If imaging is to be continued, the frame to be processed is set to the next frame and the processing returns to S1601; if not, the processing in Fig. 16 ends. Here, it is assumed that if the user has performed an operation to end imaging, it is determined not to continue imaging.

[0054] In FIG. 16, an example has been described in which a point cloud model is generated based on a captured image, but if a pre-acquired point cloud model is used, S1600 is omitted and the pre-acquired point cloud model is acquired in S1601.

[0055] 16, an example in which a bone model is generated from a point cloud model has been described, but if a bone model can be generated, it is not necessary to generate it from a point cloud model. For example, the bone generation unit 202 may generate a bone model from a silhouette image.

[0056] 17 is a flowchart showing an example of a series of processes performed by the image processing device 103 according to this embodiment, from receiving an encoded projected two-dimensional image to decoding it and outputting a regenerated point cloud model. A detailed description will be given below with reference to the flowchart in FIG.

[0057] In S1700, the receiving unit 210 receives the coded header information and the coded data of the projected two-dimensional image transmitted from the image processing device 102.

[0058] In S1701, the decoding unit 211 performs a decoding process on the encoded projected two-dimensional image received in S1700. In S1702, the model regeneration unit 212 regenerates a division point cloud model based on the projected two-dimensional image decoded in S1701 and information necessary for reconstructing the region.

[0059] In S1703, the model synthesis unit 213 synthesizes point cloud models using the divided point cloud models regenerated in S1702. Information necessary for synthesis is received in S1700.

[0060] In S1704, the output unit 214 outputs the point cloud model synthesized in S1703. In S1705, the receiving unit 210 checks whether an encoded projected two-dimensional image for the next frame has been transmitted. If no projected two-dimensional image has been transmitted, it is determined that imaging has ended, and the processing ends. If a new projected two-dimensional image has been transmitted, the processing returns to S1700, and the new projected two-dimensional image is set as the processing target.

[0061] This process allows the bone model of a 3D model to be divided into parts, and a projection plane to be set for each divided part based on the divided bone model. Then, a projected 2D image can be generated and encoded based on each of the projection planes set for each divided part. Therefore, when compressing video, it is possible to reduce the difference in the time direction in the projected image and improve the compression rate compared to when a projection plane is set for the entire imaged object. Furthermore, by setting the front direction for each part, it is possible to suppress changes in shape over time, thereby improving the encoding efficiency of motion compensation encoding.

[0062] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) an acquisition means for acquiring a bone model that indicates the connection relationship of parts of the three-dimensional model; a first division means for dividing the bone model into parts; a setting means for setting a projection plane for projecting the three-dimensional model for each divided part based on the divided bone model; a generating means for generating a two-dimensional image onto which the three-dimensional model is projected based on each of the projection planes set for each of the divided regions; an encoding means for encoding the two-dimensional image for each of the divided regions; An image processing device comprising: (Item 2) Item 1. The image processing device according to item 1, wherein the setting means further sets a front direction at the time of projection for each of the divided parts, and sets the projection surface based on the front direction. (Item 3) The image processing device described in item 2, characterized in that the setting means sets each face of a rectangular parallelepiped having a face perpendicular to the axis of the front direction as the projection plane, with the front direction as one of the coordinate axes of a spatial coordinate system. (Item 4) 4. The image processing device according to item 3, wherein the rectangular parallelepiped is a rectangular parallelepiped that surrounds one of the divided parts of the three-dimensional model. (Item 5) 2. The image processing device according to claim 1, further comprising a transmitting means for transmitting the encoded two-dimensional image and conversion information for decoding the encoded two-dimensional image to an external device. (Item 6) further comprising a second division means for dividing the three-dimensional model based on the divided bone model; Item 1. The image processing device according to item 1, wherein the setting means sets the projection plane for each divided part based on the three-dimensional model divided by the second dividing means. (Item 7) 2. The image processing device according to item 1, wherein the three-dimensional model is a three-dimensional point cloud model. (Item 8) an acquisition means for acquiring a plurality of encoded two-dimensional images; decoding means for decoding the encoded two-dimensional images; a generation means for generating a division model by dividing a three-dimensional model into parts based on the decoding results of each of the plurality of two-dimensional images; a synthesis means for generating the three-dimensional model by synthesizing the divided models; An image processing device comprising: (Item 9) obtaining a bone model that indicates the connection relationship of parts of the three-dimensional model; Dividing the bone model into parts; setting a projection plane for projecting the three-dimensional model for each divided part based on the divided bone model; generating a two-dimensional image by projecting the three-dimensional model based on each of the projection planes set for each of the divided regions; encoding the two-dimensional image for each of the divided regions; An image processing method comprising: (Item 10) obtaining a plurality of encoded two-dimensional images; decoding the encoded two-dimensional images; generating a division model by dividing the three-dimensional model into parts based on the decoding results of the plurality of two-dimensional images; generating the three-dimensional model by combining the segmented models; An image processing method comprising: (Item 11) A program for causing a computer to function as each means of the image processing device described in any one of items 1 to 7. (Item 12) A program for causing a computer to function as each means of the image processing device described in item 8.

[0063] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0064] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0065] 101: Image processing system, 102: Image processing device, 103: Image processing device, 104: Imaging range, 105: Imaging target, 106: Camera, 107: Image generating device

Claims

1. an acquisition means for acquiring a bone model that indicates the connection relationship of parts of the three-dimensional model; a first dividing means for dividing the bone model into parts; a setting means for setting a projection plane for projecting the three-dimensional model for each divided part based on the divided bone model; a generating means for generating a two-dimensional image by projecting the three-dimensional model based on each of the projection planes set for each of the divided regions; an encoding means for encoding the two-dimensional image for each of the divided regions; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein said setting means further sets a front direction at the time of projection for each of said divided portions, and sets said projection surface based on said front direction.

3. 3. The image processing device according to claim 2, wherein the setting means sets the front direction as one of the coordinate axes of a spatial coordinate system, and sets each face of a rectangular parallelepiped having a face perpendicular to the axis of the front direction as the projection plane.

4. The image processing device according to claim 3 , wherein the rectangular parallelepiped is a rectangular parallelepiped that surrounds one of the divided portions of the three-dimensional model.

5. 2. The image processing apparatus according to claim 1, further comprising a transmitting means for transmitting the encoded two-dimensional image and conversion information for decoding the encoded two-dimensional image to an external device.

6. further comprising a second division means for dividing the three-dimensional model based on the divided bone model; 2. The image processing apparatus according to claim 1, wherein said setting means sets the projection plane for each divided part based on the three-dimensional model divided by said second dividing means.

7. 2. The image processing device according to claim 1, wherein the three-dimensional model is a three-dimensional point cloud model.

8. an acquisition means for acquiring a plurality of encoded two-dimensional images; decoding means for decoding the encoded two-dimensional images; a generation means for generating a division model by dividing a three-dimensional model into parts based on the decoding results of the plurality of two-dimensional images; a synthesis means for generating the three-dimensional model by synthesizing the divided models; An image processing device comprising:

9. obtaining a bone model that indicates the connection relationship of parts of the three-dimensional model; Dividing the bone model into parts; setting a projection plane for projecting the three-dimensional model for each divided part based on the divided bone model; generating a two-dimensional image by projecting the three-dimensional model based on each of the projection planes set for each of the divided regions; encoding the two-dimensional image for each of the divided regions; An image processing method comprising:

10. obtaining a plurality of encoded two-dimensional images; decoding the encoded two-dimensional images; generating a division model by dividing the three-dimensional model into parts based on the decoding results of the plurality of two-dimensional images; generating the three-dimensional model by combining the segmented models; An image processing method comprising:

11. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 7.

12. A program for causing a computer to function as each of the means of the image processing apparatus according to claim 8.

Images