Image processing device, image processing method, and program

Abstract

The present invention improves compression efficiency in the temporal direction of a video. In the present invention, a bone model indicating a connection relationship between three-dimensional model parts is acquired. The bone model is divided into units of parts. On the basis of the bone model having been divided, a projection surface for projecting the three-dimensional model is set for each divided part. On the basis of the projection surface set for each divided part, a two-dimensional image which results from projection of the three-dimensional model is generated. Encoding of the two-dimensional image is performed for each divided part.

Description

Image processing device, image processing method, and program

[0001] The present invention relates to an image processing device, an image processing method, and a program.

[0002] A virtual viewpoint video generation system is a technology that can create an image seen from a virtual viewpoint specified by a user from images captured by an imaging system using multiple cameras and play it as a virtual viewpoint video. In recent years, a technology that compresses and distributes three-dimensional model data of a subject generated by the virtual viewpoint video generation system and allows users to display the virtual viewpoint video on their own devices such as PCs, HMDs (head-mounted displays), or tablets has been attracting attention.

[0003] In the technology disclosed in Patent Document 1, parameters related to multiple point cloud models are converted, and a hexahedral projection surface is set for the multiple point cloud models. Next, a two-dimensional planar image projected from the set projection surface is generated, and the generated two-dimensional image is encoded to generate coded data of the two-dimensional image and a bitstream including information related to the parameter conversion. By decoding the coded two-dimensional image, it is possible to display a virtual viewpoint video on a user's terminal.

[0004] International Publication No. 2020 / 012968 Japanese Patent Application Laid-Open No. 2020-29777

[0005] In the technology described in Patent Document 1, a hexahedron that covers each subject is basically set as the projection surface, but if part of the subject is hidden, the shape cannot be encoded correctly. Furthermore, in the technology described in Patent Document 1, when performing motion compensation encoding on a two-dimensional planar image, if the subject moves quickly, the difference in the time direction during video compression increases, resulting in a decrease in the overall compression rate. A decrease in compression rate increases the amount of data transmitted, which may hinder smooth distribution.

[0006] An object of the present invention is to improve the efficiency of video compression in the time direction.

[0007] To achieve the object of the present invention, for example, an image processing device according to one embodiment includes the following configuration: an acquisition means for acquiring a bone model that indicates the connection relationships between parts of a three-dimensional model, a first division means for dividing the bone model into parts, a setting means for setting, for each divided part, a projection plane onto which the three-dimensional model is projected based on the divided bone model, a generation means for generating, based on each of the projection planes set for each divided part, a two-dimensional image onto which the three-dimensional model is projected, and an encoding means for encoding the two-dimensional image for each divided part.

[0008] To improve the efficiency of video compression in the time direction.

[0009] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0010] The accompanying drawings are included in the specification, constitute a part thereof, illustrate embodiments of the present invention, and are used, together with the description thereof, to explain the principles of the present invention. A diagram showing an example of the configuration of an image processing system. A block diagram showing an example of the functional configuration of an image processing system. , and a diagram for explaining part information and part connection information of a 3D model. A diagram for explaining the relationship between a 3D model and a bone model. , and a diagram for explaining part division of a bone model. , and a diagram for explaining division of a 3D model. , and a diagram for explaining processing for setting the front direction in a divided point cloud model. , and a diagram for explaining processing for setting a projection plane for each divided part. A diagram for explaining processing for generating a 2D image on a projection plane. A diagram showing an example of the format of transmission data. A diagram showing an example of the configuration of a sequence header. A diagram showing an example of the configuration of a frame header. A diagram showing an example of the configuration of a header for data in units of divided parts. A diagram showing an example of the configuration of encoded data of a projected 2D image. A diagram showing a connection image when combining divided point cloud models. A flowchart showing an example of image processing by an image processing device performing encoding. A flowchart showing an example of image processing by an image processing device performing decoding.

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] 1 is a schematic diagram showing an example of an image processing system 101 including an image processing device according to this embodiment. The image processing system 101 according to this embodiment includes an image processing device 102, an image processing device 103 connected to the image processing device 102, and a video generation device 107.

[0013] The image processing device 102 according to this embodiment acquires a bone model that indicates the connection relationships between parts of a three-dimensional (3D) model, and divides the acquired bone model into parts. The image processing device 102 then sets a projection plane for projecting the 3D model for each divided part, and generates a two-dimensional image onto which the 3D model is projected based on each of the projection planes. Furthermore, the image processing device 102 encodes the two-dimensional image for each divided part.

[0014] In FIG. 1 , multiple cameras (image capture devices) 106 capture multiple captured images of an image capture target 105, and a 3D point cloud model of the image capture target is generated based on these captured images. In this embodiment, an example is described in which a 3D point cloud model of a human being generated in this manner is used as the 3D model to be processed, but this is not particularly limited as long as a bone model of the 3D model can be acquired. Herein, this 3D point cloud model is generated based on images captured by the cameras 106, but a 3D point cloud model of the image capture target 105 may be acquired in advance from an external device (not shown). Hereinafter, when simply referred to as a "point cloud model," this refers to the 3D point cloud model of the image capture target 105 to be processed, acquired in this manner.

[0015] The image processing device 102 divides the bone model, sets a projection plane for projecting a point cloud model for each divided region, and generates a two-dimensional image onto which the point cloud model is projected based on each of the projection planes. The image processing device 102 then encodes the two-dimensional image for each divided region. Meanwhile, the image processing device 103 decodes the two-dimensional image encoded by the image processing device 102. In this embodiment, the image generation device 107 regenerates the point cloud model based on the decoding result by the image processing device 103 and presents it to the user as a virtual viewpoint image. The image generation device 107 accepts user operations via a virtual viewpoint operation unit and generates a virtual viewpoint image based on the user input. Note that, although the following description is given assuming that the image processing device 103 performs the decoding process, the image processing device 102 may also be capable of performing a similar decoding process.

[0016] 2 is a block diagram showing an example of the functional configuration of the image processing system 101 of this embodiment. The image processing device 102 includes a model acquisition unit 201, a bone generation unit 202, a bone division unit 203, a division model generation unit 204, a projection plane direction setting unit 205, a projection plane setting unit 206, a two-dimensional (2D) image generation unit 207, a 2D image encoding unit 208, and a transmission unit 209.

[0017] The model acquisition unit 201 acquires a 3D model (point cloud model) to be processed. As described above, the model acquisition unit 201 can generate a point cloud model based on captured images captured by multiple cameras 106. The process of generating a point cloud model based on captured images can be performed using any known technology. For example, the model acquisition unit 201 separates a foreground region and a background region from the captured image using a commonly used known foreground / background separation technology. Next, the model acquisition unit 201 generates a foreground silhouette image by taking a logical product of the foreground region and the 3D point cloud model generation space projection region. Here, the 3D point cloud model generation space projection region is, for example, a predefined 3D point cloud model generation space reprojected onto the angle of view of each imaging device, and is set in advance via a UI screen (not shown). The generated foreground silhouette image may be an image with the same resolution as the captured image, or may be one or more images obtained by cutting out regions corresponding to one or more foreground regions with circumscribing rectangles. The model acquisition unit 201 can generate a three-dimensional point cloud model by a volume intersection method using the generated foreground silhouette image and preset camera parameters of the camera 106 .

[0018] The bone generation unit 202 generates a bone model of the imaging target 105 from the point cloud model acquired by the model acquisition unit 201. The bone model can be generated by a known method, and the method is not particularly limited. For example, the model generation unit 202 may estimate the bone model of the point cloud model by obtaining a posture model described in Patent Document 2.

[0019] The bone generation unit 202 stores in advance a three-dimensional standard shape model and its posture model that mimic a standard human shape. The standard shape model and posture model can be general models used in CG, and their file formats can be general formats such as FBX. The three-dimensional standard shape model is expressed as a three-dimensional point cloud model that includes information on the IDs (vertex IDs) of vertices that make up triangular or quadrangular faces and the coordinates of those vertices (vertex coordinates). Meanwhile, the bone model according to this embodiment is composed of part information that represents each part of the human body, such as the head, neck, arms, or ankles, and part connection information that represents the connection relationships between each part.

[0020] 3A to 3C are diagrams showing examples of part information and part connection information. As shown in FIG. 3A, part information according to this embodiment is composed of a 3D model ID, part codes, and joint point coordinates. The 3D model ID is a unique ID assigned to each bone model. The part code is a number assigned to each part of the bone model, and is predetermined here as shown in FIG. 3B. The part connection information according to this embodiment is composed of the 3D model ID, part codes, and junction part codes that represent the part codes of the parts to which each part is connected. By associating the parts of the standard shape model with the parts of the bone model, it is possible to change the posture of the bone model according to the posture of the standard shape model.

[0021] FIG. 4 is a diagram illustrating the relationship between a three-dimensional model (standard shape model) 401 and a bone model 402. The black circles in the bone model 402 represent joints, and the white circles represent junctions and end portions. The bone model 402 is composed of bones connecting the joints, junctions, and end portions. In this example, the position of the white circle representing the end portion of the head is the center of gravity of the head. In the posture estimation process, the standard shape model corresponding to the bone model is deformed so that the image projected onto each camera image matches the foreground area of ​​each camera, and the posture when the two images most closely match is determined as the bone model of the subject. Note that the bone model estimation method is not limited to this, and any method for generating bones for a 3D model can be used. For example, the bone generation unit 202 may estimate the two-dimensional posture of the imaged subject based on the two-dimensional images obtained from each camera, and then estimate the bone model based on the relationship between the cameras.

[0022] The bone splitting unit 203 splits the bone model generated by the bone generation unit 202 into parts (here, for each part) using part information representing each part of the bone model set by the bone generation unit 202 and part connection information representing the connection relationships between the parts. FIGS. 5A to 5C illustrate how the bone model is split into parts. FIG. 5A shows a bone model of an arm, and FIG. 5B shows a bone model obtained by splitting the bone model shown in FIG. 5A into each part. In this embodiment, the bone splitting unit 203 splits the bone model based on the coordinates of the joint points shown in FIG. 3A. When splitting the bone model, the joints connecting the parts may be included in both adjacent split parts as shown in FIG. 5B, or may be included in only one part as shown in FIG. 5C. Furthermore, a normal plane may be set at the joint point relative to a line connecting the joint points, and the part included between these normal planes may be considered as a single part after splitting (for example, the torso part may be a single part consisting of multiple joints / joints).

[0023] The model generation unit 204 divides the point cloud model acquired by the model acquisition unit 201 into parts using the bone model divided into parts generated by the bone division unit 203. Specifically, the model generation unit 204 divides the 3D point cloud model corresponding to the bone model in accordance with the division of the bone model by the bone division unit 203. FIGS. 6A and 6B show conceptual diagrams of the division of the 3D model. FIG. 6A is a diagram showing an arm bone model corresponding to FIG. 5A , and FIG. 6B shows a bone model obtained by dividing the bone model shown in FIG. 6A into parts corresponding to FIG. 5B . The point cloud model is divided from the point cloud model shown in FIG. 6A into the point cloud model shown in FIG. 6B using a plane normal to a line connecting joint points as a cutting plane. In the examples of FIGS. 6A and 6B , the right upper arm of the bone model 402 shown in FIG. 6A is divided into divided bone models 601a, 601b, and 601c in FIG. 6B . Based on these bone models 601a, 601b, and 601c, the model generation unit 204 divides the right upper arm of the point cloud model 401 into point cloud models 602a, 602b, and 602c, and acquires each of these as a divided point cloud model (hereinafter, sometimes simply referred to as a divided point cloud model). Hereinafter, some parts of the bone models and point cloud models divided in this way may be referred to as "divided parts."

[0024] The projection plane direction setting unit 205 defines the direction of the coordinate axes for setting the projection plane for each divided region (each divided point cloud model) generated by the model generation unit 204. By defining the direction of the coordinate axes for each region and defining the front direction during projection, it is possible to always obtain a projected 2D image of that region from the same direction. Here, for a certain region, the front direction is set as one of the coordinate axes of the spatial coordinate system (hereinafter, the positive x-axis direction). The method of defining the direction of the coordinate axes and the front direction may differ for each region or may be consistent for all regions. Below, with reference to Figures 7A and 7B, an example of the process of defining the direction of the coordinate axes and the front direction by the projection plane direction setting unit 205 will be specifically described.

[0025] 7A and 7B are diagrams illustrating conceptual directions defined for each divided point cloud model. For example, for the head, the projection plane direction setting unit 205 determines the direction of the front of the face as the forward direction (positive x-axis direction) based on the shape of the point cloud model, as shown in FIG. 7A , and defines the line connecting the neck joint point and the head endpoint on the bone model as the z-axis. Here, the projection plane direction setting unit 205 defines the head endpoint side on the bone model as the positive z-axis direction, the line normal to the x-axis and normal to the z-axis as the y-axis, and defines the right side of the forward direction as the positive y-axis direction. For the upper arm, the projection plane direction setting unit 205 defines the line connecting the shoulder joint point and the elbow joint point on the bone model as the z-axis, as shown in FIG. 7B , and defines the elbow joint point side on the bone model as the positive z-axis direction. Next, the projection plane direction setting unit 205 defines the line perpendicular to the z-axis and parallel to the wrist as the x-axis, and defines the wrist side on the bone model as the positive x-axis direction (forward direction). The projection surface direction setting unit 205 defines the y-axis as a line that is normal to both the x-axis and the z-axis, and defines the right side of the forward direction as the positive direction of the y-axis. For the right and left hands, the palm may be defined as the forward direction, or the upper arm may be defined as the joint point and the direction facing the chest. In this manner, in this embodiment, the direction is defined using the lines connecting the joint points and the endpoints, and for the hands and feet, the forward direction is determined using the z-axis as a line connecting the wrist joint point and the endpoint of the middle finger.

[0026] In this way, the definition of the coordinate axes for setting the projection plane according to this embodiment differs for each part. Furthermore, the projection plane direction setting unit 205 may perform the above-described direction definition based on the bone model alone, or may perform the direction definition using both the bone model and the point cloud model. Furthermore, the method of direction definition is not limited to the above-described example, and the user may be able to arbitrarily set the direction for each part.

[0027] The projection plane setting unit 206 sets a projection plane for projecting the point cloud model for each divided body part based on the divided bone model. Here, the projection plane setting unit 206 sets multiple projection planes for each body part using the divided point cloud model generated by the model generation unit 204 and the coordinate axes for each body part set by the projection plane direction setting unit 205. Images of the projection plane setting process are shown in FIGS. 8A and 8B . FIG. 8A shows a 3D point cloud model of the head in which the directions of the coordinate axes are defined by the projection plane direction setting unit 205. Using these coordinate axes, projection planes (here, six for one body part) are set to surround the 3D point cloud model, as shown in FIG. 8B . Here, six projection planes are generated so that each face of a rectangular parallelepiped having a surface perpendicular to the coordinate axis (front direction) serves as a projection plane.

[0028] An example of the projection plane setting process shown in FIG. 8B will be described in detail below. The projection plane setting unit 206 first determines a projection range for each part. Here, the projection plane setting unit 206 sets a projection range so as to cover the division point cloud model using the division point cloud model generated by the model generation unit 204. Next, the projection plane setting unit 206 sets a rectangular parallelepiped that includes the set projection range. Each face of the rectangular parallelepiped is set so as to be parallel to one of the xy plane, yz plane, and zx plane using the coordinate axes for each part set by the projection plane direction setting unit 205. Each face of the set rectangular parallelepiped becomes the projection plane for that part.

[0029] The 2D image generation unit 207 generates a two-dimensional image (projected two-dimensional image) by projecting the point cloud model based on each of the multiple projection planes set by the projection plane setting unit 206. In this embodiment, the 2D image generation unit 207 generates a projected two-dimensional image by projecting the surface of the point cloud model onto the projection plane. FIG. 9 shows an image diagram of the image generation process. FIG. 9 is a diagram showing a three-dimensional point cloud model of the head for which the projection planes have been set by the projection plane setting unit 206. Projection plane X+ is the projection plane located in the positive direction of two projection planes parallel to the xy plane, and projection plane X- is the projection plane located in the negative direction of the two. Similarly, projection plane Y+ is the projection plane located in the positive direction of two projection planes parallel to the xy plane, and projection plane X- is the projection plane located in the negative direction of the two, and the same applies to Z+ and Z-. Specifically, the projection plane setting unit 206 generates a projected two-dimensional image by projecting the divided point cloud model onto a two-dimensional plane defined as the projection plane. In this embodiment, for ease of explanation, six projection planes are set for one part to form a hexahedron of projection planes, but the number and directions of the projection planes are not limited to this. For example, twelve projection planes may be set for one part, and the projection planes may form a regular dodecahedron surrounding the point cloud model.

[0030] The 2D image encoding unit 208 encodes the projected two-dimensional image for each division point cloud model generated by the 2D image generation unit 207. The 2D image encoding unit 208 can encode the projected two-dimensional image by arbitrarily adopting a commonly used image encoding method. For example, the 2D image encoding unit 208 may encode the projected two-dimensional image by an encoding method for two-dimensional planar images, such as JPEG (Joint Photographic Experts Group) or AVC (Advanced Video Coding).

[0031] When encoding video, if a part of the subject is hidden (for example, if the face is covered by the hand during dancing) or if the subject is moving at high speed, the difference in the time direction of the video tends to increase, resulting in a lower compression rate. On the other hand, by encoding the projected 2D image for each segmented point cloud model as described above, it is no longer necessary to consider unnecessary differences in the time direction for the entire subject, making it possible to further improve the compression rate. Furthermore, by setting the front direction for each part, it is possible to suppress changes in shape over time, and the encoding efficiency can be improved through motion compensation encoding.

[0032] The transmission unit 209 transmits the 2D image encoded by the 2D image encoding unit 208 and the conversion information required for decoding to an external device (here, the image processing device 103 that performs the decoding process). FIG. 10 shows an example of the format of data transmitted to the image processing device 103. Hereinafter, the entire data transmitted from the image processing device 102 to the image processing device 103 will be referred to as a sequence. A sequence begins with a Sequence Header. The Sequence Header contains information about the point cloud model to be encoded. Here, the point cloud model is encoded frame by frame, and the sequence contains an item, Frame Data, for managing information on such frame by frame. The Frame Data begins with a Frame Header containing information about the frame, and the Frame Header contains Part Data divided and encoded for each part. The Part Data contains a Parts Header containing information about the part and information required for decoding. The Part Data also contains a Projection Image Header representing the content of each projected 2D image, and the Projection Image Header is followed by the Projection Image Data, which is the encoded data for the actual image.

[0033] FIG. 11 shows an example of the structure of a Sequence Header. In the example of FIG. 11, the Sequence Header includes Sequence Start_Code, Model_ID, Model_location, Start_Frame_Time, Frame_rate, number_of_frames, user_data_frag, and user_data, in this order. Sequence_Start_Code indicates the beginning of the Sequence Header. Model_ID is information for identifying the encoded 3D point cloud model. Model_location is information indicating the location of the encoded point cloud model in virtual space. Start_Frame_Time indicates the date and time or time of the first frame of the point cloud model. The expression of date and time is not particularly limited; for example, Greenwich Mean Time may be used to express date and time or hours, minutes, and seconds, with a time code written at the end. The same applies to the expression of date and time below. Frame_rate indicates the frame rate of the video, and number_of_frame indicates the number of frames included in the sequence. user_data_flag is flag information that indicates, with 0 or 1, whether meta information is included as user_data, and user_data is assigned when user_data_flag is 1. user_data is user data generated in units that can be collectively managed, such as the event, venue, or date and time at which the 3D point cloud model was generated. Note that the data stored in the Sequence Header is not limited to these, and may include additional information or may be omitted in part.

[0034] Next, the Frame Data will be described in detail. The Frame Data includes a Frame Header that represents a frame, followed by Part Data, which is data for each divided part of the point cloud model. FIG. 12 shows an example of the configuration of a Frame Header. In the example of FIG. 12, the Frame Header includes Frame_Start_Code, Frame_Data_Time, number_of_Part_Data, and size_of Part_Data[1..p], in this order. Frame_Start_Code indicates the beginning of the Frame Header. Frame_Date_Time indicates the date and time of the first frame of the point cloud model. number_of_Part_Data indicates the number p of part data included in the frame. size_of Part_Data is an item that stores the data size (e.g., in bytes) of the Part Data for each part in encoded order. This data structure enables access to the Part Data for each part. Furthermore, the size of the Frame Data can be calculated by adding up the sizes of all the Part Data, enabling access to the next Frame Data.

[0035] Next, the details of the Part Data will be explained. The Part Data includes a Part Header that represents a frame, followed by Part Data, which is data for each divided part of the point cloud model. Fig. 13 shows an example of the configuration of the Part Header. In the example of Fig. 13, the Part Header includes Part_Start_Code, Part_ID, number_of_Parts, Part_code[1..c], Part_location, Connection_1 st _Point_location, Connection_1 st _Parts_ID, Connection_2 nd _Point_location, Connection_2 nd_Parts_ID, Frontal_vector, number_of_Projection_Images, and size_of_Projection_Image[1..pr] are included in this order. Part_Start_Code indicates the start of the Part Header. Part_ID includes an ID that identifies the data of the divided parts within the frame. Since one Part Data can include multiple parts, the number c of parts included in the division point cloud model is stored in number_of_Parts. Part_code stores the code of the part included in the division point cloud model, for example, the code shown in Figure 3B. Part_location stores the position of the division point cloud model, for example, the position of the center of gravity, in three-dimensional coordinates. Connection_1 st _Point_location stores the coordinates of the joint point between the divided part and the adjacent part, for example, joint point 1 in FIG. 3A, and the ID of the adjacent part that joins at that joint point is Connection_1. st _Part_ID. Furthermore, the coordinates of the joint point between the divided part and the adjacent part, for example, joint point 2 in FIG. 3A, are stored in Connection_2. nd _Point_location, and the ID of the adjacent part that joins at that junction point is stored in Connection_2 nd The ID of the adjacent part to be joined may be omitted. Frontal_vector contains information indicating the front direction of the part. There are no particular limitations on the method of describing Frontal_vector, and it may be expressed, for example, in polar coordinates indicating the front direction from the position of the division point cloud model indicated in Part_location. number_of_Projection_Images contains the number pr of encoded projection 2D images, and size_of_Projection_Image describes the size of the encoded data in the order in which it is encoded and stored.

[0036] Finally, we will explain the coded data of a projection 2D image. Hereinafter, when simply referred to as "coded data," it refers to data obtained by coding a projection 2D image. The coded data consists of a Projection Image Header item that describes information necessary for decoding and reconstructing point cloud data of a region, and coded data of the actual image, Projection Image Data.

[0037] FIG. 14 shows an example of the configuration of a Projection Image Header. In the example of FIG. 14, the Projection Image Header includes Projection_Image_Start_Code, Projection_Image_ID, Projection_Image_Coding_method, Visual_camera_Location, and Visual_camera_Rotation_Quaternion, in this order. Projection_Image_Start_Code represents the beginning of the Projection Image Header. Projection_Image_ID includes an ID that identifies the projected 2D image data. Projection_Image_Coding_method includes information indicating the coding method for the projected 2D image. The description method in Projection_Image_Coding_method is not particularly limited, and may be, for example, the ISO number of the coding method, the name of the coding method, or a list number corresponding to the coding method. Virtual_camera_Location includes information indicating the position of the projection point (generally a virtual camera in a virtual viewpoint video) that generates the projected 2D image. Here, the position of the projection point is expressed in three-dimensional coordinates. The position of the projection point according to this embodiment may be expressed in the world coordinate system, or may be expressed as a relative position from the position of the corresponding divided portion. Virtual_camera_Rotation_Quaternion represents the orientation of the virtual camera, and here, the orientation of the virtual camera is represented by three values ​​of quaternion.

[0038] In addition, if the encoding method for the projected two-dimensional image is a motion compensation encoding method such as MPEG or multi-view encoding, encoding can be achieved by including in the Projection Image Header the date and time or time information of the frame indicating the referenced Part Data, an identification ID of the division part, or an ID that identifies the projected two-dimensional image.

[0039] Next, we will explain the decoding process executed by the image processing device 103. As described above, the image processing device 103 decodes the two-dimensional image encoded by the image processing device 102, and includes a receiving unit 210, a decoding unit 211, a model regeneration unit 212, a model synthesis unit 213, and an output unit 214.

[0040] The receiving unit 210 receives multiple coded 2D images and transformation information transmitted from the transmitting unit 209 of the image processing device 102. The decoding unit 211 encodes the 2D images for each divided portion. Here, the decoding unit 211 receives the Sequence data received by the receiving unit 210 and decodes each header to obtain information necessary for decoding and synthesizing the 2D images. Next, the decoding unit 211 decodes the coded data (Projection Image Data) of the projected 2D images using the decoding method specified by Projection_Image_Coding_method. The decoding unit 211 decodes all projected 2D images indicated by number_of_Projection Images.

[0041] The model regeneration unit 212 generates (regenerates) a division model (division point cloud model) by dividing the 3D model of the imaging target 105 into parts, using the projected 2D images decoded by the decoding unit 211. Specifically, the model regeneration unit 212 generates a point cloud model by projecting the projected 2D images for each divided part into a 3D space based on the descriptions of Virtual_camera_Location and Virtual_camera_Rotation Quaternion. The method used by the model regeneration unit 212 to generate the point cloud model is not particularly limited, and the above-mentioned volume intersection method may be used, for example.

[0042] The model synthesis unit 213 synthesizes the divided point cloud models generated by the model regeneration unit 212 into a single point cloud model. Specifically, the model synthesis unit 213 determines how to connect the divided point cloud models and synthesizes them using information such as the position of the part, the joint position, and the front direction described in the Part Header received by the receiving unit 210. FIG. 15 shows an image of connections when synthesizing divided point cloud models. For example, the model synthesis unit 213 can generate a single point cloud model by acquiring the coordinates of the joint points included in each divided point cloud model as shown in FIG. 15 from Part_location and arranging the divided point cloud models based on these coordinates. Here, the model synthesis unit 213 sets the front direction to the direction described in Frontal_vector and st Connection_2 from the first connection position of the part included in _Point_location nd The point cloud model is regenerated by arranging each divided point cloud model at the second joining position included in _Point_location. Note that the process of regenerating the point cloud model is not limited to this, and for example, the point cloud model may be generated using information on the bone model. For example, the bone model information can be transmitted from the image processing device 102 to the image processing device 103 by entering all of the data shown in FIG. 3A in user_data shown in FIG. 11.

[0043] The output unit 214 outputs the 3D point cloud model generated by the model synthesis unit 213 to the image generation device 107. The image generation device 107 generates a virtual viewpoint image using the 3D point cloud model in a virtual viewpoint image generation unit. This generation process can be achieved by arranging bone models so as to correspond to the part code Part_code[1..c] of each part. The image generation process and the generated images can be performed using known techniques used for image generation using 3D models, so detailed explanations will be omitted here.

[0044] 16 is a flowchart showing an example of a series of processes in the image processing system 101 of this embodiment, from acquiring a point cloud model from a captured image to transmitting an encoded two-dimensional image to the image processing device 103. A detailed description will be given below with reference to the flowchart in FIG.

[0045] In S1600, the image processing system 101 synchronizes and controls the multiple cameras 106 to start capturing an image of the object 105 within the imaging range 104. In S1601, the model acquisition unit 201 generates a foreground silhouette from the image captured in the frame to be processed, and then acquires a point cloud model. By default, the frame to be processed is the first frame after imaging has started.

[0046] In S1602, the bone generation unit 202 determines whether a bone model can be generated from the point cloud model generated in S1601. If a bone model can be generated, the process proceeds to S1603; if not, the process proceeds to S1604. The bone generation unit 202 according to this embodiment determines that a bone model can be generated if bone models of all parts of the imaging target 105 can be acquired based on the captured image acquired in S1600. An example in which a bone model cannot be generated is when there are multiple imaging targets 105 within the imaging range 104 and these imaging targets are crowded together, so that part of the body of the imaging target is hidden and a bone model cannot be acquired.

[0047] In S1603, the bone generation unit 202 generates a bone model from the point cloud model generated in S1601, and the process proceeds to S1605. In S1604, the projection plane setting unit 206 sets a projection plane for the entire point cloud model as a single processing target, and the process proceeds to S1611.

[0048] In S1605, the bone dividing unit 203 divides the generated bone model into parts based on the joint points and end points, and generates a bone model for each divided part. In S1606, the model generating unit 204 generates a point cloud model for each divided part based on the bone model for each divided part generated in S1603.

[0049] In S1607, the image processing device 102 acquires information necessary to be written in the Part Header, i.e., information necessary for reconstructing the part, such as the part's identification ID within the frame, the part's code, and information on the joining position, for each point cloud model for each divided part generated in S1606.

[0050] In S1608, the projection plane direction setting unit 205 sets coordinate axes for setting the projection plane (setting the coordinates of the virtual camera) for each point cloud model for each divided portion generated in S1606. In S1609, the projection plane direction setting unit 205 determines the front direction for each point cloud model divided between frames, and saves the coordinate settings set in S1608. Here, if coordinate settings have already been saved, the saved coordinate settings are reflected.

[0051] In S1610, the projection plane setting unit 206 sets a projection plane for each divided point cloud model using the coordinates and orientation information of the multiple virtual cameras for each determined divided point cloud model.

[0052] In S1611, the 2D image generation unit 207 generates a projected 2D image for each division point cloud model based on the projection plane set in S1610.

[0053] In S1612, the 2D image encoding unit 208 encodes the projected 2D image generated in S1611 for each division point group model.

[0054] In S1613, the transmission unit 209 transmits the information necessary for reconstructing the body part and the projected two-dimensional image generated in S1607 and S1612 to the image processing device 103 that performs decoding processing. In S1614, the image processing device 102 determines whether to continue imaging. If imaging is to be continued, the frame to be processed is set to the next frame, and the process returns to S1601; if not, the process in Fig. 16 ends. Here, it is assumed that if the user has performed an operation to end imaging, it is determined that imaging will not be continued.

[0055] Note that in Figure 16, an example has been described in which a point cloud model is generated based on a captured image, but if a pre-acquired point cloud model is used, S1600 is omitted and the pre-acquired point cloud model is acquired in S1601.

[0056] 16 has been described as an example in which a bone model is generated from a point cloud model, but if a bone model can be generated, it is not necessary to generate it from a point cloud model. For example, the bone generation unit 202 may generate a bone model from a silhouette image.

[0057] 17 is a flowchart showing an example of a series of processes performed by the image processing device 103 according to this embodiment, from receiving an encoded projected two-dimensional image to decoding it and outputting a regenerated point cloud model. A detailed description will be given below with reference to the flowchart in FIG.

[0058] In S1700, the receiving unit 210 receives the coded header information and the coded data of the projected two-dimensional image transmitted from the image processing device 102.

[0059] In S1701, the decoding unit 211 performs a decoding process on the encoded projected two-dimensional image received in S1700. In S1702, the model regeneration unit 212 regenerates a division point cloud model based on the projected two-dimensional image decoded in S1701 and information necessary for reconstructing the region.

[0060] In S1703, the model synthesis unit 213 synthesizes point cloud models using the divided point cloud models regenerated in S1702. Information necessary for synthesis is received in S1700.

[0061] In S1704, the output unit 214 outputs the point cloud model synthesized in S1703. In S1705, the receiving unit 210 checks whether an encoded projected two-dimensional image for the next frame has been transmitted. If no projected two-dimensional image has been transmitted, it is determined that imaging has ended, and the processing ends. If a new projected two-dimensional image has been transmitted, the processing returns to S1700, and the new projected two-dimensional image is set as the processing target.

[0062] According to this processing, the bone model of the 3D model can be divided into parts, and a projection plane can be set for each divided part based on the divided bone model. Then, a projected 2D image can be generated and encoded based on each of the projection planes set for each divided part. Therefore, when compressing video, it is possible to reduce the difference in the time direction in the projected image and improve the compression rate compared to when a projection plane is set for the entire image capture target. Furthermore, by setting the front direction for each part, it is possible to suppress changes in shape over time, thereby improving the encoding efficiency of motion compensation encoding.

[0063] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0064] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0065] This application claims priority based on Japanese Patent Application No. 2024-060968, filed April 4, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: an acquisition means for acquiring a bone model that indicates the connection relationships of parts of a three-dimensional model; a first division means for dividing the bone model into parts; a setting means for setting a projection plane for projecting the three-dimensional model for each divided part based on the divided bone model; a generation means for generating a two-dimensional image onto which the three-dimensional model is projected based on each of the projection planes set for each divided part; and an encoding means for encoding the two-dimensional image for each divided part.

2. The image processing device according to claim 1, wherein said setting means further sets a front direction at the time of projection for each of said divided portions, and sets said projection surface based on said front direction.

3. The image processing device according to claim 2, wherein the setting means sets the front direction as one of the coordinate axes of a spatial coordinate system, and sets each face of a rectangular parallelepiped having a face perpendicular to the axis of the front direction as the projection plane.

4. The image processing device according to claim 3, wherein the rectangular parallelepiped is a rectangular parallelepiped that surrounds one of the divided portions of the three-dimensional model.

5. An image processing device according to any one of claims 1 to 4, further comprising a transmitting means for transmitting the encoded two-dimensional image and conversion information for decoding the encoded two-dimensional image to an external device.

6. An image processing device as claimed in any one of claims 1 to 5, further comprising a second division means for dividing the three-dimensional model based on the divided bone model, and wherein the setting means sets the projection plane for each divided part based on the three-dimensional model divided by the second division means.

7. The image processing device according to any one of claims 1 to 6, wherein the three-dimensional model is a three-dimensional point cloud model.

8. An image processing device comprising: an acquisition means for acquiring a plurality of encoded two-dimensional images; a decoding means for decoding the encoded two-dimensional images; a generation means for generating a division model by dividing a three-dimensional model into parts based on the decoding results of each of the plurality of two-dimensional images; and a synthesis means for generating the three-dimensional model by synthesizing the division models.

9. An image processing method comprising: a step of obtaining a bone model that indicates the connection relationships of parts of a three-dimensional model; a step of dividing the bone model into parts; a step of setting a projection plane for projecting the three-dimensional model for each divided part based on the divided bone model; a step of generating a two-dimensional image onto which the three-dimensional model is projected based on each of the projection planes set for each divided part; and a step of encoding the two-dimensional image for each divided part.

10. An image processing method comprising: a step of acquiring a plurality of encoded two-dimensional images; a step of decoding the plurality of encoded two-dimensional images; a step of generating a division model by dividing a three-dimensional model into parts based on the decoding results of each of the plurality of two-dimensional images; and a step of generating the three-dimensional model by combining the division models.

11. A program for causing a computer to function as each means of the image processing device according to any one of claims 1 to 7.

12. A program for causing a computer to function as each means of the image processing device according to claim 8.

Images