Information processing apparatus, information processing method, and program

Abstract

To provide a format of volumetric video with which selective drawing of a specific object can be efficiently performed.SOLUTION: An information processing apparatus acquires volumetric video data consisting of the sequence of frames including shape data of objects, and tracking information indicating a change between the frames of the objects corresponding to the shape data. The information processing apparatus generates, based on the tracking information, meta data that associates the objects and the shape data in the frames with each other, and outputs the volumetric video data including the meta data.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The techniques of this disclosure relate to file formats for volumetric video. [Background technology]

[0002] Patent Document 1 discloses a method for associating tracking information indicating the position and time of a subject (object) with each frame of a video as metadata, for the purpose of analyzing highlight scenes of sports, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-106661 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, volumetric video technology has been gaining attention. This technology converts the entire captured space into 3D data, enabling the generation of images from various angles in a virtual space created within a computer. This volumetric video allows viewers to control the viewing viewpoint (e.g., the position, orientation, and field of view of a virtual camera; hereafter referred to as the "virtual viewpoint"). To achieve optimal viewing of volumetric video of a soccer match held in a large space such as a stadium, for example, the virtual viewpoint must be constantly controlled according to the scene being viewed. Specifically, precise control is required to track the movements of a player of interest and keep the ball within the field of view. Furthermore, when viewing only a specific object in volumetric video, in addition to controlling the virtual viewpoint, a process is required to selectively render only that specific object.

[0005] In volumetric video data, one frame generally contains shape data representing the three-dimensional shapes of objects, the number of which is equal to the number of objects shown in the frame. When selectively rendering a specific object according to the input virtual viewpoint, it is necessary to determine which shape data to process and how. However, this cannot be achieved by simply associating tracking information with each frame using the method of Patent Document 1.

[0006] An object of the present disclosure is to enable easy selective drawing of specific objects. [Means for solving the problem]

[0007] The information processing device according to the present disclosure is characterized by having an acquisition means for acquiring volumetric video data consisting of a sequence of frames including shape data of objects, an acquisition means for acquiring tracking information indicating changes between frames of each object corresponding to the shape data, a generation means for generating metadata that associates each of the objects with the shape data in each frame based on the tracking information, and an output means for outputting the volumetric video data including the metadata. [Effects of the Invention]

[0008] According to the technology of the present disclosure, it is possible to easily perform selective drawing of a specific object. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 2] Block diagram showing the software configuration of the information processing device [Figure 3] Flowchart showing the process of adding metadata to volumetric video [Figure 4] (a) and (b) are schematic diagrams showing object tracking. [Figure 5] FIG. 1A is a diagram showing an example of object list information, and FIG. 1B is a diagram showing an example of metadata. [Figure 6] Block diagram showing the software configuration of the information processing device [Figure 7] Flowchart showing the process of adding metadata to volumetric video [Figure 8] Diagram illustrating identification of chapter objects [Figure 9] A diagram showing an example of metadata [Figure 10] A diagram showing an example of the result of sorting processing [Figure 11] A diagram showing an example of metadata [Figure 12] A diagram showing an example of metadata [Figure 13] 10(a) and 10(b) are diagrams showing an example of a UI screen used by an operator; [Figure 14] Block diagram showing the software configuration of the information processing device [Figure 15] A flowchart showing the flow of a process for selectively playing back an object. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following describes embodiments of the present invention with reference to the drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0011] Before describing the embodiments, let us first review the concept of "volumetric video." In this specification, volumetric video refers to shape data (also called a "3D model") that represents the three-dimensional shape of an object (subject), as opposed to conventional two-dimensional video data. ) may be added, and audio data may also be added. 3D models come in various data formats, such as volume data that represents the shape of an object using voxels, point cloud data that represents a set of points, and mesh data that represents a set of polygons. In this embodiment, mesh data will be used for explanation, but the data format of 3D models is not limited to this. 3D models may also have color information. For example, in volume data, each voxel typically has color and transparency information, in point cloud data, each point typically has color information, and in mesh models, each mesh typically has texture information.

[0012] In volumetric video, one or more 3D models representing the three-dimensional shapes of one or more objects are associated with each frame. Information about the shapes of the objects shown in a frame sequence, which is a chronological arrangement of such frames, is stored. An example of this type of volumetric video is Microsoft's (registered trademark) "High-quality Streamable free-viewpoint video." This stores texture images as video in the video track of the MPEG-DASH protocol, and stores mesh-format 3D models in an extended NAL (Network Abstraction Layer) unit.

[0013] [Embodiment 1] <Hardware configuration> FIG. 1 is a diagram illustrating an example of the hardware configuration of an information processing device 100 according to this embodiment. In FIG. 1, a CPU 101 uses a RAM 102 as a work memory, executes programs stored in a ROM 103 and a hard disk drive (HDD) 105, and controls the operation of each block (described later) via a system bus 112. An HDD I / F 104 is an interface, such as a serial ATA (SATA), that connects a secondary storage device, such as the HDD 105 or an optical disk drive. The CPU 101 can read data from and write data to the HDD 105 via the HDD I / F 104. Furthermore, the CPU 101 can load data stored in the HDD 105 into the RAM 102, and conversely, can save data loaded in the RAM 102 to the HDD 105. The CPU 101 can then execute the data loaded in the RAM 102 as a program. An input I / F 106 is a serial bus interface, such as a USB or IEEE 1394, that connects an input device 107, such as a keyboard, mouse, digital camera, or scanner. The CPU 101 can read data from an input device 107 via an input I / F 106. The output I / F 108 is a video output interface, such as DVI or HDMI (registered trademark), that connects the information processing device 100 to a display, which is an output device. The CPU 101 can send data to the display via the output I / F 108 and display a predetermined image on the display. The network I / F 110 is an interface that connects to a network such as the Internet or a LAN. The CPU 101 can exchange various types of data with an external server, etc., connected to the LAN or the Internet via the network I / F 110.

[0014] <Functional configuration and processing flow> 2 and 6 are block diagrams showing the software configuration of the information processing device 100, more specifically, the functional configuration for attaching metadata suitable for selectively rendering a specific object to a volumetric video. The various functions shown in FIGS. 2 and 6 are realized by the CPU 101 executing dedicated programs, and the types of metadata generated differ between FIGS. 2 and 6. Also, FIGS. 3 and 7 are flowcharts showing the flow of processing for attaching the above metadata to a volumetric video, and correspond to FIGS. 2 and 6, respectively.

[0015] Type 1 First, the process of adding suitable metadata to selectively drawing a specific object, which is realized by the functional configuration shown in the block diagram of Fig. 2, will be described with reference to the flowchart of Fig. 3. In the following description, the symbol "S" denotes a step.

[0016] In S301, the video reading unit 201 reads a frame sequence of the volumetric video to be processed from the HDD 105 or the like. Each frame constituting the frame sequence read here includes data of a 2D image at a specific time and data of a 3D model of an object shown in the 2D image. In this case, if multiple objects are shown in the 2D image, each 3D model generally represents the three-dimensional shape of a single object. However, if multiple objects are close to each other due to contact or the like, the 3D model represents the three-dimensional shape of the multiple objects combined together.

[0017] In S302, the tracking information acquisition unit 202 extracts and tracks objects from the 2D images of each frame constituting the frame sequence read in S301. For example, if three players and a ball are captured in a 2D image, four objects are extracted, and the 3D models of each frame are analyzed to determine the inter-frame correspondence for each of the four objects. At this time, the 3D models and objects may not have a one-to-one relationship. For example, a 3D model of a player touching the ball represents a single three-dimensional shape combining the player and the ball. Care must be taken because multiple objects may be assigned to a single 3D model. Figures 4(a) and 4(b) are schematic diagrams illustrating object tracking. Figure 4(a) shows mesh-format 3D models (hereinafter referred to as "mesh models") for six frames of a captured scene showing three players and a ball. Figure 4(b) shows the tracking results of one player (human) and the ball, respectively, using solid and dashed arrows. For the human, model indexes of "00", "00", "00", "01", "00", and "02" are assigned to each mesh model in frames 0 to 5. For the ball, model indexes of "02", "01", "00", "03", "01", and "03" are assigned to each mesh model in frames 0 to 5. Note that in the second frame, the human and the ball are represented by a single mesh model. The object tracking results obtained in this way are provided to the metadata generation unit 203 as tracking information indicating the transition of each object between frames.

[0018] In S303, the tracking information acquisition unit 202 generates list information of objects appearing in the 2D images of the frame sequence based on the tracking information obtained in S302. FIG. 5(a) shows an example of the object list information. In the list information shown in FIG. 5(a), object indices (here, "001" to "004") that identify each of the four objects shown in FIG. 4(a) are assigned, and each object index is associated with a model index in each frame. Note that for object index "002," the model index for the 0th frame is "-1," which means that no corresponding player object was present in the 0th frame.

[0019] In S304, the list information generated in S303 is displayed on the display (output device 109). At this time, a representative mesh model (for example, the one with the largest surface area) corresponding to each object index is also displayed.

[0020] In the next step S305, the metadata generation unit 203 sets attributes and tags for each object described in the list information, and generates metadata to be added to the frame sequence. In this embodiment, the operator refers to the list information and a representative mesh model displayed on the display, and inputs, using a keyboard or the like, whether the "attribute" is "human (person)" or a non-human "object." Furthermore, the operator inputs, using a keyboard or the like, identification information for distinguishing objects with the same attribute as the "tag." Based on the input results, the contents of the "attribute" and "tag" are set for each object represented by the object index, as shown in FIG. 5(b). In the example of FIG. 5(b), object indexes "001" to "003" correspond to the respective players, and object index "004" corresponds to the ball. In this case, the operator specifies "human" as the "attribute" for each of object indexes "001" to "003," and further adds, as the "tag," player-specific information such as the person's (player's) name, the name of the team they belong to, and their uniform number. Furthermore, for the object index "004," "thing" is specified as its "attribute," and information such as a common noun (here, "ball") that specifically identifies the object is added as a "tag." This makes it possible to identify the corresponding mesh model simply by specifying the "attribute" and "tag" content. In this way, the "attribute" and "tag" information set for each object index is obtained as metadata. Note that instead of having the operator manually input the attribute and tag information, they can be set automatically, for example, by analyzing the 2D image or mesh model in the frame. Specifically, the "attribute" and "tag" content can be automatically set based on the results obtained by character detection, face recognition, or detection of the size and color of the shape represented by the mesh model. Furthermore, the results of this automatic setting can be displayed on a display, allowing the operator to edit them.

[0021] In S306, the video output unit 204 adds the metadata obtained in S305 to the frame sequence read in S301 and outputs the frame sequence.

[0022] In this way, a volumetric video with metadata suitable for selectively drawing a specific object can be obtained.

[0023] Type 2 Next, the process of adding suitable metadata to selectively drawing a specific object, which is realized by the functional configuration shown in the block diagram of Fig. 6, will be described with reference to the flowchart of Fig. 7. In the following description, the symbol "S" denotes a step. In S701, similar to S301 in the flow of FIG. 3, the video reading unit 201 reads a frame sequence of the volumetric video to be processed from the HDD 105 or the like.

[0024] In the next step S702, the chapter division unit 601 divides the frame sequence read in S701 into a plurality of chapters. The chapter division unit 601 performs scene analysis on the input frame sequence, for example, to determine a group of temporally consecutive frames, such as each play segment, as a chapter section, and divides the frame sequence into chapters. For example, the six frames shown in FIG. 4(a) described in the first embodiment form one chapter. The method for determining chapter sections is not limited to this. For example, an operator may specify chapter sections with a mouse or the like while playing back the frame sequence to be processed, or may manually modify chapter sections automatically generated by scene analysis, or may divide the sequence frames at predetermined fixed time intervals.

[0025] In S703, the tracking information acquisition unit 202' extracts objects and tracks each extracted object for each divided chapter. The objects extracted, tracked, and identified in a specific chapter are called "chapter objects." The tracking results for each chapter object obtained in this step are provided to the metadata generation unit 203' and sorting unit 602 as tracking information.

[0026] In S704, the tracking information acquisition unit 202' identifies chapter objects between the previous and next chapters based on the tracking information acquired in S703, and generates a list of identified chapter objects. Specifically, for two adjacent chapters, the position of each chapter object in the last frame of the previous chapter and the position of each chapter object in the first frame of the next chapter are analyzed, and objects located close to each other are first identified as the same object. FIG. 8 is a diagram illustrating the identification of chapter objects. As shown in FIG. 8, if a video is divided into four chapters, identification is performed for the pair of chapters 1 and 2, the pair of chapters 2 and 3, and the pair of chapters 3 and 4. In the example of FIG. 8, chapter object 1A, chapter object 2A, chapter object 3B, and chapter object 4C are identified as the same object. Similarly, chapter object 1B, chapter object 2B, chapter object 3A, and chapter object 4B are identified as the same object. Similarly, chapter object 1C, chapter object 2C, chapter object 3C, and chapter object 4A are identified as the same object. Similarly, chapter object 1D and chapter object 2D are identified as the same object. A chapter object identified as the same object across multiple chapters in this way is called a "global object." Note that the method for identifying objects between chapters is not limited to the above-mentioned method; for example, the size of a mesh model (surface area or number of polygons) may also be used. Then, list information similar to that shown in FIG. 5(a) is generated for the global objects identified as described above. That is, an object index identifying each identified global object is assigned, and each object index is associated with a model index in each frame.

[0027] In S705, the global object list information generated in S704 is displayed on the display (output device 109), similar to S304 in the flow of Fig. 3. At this time, a representative mesh model (e.g., the one with the largest surface area) indicated by each object index is also displayed.

[0028] In S706, the metadata generation unit 203′ sets attributes and tags for each global object described in the global object list information, and generates metadata to be added to the frame sequence. As in S305 in the flow of FIG. 3, the operator references the list information and a representative mesh model displayed on the display, and inputs, using a keyboard or the like, information indicating a person or object as an “attribute” and details such as a name and affiliation as a “tag.” Based on the input results, the contents of the “attribute” and “tag” are set for each global object identified by the object index, as shown in FIG. 9. In the example of FIG. 9, object indexes “001” to “004” correspond to each player, and object index “005” corresponds to the ball. In this case, the operator specifies “person” as the “attribute” for the global objects indicated by each object index “001” to “004.” Furthermore, player-specific information such as the person’s (player’s) name, the name of the team they belong to, and their uniform number is added as a “tag.” Furthermore, for the object index "005," "thing" is specified as its "attribute," and information such as a common noun (in this case, "ball") that identifies the specific object is added as a "tag." This makes it possible to identify the corresponding mesh model simply by specifying the contents of the "attribute" and "tag." In this way, the "attribute" and "tag" information set for each object index of the global object is obtained as metadata.

[0029] In S707, the sorting unit 602 performs a process of rearranging the order of 3D models in each chapter based on the tracking information obtained in S703. In the rearrangement process of this embodiment, the surface area of ​​the mesh model corresponding to each chapter object is estimated, and the order within the chapter is rearranged so that the mesh models are arranged in descending order of surface area. The model index is also changed accordingly. FIG. 10 shows the results of the rearrangement process assuming that the six frames shown in FIG. 4(a) form one chapter. Here, if multiple chapter objects are in contact with each other and are represented by a single mesh model, the chapter object with the largest proportion of the total surface area is associated with the chapter object. In the example of FIG. 10, the mesh model 1001, in which a player and a ball are integrated, has a larger proportion of the portion representing the player's shape, and is therefore associated with the player's chapter object (model index "00").

[0030] In S708, the chapter metadata generation unit 603 generates metadata for the 3D model that has been rearranged by chapter. In this embodiment, where a mesh model is used as the 3D model, the following process is performed. First, a chapter of interest is selected, and for each model index in the chapter, the maximum number of polygons, the maximum number of vertices, and the circumscribing rectangle of the mesh model in each frame are calculated, and metadata associated with each mesh model is generated. FIG. 11 shows the data structure of a "chapter mesh model table" generated as metadata for each mesh model. In the chapter mesh model table shown in FIG. 11, "Bbox" refers to the circumscribing rectangle (bounding box) of the mesh model and is associated with the model index. This chapter mesh model table enables efficient rendering of mesh models. For example, the buffer size for mesh processing can be determined based on the maximum number of vertices and the number of polygons. In addition, the bounding rectangle for each frame enables efficient culling during rendering (a technique for disabling the rendering of an object when it is obscured by other objects and cannot be seen from a virtual viewpoint). Furthermore, the chapter metadata generation unit 603 generates metadata associated with each chapter object, including the index of the global object corresponding to the chapter object, three-dimensional position information for each frame, and a model index. If a mesh model corresponds to multiple objects, the unit analyzes the integration status of the objects and includes information about the integrated objects in the metadata. Figure 12 shows the data structure of a "chapter object table" generated as metadata for each chapter object in the target chapter (see Figure 10 above) after the sorting process. As shown in Figure 12, additional information "GO" is added to identify the global object from the chapter object, and additional information "CONV" is added to identify the other chapter objects with which the chapter object will be integrated in the case of an integrated relationship. In this case, the index of the global object is written in "GO," and the index of the other chapter object to be integrated is written in "Conv."

[0031] In S709, the video output unit 204 adds the metadata obtained in S706 and S708 to the frame sequence made up of each chapter that has been rearranged in S707, and outputs the frame sequence.

[0032] The above is the operation of each part and the chronological processing flow for generating volumetric video with type 2 metadata.

[0033] Volumetric video in the Type 2 format allows for chapter-based management and efficient management of 3D models, including the ability to swap objects between chapters. Furthermore, mesh models can be organized based on the tracking results of each object, and metadata can be added to each mesh model, enabling more efficient rendering.

[0034] <Selective playback of objects> Next, a method for selectively playing back an arbitrary object using metadata added to a volumetric video will be described using the above-mentioned pattern 2 as an example.

[0035] 13(a) and 13(b) are diagrams showing an example of a UI screen used by an operator, FIG. 14 is a block diagram showing the software configuration (functional configuration) of the information processing device 100, and FIG. 15 is a flowchart showing the flow of selective playback processing. The various functions shown in FIG. 14 are realized by the CPU 101 executing dedicated programs. Selective playback of objects using volumetric video metadata will be described below with reference to the block diagram in FIG. 14 and the flowchart in FIG. 15. In the following description, the symbol "S" means step.

[0036] In S1501, data of the volumetric video to be played is read from the HDD 105 or the like, and a user interface screen (UI screen) showing a frame of interest in a chapter of interest is displayed on the display based on an instruction from the operator. When the operator specifies a chapter to watch from among the chapters making up the volumetric video on the UI screen of FIG. 13(a), a representative frame (e.g., the first frame) in the specified chapter is displayed on the UI screen. The operator uses a mouse or the like to select a desired object (here, object 1301 with the attribute "person," the player name "Goto," and the team "Team A") from among the objects present on the UI screen.

[0037] In S1502, operation information of the operator who selects an arbitrary object via the input device 107 is acquired. For example, when it is detected that a click operation is performed with the mouse hovering over an arbitrary object among the objects present on the UI screen of Fig. 13(a), information on the detected click operation is acquired.

[0038] In S1503, the selected model determination unit 1401 identifies the 3D model of the object selected by the operator based on the operation information acquired in S1502. Specifically, the foreground mesh model that collides with the ray corresponding to the pixel position indicated by the operation information is identified as the mesh model of the object selected by the operator. Note that selection conditions may be set in advance to identify the mesh model of an automatically selected object.

[0039] In S1504, the rendering model setting unit 1402 sets a global object to be the main rendering target based on the model index of the 3D model identified in S1503. Specifically, by referring to the attached information "GO" as metadata, the rendering model setting unit 1402 sets the global object corresponding to the chapter object of the model index of the 3D model identified in S1503 as the object to be the main rendering target.

[0040] In S1505, the rendering model setting unit 1402 sets a global object to be a sub-object to be rendered from among other global objects in the chapter of interest. Specifically, by referencing the attached information "Conv" as metadata, the rendering model setting unit 1402 sets a global object that is integrated with the global object set as the main object to be rendered in S1504 as a sub-object to be rendered.

[0041] In S1506, the camera path generation unit 1403 generates information (camera path) indicating the movement path of the virtual viewpoint within the chapter of interest, allowing the operator to optimally view the object selected by the operator. The camera path generation uses information (object position, attributes, circumscribing rectangle of the mesh model) in the metadata of the global object set as the main rendering target. For example, if the attribute is a person, a camera path capturing the selected object from the front, a camera path capturing the object from behind, or a camera path reproducing the line of sight of the person (player) is selected and generated based on a user instruction or the like. If the attribute is a ball, a camera path capturing the entire imaging space (e.g., the entire field) from the side, a camera path capturing the entire field of view from directly above, or a camera path that always keeps the ball and goal within the field of view is generated based on a user instruction or the like. Alternatively, the camera path may be generated by receiving operation information specifying the position and orientation of the virtual viewpoint (virtual camera) from a virtual viewpoint controller (not shown) via the network I / F 110.

[0042] In S1507, the rendering unit 1404 performs rendering processing using the 3D model of the global object to be rendered set in S1504 and S1505, according to the camera path generated in S1506. FIG. 13B shows an example of the rendering result when a player 1301 is selected on the UI screen of FIG. 13A. In the example of FIG. 13B, player objects 1301 and 1302 and a ball object 1303, which are integrated with each other in the chapter of interest, are rendered and displayed. However, player object 1304, which is not integrated with other players or the ball, is not rendered and displayed. "Integration" here means that multiple objects are represented by a single mesh model. The attribute and tag information of the selected object, player object 1301, is displayed in the upper left corner of the screen based on metadata. In this way, rendering and displaying non-selected objects that are integrated with the object selected by the operator in the chapter of interest can prevent a virtual viewpoint image from appearing unnatural. In other words, if non-selected objects are displayed only in the frame at the moment when the selected object comes into contact with another object, an object that was not present in the previous frame will suddenly appear. Such an image will be very strange to the viewer, so to prevent this, it is desirable to perform the rendering and display control described above. Furthermore, for non-selected objects that are momentarily integrated with the selected object, measures can be taken to ensure that the selected object is not obstructed, for example by rendering and displaying them semi-transparently in the frames when the 3D models are not integrated.

[0043] The above is the flow of processing for selectively playing back any object using volumetric video with metadata of pattern 2.

[0044] <Modification> In the above-described embodiment, a system configuration was described assuming that volumetric video data with metadata is stored in an operator's client environment, but this is not limiting. For example, a system configuration may be used in which the data is stored in a server environment, and the results of processing in the server environment in response to an operator's operation are received and viewed on the operator's client PC. Furthermore, when transmitting a mesh model from the server to selectively play back an arbitrary object on a client PC, the communication load can be reduced by transmitting only the 3D shape data of the selected object.

[0045] As described above, according to this embodiment, metadata linking objects and mesh models is generated and added to a volumetric video frame sequence for provision. Volumetric video data in this format allows for efficient selective rendering of specific objects according to the virtual viewpoint. It also makes it possible to flexibly handle cases where multiple objects are represented by a single 3D model at a given moment.

[0046] (Other Examples) The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0047] 100 Information processing device 201 Video reading section 202 Tracking information acquisition unit 203 Metadata Generation Unit 204 Video output section

Claims

[Claim 1] acquisition means for acquiring volumetric video data comprising a sequence of frames containing shape data of an object; an acquisition means for acquiring tracking information indicating changes between frames of each object corresponding to the shape data; a generating means for generating metadata that associates each of the objects with the shape data in each frame based on the tracking information; an output means for outputting the volumetric video data including the metadata; An information processing device comprising:

Images