Information processing device, information processing method, and program

Abstract

The present invention switches between a reproduction process based on 2D data and one based on 3D data, for example.　This information processing device has a switching control unit that acquires 2D data and 3D data related to the same photographic subject, and that controls switching between a first reproduction process that is based on the 2D data and a second reproduction process that is based on the 3D data.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Techniques have been proposed for rendering a 3D model in a model space (for example, the technique described in Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-291120 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, rendering a 3D model with a large number of polygons requires a large computational load and a large amount of computing resources. Therefore, it is desirable to store two-dimensional data (2D data), which has a relatively small computational load for processing, and use it appropriately for processing three-dimensional data (3D data).

[0005] An object of the present disclosure is to provide an information processing device, an information processing method, and a program that appropriately switch between a first reproduction process based on 2D data and a second reproduction process based on 3D data. [Means for solving the problem]

[0006] The present disclosure provides, for example, a switching control unit that acquires 2D data and 3D data relating to the same subject and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data; and, a drawing unit that performs drawing processing in accordance with the control of the switching control unit; With death, The switching control unit controls the switching in response to a change in the user's viewpoint; The drawing unit performs control to bring the posture of the subject before switching and the posture of the subject after switching closer to each other before switching is performed by the switching control unit. It is an information processing device.

[0007] The present disclosure provides, for example, A switching control unit acquires 2D data and 3D data relating to the same subject, and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data. death, The switching control unit controls the switching in response to a change in the user's viewpoint; The drawing unit performs control to bring the posture of the subject before switching and the posture of the subject after switching closer to each other before switching is performed by the switching control unit. It is an information processing method.

[0008] The present disclosure provides, for example, A switching control unit acquires 2D data and 3D data relating to the same subject, and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data. death, The switching control unit controls the switching in response to a change in the user's viewpoint; The drawing unit performs control to bring the posture of the subject before switching and the posture of the subject after switching closer to each other before switching is performed by the switching control unit. It is a program that causes a computer to execute an information processing method. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram that will be referred to when describing the technology related to the present disclosure. [Figure 2] FIG. 2 is a diagram to be referred to when describing the technology related to the present disclosure. [Figure 3] FIG. 3 is a diagram that will be referred to when describing the technology related to the present disclosure. [Figure 4] FIG. 4 is a diagram that will be referred to when describing the technology related to the present disclosure. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating a data storage format in the storage unit of the first embodiment. [Figure 7]FIG. 7 is a flowchart illustrating an example of the operation of the information processing apparatus according to the first embodiment. [Figure 8] 8A and 8B are diagrams for explaining a modified example of the first embodiment. [Figure 9] 9A and 9B are diagrams for explaining a modified example of the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of an information processing device according to the second embodiment. [Figure 11] FIG. 11 is a diagram for explaining a data storage format in the storage unit of the second embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of the operation of the information processing apparatus according to the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to the third embodiment. [Figure 14] FIG. 14 is a diagram illustrating a data storage format in the storage unit of the third embodiment. [Figure 15] FIG. 15 is a sequence diagram for explaining the processing performed in the third embodiment. [Figure 16] FIG. 16 is a sequence diagram for explaining the processing performed in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The description will be made in the following order. <Technologies related to the present disclosure> First Embodiment <Second embodiment> <Third embodiment> <Modification> <Application example> The embodiments and the like described below are preferred specific examples of the present disclosure, and the contents of the present disclosure are not limited to these embodiments and the like.

[0011] <Technologies related to the present disclosure> First, to facilitate understanding of the present disclosure, techniques related to the present disclosure will be described. The techniques related to the present disclosure described below can be applied to the present disclosure.

[0012] [Outline of information processing system] FIG. 1 shows an overview of an information processing system to which the present technology is applied. A data acquisition unit 1 acquires image data for generating a 3D model of a subject. For example, as shown in FIG. 2, a plurality of viewpoint images captured by a plurality of imaging devices 8B arranged to surround a subject 8A are acquired as image data. In this case, the plurality of viewpoint images are preferably images captured synchronously by a plurality of cameras. Alternatively, the data acquisition unit 1 may acquire, as image data, a plurality of viewpoint images captured by a single camera from a plurality of viewpoints of the subject 8A. Alternatively, the data acquisition unit 1 may acquire, as image data, a single captured image of the subject 8A. In this case, a 3D model generation unit 2, which will be described later, generates a 3D model using, for example, machine learning.

[0013] The data acquisition unit 1 may perform calibration based on image data to acquire internal parameters and external parameters of each image capture device 8 B. The data acquisition unit 1 may also acquire, for example, multiple pieces of depth information indicating the distances from multiple viewpoints to the subject 8 A.

[0014] The 3D model generation unit 2 generates a model having three-dimensional information of the subject 8A based on image data for generating a 3D model of the subject 8A. The 3D model generation unit 2 generates a 3D model of the subject 8A by carving out the three-dimensional shape of the subject 8A using images from multiple viewpoints (e.g., silhouette images from multiple viewpoints) using, for example, a so-called visual hull. In this case, the 3D model generation unit 2 can further deform the 3D model generated using the visual hull with high precision using multiple pieces of depth information indicating the distances from multiple viewpoints to the subject 8A. Also, for example, the 3D model generation unit 2 may generate a 3D model of the subject 8A from a single captured image of the subject 8A. The 3D model generated by the 3D model generation unit 2 can also be referred to as a video of the 3D model because it is generated frame by frame in a time series. Furthermore, the 3D model can also be referred to as a live-action 3D model because it is generated using images captured by the imaging device 8B. The 3D model can express shape information representing the surface shape of the subject 8A in the form of mesh data, which is called a polygon mesh and is expressed by connections between vertices. The method of expressing the 3D model is not limited to this, and it may also be described using a so-called point cloud expression method, which is expressed by position information of points.

[0015] Color information data is also generated as textures linked to these 3D shape data. For example, there are view-independent textures, which have a constant color no matter what direction you look at them from, and view-dependent textures, which change color depending on the viewing direction.

[0016] The formatting unit 3 (encoding unit) converts the data of the 3D model generated by the 3D model generation unit 2 into a format suitable for transmission and storage. For example, the 3D model generated by the 3D model generation unit 2 may be converted into multiple 2D images by perspective projection from multiple directions. In this case, the 3D model may be used to generate depth information, which is a 2D depth image from multiple viewpoints. The depth information and color information of this 2D image state are compressed and output to the transmission unit 4. The depth information and color information may be transmitted side by side as a single image, or as two separate images. In this case, because the data is in the form of 2D image data, it can also be compressed using a 2D compression technology such as AVC (Advanced Video Coding).

[0017] Also, for example, the 3D data may be converted into a point cloud format and output as 3D data to the transmitting unit 4. In this case, for example, the 3D compression technology of the Geometry-based Approach discussed in MPEG can be used.

[0018] The transmitting unit 4 transmits the transmission data formed by the formatting unit 3 to the receiving unit 5. The transmitting unit 4 performs a series of processes by the data acquiring unit 1, the 3D model generating unit 2, and the formatting unit 3 offline, and then transmits the transmission data to the receiving unit 5. The transmitting unit 4 may also transmit the transmission data generated by the above-mentioned series of processes to the receiving unit 5 in real time.

[0019] The receiving unit 5 receives the transmission data transmitted from the transmitting unit 4 .

[0020] The rendering unit 6 performs rendering using the transmitted data received by the receiving unit 5. For example, it projects the mesh of the 3D model from the viewpoint of the camera that renders it, and performs texture mapping to apply textures that represent colors and patterns. The rendering at this time can be set arbitrarily, regardless of the camera position at the time of shooting, and can be viewed from any viewpoint.

[0021] The rendering unit 6 performs texture mapping, for example, by applying a texture representing the color, pattern, and texture of a mesh according to the position of the mesh of the 3D model. Texture mapping can be performed in two ways: a view-dependent method that takes into account the user's viewing viewpoint, and a view-independent method that does not. The view-dependent method has the advantage of achieving higher quality rendering than the view-independent method because it changes the texture applied to the 3D model depending on the position of the viewing viewpoint. On the other hand, the view-independent method has the advantage of requiring less processing than the view-dependent method because it does not take into account the position of the viewing viewpoint. Note that the viewing viewpoint data is input from the display device to the rendering unit 6 after the display device detects the user's viewing position (region of interest). The rendering unit 6 may also employ billboard rendering, which renders objects so that they maintain a vertical orientation relative to the viewing viewpoint. For example, when rendering multiple objects, objects of low viewer interest can be rendered using billboards, while other objects can be rendered using a different rendering method.

[0022] The display unit 7 displays the result of rendering by the rendering unit 6 on the display unit 7 of the display device. The display device may be, for example, a head-mounted display, a spatial display, a mobile phone, a television, a PC, or any other 2D or 3D monitor.

[0023] The information processing system in Figure 1 shows a series of processes from a data acquisition unit 1 that acquires captured images, which are materials for generating content, to a display control unit that controls the display device on which the user views the content. However, this does not mean that all functional blocks are required to implement the present technology; the present technology may be implemented for each functional block or a combination of multiple functional blocks. For example, Figure 1 includes a transmitter 4 and a receiver 5 to show a series of processes from the side that creates the content to the side that views the content through the distribution of content data, but if the entire process from content creation to viewing is carried out on the same information processing device (e.g., a personal computer), it is not necessary to include an encoder, transmitter 4, decoder, or receiver 5.

[0024] When implementing this information processing system, the same implementer may implement all functions, or different implementers may implement each functional block. For example, provider A generates 3D content using a data acquisition unit 1, a 3D model generation unit 2, and a formatting unit 3. The 3D content is then distributed through a transmission unit 4 (platform) of provider B, and a display device of provider C receives, renders, and controls the display of the 3D content.

[0025] Furthermore, each functional block can be implemented on the cloud. For example, the rendering unit 6 can be implemented within the display device or on a server. In this case, information is exchanged between the display device and the server.

[0026] 1 illustrates an information processing system that collectively includes a data acquisition unit 1, a 3D model generation unit 2, a formatting unit 3, a transmission unit 4, a reception unit 5, a rendering unit 6, and a display unit 7. However, the information processing system in this specification is also referred to as an information processing system if two or more functional blocks are involved, and for example, the data acquisition unit 1, the 3D model generation unit 2, the encoding unit, the transmission unit 4, the reception unit 5, the decoding unit, and the rendering unit 6 can also be collectively referred to as an information processing system, excluding the display unit 7.

[0027] [Processing flow of information processing system] An example of the processing flow of the information processing system will be described with reference to the flowchart of FIG. When the process starts, in step S101, the data acquisition unit 1 acquires image data for generating a 3D model of the subject 8A. In step S102, the 3D model generation unit 2 generates a model having three-dimensional information about the subject 8A based on the image data for generating the 3D model of the subject 8A. In step S103, the formatting unit 3 encodes the shape and texture data of the 3D model generated by the 3D model generation unit 2 into a format suitable for transmission and storage. In step S104, the transmission unit 4 transmits the encoded data, and in step S105, the reception unit 5 receives the transmitted data. In step S106, the decoding unit performs decoding processing and converts the data into shape and texture data required for display, and the rendering unit 6 performs rendering using the shape and texture data. In step S107, the display unit 7 displays the rendering result. When the process of step S107 ends, the process of the information processing system ends.

[0028] FIG. 4 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. In the computer shown in FIG. 4, a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, and a RAM (Random Access Memory) 13 are interconnected via a bus 14. An input / output interface 15 is also connected to the bus 14. An input unit 16, an output unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15. The input unit 16 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 17 includes, for example, a display, a speaker, and an output terminal. The storage unit 18 includes, for example, a hard disk, a RAM disk, and a nonvolatile memory. The communication unit 19 includes, for example, a network interface. The drive 20 drives removable media such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0029] In the computer configured as above, the CPU 11 performs the above-described series of processes by, for example, loading a program stored in the storage unit into RAM 13 via input / output interface 15 and bus 14 and executing the program. RAM 13 also stores data necessary for CPU 11 to execute various processes as needed.

[0030] The program executed by the computer can be applied by recording it on removable media such as package media. In this case, the program can be installed in the storage unit 18 via the input / output interface 15 by inserting the removable media into the drive 20. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 19 and installed in the storage unit 18.

[0031] First Embodiment [Configuration example of information processing device] 5 is a diagram illustrating an example of the configuration of an information processing device (information processing device 100) according to the first embodiment. The information processing device 100 may be configured as, for example, a head-mounted display that is worn on the user's head. The information processing device 100 may also be configured as a display in other forms, such as glasses.

[0032] The information processing device 100 has a user interface processing unit 101, a 2D / 3D switching determination unit 102, a 2D / 3D switching control unit 103, a memory unit 104, a 2D video decoding unit 105, a 2D video drawing unit 106, a 3D model reading unit 107, a 3D model drawing unit 108, a rendering unit 109, and a display unit 110.

[0033] The user interface processing unit 101 processes operation input to an operation input unit (not shown) such as a controller. For example, when head tracking or a predetermined operation is performed on the controller, the user interface processing unit 101 detects the operation and generates an operation input signal corresponding to the operation. The user interface processing unit 101 then supplies the generated operation input signal to the 2D / 3D switching determination unit 102.

[0034] The 2D / 3D switching determination unit 102 determines whether to switch between a first playback process based on 2D data and a second playback process based on 3D data. In this embodiment, the 2D / 3D switching determination unit 102 switches the above-mentioned playback processes when a predetermined operation is performed on the controller and an operation input signal corresponding to the operation is supplied from the user interface processing unit 101. The 2D / 3D switching determination unit 102 supplies a switching determination signal for switching the playback processes to the 2D / 3D switching control unit 103.

[0035] The 2D / 3D switching control unit 103 controls switching between a first playback process based on 2D data and a second playback process based on 3D data, with respect to 2D data and 3D data related to the same subject (also referred to as an object) stored in the information processing device 100. Specifically, the 2D / 3D switching control unit 103 selectively operates either the 2D video rendering unit 106 or the 3D model rendering unit 108.

[0036] The storage unit 104 holds (stores) 2D data related to the same subject and 3D data, which is a 3D model. Specifically, 2D data of a specific subject is stored in storage unit 104A of the storage unit 104, and 3D data including 3D model data of the same subject is stored in storage unit 104B of the storage unit 104. While separate storage units are illustrated for ease of explanation, the 2D data and 3D data may be stored in the same storage unit. The 2D data may be an image obtained by rendering a 3D model using a virtual camera, or an image obtained by extracting only the subject from the results of capturing images of the subject from multiple viewpoints using a physical camera. The 2D data may also be video data rendered using a monocular virtual camera or captured using a physical camera, or may be rendered or captured using a stereo system. When stereo images obtained using a right-eye camera and a left-eye camera are used as the 2D data, the user can experience a sense of depth when the 2D image is displayed.

[0037] 2D video decoding unit 105 decodes the 2D video read from storage unit 104A.

[0038] The 2D video rendering unit 106 performs processing to render 2D video images under the control of the 2D / 3D switching control unit 103. For example, the 2D video rendering unit 106 performs processing to perform billboard rendering of a predetermined subject as the foreground.

[0039] The 3D model reading unit 107 generates a 3D model based on the 3D data read from the storage unit 104B.

[0040] The 3D model rendering unit 108 performs processing to render a 3D model under the control of the 2D / 3D switching control unit 103. For example, the 3D model rendering unit 108 renders the 3D model generated by the 3D model reading unit 107 as the foreground.

[0041] The rendering unit 109 renders a background based on the background data for the foreground rendered by the 2D video rendering unit 106 or the foreground rendered by the 3D model rendering unit 108, and combines the foreground and the background. Note that at least the processing performed by the 2D video rendering unit 106 corresponds to a first reproduction process (e.g., billboard rendering), and at least the processing performed by the 3D model rendering unit 108 corresponds to a second reproduction process (e.g., 3D model rendering). Note that the background data may be stored in the storage unit 104. The background data may be anything, such as a 2D model in a virtual space, a celestial sphere photograph, or a celestial sphere video.

[0042] The display unit 110 is a display that displays the processing results of the rendering unit 109.

[0043] [Data retention format] 6 is a diagram for explaining the format of data storage in the storage unit 104 according to the first embodiment. In this embodiment, as shown in FIG. 6, the 2D data and the 3D model data have the same time-series display content at a predetermined timing indicated by a black circle.

[0044] [Example of operation of information processing device] Next, an example of the operation of the information processing device 100 will be described with reference to the flowchart shown in Fig. 7. The process shown in Fig. 7 is started, for example, when an instruction to play a video is given in the information processing device 100. The process shown in Fig. 7 is performed, for example, on a frame-by-frame basis. The start timing of the process can be changed as appropriate. The processing unit can also be changed as appropriate, such as to multiple frames. In this example, it is assumed that switching between 2D data playback processing and 3D data playback processing is performed, for example, when a switching button on the operation input unit is pressed.

[0045] In step S10, the user interface processing unit 101 processes information from the operation input unit. For example, the user interface processing unit 101 outputs a logical "1" as the operation input signal when the switching button is pressed, and outputs a logical "0" as the operation input signal when the switching button is not pressed. The operation input signal is output to the 2D / 3D switching determination unit 102. Then, the process proceeds to step S11.

[0046] In step S11, the 2D / 3D switching determination unit 102 determines, based on the operation input signal, whether to use a display method based on the first playback process or the second playback process. For example, if the operation input signal indicates "1," the 2D / 3D switching determination unit 102 determines to use a display method based on the first playback process, and if the operation input signal indicates "0," it determines to use a display method based on the second playback process. The 2D / 3D switching determination unit 102 then supplies a switching determination signal indicating the determination result to the 2D / 3D switching control unit 103. The process then proceeds to step S12.

[0047] In step S12, the 2D / 3D switching control unit 103 determines whether the display method for the current frame is the same as the display method for the previous frame based on the switching determination signal. If the display method for the current frame is different from the display method for the previous frame, the process proceeds to step S13. If the display method for the current frame is the same as the display method for the previous frame, the process proceeds to step S14.

[0048] In step S13, 2D / 3D switching control unit 103 controls switching of the display method. Specifically, 2D / 3D switching control unit 103 selects 2D video rendering unit 106 or 3D model rendering unit 108 to operate so that the post-switching display method is performed. Then, the process proceeds to step S14.

[0049] In step S14, a process is performed to render the foreground in the determined display format. That is, under the control of 2D / 3D switching control unit 103, 2D video rendering unit 106 renders the object as the foreground using billboard rendering, or 3D model rendering unit 108 renders the 3D model as the foreground. Then, the process proceeds to step S15.

[0050] In step S15, the rendering unit 109 draws the background, and the process then proceeds to step S16.

[0051] In step S16, the drawn foreground and background are combined, and the process then proceeds to step S17.

[0052] In step S17, the combined data is displayed on the display unit 110.

[0053] According to the present embodiment described above, it is possible to switch between playback processing based on 2D data and playback processing based on 3D data at any time. Furthermore, by performing playback processing based on 2D data, the calculation load can be reduced compared to always performing playback processing based on 3D data.

[0054] [Modification of the first embodiment] In the above description, the playback process based on 2D data and the playback process based on 3D data are switched at the timing when a predetermined operation is performed on the controller. However, the predetermined timing may be determined based on the relative positional relationship between a predetermined position in the virtual space and the user's position, rather than the timing when the predetermined operation is performed on the controller.

[0055] An example of such a relative positional relationship is the distance between a predetermined position and a user position. The predetermined position is, for example, a position in a virtual space where an object is displayed, and the user position is the user's position within the virtual space. A threshold Dth is set as a distance threshold. As shown in FIG. 8A, when the distance DA between the display position of an object (e.g., two people skating) and the user position is greater than the threshold Dth, the object is reproduced by a reproduction process based on 2D data. As shown in FIG. 8B, when the user position approaches the display position of the object and the distance DB becomes smaller than the threshold Dth, the object is reproduced as a three-dimensional model by a reproduction process based on 3D data. Note that the change in the user position may be a virtual change in position caused by the user operating a controller or the like, or a change in position caused by the actual movement of the user wearing the information processing device 100. By performing such processing, distant objects can be displayed by a reproduction process based on 2D data, which requires a low computational load, and close-up objects that require a three-dimensional effect can be displayed by a reproduction process based on 3D data, which can more accurately reproduce the three-dimensional effect.

[0056] The relative positional relationship may also be the positional relationship between the user's position and an angle range set with respect to a predetermined position. For example, as shown in FIG. 9A, an angle θ is defined in the forward direction based on the position of an object in the virtual space. The angle θ is defined so that the area is in front of the plane displaying the two-dimensional image of the object to be viewed. If the user's position is within the range corresponding to the angle θ, the object is displayed by a playback process based on 2D data. As shown in FIG. 9B, if the user's position is not within the range corresponding to the angle θ, the object is displayed as a 3D model by a playback process based on 3D data. The user's position can be determined by the viewpoint relative to the object or by a position sensor provided in the information processing device 100.

[0057] The two examples of the relative positional relationship described above may be combined. For example, even if the user position is within the area displayed in 2D images, if the distance between the object and the user is smaller than a set threshold Dth, a determination may be made to switch to displaying the object in 3D display format.

[0058] <Second embodiment> Next, a second embodiment will be described. The matters described in the first embodiment can be applied to the second embodiment unless otherwise specified. Furthermore, the same reference numerals are used for the same or similar components as those described in the first embodiment, and duplicated descriptions will be omitted as appropriate.

[0059] 10 is a block diagram showing an example of the configuration of an information processing device (information processing device 200) according to the second embodiment. The information processing device 200 differs from the information processing device 100 in that the information processing device 200 includes a playback time storage unit 201 and a timing data reading unit 202. The storage unit 104 of the information processing device 200 also includes a storage unit 104C. The storage unit 104C stores switching timing data indicating the timing for switching between playback processing based on 2D data and playback processing based on 3D data. The switching timing data is, for example, set in advance.

[0060] The playback time storage unit 201 stores the playback time, which is the time from the playback start timing of the video. The playback time is supplied to the 2D / 3D switching determination unit 102.

[0061] The timing data reading unit 202 reads the switching timing data from the storage unit 104C, and supplies the result to the 2D / 3D switching determination unit 102.

[0062] Fig. 11 is a diagram illustrating a data storage format in the storage unit 104 of the second embodiment. As shown in Fig. 11, in a section (a section from the start of playback to t1 seconds later) where a first playback process is performed based on switching timing data (for example, switching timing data for displaying 3D data instead of 2D data at t1 seconds), only 2D data corresponding to that section is stored. Also, in a section (a section from t1 seconds later to t2 seconds later) where a second playback process is performed based on switching timing data (for example, switching timing data for displaying 2D data instead of 3D data at t2 seconds), only 3D data corresponding to that section is stored. This makes it possible to save storage capacity.

[0063] FIG. 12 is a flowchart illustrating an example of the operation of the information processing device 200 according to the second embodiment.

[0064] In step S21, the 2D / 3D switching determination unit 102 acquires the current playback time from the playback time storage unit 201. Then, the process proceeds to step S22.

[0065] In step S22, the switching timing data is read by the switching timing data reading unit 202. The 2D / 3D switching determination unit 102 acquires the read switching timing data. Note that the 2D / 3D switching determination unit 102 may acquire the switching timing data before playback of the video data, or may acquire it in real time. Then, the process proceeds to step S23.

[0066] In step S23, the 2D / 3D switching determination unit 102 determines whether the current playback time is the switching timing indicated by the switching timing data. If it is not the switching timing, the 2D / 3D switching determination unit 102 notifies the 2D / 3D switching control unit 103 of this fact. Then, the process proceeds to step S24.

[0067] In step S24, 2D / 3D switching control unit 103 performs control so that the foreground is drawn in the display format of the previous frame. For example, if the previous frame was played back using a playback process based on 2D data, 2D video rendering unit 106 draws the object in the foreground using billboard rendering under the control of 2D / 3D switching control unit 103. Also, for example, if the previous frame was played back using a playback process based on 3D data, 3D model rendering unit 108 draws the object in the foreground using a 3D model under the control of 2D / 3D switching control unit 103. Then, the process proceeds to step S15.

[0068] In step S23, if the current playback time coincides with the switching timing indicated by the switching timing data, the 2D / 3D switching determination unit 102 notifies this fact to the 2D / 3D switching control unit 103. Then, the process proceeds to step S25.

[0069] In step S25, 2D / 3D switching control unit 103 performs control to switch the display format. For example, if playback processing based on 2D data has been performed up until then, 2D / 3D switching control unit 103 performs control to operate 3D model rendering unit 108 instead of 2D video rendering unit 106. Conversely, if playback processing based on 3D data has been performed up until then, 2D / 3D switching control unit 103 performs control to operate 2D video rendering unit 106 instead of 3D model rendering unit 108. Then, the process proceeds to step S26.

[0070] In step S26, the 2D video rendering unit 106 or the 3D model rendering unit 108 corresponding to the switched display format operates. As a result, the object is rendered in the foreground as a billboard rendering or a 3D model. Then, the process proceeds to step S15.

[0071] The processes in steps S15 to S17 are the same as those in the first embodiment, and therefore a duplicated description will be omitted.

[0072] <Third embodiment> Next, a third embodiment will be described. The matters described in the first and second embodiments can be applied to the third embodiment unless otherwise specified. Furthermore, the same reference numerals are used to designate components that are the same as or of the same quality as those described in the first and second embodiments, and duplicated descriptions will be omitted as appropriate.

[0073] Fig. 13 is a diagram showing an example of the configuration of an information processing device (information processing device 300) according to the third embodiment. The information processing device 300 has a 2D image selection unit 301. In addition, a storage unit 104 of the information processing device 300 stores a set of 2D images generated by photographing a 3D model from multiple viewpoints. More specifically, as shown in Fig. 14, 2D data and 3D model data corresponding to each viewpoint at a predetermined timing are stored in the storage unit 104.

[0074] An example of the operation of the information processing device 300 will be described. For example, assume that a user performs an operation to change the viewpoint while playback processing based on 2D data is being performed. The user interface processing unit 101 detects this operation, and an operation input signal corresponding to the operation is supplied to the 2D / 3D switching control unit 103 and the 2D image selection unit 301.

[0075] The 2D image selection unit 301 selects and reads out 2D data corresponding to the viewpoint corresponding to the operation input signal, and supplies the read out 2D data to the 2D video decoding unit 105. This control makes it possible to display images that do not look strange to the eye even if the user's viewpoint changes.

[0076] However, when using multi-viewpoint 2D data, the posture of an object based on the 2D data may not match the posture of an object based on a 3D model. Therefore, when switching the playback process, the posture of the displayed object may be different, which may cause the user to feel uncomfortable. Therefore, in the information processing device 300, before switching by the 2D / 3D switching control unit 103, control may be performed to bring the posture of the object before switching closer to the posture of the object after switching.

[0077] An example of such control will be specifically described with reference to the sequence diagrams shown in Fig. 15 and Fig. 16. The example shown in Fig. 15 is an example in which the 2D / 3D switching determination unit 102 determines to switch from playback processing based on 2D data to playback processing based on 3D data.

[0078] In step S31, the 2D / 3D switching determination unit 102 notifies the 2D / 3D switching control unit 103 of the determination result. In the following step S32, the 2D / 3D switching control unit 103 controls the 3D model rendering unit 108. In response to this control, the 3D model rendering unit 108 rotates the 3D model to an orientation corresponding to the object being displayed as a 2D image, thereby bringing the orientation closer to that of the object. In step S33, the 3D model rendering unit 108 notifies the 2D / 3D switching control unit 103 that the rotation of the 3D model has ended. In step S34, the 2D / 3D switching control unit 103 operates the 3D model rendering unit 108 in place of the 2D video rendering unit 106. As a result, in step S35, display of the 3D model begins.

[0079] The example shown in FIG. 16 is an example in which the 2D / 3D switching determination unit 102 determines to switch from playback processing based on 3D data to playback processing based on 2D data.

[0080] In step S41, the 2D / 3D switching determination unit 102 notifies the 2D / 3D switching control unit 103 of the determination result. In the following step S42, the 2D / 3D switching control unit 103 controls the 2D video rendering unit 106. In response to this control, the 2D video rendering unit 106 rotates the billboard on which the object is displayed until it faces the user directly. In step S43, the 2D video rendering unit 106 notifies the 2D / 3D switching control unit 103 that the billboard rotation has ended.

[0081] In step S44, the 2D / 3D switching control unit 103 controls the 3D model rendering unit 108. In response to this control, the 3D model rendering unit 108 rotates the 3D model to an angle corresponding to the angle of the billboard. In step S45, the 3D model rendering unit 108 notifies the 2D / 3D switching control unit 103 that rotation of the 3D model has been completed. In step S46, the 2D / 3D switching control unit 103 operates the 2D video rendering unit 106 instead of the 3D model rendering unit 108. This completes the display of the 3D model. Then, in step S47, playback processing based on the 2D data is performed, and the object of the 2D image is displayed.

[0082] By performing the above-described processing, it is possible to minimize the misalignment of posture that may occur between a 2D image and a 3D model of the same subject when switching between them, thereby minimizing the visual discomfort felt by the user.

[0083] In the above example, the posture of the 3D model based on 3D data was changed, but the posture of the object based on 2D data may also be changed by rotating the billboard to match the angle of the 3D model.

[0084] <Modification> Although several embodiments of the present disclosure have been specifically described above, the contents of the present disclosure are not limited to the above-described embodiments, and various modifications based on the technical ideas of the present disclosure are possible.

[0085] In the above-described embodiment, 3D model data is stored in advance, but 3D model data may be generated in real time based on 2D data. Also, in the above-described embodiment, an example has been described in which 2D data and 3D data related to the same subject that are stored in advance are acquired by the information processing device 100, but the information processing device 100 may acquire 2D data and 3D data related to the same subject from a server device or the like via communication.

[0086] A part of the configuration and functions of the information processing device according to the above-described embodiment may exist in a device different from the information processing device (for example, a server device on a network). For example, the storage unit 104 may be a server device that can be connected to the information processing device via a network.

[0087] Furthermore, for example, the program that realizes the above-described functions may be executed by any device. In that case, it is sufficient that the device has the necessary functional blocks and is able to obtain the necessary information. Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, those multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can also be executed as multiple step processes. Conversely, processes described as multiple steps can also be executed collectively as a single step.

[0088] Furthermore, for example, the processing of steps describing a program executed by a computer may be executed chronologically according to the order described in this specification, or may be executed in parallel or individually at a necessary timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of steps describing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program. Also, for example, multiple technologies related to the present technology can be implemented independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described technologies can be implemented in combination with other technologies not described above.

[0089] It should be noted that the contents of the present disclosure should not be construed as being limited to the effects exemplified in this specification.

[0090] The present disclosure may also have the following configurations. (1) The apparatus has a switching control unit that holds 2D data and 3D data relating to the same subject and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data. Information processing device. (2) The switching control unit switches between the first playback process and the second playback process based on a predetermined timing. The information processing device described in (1). (3) The predetermined timing is the timing when a predetermined operation is performed on the controller. (2) An information processing device according to the present invention. (4) The predetermined timing is determined according to the relative positional relationship between a predetermined position and the user's position in the virtual space. (2) An information processing device according to the present invention. (5) The relative positional relationship is the distance between the predetermined position and the user position. (4) An information processing device according to the present invention. (6) The switching control unit selects the first reproduction process when the distance is greater than a threshold value, and selects the second reproduction process when the distance is smaller than a threshold value. (5) An information processing device according to the present invention. (7) The relative positional relationship is a positional relationship between the user position and a range of angles set based on the predetermined position. (4) An information processing device according to the present invention. (8) The switching control unit selects the first playback process when the user position is within a predetermined angle range, and selects the second playback process when the user position is not within the predetermined angle range. (7) An information processing device according to (7). (9) The predetermined timing is a timing that is set in advance. (2) An information processing device according to the present invention. (10) Holds switching timing data indicating the preset timing (9) An information processing device according to (9). (11) The 2D data corresponding to the section where the first reproduction process is performed according to the switching timing data is held, and the 3D data corresponding to the section where the second reproduction process is performed according to the switching timing data is held. (10) An information processing device according to (10). (12) Before the switching is performed by the switching control unit, control is performed to bring the posture of the subject before the switching closer to the posture of the subject after the switching. An information processing device according to any one of (1) to (11). (13) Before the switching is performed by the switching control unit, the posture of the subject based on the 3D data is changed. (12) An information processing device according to (12). (14) Before the switching is performed by the switching control unit, the posture of the subject based on the 2D data is changed. (12) An information processing device according to (12). (15) a storage unit for storing 2D data and 3D data relating to the same subject; An information processing device according to any one of (1) to (14). (16) A switching control unit holds 2D data and 3D data relating to the same subject, and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data. Information processing methods. (17) A switching control unit holds 2D data and 3D data relating to the same subject, and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data. A program that causes a computer to execute an information processing method.

[0091] <Application example> The technology disclosed herein can be applied to a variety of products and services. (Content Creation) For example, new video content may be produced by combining the 3D model of the subject generated in the above-described embodiment with 3D data managed on another server. Furthermore, if background data acquired by an imaging device such as Lidar is present, the 3D model of the subject generated in the above-described embodiment may be combined with the background data to produce content that makes the subject appear as if they are in the location indicated by the background data. The video content may be three-dimensional video content or two-dimensional video content converted to two dimensions. The 3D model of the subject generated in the above-described embodiment may be, for example, a 3D model generated by the 3D model generation unit 2 or a 3D model reconstructed by the rendering unit 6.

[0092] (Virtual space experience) For example, a subject (e.g., a performer) generated in this embodiment can be placed in a virtual space where a user can communicate as an avatar. In this case, the user can view a live-action subject in the virtual space as an avatar.

[0093] (Application to remote communication) For example, by transmitting the 3D model of the subject generated by the 3D model generation unit 2 from the transmission unit 4 to a remote location, a user at the remote location can view the 3D model of the subject through a playback device at the remote location. For example, by transmitting the 3D model of the subject in real time, the subject and a user at the remote location can communicate in real time. For example, a case can be envisioned where the subject is a teacher and the user is a student, or the subject is a doctor and the user is a patient.

[0094] (others) For example, a free viewpoint video of a sport or the like can be generated based on the 3D models of multiple subjects generated in the above-described embodiment, or an individual can distribute the 3D model of themselves generated in the above-described embodiment to a distribution platform. In this way, the content of the embodiments described in this specification can be applied to various technologies and services. [Explanation of symbols]

[0095] 100, 200, 300... Information processing equipment 103 2D / 3D switching control unit 104...Storage section 106···2D image rendering section 108···3D model drawing section

Claims

1. a switching control unit that acquires 2D data and 3D data related to the same subject and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data; a drawing unit that performs drawing processing in accordance with control of the switching control unit; and the switching control unit controls the switching in response to a change in a user's viewpoint; The drawing unit performs control to bring the posture of the subject before switching and the posture of the subject after switching closer to each other before switching is performed by the switching control unit. Information processing device.

2. The rendering unit changes the posture of the subject based on the 3D data. The information processing device according to claim 1 .

3. The rendering unit changes the posture of the subject based on the 2D data. The information processing device according to claim 1 .

4. a storage unit for storing 2D data and 3D data relating to the same subject; The information processing device according to claim 1 .

5. a switching control unit that acquires 2D data and 3D data related to the same subject and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data; the switching control unit controls the switching in response to a change in a user's viewpoint; A drawing unit performs control to bring the posture of the subject before switching and the posture of the subject after switching closer to each other before switching is performed by the switching control unit. Information processing methods.

6. a switching control unit that acquires 2D data and 3D data related to the same subject and controls switching between a first reproduction process based on the 2D data and a second reproduction process based on the 3D data; the switching control unit controls the switching in response to a change in a user's viewpoint; A drawing unit performs control to bring the posture of the subject before switching and the posture of the subject after switching closer to each other before switching is performed by the switching control unit. A program that causes a computer to execute an information processing method.

Images