Information processing apparatus, terminal apparatus, information processing method, and program

Abstract

[Problem] To provide a mechanism capable of further improving the estimation accuracy of the camera parameters. [Solution] An information processing apparatus includes a control unit that estimates camera parameters of a live video based on a first feature quantity of a moving body present in a target space and a second feature quantity of the moving body depicted in the live video obtained by imaging the target space.

Description

Technical Field

[0001] This disclosure relates to information processing devices, terminal devices, information processing methods, and procedures. Background Technology

[0002] Conventionally, techniques have been developed for estimating camera device parameters of a camera device that has captured video. For example, Patent Document 1 below discloses a technique for estimating camera device parameters based on the skeleton of a person appearing in a video.

[0003] Citation List

[0004] Patent documents

[0005] Patent Document 1: WO 2022 / 259618 Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] However, the technology disclosed in Patent Document 1 is still in its early stages since its development, and there is room for improvement from various perspectives. Improving the accuracy of the estimation of camera device parameters is one such perspective.

[0008] Therefore, this disclosure was made in view of the above problems, and the purpose of this disclosure is to provide a mechanism that can further improve the accuracy of the estimation of camera device parameters.

[0009] Solution to the problem

[0010] According to this disclosure, an information processing apparatus is provided, which includes a control unit configured to estimate camera parameters of a live video based on a first feature of a moving object present in a target space and a second feature of a moving object appearing in a live video obtained by imaging the target space.

[0011] Furthermore, according to this disclosure, a terminal device is provided, the terminal device including a control unit configured to generate a playback screen of a reproduced video synchronized with a live-shot video obtained by imaging a target space, the reproduced video being generated by rendering a 3D model generated based on a first feature quantity of a moving object present in the target space as a 2D image based on camera device parameters of the live-shot video.

[0012] Furthermore, according to this disclosure, an information processing method is provided, which includes estimating camera device parameters of a live video based on a first feature quantity of a moving object existing in a target space and a second feature quantity of a moving object appearing in a live video obtained by imaging the target space.

[0013] Furthermore, according to this disclosure, an information processing method is provided, which includes generating a playback frame of a reconstructed video synchronously with a live-action video obtained by imaging a target space, the reconstructed video being generated by rendering a 3D model generated based on a first feature quantity of a moving object existing in the target space as a 2D image based on camera device parameters of the live-action video.

[0014] Furthermore, according to this disclosure, a program is provided for using a computer as a control unit, the control unit being configured to estimate camera device parameters of a live video based on a first feature quantity of a moving object present in a target space and a second feature quantity of a moving object appearing in a live video obtained by imaging the target space.

[0015] Furthermore, according to this disclosure, a program is provided for using a computer as a control unit, the control unit being configured to generate playback frames of a reproduced video synchronized with a live-action video obtained by imaging a target space, the reproduced video being generated by rendering a 3D model generated based on a first feature quantity of a moving object present in the target space as a 2D image using camera device parameters based on the live-action video. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating an overview of system 1 according to one embodiment of the present disclosure.

[0017] Figure 2 This is a block diagram illustrating the configuration of system 1, according to a first embodiment of the present disclosure, in relation to video reproduction.

[0018] Figure 3 This is a block diagram illustrating the configuration of System 1 related to live-action video according to an embodiment.

[0019] Figure 4 This is a block diagram illustrating a configuration example of server 2 according to an implementation method.

[0020] Figure 5 This is a diagram used to illustrate the parameters of the camera device in the implementation method.

[0021] Figure 6 This is a block diagram illustrating a configuration example of user terminal 3 according to an embodiment.

[0022] Figure 7 This is a diagram illustrating an example of a playback screen displayed by a user terminal 3 according to an embodiment.

[0023] Figure 8 This is a diagram illustrating an example of a playback screen displayed by a user terminal 3 according to an embodiment.

[0024] Figure 9 This is a diagram illustrating an example of a playback screen displayed by a user terminal 3 according to an embodiment.

[0025] Figure 10 This is a block diagram illustrating an example configuration of the camera device parameter estimation unit 34 according to an embodiment.

[0026] Figure 11 This is a diagram illustrating a specific example of the processing performed by the camera device parameter estimation unit 34 according to the embodiment.

[0027] Figure 12 This is a diagram illustrating a specific example of the processing performed by the camera device parameter estimation unit 34 according to the embodiment.

[0028] Figure 13 This is a diagram illustrating a specific example of the processing performed by the camera device parameter estimation unit 34 according to the embodiment.

[0029] Figure 14 This is a diagram illustrating a specific example of the processing performed by the camera device parameter estimation unit 34 according to the embodiment.

[0030] Figure 15 This is a block diagram illustrating an example configuration of the playback speed estimation unit 35 according to an embodiment.

[0031] Figure 16 This is a diagram illustrating a specific example of the processing performed by the playback speed estimation unit 35 according to the embodiment.

[0032] Figure 17 This is a diagram illustrating a specific example of the processing performed by the playback speed estimation unit 35 according to the embodiment.

[0033] Figure 18 This is a block diagram illustrating an example of the flow of camera device parameter estimation processing performed by server 2 according to an embodiment.

[0034] Figure 19 This is a block diagram illustrating an example of the flow of synchronous playback processing performed by user terminal 3 according to an embodiment.

[0035] Figure 20 This is a block diagram illustrating the configuration of system 1, according to a second embodiment of the present disclosure, related to the reproduction of video.

[0036] Figure 21 This is a block diagram illustrating the configuration of System 1 related to live-action video according to an embodiment.

[0037] Figure 22This is a block diagram illustrating an example of the hardware configuration of an information processing device according to each of the above embodiments. Detailed Implementation

[0038] Preferred embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are indicated by the same reference numerals, and redundant descriptions are omitted.

[0039] Note that the descriptions will be given in the following order.

[0040] 1. Technical Features

[0041] 2. First Implementation Method

[0042] 2.1. Configuration Example

[0043] 2.2. Detailed Configuration

[0044] 2.3. Processing Flow

[0045] 3. Second Implementation Method

[0046] 4. Hardware Configuration Example

[0047] 5. Supplement

[0048] <1. Technical Features>

[0049] Figure 1 This is a diagram illustrating an overview of system 1 according to one embodiment of this disclosure. (See diagram for example.) Figure 1 As shown, System 1 generates a reconstructed video based on motion data and metadata obtained from moving objects, such as humans, and plays back the reconstructed video together with live-action video obtained by actually imaging the moving object.

[0050] Recreated video can be a 3D computer graphics (3DCG) animation of a 3D model of a moving object. Specifically, 2D images constituting each frame of the recreated video are generated by rendering the motion-data-based 3D model as a 2D image using the same camera parameters as those used in the live-action video. Therefore, users can view both live-action and recreated videos captured or generated with the same camera parameters.

[0051] Furthermore, System 1 performs synchronized playback of the reconstructed video and the live-action video. This synchronized playback means that the acquisition time of the motion data used as the basis for the reconstructed video matches or nearly matches the capture time of the live-action video. Therefore, users can watch both the reconstructed and live-action videos without any sense of incongruity.

[0052] Here, including the technology disclosed in Patent Document 1 above, various prior art techniques have been proposed for estimating camera device parameters of live-action video through image analysis. As an example, there are methods for estimating camera device parameters based on the identification of lines and vanishing points appearing in live-action video. As another related technique, there are techniques for estimating 3D structures based on semantic segmentation of 2D images.

[0053] However, each of the existing technologies has various problems. As an example, in some cases, live-action video from which camera parameters can be estimated with high accuracy is limited to video obtained by capturing, for example, a football match where field lines may appear, from an appropriate perspective including lines, etc. As another example, it is difficult to estimate camera parameters with high accuracy for live-action video that includes camera switching, zooming operations, and overlapping (i.e., occlusion) of people.

[0054] From this perspective, a system 1 according to one embodiment of the present disclosure has been invented. System 1 uses motion data and metadata of a moving object to estimate camera device parameters of live video. Therefore, while eliminating the problems existing in the prior art of estimating camera device parameters through image analysis, System 1 can further improve the accuracy of camera device parameter estimation.

[0055] The following will refer to Figure 1 The technical features of System 1 will be described again. For example... Figure 1 As shown, system 1 includes server 2 and user terminal 3.

[0056] (Server 2)

[0057] Server 2 is an information processing device with a control unit that controls various types of processing for the synchronized playback of live-action and replayed video. The function of the control unit in Server 2 is achieved through the collaboration between hardware such as a processor and software included within Server 2. Server 2 generates synchronized playback information based on motion data and metadata of moving objects and outputs this information to user terminal 3. Server 2 can be, for example, a cloud server.

[0058] Server 2 estimates the camera device parameters of the live-action video based on a first feature of the moving objects present in the target space and a second feature of the moving objects appearing in the live-action video obtained by imaging the target space. The target space is not limited to real space and can be a virtual space, or a space where real and virtual spaces interfere with each other. The moving objects can be, for example, humans, animals, balls, vehicles, or plants. By using the first and second feature values, Server 2 can eliminate the problems existing in the prior art of estimating camera device parameters through image analysis.

[0059] The first feature quantity is motion data, which includes information indicating the three-dimensional position and orientation of each of the one or more parts constituting a moving object. Motion data has a smaller data volume and higher portability than images. Therefore, camera device parameters can be estimated at a higher speed compared to estimating camera device parameters by applying image analysis to live-action video. Consequently, reconstructed videos corresponding to live-action videos can be generated at a high speed, and both live-action and reconstructed videos can be played back synchronously.

[0060] The second feature is the pose of the moving object appearing in the 2D image. Server 2 renders the 3D model of the moving object based on the first feature into a 2D image based on candidate camera device parameters, and estimates the camera device parameters of the live-action video based on the similarity between the pose of the moving object appearing in the rendered 2D image and the pose of the moving object appearing in the 2D image constituting the live-action video. For example, server 2 estimates the candidate camera device parameters with the highest similarity or similarity exceeding a predetermined threshold as the camera device parameters of the live-action video. With this configuration, the estimation accuracy of the camera device parameters of the live-action video can be improved.

[0061] Server 2 sets up a first search range based on the metadata of the live-action video, including one or more candidate camera device parameters, and estimates the camera device parameters according to the set first search range. The metadata includes information for identifying the imaging location of the live-action video. Server 2 sets the vicinity of the imaging location of the live-action video indicated by the metadata as the first search range. Then, Server 2 searches for camera device parameters within the first search range, while repeatedly setting up candidates for camera device parameters and calculating similarity. With this configuration, the computational load associated with searching for camera device parameters can be reduced, and camera device parameters can be estimated at high speed.

[0062] The first feature can be associated with the identification information of the moving object. Then, server 2 can calculate the similarity of moving objects associated with the same identification information. More specifically, server 2 can calculate the similarity of the poses of moving objects associated with the same identification information and appearing in the 2D image based on motion data and the 2D image constituting the live-action video. The identification information of the moving object can be, for example, an athlete's name or jersey number. The identification information of the moving objects appearing in the 2D image constituting the live-action video can be included in metadata or estimated through image analysis. With this configuration, the accuracy of the estimation of the camera device parameters can be further improved.

[0063] Server 2 renders the 3D model of the moving object corresponding to the first feature value acquired at the acquisition time corresponding to the capture time of the live video as a 2D image based on the candidate camera device parameters, and calculates the similarity. The acquisition time corresponding to the capture time refers to the acquisition time that matches or approximately matches the capture time. With this configuration, the similarity can be appropriately calculated, and therefore, the camera device parameters can be appropriately estimated.

[0064] Server 2 sets a second search range, including one or more candidate acquisition times, based on the playback speed of the live-action video, and estimates the acquisition time according to the set second search range. For example, if the playback speed is 1x, Server 2 can determine that the live-action video is a live broadcast video, and can set the range corresponding to several frames before and after the current time as the second search range. On the other hand, if the playback speed is less than 1x, Server 2 can determine that the live-action video is a slow-motion playback of past video, and can set the range from the current time to a predetermined time in the past as the second search range. If the metadata includes information for identifying the capture time of the live-action video, Server 2 can also set the second search range based on the metadata. Server 2 can then search for acquisition times within the second search range, while repeatedly setting candidates for acquisition times, and calculating similarity based on a first feature value acquired at the candidate acquisition time. With this configuration, the computational load related to searching for acquisition times can be reduced, and the acquisition time can be estimated at high speed.

[0065] Server 2 generates synchronized playback information that correlates the captured video, a first feature acquired at the acquisition time corresponding to the capture time of the captured video, and the camera device parameters of the captured video. More specifically, the synchronized playback information correlates the 2D image constituting the captured video, the first feature at the acquisition time corresponding to the capture time of the 2D image, and the camera device parameters of the 2D image. Based on this configuration, using the synchronized playback information, the captured video and the reproduced video can be simultaneously played back in user terminal 3.

[0066] Synchronous playback information can include the playback speed of the live-action video. With this configuration, when performing synchronous playback of the live-action and reproduced video, user terminal 3 can display the playback speed simultaneously.

[0067] (User terminal 3)

[0068] User terminal 3 is a terminal device with a control unit that controls the processing for generating playback frames that simultaneously play back the live-action video and the replayed video. The function of the control unit in user terminal 3 is achieved through the collaboration between hardware such as a processor included in user terminal 3 and the program. User terminal 3 generates replayed video based on motion data of the moving object and synchronous playback information, and performs synchronous playback of the replayed video and the live-action video. User terminal 3 can be, for example, a smartphone or tablet computer.

[0069] User terminal 3 generates a playback screen on which a reconstructed video is played back synchronously with the live-action video. This reconstructed video is generated by rendering a 2D image from a 3D model based on a first feature quantity of a moving object existing in the target space, based on camera device parameters obtained by imaging the live-action video of the target space. With this configuration, a user of user terminal 3 can view the synchronized live-action video and the reconstructed video.

[0070] User terminal 3 generates playback screens that simultaneously display live-action and replayed videos, or display either live-action or replayed videos in a switchable manner. With this configuration, users can simultaneously view and compare live-action and replayed videos, or view either live-action or replayed videos as needed.

[0071] User terminal 3 generates a playback screen based on the synchronized playback information, and plays back the reproduced video synchronously with the live-action video on this playback screen. More specifically, user terminal 3 plays back the live-action video, and while playing back the live-action video, it also plays back the reproduced video generated based on a first feature value acquired at the acquisition time corresponding to the capture time of the live-action video and the camera device parameters based on the live-action video. With this configuration, since user terminal 3 does not need to perform estimation of the acquisition time and estimation of the camera device parameters, the processing load associated with the synchronized playback of the reproduced video and the live-action video can be reduced.

[0072] The playback footage can include the playback speed of the live-action video. With this configuration, users can watch the replayed video while simultaneously recognizing the playback speed of the live-action video.

[0073] The technical features of System 1 have been described above.

[0074] <2. First Implementation Method>

[0075] The first implementation method is to simultaneously replay live footage and replay video of a football match held on a football field.

[0076] <2.1. Configuration Example>

[0077] (Configuration example for distributing reproduced videos)

[0078] Figure 2 This is a block diagram illustrating the configuration of system 1 according to this embodiment related to video playback. For example... Figure 2 As shown, system 1 includes multiple camera devices 11, a tracking system 12, a streamer 13, a converter 14, a metadata tagger 15, multiple players 16, and multiple renderers 17.

[0079] Multiple camera devices 11 are arranged around the football field to capture the football match from various angles and output the acquired video data to the tracking system 12.

[0080] The tracking system 12 acquires motion data of the match based on video data obtained by multiple camera devices 11, and outputs the motion data to the fluidizer 13. The motion data may include, for example, the three-dimensional position and orientation of various parts of a football player (e.g., the bones of a skeletal model), and the three-dimensional position of the football. The motion data, along with time data indicating the time when the motion data was acquired, is output as time-series data.

[0081] Fluidizer 13 distributes the motion data stream of the game. More specifically, fluidizer 13 distributes the motion data of all players and the ball as a stream each time.

[0082] Converter 14 converts the motion data distributed from streamer 13 into a predetermined format and distributes the converted motion data. An example of the predetermined format is a format suitable for distribution and playback, such as the MPEG-DASH standard or a proprietary format based on the MPEG-DASH standard. Converter 14 connects to the Internet and provides an interface through which multiple players 16 can simultaneously acquire the converted data.

[0083] Metadata tagger 15 distributes the match's metadata. Specifically, metadata tagger 15 outputs match metadata in association with the motion data output from converter 14. The metadata is distributed as time-series data along with time data indicating when the metadata was added. Note that the metadata can be added manually or automatically via the Internet or similar means. Similar to converter 14, metadata tagger 15 connects to the Internet and provides an interface through which multiple players 16 can simultaneously acquire metadata.

[0084] When a football match begins, the match's motion data and metadata are distributed via the Internet with a delay of a few seconds.

[0085] Player 16 acquires the distributed motion data and metadata.

[0086] Renderer 17 renders and reproduces the video based on the motion data stream acquired by player 16 and displays the rendered video. More specifically, renderer 17 renders a 3D model based on motion data as a 2D image and displays the 2D image in a time series.

[0087] (Configuration example for broadcasting live video footage)

[0088] Figure 3 This is a block diagram illustrating the configuration of system 1 according to this embodiment related to live-action video. For example... Figure 3 As shown, system 1 includes multiple camera devices 21, switcher 22, editor 23, encoder 24, transmitter 25, receiver 26, decoder 27 and renderer 28.

[0089] Multiple camera devices 21 are deployed near the field to simultaneously capture the football match from various angles and output video data (i.e., time-series data of 2D images) of the captured live video.

[0090] Switcher 22 selects and outputs the video data of the live video to be broadcast from the video data of the live video captured by multiple camera devices 21.

[0091] Editor 23 edits video data from live-action videos. For example, Editor 23 sets the playback speed of live-action videos.

[0092] Encoder 24 encodes the video data of the live-action video edited by editor 23 into a predetermined format suitable for broadcasting, and outputs the encoded video data to transmitter 25.

[0093] Transmitter 25 broadcasts encoded video data of the live-shot video.

[0094] Receiver 26 receives the broadcast video data of the live video and outputs the broadcast video data to decoder 27.

[0095] Decoder 27 decodes the video data of the live-action video and outputs the decoded video data to renderer 28.

[0096] Renderer 28 renders the live-action video based on the decoded data and displays the rendered video.

[0097] (Configuration example for server 2)

[0098] Figure 4 This is a block diagram illustrating a configuration example of server 2 according to this embodiment. Figure 4 As shown, server 2 includes a player 31, a receiver 32, a decoder 33, a camera device parameter estimation unit 34, a playback speed estimation unit 35, and a synchronous playback information generation unit 36.

[0099] Player 31 obtained from Figure 2 The converter 14 and metadata tagger 15 distributed motion data and metadata, and output the motion data and metadata to the camera device parameter estimation unit 34. The player 31 corresponds to... Figure 2 The player shown is 16.

[0100] Receiver 32 receives from Figure 3 The transmitter 25 broadcasts the video data of the captured video and outputs the video data to the decoder 33. The receiver 32 corresponds to... Figure 3 The receiver 26 is shown.

[0101] Decoder 33 decodes the video data of the live-shot video and outputs the decoded video data to camera device parameter estimation unit 34 and playback speed estimation unit 35. Decoder 33 corresponds to Figure 3 The decoder 27 is shown.

[0102] The camera device parameter estimation unit 34 estimates the camera device parameters of the live video (i.e., the camera device parameters of the camera device 21 that captured the live video) based on motion data, metadata, and video data of the live video. The detailed configuration of the camera device parameter estimation unit 34 will be described in detail later.

[0103] Reference Figure 5 An example describing the parameters of a camera device used to shoot live video.

[0104] Figure 5 This is a diagram used to illustrate the parameters of the camera device in this embodiment. For example... Figure 5 As shown, the camera device parameters may include the three-dimensional position (x, y, z) of the camera device 21 relative to the origin O of the world coordinate system, consisting of its x, y, and z coordinates. The camera device parameters may also include the orientation (θ, ω, γ) of the camera device 21 in the world coordinate system, consisting of its roll angle θ, pitch angle ω, and yaw angle γ. Furthermore, the camera device parameters may include the viewing angle α of the camera device 21. The camera device parameters for the live-action video only need to include at least one of the three-dimensional position, orientation, or viewing angle of the camera device 21 that captured the live-action video.

[0105] The playback speed estimation unit 35 estimates the playback speed of the live-action video. For example, the playback speed estimation unit 35 estimates the playback speed of live-action video broadcast in real-time as 1x, estimates the playback speed of live-action video replayed at 2x, 3x, etc. as greater than 1x, and estimates the playback speed of live-action video replayed in slow motion as less than 1x. The detailed configuration of the playback speed estimation unit 35 will be described in detail later.

[0106] The synchronous playback information generation unit 36 ​​generates synchronous playback information based on the camera device parameters estimated by the camera device parameter estimation unit 34 and the playback speed estimated by the playback speed estimation unit 35. Then, the synchronous playback information generation unit 36 ​​sends the generated synchronous playback information.

[0107] Table 1 below shows an example of synchronized playback information.

[0108] [Table 1]

[0109] Table 1. An example of synchronized playback information

[0110]

[0111] The video frame number is the identifier of the 2D image that constitutes the live-action video. The motion frame number is the identifier of each item of the motion data that constitutes the motion data stream. In Table 1, the video frame number, motion frame number, and camera device parameters stored in the same row indicate the 2D image that constitutes the live-action video, the motion data acquired at the acquisition time corresponding to the capture time of the 2D image, and the camera device parameters of the 2D image.

[0112] (Configuration example for user terminal 3)

[0113] Figure 6 This is a block diagram illustrating a configuration example of user terminal 3 according to this embodiment. For example... Figure 6 As shown, user terminal 3 includes receiver 41, decoder 42, synchronized player 43, and renderer 44. These components can be integrated into, for example, an application or browser.

[0114] Receiver 41 receives the video data from the broadcast live video and outputs the video data to decoder 42. Receiver 41 corresponds to... Figure 3 The receiver 26 is shown.

[0115] Decoder 42 decodes the video data of the live-action video and outputs the decoded video data to the synchronous player 43. Decoder 42 corresponds to... Figure 3 The decoder 27 is shown.

[0116] Synchronous Player 43 obtained from Figure 2 The motion data and metadata distributed by the converter 14 and metadata tagger 15 shown, as well as the motion data and metadata distributed by the converter 14 and metadata tagger 15 shown, are... Figure 4The synchronous playback information generation unit 36, as shown, sends synchronous playback information. Then, based on the synchronous playback information, the synchronous player 43 outputs motion data synchronized with the live-action video (i.e., motion data acquired at the acquisition time corresponding to the capture time of the live-action video), camera device parameters of the live-action video, and playback speed of the live-action video. More specifically, in the synchronous playback information, the synchronous player 43 outputs motion data, camera device parameters, and playback speed corresponding to the motion frame number of the live-action video output from the decoder 42, as well as the playback speed.

[0117] Renderer 44 renders and displays both the live-action video and the replay video. More specifically, renderer 44 renders the live-action video based on the video data output from decoder 42 and displays the rendered video. Simultaneously, renderer 44 renders a 3D model based on motion data output from synchronizer 43 into a 2D image based on camera device parameters output from synchronizer 43, and displays the rendered 2D image. Furthermore, renderer 44 renders and displays the playback speed output from synchronizer 43.

[0118] Reference Figures 7 to 9 An example of a playback screen displayed by user terminal 3. Figures 7 to 9 This is a diagram showing an example of a playback screen displayed by the user terminal 3 according to this embodiment.

[0119] like Figure 7 As shown, user terminal 3 can simultaneously display the reproduced video V1 and the live-action video V2 on a single playback screen D1. Figure 7 As shown, the reproduced video V1 is rendered using the same camera parameters as the live-action video V2. Figure 8 As shown, when the playback speed specified in the synchronous playback information is other than 1x, the playback speed information D21 can be clearly displayed on the playback screen D2.

[0120] like Figure 9 As shown, on a playback screen D3, either the replayed video V1 or the live-action video V2 can be displayed. The user terminal 3 can seamlessly switch between the videos to be played back based on user operations.

[0121] Furthermore, user terminal 3 can accept user operations on camera device parameters. In this case, user terminal 3 renders a 3D model based on motion data into a 2D image based on the camera device parameters specified by the user, and displays the rendered 2D image as replay video V1. With this configuration, the user can freely move the viewpoint and perform zoom in / out to view the replay video V1, which is played back synchronously with the live-action video V2.

[0122] <2.2. Detailed Configuration>

[0123] (1) Camera device parameter estimation unit 34

[0124] Figure 10 This is a block diagram illustrating an example configuration of the camera device parameter estimation unit 34 according to this embodiment. Figure 10 As shown, the camera device parameter estimation unit 34 includes an input interface 110, a camera device parameter analysis block 120, a motion data analysis block 130, an image analysis block 140, a buffer memory 150, and an output interface 160.

[0125] (Input interface 110)

[0126] Input interface 110 is an interface for receiving information input to the camera device parameter estimation unit 34. For example... Figure 10 As shown, the input interface 110 includes a metadata input unit 111, a motion data input unit 112, and a video data input unit 113.

[0127] Metadata input unit 111 acquires metadata and outputs the metadata to camera device parameter analysis block 120.

[0128] The motion data input unit 112 acquires motion data and outputs the motion data to the motion data analysis block 130.

[0129] The video data input unit 113 acquires the video data of the real-shot video and outputs the video data to the image analysis block 140.

[0130] (Camera device parameter analysis block 120)

[0131] Camera device parameter analysis block 120 is a processing block for analyzing camera device parameters. For example... Figure 10 As shown, the camera device parameter analysis block 120 includes a metadata analysis unit 121 and a camera device parameter search unit 122.

[0132] The metadata analysis unit 121 analyzes the metadata input by the metadata input unit 111 and sets a search range (corresponding to a first search range) for the camera device parameters of the captured video. The metadata includes information for identifying the capture time and imaging position of the captured video. Therefore, the metadata analysis unit 121 sets the search range to include camera device parameters capable of capturing imaging positions identified by the metadata.

[0133] Metadata can include absolute information, such as timestamps, latitude, longitude, and altitude, or information that can be converted to absolute information, such as "a football match between DEF and GHI held at ABC Stadium on February 24, 2024." Additionally, metadata can include information that can be used to identify moving objects appearing in the live-action video, such as the names of the participating teams, player names, player jersey numbers, jersey colors, scoring information, referee information such as fouls, and the ball's position at the time of imaging. Furthermore, metadata can include data beyond those mentioned above.

[0134] The metadata analysis unit 121 can also set the search range of camera device parameters based on the live-action video. As an example, if a ball appears in the live-action video, the metadata analysis unit 121 can set the range within which the ball fits in the field of view as the search range. As another example, the metadata analysis unit 121 can set the range within which an athlete appearing in the live-action video fits in the field of view as the search range.

[0135] The camera device parameter search unit 122 searches for camera device parameters of the actual video from the search range. More specifically, the camera device parameter search unit 122 outputs candidate camera device parameters within the search range to the pose comparison unit 132. Then, the camera device parameter search unit 122 identifies the candidate camera device parameters whose comparison results from the pose comparison unit 132 meet predetermined conditions as the camera device parameters of the actual video.

[0136] (Image Analysis Block 140)

[0137] Image analysis block 140 is a processing block that analyzes the 2D image of each video frame that constitutes the video data of the live-action video. For example... Figure 10 As shown, the image analysis block 140 includes a video data decoding unit 141 and a pose estimation unit 142.

[0138] The video data decoding unit 141 decodes the video data of the live-shot video input by the video data input unit 113. For example, the video data decoding unit 141 decodes the video data of the live-shot video into a predetermined format.

[0139] The buffer memory 150 buffers the 2D images of each video frame constituting the live-action video, which are decoded by the video data decoding unit 141.

[0140] Pose estimation unit 142 estimates the pose of a moving object appearing in a 2D image constituting the live-action video. The pose here can be the position and orientation of each of one or more parts of the moving object appearing in the 2D image. For example, pose estimation unit 142 estimates the pose of the moving object appearing in the live-action video by applying any pose recognition algorithm such as OpenPose to the 2D image cached by buffer memory 150. Pose estimation unit 142 outputs the estimated pose of the moving object to pose comparison unit 132.

[0141] (Motion data analysis block 130)

[0142] Motion data analysis block 130 is a processing block for analyzing motion data. For example... Figure 10 As shown, the motion data analysis block 130 includes a motion data decoding unit 131 and a pose comparison unit 132.

[0143] The motion data decoding unit 131 decodes the motion data. For example, the motion data decoding unit 131 decodes the motion data into a predetermined format by setting the number of bones to a predetermined value.

[0144] The pose comparison unit 132 compares the pose of a moving object appearing in a 2D image constituting the live-action video with the pose of a 3D model based on motion data. More specifically, the pose comparison unit 132 renders a 3D model based on motion data decoded by the motion data analysis block 130 into a 2D image based on candidate camera device parameters output from the camera device parameter search unit 122. Then, the pose comparison unit 132 calculates the similarity between the pose of the moving object estimated by the pose estimation unit 142 and the pose of the 3D model in the 2D image rendered based on motion data, and outputs the similarity to the camera device parameter search unit 122.

[0145] If the similarity calculated by the pose comparison unit 132 is equal to or less than a predetermined threshold, the camera device parameter search unit 122 instructs the pose comparison unit 132 to continue calculating the similarity while changing the candidates for camera device parameters. Then, the camera device parameter search unit 122 identifies the candidates for camera device parameters whose similarity calculated by the pose comparison unit 132 exceeds the predetermined threshold as the camera device parameters of the live-action video. In this way, the camera device parameter search unit 122 sequentially searches for the camera device parameters of the live-action video by changing the candidates for camera device parameters until the similarity calculated by the pose comparison unit 132 exceeds the predetermined threshold.

[0146] Note that constant values ​​can always be used as candidate values ​​for the threshold and camera device parameters mentioned above, or the threshold and values ​​can be appropriately changed. For example, at the start of the search, the threshold can be set relatively low, while the values ​​can be set relatively large. Subsequently, if the similarity exceeds the threshold, the threshold can be updated to a relatively high value, the values ​​can be updated to a relatively small value, and the search can continue.

[0147] (Replenish)

[0148] While the above description describes the search for camera device parameters, a similar approach can be used to search for the acquisition time of motion data corresponding to the capture time of the live video. For example, the metadata analysis unit 121 can set a search range (corresponding to a second search range) for the acquisition time of motion data corresponding to the capture time of the live video based on metadata. Then, the camera device parameter search unit 122 can search for the acquisition time of motion data corresponding to the capture time of the live video from the search range.

[0149] More specifically, the camera device parameter search unit 122 can output candidate camera device parameters within a first search range and candidate acquisition times within a second search range to the pose comparison unit 132. In this case, the pose comparison unit 132 renders a 3D model based on motion data acquired at candidate acquisition times into a 2D image based on the candidate camera device parameters and calculates similarity. Then, the camera device parameter search unit 122 identifies combinations of candidate camera device parameters and candidate acquisition times with similarity exceeding a predetermined threshold as the camera device parameters of the live video and the acquisition time of the motion data corresponding to the capture time.

[0150] (Output interface 160)

[0151] Output interface 160 is an interface for outputting information from the camera device parameter estimation unit 34. For example... Figure 10 As shown, the output interface 160 includes a camera device parameter output unit 161.

[0152] The camera device parameter output unit 161 outputs the camera device parameters of the real-shot video identified by the camera device parameter search unit 122.

[0153] (Specific example)

[0154] Figures 11 to 14 This is a diagram illustrating a specific example of the processing performed by the camera device parameter estimation unit 34 according to this embodiment.

[0155] like Figure 11As shown, the metadata analysis unit 121 sets the search range 51 for camera device parameters of the live-action video. Next, the camera device parameter search unit 122 sets candidates for camera device parameters from the search range 51. Then, the pose comparison unit 132 renders the motion data-based 3D model 52 into a 2D image 53 based on the candidates for camera device parameters.

[0156] In addition, such as Figure 12 As shown, the pose estimation unit 142 analyzes the 2D image 54 of the live video and estimates the poses 541 to 544 of the moving objects appearing in the 2D image 54 of the live video.

[0157] Then, the pose comparison unit 132 compares the pose of the 3D model 52 appearing in the 2D image 53 based on motion data with the poses 541 to 544 of the moving object appearing in the 2D image 54 of the live-action video. The camera device parameter search unit 122 searches for camera device parameters in the live-action video based on the comparison result of the pose comparison unit 132 while changing the candidate camera device parameters.

[0158] For example, such as Figure 13 As shown, when the poses 531 to 534 of the 3D model 52 appearing in the 2D image 53 based on motion data are dissimilar to the poses 541 to 544 of the moving object appearing in the 2D image 54 of the live-action video, the camera device parameter search unit 122 changes the candidates for camera device parameters and continues the search. On the other hand, as Figure 14 As shown, when the poses 531 to 534 of the 3D model 52 appearing in the 2D image 53 based on motion data are similar to the poses 541 to 544 of the moving object appearing in the 2D image 54 of the live-action video, the camera device parameter search unit 122 identifies the candidate camera device parameters as the camera device parameters of the live-action video.

[0159] (2) Playback speed estimation unit 35

[0160] Figure 15 This is a block diagram illustrating an example configuration of the playback speed estimation unit 35 according to this embodiment. Figure 15 As shown, the playback speed estimation unit 35 includes a motion database 210, a playback speed learning block 220, an input interface 230, a playback speed estimation block 240, a learned network 250, and an output interface 260.

[0161] (Network 250 learned)

[0162] The learned network 250 is a network that outputs the playback speed of a video given a video showing a moving object as input. The learned network 250 can be, for example, a neural network or a recurrent neural network. The learned network 250 is learned by the learning unit 223, which will be described later.

[0163] (Sports Database 210)

[0164] The motion database 210 is a time-series database that stores motion data in real-time playback (i.e., playback speed of 1x).

[0165] (Replay speed learning block 220)

[0166] The playback speed learning block 220 is a processing block for building the learned network 250. For example... Figure 15 As shown, the playback speed learning block 220 includes a motion data decoding unit 221, a change generation unit 222, and a learning unit 223.

[0167] The motion data decoding unit 221 decodes the time-series data of the motion data. Then, the motion data decoding unit 221 outputs the decoded time-series data of the motion data in association with the playback speed of 1 to the motion data decoding unit 221 and the change generation unit 222.

[0168] The variation generation unit 222 generates time-series data of motion data. For example, the variation generation unit 222 generates time-series data of motion data at playback speeds other than 1x, and outputs the generated time-series data in association with the playback speed to the learning unit 223.

[0169] Learning unit 223 learns the parameters of the learned network 250 based on a combination of playback speed and time-series motion data. More specifically, learning unit 223 uses a combination of playback speed and 3DCG animation generated from time-series motion data as teacher data to learn the parameters of the learned network 250.

[0170] (Input interface 230)

[0171] Input interface 230 is an interface that receives input information to the playback speed estimation unit 35. For example... Figure 15 As shown, the input interface 230 includes a video data input unit 231.

[0172] The video data input unit 231 acquires the video data of the real-shot video and outputs the video data to the playback speed estimation block 240.

[0173] (Playback speed estimated at 240)

[0174] Playback speed estimation block 240 is a processing block that estimates the playback speed of the live-action video. For example... Figure 15 As shown, the playback speed estimation block 240 includes a video data decoding unit 241 and an inference unit 242.

[0175] The video data decoding unit 241 decodes the video data of the real-shot video and outputs the decoded video data to the inference unit 242.

[0176] The inference unit 242 estimates the playback speed of the live-action video by performing inference using the learned network 250. More specifically, the inference unit 242 estimates the poses of moving objects appearing in the 2D images that constitute each video frame of the live-action video. Then, the inference unit 242 inputs an animation consisting of time-series data of the estimated poses of the moving objects into the learned network 250 and obtains the output playback speed as the estimated playback speed of the live-action video. The inference unit 242 outputs the estimated playback speed of the live-action video to the output interface 260.

[0177] (Output interface 260)

[0178] Output interface 260 is an interface for outputting information from playback speed estimation unit 35. For example... Figure 15 As shown, the output interface 260 includes a playback speed output unit 261.

[0179] Playback speed output unit 261 outputs the playback speed output from inference unit 242.

[0180] (Specific example)

[0181] Figure 16 and Figure 17 This is a diagram illustrating a specific example of the processing performed by the playback speed estimation unit 35 according to this embodiment.

[0182] like Figure 16 As shown, learning unit 223 generates 3DCG animation 61 based on time-series data of motion data at 1x playback speed and various playback speeds. Then, learning unit 223 uses the combination of playback speed and 3DCG animation 61 as teacher data to learn the parameters of the learned network 250.

[0183] like Figure 17 As shown, the inference unit 242 estimates the pose of a moving object appearing in the live-action video for each 2D image constituting the broadcast, and generates an animation 62 as time-series data of the estimated pose of the moving object. The inference unit 242 then estimates the playback speed by inputting the generated animation 62 into the learned network 250.

[0184] <2.3. Processing Flow>

[0185] (Camera device parameter estimation and processing)

[0186] Figure 18 This is a block diagram illustrating an example of the process of camera device parameter estimation performed by server 2 according to this embodiment.

[0187] like Figure 18 As shown, firstly, server 2 obtains the video data, metadata, and motion data of the real-shot video (step S102).

[0188] Next, server 2 selects the video frame to be processed (step S104). With an initial value of -1, the video frame number of the video frame to be processed is incremented each time step S104 is executed.

[0189] Next, server 2 inputs a 2D image of a frame to be processed (step S106) and estimates the pose of moving objects appearing in the 2D image (step S108).

[0190] In parallel, server 2 estimates the playback speed of the live video (step S110).

[0191] Next, server 2 sets candidate times for acquiring motion data corresponding to the capture time of the live-action video based on the estimated playback speed of the live-action video (step S112). As an example, if the playback speed is 1x, server 2 can determine that the live-action video is a live-streamed video, and can use a time within a range corresponding to several frames before and after the current time, including the current time, as a candidate time for acquiring motion data corresponding to the capture time of the live-action video. On the other hand, if the playback speed is less than 1x, server 2 can determine that the live-action video is a slow-motion playback of a past video, and can use a time within a predetermined time range from the current time to the past as a candidate time for acquiring motion data corresponding to the capture time of the live-action video. Furthermore, if the playback speed is greater than 1x, server 2 can determine that the live-action video is a high-speed playback of a past video, and can use a time within a predetermined time range from the current time to the past as a candidate time for acquiring motion data corresponding to the capture time of the live-action video.

[0192] Next, server 2 sets the search range for camera device parameters based on metadata (step S114).

[0193] Next, server 2 obtains motion data of a frame from the candidate location of the motion data acquisition time corresponding to the capture time of the real-shot video set in step S112 (step S116).

[0194] Next, server 2 determines whether there are any unverified camera device parameters within the search range of the camera device parameters set in step S114 (step S118). Note that verification here refers to the similarity calculation and threshold comparison in step S124.

[0195] If it is determined that there are no unverified camera device parameters (step S118: No), the process proceeds to step S128.

[0196] If it is determined that there are unverified camera device parameters (step S118: Yes), the server 2 sets the unverified camera device parameters as candidates for camera device parameters (step S120).

[0197] Next, server 2 renders the 3D model based on the motion data obtained in step S116 into a pose on a 2D image based on the candidate camera device parameters set in step S120 (step S122).

[0198] Next, server 2 calculates the similarity between the pose estimated in step S108 and the pose rendered in step S122. Then, server 2 determines whether the calculated similarity is equal to or greater than a threshold (step S124).

[0199] If the similarity is determined to be less than the threshold (step S124: No), the process returns to step S118.

[0200] If the similarity is determined to be equal to or greater than the threshold (step S124: Yes), server 2 outputs the candidate camera device parameters used for rendering in step S122 as the camera device parameters of the live video (step S126).

[0201] At this point, server 2 can generate synchronized playback information that associates the video frame selected in step S104, the motion data obtained in step S116, and the camera device parameters output in step S126.

[0202] After that, server 2 determines whether there is a next video frame (step S128).

[0203] If it is determined that there is a next video frame (step S128: Yes), the process returns to step S104.

[0204] On the other hand, if it is determined that there is no next video frame (step S128: No), the process ends.

[0205] (Synchronous playback processing)

[0206] Figure 19This is a block diagram illustrating an example of the flow of synchronous playback processing performed by user terminal 3 according to this embodiment.

[0207] like Figure 19 As shown, user terminal 3 obtains synchronous playback information (step S202).

[0208] Next, user terminal 3 reads the synchronization playback information of one frame (step S204).

[0209] Next, user terminal 3 obtains the video frame number of the synchronization playback information read in step S204 (step S206).

[0210] Next, user terminal 3 acquires motion data of the motion frame number corresponding to the video frame number acquired in step S206 (step S208).

[0211] Next, user terminal 3 obtains the camera device parameters corresponding to the video frame number obtained in step S206 (step S210).

[0212] Next, user terminal 3 renders a reproduced video based on the 3D model obtained in step S208, using the camera device parameters obtained in step S210 (step S212).

[0213] On the other hand, user terminal 3 obtains the video data of the real-shot video with the video frame number obtained in step S206 (step S214).

[0214] Next, user terminal 3 renders the real-shot video based on the video data of the real-shot video obtained in step S214 (step S216).

[0215] Then, user terminal 3 synchronously displays the reproduced video rendered in step S212 and the real-shot video rendered in step S216 on the playback screen (step S218).

[0216] On the other hand, user terminal 3 obtains the playback speed of the synchronous playback information read in step S204 (step S220).

[0217] Next, user terminal 3 displays the playback speed obtained in step S220 on the playback screen (step S222).

[0218] After this, user terminal 3 determines the termination of synchronous playback (step S224).

[0219] If it is determined that the synchronous playback will not be terminated (step S224: No), the process returns to step S204.

[0220] On the other hand, if it is determined that synchronous playback will be terminated (step S224: Yes), the process ends.

[0221] <3. Second Implementation Method>

[0222] System 1 according to the second embodiment is an implementation for simultaneously playing back live-action video and reconstructed video for a user wearing a motion sensor. More specifically, System 1 simultaneously plays back live-action video captured by the user terminal 3 and reconstructed video based on motion data obtained by the motion sensor. System 1 according to this embodiment can, for example, be used to improve the form of sports coaching.

[0223] (Configuration example for distributing reproduced videos)

[0224] Figure 20 This is a block diagram illustrating the configuration of system 1 according to this embodiment related to video playback. For example... Figure 20 As shown, system 1 includes multiple motion sensors 71, data receivers 72, motion capture devices 73, and transmitters 74.

[0225] Multiple motion sensors 71 are attached to various parts of the user's body. For example, multiple motion sensors 71 are attached to the user's head, arms, hands, waist, and feet. The multiple motion sensors 71 acquire sensor data related to the user's motion data and transmit the sensor data to a data receiver 72. The motion sensors 71 may include, for example, accelerometers and angular velocity sensors, and the sensor data may be, for example, time-series data of acceleration and angular velocity.

[0226] The data receiver 72 receives sensor data from multiple motion sensors 71 and outputs the sensor data to the motion capture unit 73.

[0227] The motion capture device 73 acquires the user's motion data based on sensor data obtained by multiple motion sensors 71.

[0228] Transmitter 74 transmits time-series data of motion data acquired by motion capture unit 73.

[0229] (Configuration example for distributing live-action videos)

[0230] Figure 21 This is a block diagram illustrating the configuration of system 1 according to this embodiment related to live-action video. For example... Figure 21 As shown, system 1 includes a camera device 81, an encoder 82, and a transmitter 83. These components can be arranged in user terminal 3.

[0231] The camera device 81 captures the user wearing the motion sensor 71 and outputs the video data of the captured live video.

[0232] The encoder 82 encodes the video data of the real-shot video acquired by the camera device 81 into a predetermined format suitable for distribution, and outputs the encoded video data to the transmitter 83.

[0233] Transmitter 83 transmits encoded video data of the live-shot video.

[0234] Note that you can perform edits, such as cutting and changing playback speed, before sending the live video data.

[0235] Furthermore, user terminal 3 can send metadata associated with the video data of the live-action video. The metadata may include, for example, the capture time and location information at the time of capture. Additionally, the metadata may include, for example, information that can be used to identify moving objects appearing in the live-action video, such as the name of the person being filmed.

[0236] (Configuration example for server 2)

[0237] The configuration example for server 2 is shown in the first implementation method. Figure 4 As mentioned above.

[0238] In other words, server 2 estimates the camera device parameters of the captured video (i.e., the camera device parameters of the camera device 81 that captured the video) based on the video data of the captured video obtained by camera device 81 and the motion data obtained by motion sensor 71. Then, server 2 generates synchronized playback information that associates the captured video, the motion data acquired at the time corresponding to the capture time of the captured video, and the camera device parameters of the captured video. Subsequently, server 2 distributes the generated synchronized playback information in association with the captured video and the motion data.

[0239] In addition, server 2 can estimate the playback speed of the live-action video. Then, server 2 can generate synchronized playback information that includes the estimated playback speed.

[0240] (Configuration example for user terminal 3)

[0241] The configuration example of user terminal 3 is shown in the first embodiment. Figure 6 As mentioned above.

[0242] In other words, user terminal 3 receives live-action video, motion data, and synchronous playback information, and synchronously plays back the live-action video and the replayed video based on the received information. More specifically, user terminal 3 generates the replayed video by rendering a 3D model of motion data acquired at the acquisition time corresponding to the capture time of the live-action video into a 2D image using the camera device parameters based on the live-action video, and plays back the replayed video together with the live-action video.

[0243] (Detailed configuration)

[0244] The detailed configuration of the camera device parameter estimation unit 34 and playback speed estimation unit 35 according to this embodiment is as described in the first embodiment. Figure 10 , Figure 15 As mentioned above.

[0245] (Processing flow)

[0246] The processing flow performed by server 2 and user terminal 3 according to this embodiment is as described in the first embodiment. Figure 18 and Figure 19 As stated above.

[0247] <4. Hardware Configuration Example>

[0248] Finally, refer to Figure 22 The hardware configuration of the information processing apparatus according to the above embodiments will be described. Figure 22 This is a block diagram illustrating an example of the hardware configuration of an information processing apparatus according to the above embodiments. Note that... Figure 22 The information processing device 900 shown can achieve, for example... Figure 4 Server 2 shown Figure 5 The user terminal 3 shown. The information processing performed by the server 2 or user terminal 3 according to the above embodiment is achieved through the cooperation between the following software and hardware.

[0249] like Figure 22 As shown, the information processing device 900 includes a central processing unit (CPU) 901, a read-only memory (ROM) 902, a random access memory (RAM) 903, and a host bus 904a. Furthermore, the information processing device 900 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a driver 909, a connection port 910, and a communication device 912. Instead of or together with the CPU 901, the information processing device 900 may have processing circuitry such as circuitry, a DSP, or an ASIC.

[0250] The CPU 901 serves as an arithmetic processing and control device, and controls the overall operation within the information processing unit 900 according to various programs. Furthermore, the CPU 901 may be a microprocessor. The ROM 902 stores programs used by the CPU 901, operating parameters, etc. The RAM 903 temporarily stores programs used by the CPU 901 during execution, parameters that change appropriately during execution, etc. The CPU 901 may form, for example, a control unit for controlling the overall processing of the server 2 or a control unit for controlling the overall processing of the user terminal 3.

[0251] CPU 901, ROM 902, and RAM 903 are interconnected via a host bus 904a, which includes the CPU bus. The host bus 904a is connected to an external bus 904b, such as a Peripheral Component Interconnect / Interface (PCI) bus, via a bridge 904. Note that the host bus 904a, bridge 904, and external bus 904b do not necessarily need to be provided separately, and these functions can be implemented on a single bus.

[0252] Input device 906 is implemented by, for example, a device through which a user inputs information (e.g., a mouse, keyboard, touch panel, button, microphone, switch, and joystick). Furthermore, input device 906 can be, for example, a remote control device using infrared or other radio waves, or an external connection device such as a mobile phone or PDA corresponding to the operation of information processing device 900. Additionally, input device 906 may include, for example, an input control circuit that generates an input signal based on the information input by the user using the aforementioned input device and outputs the input signal to CPU 901. The user of information processing device 900 can input various types of data or instruct processing operations to information processing device 900 by operating input device 906.

[0253] Output device 907 is formed by a device capable of notifying a user of acquired information visually or audibly. Examples of such devices include: display devices, such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, laser projectors, LED projectors, and lamps; audio output devices, such as speakers and headphones; and printers. Output device 907 outputs, for example, the results obtained through various types of processing performed by information processing device 900. Specifically, the display device visually displays the results obtained through various types of processing performed by information processing device 900 in various formats such as text, images, tables, and graphics. On the other hand, the audio output device converts audio signals, including reproduced speech data, acoustic data, etc., into analog signals and outputs the analog signals audibly. The above-described display device can, for example, simultaneously play back live video and reproduced video in user terminal 3.

[0254] Storage device 908 is an example data storage device formed as a storage unit in information processing apparatus 900. Storage device 908 is implemented using, for example, magnetic storage devices such as HDDs, semiconductor storage devices, optical storage devices, or magneto-optical storage devices. Storage device 908 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. This storage device 908 stores programs executed by CPU 901, various types of data, and various types of data acquired from external sources. Storage device 908 may temporarily store motion data, metadata, and video data, for example, in server 2 or user terminal 3.

[0255] Drive 909 is a reader / writer for storage media and is either built into or externally attached to the information processing device 900. Drive 909 reads information recorded on removable storage media such as attached magnetic disks, optical disks, magneto-optical disks, or semiconductor memories and outputs that information to RAM 903. Furthermore, drive 909 can also write information to removable storage media.

[0256] Connection port 910 is an interface for connecting to external devices, and is a connection port for external devices, such as those capable of data transfer via Universal Serial Bus (USB).

[0257] Communication device 912 is, for example, a communication interface formed by communication devices for connecting to network 920. Communication device 912 is, for example, a communication card for wired or wireless local area networks (LANs), Long Term Evolution (LTE), Bluetooth (registered trademark), or Wireless USB (WUSB). Furthermore, communication device 912 can be a router for optical communication, a router for Asymmetric Digital Subscriber Line (ADSL), a modem for various types of communication, etc. This communication device 912 can, for example, send signals to and receive signals from the Internet or other communication devices according to predetermined protocols such as TCP / IP. Communication device 912 can, for example, send and receive synchronization playback information in server 2 or user terminal 3.

[0258] Note that network 920 is a wired or wireless transmission path used to send information from devices connected to network 920. For example, network 920 may include public networks such as the Internet, telephone line networks, or satellite communication networks, as well as various local area networks (LANs) including Ethernet (registered trademark), wide area networks (WANs), etc. Furthermore, network 920 may include private line networks, such as Internet Protocol-Virtual Private Networks (IP-VPNs).

[0259] Examples of hardware configurations capable of implementing the functions of the information processing apparatus 900 according to the embodiments described above have been described above. Each of the components described above can be implemented using general-purpose components, or can be implemented using hardware dedicated to the function of each component. Therefore, the hardware configuration to be used can be appropriately changed depending on the level of technology at the time of implementing the embodiments described above.

[0260] <5. Supplement>

[0261] While preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that those skilled in the art will conceive of various modifications or alterations within the scope of the technical concept described in the claims, and such modifications or alterations are naturally understood to fall within the technical scope of the present disclosure.

[0262] For example, the above implementation describes an example of distributing and playing back video data from live-action videos, but audio data can be distributed and played back along with the video data.

[0263] For example, the above implementation describes an application of System 1 to sports broadcasting or sports coaching, but the technology is not limited to such examples. As an example, System 1 can be applied to the broadcasting of live music performances or the recording / playback of stage performances such as dance or drama. As another example, System 1 can be applied to computer games in which live-action video and 3DCG animation seamlessly switch, or to augmented reality (AR) applications.

[0264] Note that each device described in this specification may be implemented as a single device, and some or all of the devices may be implemented as separate devices. Furthermore, at least a portion of the processes described in this specification that are performed by a particular device may be performed by any other device. As an example, user terminal 3 may have a player 31, a receiver 32, a decoder 33, a camera device parameter estimation unit 34, a playback speed estimation unit 35, and a synchronous playback information generation unit 36, and may generate synchronous playback information. As another example, server 2 may have a receiver 41, a decoder 42, a synchronous player 43, and a renderer 44, and user terminal 3 may display video rendered by server 2 on the Web.

[0265] Note that the series of processes performed by each device described in this specification can be implemented using software, hardware, or a combination of both. The program constituting the software is pre-stored in a recording medium (specifically, a computer-readable non-transitory storage medium) provided, for example, internally or externally to each device. Each program is then read into, for example, random access memory (RAM) during execution by a computer controlling each device described in this specification, and executed by processing circuitry such as a central processing unit (CPU). The recording medium is, for example, a magnetic disk, optical disk, magneto-optical disk, or flash memory. Furthermore, the computer program described above can be distributed, for example, via a network without using a recording medium. Furthermore, the computer described above can be an application-specific integrated circuit (ASIC), a general-purpose processor that executes functions by loading software programs, or a computer on a server used for cloud computing. Furthermore, the series of processes performed by each device described in this specification can be centrally processed by a single computer or distributed by multiple computers. Furthermore, in the above embodiments, two or more communication devices present in a single device can be physically implemented using a single medium.

[0266] Furthermore, the processes described using flowcharts or sequence diagrams in this specification do not necessarily need to be executed in the order shown. Several processing steps can be executed in parallel. In addition, additional processing steps can be used, and some processing steps can be omitted.

[0267] Furthermore, the effects described in this specification are descriptive or exemplary only, and not limiting. That is, the technology according to this disclosure can be used in conjunction with or in lieu of the effects described above to achieve other effects that will be apparent to those skilled in the art as described in this specification.

[0268] Note that the following configurations also fall within the technical scope of this disclosure.

[0269] (1) An information processing device, comprising:

[0270] A control unit configured to estimate camera parameters of the live-action video based on a first feature of a moving object present in a target space and a second feature of the moving object appearing in the live-action video obtained by imaging the target space.

[0271] (2) The information processing apparatus according to (1), wherein,

[0272] The first feature includes information indicating the three-dimensional position and orientation of a corresponding one or more parts constituting the moving object.

[0273] (3) The information processing apparatus according to (2), wherein,

[0274] The control unit renders the 3D model of the moving object based on the first feature quantity into a 2D image based on the candidate parameters of the camera device, and estimates the camera device parameters of the live-action video based on the similarity between the pose of the moving object appearing in the rendered 2D image and the pose of the moving object appearing in the 2D image constituting the live-action video.

[0275] (4) The information processing apparatus according to (3), wherein,

[0276] When the similarity is equal to or less than a predetermined threshold, the control unit continues to calculate the similarity while changing the candidates for the camera device parameters, and identifies the candidates for the camera device parameters whose similarity exceeds the predetermined threshold as the camera device parameters of the live video.

[0277] (5) The information processing apparatus according to (3) or (4), wherein,

[0278] The control unit sets a first search range including one or more candidates for the camera device parameters based on the metadata of the real-shot video, and estimates the camera device parameters from the set first search range.

[0279] (6) The information processing apparatus according to (5), wherein,

[0280] The metadata includes at least one of the following: information for identifying the imaging location of the live-action video, information for identifying the capture time of the live-action video, or information that can be used to identify the moving object appearing in the live-action video.

[0281] (7) The information processing apparatus according to any one of (3) to (6), wherein,

[0282] The first feature quantity is associated with the identification information of the moving object, and

[0283] The control unit calculates the similarity for moving objects associated with the same identification information.

[0284] (8) The information processing apparatus according to any one of (3) to (7), wherein,

[0285] The control unit renders the 3D model of the moving object corresponding to the first feature value acquired at the acquisition time corresponding to the capture time of the real-shot video into a 2D image based on the candidate parameters of the camera device, and calculates the similarity.

[0286] (9) The information processing apparatus according to (8), wherein,

[0287] The control unit sets a second search range including one or more candidates for the acquisition time based on the playback speed of the real-shot video, and estimates the acquisition time from the set second search range.

[0288] (10) The information processing apparatus according to any one of (1) to (9), wherein,

[0289] The control unit generates synchronous playback information, which associates the live-shot video, the first feature value acquired at the acquisition time corresponding to the capture time of the live-shot video, and the camera device parameters of the live-shot video.

[0290] (11) The information processing apparatus according to (10), wherein,

[0291] The synchronized playback information includes the playback speed of the actual video.

[0292] (12) The information processing apparatus according to any one of (1) to (11), wherein,

[0293] The camera device parameters include at least one of the camera device's three-dimensional position, orientation, or viewing angle.

[0294] (13) A terminal device, comprising:

[0295] A control unit configured to generate a playback screen on which a reproduced video is played back synchronously with a live-action video obtained by imaging a target space, the reproduced video being generated by rendering a 3D model generated based on a first feature quantity of a moving object present in the target space as a 2D image based on camera device parameters of the live-action video.

[0296] (14) The terminal device according to (13), wherein,

[0297] The control unit generates a playback screen that simultaneously displays the live-action video and the replayed video, or displays either the live-action video or the replayed video in a switchable manner.

[0298] (15) The terminal device according to (13) or (14), wherein,

[0299] The control unit generates the playback screen based on the synchronous playback information, and plays back the reproduced video synchronously with the live-action video on the playback screen. The synchronous playback information associates the live-action video, the first feature value acquired at the acquisition time corresponding to the capture time of the live-action video, and the camera device parameters of the live-action video.

[0300] (16) The terminal device according to (15), wherein,

[0301] The synchronized playback information includes the playback speed of the live-action video, and

[0302] The playback footage includes the playback speed of the actual video.

[0303] (17) An information processing method, comprising:

[0304] The camera device parameters of the live-shot video are estimated based on a first feature of the moving object existing in the target space and a second feature of the moving object appearing in the live-shot video obtained by imaging the target space.

[0305] (18) An information processing method, comprising:

[0306] A playback screen is generated, on which a reproduced video is played back synchronously with a live-action video obtained by imaging the target space. The reproduced video is generated by rendering a 3D model, generated based on a first feature quantity of a moving object existing in the target space, into a 2D image based on camera device parameters of the live-action video.

[0307] (19) A program for causing a computer to function as:

[0308] A control unit configured to estimate camera parameters of the live-action video based on a first feature of a moving object present in a target space and a second feature of the moving object appearing in the live-action video obtained by imaging the target space.

[0309] (20) A program for causing a computer to function as:

[0310] A control unit configured to generate a playback screen on which a reproduced video is played back synchronously with a live-action video obtained by imaging a target space, the reproduced video being generated by rendering a 3D model generated based on a first feature quantity of a moving object present in the target space as a 2D image based on camera device parameters of the live-action video.

[0311] List of reference numerals

[0312] 1 System

[0313] 2 servers

[0314] 3 User Terminals

Claims

1. An information processing apparatus, comprising: A control unit configured to estimate camera parameters of the live-action video based on a first feature of a moving object present in a target space and a second feature of the moving object appearing in the live-action video obtained by imaging the target space.

2. The information processing apparatus according to claim 1, wherein, The first feature includes information indicating the three-dimensional position and orientation of a corresponding one or more parts constituting the moving object.

3. The information processing apparatus according to claim 2, wherein, The control unit renders the 3D model of the moving object based on the first feature quantity into a 2D image based on the candidate parameters of the camera device, and estimates the camera device parameters of the live-action video based on the similarity between the pose of the moving object appearing in the rendered 2D image and the pose of the moving object appearing in the 2D image constituting the live-action video.

4. The information processing apparatus according to claim 3, wherein, When the similarity is equal to or less than a predetermined threshold, the control unit continues to calculate the similarity while changing the candidates for the camera device parameters, and identifies the candidates for the camera device parameters whose similarity exceeds the predetermined threshold as the camera device parameters of the live video.

5. The information processing apparatus according to claim 3, wherein, The control unit sets a first search range including one or more candidates for the camera device parameters based on the metadata of the real-shot video, and estimates the camera device parameters from the set first search range.

6. The information processing apparatus according to claim 5, wherein, The metadata includes at least one of the following: information for identifying the imaging location of the live-action video, information for identifying the capture time of the live-action video, or information that can be used to identify the moving object appearing in the live-action video.

7. The information processing apparatus according to claim 3, wherein, The first feature quantity is associated with the identification information of the moving object, and The control unit calculates the similarity for moving objects associated with the same identification information.

8. The information processing apparatus according to claim 3, wherein, The control unit renders the 3D model of the moving object corresponding to the first feature value acquired at the acquisition time corresponding to the capture time of the real-shot video into a 2D image based on the candidate parameters of the camera device, and calculates the similarity.

9. The information processing apparatus according to claim 8, wherein, The control unit sets a second search range including one or more candidates for the acquisition time based on the playback speed of the real-shot video, and estimates the acquisition time from the set second search range.

10. The information processing apparatus according to claim 1, wherein, The control unit generates synchronous playback information, which associates the live-shot video, the first feature value acquired at the acquisition time corresponding to the capture time of the live-shot video, and the camera device parameters of the live-shot video.

11. The information processing apparatus according to claim 10, wherein, The synchronized playback information includes the playback speed of the actual video.

12. The information processing apparatus according to claim 1, wherein, The camera device parameters include at least one of the camera device's three-dimensional position, orientation, or viewing angle.

13. A terminal device, comprising: A control unit configured to generate a playback screen on which a reproduced video is played back synchronously with a live-action video obtained by imaging a target space, the reproduced video being generated by rendering a 3D model generated based on a first feature quantity of a moving object present in the target space as a 2D image based on camera device parameters of the live-action video.

14. The terminal device according to claim 13, wherein, The control unit generates a playback screen that simultaneously displays the live-action video and the replayed video, or displays either the live-action video or the replayed video in a switchable manner.

15. The terminal device according to claim 13, wherein, The control unit generates the playback screen based on the synchronous playback information, and plays back the reproduced video synchronously with the live-action video on the playback screen. The synchronous playback information associates the live-action video, the first feature value acquired at the acquisition time corresponding to the capture time of the live-action video, and the camera device parameters of the live-action video.

16. The terminal device according to claim 15, wherein, The synchronized playback information includes the playback speed of the live-action video, and The playback footage includes the playback speed of the actual video.

17. An information processing method, comprising: The camera device parameters of the live-shot video are estimated based on a first feature of the moving object existing in the target space and a second feature of the moving object appearing in the live-shot video obtained by imaging the target space.

18. An information processing method, comprising: A playback screen is generated, on which a reproduced video is played back synchronously with a live-action video obtained by imaging the target space. The reproduced video is generated by rendering a 3D model, generated based on a first feature quantity of a moving object existing in the target space, into a 2D image based on camera device parameters of the live-action video.

19. A program that enables a computer to function as: A control unit configured to estimate camera parameters of the live-action video based on a first feature of a moving object present in a target space and a second feature of the moving object appearing in the live-action video obtained by imaging the target space.

20. A program that causes a computer to function as: A control unit configured to generate a playback screen on which a reproduced video is played back synchronously with a live-action video obtained by imaging a target space, the reproduced video being generated by rendering a 3D model generated based on a first feature quantity of a moving object present in the target space as a 2D image based on camera device parameters of the live-action video.

Images