Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that make it possible to easily synchronize a plurality of moving images. An information processing device according to the present technology comprises: a synchronization parameter estimation unit that estimates a synchronization parameter for synchronizing a plurality of moving images on the basis of time-series information, the time-series information indicates the posture of the subject in a frame image constituting each of a plurality of moving images in which the same subject is captured from a plurality of viewpoints; an integration unit that integrates time series information of each of the plurality of viewpoints synchronized by means of the synchronization parameter; and an updating unit that updates the synchronization parameter on the basis of an evaluation value that evaluates the time series information after integration. The present technology can be applied, for example, to a gaming machine that generates virtual avatar data for reproducing a motion of a subject on the basis of a plurality of moving images.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and particularly to an information processing device, an information processing method, and a program capable of easily synchronizing a plurality of video images. Background Art

[0002] As a method for 3D modeling objects with three-dimensional shapes, there is a technology called SfM (Structure from Motion), which generates a 3D model based on multiple images of the object obtained by photographing it from multiple directions. There is also a technology called NeRF (Neural Radiant Field), which generates an image or 3D model viewed from an arbitrary viewpoint based on the viewpoint and image of the camera that photographed the object.

[0003] When using these technologies to generate a 3D model based on video images of the same subject captured from multiple directions, it is necessary to synchronize the timing of capturing the frame images that make up the video images between all the cameras. To synchronize the timing of capturing the frame images between multiple cameras, for example, a method of sending a synchronization signal to each camera is used (for example, see PTL 1).

[0004] Reference List

[0005] Patent Literature

[0006] [PTL 1]

[0007] JP 2021-184540A Summary of the Invention

[0008] Technical issues

[0009] When using a synchronization signal to synchronize multiple video images, each camera device needs to be equipped with a mechanism for receiving the synchronization signal. Camera devices equipped with a mechanism for receiving the synchronization signal are usually expensive, so using a synchronization signal to synchronize multiple video images will bring high costs.

[0010] Furthermore, constructing an environment for transmitting synchronization signals to multiple cameras requires specialized knowledge and is time-consuming.

[0011] The present technology has been proposed in consideration of such a situation, and can easily synchronize a plurality of video images.

[0012] Solution to the problem

[0013] An information processing device according to one aspect of the present technology includes: a synchronization parameter estimation unit that estimates synchronization parameters for synchronizing multiple video images based on time series information, the time series information indicating the posture of a subject captured in a frame image of each of a plurality of video images constituting the same subject captured from a plurality of viewpoints; an integration unit that integrates the time series information of each of a plurality of viewpoints synchronized using the synchronization parameters; and an update unit that updates the synchronization parameters based on an evaluation value obtained by evaluating the time series information after integration.

[0014] In an information processing method according to one aspect of the present technology, an information processing device performs: estimating synchronization parameters for synchronizing multiple video images based on time series information, the time series information indicating the posture of a subject captured in a frame image of each of a plurality of video images constituting the same subject captured from a plurality of viewpoints; integrating the time series information of each of the plurality of viewpoints synchronized using the synchronization parameters; and updating the synchronization parameters based on an evaluation value obtained by evaluating the time series information after integration.

[0015] A program according to one aspect of the present technology causes a computer to perform the following processing: estimating synchronization parameters for synchronizing a plurality of video images based on time series information indicating a posture of a subject captured in a frame image constituting each of a plurality of video images capturing the same subject from a plurality of viewpoints; integrating the time series information of each of the plurality of viewpoints synchronized using the synchronization parameters; and updating the synchronization parameters based on an evaluation value obtained by evaluating the time series information after integration.

[0016] In one aspect of the present technology, synchronization parameters for synchronizing multiple video images are estimated based on time series information, which indicates the posture of a subject captured in a frame image of each of a plurality of video images constituting the same subject captured from a plurality of viewpoints, the time series information of each of the plurality of viewpoints synchronized using the synchronization parameters is integrated; and the synchronization parameters are updated based on an evaluation value obtained by evaluating the time series information after the integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] [ Figure 1 ]

[0018] Figure 1 : is a block diagram showing an example of the configuration of a photographing system according to an embodiment of the present technology.

[0019] [ Figure 2 ]

[0020] Figure 2 is a diagram showing an example of the arrangement of imaging devices.

[0021] [ Figure 3 ]

[0022] Figure 3 is a diagram illustrating an example of a process flow for synchronizing multiple video images.

[0023] [ Figure 4 ]

[0024] Figure 4 is a diagram illustrating an example of frame matching.

[0025] [ Figure 5 ]

[0026] Figure 5 is a diagram illustrating an example of sequence synthesis.

[0027] [ Figure 6 ]

[0028] Figure 6 is a diagram illustrating an example of a process for returning a composite sequence to the viewpoint of a camera.

[0029] [ Figure 7 ]

[0030] Figure 7 is a diagram illustrating an example of a general method for synchronizing a plurality of video images.

[0031] [ Figure 8 ]

[0032] Figure 8 is a diagram illustrating an example of a synchronization method of the present technology.

[0033] [ Figure 9 ]

[0034] Figure 9 is a diagram illustrating an example of a method for using synchronized video images.

[0035] [ Figure 10 ]

[0036] Figure 10 is a block diagram illustrating an example of a functional configuration of a calculator.

[0037] [ Figure 11 ]

[0038] Figure 11 is a flowchart illustrating the processing performed by the imaging system.

[0039] [ Figure 12 ]

[0040] Figure 12 is a diagram showing another example of the configuration of the imaging system.

[0041] [ Figure 13 ]

[0042] Figure 13 is a block diagram showing a configuration example of computer hardware. DETAILED DESCRIPTION

[0043] Hereinafter, embodiments for implementing the present technology will be described. The description will be given in the following order.

[0044] 1. Implementation Methods of the Present Technology

[0045] 2. Configuration and operation of each device

[0046] 3. Modification

[0047] <1. Implementation Methods of the Present Technology>

[0048] Figure 1 : is a block diagram showing an example of the configuration of a photographing system according to an embodiment of the present technology.

[0049] like Figure 1 As shown, the photographing system is configured by connecting the camera 1 - 1 to the camera 1 - 3 and the display 3 to the information processing apparatus 2 .

[0050] The camera 1-1 to the camera 1-3 are, for example, RGB cameras. The camera 1-1 to the camera 1-3 shoot the hands, the whole body, or other objects of a person as a subject and acquire video images. The camera 1-1 to the camera 1-3 shoot the same person or object as a subject. Figure 1 In the embodiment, three cameras 1-1 to 1-3 are connected to the information processing device 2, but in practice, any number of cameras can be connected to the information processing device 2. Hereinafter, when there is no need to distinguish between the camera 1-1 to the camera 1-3, they will be simply referred to as the camera 1.

[0051] The information processing device 2 includes a PC, a smartphone, a tablet terminal, a game console, etc. The information processing device 2 includes a memory 11 , a calculator 12 , and a display control unit 13 .

[0052] The memory 11 stores a plurality of video images captured by the camera 1 - 1 to the camera 1 - 3 .

[0053] The calculator 12 performs processing to synchronize the plurality of video images stored in the memory 11 .

[0054] The display control unit 13 controls the display 3 , and generates avatar data that reproduces the motion of a subject captured in a video image based on a plurality of video images synchronized by the calculator 12 , for example, and displays the avatar on the display 3 .

[0055] The display 3 displays various pieces of information under the control of the display control unit 13 .

[0056] Figure 2 An example of the arrangement of the imaging devices 1 - 1 to 1 - 3 is shown.

[0057] like Figure 2 As shown, three camera devices 1-1 to 1-3 are arranged around a person as a subject, for example, and capture the subject from multiple viewpoints to obtain video images. A display 3 is arranged in front of the person as a subject, for example.

[0058] Camera 1-1 is, for example, a web camera mounted on display 3 and captures a subject from the front. Camera 1-2 is, for example, a camera mounted on a smartphone and captures a subject from one side. Camera 1-3 is, for example, a digital camera and captures a subject from the other side.

[0059] Note that the camera 1 - 1 to the camera 1 - 3 have been calibrated, and therefore it is assumed that the viewpoints (positions and orientations) of the camera 1 - 1 to the camera 1 - 3 are known.

[0060] As described above, the cameras 1-1 to 1-3 are, for example, different types of cameras, and therefore the frame rates of the plurality of video images captured by the cameras 1-1 to 1-3 are different. Furthermore, if an environment for synchronizing the timing of capturing frame images is not prepared, the plurality of video images captured by the cameras 1-1 to 1-3 will not be synchronized.

[0061] The information processing apparatus 2 performs a process of synchronizing a plurality of video images that have different frame rates and are not synchronized as described above.

[0062] Figure 3 is a diagram illustrating an example of a process flow for synchronizing multiple video images.

[0063] First, if Figure 3As shown in #1 of , the information processing device 2 estimates the posture of the subject captured in all frame images constituting the plurality of video images captured by the camera devices 1-1 to 1-3. Hereinafter, the temporal position of a frame image in a video image is referred to as a frame, and the frame in which the frame image is captured is referred to as a captured frame. By estimating the posture of the subject, a sequence (time series information) is generated, which is information indicating the posture of the subject in the captured frames in the time series. For example, the posture of the subject in each captured frame is represented in three-dimensional coordinates based on the viewpoint of the camera device 1 that captured the frame image.

[0064] Next, if Figure 3 As shown in #2 of FIGURE 2, information processing device 2 performs frame interpolation to supplement frames between the captured frames included in the sequence for each viewpoint of camera 1-1 to camera 1-3. Hereinafter, the frames supplemented between the captured frames are referred to as interpolated frames. For example, in the sequence of viewpoints of camera 1-1, information processing device 2 estimates the coordinates of the subject's posture in the interpolated frames between these captured frames based on the coordinates of the subject's posture in the captured frame and the coordinates of the subject's posture in the next captured frame. The subject's posture in the interpolated frames is estimated through interpolation, extrapolation, prediction, or the like.

[0065] Next, the information processing device 2 performs frame matching on the sequence of viewpoints of each camera 1, as shown in FIG. Figure 3 As shown in #3.

[0066] Figure 4 A diagram showing an example of frame matching. Figure 4 In the figure, a group of solid or dotted circles connected by a single arrow indicates a sequence of viewpoints, and each circle indicates the coordinates of the pose of the subject in the corresponding frame. Solid circles indicate coordinates in the captured frame, and dotted circles indicate coordinates in the interpolated frame.

[0067] In frame matching, the reference viewpoint is determined, and a time window W1 is set, which includes K frames (K is an arbitrary number) starting from the starting frame of the sequence of the reference viewpoint, such as Figure 4 As shown in the upper part of . The reference viewpoint can be any one of the viewpoints of camera 1-1 to camera 1-3, or can be another viewpoint. For example, the viewpoint of camera 1-3 is determined as the reference viewpoint. When the viewpoint of camera 1 is set as the reference viewpoint, if the viewpoint of camera 1 with a high frame rate is determined as the reference viewpoint, the number of captured frames included in time window W1 increases, and the accuracy of frame matching is considered to be high.

[0068] exist Figure 4, the viewpoint sequence of cameras 1-3 includes the coordinates of the subject's posture in frames F1 to F10. Frames F1, F5, and F9 are captured frames, and frames F2, F3, F4, F6, F7, F8, and F10 are interpolated frames. Time window W1 includes three frames: frame F1, which is a captured frame, and frames F2 and F3, which are interpolated frames.

[0069] For example, the sequence of the viewpoint of the imaging device 1-1 as the subject of frame matching includes the coordinates of the posture of the subject in the frames F51 to F62, as shown in FIG. Figure 4 Frames F51, F56, and F61 are captured frames, and frames F52, F53, F54, F55, F57, F58, F59, F60, and F62 are interpolated frames.

[0070] In the sequence of viewpoints of the camera 1-1, as shown in FIG. Figure 4 As shown by the arrow A1, the information processing device 2 slides the time window W11 including three frames (K frames), and searches for three frames corresponding to frames F1 to F3 included in the time window W1 as shown by the arrow A2 (three frames shot at the same timing as frames F1 to F3).

[0071] When searching for a frame, for example, first, the coordinates of frames F51 to F62 based on the viewpoint of camera 1-1 are converted to coordinates based on the viewpoint of camera 1-3. Then, the errors or correlation coefficients between the coordinates of frames F1 to F3 and the coordinates of three frames included in time window W11 are calculated as similarities, and three frames similar to frames F1 to F3 are searched.

[0072] Hereinafter, for example, frames F55 to F57 are searched as three frames corresponding to frames F1 to F3 .

[0073] As described above, in frame matching, corresponding frames between the sequence of the reference viewpoint and the sequence of the viewpoint of the target camera 1 are matched based on the similarity between the time window W1 and the time window W11. In other words, a tentative offset of the video image relative to the target camera 1 is estimated. Here, the offset indicates the frame shift between the starting frame in the sequence of the reference viewpoint and the frame corresponding to the starting frame in the sequence (video image) of the viewpoint of the camera 1.

[0074] Back to Figure 3 Next, as shown in #4, the information processing device 2 optimizes the synchronization parameters. The synchronization parameters include the offset of each video image (sequence), the number of interpolation frames between shooting frames, and the like.

[0075] When estimating the synchronization parameters, first, as Figure 3As shown in #11 of , sequence synthesis is performed to integrate the sequence of the viewpoint of each camera 1.

[0076] Figure 5 is a diagram showing an example of sequence synthesis. Figure 5 , an example of integrating the sequence of the viewpoint of the camera 1 - 3 and the sequence of the viewpoint of the camera 1 - 1 is described.

[0077] As described above, frame matching determined that frames F1 to F13 correspond to frames F55 to F67, respectively. In this case, information processing device 2 synchronizes the viewpoint sequence of camera 1-3 with the viewpoint sequence of camera 1-1 by associating frames F1 to F13 with frames F55 to F67, respectively. Information processing device 2 then generates a composite sequence by referencing the coordinates of the captured frame within the coordinates of each frame included in the viewpoint sequence of each camera 1. The composite sequence is a sequence in which the viewpoint sequence of each camera 1 is integrated.

[0078] exist Figure 5 In the example, the coordinates of the posture of the subject in frames F1, F56, F5, F61, F9, F66, and F13 are converted into coordinates based on the viewpoint of the synthetic sequence, and the converted coordinates are set as the coordinates of the posture of the subject in frames F101 to F107 of the synthetic sequence.

[0079] Then, as shown by the tip of arrow A11, frame interpolation is performed again to supplement frames between frames F102 and F103, between frames F103 and F104, between frames F104 and F105, and between frames F105 and F106, and a synthetic sequence including frames F101c to F113c is generated. Frames F101c, F102c, F105c, F107c, F109c, F112c, and F113c correspond to frames F101 to F107, respectively, and frames F103c, F104c, F106c, F108c, F110c, and F111c are interpolated frames.

[0080] Back to Figure 3 , in the optimization of the synchronization parameters, next, as shown in #12, a process of returning the synthesized sequence to the viewpoint of each camera 1 is performed.

[0081] Figure 6 1 - 1 is a diagram illustrating an example of a process of returning a composite sequence to the viewpoint of the camera 1 - 1 .

[0082] exist Figure 6In the example shown in FIG, the synthesized sequence includes the coordinates of the subject's posture in frames F121c to F132c, and the original (at the time of shooting) sequence of the camera 1-1 includes the coordinates of the subject's posture in frames F71 to F73. Frames F71, F72, and F73 correspond to frames F124c, F128c, and F132c, respectively.

[0083] The information processing apparatus 2 generates a final sequence of the camera 1-1, which indicates the coordinates of the posture of the subject in frames F71a to F82a, by returning the synthesized sequence to the viewpoint of the camera 1-1.

[0084] Specifically, the information processing device 2 uses the coordinates of frames F71, F72, and F73 included in the original sequence of the camera 1-1 as they are as the coordinates of frames F74a, F78a, and F82a (captured frames) included in the final sequence. Furthermore, the information processing device 2 uses the coordinates of the frames corresponding to the interpolated frames included in the synthesized sequence as the coordinates of the interpolated frames between the captured frames included in the final sequence, and converts these coordinates into coordinates based on the viewpoint of the camera 1-1.

[0085] exist Figure 6 , circles indicating coordinates in the frame F71 , the frame F72 , the frame F73 , the frame F74 a , the frame F78 a , and the frame F82 a are surrounded by rectangles to indicate that the frame images in the frames are captured by the camera 1 - 1 .

[0086] The information processing device 2 may generate an interpolation image based on at least one of the frame image captured in the captured frame and the coordinates of the posture of the subject in the interpolation frame, which is a frame image in the interpolation frame without the captured frame image.

[0087] Specifically, the information processing device 2 obtains an interpolated image by inputting the frame image captured in the captured frame, the coordinates of the subject's posture in the interpolated frame, etc. into a DNN (deep neural network). For example, the information processing device 2 generates an interpolated image using a technology for generating an image that complements a frame between two frame images based on a frame image (e.g., frame interpolation) or a technology for generating an image based on a subject's posture and a sample image (e.g., gesture GAN (generative adversarial network)).

[0088] By generating the interpolated image, the information processing apparatus 2 can obtain a video image having a higher frame rate than the original video image captured by the camera 1-1 from the viewpoint of the camera 1-1.

[0089] Back to Figure 3 ,In the optimization of the synchronization parameters, next, as shown in #13, a sequence evaluation is performed to ,evaluate the synthesized sequence.

[0090] Specifically, when shooting is performed by three or more cameras 1 (when shooting is performed from three or more viewpoints), the information processing device 2 estimates (reintegrates) the three-dimensional coordinates indicating the posture of the subject in each frame based on the final sequence of the viewpoints of the three or more cameras 1, and converts the estimated coordinates back into coordinates based on the viewpoint of each camera 1. The information processing device 2 calculates a reprojection error by comparing the coordinates of the posture of the subject captured in the final sequence with the coordinates of the posture of the subject obtained by the reconversion. The average or median value of the reprojection errors of each camera 1 is used as an evaluation value for the synthesized sequence.

[0091] When shooting is performed by two cameras 1 (when shooting is performed from two viewpoints), the information processing device 2 calculates the naturalness of the synthesized sequence. The naturalness of the synthesized sequence is represented by, for example, the smoothness of the subject's motion defined based on acceleration error and variance, the authenticity determination result of the DNN, etc., and is used as an evaluation value of the synthesized sequence.

[0092] After performing sequence evaluation, the synchronization parameters are updated based on the evaluation values. For example, when the evaluation values are used as costs, the synchronization parameters are updated through discrete optimization or gradient descent. After the synchronization parameters are updated, the process after sequence synthesis (process #11 to process #13) is repeated to optimize the synchronization parameters.

[0093] The information processing device 2 can synchronize the video images captured by the cameras 1 - 1 to 1 - 3 using the optimized synchronization parameters.

[0094] Figure 7 is a diagram showing an example of a general method for synchronizing multiple video images. Figure 7 In the following, an example of synchronizing two video images captured by camera A and camera B will be described. The video image from camera A includes frame images captured in frames F1A to F3A, and the video image from camera B includes frame images captured in frames F1B to F3B. These two video images have different frame rates and are not synchronized.

[0095] In this case, if Figure 7 As shown in the upper part of , the frame shift (offset of the video image from camera B) between frame F1A is calculated, where frame F1A is the starting frame of the video image from camera A and frame F1B is the starting frame of the video image from camera B. Then, as Figure 7 As shown at the tip of the white arrow in the middle, the video image from the camera A and the video image from the camera B are synchronized by making the frame F1B correspond to the frame F1A.

[0096] In reference Figure 7The synchronization method described above can only synchronize video images by matching the timing (frames) of capturing frame images. Therefore, if the capturing timing of frame F1A actually matches the timing t1 between, for example, frames F1B and F2B, this synchronization method cannot correctly synchronize the video images from camera A with the video images from camera B.

[0097] In addition, since the video image from camera A and the video image from camera B have different frame rates, even if the timing of the start frames in the two video images are aligned, frames other than the start frame will not be aligned.

[0098] Figure 8 : is a diagram showing an example of a synchronization method of the present technology. Figure 8 In, with Figure 7 Likewise, an example of synchronizing two video images captured by camera A and camera B will be described.

[0099] In the synchronization method of this technology, if Figure 8 As shown, corresponding frames are searched among the captured frames (frames F1A to F3A and frames F1B to F3B) and the interpolated frames (frames F1A-1 to F1A-3, frames F2A-1 to F2A-3, frame F3A-1, frames F1B-1 to F1B-3, frames F2B-1 to F2B-3, and frame F3B-1).

[0100] For example, by adding frames between captured frames so that the frame rates of the video images from camera A and camera B are the same, the frame in the video image from camera B corresponding to frame F1A in the video image from camera A can be accurately searched, and there will be no deviation in each frame thereafter.

[0101] Figure 9 is a diagram illustrating an example of a method of using synchronized video images.

[0102] The information processing device 2 synchronizes a plurality of video images having different frame rates and being asynchronous, thereby acquiring data of a plurality of synchronized video images (synchronization data), such as Figure 9 The synchronization data is data of high frame rate video images as if they were shot from multiple viewpoints.

[0103] The information processing device 2 can use such synchronization data to restore the 3D shape of the subject and obtain high frame rate virtual avatar data representing the movement of the subject, such as Figure 9 Note that the use of a high frame rate video image data and synchronization data is not limited to generating avatar data, and such data can be used for gaming, metaverse, video playback, etc.

[0104] Conventionally, to obtain a 3D model of a subject using video images of the same subject captured synchronously from multiple directions, methods have been used. For example, a synchronization signal is sent to each camera to synchronize the capture timing of all frame images within the camera. Therefore, each camera must have a mechanism for receiving the synchronization signal. Cameras equipped with a mechanism for receiving synchronization signals are typically expensive, so synchronizing multiple video images using synchronization signals is costly. Furthermore, establishing an environment for transmitting synchronization signals to multiple cameras requires specialized knowledge and is time-consuming.

[0105] With this technology, even if a user uses multiple camera devices 1 with different prices and frame rates (e.g., webcams, cameras installed on smartphones, and digital cameras) to take pictures, the user can obtain multiple synchronized video images with a high frame rate or avatar data with a high frame rate. Therefore, the user can easily obtain multiple synchronized video images without having to prepare an expensive camera device equipped with a mechanism for receiving synchronization signals or constructing an environment for transmitting multiple synchronization signals.

[0106] <2. Configuration and Operation of Each Device>

[0107] Figure 10 is a block diagram showing an example of the functional configuration of the calculator 12 .

[0108] like Figure 10 As shown, the calculator 12 includes a data acquisition unit 21 , a subject pose estimation unit 22 , an interpolation unit 23 , an offset estimation unit 24 , a sequence processing unit 25 , an interpolated image generation unit 26 , an evaluation unit 27 , and an optimization unit 28 .

[0109] The data acquisition unit 21 acquires a plurality of video images captured by a plurality of cameras 1 from the memory 11 , and supplies them to the subject pose estimation unit 22 and the interpolation image generation unit 26 .

[0110] The subject posture estimation unit 22 estimates the posture of the subject captured in all frame images constituting the plurality of video images supplied from the data acquisition unit 22 and generates a sequence of coordinates indicating the posture of the subject in the captured frames in time series based on the viewpoint of each camera 1 .

[0111] The pose of a subject is estimated using a technique called 3D pose estimation, which identifies information about a person's pose based on still and video images of the person. 3D pose estimation estimates the three-dimensional coordinates of a person's joints based on still and video images taken from a single viewpoint or multiple viewpoints. These three-dimensional coordinates include the world coordinates of the joints, camera coordinates, and relative coordinates with the waist as the origin.

[0112] The subject pose estimation unit 22 supplies the sequence of viewpoints of each camera 1 to the interpolation unit 23 .

[0113] Interpolation unit 23 performs frame interpolation for each viewpoint in the multiple sequences provided by subject pose estimation unit 22. If synchronization parameter optimization has not yet been performed, interpolation unit 23 estimates the number of interpolated frames between captured frames based on the frame rate of each video image and performs frame interpolation. Interpolation unit 23 estimates the number of interpolated frames between captured frames so that, for example, the frame rate of each sequence is the same. Furthermore, if synchronization parameters have already been optimized, interpolation unit 23 sets the number of interpolated frames between captured frames under the control of optimization unit 28 and performs frame interpolation.

[0114] The interpolation unit 23 supplies a plurality of sequences in which frames between captured frames have been supplemented to the offset estimation unit 24 and the sequence processing unit 25 .

[0115] The offset estimation unit 24 performs frame matching on each of the plurality of sequences supplied from the interpolation unit 23 to estimate the offset of each sequence, and supplies information indicating the offset of each sequence to the sequence processing unit 25 .

[0116] Note that the interpolation unit 23 and the offset estimation unit 24 function as a synchronization parameter estimation unit that estimates a tentative offset for each video image (sequence) and a tentative synchronization parameter including a tentative number of interpolation frames between captured frames.

[0117] When the synchronization parameter is not optimized, the sequence processing unit 25 synchronizes the multiple sequences supplied from the interpolation unit 23 using the offset estimated by the offset estimation unit 24. When the synchronization parameter is optimized, the sequence processing unit 25 synchronizes the multiple sequences using the offset set according to the control of the optimization unit 28.

[0118] After synchronizing the multiple sequences, the sequence processing unit 25 performs sequence synthesis to generate a synthesized sequence. The sequence processing unit 25 functions as an integration unit that integrates the multiple sequences. The sequence processing unit 25 returns the synthesized sequence to the viewpoint of each camera 1 to generate a final sequence for the viewpoint of each camera 1.

[0119] The sequence processing unit 25 supplies the final sequence for the viewpoint of each camera 1 to the interpolation image generation unit 26. The sequence processing unit 25 also supplies the synthesized sequence and the final sequence for the viewpoint of each camera 1 to the evaluation unit 27.

[0120] The interpolation image generation unit 26 generates an interpolation image based on the multiple video images supplied from the data acquisition unit 21 and the final sequence for each viewpoint of the camera 1 supplied from the sequence processing unit 25 , and acquires multiple video images with a high frame rate.

[0121] The evaluation unit 27 performs sequence evaluation based on the various sequences supplied from the sequence processing unit 25 to calculate evaluation values, and supplies information indicating the evaluation values to the optimization unit 28 .

[0122] The optimization unit 28 controls the interpolation unit 23 and the sequence processing unit 25 to update the synchronization parameter based on the evaluation value calculated by the evaluation unit 27. The optimization unit 28 functions as an updating unit that updates the synchronization parameter. The optimization unit 28 optimizes the synchronization parameter by repeatedly updating the synchronization parameter.

[0123] Next, we will refer to Figure 11 The flowchart describes the processing performed by the photography system.

[0124] In step S1 , the photographic system performs camera calibration, and the information processing device 2 acquires viewpoints of a plurality of cameras 1 .

[0125] In step S2 , each of the plurality of camera devices 1 captures an image and acquires a plurality of video images.

[0126] In step S3 , the subject pose estimation unit 22 of the information processing device 2 estimates the poses of the subject captured in all frame images constituting a plurality of video images (poses of the subject in all captured frames) and generates a sequence of viewpoints of each camera 1 .

[0127] In step S4 , the interpolation unit 23 of the information processing device 2 performs frame interpolation for each viewpoint of the camera 1 .

[0128] In step S5, the offset estimation unit 24 of the information processing device 2 performs frame matching on the sequence of viewpoints of each camera 1, which supplements the frames between the captured frames, and estimates a tentative offset for each sequence. The sequence processing unit 25 of the information processing device 2 uses the offset estimated by the offset estimation unit 24 to synchronize each sequence.

[0129] In step S6 , the sequence processing unit 25 performs sequence synthesis and generates a synthesized sequence.

[0130] In step S7 , the sequence processing unit 25 returns the synthesized sequence to the viewpoint of each camera 1 , and generates a final sequence of the viewpoint of each camera 1 .

[0131] In step S8 , the interpolation image generation unit 26 of the information processing device 2 generates an interpolation image, and acquires a plurality of high frame rate video images.

[0132] In step S9 , the evaluation unit 27 of the information processing device 2 performs sequence evaluation and calculates an evaluation value.

[0133] In step S10 , the optimization unit 28 of the information processing device 2 determines whether the evaluation value is equal to a threshold value or greater than the threshold value.

[0134] If it is determined in step S10 that the evaluation value is less than the threshold, then in step S11, the optimization unit 28 updates the synchronization parameters. After updating the synchronization parameters, the process returns to step S6 and the process from step S6 is repeated until the evaluation value becomes equal to or greater than the threshold.

[0135] On the other hand, if it is determined in step S10 that the evaluation value is equal to or greater than the threshold, for example, when the evaluation value becomes equal to or greater than the threshold, the calculator 12 synchronizes the plurality of high frame rate video images using the offset, and the process ends.

[0136] When the process of step S6 to step S11 has been performed a predetermined number of times, even if the evaluation value does not become equal to the threshold value or greater than the threshold value, Figure 11 The process can also be terminated. The number of times the process is terminated is, for example, preset by the user.

[0137] In the above process, synchronization parameters for synchronizing multiple video images are estimated based on time series information indicating the posture of a subject captured in frame images constituting multiple video images of the same subject captured from multiple viewpoints, and the time series information of each viewpoint synchronized using the synchronization parameters is integrated, and the synchronization parameters are updated based on an evaluation value obtained by evaluating the time series information after integration.

[0138] The user can obtain multiple synchronized video images by simply installing a web camera or the like that the user owns at home and taking pictures, without preparing an expensive camera equipped with a mechanism for receiving synchronization signals or constructing an environment for transmitting multiple synchronization signals.

[0139] <3. Modification>

[0140] Figure 12 is a diagram showing another example of the configuration of the photographic system. Figure 12 In the figure, the same reference numerals are used to represent Figure 1 Components that are identical to those in . Redundant descriptions will be omitted as appropriate.

[0141] Figure 12 The photography system and Figure 1 The photographic system of is different in that the information processing device 2 is not provided, and the camera 1 - 2 and the camera 1 - 3 and the display 3 are connected to the camera 1 - 1 . Figure 12 The photography system and Figure 1 The photographing system of FIG. 1 is also different in that a memory 11A, a calculator 12A, and a display control unit 13A are provided in the image pickup apparatus 1 - 1 .

[0142] The memory 11A, the calculator 12A, and the display control unit 13A correspond to Figure 1 The memory 11, the calculator 12, and the display control unit 13 of the information processing device 2 are similar to each other and have similar functions.

[0143] In this manner, some or all of the functions of the information processing device 2 can be provided in devices other than the information processing device 2 , such as the camera 1 and the display 3 .

[0144] In the above, an example has been described in which the image pickup device 1 is configured as an RGB image pickup device, but the image pickup device 1 may be configured as a grayscale image pickup device or a depth sensor such as an iToF (indirect time of flight) sensor.

[0145] The imaging device 1 and the information processing device 2 may be connected via a network such as the Internet.

[0146] About computers

[0147] The above series of processing can be executed by hardware or software. When the series of processing is executed by software, the program constituting the software is installed from a program recording medium to a computer embedded in dedicated hardware, a general-purpose personal computer, or the like.

[0148] Figure 13 : is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes according to a program.

[0149] A central processing unit (CPU) 501 , a read only memory (ROM) 502 , and a random access memory (RAM) 503 are connected to one another via a bus 504 .

[0150] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, and the like, and an output unit 507 including a display, a speaker, and the like are connected to the input / output interface 505. Furthermore, a storage unit 508 including a hard disk, a nonvolatile memory, and the like, a communication unit 509 including a network interface and the like, and a drive 510 that drives a removable medium 511 are connected to the input / output interface 505.

[0151] In the computer configured as described above, for example, the CPU 501 performs the above-described series of processing by loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing the program.

[0152] The program executed by the CPU 501 is recorded on, for example, the removable medium 511 , or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and installed in the storage unit 508 .

[0153] The program executed by the computer may be a program that chronologically executes processing in the order described herein, or may be a program that executes processing in parallel or when necessary, such as when a call is made.

[0154] At the same time, as used herein, a system means a collection of multiple components (e.g., devices, modules (components), etc.), and all components may be located in the same housing or not. Therefore, a plurality of devices housed in separate housings and connected via a network and a single device housing multiple modules in a single housing both constitute a system.

[0155] The effects described herein are merely exemplary and non-limiting, and other effects may be obtained.

[0156] The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

[0157] For example, the present technology may be configured as cloud computing in which individual functions are collaboratively processed in a distributed manner via a network.

[0158] Furthermore, each step described in the flowcharts discussed above may be performed by a single device, or may be performed by multiple devices in a distributed manner.

[0159] Furthermore, when a single step includes a plurality of processing types, the plurality of processing types included in the single step may be performed by a single device, or may be performed by a plurality of devices in a distributed manner.

[0160] Configuration combination example

[0161] The present technology can be configured as follows. (1)

[0163] An information processing device, comprising:

[0164] a synchronization parameter estimation unit that estimates synchronization parameters for synchronizing a plurality of video images based on time series information indicating a posture of a subject captured in frame images constituting each of a plurality of video images capturing the same subject from a plurality of viewpoints;

[0165] an integrating unit that integrates time series information of each of a plurality of viewpoints synchronized using a synchronization parameter; and

[0166] An updating unit updates a synchronization parameter based on an evaluation value obtained by evaluating the time series information after integration. (2)

[0168] The information processing device according to (1), further comprising:

[0169] an interpolation unit that estimates a posture of a subject in an interpolation frame in time series information of each of a plurality of viewpoints, the interpolation frame supplementing a frame between captured frames of the captured frame image, wherein

[0170] The synchronization parameter estimation unit estimates the synchronization parameter based on the time series information supplemented with the posture of the subject in the interpolation frame. (3)

[0172] The information processing device according to (2), wherein

[0173] The synchronization parameters include the offset of the multiple video images and the number of interpolated frames between captured frames. (4)

[0175] The information processing device according to (3), wherein

[0176] The synchronization parameter estimation unit estimates the offset of each video image captured from a predetermined viewpoint based on the similarity between a first time window including a predetermined number of frames from a start frame of time series information of a reference viewpoint and a second time window including a predetermined number of frames included in the time series information of the predetermined viewpoint. (5)

[0178] The information processing device according to (3) or (4), wherein

[0179] The synchronization parameter estimation unit estimates the number of interpolation frames between captured frames based on a frame rate of each of the plurality of video images. (6)

[0181] The information processing apparatus according to any one of (2) to (5), further comprising:

[0182] An interpolation image generating unit generates an interpolation image as each frame image in the interpolation frame. (7)

[0184] The information processing device according to any one of (2) to (6), wherein

[0185] The integration unit generates integrated time series information by referring to the posture of the subject in the captured frame among the postures of the subject in the frames included in the time series information of each viewpoint among multiple viewpoints, and estimates the posture of the subject in the interpolated frames that are supplementary frames between the captured frames included in the time series information after integration. (8)

[0187] The information processing apparatus according to any one of (1) to (7), further comprising:

[0188] An evaluation unit that calculates an error between final time series information of each of a plurality of viewpoints and time series information generated based on the time series information after reintegration for each of the plurality of viewpoints as an evaluation value, wherein the final time series information is generated based on the time series information after integration and the time series information before integration when a subject is photographed from three or more viewpoints, and the time series information after reintegration is obtained by reintegrating the final time series information of each of the plurality of viewpoints. (9)

[0190] The information processing apparatus according to any one of (1) to (7), further comprising:

[0191] An evaluation unit that calculates, when a subject is captured from two viewpoints, a naturalness of time series information after integration as an evaluation value. (10)

[0193] The information processing device according to any one of (1) to (9), wherein

[0194] an integrating unit integrating time series information of each of the plurality of viewpoints synchronized using the synchronization parameters updated by the updating unit, and

[0195] The updating unit optimizes the synchronization parameters by repeatedly updating the synchronization parameters. (11)

[0197] The information processing device according to (10), wherein

[0198] The updating unit repeats the updating of the synchronization parameter until the evaluation value becomes equal to or greater than a predetermined threshold value, or until the synchronization parameter has been updated a predetermined number of times. (12)

[0200] The information processing apparatus according to any one of (1) to (11), further comprising:

[0201] The camera device is used to shoot a subject and obtain one video image among a plurality of video images. (13)

[0203] An information processing method, wherein an information processing device performs:

[0204] estimating a synchronization parameter for synchronizing a plurality of video images based on time series information indicating a posture of a subject captured in a frame image constituting each of a plurality of video images capturing the same subject from a plurality of viewpoints;

[0205] integrating time series information of each of the plurality of viewpoints synchronized using a synchronization parameter; and

[0206] The synchronization parameter is updated based on the evaluation value obtained by evaluating the time series information after integration. (14)

[0208] A program that causes a computer to execute the following processing:

[0209] estimating a synchronization parameter for synchronizing a plurality of video images based on time series information indicating a posture of a subject captured in a frame image constituting each of a plurality of video images capturing the same subject from a plurality of viewpoints;

[0210] integrating time series information of each of the plurality of viewpoints synchronized using a synchronization parameter; and

[0211] The synchronization parameter is updated based on the evaluation value obtained by evaluating the time series information after integration.

[0212] Reference Signs List

[0213] 1-1, 1-2, 1-3 Camera Device

[0214] 2Information processing device

[0215] 3 monitors

[0216] 11 Memory

[0217] 12Calculator

[0218] 13 Display control unit

[0219] 21 Data acquisition unit

[0220] 22 Subject pose estimation unit

[0221] 23 interpolation units

[0222] 24 offset estimation units

[0223] 25 sequence processing units

[0224] 26 interpolation image generation unit

[0225] 27 evaluation units

[0226] 28 optimization units

Claims

1. An information processing device, comprising: a synchronization parameter estimation unit that estimates synchronization parameters for synchronizing a plurality of video images based on time series information indicating a posture of a subject captured in frame images constituting each of the plurality of video images capturing the same subject from a plurality of viewpoints; an integrating unit that integrates time series information of each of the plurality of viewpoints synchronized using the synchronization parameter; as well as An updating unit updates the synchronization parameter based on an evaluation value obtained by evaluating the time series information after integration.

2. The information processing apparatus according to claim 1, further comprising: an interpolation unit that estimates the posture of the subject in an interpolation frame from time series information of each of the plurality of viewpoints, the interpolation frame supplementing a frame between captured frames for capturing the frame image, wherein The synchronization parameter estimation unit estimates the synchronization parameter based on time series information supplemented with the posture of the subject in the interpolation frame.

3. The information processing device according to claim 2, wherein: The synchronization parameters include an offset of the plurality of video images and a number of interpolated frames between the captured frames.

4. The information processing device according to claim 3, wherein: The synchronization parameter estimation unit estimates the offset of each of the video images taken from a predetermined viewpoint based on the similarity between a first time window and a second time window, wherein the first time window includes a predetermined number of frames from the starting frame of the time series information of the reference viewpoint, and the second time window includes a predetermined number of frames included in the time series information of the predetermined viewpoint. The information processing apparatus according to claim 3 , wherein: The synchronization parameter estimation unit estimates the number of the interpolation frames between the captured frames based on a frame rate of each of the plurality of video images.

6. The information processing apparatus according to claim 2, further comprising: An interpolation image generating unit generates an interpolation image as each of the frame images in the interpolation frames.

7. The information processing apparatus according to claim 2, wherein: The integration unit generates the integrated time series information by referring to the posture of the subject in the captured frame among the postures of the subject in the frames included in the time series information of each of the multiple viewpoints, and estimates the posture of the subject in the interpolated frames that are supplementary frames between the captured frames included in the integrated time series information.

8. The information processing apparatus according to claim 1, further comprising: An evaluation unit that calculates an error between final time series information of each of the multiple viewpoints and time series information generated based on the reintegrated time series information of each of the multiple viewpoints as an evaluation value, wherein the final time series information is generated based on the time series information after integration and the time series information before integration when the subject is photographed from three or more viewpoints, and the reintegrated time series information is obtained by reintegrating the final time series information of each of the multiple viewpoints.

9. The information processing apparatus according to claim 1, further comprising: An evaluation unit calculates a naturalness of the time series information after integration as an evaluation value when the subject is photographed from two viewpoints.

10. The information processing apparatus according to claim 1, wherein: The integrating unit integrates the time series information of each of the plurality of viewpoints synchronized using the synchronization parameter updated by the updating unit, and The updating unit optimizes the synchronization parameter by repeatedly updating the synchronization parameter. The information processing apparatus according to claim 10 , wherein: The updating unit repeats the updating of the synchronization parameter until the evaluation value becomes equal to or greater than a predetermined threshold value, or until the synchronization parameter has been updated a predetermined number of times.

12. The information processing apparatus according to claim 1, further comprising: A camera device is used to shoot the subject and obtain one video image among the multiple video images.

13. An information processing method, wherein: The information processing device executes: estimating a synchronization parameter for synchronizing a plurality of video images based on time series information indicating a posture of a subject captured in a frame image constituting each of the plurality of video images capturing the same subject from a plurality of viewpoints; integrating time series information of each of the plurality of viewpoints synchronized using the synchronization parameter; as well as The synchronization parameter is updated based on an evaluation value obtained by evaluating the time series information after integration.

14. A program for causing a computer to execute the following processing: estimating a synchronization parameter for synchronizing a plurality of video images based on time series information indicating a posture of a subject captured in a frame image constituting each of the plurality of video images capturing the same subject from a plurality of viewpoints; integrating time series information of each of the plurality of viewpoints synchronized using the synchronization parameter; as well as The synchronization parameter is updated based on an evaluation value obtained by evaluating the time series information after integration.

Images