Information processing device, information processing method, and program

Abstract

To generate a virtual viewpoint path that smoothly changes the virtual viewpoint. [Solution] A key frame setting unit reads out a generated virtual viewpoint path that serves as a base and is stored in a virtual viewpoint path storage unit. A key frame number receiving unit acquires an input value as the number of key frames. The key frame setting unit selects virtual viewpoint parameters for the acquired number of key frames from the virtual viewpoint parameters for each frame included in the read virtual viewpoint path, and sets them as virtual viewpoint parameters of the key frames. A virtual viewpoint path regeneration unit interpolates virtual viewpoint parameters corresponding to each frame between key frames based on the virtual viewpoint parameters of the key frames stored in the key frame storage unit.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an information processing technology for generating a virtual viewpoint image. [Background technology]

[0002] There is a technology that generates a virtual viewpoint image representing the view from a virtual viewpoint virtually located in three-dimensional space using multiple captured images obtained by synchronously capturing the same subject using multiple imaging devices installed at different locations. This technology allows viewers of the virtual viewpoint image to generate a virtual viewpoint path themselves, and virtual viewpoint images rendered based on the generated virtual viewpoint path can be distributed to other viewers. One method of generating a virtual viewpoint path involves a user manually specifying position information indicating the position of the virtual viewpoint at short intervals, such as every frame, when specifying time-series virtual viewpoint parameters that constitute the virtual viewpoint path. This method of manually specifying position information allows the user to continuously input virtual viewpoint parameters corresponding to each of multiple consecutive frames, making it easy to generate a virtual viewpoint path. However, the position and orientation information of the virtual viewpoint specified manually by the user is stored as is as the time-series virtual viewpoint parameters that constitute the virtual viewpoint path. Therefore, a virtual viewpoint path that includes a sudden change in the position or orientation of the virtual viewpoint, such as unintentional blurring due to user error, may be stored.

[0003] Patent Document 1 discloses a method for suppressing abrupt changes in a virtual viewpoint image by changing the playback speed of the virtual viewpoint image in order to reduce visual sickness caused by abrupt changes in the generated virtual viewpoint path. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-173577 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology disclosed in Patent Document 1, the virtual viewpoint path, which includes unintentional blurring due to user operation errors, remains in the state set by the user, and there is a problem that abrupt changes in the virtual viewpoint image may occur depending on the playback speed. [Means for solving the problem]

[0006] The present disclosure is characterized by comprising an acquisition means for acquiring a first virtual viewpoint path that indicates the trajectory of change of a virtual viewpoint used to generate a virtual viewpoint image, a selection means for selecting a plurality of virtual viewpoints included in the first virtual viewpoint path, and a generation means for generating a second virtual viewpoint path by interpolating between the plurality of virtual viewpoints selected by the selection means. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to generate a virtual viewpoint path that allows the virtual viewpoint to change smoothly. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of a virtual viewpoint image generation system including an information processing device according to a first embodiment of the present disclosure. [Figure 3] FIG. 10 illustrates an example UI for editing a virtual viewpoint path according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a diagram illustrating an example of a UI for regenerating a virtual viewpoint path according to the first embodiment of the present disclosure. [Figure 5] Flowchart for explaining a process for regenerating a virtual viewpoint path according to the first embodiment of the present disclosure. [Figure 6] An example of setting keyframe information from specified virtual viewpoint path information. [Figure 7]FIG. 1 is a schematic diagram illustrating a virtual viewpoint path before and after regeneration according to the first embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram illustrating an example of the configuration of a virtual viewpoint image generation system including an information processing device according to a second embodiment of the present disclosure. [Figure 9] FIG. 10 is a diagram illustrating an example of a UI for regenerating a virtual viewpoint path according to the second embodiment of the present disclosure. [Figure 10] Flowchart for explaining a virtual viewpoint path regeneration process according to the second embodiment of the present disclosure [Figure 11] 10 is a flowchart illustrating a process for detecting a sudden change in a regeneration interval according to a second embodiment of the present disclosure. [Figure 12] 10 is a flowchart illustrating a process for setting a key frame excluding a sharp change portion according to a second embodiment of the present disclosure. [Figure 13] An example of setting keyframe information from specified virtual viewpoint path information. [Figure 14] FIG. 10 is a schematic diagram illustrating a virtual viewpoint path before and after regeneration according to a second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The following embodiments do not limit the present disclosure, and not all of the combinations of features described in the present embodiments are necessarily essential to the solutions of the present disclosure. Various other embodiments are also included in the present disclosure without departing from the gist of the present disclosure, and parts of the following embodiments can also be combined as appropriate.

[0010] [Embodiment 1] 1-1 Overview Hereinafter, a virtual viewpoint path regeneration process using the information processing device 10 according to the first embodiment of the present disclosure will be described with reference to FIGS.

[0011] 1-2 Configuration The hardware configuration and system configuration of an information processing device 10 according to the first embodiment of the present disclosure will be described with reference to FIGS.

[0012] Fig. 1 shows an example of the hardware configuration of an information processing device 10 according to the first embodiment of the present disclosure. Fig. 2 shows an example of the configuration of a virtual viewpoint image generation system including the information processing device 10 according to the first embodiment of the present disclosure.

[0013] As shown in FIG. 1, the hardware of the information processing device 10 includes a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, and input / output interfaces such as a display unit 105, an operation unit 106, a communication I / F 107, and a system bus 108.

[0014] The CPU 101 controls the entire information processing device 10 using computer programs and data stored in the ROM 102 and RAM 103. The information processing device 10 may have one or more dedicated hardware or GPUs different from the CPU 101, and at least part of the processing by the CPU 101 may be performed by the GPUs or dedicated hardware. Examples of dedicated hardware include an ASIC (application-specific integrated circuit) and a DSP (digital signal processor).

[0015] The ROM 102 holds computer programs and data that do not require modification.

[0016] The RAM 103 temporarily stores computer programs and data read from the ROM 102, and data acquired from the outside via the communication I / F 107. The auxiliary storage device 104 is configured, for example, by a hard disk drive or the like, and stores various data such as image data and acoustic data.

[0017] The display unit 105 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI or the like for the user to give instructions to the information processing device 10. The operation unit 106 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives operations by the user to input various instructions to the CPU 101. The CPU 101 operates as a display control unit that controls the display unit 105 and an operation control unit that controls the operation unit 106.

[0018] The communication I / F 107 is used for communication with devices external to the information processing device 10, such as a virtual viewpoint image imaging device group 211, a virtual viewpoint image generation unit 212, and a virtual viewpoint operation unit 201, which will be described later. When the information processing device 10 has a function of performing wireless communication with external devices, the communication I / F 107 includes an antenna.

[0019] The system bus 108 connects the various components of the information processing device 10 to transmit information.

[0020] In this embodiment, at least one of the display unit 105 and the operation unit 106 may exist as an external device to the information processing device 10.

[0021] 2, the virtual viewpoint image generation system including the information processing device 10 includes a virtual viewpoint path editing unit 200, a group of imaging devices for virtual viewpoint images 211, and a virtual viewpoint image generation unit 212. In this embodiment, at least the virtual viewpoint path editing unit 200 is realized as a software configuration on the information processing device 10, and the group of imaging devices for virtual viewpoint images 211 is realized as a software configuration on the information processing device 10 or another information processing device.

[0022] The virtual viewpoint path editing unit 200 includes a virtual viewpoint operation unit 201 , a virtual viewpoint path generation unit 202 , a virtual viewpoint path storage unit 203 , a key frame number reception unit 204 , a key frame setting unit 205 , a key frame saving unit 206 , and a virtual viewpoint path regeneration unit 207 .

[0023] The virtual viewpoint path editing unit 200 controls the virtual viewpoint and determines a series of virtual viewpoint parameters that constitute the virtual viewpoint path. The virtual viewpoint path includes a time code assigned to each frame and parameters indicating the virtual viewpoint parameters associated with the time code, and indicates the trajectory of change of the virtual viewpoint. Time expressed as a time code is a uniaxial parameter. The time code assigned to each frame is determined according to the frame rate and playback speed of the virtual viewpoint image.

[0024] The virtual viewpoint parameters include at least parameters that indicate the position and orientation of the virtual viewpoint. However, the virtual viewpoint parameters are not limited to this, and may include parameters that define other elements, such as the zoom and angle of view of the virtual viewpoint. The position of the virtual viewpoint may indicate three-dimensional coordinates, and may be expressed, for example, by a three-axis Cartesian coordinate system having an X axis, a Y axis, and a Z axis. In this embodiment, the virtual viewpoint parameters that specify the position of the virtual viewpoint are three parameters that represent the X coordinate, the Y coordinate, and the Z coordinate of the Cartesian coordinate system. Note that the origin of the coordinate system may be any position within three-dimensional space. Furthermore, in this embodiment, the virtual viewpoint parameters that specify the orientation of the virtual viewpoint are three parameters that represent the angles of rotation (pan, tilt, and roll) around three mutually orthogonal axes at the virtual viewpoint.

[0025] Therefore, the virtual viewpoint path has at least seven parameters for each frame: X coordinate, Y coordinate, Z coordinate, pan, tilt, roll, and time code. The virtual viewpoint path editing unit 200 can control these seven parameters.

[0026] The virtual viewpoint path editing unit 200 outputs a virtual viewpoint path generated based on a series of virtual viewpoint parameters input by the user to the virtual viewpoint image generation unit 212. The virtual viewpoint image imaging device group 211 performs synchronized imaging using multiple imaging devices installed so as to surround an imaging target area such as a competition field, and outputs the captured images obtained by imaging to the virtual viewpoint image generation unit 212. The virtual viewpoint image generation unit 212 generates a virtual viewpoint image based on the input virtual viewpoint path and captured images.

[0027] The virtual viewpoint operation unit 201 samples the user input at a predetermined sampling rate, converts the sampled input values ​​into virtual viewpoint parameters, and outputs them to the virtual viewpoint path generation unit 202 as a virtual viewpoint parameter group.

[0028] The virtual viewpoint path generation unit 202 generates a virtual viewpoint path by associating each virtual viewpoint parameter included in the virtual viewpoint parameter group input from the virtual viewpoint operation unit 201 with a time code corresponding to a specific frame.

[0029] The virtual viewpoint path storage unit 203 stores the virtual viewpoint path generated by the virtual viewpoint path generation unit 202 .

[0030] The key frame number receiving unit 204 receives input indicating the number of key frames from the user and outputs it to the key frame setting unit 205. Note that the number of selectable key frames is equal to or less than the number of virtual viewpoints per frame included in the base virtual viewpoint path, but as will be described later, the larger the number of key frames, the closer it gets to the base virtual viewpoint path. Therefore, it is desirable that the number of key frames be smaller than the number of virtual viewpoints per frame included in the base virtual viewpoint path.

[0031] The key frame setting unit 205 selects virtual viewpoint parameters for the number of key frames input to the key frame number receiving unit 204 from the virtual viewpoint path stored in the virtual viewpoint path storage unit 203, and sets key frame information based on the selected virtual viewpoint parameters. The key frame setting unit 205 outputs the set key frame information to the key frame saving unit 206. Details of how to set the key frame information will be described later. The key frame information includes the time code of the key frame, the virtual viewpoint parameters associated with that time code, and the playback speed between adjacent key frames.

[0032] The key frame storage unit 206 stores the key frame information output by the key frame setting unit 205. The key frame information stored in the key frame storage unit 206 is extracted by the virtual viewpoint path regeneration unit 207 as appropriate.

[0033] The virtual viewpoint path regeneration unit 207 acquires key frame information from the key frame storage unit 206 and regenerates a virtual viewpoint path by interpolating virtual viewpoint parameters associated with time codes corresponding to frames between the key frames. Here, the virtual viewpoint path is represented by virtual viewpoint parameters associated with time codes corresponding to consecutive frames. When regenerating the virtual viewpoint path, the virtual viewpoint path regeneration unit 207 associates the interpolated virtual viewpoint parameters with the time codes so that the virtual viewpoint parameters corresponding to each frame can be identified.

[0034] The virtual viewpoint image generation unit 212 generates a three-dimensional model from multi-viewpoint images obtained by synchronously capturing images using the group of virtual viewpoint image capturing devices 211. The virtual viewpoint image generation unit 212 then generates a virtual viewpoint image by mapping textures at the virtual viewpoints (position, posture, and angle of view of the virtual viewpoint) in the virtual viewpoint paths generated by the virtual viewpoint path generation unit 202 and the virtual viewpoint path regeneration unit 207.

[0035] The information processing device 10 of the present disclosure can also be used to edit a virtual viewpoint path of a CG image. In this case, a CG renderer is provided instead of the virtual viewpoint image imaging device group 211 and the virtual viewpoint image generation unit 212. In this case, the virtual viewpoint represents a viewpoint placed in a CG space for generating a CG image, and can be moved to any position within the CG space.

[0036] 1-3 Virtual Viewpoint Path A virtual viewpoint path defines the movement of a virtual viewpoint in a video generated by sequentially playing multiple virtual viewpoint images or CG images. This virtual viewpoint path is managed by frames and a timeline. Frames hold the information necessary to generate each image that makes up the video. Specifically, they hold the scene time (time code) and virtual viewpoint parameters. Scene time is expressed as a time code that shares a common reference with the captured images; for example, the start time of the game being captured is expressed as 00:00:00:00 frame. The number of frames included in the timeline is determined by the number of images played per second (frame rate). For example, if the frame rate is 60 frames per second, the timeline will contain a number of frames equivalent to 60 x (number of seconds in the timeline).

[0037] 3 shows an example of a UI for editing a virtual viewpoint path according to this embodiment. A virtual viewpoint image display unit 301 displays a virtual viewpoint image generated by the virtual viewpoint image generation unit 212, that is, a virtual viewpoint image that represents the view from the virtual viewpoint. A GUI display unit 302 displays the virtual viewpoint path, key frame information, etc. A virtual viewpoint path editing controller 303 is a controller for operating multiple operation axes of the virtual viewpoint that the user uses to edit the virtual viewpoint path.

[0038] 1-4 Keyframe information setting method and virtual viewpoint path regeneration method 4 shows an example of a UI 400 for regenerating a virtual viewpoint path in this embodiment. The UI 400 includes a button 401, a box 402, and a button 403. When button 401 is pressed, the virtual viewpoint path editing unit 200 can read out a generated virtual viewpoint path that serves as a base and that is stored in the virtual viewpoint path storage unit 203. The key frame number receiving unit 204 can receive the number of key frames from the user by acquiring the value input in box 402. When button 403 is pressed, the virtual viewpoint path editing unit 200 can regenerate a virtual viewpoint path based on the read out virtual viewpoint path and the number of key frames held by the key frame number receiving unit 204.

[0039] 5 shows a flowchart for explaining the process of regenerating a virtual viewpoint path in this embodiment. Note that in the following explanation of the flowchart, the symbol "S" represents a step.

[0040] In S501, when the user presses the button 401 on the UI 400 shown in FIG. 4, the key frame setting unit 205 reads out the generated virtual viewpoint path that serves as the base and that is stored in the virtual viewpoint path storage unit 203.

[0041] In S502, when the user inputs a value into the box 402 on the UI 400 shown in FIG. 4, the key frame number receiving unit 204 acquires the input value as the number of key frames.

[0042] In S503, the key frame setting unit 205 selects virtual viewpoint parameters for the number of key frames acquired in S502 from the virtual viewpoint parameters for each frame included in the read virtual viewpoint path, and sets them as virtual viewpoint parameters for the key frame. Key frame information including the time code, virtual viewpoint parameters, and playback speed of the key frame is saved in the key frame saving unit 206. Note that the number of frames between key frames can be set arbitrarily, but key frames may also be set so that the number of frames between key frames is the same, for example. Furthermore, the virtual viewpoint of the first key frame is set as the first virtual viewpoint of the read virtual viewpoint path, and the virtual viewpoint of the last key frame is set as the last virtual viewpoint of the read virtual viewpoint path.

[0043] In S504, when button 403 shown in FIG. 4 is pressed, the virtual viewpoint path regeneration unit 207 interpolates virtual viewpoint parameters corresponding to each frame between key frames based on the virtual viewpoint parameters of the key frames stored in the key frame storage unit 206. The interpolation method is spline interpolation, which interpolates the virtual viewpoint parameters between key frames using a spline function so that the virtual viewpoint path between key frames moves along a smooth straight line or curve. Note that the interpolation method is not limited to this, and for example, circular interpolation, linear interpolation, or Bezier interpolation may also be used. Therefore, the virtual viewpoint path regeneration unit 207 can regenerate a virtual viewpoint path in which key frames are smoothly connected.

[0044] The above steps make it possible to regenerate a virtual viewpoint path that is smoother than an existing virtual viewpoint path, but the virtual viewpoint path regeneration unit 207 may further include a step of displaying the regeneration result and allowing the user to select whether or not to redo the regeneration. In this case, if the user selects redo, the process returns to S502, and if the user does not select redo, the process ends. Here, the display of the regeneration result may be by displaying the regenerated virtual viewpoint path, or by displaying a virtual viewpoint image generated using the regenerated virtual viewpoint path.

[0045] 1-5 Effects Fig. 6 shows an example of setting key frame information from specified virtual viewpoint path information. Fig. 6(a) shows virtual viewpoint parameters corresponding to key frames set when the number of key frames is set to 5, and Fig. 6(b) shows virtual viewpoint parameters corresponding to key frames set when the number of key frames is set to 9. Fig. 7 shows schematic diagrams of virtual viewpoint paths before and after regeneration in this embodiment. Fig. 7(a) is a schematic diagram of the virtual viewpoint path when the number of key frames is set to 5, and Fig. 7(b) is a schematic diagram of the virtual viewpoint path when the number of key frames is set to 9.

[0046] 6 includes a set of virtual viewpoint parameters corresponding to 2400 frames from time code 00:00:00:00 to 00:00:40:00. It is assumed that there is a sudden change in the set of virtual viewpoint parameters corresponding to frames with frame numbers 1101 to 1300 among the 2400 frames.

[0047] When the number of key frames is set to 5, as shown in Figures 6(a) and 7(a), five pieces of key frame information are set every 600 frames, and a new virtual viewpoint path is regenerated by interpolating the virtual viewpoint parameters between each key frame based on the set key frame information.

[0048] When the number of key frames is set to 9, as shown in Figures 6(b) and 7(b), 9 pieces of key frame information are set every 300 frames, and a new virtual viewpoint path is regenerated by interpolating the virtual viewpoint parameters between each key frame based on the set key frame information.

[0049] As described above, the virtual viewpoint path editing unit 200 selects virtual viewpoint parameters corresponding to the number of key frames specified by the user and regenerates a virtual viewpoint path by interpolating the virtual viewpoint parameters corresponding to the frames between each key frame. As shown in FIG. 7(a), if a small number of key frames are set, a virtual viewpoint path with gradual changes is generated. As shown in FIG. 7(b), if a large number of key frames are set, a virtual viewpoint path close to the original virtual viewpoint path is generated. This makes it possible to generate a smoothly changing virtual viewpoint path that can suppress abrupt changes in the virtual viewpoint image without affecting the playback speed of the virtual viewpoint image, even if the underlying virtual viewpoint path was manually specified by the user.

[0050] [Embodiment 2] 2-1 Overview Hereinafter, a virtual viewpoint path regeneration process using the information processing device 20 according to the second embodiment of the present disclosure will be described with reference to Figures 8 to 14. Components that perform the same functions as those in the first embodiment will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0051] 2-2 Configuration 8 shows an example configuration of a virtual viewpoint image generation system including an information processing device 20 according to the second embodiment of the present disclosure. Note that the hardware configuration of the information processing device 20 is similar to that of the information processing device 10, and therefore a description thereof will be omitted here.

[0052] As shown in FIG. 8, the virtual viewpoint image generating system including the information processing device 20 includes a virtual viewpoint path editing unit 800, a group of imaging devices for virtual viewpoint images 211, and a virtual viewpoint image generating unit 212.

[0053] In the virtual viewpoint path editing unit 800, the virtual viewpoint operation unit 201, virtual viewpoint path generation unit 202, virtual viewpoint path storage unit 203, key frame setting unit 205, key frame saving unit 206, and virtual viewpoint path regeneration unit 207 have the same configuration as in embodiment 1. The virtual viewpoint path editing unit 800 further includes a setting information receiving unit 801, a virtual viewpoint change amount detection unit 802, and a key frame setting section determination unit 803.

[0054] The virtual viewpoint path editing unit 800 outputs a virtual viewpoint path generated based on a series of virtual viewpoint parameters input by the user to the virtual viewpoint image generation unit 212. The virtual viewpoint image imaging device group 211 performs synchronized imaging using multiple imaging devices installed so as to surround an imaging target area such as a competition field, and outputs the captured images obtained by imaging to the virtual viewpoint image generation unit 212. The virtual viewpoint image generation unit 212 generates a virtual viewpoint image based on the input virtual viewpoint path and captured images.

[0055] The setting information receiving unit 801 receives from the user setting information indicating the number of key frames, the regeneration section of the virtual viewpoint path, and whether a steep section setting avoidance function that avoids setting key frames in sections where the virtual viewpoint changes sharply is enabled or disabled. The setting information receiving unit 801 outputs the received setting information to the key frame setting unit 205, the virtual viewpoint change amount detection unit 802, and the virtual viewpoint path regeneration unit 804. Note that if the user does not set a regeneration section, the regeneration section may be the entire virtual viewpoint path read from the virtual viewpoint path storage unit 203, or may be a section of a predetermined range that includes a steep section.

[0056] When the steep section setting avoidance function is enabled by the setting information receiving unit 801, the virtual viewpoint change amount detection unit 802 calculates the amount of change in the virtual viewpoint parameters of the regeneration section and detects sections where the amount of change is equal to or greater than a predetermined threshold as steep sections. The virtual viewpoint change amount detection unit 802 outputs the detected steep sections to the key frame setting section determination unit 803 as a time code list.

[0057] The key frame setting section determination unit 803 outputs to the key frame setting unit 205 a virtual viewpoint path obtained by excluding the virtual viewpoint parameters corresponding to the time code output from the virtual viewpoint change amount detection unit 802 from the original virtual viewpoint path.

[0058] The key frame setting unit 205 sets key frame information of the number of key frames for the virtual viewpoint path stored in the virtual viewpoint path storage unit 203 or for the regeneration section of the virtual viewpoint path input from the key frame setting section determination unit 803. The number of key frames set here is the number of key frames input from the key frame number receiving unit 204, as in the first embodiment. The key frame setting unit 205 outputs the set key frame information to the key frame saving unit 206.

[0059] The virtual viewpoint path regeneration unit 804 acquires key frame information from the key frame storage unit 206, and regenerates the virtual viewpoint path of the regeneration section by interpolating virtual viewpoint parameters associated with the time codes corresponding to the frames between the key frames. The virtual viewpoint path regeneration unit 804 generates a new virtual viewpoint path by swapping the regeneration section of the original virtual viewpoint path with the regenerated virtual viewpoint path.

[0060] 9 shows an example of a UI 900 for regenerating a virtual viewpoint path in this embodiment. The UI 900 is made up of a button 901, a box 902a, a box 902b, a box 903, a button 904, and a button 905.

[0061] When button 901 is pressed, the virtual viewpoint path editing unit 800 reads out the generated virtual viewpoint path that serves as the base from the virtual viewpoint path storage unit 203. The setting information receiving unit 801 can set the regeneration section by inputting the start frame and end frame of the original virtual viewpoint path in boxes 902a and 902b. Note that any parameter that can specify the section of the virtual viewpoint path to be read out may be used, for example, a time code. The setting information receiving unit 801 can set the number of key frames to be set based on the value input in box 903. When button 904 is pressed, the setting information receiving unit 801 switches between enabling and disabling a function to avoid setting steep sections in the regeneration section. When button 905 is pressed, the virtual viewpoint path editing unit 800 regenerates the virtual viewpoint path based on the value set in the setting information receiving unit 801.

[0062] Hereinafter, the processing procedure for regenerating a virtual viewpoint path in this embodiment will be described with reference to FIGS.

[0063] Fig. 10 shows a flowchart for explaining the process of regenerating a virtual viewpoint path in this embodiment. Fig. 11 shows a flowchart for explaining the process of detecting abrupt changes within a regeneration section in this embodiment. Fig. 12 shows a flowchart for explaining the process of setting key frames excluding abrupt changing sections in this embodiment. Note that in the following explanation of the flowcharts, the symbol "S" represents a step.

[0064] In S1001, when the user presses the button 901, the virtual viewpoint path editing unit 800 specifies a generated virtual viewpoint path that is stored in the virtual viewpoint path storage unit 203 and serves as the basis.

[0065] In S1002, the setting information receiving unit 801 receives the number of key frames to be set, the regeneration section, and whether the steep section setting avoidance function is enabled or disabled.

[0066] In S1003, the setting information receiving unit 801 determines whether or not a regeneration section has been designated. If a regeneration section has been designated, the process proceeds to S1004, and if not, the process proceeds to S1005.

[0067] In S1004, the setting information receiving unit 801 sets the virtual viewpoint path of the section designated for the original virtual viewpoint path as a regeneration section.

[0068] In S1005, the setting information receiving unit 801 sets the entire original virtual viewpoint path as a regeneration section.

[0069] In S1006, the setting information receiving unit 801 determines whether the steep section setting avoidance function is enabled or disabled. If the steep section setting avoidance function is enabled, the process proceeds to S1007, and if not, the process proceeds to S1010.

[0070] In S1007, the virtual viewpoint change amount detection unit 802 detects a sudden change within the reproduction section. Details of the processing in S1007 will be described later with reference to FIG.

[0071] In S1008, the virtual viewpoint change amount detection unit 802 determines whether or not there is a steep change. If a steep section has been acquired in S1007, the process proceeds to S1009, and if not, the process proceeds to S1010.

[0072] In S1009, the key frame setting section determination unit 803 sets virtual viewpoint parameters corresponding to the set number of key frames from a group of virtual viewpoint parameters of a regeneration section obtained by excluding sharply changing sections from the base virtual viewpoint path. Details of the processing of S1009 will be described later with reference to FIG.

[0073] In S1010, the key frame setting unit 205 selects virtual viewpoint parameters corresponding to key frames so as to divide the frames of the regeneration section equally by the number of key frames read out, and sets key frame information based on the selected virtual viewpoint parameters. The key frame setting unit 205 saves the set key frame information in the key frame saving unit 206. Note that the first key frame has the time code and virtual viewpoint parameters of the first frame of the regeneration section, and the last key frame has the time code and virtual viewpoint parameters of the last frame of the regeneration section. Note that the key frame setting method does not necessarily have to be equal division; for example, the number of frames between key frames may be divided to a predetermined number of frames.

[0074] In S1011, when button 905 is pressed, the virtual viewpoint path regeneration unit 207 interpolates virtual viewpoint parameters corresponding to the frames between the key frames, based on the virtual viewpoint parameters of the key frames stored in the key frame storage unit 206. If the regeneration section is set as part of the base virtual viewpoint path, the virtual viewpoint path regeneration unit 207 replaces the virtual viewpoint parameter group of the regeneration section of the base virtual viewpoint path with the virtual viewpoint parameter group of the regenerated virtual viewpoint path. Because the first and last frames of the regeneration section are set as key frames, it is possible to regenerate a virtual viewpoint path with a smooth connection between the frames of the base virtual viewpoint path and the regenerated virtual viewpoint path.

[0075] The process of detecting abrupt changes within the regeneration interval in S1007 will be described in detail below with reference to FIG.

[0076] In S1101, the virtual viewpoint change amount detection unit 802 acquires the number of frames N of the virtual viewpoint path set in the regeneration section.

[0077] In S1102, the virtual viewpoint change amount detection unit 802 performs the processes of S1103 to S1105 on the virtual viewpoint parameters corresponding to the (n-1)th to (n+1)th frames in the virtual viewpoint path set in the reproduction section, where n is an integer between 2 and N-1.

[0078] In S1103, the virtual viewpoint change amount detection unit 802 calculates the time code between the n-1th frame and the nth frame and the amount of movement Dn of the position of the virtual viewpoint corresponding to those frames. Similarly, the time code between the nth frame and the n+1th frame and the amount of movement Dn+1 of the position of the virtual viewpoint corresponding to those frames are calculated. The virtual viewpoint change amount detection unit 802 then calculates a change amount ΔD (=Dn+1-Dn), which is the difference between these amounts of movement per frame. Note that the change amount ΔD here is not limited to the difference in the amount of movement of the position of the virtual viewpoint, but may also be the difference in the amount of change in the attitude of the virtual viewpoint.

[0079] In S1104, the virtual viewpoint change amount detection unit 802 determines whether the difference in movement amount ΔD exceeds a predetermined threshold. If the difference in movement amount ΔD exceeds the predetermined threshold, the process proceeds to S1105, and if ΔD does not exceed the predetermined threshold, the process returns to S1102.

[0080] In S1105, the virtual viewpoint change amount detection unit 802 sets a frame in which the difference in movement amount ΔD exceeds a predetermined threshold as a steep section. The processes of S1103 to S1105 are repeated until n reaches N−1.

[0081] In S1106, the virtual viewpoint change amount detection unit 802 outputs the frame information set in the steep section, and the process returns to S1008.

[0082] Hereinafter, details of the process in S1009 for setting the virtual viewpoint parameters of the regeneration section, avoiding steep sections, as the virtual viewpoint parameters corresponding to the key frame will be described with reference to FIG.

[0083] In S1201, the key frame setting section determination unit 803 acquires information on regeneration sections and steep sections.

[0084] In S1202, the key frame setting section determination unit 803 acquires a virtual viewpoint path in which virtual viewpoint parameters corresponding to frames in steep sections are excluded from the regeneration section.

[0085] In S1203, the key frame setting section determination unit 803 acquires the number of frames of the virtual viewpoint path excluding the virtual viewpoint parameters corresponding to the frames in the steep section, and sets key frame information in the same manner as in S1010.

[0086] In S1204, the key frame setting section determination unit 803 outputs the key frame information, and the process returns to S1011.

[0087] 2-3 Effects The effects of this embodiment will be described using a specific example with reference to FIGS.

[0088] FIG. 13 is a diagram showing an example of setting key frame information from specified virtual viewpoint path information. FIG. 13(a) shows a case where no regeneration section is set and the steep section setting avoidance function is disabled, and FIG. 13(b) shows a case where no regeneration section is set and the steep section setting avoidance function is enabled. FIG. 13(c) shows a case where a regeneration section is set and the steep section setting avoidance function is enabled. The number of key frames is set to 5 in all of FIGS. 13(a) to 13(c). FIG. 14 is a diagram showing the effect of virtual viewpoint path regeneration in this embodiment. The settings in FIGS. 14(a) to 14(c) are the same as those in FIGS. 13(a) to 13(c), respectively.

[0089] It is assumed that the specified virtual viewpoint path as shown in FIG. 13 is a group of virtual viewpoint parameters for 2400 frames from time code 00:00:00:00 to 00:00:40:00.

[0090] As shown in Fig. 13(a), when the number of key frames is set to 5, five pieces of key frame information are set for every 600 frames. As shown in Fig. 14(a), the virtual viewpoint path regeneration unit 804 interpolates virtual viewpoint parameters corresponding to frames between key frames based on the set key frame information, and generates new virtual viewpoint paths for all sections of the original virtual viewpoint path.

[0091] As shown in FIG. 13(b), a case will be described in which the number of key frames is 5 and the steep section setting avoidance function is enabled. If the steep section detected in S1007 is, for example, 120 frames from 1001 to 1120, key frames are set at frames 0, 570, 1260, 1830, and 2400 of the original virtual viewpoint path. In this case, since key frames are set for frames excluding the steep section, the positions of the key frames set in the case of FIG. 13(a) are different. As shown in FIG. 14(b), the virtual viewpoint path regeneration unit 804 interpolates virtual viewpoint parameters corresponding to frames between key frames based on the set key frame information, and generates a new virtual viewpoint path for the entire section of the original virtual viewpoint path.

[0092] 13(c), a case will be described in which a regeneration section is specified, the number of key frames is set to 5, and the steep section setting avoidance function is enabled. The virtual viewpoint change amount detection unit 802 extracts a group of virtual viewpoint parameters for the regeneration section set in the base virtual viewpoint path. If the steep section is from frames 1001 to 1120 and the regeneration section is from frames 400 to 2000, key frames are set at frames 400, 770, 1260, 1630, and 2000 of the base virtual viewpoint path.

[0093] 14(c), the virtual viewpoint path regeneration unit 804 generates a new virtual viewpoint path by interpolating virtual viewpoint parameters corresponding to frames between key frames based on the set key frame information. The virtual viewpoint path regeneration unit 804 connects the generated virtual viewpoint path of the regeneration section with a virtual viewpoint path outside the regeneration section of the base virtual viewpoint path to generate a new virtual viewpoint path.

[0094] As described above, the virtual viewpoint path editing unit 800 sets any number of key frames for a partial section of the base virtual viewpoint path, and interpolates the virtual viewpoint parameters corresponding to the frames between the key frames to regenerate the virtual viewpoint path. Because interpolation processing can be performed only on unintentionally specified sections in this way, it is possible to generate a virtual viewpoint path that suppresses abrupt changes in the virtual viewpoint image without affecting the playback speed setting of the virtual viewpoint image, without changing the section that was specified as intended.

[0095] (Other Examples) The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] The present disclosure includes the following configurations and methods. [Configuration 1] an acquisition means for acquiring a first virtual viewpoint path that indicates a trajectory of a change in the virtual viewpoint used to generate the virtual viewpoint image; a selection means for selecting a plurality of virtual viewpoints included in the first virtual viewpoint path; a generating means for generating a second virtual viewpoint path by interpolating between the plurality of virtual viewpoints selected by the selecting means; An information processing device comprising: [Configuration 2] The change in the virtual viewpoint is a change in at least one of the position and the orientation of the virtual viewpoint. 2. The information processing device according to configuration 1, [Configuration 3] the number of the virtual viewpoints selected by the selection means is less than the number of virtual viewpoints included in the first virtual viewpoint path; 3. The information processing device according to configuration 1 or 2. [Configuration 4] the selection means selects a range of the first virtual viewpoint path for selecting the plurality of virtual viewpoints; 4. The information processing device according to any one of configurations 1 to 3. [Configuration 5] the generating means replaces the range of the first virtual viewpoint path selected by the selecting means with the generated second virtual viewpoint path; 5. The information processing device according to configuration 4. [Configuration 6] a receiving means for receiving a user input specifying the number of the plurality of virtual viewpoints to be selected by the selecting means; 6. The information processing device according to any one of configurations 1 to 5, further comprising: [Configuration 7] the receiving means includes operation means for operating a plurality of operation axes of a virtual viewpoint arranged in a three-dimensional space, 7. The information processing device according to configuration 6. [Configuration 8] a detection means for detecting a steep section of the change in the first virtual viewpoint path where the change in the first virtual viewpoint path exceeds a predetermined magnitude; Furthermore, the selection means selects the plurality of virtual viewpoints in a section of the first virtual viewpoint path that is different from the section of the steep change detected by the detection means. 8. The information processing device according to any one of configurations 1 to 7. [Configuration 9] the generating means performs interpolation between the plurality of virtual viewpoints using any one of spline interpolation, circular interpolation, linear interpolation, and Bezier interpolation; 9. The information processing device according to any one of configurations 1 to 8. [Configuration 10] acquiring a first virtual viewpoint path indicating a trajectory of a change in the virtual viewpoint in the virtual viewpoint image; selecting a plurality of virtual viewpoints included in the first virtual viewpoint path; generating a second virtual viewpoint path by interpolating between the plurality of virtual viewpoints selected by the selecting step; An information processing method comprising: [Configuration 11] A program for causing a computer to function as the information processing device according to any one of configurations 1 to 8.

Claims

1. an acquisition means for acquiring a first virtual viewpoint path that indicates a trajectory of a change in the virtual viewpoint used to generate the virtual viewpoint image; a selection means for selecting a plurality of virtual viewpoints included in the first virtual viewpoint path; a generating means for generating a second virtual viewpoint path by interpolating between the plurality of virtual viewpoints selected by the selecting means; An information processing device comprising:

2. The change in the virtual viewpoint is a change in at least one of the position and the orientation of the virtual viewpoint.

2. The information processing apparatus according to claim 1, wherein:

3. the number of the plurality of virtual viewpoints selected by the selection means is less than the number of virtual viewpoints included in the first virtual viewpoint path; 2. The information processing apparatus according to claim 1, wherein:

4. the selection means selects a range of the first virtual viewpoint path for selecting the plurality of virtual viewpoints; 2. The information processing apparatus according to claim 1, wherein:

5. the generating means replaces the range of the first virtual viewpoint path selected by the selecting means with the generated second virtual viewpoint path; 5. The information processing apparatus according to claim 4,

6. a receiving means for receiving a user input specifying the number of the plurality of virtual viewpoints to be selected by the selecting means; 2. The information processing apparatus according to claim 1, further comprising:

7. the receiving means includes operation means for operating a plurality of operation axes of a virtual viewpoint arranged in a three-dimensional space, 7. The information processing apparatus according to claim 6,

8. a detection means for detecting a steep section of the change in the first virtual viewpoint path where the change in the first virtual viewpoint path exceeds a predetermined magnitude; Furthermore, the selection means selects the plurality of virtual viewpoints in a section of the first virtual viewpoint path that is different from the section of the steep change detected by the detection means.

2. The information processing apparatus according to claim 1, wherein:

9. the generating means performs interpolation between the plurality of virtual viewpoints using any one of spline interpolation, circular interpolation, linear interpolation, and Bezier interpolation; 2. The information processing apparatus according to claim 1, wherein:

10. acquiring a first virtual viewpoint path indicating a trajectory of a change in the virtual viewpoint in the virtual viewpoint image; selecting a plurality of virtual viewpoints included in the first virtual viewpoint path; generating a second virtual viewpoint path by interpolating between the plurality of virtual viewpoints selected by the selecting step; An information processing method comprising:

11. A program for causing a computer to function as the information processing device according to any one of claims 1 to 9.

Images