Information processing device, information processing system, information processing program, and information processing method

Abstract

To provide an information processing device that can display to the user the change in the effective field of view for each frame in a video. [Solution] The information processing device includes a display control means that displays a timeline area corresponding to the playback time of a plurality of frame images constituting video content, an image processing means that performs blur correction processing on a first frame image based on the amount of blur of the first frame image included in the plurality of frame images, and an acquisition means that acquires a first field of view that is displayed when the first frame image to which the blur correction processing has been applied is played back. The display control means controls the timeline area to distinguish and display sections in which the field of view satisfies predetermined conditions and sections in which the field of view does not satisfy predetermined conditions, based on the field of view that is displayed when each of the plurality of frame images to which the blur correction processing has been applied is played back.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus.

Background Art

[0002] As a video for VR, a VR180 video that displays two images with a viewing angle of 180 degrees side by side is used. When viewing a VR180 video with a head-mounted display, if the video is blurred, VR sickness will occur, so it is necessary to apply blur correction during video editing. In a general video, when an invalid area, which is an area where an image is not reflected due to blur correction, occurs, a process such as cutting out a part of the image is performed so that the invalid area does not enter the video. Patent Document 1 discloses a method of displaying a blur correction frame indicating an area of an image cut out to correct blur according to the maximum amount of blur in the video.

[0003] As described above, in a VR180 video, since it is necessary to use a viewing angle of 180 degrees, there is a limit to the area of the image that can be cut out, and in the 180-degree viewing angle, an invalid area, which is an area where an actual image is not reflected, may occur. In addition, since the amount of blur correction varies depending on the frame, the size of the invalid area also varies depending on the frame, and the effective area (or effective viewing angle), which is the area where an actual image is reflected within the 180-degree viewing angle video area, also varies. When viewing such a video with a head-mounted display, the effective viewing angle continues to change, and there may be a frame in which the invalid area is reflected during viewing, which impairs the viewing experience.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] To avoid compromising the viewing experience described above, it is necessary to prevent invalid areas from appearing in the image when viewing with a head-mounted display. However, it is time-consuming for video editors to manually search for and cut out scenes with a narrow effective field of view.

[0006] The present invention has been made in view of the above problems, and aims to provide an information processing device that can display to the user the change in the effective field of view for each frame in a video. [Means for solving the problem]

[0007] To achieve the above objective, the information processing apparatus of the present invention includes: display control means for displaying a timeline area on a screen corresponding to the playback time of a plurality of frame images constituting video content; image processing means for performing blur correction processing on a first frame image included in the plurality of frame images based on the amount of image blur due to the movement of an imaging device that captures the video content; and acquisition means for acquiring a first field of view displayed when the first frame image to which the blur correction processing has been applied is played back. The display control means is characterized in that, based on the field of view displayed when each of the plurality of frame images to which the blur correction processing has been applied is displayed, a first section which is a section of frame images where the field of view satisfies a predetermined condition and a second section which is a section of frame images where the field of view does not satisfy the predetermined condition are distinguished in the timeline area and displayed on the screen. [Effects of the Invention]

[0008] According to the present invention, the change in the effective field of view for each frame in a video can be displayed to the user. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the internal configuration of an information processing device according to the first embodiment of the present invention. [Figure 2]This figure shows an example of converting a two-lens fisheye video to an equirectangular projection video for VR180 while applying the image stabilization method used in this embodiment. [Figure 3] This is a flowchart illustrating an information processing device according to the first embodiment of the present invention, which performs predetermined image processing while applying image stabilization to a two-camera video and records the effective field of view after image processing. [Figure 4] This is a flowchart diagram of the process of an information processing device according to the first embodiment of the present invention, in which a user specifies conditions related to the field of view, and scenes that do not meet the conditions are clearly displayed on the editing screen. [Figure 5] This is an example of a user interface screen for editing content videos according to the first embodiment of the present invention. [Figure 6] This is a flowchart illustrating the process of exporting a video based on the specified conditions for an information processing device according to the first embodiment of the present invention, in which a user specifies conditions related to the field of view. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted as appropriate. Furthermore, some components, members, and processes will be omitted in each drawing. The following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (operating conditions, operating environment, etc.). In the following embodiments, the same components will be denoted by the same reference numerals.

[0011] (First embodiment) <Internal configuration of information processing device> Figure 1 shows a block diagram illustrating the internal configuration of PC100, an information processing device to which the present invention can be applied. In PC100, the control unit 101, ROM 102, RAM 103, external storage device 104, operation unit 105, display unit 106, and communication unit 107 are connected to the system bus 108. Note that each component of PC100 may be an information processing system composed of individual hardware.

[0012] The control unit 101 is a CPU (Central Processing Unit) that controls the entire PC 100. It executes the programs recorded in the ROM 102, which will be described later, to realize each of the processes of this embodiment, which will be described later. Furthermore, the control unit 101 controls the display by controlling the RAM 103, the external storage device 104, and the display unit 106. Alternatively, instead of the control unit 101 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0013] The ROM (Read Only Memory) 102 is an electrically erasable and recordable non-volatile memory, such as an EEPROM. The ROM 102 stores constants for the operation of the control unit 101, programs, etc. The program referred to here is a program for executing the various flowcharts described later in this embodiment.

[0014] The RAM (Random Access Memory) 103 is a volatile memory that stores constants, variables, and programs read from the ROM 102 for the operation of the control unit 101. The RAM 103 consists of, for example, a volatile memory (DRAM) that utilizes semiconductor elements.

[0015] The external storage device 104 is a hard disk or flash memory fixed to the PC 100, or a memory card that can be attached to or removed from the PC 100. In this embodiment, images and video files captured by an imaging device such as a camera are written to the external storage device 104.

[0016] The operation unit 105 is an input device for receiving user operations, including a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, buttons, dials, joysticks, touch sensors, and touch pads. The operation unit 105 transmits the user's operations to the control unit 101.

[0017] The display unit 106 is a display for displaying images, GUI (Graphical User Interface) screens that make up the GUI, etc. based on the control of the control unit 101. The control unit 101 generates a display control signal according to a program, generates a video signal for display to the display unit 106, and controls each part of the PC 100 to output it to the display unit 106. The display unit 106 displays a video based on the output video signal. Note that the configuration of the PC 100 itself may include up to an interface for outputting a video signal for display on the display unit 106, and the display unit 106 may be configured with an external monitor (such as a TV).

[0018] The communication unit 107 is an interface for communicating directly with external devices or via the Internet to transmit and receive various data such as files and commands. Note that wireless communication may be performed using a wireless communication method such as Wi-Fi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark), or communication with external devices may be performed by wire.

[0019] <Process of converting a binocular fisheye video into a video in the orthographic cylindrical projection format for VR180> Referring to FIGS. 2(a) and 2(b), an example of converting a video of binocular fisheye images into a video in the orthographic cylindrical projection format for VR180 while applying the blur correction process according to the first embodiment will be described.

[0020] In FIG. 2(a), the image 200 is a binocular fisheye image. The image 200 includes a circular image area 201 and an image area 202. Also, the image 200 represents the M-th frame image of the video content. Note that the circular image area 201 and the image area 202 are areas where subject images are formed via an optical system, and the image 200 may be an image generated by forming two subject images on one imaging device. Note that the image 200 may be synthesized from images generated by forming subject images on one imaging device via one optical system. Also, the invalid area 203 is an area in the image 200 that is different from the circular image area 201 and the image area 202. This area is an area where no subject image is formed via the optical system and is composed of black pixels.

[0021] Here, assume that the image 200 is an image in which image blur has occurred due to the movement of the imaging device when imaging the video content. When such image blur has occurred, correction is performed based on the amount of image blur due to the movement of the imaging device in each frame of the video content. Also, assume that in the M-th frame, the scene has more image blur than in the N-th frame. Furthermore, since the image 200 is a fisheye image with distortion, it is converted into the orthographic cylindrical projection format.

[0022] The image 205 is an image obtained by performing the above-described image blur correction (blur correction processing) and conversion into the orthographic cylindrical projection format on the image 200. The image area 206 corresponds to the circular image area 201 in the image 200, and the image area 207 corresponds to the circular image area 202 in the image 200. Also, the area of the image 205 that is different from the image area 206 and the image area 207 is an invalid area and is composed of black pixels.

[0023] In Figure 2(b), image 210 is a binocular fisheye image. Image 210 includes circular image regions 211 and 212. Image 210 also represents the Nth frame of a video content. The circular image regions 211 and 212 are regions where subject images are formed via the optical system, and image 210 may be an image generated by forming two subject images on a single image sensor. Alternatively, image 210 may be a composite of images generated by forming subject images on a single image sensor via a single optical system. The invalid region 213 is a region in image 210 that is different from the circular image regions 211 and 212. This region is an area where no subject image is formed via the optical system and is composed of black pixels.

[0024] Here, we assume that image 210 is an image in which image blur occurs due to the movement of the imaging device when capturing video content. Furthermore, we assume that the Nth frame is a scene with less image blur than the Mth frame. In addition, since image 210 is a distorted fisheye image, it is converted to equirectangular projection.

[0025] Image 215 is an image obtained by applying the image blur correction (blur correction processing) described above to Image 210 and converting it to equirectangular projection format. Image region 216 corresponds to the circular image region 211 in Image 210, and image region 217 corresponds to the circular image region 212 in Image 210. Furthermore, the areas in Image 215 that are different from image regions 216 and 217 are invalid regions and are composed of black pixels.

[0026] Here, since image 200 has greater blur than image 210, the amount of blur correction required is also greater. Furthermore, the greater the blur, the narrower the image area (effective field of view) becomes after applying blur correction. In other words, the ineffective area (area of ​​black pixels) becomes larger. Therefore, the image area of ​​image 205 with blur correction applied has a narrower field of view than the image area of ​​image 215 with blur correction applied.

[0027] As described above, applying image stabilization to VR180 video results in varying amounts of blur depending on the scene, and therefore the effective field of view also varies from scene to scene. Video editors need to mask or cut scenes according to these changes in the effective field of view, so it is necessary to clearly display the changes in the effective field of view for each frame in the video to the user.

[0028] <A workflow for applying image stabilization to dual-camera video while performing predetermined image processing, and recording the effective field of view after image processing.> Referring to Figure 3, a flowchart will be described for performing predetermined image processing while applying blur correction to a video of a two-lens circular fisheye image used in this embodiment, and recording the effective field of view after image processing. The control unit 101 executes this process when it receives an operation from the user to read video content data from an external storage device or to receive video content data via the communication unit.

[0029] In step S300, the control unit 101 determines whether or not a video file has been loaded. If the control unit 101 determines that a video file has been loaded, it proceeds to step S305; if it determines that a video file has not been loaded, it proceeds to step S300.

[0030] In step S305, the control unit 101 determines whether the video file read in step S300 is a video recorded with two cameras. If the control unit 101 determines that it is a video recorded with two cameras, it proceeds to step S310; otherwise, it terminates this flowchart. Alternatively, whether a video file is a video recorded with two cameras can be determined by checking whether the metadata attached to the video contains information indicating that it is a video recorded with two cameras. Alternatively, whether a video file is a binocular image can be determined by analyzing a predetermined frame image of the video file. For example, if a predetermined frame image of a video file is a binocular image with a predetermined parallax between the left and right sides, when the image is divided into left and right image regions, there is a high probability that the two image regions are capturing the same subject, and it can be considered that there is a correlation between the two image regions. Thus, whether a video file is a video recorded with two cameras can be determined by analyzing whether a correlation can be observed from the pixels of the two image regions.

[0031] Alternatively, whether or not a video file is a binarized image can be determined by analyzing a predetermined frame image of the video file. A threshold value close to the brightness value of a black pixel is applied to the brightness value of each pixel in the image data to generate binarized image data. Then, if the control unit 101 determines that the image represented by the binarized image data contains two white circular regions, it determines that the type of video file is a video file containing two circular fisheye images. By using this method, it is possible to determine whether or not a video file is recorded with two cameras, even if metadata is not attached to the video file.

[0032] In step S310, the control unit 101 sets the frame count N of the PC 100 to 1 and proceeds to step S315.

[0033] In step S315, the control unit 101 obtains the Nth frame image from the video file read by the PC 100 and proceeds to step S320.

[0034] In step S320, the control unit 101 determines the amount of image stabilization for the Nth frame image read by the PC 100 in step S315, and proceeds to step S325. The amount of image stabilization is determined using the camera's gyro information (inertial information) recorded in the video file at the time of shooting, lens metadata, and changes in image features.

[0035] In step S325, the control unit 101 performs predetermined image processing on the Nth frame image using the blur correction amount obtained by the PC 100 in step S320, and proceeds to step S330. As predetermined image processing, the control unit 101 performs geometric transformation processing, for example, to convert the video content into an image with reduced distortion. For example, as shown in Figure 2, one method is to convert a two-lens fisheye video into an equirectangular projection format while applying blur correction (equirectangular transformation processing). Another method is to convert it into a perspective projection format instead of an equirectangular projection format (perspective projection transformation processing).

[0036] In step S330, the control unit 101 determines and records the effective field of view for the image after the PC 100 has performed predetermined image processing in step S325, and then proceeds to step S335.

[0037] In step S335, the control unit 101 determines whether the Nth frame being processed is the final frame. If the control unit 101 determines that it is the final frame of the video file, it terminates this flowchart. If the control unit 101 determines that it is not the final frame of the video file, it proceeds to step S340.

[0038] In step S340, the control unit 101 adds 1 to the frame count N and proceeds to step S315.

[0039] <A process where the user specifies conditions regarding the field of view, and scenes that do not meet those conditions are displayed on the editing screen.> Referring to Figures 4 and 5, the process of displaying scenes on the editing screen that do not meet the conditions specified by the user regarding the field of view will be explained. Figure 4 is a flowchart showing the process of displaying scenes on the editing screen that do not meet the conditions specified by the user regarding the field of view. The control unit 101 executes this process when it receives an operation from the user to read video content data for which an effective field of view has been set from an external storage device, and when the user has set conditions regarding the effective field of view of the video content. Alternatively, the control unit 101 executes this process when it receives an operation from the user to receive video content data for which an effective field of view has been set via the communication unit, and when the user has set conditions regarding the effective field of view of the video content.

[0040] In step S400, the control unit 101 reads the conditions regarding the field of view specified by the user and proceeds to step S405.

[0041] In step S405, the control unit 101 sets the frame count N to 1 and proceeds to step S410.

[0042] In step S410, the control unit 101 obtains the effective field of view of the Nth frame image recorded in step S330, and proceeds to step S415.

[0043] In step S415, the control unit 101 determines whether the effective field of view of the Nth frame acquired in step S410 satisfies the field of view conditions specified in step S400. If the control unit 101 determines that the effective field of view of the Nth frame acquired in step S410 satisfies the field of view conditions specified in step S400, it proceeds to step S420. If the control unit 101 does not determine that the effective field of view of the Nth frame acquired in step S410 satisfies the field of view conditions specified in step S400, it proceeds to step S430.

[0044] In step S420, the control unit 101 places a marker at the Nth frame on the seek bar of the editing screen and proceeds to step S425. Here, the conditions regarding the field of view include, for example, "whether the effective field of view does not satisfy either the horizontal field of view or the vertical field of view." In other words, if the field of view becomes narrower than the set effective field of view, a marker will be placed in step S420.

[0045] In step S425, the control unit 101 determines whether the Nth frame being processed is the final frame. If the control unit 101 determines that the Nth frame being processed is the final frame, it terminates this flowchart. If the control unit 101 does not determine that the Nth frame being processed is the final frame, it proceeds to step S430.

[0046] In step S430, the control unit 101 adds 1 to the frame count N and proceeds to step S410.

[0047] As described above, by following the flow in Figure 4, it is possible to visually identify the playback section of the video that meets the conditions set by the user, reducing the effort required for the user to search for sections of the video that do not meet the effective field of view when editing the effective field of view.

[0048] Furthermore, Figure 5 shows an example of a user interface screen implemented by the flowchart in Figure 4.

[0049] The display unit is designed to show an image 500, allowing the user to set the effective field of view and edit the video content while viewing this image. Image 500 includes a video display area 505, a timeline area 510, a playback position 515, markers 520 indicating sections that meet user-defined conditions, a play button 525, and an effective field of view setting area 530.

[0050] Timeline area 510 corresponds to the playback time of the loaded video content and represents the temporal changes in the VR content (changes according to the passage of playback time) over at least a portion of the playback period.

[0051] The playback position 515 indicates the position on the timeline area 510 that corresponds to the playback time of the currently playing video content, and the frame image of the video content corresponding to that position is displayed in the video display area 505.

[0052] Marker 520, which indicates the section that satisfies the conditions set by the user, indicates the section that satisfies the conditions set by the user, and the conditions are set in the effective field of view setting area 530. Here, as conditions related to the field of view, horizontal 140 degrees and vertical 130 degrees are set as thresholds.

[0053] Furthermore, the play / rewind button 525 includes buttons for playing and stopping video content, buttons for fast-forwarding and rewinding, and a skip button. Other user interfaces may also be provided to allow users to view frame images of the video content.

[0054] By performing this process, it becomes possible to distinguish and display to the user sections that satisfy the user-specified field of view conditions and sections that do not, such as the marker 520 that indicates sections that satisfy the user-specified conditions for the field of view.

[0055] To rephrase the above example, it is possible to display sections that satisfy the conditions set by the user, i.e., sections where the effective field of view becomes narrower than the field of view set by the user due to image stabilization, and where invalid areas are captured when viewed with a head-mounted display or the like. Furthermore, for sections where the effective field of view is narrower than the field of view set by the user, it is possible to trim and delete them from the video, or widen the effective field of view to prevent invalid areas from being captured.

[0056] Furthermore, if the conditions related to the field of view are changed, that is, if the numerical value of the effective field of view setting area 530 is changed, the control unit 101 will re-execute the process described in Figure 4 based on the changed conditions related to the field of view.

[0057] (Second Embodiment) <A workflow for exporting a video based on conditions specified by the user regarding the field of view.> In the first embodiment described above, an example was explained in which the user is shown sections that satisfy the user-specified field of view conditions and sections that do not satisfy the user-specified field of view conditions. In the second embodiment, an example is further explained in which sections that satisfy the user-specified field of view conditions and sections that do not satisfy the user-specified field of view conditions are saved separately.

[0058] Referring to Figure 6, the process of exporting a video based on the conditions specified by the user regarding the field of view will be explained. The control unit 101 executes this process when it receives an operation from the user to read video content data with a set effective field of view from an external storage device, and when the user has set conditions regarding the effective field of view of the video content. Alternatively, the control unit 101 executes this process when it receives an operation from the user to receive video content data with a set effective field of view via the communication unit, and when the user has set conditions regarding the effective field of view of the video content.

[0059] In step S600, the control unit 101 determines whether or not a video file has been loaded. If the control unit 101 determines that a video file has been loaded, it proceeds to step S601; otherwise, it proceeds to step S600.

[0060] In step S601, the control unit 101 reads the conditions regarding the field of view specified by the user in the PC 100.

[0061] In step S605, the control unit 101 sets the frame count N to 1 and proceeds to step S610.

[0062] In step S610, the control unit 101 obtains the image of the Nth frame from the read video file and the effective field of view of the Nth frame recorded in step S330, and proceeds to step S615.

[0063] In step S615, the control unit 101 determines whether the effective field of view of the Nth frame acquired in step S610 satisfies the field of view conditions specified in step S601. If the control unit 101 determines that the effective field of view of the Nth frame acquired in step S610 satisfies the field of view conditions specified in step S601, it proceeds to step S630. If the control unit 101 determines that the effective field of view of the Nth frame acquired in step S610 does not satisfy the field of view conditions specified in step S601, it proceeds to step S620. For example, consider a case where 140 degrees horizontally and 130 degrees vertically are set as thresholds, and the field of view condition is that the effective field of view is narrower than the thresholds. In this case, frame images with a field of view of 140 degrees horizontally or 130 degrees vertically or more are determined to be frame images that do not satisfy the field of view conditions. Frame images with a field of view of less than 140 degrees horizontally or less than 130 degrees vertically are determined to be frame images that satisfy the field of view conditions.

[0064] In step S620, the control unit 101 performs a masking process on the Nth frame image read by the PC 100, corresponding to the specified field of view, and proceeds to step S625. For example, if 140 degrees horizontally and 130 degrees vertically are set as thresholds for the field of view, the frame images that satisfy the predetermined conditions are those with little camera shake and a wide effective field of view. In such cases, the control unit performs a process to black out the video portion with a field of view of 140 degrees horizontally or 130 degrees vertically or more, so that the effective field of view of the exported video file is unified for each frame image.

[0065] In step S625, the control unit 101 exports the Nth frame image from PC 100 to video file 1 and proceeds to step S635. That is, video file 1 with an effective field of view set according to predetermined conditions for field of view is generated from a group of frame images that do not satisfy predetermined conditions for field of view.

[0066] In step S630, the control unit 101 exports the Nth frame image to the video file 2 and proceeds to step S635. That is, a video file 2 that satisfies predetermined conditions regarding the field of view is generated.

[0067] In this way, the processing in steps S625 and S630 exports scenes that meet the user-specified field of view conditions and scenes that do not meet those conditions to separate video files. The export destination may be RAM 103, external storage device 104, or other devices connected via the communication unit.

[0068] In step S635, the control unit 101 determines whether the Nth frame being processed is the final frame. If the control unit 101 determines that the Nth frame being processed is the final frame, it terminates this flowchart. If the control unit 101 does not determine that the Nth frame being processed is the final frame, it proceeds to step S640.

[0069] In step S640, the control unit 101 adds 1 to the frame count N and proceeds to step S610.

[0070] In this way, by following the process described above, it is possible to generate a video content that extracts only the time segments from the video content that have minimal camera shake and require minimal image stabilization, and that have the effective field of view set by the user. Furthermore, although video file 1 had segments with a field of view greater than the effective field of view set by the user, this process makes it possible to generate video file 1 that is unified with the effective field of view set by the user.

[0071] Alternatively, you may choose to export either video file 1 or video file 2, and leave the other file unexported.

[0072] Although the description above explains how video file 1 is generated as a video file with the effective field of view set by the user, it is also possible to generate a video file consisting of frame images with a field of view greater than or equal to the effective field of view set by the user, without performing step S620 described above.

[0073] (modified version) Furthermore, the user interface example shown in Figure 5 may also include a button in the timeline area to toggle the display of either sections that meet user-defined conditions or sections that do not. Additionally, for example, if the user instructs to hide sections that meet user-defined conditions in the timeline area, video content consisting of the hidden sections may be generated and made viewable. This allows for the generation of video content composed of frame images from sections that do not meet user-defined conditions, without the user having to review each individual frame image.

[0074] Furthermore, the user interface example shown in Figure 5 may also include a button to export the video file of the section currently displayed in the timeline area. For example, if the section that meets the conditions set by the user is hidden as described above, and the user requests to export a video file, the system may export a video file with the hidden section removed.

[0075] (Other embodiments) Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or control unit or MPU, etc.) of the system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0076] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0077] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0078] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0079] Furthermore, in each of the examples described above, "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and specialized processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0080] This embodiment includes the following configurations, methods, and programs.

[0081] [Configuration 1] A display control means that displays on the screen the playback time of multiple frame images that make up the video content and the corresponding timeline area, Image processing means that performs blur correction processing on the first frame image based on the amount of image blur of the first frame image included in the plurality of frame images due to the movement of the imaging device that captures the video content, The system includes an acquisition means for acquiring a first field of view that is displayed when the first frame image to which the image stabilization processing has been applied is played back. The display control means controls the timeline area to distinguish between a first section, which is a section of frame images whose field of view satisfies a predetermined condition, and a second section, which is a section of frame images whose field of view does not satisfy the predetermined condition, based on the field of view displayed when the multiple frame images to which the blur correction processing has been applied are played back, and to display them on the screen. An information processing device characterized by the following:

[0082] [Configuration 2] The system further includes a determination means for determining that if the field of view to which the blur correction processing is applied is narrower than a predetermined field of view, it is a section of the frame image that satisfies the predetermined conditions. The information processing device according to configuration 1, characterized by the above.

[0083] [Configuration 3] The first section is a section of frame images in which the field of view displayed during playback, as determined by the image stabilization process, is smaller than a predetermined field of view. An information processing apparatus according to configuration 1 or 2, characterized by the above.

[0084] [Structure 4] The system further includes a second means for obtaining an instruction to set the predetermined field of view, When the predetermined viewing angle is changed, the display control means distinguishes between the first section and the second section and displays them based on the changed predetermined viewing angle. The information processing apparatus according to configuration 3, characterized by the features described herein.

[0085] [Composition 5] The display control means displays the currently playing frame image on the screen and displays the position indicating the playback time corresponding to the frame image in the timeline area. An information processing apparatus according to any one of configurations 1 to 4, characterized by the above.

[0086] [Composition 6] The system further includes a determination means for determining the amount of image blur based on the inertial information of the imaging device when the video content was captured. An information processing device according to any one of configurations 1 to 5, characterized by the above.

[0087] [Composition 7] The system further includes a determination means for determining the amount of image blur based on metadata attached to the video content. An information processing device according to any one of configurations 1 to 6.

[0088] [Structure 8] The system further includes a determination means for determining the amount of image blur based on the image features contained in the video content. An information processing device according to any one of configurations 1 to 7, characterized by the above.

[0089] [Composition 9] The system further comprises a generation means for generating a second video content consisting of the second section, from which the first section has been cut out. An information processing device according to any one of configurations 1 to 8.

[0090] [Configuration 10] The generation means generates the second video content and a third video content consisting of the first section from which the second section has been cut out. The information processing apparatus according to configuration 9, characterized by the features described therein.

[0091] [Composition 11] The system further includes a generation means for generating a second video content by cropping a section of each of the multiple frame images to which the image stabilization process has been applied, based on the field of view displayed when the image is played back, where the field of view is smaller than a predetermined field of view, and using the frame images of the section where the field of view is equal to or greater than the predetermined field of view as frame images with a predetermined field of view. An information processing apparatus according to any one of configurations 1 to 10, characterized by the above.

[0092] [Composition 12] The timeline area further includes a second acquisition means for acquiring an instruction to switch the display and hiding of the first section or the second section. The display control means controls the display of the timeline based on the instruction. An information processing device according to any one of configurations 1 to 11, characterized by the features described herein.

[0093] [Composition 13] The second acquisition means acquires an instruction to generate video content while the first or second section of the timeline area is hidden, In response to the second acquisition means receiving an instruction to generate the video content, it generates a second video content in which the hidden section has been cut out. The information processing device according to configuration 12, characterized by the features described herein.

[0094] [Composition 14] The aforementioned video content is a video of images with distortion, The image processing means further performs a geometric transformation process to convert the video content into an image with reduced distortion. An information processing device according to any one of configurations 1 to 13, characterized by the above.

[0095] [Composition 15] The aforementioned video content is a circular fisheye image, and the aforementioned geometric transformation process is a process that performs perspective projection transformation or an equirectangular transformation. The information processing apparatus according to configuration 14, characterized by the features described herein.

[0096] [method] A display control step that displays the playback time of multiple frame images constituting the video content and the corresponding timeline area on the screen, An image processing step in which blur correction processing is performed on the first frame image, which is included in the plurality of frame images, based on the amount of image blur caused by the movement of the imaging device that captures the video content, The process includes an acquisition step of acquiring a first field of view that is displayed when the first frame image to which the image stabilization process has been applied is played back, In the display control step, based on the field of view displayed when the multiple frame images to which the blur correction processing has been applied are played back, the system controls the timeline area to distinguish between a first section, which is a section of frame images whose field of view satisfies a predetermined condition, and a second section, which is a section of frame images whose field of view does not satisfy the predetermined condition, and to display them on the screen. An information processing method characterized by the following:

[0097] [program] A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 15.

[0098] [system] An imaging device that generates video content, A display control device that displays a timeline area on the screen corresponding to the playback time of multiple frame images constituting the video content, An image processing device that performs blur correction processing on the first frame image based on the amount of image blur of the first frame image included in the plurality of frame images due to the movement of the imaging device, The device includes an acquisition device that acquires a first field of view that is displayed when the first frame image to which the blur correction processing has been applied is played back. The display control device controls the timeline area to distinguish between a first section, which is a section of frame images whose field of view satisfies a predetermined condition, and a second section, which is a section of frame images whose field of view does not satisfy the predetermined condition, based on the field of view displayed when each of the plurality of frame images to which the blur correction processing has been applied is played back, and to display them on the screen. An information processing system characterized by the following:

Claims

1. A display control means that displays on the screen the playback time of multiple frame images that make up the video content and the corresponding timeline area, Image processing means that performs blur correction processing on the first frame image based on the amount of image blur of the first frame image included in the plurality of frame images due to the movement of the imaging device that captures the video content, The system includes an acquisition means for acquiring a first field of view that is displayed when the first frame image to which the image stabilization processing has been applied is played back. The display control means controls the timeline area to distinguish between a first section, which is a section of frame images whose field of view satisfies a predetermined condition, and a second section, which is a section of frame images whose field of view does not satisfy the predetermined condition, based on the field of view displayed when the multiple frame images to which the blur correction processing has been applied are played back, and to display them on the screen. An information processing device characterized by the following:

2. The system further includes a determination means for determining that if the field of view to which the blur correction processing is applied is narrower than a predetermined field of view, it is a section of the frame image that satisfies the predetermined conditions. The information processing apparatus according to feature 1.

3. The first section is a section of frame images in which the field of view displayed during playback, as determined by the image stabilization process, is smaller than a predetermined field of view. The information processing apparatus according to feature 1.

4. The system further includes a second means for obtaining an instruction to set the predetermined field of view, When the predetermined viewing angle is changed, the display control means distinguishes between the first section and the second section and displays them based on the changed predetermined viewing angle. The information processing apparatus according to claim 3.

5. The display control means displays the currently playing frame image on the screen and displays the position indicating the playback time corresponding to the frame image in the timeline area. The information processing apparatus according to feature 1.

6. The system further includes a determination means for determining the amount of image blur based on the inertial information of the imaging device when the video content was captured. The information processing apparatus according to feature 1.

7. The system further includes a determination means for determining the amount of image blur based on metadata attached to the video content. The information processing apparatus according to feature 1.

8. The system further includes a determination means for determining the amount of image blur based on the image features contained in the video content. The information processing apparatus according to feature 1.

9. The system further comprises a generation means for generating a second video content consisting of the second section, from which the first section has been cut out. The information processing apparatus according to feature 1.

10. The generation means generates the second video content and a third video content consisting of the first section from which the second section has been cut out. The information processing apparatus according to feature 9.

11. The system further includes a generation means for generating a second video content by cropping a section of each of the multiple frame images to which the image stabilization process has been applied, based on the field of view displayed when the image is played back, where the field of view is smaller than a predetermined field of view, and using the frame images of the section where the field of view is equal to or greater than the predetermined field of view as frame images with a predetermined field of view. The information processing apparatus according to feature 1.

12. The timeline area further includes a second acquisition means for acquiring an instruction to switch the display and hiding of the first section or the second section. The display control means controls the display of the timeline based on the instruction. The information processing apparatus according to feature 1.

13. The second acquisition means acquires an instruction to generate video content while the first or second section of the timeline area is hidden, In response to the second acquisition means receiving an instruction to generate the video content, it generates a second video content in which the hidden section has been cut out. The information processing apparatus according to feature 12.

14. The aforementioned video content is a video of images with distortion, The image processing means further performs a geometric transformation process to convert the video content into an image with reduced distortion. The information processing apparatus according to feature 1.

15. The aforementioned video content is a circular fisheye image, and the aforementioned geometric transformation process is a process that performs perspective projection transformation or an equirectangular transformation. The information processing apparatus according to feature 14.

16. A display control step that displays the playback time of multiple frame images constituting the video content and the corresponding timeline area on the screen, An image processing step in which blur correction processing is performed on the first frame image, which is included in the plurality of frame images, based on the amount of image blur caused by the movement of the imaging device that captures the video content, The process includes an acquisition step of acquiring a first field of view that is displayed when the first frame image to which the image stabilization process has been applied is played back, In the display control step, based on the field of view displayed when the multiple frame images to which the blur correction processing has been applied are played back, the system controls the timeline area to distinguish between a first section, which is a section of frame images whose field of view satisfies a predetermined condition, and a second section, which is a section of frame images whose field of view does not satisfy the predetermined condition, and to display them on the screen. An information processing method characterized by the following:

17. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

18. An imaging device that generates video content, A display control device that displays a timeline area on the screen corresponding to the playback time of multiple frame images constituting the video content, An image processing device that performs blur correction processing on the first frame image based on the amount of image blur of the first frame image included in the plurality of frame images due to the movement of the imaging device, The device includes an acquisition device that acquires a first field of view that is displayed when the first frame image to which the blur correction processing has been applied is played back. The display control device controls the timeline area to distinguish between a first section, which is a section of frame images whose field of view satisfies a predetermined condition, and a second section, which is a section of frame images whose field of view does not satisfy the predetermined condition, based on the field of view displayed when each of the plurality of frame images to which the blur correction processing has been applied is played back, and to display them on the screen. An information processing system characterized by the following:

Images