Electronic apparatus, method for controlling electronic apparatus, and program

Abstract

To generate a VR moving image reduced in flickering during reproduction.SOLUTION: An electronic apparatus for converting a first moving image of a first display system, the first moving image including a photographing area expressing the result of photographing by an imaging apparatus to a second moving image of a second display system, the second moving image having a plurality of frames includes: acquisition means for acquiring the first moving image; setting means for setting an invalid area having the same shape at the same position in the plurality of frames of the second moving image; and generation means for generating the second moving image by converting the first moving image, so as to set a pixel value indicating invalidity at the invalid area for each of the plurality of frames of the second moving image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an electronic device, a control method for an electronic device, and a program.

Background Art

[0002] Generally, a VR image (VR video) includes two images (a left-eye image and a right-eye image) that allow a user to view in stereoscopy by utilizing parallax when reproduced on a head-mounted display (HMD). The VR image includes, for example, two orthographic cylindrical projection images that are the results of performing orthographic cylindrical transformation on a circumferential fisheye image captured via two circumferential fisheye lenses, respectively.

[0003] When the attachment of the two circumferential fisheye lenses is not appropriate, the parallax between the two circumferential fisheye images may not be appropriate for stereoscopy. In this case, by adjusting the parameters used for the orthographic cylindrical transformation that converts the circumferential fisheye image into an orthographic cylindrical projection image, it is possible to achieve conversion into an orthographic cylindrical projection image in which the parallax is suitable for stereoscopy. However, as a result of this parameter adjustment, there may be no effective region of the circumferential fisheye image corresponding to a part of the region of the orthographic cylindrical projection image, and a part of the region of the orthographic cylindrical projection image may be set as an invalid region having invalid pixel values.

[0004] Patent Document 1 discloses a technique of making the shapes of the invalid regions of the left and right images the same during the orthographic cylindrical transformation of a VR image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] When generating a video file of a VR image, there is a method of realizing camera shake correction by adjusting parameters used in the orthographic cylindrical transformation so as to correct motion blur between the previous and subsequent frames and the corresponding frame. As described above, depending on the value of the parameter used in the orthographic cylindrical transformation, a part of the area of the orthographic cylindrical projection image of the conversion result may become an invalid area. Further, when the parameters used in the orthographic cylindrical transformation are different for each frame, there is a possibility that the invalid area of the orthographic cylindrical projection image moves during the playback of the video by performing camera shake correction.

[0007] As a result, when camera shake correction is performed, flicker occurs in the orthographic cylindrical projection image. However, the technique described in Patent Document 1 cannot suppress this flicker.

[0008] Therefore, an object of the present invention is to enable the generation of a VR video with little flicker during playback.

Means for Solving the Problems

[0009] One aspect of the present invention is An electronic device that converts a first video of a first display method, which is a first video including a shooting area representing the result of shooting by an imaging device, into a second video that is a video of a second display method having a plurality of frames, An acquisition means for acquiring the first video; Setting means for setting invalid areas of the same shape at the same position in the plurality of frames of the second video; Generating means for generating the second video by converting the first video so as to set pixel values indicating invalidity in the invalid areas for each of the plurality of frames of the second video; An electronic device characterized by having

[0010] One aspect of the present invention is A control method for an electronic device that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video that is a video of a second display method having a plurality of frames. An acquisition step of acquiring the first video; A setting step of setting invalid areas having the same shape at the same position in the plurality of frames of the second video; A generation step of generating the second video by converting the first video so that pixel values indicating invalidity are set in the invalid areas for each of the plurality of frames of the second video; A control method for an electronic device, characterized by comprising the above.

Effect of the Invention

[0011] According to the present invention, a VR video with little flicker during playback can be generated.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Modes for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0014] FIG. 1 shows an example of the configuration of an editing device 100, which is an image display device (video editing device) common to all embodiments. The editing device 100 is an electronic device such as, for example, a computer, a smartphone, or an imaging device (such as a digital camera).

[0015] The editing device 100 includes a control unit 101, a ROM 102, a RAM 103, an external storage device 104, an operation unit 105, a display unit 106, a communication unit 107, and a system bus 108.

[0016] The control unit 101 controls the entire editing device 100. The control unit 101 is, for example, a Central Processing Unit (CPU). The control unit 101 is also capable of performing image generation, setting processing, and calculation processing. Therefore, hereinafter, the control unit 101 can also operate as a "generation unit", a "setting unit", and a "coordinate calculation unit".

[0017] The ROM 102 is a Read Only Memory (ROM) that stores information (programs and parameters) that do not require modification.

[0018] The RAM 103 is a Random Access Memory (RAM) that temporarily stores information (programs and data) supplied from external devices and the like.

[0019] The external storage device 104 is fixedly installed in the editing device 100. The external storage device 104 is a hard disk or a flash memory. Alternatively, the external storage device 104 has an optical disk such as a floppy disk (FD) or a Compact Disk (CD) that is detachable from the editing device 100. The external storage device 104 may have a magnetic card, an optical card, an IC card, or a memory card.

[0020] The operation unit 105 receives instructions according to the operations of the user. The operation unit 105 has operation members (buttons or touch panels) that can be operated by the user.

[0021] The display unit 106 displays the data held by the editing device 100 and the data supplied to the editing device 100.

[0022] The communication unit 107 communicates with external devices such as cameras (imaging devices; digital cameras).

[0023] The system bus 108 communicably connects between each component (unit).

[0024] Video files including image data and metadata are stored (recorded) in the external storage device 104. Also, the video files are readable from the external storage device 104. The communication unit 107 can also receive video files from external devices such as cameras. Therefore, hereinafter, the "input unit" or "acquisition unit" of data is a name collectively referring to the external storage device 104 and the communication unit 107.

[0025] Referring to the flowchart of FIG. 2, the operation during shooting of a camera (imaging device) communicable with the editing device 100 will be described.

[0026] In step S201, the camera determines whether sensor data during shooting can be stored (recorded) in a video file. Here, the sensor data is, for example, inertial data indicating acceleration measured by a gyro sensor. The camera can record the movement of the camera during shooting by periodically storing the inertial data of the gyro sensor in the video file. If it is determined that the sensor data can be stored, the process proceeds to step S202. If it is determined that the sensor data cannot be stored, the process proceeds to step S203. If it is determined that the sensor data cannot be stored, the process proceeds to step S203.

[0027] In step S202, the camera stores the sensor data in the video file as time-domain metadata (metadata having information in time series).

[0028] In step S203, the camera determines whether to store audio (voice) in the video file. Depending on the shooting mode, the control unit 101 may not store the audio in the video file. If it is determined that the audio is to be stored, the process proceeds to step S204. If it is determined that the audio is not to be stored, the process proceeds to step S205.

[0029] In step S204, the camera stores the audio data (voice data) as audio sampling data in the video file.

[0030] In step S205, the camera stores one frame of the image as video sampling data in the video file.

[0031] In step S206, the camera determines whether shooting has been instructed to end. If it is determined that shooting has been instructed to end, the processing of this flowchart ends. If it is determined that shooting has not been instructed to end, the process returns to step S201.

[0032] Figure 3 shows the structure of the video file acquired by the camera. The video file includes a video header area 301, a file metadata area 302, and a sampling data area 303.

[0033] The video header area 301 holds information regarding the entire video (hereinafter referred to as "video-related information"). For example, the video-related information includes at least any one of information such as the image size (width and height) of the video image, the frame rate, the sample frequency of the audio, the sample size, and the number of channels. The video-related information may include information such as the position (playback position; playback time) and size of the sample of each media (video, audio, time domain metadata, etc.) (in the case of video, the data of one frame).

[0034] The file metadata area 302 holds metadata regarding the file itself and the entire video. For example, information about the lens at the time of shooting, design values (such as the center position or radius of the image circle), or measured values obtained by calibration are stored as metadata in the file metadata area 302.

[0035] The sampling data area 303 stores the sampling data of each medium. The sampling data area 303 includes a video sampling area 3031, an audio sampling area 3032, and a time-domain metadata sampling area 3033.

[0036] The video sampling area 3031 stores the sampling data of the video medium. The sampling data of the video medium is the image data stored in step S205 of FIG. 2. For example, when a VR video is acquired by shooting with a camera equipped with two fish-eye lenses, two circular fisheye images are stored as one frame of video sample in the video sampling area 3031.

[0037] The audio sampling area 3032 stores the sampling data of the audio medium. The sampling data of the audio medium is the audio data stored in step S204 of FIG. 2. If there is no need to record audio data, the audio sampling area 3032 may not exist.

[0038] The time-domain metadata sampling area 3033 stores the sampling data of the time-domain metadata medium. The sampling data of the time-domain metadata medium is, for example, the sensor data stored in step S202 of FIG. 2. If there is no need to record time-domain metadata, the time-domain metadata sampling area 3033 may not exist.

[0039] Figures 4A to 4C show a circumferential fisheye image recorded by a camera and an orthographic cylindrical projection image (hereinafter referred to as an "orthographic cylindrical image") as a result of converting the circumferential fisheye image by the editing device 100. Hereinafter, with reference to FIGS. 4A and 4B, an orthographic cylindrical image subjected to shake correction will be described, and with reference to FIG. 4C, an orthographic cylindrical image subjected to shake correction according to each of the embodiments described later will be described. For the sake of simplicity of explanation, one circumferential fisheye image recorded by a camera equipped with one fisheye lens and one orthographic cylindrical image generated from the circumferential fisheye image will be described. However, the same features as the following description also occur for two circumferential fisheye images recorded by a camera equipped with two fisheye lenses and two orthographic cylindrical images generated from the two circumferential fisheye images.

[0040] (FIG. 4A) The circumferential fisheye image 401 is an example of one circumferential fisheye image recorded by a camera equipped with a fisheye lens.

[0041] The image circle 4011 is an area (shooting area) where a subject is photographed by a camera through a fisheye lens. Inside the image circle 4011, the subject is recorded. The outside of the image circle 4011 is an invalid area where no image is reflected, and a black or near-black color is recorded. The invalid area is an area that has nothing to do with the result of shooting. The position information (information such as the center position and radius) of the image circle 4011 in the circumferential fisheye image is stored in the file metadata area 302 as file metadata. Usually, the position of the image circle 4011 is a fixed position because the position of the lens does not change during shooting.

[0042] The area 4012 is an area to be subjected to orthographic cylindrical conversion. The inside of the area 4012 is developed into an orthographic cylindrical image. The information representing the size of the area 4012 is also stored in the file metadata area 302 as file metadata. The position of the area 4012 is, by default, a position having the same center as the center of the image circle 4011, but can be changed.

[0043] The orthographic cylindrical image 402 is an orthographic cylindrical image that is the result of orthographically projecting the area 4012 within the image circle 4011.

[0044] (Fig. 4B) The circumferential fisheye image 403 is an example of a circumferential fisheye image recorded by the camera, similar to the circumferential fisheye image 401.

[0045] The image circle 4031 is an image circle, similar to the image circle 4011. The area 4032 is an area to be orthographically projected, similar to the area 4012.

[0046] In the circumferential fisheye image 403, compared to the circumferential fisheye image 401, the camera is turned slightly to the left, so the subject is photographed at a position slightly shifted to the right from the center inside the image circle 4031.

[0047] The orthographic cylindrical image 404 is an orthographic cylindrical image that is the result of orthographically projecting the area 4032 within the circumferential fisheye image 403. In the circumferential fisheye image 403, the position of the area 4032 with respect to the image circle 4031 is the same as the position of the area 4012 with respect to the image circle 4011 in the circumferential fisheye image 401. Therefore, in the orthographic cylindrical image 404, the subject is arranged at a position slightly shifted to the right from the center of the image.

[0048] (Fig. 4C) The circumferential fisheye image 405 is an example of a circumferential fisheye image recorded by the camera, similar to the circumferential fisheye image 401 and the circumferential fisheye image 403.

[0049] The image circle 4051 is an image circle, similar to the image circle 4011 and the image circle 4031.

[0050] The area 4052 is an area to be orthographically projected, similar to the area 4012 and the area 4032. The position of the area 4052 is slightly shifted to the right compared to the positions of the area 4012 and the area 4032.

[0051] The orthographic cylindrical image 406 is an orthographic cylindrical image that is the result of orthographically projecting the area 4052 within the circumferential fisheye image 405. Since the position of the area 4052 is closer to the right side of the image circle 4051, the subject is positioned at the center in the orthographic cylindrical image. Thus, in each of the following embodiments, when generating an orthographic cylindrical image from a circumferential fisheye image, the area to be orthographically projected in the circumferential fisheye image is changed. By doing so, an effect of reducing the influence of camera shake in the orthographic cylindrical image can be obtained.

[0052] On the other hand, since a part of the right side of the area 4052 protrudes from the outside of the image circle 4051 (= invalid area), a part of the area on the right side in the orthographic cylindrical image 406, which is the result of the orthographic projection, refers to the invalid area of the circumferential fisheye image 405. Since there are no pixel values of the valid circumferential fisheye image 405 in the invalid area, there are pixels (for example, black pixels) with pixel values indicating the invalid area in the orthographic cylindrical image 406.

[0053] <Embodiment 1> With reference to the flowchart of FIG. 5, a process for generating a VR video (VR180 video), which is an orthographic cylindrical image, will be described. In Embodiment 1, a VR video is generated from a video that is a circumferential fisheye image with a different display format from the orthographic cylindrical image. Note that Embodiment 1 is applicable not only when generating an orthographic cylindrical image from a circumferential fisheye image, but also when generating an image with a display format different from a certain display format from an image with that display format.

[0054] In step S501, the control unit 101 reads out the common metadata (metadata common to all frames) stored in the file metadata area 302 of the video file through the input unit. For example, the common metadata includes information such as the center position of the image circle, the radius of the image circle, and the radius of the target area of the orthographic cylindrical transformation. Further, the control unit 101 also reads out the parameter values (manual correction parameter values) set by the user using the operation unit 105 as metadata. These metadata are used in a later process to generate an orthographic cylindrical image from the circumferential fisheye image.

[0055] In step S502, the control unit 101 calculates the blur correction value for each frame. The process of step S502 will be described in detail with reference to the flowchart of FIG. 6. The blur correction value is a value for correcting the position deviation in the orthographic cylindrical image caused by hand shake (camera shake).

[0056] In step S503, the control unit 101 calculates the invalid area of the orthographic cylindrical image for all frames. The process of step S503 will be described in detail with reference to the flowchart of FIG. 7.

[0057] In step S504, the control unit 101 generates the orthographic cylindrical image for each frame and outputs the generated orthographic cylindrical image. The process of step S504 will be described in detail with reference to the flowchart of FIG. 8.

[0058] (Process of step S502) With reference to the flowchart of FIG. 6, the calculation process of the blur correction value will be described. Hereinafter, the control unit 101 calculates the blur correction value for each frame. By using the blur correction value when generating the orthographic cylindrical image from the circumferential fisheye image, it is possible to correct the influence of blur (hand shake; camera shake) in the orthographic cylindrical image for each frame.

[0059] In step S601, the control unit 101 sets a variable i representing the frame number of the image data to be read and a variable n representing the frame number of the image data for which the blur correction value is to be calculated, each to 0.

[0060] In step S602, the control unit 101 determines whether all the frame image data (the image data of the circumferential fisheye image) stored in the video file has been read. If it is determined that all the frame image data has been read, the process proceeds to step S612. If it is determined that the image data of at least one frame has not been read, the process proceeds to step S603.

[0061] In step S603, the control unit 101 reads out the image data of the circumferential fisheye image of the frame with frame number i (= frame i) from the video sampling area 3031 of the video file. Here, if the image data stored in the video sampling area 3031 is compressed, the control unit 101 obtains RGB-formatted image data by decoding the image data.

[0062] In step S604, the control unit 101 analyzes the read image data (performs image processing on the image data) to extract feature points in the circumferential fisheye image.

[0063] In step S605, the control unit 101 determines whether the sensor data of frame i is stored in the time-domain metadata sampling area 3033 of the video file. If it is determined that the sensor data of frame i is stored, the process proceeds to step S606. If it is determined that the sensor data of frame i is not stored, the process proceeds to step S607.

[0064] In step S606, the control unit 101 reads out the sensor data of frame i (sensor data regarding the movement of the camera at the time of shooting the circumferential fisheye image of frame i) from the time-domain metadata sampling area 3033.

[0065] In step S607, the control unit 101 adds 1 to the variable i representing the read frame number.

[0066] In step S608, the control unit 101 determines whether image data of the number of frames necessary for calculating the amount of blur of the frame (= frame n) of frame number n has been read. For example, the amount of blur is the "amount of camera movement or the amount of movement of the subject in the image data" caused by hand shake during a specific period centered on the time of one frame. Therefore, for example, in order to calculate the amount of blur of one frame, the frames for one second before and after the one frame of image data are required. If it is determined that the image data of the required number of frames has been read, the process proceeds to step S609. If it is determined that the image data of the required number of frames has not been read, the process returns to step S602.

[0067] In step S609, the control unit 101 calculates the amount of blur of frame n. For example, the control unit 101 calculates the amount of blur of frame n based on the "feature point information and sensor data" of frame n and its preceding and succeeding frames. At this time, the control unit 101 may calculate the amount of blur based only on the feature point information or only on the sensor data.

[0068] The movement of the camera due to unintentional hand shake by the user is different from the movement of the camera as camera work. For example, when hand shake occurs, the camera makes reciprocating movements such as left - right or up - down in a short cycle. Therefore, the control unit 101 can calculate the amount of blur based on the movement pattern of feature points between a plurality of frames or the values of the gyro sensor mounted on the camera.

[0069] In step S610, the control unit 101 calculates a blur correction value for correcting the blur of the amount of blur calculated in step S609 based on the amount of blur of frame n.

[0070] In step S611, the control unit 101 adds 1 to the variable n.

[0071] As described above, in order to calculate the amount of blur correction for one frame, information on the frames before and after that frame is also required. Therefore, when all the frames have been read, the blur correction values for all the frames have not been calculated yet, and there are a plurality of frames for which the blur correction values are not calculated. Therefore, after step S612, the blur correction values for these remaining frames are calculated.

[0072] In step S612, the control unit 101 determines whether the blur correction values for all the frames have been calculated. If it is determined that the blur correction values for all the frames have been calculated, the processing of this flowchart ends. If it is determined that the blur correction values for all the frames have not been calculated, the process proceeds to step S613.

[0073] In step S613, the control unit 101 calculates the amount of blur of frame n. The process of step S613 is the same as the process of step S609.

[0074] In step S614, the control unit 101 calculates a blur correction value for correcting the blur of the amount of blur calculated in step S613 based on the amount of blur of frame n. The process of step S614 is the same as the process of step S610.

[0075] In step S615, the control unit 101 adds 1 to the variable n.

[0076] (Process of step S503) With reference to the flowchart of FIG. 7, the calculation process of the invalid area (the invalid area common to all frames) in the orthographic cylindrical image will be described.

[0077] In step S701, the control unit 101 sets the variable i representing the frame number to 0.

[0078] In step S702, the control unit 101 determines whether the calculation of invalid pixels in the orthographic cylindrical images of all frames has been completed. Here, an invalid pixel is a pixel for which an invalid pixel value (a specific pixel value) is set in a subsequent process. If it is determined that the calculation of invalid pixels in the orthographic cylindrical images of all frames has been completed, the processing of this flowchart ends. If it is determined that the calculation of at least one of the invalid pixels in the orthographic cylindrical images of all frames has not been completed, the process proceeds to step S703. If it is determined that the calculation of at least one of the invalid pixels in the orthographic cylindrical images of all frames has not been completed, the process proceeds to step S703.

[0079] In step S703, the control unit 101 acquires the blur correction value for frame i. The blur correction value is used to correct the position of the region of interest for orthographic cylindrical conversion in the circumferential fisheye image 405 shown in FIG. 4C.

[0080] In step S704, the control unit 101 sets the coordinates (x, y) of the pixel to be processed in the orthographic cylindrical image of frame i to (0, 0). Here, in one loop (steps S705 to S710), the control unit 101 processes both monocular images (both the left-eye image and the right-eye image) in the orthographic cylindrical image. In the monocular image of the VR image, the width and height are equal to each other. Therefore, if the jth pixel of the monocular image and the height of the monocular image is j rows, the coordinates of the monocular image are included in the range from the upper left (0, 0) to the lower right (j - 1, j - 1) of the monocular image.

[0081] In step S705, the control unit 101 determines whether the processing of steps S706 to S710 has been executed for all the pixels in the orthographic cylindrical image of one frame. If it is determined that the processing of steps S706 to S710 has been executed for all the pixels, the process proceeds to step S711. If it is determined that the processing of steps S706 to S710 has not been executed for at least one of all the pixels, the process proceeds to step S706.

[0082] In step S706, the control unit 101 obtains the coordinates (xL, yL) of the circumferential fisheye image corresponding to the coordinates (x, y) of the pixel to be processed in the left-eye image. The coordinates of the circumferential fisheye image corresponding to the coordinates of the orthographic cylindrical image can be calculated by performing the inverse transformation of the orthographic cylindrical transformation. At this time, the control unit 101 obtains the coordinates (xL, yL) of the circumferential fisheye image based on the common metadata read in step S501 and the manual correction parameter values set by the user. In addition, the control unit 101 obtains the coordinates (xL, yL) of the circumferential fisheye image based on the blur correction value obtained in step S703 in consideration of the blur correction in the orthographic cylindrical image.

[0083] In step S707, the control unit 101 obtains the coordinates (xR, yR) of the circumferential fisheye image corresponding to the coordinates (x, y) of the pixel to be processed in the right-eye image. This is the same process as step S706.

[0084] In step S708, the control unit 101 determines whether both of the two coordinates (xL, yL) and the coordinates (xR, yR) belong to the valid region. In Embodiment 1, the valid region is the image circle (shooting region). If it is determined that both coordinates are coordinates within the image circle, the process proceeds to step S710. If it is determined that at least one of the two coordinates is not a coordinate within the image circle, the process proceeds to step S709.

[0085] In step S709, the control unit 101 determines that the pixel of the orthographic cylindrical image to be processed is an invalid pixel, and stores the coordinates (x, y) of the pixel in the invalid coordinate table. The invalid coordinate table is commonly used throughout all frames of the VR video. Therefore, the coordinates of the pixels determined to be invalid pixels in at least one frame are registered in the invalid coordinate table. Thus, by controlling the invalid coordinate table in this way, the control unit 101 can set the set of the coordinates (invalid coordinates) of the pixels determined to be invalid pixels in at least one frame as the invalid region.

[0086] In step S710, the control unit 101 updates the coordinates (x, y) to the coordinates of the next pixel. The coordinates of the next pixel are the coordinates obtained by adding 1 to x which is the X coordinate. However, when the X coordinate obtained by adding 1 to x reaches the width j, it is the coordinates obtained by adding 1 to y which is the Y coordinate and resetting x which is the X coordinate to 0.

[0087] In step S711, the control unit 101 adds 1 to the variable i representing the frame number.

[0088] (Processing of step S504) With reference to the flowchart of FIG. 8, the process of generating a perspective cylindrical image from a circumferential fisheye image will be described for all frames.

[0089] In step S801, the control unit 101 sets the variable i representing the frame number to 0.

[0090] In step S802, the control unit 101 determines whether or not the perspective cylindrical images of all the frames constituting the VR video have been output. If it is determined that the perspective cylindrical images of all the frames have been output, the processing of this flowchart ends. If it is determined that at least one of the perspective cylindrical images of all the frames has not been output, the process proceeds to step S803.

[0091] In step S803, the control unit 101 reads out the image data of the circumferential fisheye image of frame i from the video sampling area 3031 through the input unit. When the image data stored in the video sampling area 3031 is compressed, the control unit 101 converts the image data into the RGB format by decoding the image data.

[0092] In step S804, the control unit 101 acquires the blur correction value of frame i. The process of step S804 is the same as the process of step S703.

[0093] In step S805, the control unit 101 sets the coordinates (x, y) of the pixel to be processed in the orthographic cylindrical image to (0, 0). Here too, similar to the processing in the flowchart of FIG. 7, the processing for both the left-eye image and the right-eye image is performed in one loop (steps S806 to S812). Therefore, if the coordinates of a single-eye image are j pixels in width and j rows in height for a single-eye image, the upper left corner takes the range from (0, 0) to the lower right (j - 1, j - 1).

[0094] In step S806, the control unit 101 determines whether the pixel values of all the pixels in the orthographic cylindrical image of frame i are set. If it is determined that the pixel values of all the pixels in the orthographic cylindrical image of frame i are set, the process proceeds to step S813. If it is determined that the pixel values of at least any one of all the pixels in the orthographic cylindrical image of frame i are not set, the process proceeds to step S807.

[0095] In step S807, the control unit 101 determines whether the coordinates (x, y) of the pixel to be processed are registered in the invalid coordinate table (that is, the pixel at the coordinates (x, y) is included in the invalid area). If it is determined that the coordinates (x, y) are registered in the invalid coordinate table, the process proceeds to step S808. If it is determined that the coordinates (x, y) are not registered in the invalid coordinate table, the process proceeds to step S810.

[0096] In step S808, the control unit 101 sets a pixel value indicating invalidity (an invalid pixel value) to the pixel value of the pixel at the coordinates (x, y) of the left-eye image in the orthographic cylindrical image. For example, the pixel value indicating invalidity is a pixel value of a specific value such as a pixel value representing black, a pixel value representing white, or a pixel value representing blue.

[0097] In step S809, the control unit 101 sets a pixel value indicating invalidity to the pixel value of the pixel at the coordinates (x, y) of the right-eye image in the orthographic cylindrical image. According to steps S808 and S809, a pixel value indicating invalidity is set in the invalid areas of the same position and the same shape in the left-eye image and the right-eye image in the orthographic cylindrical image.

[0098] In step S810, the control unit 101 sets the pixel value of the coordinate (x, y) of the left-eye image in the orthoscopic cylindrical image based on the blur correction value. In step S811, the control unit 101 sets the pixel value of the coordinate (x, y) of the right-eye image in the orthoscopic cylindrical image based on the blur correction value. In this way, by setting the pixel values of the pixels of the orthoscopic cylindrical image based on the blur correction value, it becomes possible to generate an orthoscopic cylindrical image with reduced influence of camera shake. Therefore, when the subject is not moving and camera shake occurs, in a plurality of consecutive frames, the position where the subject appears in the orthoscopic cylindrical image can be maintained at a fixed position. The details of the processing in steps S810 and S811 will be described later with reference to the flowchart of FIG. 9.

[0099] In step S812, the control unit 101 updates the coordinate (x, y) to the coordinate of the next pixel. The coordinate of the next pixel is the coordinate obtained by adding 1 to x which is the X coordinate. However, when the X coordinate obtained by adding 1 to x reaches the width j, it is the coordinate obtained by adding 1 to y which is the Y coordinate and resetting x which is the X coordinate to 0. By repeating the processing of steps S807 to S812 for each pixel of the orthoscopic cylindrical image of one frame, the pixel value of each pixel of the orthoscopic cylindrical image of that frame is set, and as a result, the orthoscopic cylindrical image of that frame is generated.

[0100] In step S813, the control unit 101 outputs the orthoscopic cylindrical image of frame number i generated in steps S803 to S812. When the orthoscopic cylindrical image is saved in a file, the orthoscopic cylindrical image is output to the external storage device 104. When the orthoscopic cylindrical image is displayed, the orthoscopic cylindrical image is output to the display unit 106.

[0101] In step S814, the control unit 101 adds 1 to the variable i representing the frame number.

[0102] With reference to the flowchart of FIG. 9, the processing for setting the pixel value of one pixel of the orthoscopic cylindrical image will be described.

[0103] In step S901, the control unit 101 calculates the coordinates (X, Y) of the circumferential fisheye image corresponding to the coordinates (x, y) of the orthographic cylindrical image based on the blur correction value and the like. The process of step S901 is the same as the processes of steps S706 and S707 in FIG. 7.

[0104] In step S902, the control unit 101 acquires the pixel value of the coordinates (X, Y) of the circumferential fisheye image. Alternatively, the control unit 101 acquires, as the pixel value of the coordinates (X, Y), the pixel value interpolated by the bicubic method or the like based on the number of pixels around the coordinates (X, Y) obtained in step S901.

[0105] In step S903, the control unit 101 sets the pixel value acquired in step S902 as the pixel value of the coordinates (x, y) of the orthographic cylindrical image.

[0106] According to Embodiment 1, in the case where different blur corrections are performed for each frame, even when the orthographic cylindrical image is generated from the circumferential fisheye image, the positions and shapes of the invalid areas generated in the orthographic cylindrical image can be made the same positions and the same shapes throughout all frames. Thereby, when playing back the VR video (orthographic cylindrical image), it is possible to suppress the occurrence of flickering due to changes in the shape and position of the invalid area.

[0107] <Embodiment 2> FIG. 10A is a diagram showing an example of a binocular fisheye lens according to Embodiment 2. FIG. 10B is a diagram showing an example of a circumferential fisheye image acquired by a camera equipped with the binocular fisheye lens.

[0108] In FIG. 10A, the binocular fisheye lens 1010 is a binocular fisheye lens viewed from above. The binocular fisheye lens 1010 is used to acquire two circumferential fisheye images. The binocular fisheye lens 1010 includes a glass part 1011, a support 1012, a glass part 1013, a support 1014, and a support 1015.

[0109] The glass part 1011 is the glass part of the left-eye fisheye lens. The support 1012 is the support of the glass part 1011 of the left-eye fisheye lens.

[0110] The glass part 1013 is the glass part of the fish-eye lens for the right eye. The support 1014 is the support for the glass part 1013 of the fish-eye lens for the right eye. The support 1015 is the support for the two fish-eye lenses on the left and right.

[0111] The angular field of view 1016 connecting the line segment AB and the line segment BC is the angular field of view of the shooting range of the fish-eye lens for the left eye. In the fish-eye lens for the left eye, the angular field of view 1016 is an angular field of view of 180 degrees. The angular field of view 1017 connecting the line segment DE and the line segment EF is the angular field of view of the shooting range of the fish-eye lens for the right eye. Similar to the angular field of view 1016, the angular field of view 1017 is an angular field of view of 180 degrees.

[0112] The circumferential fish-eye image 1020 is an example of a circumferential fish-eye image obtained by a camera equipped with the binocular fish-eye lens 1010.

[0113] The image circle 1021 is the image circle corresponding to the glass part 1011. The image circle 1022 is the image circle corresponding to the glass part 1013.

[0114] When the positions of the two lenses are close to each other like the binocular fish-eye lens 1010 and the angular field of view of the shooting range of each lens is close to 180 degrees, the other lens will enter the shooting range using one lens.

[0115] The image 1023 is the image of the glass part 1013 reflected in the glass part 1011. Similarly, the image 1024 is the image of the glass part 1011 reflected in the glass part 1013. Since these images are within the image circle, they can be the target of orthographic cylindrical transformation. Also, due to the parameter values for shake correction (the parameter values for orthographic cylindrical transformation), the reflection positions of these images in the orthographic cylindrical image are not constant. On the other hand, when the two fish-eye lenses are in a fixed positional relationship like the binocular fish-eye lens 1010, the reflection positions of these images in the circumferential fish-eye image are constant.

[0116] Referring to FIG. 11, the calculation process of the invalid region of the orthographic cylindrical image of all frames in Embodiment 2 will be described. This is a process corresponding to step S503 in the flowchart of FIG. 5 according to Embodiment 1. The processes other than step S503 are the same as those in each step of the flowchart of FIG. 5. Also, in steps S701 to S711 in the flowchart of FIG. 11, the same processes as the steps with the same names in the flowchart of FIG. 7 are performed.

[0117] In step S1101, the control unit 101 acquires information regarding a preset exclusion region in the image circle of the circumferential fisheye image. The information regarding the exclusion region is, for example, information regarding the coordinates of images (images 1023 and 1024) that are unnecessary as VR images within the image circle of the circumferential fisheye image 1120 as shown in FIG. 10B. Since the position of the lens reflection region is constant in the circumferential fisheye image, it is possible to calculate the coordinates of this position in advance and store them in advance in a form such as a coordinate table or a mask image.

[0118] In step S1110, the control unit 101 determines whether each of the two coordinates (xL, yL) and (xR, yR) of the circumferential fisheye image obtained in steps S706 and S707 is outside the exclusion region. If it is determined that both coordinates are outside the exclusion region, the process proceeds to step S710. If it is determined that at least one of the coordinates is within the exclusion region, the process proceeds to step S709.

[0119] According to Embodiment 2, the coordinates of the exclusion region within the image circle of the circumferential fisheye image are also treated as regions that are not valid regions, similar to the regions outside the image circle. That is, the region in the orthographic cylindrical image corresponding to the exclusion region is also treated as an invalid region. Therefore, unnecessary images corresponding to the exclusion region are not displayed in the orthographic cylindrical image, and since the shape of the invalid region does not change in each frame, it is possible to suppress the flickering of the VR video and improve the user's sense of immersion in the VR video.

[0120] <Embodiment 3> In Embodiments 1 and 2, in order to calculate the invalid region, calculations in the reverse direction of the orthographic cylindrical transformation are performed. These processes are considered to have a large amount of calculation and take a long processing time. However, in some cases, the accuracy of the invalid region may be low, but it is desirable to calculate the invalid region of the orthographic cylindrical image in a short processing time. Therefore, in Embodiment 3, an editing device 100 that can calculate the invalid region of the orthographic cylindrical image in a short processing time by calculating the invalid region according to the blur in four directions of the camera during shooting will be described.

[0121] With reference to the flowchart of FIG. 12, the calculation process of the blur correction value (the process corresponding to step S502 in the flowchart of FIG. 5 according to Embodiment 1) in Embodiment 3 will be described.

[0122] In the flowchart of FIG. 12, the processes of steps S601, S602, S606 to S615 are the same as the processes with the same names in the flowchart of FIG. 6. On the other hand, in the flowchart of FIG. 12, if it is determined in step S602 that the image data of any one of all the frames has not been read, the process proceeds to step S606. That is, since the processes of steps S603 to S605 (more specifically, the feature point extraction process in step S604) are not performed, the number of processes of the editing device 100 can be reduced.

[0123] With reference to the flowchart of FIG. 13, the calculation process (the process of step S503) of the invalid region of the orthographic cylindrical image of all the frames in Embodiment 3 will be described. The processes of steps S701 to S703 and S711 in the flowchart of FIG. 13 are the same as the processes of the steps with the same names in the flowchart of FIG. 7.

[0124] In step S1305, the control unit 101 obtains the maximum shake correction value in each of the four directions among the shake correction values obtained in step S703. The maximum shake correction values in the four directions are the maximum shake correction value in the positive direction of the X axis, the maximum shake correction value in the negative direction of the X axis, the maximum shake correction value in the positive direction of the Y axis, and the maximum shake correction value in the negative direction of the Y axis. The maximum shake correction values in the four directions are values corresponding to the magnitudes of the camera shakes in the four directions caused by hand shake.

[0125] In step S1306, the control unit 101 determines an invalid area corresponding to the maximum shake correction value in each direction determined in step S1305. This can be achieved, for example, by preparing in advance mask images of the invalid areas at several levels corresponding to the shake correction values in each direction, and selecting the mask image of the invalid area corresponding to a shake correction value greater than or equal to the maximum shake correction value in that direction.

[0126] For example, FIG. 14 shows four types of mask images of invalid areas prepared in advance according to the shake correction value in the positive direction of the X axis. The mask image 1401 is the mask image when the shake correction value in the positive direction of the X axis is 0. The mask image 1402 is the mask image when the shake correction value in the positive direction of the X axis is small. The mask image 1403 is the mask image when the shake correction value in the positive direction of the X axis is medium. The mask image 1404 is the mask image when the shake correction value in the positive direction of the X axis is large. In each mask image, the area represented in black is the invalid area.

[0127] Thus, in the mask image according to the shake correction value in the positive direction of the X axis, the area representing the invalid area appears on the right side of the image. Also, for example, in the mask image according to the shake correction value in the negative direction of the X axis, the area representing the invalid area appears on the left side of the image.

[0128] Similarly, in the mask image according to the shake correction value in the positive direction of the Y axis, the area representing the invalid area appears on the lower side of the image, and in the mask image according to the shake correction value in the negative direction of the Y axis, the area representing the invalid area appears on the upper side of the image.

[0129] In step S1307, the control unit 101 obtains the logical sum of the mask images of the four-direction invalid regions determined in step S1306 as the common invalid region of the VR video (all frames).

[0130] For example, assume that a mask image when the blur correction value in the positive direction of the X-axis is small, a mask image when the blur correction value in the negative direction of the X-axis is large, a mask image when the blur correction value in the positive direction of the Y-axis is medium, and a mask image when the blur correction value in the negative direction of the Y-axis is 0 are selected. In this case, as shown in FIG. 15, the control unit 101 obtains a mask image 1500 that indicates an invalid region (region represented in black) that is the logical sum of the four mask images 1500.

[0131] According to Embodiment 3, since there is no per-pixel loop process in steps S705 to S710 in the flowchart of FIG. 7, the invalid region of the orthographic cylindrical image can be calculated with a small processing amount and a short processing time.

[0132] <Embodiment 4> The information on the invalid region in the orthographic cylindrical image calculated in the past can be reused when specific conditions are met. The case where specific conditions are met is, for example, when the calculation method of the invalid region is the same and the manual correction parameters are also the same.

[0133] With reference to the flowchart of FIG. 16, the process of generating a VR video according to Embodiment 4 will be described. Note that in steps S501 to S504 in the flowchart of FIG. 16, the same processes as the steps with the same names in the flowchart of FIG. 5 are performed.

[0134] In step S1603, the control unit 101 determines whether invalid area information (information including the invalid area and settings for calculating the invalid area, etc.) of the orthographic cylindrical image is stored in a file. If it is determined that the invalid area information is stored in the file (that is, information regarding the invalid area calculated during past image conversion is stored in the file), the process proceeds to step S1604. If it is determined that the invalid area information is not stored in the file, the process proceeds to step S503. Here, the file storing the invalid area information of the orthographic cylindrical image may be the video file itself or a file separate from the video file.

[0135] In step S1604, the control unit 101 determines whether the invalid area of the orthographic cylindrical image stored in the file is calculated with the same settings as the current settings. Here, the "settings" refer to, for example, the calculation method of the invalid area or the manual correction parameter value. If it is determined that the invalid area is calculated with the same settings as the current settings, the process proceeds to step S1605. If it is determined that the invalid area is not calculated with the same settings as the current settings, the process proceeds to step S503.

[0136] In step S1605, the control unit 101 reads the invalid area information of the orthographic cylindrical image stored in the file. By this process, in the subsequent step S504, an orthographic cylindrical image (VR video) is generated based on the invalid area information.

[0137] In step S1607, the control unit 101 stores, as invalid area information, the invalid area of the orthographic cylindrical image of the video file calculated in step S503, the settings for calculating the invalid area, and the manual correction parameters, etc. in a file. Here, the destination file for storage may be the video file itself or a file separate from the video file.

[0138] According to Embodiment 4, when appropriate invalid area information is stored in advance, the calculation process of the invalid area can be omitted, so that an orthographic cylindrical image can be generated at a higher speed.

[0139] <Embodiment 5> Multiple video files may be played continuously as one video. Therefore, in Embodiment 5, the editing device 100 controls so that the position and shape of the invalid area do not change in the playback unit (multiple video files) rather than in the video file unit.

[0140] Referring to the flowchart of FIG. 17, a method for generating a perspective cylindrical image from a circular fisheye image and generating a VR video for multiple video files in Embodiment 5 will be described. Steps S501 to S504 in the flowchart of FIG. 17 are the same as steps S501 to S504 in the flowchart of FIG. 5. In steps S501 to S504 in the flowchart of FIG. 17, processing is executed for the video file with file number n.

[0141] In step S503, the same invalid coordinate table is used for all video files that are played continuously (all video files that can be played continuously). As a result, the coordinates of the pixels determined to be invalid pixels in any frame of any video file are registered in the invalid coordinate table. Also in step S504, a perspective cylindrical image is output (generated) using the same invalid coordinate table for any video file.

[0142] In step S1701, the control unit 101 sets a variable n representing the file number of the video file to 0.

[0143] In step S1702, the control unit 101 determines whether the processing of steps S501 to S503 has been executed for all video files. If it is determined that the processing of steps S501 to S503 has been executed for all video files, the process proceeds to step S1707. If it is determined that the processing of steps S501 to S503 has not been executed for at least any one of all video files, the process proceeds to step S501.

[0144] In step S1706, the control unit 101 adds 1 to the variable n representing the file number.

[0145] In step S1707, the control unit 101 sets the variable n representing the file number to 0 again.

[0146] In step S1708, the control unit 101 determines whether the process of step S504 has been executed for all video files. If it is determined that the process of step S504 has been executed for all video files, the processing of this flowchart is completed. If it is determined that the process of step S504 has not been executed for at least any one of all video files, the process proceeds to step S504.

[0147] In step S1710, the control unit 101 adds 1 to the variable n representing the file number.

[0148] In Embodiment 5, the position and shape of the invalid region generated in the orthographic cylindrical image can be made the same position and the same shape throughout all frames of a plurality of video files. Thereby, when reproducing the orthographic cylindrical images of a plurality of video files, it is possible to suppress the occurrence of flickering due to changes in the shape of the invalid region or the like.

[0149] <Embodiment 6> In Embodiment 6, the editing device 100 calculates an orthographic cylindrical image from a circumferential fisheye image based on image correction information such as a blur correction value (correction value for shake correction). Further, the editing device 100 generates a mask image (image mask; mask) that covers a part of the orthographic cylindrical image based on the image correction information. The circumferential fisheye image in Embodiment 6 has two image circles (imaging regions).

[0150] FIG. 18 is a diagram showing the configuration of the control unit 101 in Embodiment 6. The control unit 101 includes a correction information generation unit 109, a coordinate information generation unit 110, and a mask generation unit 111. Each configuration of the control unit 101 will be described with reference to the flowchart of FIG. 19. The flowchart of FIG. 19 is a diagram for explaining image generation according to Embodiment 6.

[0151] In step S501, the control unit 101 reads out the common metadata stored in the file metadata area 302 of the video file through the input unit. The common metadata is metadata related to the shooting state of the video. For example, the common metadata includes the number of pixels of the video file, the design values of the optical system, the size and position information of the image circle on the sensor, information related to the manufacturing error of the optical system, the time-series information of the posture of the optical system during video shooting, and information related to the sensor output.

[0152] In step S502, the correction information generation unit 109 generates, for example, the blur correction value for each frame as image correction information (geometric image correction information) according to the flowchart of FIG. 6. Note that by using the blur correction value when generating a perspective cylindrical image from a circumferential fisheye image (when generating a perspective cylindrical image), it is possible to correct the influence of camera shake (camera shake; hand shake) in the perspective cylindrical image for each frame.

[0153] In Embodiment 6, in step S609 of the flowchart of FIG. 6, the correction information generation unit 109 (control unit 101) generates the blur correction value for the area of the circumferential fisheye image based on the metadata related to the shooting state of the video. Also, in step S602, whether the image data of all frames has been read may be referred to the "information on the number of frames included in the video" included in the metadata related to the shooting state of the video.

[0154] In step S1902, the mask generation unit 111 generates a mask image (such as the mask image 2200 shown in FIG. 22) based on the blur correction value which is the image correction information. The mask image includes at least "a mask area that covers the orthographic cylindrical image" and "a mask-free area that allows the orthographic cylindrical image to be displayed through it without covering it". The generation process of the mask image will be described later using the flowchart of FIG. 20.

[0155] In step S1903, the coordinate information generation unit 110 generates an orthographic cylindrical image by obtaining the coordinate values of the area of the circumferential fisheye image corresponding to the orthographic cylindrical image for each frame.

[0156] FIG. 21 shows a flowchart showing the process of generating an orthographic cylindrical image from a circumferential fisheye image. The flowchart of FIG. 21 is the flowchart obtained by removing steps S807 to S809 from the flowchart of FIG. 8. Specifically, when it is determined in step S806 that the pixel value of at least any one pixel of the orthographic cylindrical image of frame i is not set, the process proceeds to step S810 without performing the process of step S807. That is, in Embodiment 6, the invalid coordinate table is not used, and the calculation of the invalid area in the orthographic cylindrical image is not performed.

[0157] In step S1904, the coordinate information generation unit 110 generates a new orthographic cylindrical image by applying (superimposing; synthesizing) the mask image generated in step S1902 to the orthographic cylindrical image generated in step S1903.

[0158] ​The coordinate information generation unit 110 superimposes, for example, a mask image on an orthographic cylindrical image. At this time, the coordinate information generation unit 110 ensures that the area overlapping with the mask area in the orthographic cylindrical image is hidden by the mask area, and the area overlapping with the area without a mask in the orthographic cylindrical image is not hidden. For example, the coordinate information generation unit 110 generates a new orthographic cylindrical image (orthographic cylindrical image with a mask image) by adopting the darker (lower luminance value) pixel for each pixel at the same coordinates of the mask image and the orthographic cylindrical image. Thus, the coordinate information generation unit 110 may generate a new orthographic cylindrical image.

[0159] In Embodiment 6, by using metadata, the result of the shake correction process is considered in the mask image. Therefore, even if an area unintended by the photographer (area outside the image circle) appears in the orthographic cylindrical image, that area can be hidden by the mask image. Thus, the user's sense of immersion in the VR video is improved.

[0160] (Generation process of mask image; S1902) With reference to the flowchart of FIG. 20, the generation process of the mask image in step S1902 will be described.

[0161] In step S2001, the mask generation unit 111 sets a variable i indicating the position of the pixel to 0. Hereinafter, the i-th pixel refers to one pixel in the mask image. The processing after step S2001 may be sequentially processed in the X direction or the Y direction of the (X, Y) coordinates.

[0162] In step S2002, the mask generation unit 111 determines whether the pixel values of all the pixels of the mask image have been calculated. If it is determined that the pixel value of any one of all the pixels of the mask image has not been calculated, the process proceeds to step S2003. If it is determined that the pixel values of all the pixels of the mask image have been calculated, the processing of this flowchart ends.

[0163] In step S2003, the mask generation unit 111 determines whether the i-th pixel of the mask image is included in the mask area. In Embodiment 6, for determining whether a certain pixel is included in the mask area, metadata regarding the image circle 1021 and the blur correction value (geometric image correction information) calculated in step S502 are used.

[0164] For example, the coordinate information generation unit 110 calculates the coordinates of a circumferential fisheye image (see FIGS. 4A to 4C) corresponding to the coordinates of the orthographic cylindrical image. Then, the mask generation unit 111 can determine whether a certain pixel in the orthographic cylindrical image corresponds to a pixel included in the image circle 1021 by referring to the metadata regarding the image circle 1021 in the area of the circumferential fisheye image. Therefore, the mask generation unit 111 determines the area (the area of the mask image) having the same coordinates as the area "not corresponding to the pixels within the image circle 1021 in the orthographic cylindrical image" as the mask area when no geometric image correction such as shake correction is performed.

[0165] Thereafter, the mask generation unit 111 corrects (adjusts) the mask area (the position and size of the mask area) based on the blur correction value (geometric image correction information) calculated in step S502. For example, the mask generation unit 111 refers to the blur correction value for each frame and widens the mask area when no shake correction is performed by the amount of the blur correction value. Thus, the mask generation unit 111 calculates the mask area (the position and size of the mask area) taking into account the shake correction.

[0166] Referring to the mask area calculated in this way, in step S2003, the mask generation unit 111 determines whether the i-th pixel is included in the mask area. If it is determined that the i-th pixel is included in the mask area, the process proceeds to step S2005. If it is determined that the i-th pixel is not included in the mask area, the process proceeds to step S2004.

[0167] In step S2004, the mask generation unit 111 sets the i-th pixel in the mask image to a white pixel (sets the pixel value indicating white to the pixel value of the i-th pixel). When the area set to a white pixel in the mask image is superimposed on a certain area of the orthographic cylindrical image, the certain area is displayed through it (displayed without being blocked).

[0168] In step S2005, the mask generation unit 111 sets the i-th pixel in the mask image to a black pixel (sets the pixel value indicating black to the pixel value of the i-th pixel). When the area set to a black pixel in the mask image is superimposed on a certain area of the orthographic cylindrical image, the certain area is covered by attaching black to the certain area.

[0169] In step S2006, the mask generation unit 111 adds 1 to the variable i. As a result, for the next pixel of the mask image, the processes of steps S2002 to S2005 are performed.

[0170] In this way, in Embodiment 6, the pixels in the mask area are set to black pixels, and the pixels in the non-mask area are set to white pixels. By performing such processing, the mask generation unit 111 generates a mask image 2200 having a distribution of white and black pixels as shown in FIGS. 22 and 23A. Note that it is not limited to the distribution of white and black pixels as long as the mask area and the non-mask area can be distinguished.

[0171] As shown in FIGS. 22 and 23A, the mask image 2200 may be individually generated for each frame using the blur correction value (the correction value for shake correction) in the corresponding orthographic cylindrical image 402. The mask image 2200 may be generated based on the maximum correction value in all frames. In this case, the mask image 2200 may be applied to all frames of the orthographic cylindrical image. In this case, for example, a mask image is generated such that the invalid area does not appear in all frames of the orthographic cylindrical image. Also, the mask image 2200 applied to all frames of the orthographic cylindrical image is the same. Even when the blur correction value in the image of each frame changes, the shape of the masked peripheral part does not change. Therefore, it is possible to improve the user's immersion in the VR video while suppressing the flickering of the VR video. Note that a mask image in which a predetermined area is masked may be generated so as to prevent the invalid area from being reflected even if blur correction is performed without depending on the blur correction value.

[0172] FIG. 22 shows an example in which a gradation area is provided in the mask image 2200. FIG. 22 is an extracted view of the end portion of the mask image 2200. The mask image 2200 has a mask-free area 2201. The mask image 2200 has a gradation area 2202 and a mask-covered area 2203 as mask areas. The mask-free area 2201 is an area of white pixels. The mask-covered area 2203 is an area of black pixels. The gradation area 2202 has a luminance value intermediate between the mask-free area 2201 and the mask-covered area 2203. In Embodiment 6, The luminance value of the gradation area 2202 is obtained by linear interpolation of the luminance value of the mask-free area 2201 and the luminance value of the mask-covered area 2203. Thereby, the luminance value of the gradation area 2202 changes stepwise (continuously) from the mask-free area 2201 to the mask-covered area 2203.

[0173] For example, when the gradient region 2202 is superimposed on a certain region of the orthographic cylindrical image, the certain region is colored with the color of the gradient region 2202. Or, when the gradient region 2202 is superimposed on a certain region of the orthographic cylindrical image, the gradient region 2202 may display by transmitting through the certain region with a transmittance that is higher as its luminance value is higher. By applying the mask image having the gradient region 2202, the luminance change at the periphery of the VR video can be smoothed. Therefore, the sense of immersion in the VR video can be improved.

[0174] The gradient region 2202 smooths the luminance change at the boundary between the mask-free region 2201 and the masked region 2203. In Embodiment 6, the start position of the gradient region 2202 (= the position of the boundary between the gradient region 2202 and the mask-free region 2201) is determined based on the blur correction value. The mask image having the gradient region 2202 may be generated for each frame based on the blur correction value of each frame. The start position of the gradient region 2202 may be determined based on the maximum value of the blur correction values of all frames.

[0175] With reference to FIGS. 23A and 23B, the start positions of the gradient region 2202 and the masked region 2203 will be described.

[0176] FIG. 23A is a diagram excerpting pixels corresponding to one circumferential fisheye region from the mask image 2200. FIG. 23B is an enlarged view of a part of the mask image 2200 in FIG. 23A. The mask image 2200 has a mask-free region 2201, a gradation region 2202, and a mask-present region 2203. The gradation start position 2301 indicates the start position of the gradation region 2202 when correction by shake correction is not performed. In FIG. 23B, the gradation start position 2301 is the position where the gradation region 2202 starts closer to the center of the mask image 2200. The start position 2302 is the position where the mask-present region 2203 starts closer to the center of the mask image 2200. The gradation start position 2304 is the start position of the gradation region 2202 when correction by shake correction is performed.

[0177] The start position variation amount 2303, which is the difference between the gradation start position 2301 and the gradation start position 2304, corresponds to the amount of displacement of the relative position between the image circle 4011 and the region 4012 due to shake correction. That is, the gradation start position 2304 is shifted in the center direction of the mask image 2200 (the direction where the mask-free region 2201 exists) by the number of pixels corresponding to the blur correction value compared to the gradation start position 2301. In this way, by taking into account the information of the blur correction value for the start position of the gradation region 2202, the luminance change in the peripheral part can be smoothed. For this reason, it becomes possible to more naturally hide reflections and the like that the photographer does not intend in the mask image. The gradation region is determined based on, for example, white pixels and black pixels set by the mask image generation process (flowchart in FIG. 20) in step S1902. Only a region of pixels with a predetermined width in the region of black pixels may be determined as the gradation region. Also, only a region of pixels with a predetermined width in the region of white pixels may be determined as the gradation region. Further, a region of pixels with a predetermined width including both white pixels and black pixels may be determined as the gradation region.

[0178] FIG. 24 shows an example of a mask image 2200. The mask image 2200 corresponds to a circumferential fisheye image 403 (see FIG. 4B) having two image circles. The mask-free region 2201 is a region of white pixels. The mask-containing region 2203 is a region of black pixels. Between the mask-free region 2 201 and the mask-containing region 2203, a gradation region 2202 is set. The mask image 2200 has two mask-free regions 2201 corresponding to the regions of the circumferential fisheye image. The mask-containing region 2203 is arranged so as to surround the peripheral portion of the mask-free region 2201.

[0179] In the mask image 2200 shown in FIG. 24, there are two mask-free regions 2201 separated by the mask-containing region 2203. The two mask-free regions 2201 correspond to two image circles 1021 (regions of two circumferential fisheye images) in the circumferential fisheye image 403. As shown in FIG. 24, the mask image 2200 has a first mask region 2204 and a second mask region 2205. The first mask region 2204 includes a range corresponding to an invalid region outside the image circle.

[0180] By the way, when shooting a video including two image circles, for example, when using a fisheye lens with an angle of view exceeding 180°, as described with reference to FIGS. 10A and 10B, the other optical system may be reflected in the video shot by one optical system. There is a concern that the unintended reflection may reduce the sense of immersion when the viewer watches the VR video. Therefore, in Embodiment 6, the mask image 2200 has a second mask region 2205. The second mask region 2205 includes a range corresponding to the images 1023 and 1024, which are the reflection regions of the optical system.

[0181] Note that when the user watches a VR video, the videos captured by the two optical systems are viewed with each eye through an HMD. At this time, if the shape of the mask area is different between the left and right, there will be an area that can only be seen by one eye, and there is a concern that a sense of discomfort will occur in the video. In Embodiment 6, in order to make the shape of the mask area the same between the left and right, the second mask area 2205 is configured to have a symmetric shape in each of the left and right videos.

[0182] As described above, in Embodiment 6, an example of using the blur correction value as the image correction information used for generating the mask image has been shown. Here, in a VR video, it is important that the parallax between the left-eye image and the right-eye image is appropriately set in order to experience an appropriate sense of three-dimensionality. For this reason, parallax correction for adjusting the positions of the left and right images may be performed. Also, when there is an unintended tilt in the video in a VR video, there is a concern that the immersion of the viewer will be impaired. To prevent this, horizontal correction for adjusting the tilt of the video may be performed. Therefore, the geometric correction for the orthographic cylindrical image may be either horizontal correction or parallax correction, not limited to shake correction. And the image correction information may be a correction value for either horizontal correction or parallax correction.

[0183] Also, regarding shake correction, image correction information may be calculated using sensor output (sensor information) such as an acceleration sensor mounted on the imaging device as metadata. Also, information on feature points of each detected frame may be used as metadata. Also, the metadata may be information that is a combination of sensor output and information on feature points. Regarding horizontal correction, for example, the output of a sensor included in the imaging device such as a gyro sensor may be used as metadata. Regarding parallax correction, information on the deviation of the actual value of the optical system from the design value may be used as metadata. Also, information on the relative position of two image circles calculated from an image may be used as metadata. Note that in Embodiment 6, the mask image was generated so as to be applied to the orthographic cylindrical image, but the present invention is not limited to this. For example, a mask image may be generated so as to be applied to a circumferential fisheye image, and after applying the mask image to the circumferential fisheye image, an orthographic cylindrical image may be obtained.

[0184] <Embodiment 7> In Embodiment 7, the processing when an error region (a region where pixel values corresponding to an invalid region are set even though they should correspond to the valid region) occurs when obtaining an orthographic cylindrical image from a fisheye circumferential image will be described with reference to FIG. 25. The flowchart of FIG. 25 shows the processing in Embodiment 7. Note that steps in which the same processing as in each step of the flowchart of FIG. 5 or the flowchart of FIG. 19 is performed are given the same numbers, and the description thereof is omitted. In step S2501, the control unit 101 generates an orthographic cylindrical image for each of all frames, and outputs the generated orthographic cylindrical image.

[0185]

[0186] In step S2502, the mask image generated in step S1902 is applied to the orthographic cylindrical image generated in step S504 to generate a new orthographic cylindrical image.

[0187] Here, in step S2501 described above, the processing of the flowchart of FIG. 9 is performed in the same manner as in step S504.​

[0188] In step S902 of the flowchart in FIG. 9, the control unit 101 acquires the pixel values of each coordinate (X, Y) in the region of the circumferential fisheye image. At this time, the control unit 101 obtains, for example, the coordinates of the circumferential fisheye image corresponding to each pixel in the orthographic cylindrical image, and acquires the pixel value at the pixel. Here, in order to reduce the arithmetic processing in the control unit 101, interpolation processing may be performed when calculating the pixels of the circumferential fisheye video corresponding to the pixels of the orthographic cylindrical image.

[0189] With reference to FIGS. 26 and 27, an error region (a region where pixel values corresponding to the invalid region are set even though it corresponds to the valid region) caused by the interpolation processing will be described. FIG. 26 is an enlarged view of a part of the orthographic cylindrical image 406 when the interpolation processing is not performed. One rectangular region in FIG. 26 represents one pixel in the orthographic cylindrical image 406.

[0190] The coordinate information generation unit 110 calculates the "coordinates of the pixel in the region of the circumferential fisheye image" corresponding to a certain pixel of the orthographic cylindrical image 406. The invalid region boundary line 2601 is a line obtained by projecting a line indicating the boundary between the valid region and the invalid region in the region of the circumferential fisheye image onto the region of the orthographic cylindrical image. That is, in FIG. 26, for the pixels located inside (lower right) the invalid region boundary line 2601, the pixel values of the corresponding pixels in the region of the circumferential fisheye image (valid region) are input and treated as valid pixels 2605. On the other hand, for the pixels outside (upper left) the invalid region boundary line 2601, it is determined that there is no corresponding pixel (corresponding to the invalid region), and black pixels 2604 are input.

[0191] FIG. 27 shows a part of the same orthographic cylindrical image as FIG. 26, but shows a part of the orthographic cylindrical image when interpolation processing is performed. When interpolation processing is performed, the pixels in the area of the circumferential fisheye image corresponding individually to each pixel are not obtained. In this case, taking a certain block (interpolation block 2602) as one unit, the "coordinates in the area of the circumferential fisheye image" corresponding to the center 2603 of the interpolation block 2602 are obtained. In Embodiment 7, the interpolation block 2602 is a 4-pixel × 4-pixel square. At this time, as a method for obtaining the "coordinates in the area of the circumferential fisheye image" corresponding to each pixel, a method using linear interpolation can be considered. The "X coordinate in the area of the circumferential fisheye image" corresponding to each pixel can be calculated by linear interpolation using the coordinates of two adjacent centers 2603 in the X direction. Regarding the "Y coordinate in the area of the circumferential fisheye image" corresponding to each pixel, it can be calculated by linear interpolation using the coordinates of the centers 2603 of two adjacent interpolation blocks in the Y direction. Here, although it is assumed that the interpolation block is a 4-pixel × 4-pixel square, it is not limited to this. The interpolation block may be a 3-pixel × 3-pixel square, and the "coordinates in the area of the circumferential fisheye image" corresponding to the pixel at the center of the interpolation block may be used. Note that the shape of the interpolation block is not limited to a square, and may be a rectangle composed of a plurality of pixels.

[0192] Also, when it is determined that the center 2603 corresponds to the invalid area of the circumferential fisheye image, the corresponding coordinates of the center 2603 cannot be obtained. Also, the "coordinates in the area of the circumferential fisheye image" corresponding to each pixel are obtained by linear interpolation using the coordinates of two adjacent centers 2603. For this reason, black pixels 2604 as invalid pixels are input to the pixels located between the "center 2603 determined to correspond to the invalid area" and the "center 2603 adjacent thereto".

[0193] In FIG. 27, compared with FIG. 26, the number of pixels into which the black pixel 2604 is input has increased by performing interpolation processing. Further, in FIG. 27, the shape of the boundary portion between the black pixel 2604 and the valid pixel 2605 is also rough, which inhibits the sense of immersion when the viewer watches the VR video. Therefore, in Embodiment 7, the masked area 2203 is generated so as to be able to cover "the error area 2606 which is an area where an invalid pixel value is input despite corresponding to the valid pixel". More specifically, the mask generation unit 111 calculates in advance the error area 2606 generated by the interpolation processing, and generates the masked area 2203 wider by the width of the error area 2606 when generating the mask image 2200.

[0194] For this reason, in Embodiment 7, since the area generated by the calculation error in the orthographic cylindrical image can be hidden by the mask image, it is possible to prevent the sense of immersion when the viewer watches the VR video from being impaired.

[0195] Note that, in Embodiment 7, an example in which the mask image is generated after generating the orthographic cylindrical image from the result of calculating the invalid area is shown. As the order of generation, the generation of the mask image may be earlier, or the generation of the orthographic cylindrical image and the generation of the mask image may be processed simultaneously in parallel.

[0196] Also, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than B, proceed to step S2" may be read as "If A is greater than B, proceed to step S1; if A is less than or equal to B, proceed to step S2". Conversely, "If A is greater than B, proceed to step S1; if A is less than or equal to B, proceed to step S2" may be read as "If A is greater than or equal to B, proceed to step S1; if A is less than B, proceed to step S2". Therefore, as long as there is no contradiction, "A or more" may be read as "greater than (higher; longer; more) than A", and "A or less" may be read as "less than (lower; shorter; less) than A". And "greater than (higher; longer; more) than A" may be read as "A or more", and "less than (lower; shorter; less) than A" may be read as "A or less".

[0197] Note that each of the above various controls may be performed by one piece of hardware (for example, a processor or a circuit), or not. The overall control of the device may be performed by a plurality of pieces of hardware (for example, a plurality of processors, a plurality of circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0198] Also, the above processor is a processor in a broad sense and includes a general-purpose processor and a dedicated processor. General-purpose processors are, for example, CPU (Central Processing Unit), MPU (Micro Processing Unit), DSP (Digital Signal Processor), etc. Dedicated processors are, for example, GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), etc. Programmable logic devices are, for example, FPGA (Field Programmable Gate Array), CPLD (Complex Programmable Logic Device), etc.

[0199] Also, although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of this invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine the embodiments.

[0200] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit that realizes one or more functions.

[0201] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An electronic device that converts a first video of a first display method, which is a first video including a shooting area representing the result of shooting by an imaging device, into a second video that is a video of a second display method having a plurality of frames, an acquisition means for acquiring the first video; a setting means for setting invalid areas having the same shape at the same positions in the plurality of frames of the second video; a generation means for generating the second video by converting the first video so that pixel values indicating invalidity are set in the invalid areas for each of the plurality of frames of the second video; An electronic device characterized by having the above. (Configuration 2) The setting means determines invalid coordinates in the second video that do not correspond to the valid area of the first video for each frame, and sets a set of coordinates determined to be the invalid coordinates in at least any one of the plurality of frames as the invalid area. The electronic device according to Configuration 1, characterized by the above. (Configuration 3) The effective area is the imaging area, The electronic device according to Configuration 2, characterized in that. (Configuration 4) The effective area is an area that does not include a pre-specified area within the imaging area, The electronic device according to Configuration 3, characterized in that. (Configuration 5) The setting means sets, as the invalid area, an area corresponding to the blur of the imaging device when the first video is being captured, The electronic device according to Configuration 1, characterized in that. (Configuration 6) The setting means sets, as the invalid area, an area corresponding to the blur in each of the four directions of the imaging device when the first video is being captured, The electronic device according to Configuration 5, characterized in that. (Configuration 7) The generation means generates the second video that has been corrected so that the influence of hand shake in the shooting of the imaging device is reduced, The electronic device according to any one of Configurations 1 to 6, characterized in that. (Configuration 8) The generation means generates the second video that has been corrected based on the result of image processing on the first video, The electronic device according to Configuration 7, characterized in that. (Configuration 9) The acquisition means acquires sensor data related to the imaging device when the first video is being captured, together with the first video, The generation means generates the second video that has been corrected based on the sensor data, The electronic device according to Configuration 7 or 8, characterized in that. (Configuration 10) The pixel value indicating invalidity is a pixel value representing black, The electronic device according to any one of Configurations 1 to 9, characterized in that. (Configuration 11) If the information on the invalid area calculated in the past is stored in a specific file and the information on the invalid area stored in the specific file is calculated according to the same settings as the current settings, the generating means generates the second video based on the information on the invalid area stored in the specific file. The electronic device according to any one of Configurations 1 to 10, characterized by this. (Configuration 12) The first video includes a plurality of video files. The electronic device according to any one of Configurations 1 to 11, characterized by this. (Configuration 13) An electronic device that converts a first video, which is a first video of a first display method and includes a shooting area representing the result of shooting by an imaging device, into a second video, which is a video of a second display method having a plurality of frames. An acquisition means for acquiring the first video and metadata related to the shooting state of the first video. An information generation means for generating correction information related to geometric correction performed at the time of generating the second video based on the metadata. A mask generation means for generating a mask image that covers a part of the area of the second video. A generation means for generating the second video converted from the first video based on the correction information, with the mask image applied to each frame. An electronic device characterized by having these. (Configuration 14) The correction information is the maximum value of the correction values of the correction for each frame of the first video. The generation means applies the same mask image to all frames of the second video. The electronic device according to Configuration 13, characterized by this. (Configuration 15) The mask image has a first area that covers the second video by attaching a specific color to the second video and a second area that is displayed without covering the second video. The electronic device according to Configuration 13 or 14, characterized by this. (Configuration 16) The mask image has a third region between the first region and the second region, in which the luminance of the color applied to the second video continuously changes from the first region to the second region. The electronic device according to Configuration 15, characterized in that. (Configuration 17) The position of the boundary between the third region and the second region is a position based on the correction information. , The electronic device according to Configuration 16, characterized in that. (Configuration 18) When the second video includes a specific region that, although being a region corresponding to the imaging region, does not have pixel values corresponding to the imaging region, the mask image has a region covering the specific region. The electronic device according to any one of Configurations 13 to 17, characterized in that. (Configuration 19) The first video has regions of at least two circular fisheye images. The second video has orthographic cylindrical projection images corresponding to each of the at least two circular fisheye images. The mask image has a region covering the peripheral portion of the orthographic cylindrical projection image. The electronic device according to any one of Configurations 13 to 18, characterized in that. (Configuration 20) The first video has a region in which the optical system for acquiring the imaging region is reflected. The mask image has a region covering the region corresponding to the region in which the optical system is reflected in the second video. The electronic device according to any one of Configurations 13 to 19, characterized in that. (Configuration 21) The correction includes any one of shake correction, horizontal correction, and parallax correction. The electronic device according to any one of Configurations 13 to 20, characterized in that. (Configuration 22) The metadata includes at least any one of information regarding design values of an optical system that acquires the imaging region, information regarding a deviation of an actual value of the optical system from the design value, time-series information regarding the orientation of the optical system, and information regarding a relative position of at least two image circles when the imaging region has at least two image circles. The electronic device according to any one of Configurations 13 to 21, characterized in that. (Configuration 23) The generation means generates the second moving image by acquiring a pixel value of a coordinate of the first moving image corresponding to each coordinate of the second moving image based on the correction information. The electronic device according to any one of Configurations 13 to 22, characterized in that. (Configuration 24) The mask generation means generates the mask image that covers a partial region of the second moving image based on the correction information. The electronic device according to any one of Configurations 13 to 23, characterized in that. (Configuration 25) After the generation means generates the second moving image by converting the first moving image based on the correction information, the generation means covers a partial region of the second moving image by applying the mask image to each frame of the second moving image. The electronic device according to any one of Configurations 13 to 24, characterized in that. (Method 1) A control method for an electronic device that converts a first moving image of a first display format, the first moving image including an imaging region representing a result of imaging by an imaging device, into a second moving image of a second display format having a plurality of frames, An acquisition step of acquiring the first moving image; A setting step of setting an invalid region having the same shape at the same position in the plurality of frames of the second moving image; A generation step of generating the second moving image by converting the first moving image so as to set a pixel value indicating invalidity in the invalid region for each of the plurality of frames of the second moving image. A control method for an electronic device, characterized by having (Method 2) A control method for an electronic device that converts a first video of a first display format, which includes a shooting area representing the result of shooting by an imaging device, into a second video that is a video of a second display format having a plurality of frames, An acquisition step of acquiring the first video and metadata related to the shooting state of the first video; An information generation step of generating correction information related to geometric correction performed during generation of the second video based on the metadata; A mask generation step of generating a mask image that covers a partial area of the second video; A generation step of generating the second video converted from the first video based on the correction information, with the mask image applied to each frame; A control method for an electronic device, characterized by having (Program) A program for causing a computer to function as each means of the electronic device according to any one of Configurations 1 to 25.

Explanation of Signs

[0202] 100: Editing device (electronic device), 101: Control unit

Claims

1. An electronic device that converts a first video of a first display format, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display format having a plurality of frames, an acquisition means for acquiring the first video; a setting means for setting invalid areas of the same shape at the same position in the plurality of frames of the second video; a generation means for generating the second video by converting the first video so as to set pixel values indicating invalidity in the invalid areas for each of the plurality of frames of the second video; An electronic device characterized by comprising.

2. The setting means, for each frame, determines invalid coordinates in the second video that do not correspond to the valid area of the first video, and sets a set of coordinates determined to be the invalid coordinates in at least any one of the plurality of frames as the invalid area. The electronic device according to claim 1, characterized in that.

3. The valid area is the shooting area. The electronic device according to claim 2, characterized in that.

4. The valid area is an area that does not include a pre-specified area within the shooting area. The electronic device according to claim 3, characterized in that.

5. The setting means sets an area corresponding to the blur of the imaging device when shooting the first video as the invalid area. The electronic device according to claim 1, characterized in that.

6. The setting means sets areas corresponding to the blur in each of the four directions of the imaging device when shooting the first video as the invalid areas. The electronic device according to claim 5, characterized in that.

7. The generation means generates the second video that has been corrected so that the influence of camera shake in the shooting of the imaging device is reduced. The electronic device according to claim 1, characterized in that.

8. The generation means generates the second video that has been corrected based on the result of image processing on the first video. The electronic device according to claim 7, characterized in that.

9. The acquisition means acquires sensor data related to the imaging device when shooting the first video, together with the first video. The generation means generates the second video that has been corrected based on the sensor data. The electronic device according to claim 7, characterized in that.

10. The pixel value indicating the invalidity is a pixel value representing black. The electronic device according to claim 1, characterized in that.

11. When the information on the invalid area calculated in the past is stored in a specific file by the generation means, and the information on the invalid area stored in the specific file is calculated according to the same setting as the current setting, the second moving image is generated based on the information on the invalid area stored in the specific file. The electronic device according to claim 1, characterized in that.

12. The first moving image includes a plurality of moving image files. The electronic device according to claim 1, characterized in that.

13. An electronic device that converts a first moving image in a first display format, which includes a shooting area representing the result of shooting by an imaging device, into a second moving image in a second display format having a plurality of frames, An acquisition means for acquiring the first moving image and metadata related to the shooting state of the first moving image; An information generation means for generating correction information regarding geometric correction performed at the time of generating the second moving image based on the metadata; A mask generation means for generating a mask image covering a part of the area of the second moving image; A generation means for generating the second moving image converted from the first moving image based on the correction information, with the mask image applied to each frame. An electronic device characterized by comprising:

14. The correction information is the maximum value of the correction values of the correction for each frame of the first moving image, The generation means applies the same mask image to all frames of the second moving image. The electronic device according to claim 13, characterized in that.

15. The mask image has a first area that covers the second moving image by attaching a specific color to the second moving image, and a second area that is displayed without covering the second moving image. The electronic device according to claim 13, characterized in that.

16. The mask image has a third area between the first area and the second area, where the luminance of the color applied to the second moving image continuously changes from the first area to the second area. The electronic device according to claim 15, characterized in that.

17. The position of the boundary between the third area and the second area is a position based on the correction information. The electronic device according to claim 16, characterized in that.

18. If the second video includes a specific area that, although being an area corresponding to the imaging area, does not have pixel values corresponding to the imaging area, the mask image has an area covering the specific area. The electronic device according to claim 13, characterized in that.

19. The first video has areas of at least two circular fisheye images. The second video has an orthographic cylindrical projection image corresponding to each of the at least two circular fisheye images. The mask image has an area covering the peripheral portion of the orthographic cylindrical projection image. The electronic device according to claim 13, characterized in that.

20. The first video has an area in which the optical system for acquiring the imaging area is reflected. The mask image has an area covering the area corresponding to the area in which the optical system is reflected in the second video. The electronic device according to claim 13, characterized in that.

21. The correction includes any one of shake correction, horizontal correction, and parallax correction. The electronic device according to claim 13, characterized in that.

22. The metadata includes at least any one of information regarding the design values of the optical system for acquiring the imaging area, information regarding the deviation of the actual values of the optical system from the design values, time-series information regarding the attitude of the optical system, and information regarding the relative positions of the at least two image circles when the imaging area has at least two image circles. The electronic device according to claim 13, characterized in that.

23. The generation means generates the second video by acquiring the pixel values of the coordinates of the first video corresponding to the respective coordinates of the second video based on the correction information. The electronic device according to claim 13, characterized in that.

24. The mask generation means generates the mask image covering a part of the area of the second video based on the correction information. The electronic device according to claim 13, characterized in that.

25. After generating the second video by converting the first video based on the correction information, the generation means covers a part of the area of the second video by applying the mask image to each frame of the second video. The electronic device according to claim 13, characterized in that.

26. A control method for an electronic device that converts a first video of a first display format, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display format having a plurality of frames, comprising: an acquisition step of acquiring the first video; a setting step of setting invalid areas of the same shape at the same positions in the plurality of frames of the second video; a generation step of generating the second video by converting the first video so that pixel values indicating invalidity are set in the invalid areas for each of the plurality of frames of the second video; A control method for an electronic device, characterized by comprising the above steps.

27. A control method for an electronic device that converts a first video of a first display format, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display format having a plurality of frames, comprising: an acquisition step of acquiring the first video and metadata related to the shooting state of the first video; an information generation step of generating correction information related to geometric correction performed during generation of the second video based on the metadata; a mask generation step of generating a mask image that covers a partial area of the second video; a generation step of generating the second video that is converted from the first video based on the correction information with the mask image applied to each frame; A control method for an electronic device, characterized by comprising the above steps.

28. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 25.

Images