Electronic devices, methods for controlling electronic devices, programs

Abstract

Generates VR videos with minimal flickering during playback. [Solution] An electronic device that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having a plurality of frames, includes: acquisition means for acquiring the first video; setting means for setting invalid areas of the same shape at the same position in the plurality of frames of the second video; and generating means for generating the second video by converting the first video such that a pixel value indicating invalidity is set in the invalid area for each of the plurality of frames of the second video.

Description

Technical Field

[0001] The present invention relates to an electronic apparatus, a control method for an electronic apparatus, and a program.

Background Art

[0002] In general, a VR image (VR moving image) includes two images (a left-eye image and a right-eye image) that allow a user to view stereoscopically using parallax when reproduced on a head-mounted display (HMD). The VR image includes, for example, two equirectangular projection images that are results of respectively performing equirectangular conversion on circular fisheye images captured via two circular fisheye lenses.

[0003] If the two circular fisheye lenses are not properly mounted, the parallax between the two circular fisheye images may not be suitable for stereoscopic viewing. In this case, by adjusting parameters used for the equirectangular conversion that converts the circular fisheye images into equirectangular projection images, conversion into equirectangular projection images whose parallax is suitable for stereoscopic viewing can be realized. However, as a result of this parameter adjustment, there may be no valid region of the circular fisheye image corresponding to a partial region of the equirectangular projection image, and a partial region of the equirectangular projection image may be set as an invalid region having invalid pixel values.

[0004] Patent Document 1 discloses a technique of making the shapes of invalid regions of left and right images the same during equirectangular conversion of a VR image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of Invention

Problem to be Solved by the Invention

[0006] When generating video files of VR images, one method of achieving image stabilization is to adjust the parameters used in equirectangular transformation to correct motion blur between the preceding and succeeding frames and the current frame. As mentioned above, depending on the values ​​of the parameters used in equirectangular transformation, a portion of the area of ​​the resulting equirectangular projection image may become an invalid region. Furthermore, if the parameters used in equirectangular transformation differ from frame to frame, image stabilization may cause the invalid region of the equirectangular projection image to move during video playback.

[0007] As a result, when image stabilization is applied, flickering occurs in the equirectangular projection image. However, the technology described in Patent Document 1 cannot suppress this flickering.

[0008] Therefore, the present invention aims to enable the generation of VR videos with minimal flickering during playback. [Means for solving the problem]

[0009] One aspect of the present invention is, An electronic device that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, The means for acquiring the first video, Setting means for setting invalid areas of the same shape at the same position in the plurality of frames of the second video, A generation means for generating the second video by converting the first video such that for each of the plurality of frames of the second video, a pixel value indicating invalidity is set in the invalid region, This is an electronic device characterized by having [a certain feature].

[0010] One aspect of the present invention is, A control method for electronic equipment that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, The acquisition step of obtaining the first video, A setting step of setting invalid areas of the same shape at the same position in the multiple frames of the second video, A generation step of generating the second video by converting the first video such that, for each of the plurality of frames of the second video, a pixel value indicating invalidity is set in the invalid region, This is a control method for electronic equipment characterized by having [a certain feature]. [Effects of the Invention]

[0011] According to the present invention, it becomes possible to generate VR videos with minimal flickering during playback. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing the configuration of the editing device according to the embodiment. [Figure 2] This is a flowchart illustrating the operation of the camera according to the embodiment. [Figure 3] This figure shows the structure of a video file according to the embodiment. [Figure 4] This figure illustrates the circular fisheye image and equirectangular image according to the embodiment. [Figure 5] This is a flowchart of the VR video generation process according to Embodiment 1. [Figure 6] This is a flowchart of the process for calculating the blur correction value according to Embodiment 1. [Figure 7] This is a flowchart of the invalid area calculation process according to Embodiment 1. [Figure 8] This is a flowchart of the process for generating an equirectangular image according to Embodiment 1. [Figure 9] This is a flowchart of the pixel value setting process according to Embodiment 1. [Figure 10]This is a diagram illustrating a fisheye lens and a circumferential fisheye image according to Embodiment 2. [Figure 11] This is a flowchart of invalid region calculation processing according to Embodiment 2. [Figure 12] This is a flowchart of blur correction value calculation processing according to Embodiment 3. [Figure 13] This is a flowchart of invalid region calculation processing according to Embodiment 3. [Figure 14] This is a diagram illustrating a mask image according to Embodiment 3. [Figure 15] This is a diagram illustrating a mask image according to Embodiment 3. [Figure 16] This is a flowchart of VR moving image generation processing according to Embodiment 4. [Figure 17] This is a flowchart of VR moving image generation processing according to Embodiment 5. [Figure 18] This is a configuration diagram of a control unit according to Embodiment 6. [Figure 19] This is a flowchart of equirectangular image generation processing according to Embodiment 6. [Figure 20] This is a flowchart of mask image generation processing according to Embodiment 6. [Figure 21] This is a flowchart of equirectangular image generation processing according to Embodiment 6. [Figure 22] This is a diagram illustrating a gradation region according to Embodiment 6. [Figure 23] This is a diagram illustrating a gradation region according to Embodiment 6. [Figure 24] This is a diagram illustrating the configuration of a mask image according to Embodiment 6. [Figure 25] This is a flowchart of equirectangular image generation processing according to Embodiment 7. [Figure 26] This is a diagram illustrating an error region according to Embodiment 7. [Figure 27] This is a diagram illustrating an error region according to Embodiment 7. [Mode for Carrying Out the Invention]

[0013] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0014] Figure 1 shows an example of the configuration of the editing device 100, which is an image display device (video editing device) common to all embodiments. The editing device 100 is an electronic device such as a computer, smartphone, or imaging device (digital camera, etc.).

[0015] The editing device 100 includes a control unit 101, a ROM 102, a RAM 103, an external storage device 104, an operation unit 105, a display unit 106, a communication unit 107, and a system bus 108.

[0016] The control unit 101 controls the entire editing device 100. The control unit 101 is, for example, a Central Processing Unit (CPU). The control unit 101 can also perform image generation, setting, and calculation processing. For this reason, in the following, the control unit 101 can also operate as a "generation unit," a "setting unit," and a "coordinate calculation unit."

[0017] ROM102 is a Read Only Memory (ROM) that stores information (programs and parameters) that does not require modification.

[0018] RAM103 is Random Access Memory (RAM) that temporarily stores information (programs and data) supplied from external devices.

[0019] The external storage device 104 is fixedly installed on the editing device 100. The external storage device 104 is a hard disk or flash memory. Alternatively, the external storage device 104 has an optical disk such as a floppy disk (FD) or Compact Disk (CD) that can be removed from the editing device 100. The external storage device 104 may also have a magnetic card, optical card, IC card, or memory card.

[0020] The control unit 105 receives instructions in response to user operations. The control unit 105 has user-operable operating elements (buttons or touch panel).

[0021] The display unit 106 displays the data held by the editing device 100 and the data supplied to the editing device 100.

[0022] The communication unit 107 communicates with external devices such as cameras (imaging devices; digital cameras).

[0023] The system bus 108 connects each component (unit) in a way that enables communication between them.

[0024] The video file, including image data and metadata, is stored (recorded) in the external storage device 104. The video file is also readable from the external storage device 104. The communication unit 107 can also receive video files from external devices such as cameras. Therefore, in the following, the terms "data input unit" or "data acquisition unit" refer collectively to the external storage device 104 and the communication unit 107.

[0025] Referring to the flowchart in Figure 2, the operation of the camera (imaging device) that can communicate with the editing device 100 during shooting will be explained.

[0026] In step S201, the camera determines whether it is possible to store (record) the sensor data being captured in a video file. Here, sensor data refers to inertial data such as acceleration measured by a gyro sensor. The camera can record the movement of the camera during shooting by periodically storing the inertial data from the gyro sensor in a video file. If it is determined that the sensor data can be stored, the process proceeds to step S202. If it is determined that the data cannot be stored, the process proceeds to step S203.

[0027] In step S202, the camera stores the sensor data as timed metadata (metadata that holds information in a time series) in the video file.

[0028] In step S203, the camera determines whether or not to store audio (sound) in the video file. Depending on the shooting mode, the control unit 101 may choose not to store the audio in the video file. If it is determined that the audio should be stored, the process proceeds to step S204. If it is determined that the audio should not be stored, the process proceeds to step S205.

[0029] In step S204, the camera stores the audio data (sound data) as audio sampling data in the video file.

[0030] In step S205, the camera stores one frame of the image as video sampling data in a video file.

[0031] In step S206, the camera determines whether or not it has been instructed to stop shooting. If it is determined that it has been instructed to stop shooting, the process in this flowchart ends. If it is determined that it has not been instructed to stop shooting, the process returns to step S201.

[0032] Figure 3 shows the structure of a video file acquired by the camera. The video file includes a video header area 301, a file metadata area 302, and a sampling data area 303.

[0033] The video header area 301 holds information about the entire video (hereinafter referred to as "video-related information"). For example, video-related information includes at least one of the following: the image size (width and height) of the video, the frame rate, the sample frequency of the audio, the sample size, and the number of channels. Video-related information may also include information such as the position (playback position; playback time) and size of the samples (in the case of video, data for one frame) of each medium (video, audio, timed metadata, etc.).

[0034] The file metadata area 302 holds metadata about the file itself and the entire video. For example, the file metadata area 302 stores metadata such as information about the lens used during shooting, design values ​​(such as the center position or radius of the image circle), or measurements taken through calibration.

[0035] The sampling data area 303 stores sampling data for each medium. The sampling data area 303 includes a video sampling area 3031, an audio sampling area 3032, and a timed metadata sampling area 3033.

[0036] The video sampling area 3031 stores the sampling data of the video media. The sampling data of the video media is the image data stored in step S205 in Figure 2. For example, if a VR video is acquired by shooting with a camera equipped with two fisheye lenses, the video sampling area 3031 stores two circular fisheye images as one frame of video sample.

[0037] The audio sampling area 3032 stores the sampling data of the audio media. The sampling data of the audio media is the audio data stored in step S204 in Figure 2. If there is no need to record audio data, the audio sampling area 3032 may not exist.

[0038] The timed metadata sampling area 3033 stores the sampled data from the timed metadata media. The sampled data from the timed metadata media is, for example, the sensor data stored in step S202 in Figure 2. If there is no need to record timed metadata, the timed metadata sampling area 3033 may not exist.

[0039] Figures 4A to 4C show the circular fisheye image recorded by the camera and the equirectangular projection image (hereinafter referred to as "equistratic image") resulting from the conversion of the circular fisheye image by the editing device 100. Below, with reference to Figures 4A and 4B, the equirectangular image with image stabilization will be described, and with reference to Figure 4C, the equirectangular image with image stabilization according to each embodiment described later will be described. For simplicity, we will describe one circular fisheye image recorded by a camera equipped with one fisheye lens and one equirectangular image generated from that circular fisheye image. However, the same characteristics as described below also occur for two circular fisheye images recorded by a camera equipped with two fisheye lenses and two equirectangular images generated from the two circular fisheye images.

[0040] (Figure 4A) Circumferential fisheye image 401 is an example of a circular fisheye image recorded by a camera equipped with a fisheye lens.

[0041] Image circle 4011 is the area (shooting area) where the subject is captured by the camera through the fisheye lens. The subject is recorded inside image circle 4011. Outside image circle 4011 is an invalid area where no image is captured, and black or a color close to black is recorded. The invalid area is an area that does not affect the result of the shooting. In a circular fisheye image, the position information of image circle 4011 (information such as the center position and radius) is stored as file metadata in the file metadata area 302. Normally, the position of image circle 4011 is constant because the position of the lens does not change during shooting.

[0042] Region 4012 is the region subject to the equirectangular transformation. The area inside region 4012 is unfolded into an equirectangular image. Information representing the size of region 4012 is also stored as file metadata in the file metadata region 302. By default, the position of region 4012 has the same center as the center of the image circle 4011, but this can be changed.

[0043] The equirectangular image 402 is the equirectangular image obtained by performing an equirectangular transformation on region 4012 of the image circle 4011.

[0044] (Figure 4B) Circular fisheye image 403, like circular fisheye image 401, is an example of a circular fisheye image recorded by a camera.

[0045] Image circle 4031 is an image circle, just like image circle 4011. Region 4032 is a region subject to equirectangular transformation, just like region 4012.

[0046] In circular fisheye image 403, the subject is photographed slightly to the right of the center of the image circle 4031, due to the camera being oriented slightly to the left compared to circular fisheye image 401.

[0047] The equirectangular image 404 is obtained by performing an equirectangular transformation on region 4032 of the circular fisheye image 403. This is the resulting equirectangular image. In the circular fisheye image 403, the position of region 4032 relative to the image circle 4031 is the same as the position of region 4012 relative to the image circle 4011 in the circular fisheye image 401. Therefore, in the equirectangular image 404, the subject is positioned slightly to the right of the center of the image.

[0048] (Figure 4C) Circumferential fisheye image 405, like circular fisheye images 401 and 403, is an example of a circular fisheye image recorded by a camera.

[0049] Image circle 4051 is an image circle, just like image circle 4011 and image circle 4031.

[0050] Region 4052, like regions 4012 and 4032, is a region subject to equirectangular transformation. The position of region 4052 is shifted slightly to the right compared to the positions of regions 4012 and 4032.

[0051] The equirectangular image 406 is an equirectangular image obtained by performing an equirectangular transformation on region 4052 of the circular fisheye image 405. Since region 4052 is located to the right of the image circle 4051, the subject is centered in the equirectangular image. Thus, in each of the following embodiments, when generating an equirectangular image from a circular fisheye image, the region targeted for equirectangular transformation of the circular fisheye image is changed. This reduces the effect of camera shake in the equirectangular image.

[0052] On the other hand, because a portion of the right side of region 4052 extends beyond the outside of the image circle 4051 (= invalid region), a portion of the right side of the equirectangular image 406, which is the result of the equirectangular transformation, refers to the invalid region of the circular fisheye image 405. Since there are no valid pixel values ​​of the circular fisheye image 405 in the invalid region, the equirectangular image 406 contains pixels with pixel values ​​that indicate the invalid region (for example, black pixels).

[0053] <Embodiment 1> Referring to the flowchart in Figure 5, the process for generating a VR video (VR180 video) which is an equirectangular image will be explained. In Embodiment 1, a VR video is generated from a video which is a circular fisheye image, which has a different display method than an equirectangular image. Embodiment 1 is applicable not only when generating an equirectangular image from a circular fisheye image, but also when generating an image with a different display method from an image with a certain display method.

[0054] In step S501, the control unit 101 reads common metadata (metadata common to all frames) stored in the file metadata area 302 of the video file through the input unit. For example, the common metadata includes information such as the center position of the image circle, the radius of the image circle, and the radius of the target area for equirectangular transformation. The control unit 101 also reads parameter values ​​(manual correction parameter values) set by the user using the operation unit 105 as metadata. This metadata is used in a later process when generating an equirectangular image from a circular fisheye image.

[0055] In step S502, the control unit 101 calculates a blur correction value for each frame. The process in step S502 will be explained in detail with reference to the flowchart in Figure 6. The blur correction value is a value used to correct the positional shift in the equirectangular image caused by camera shake.

[0056] In step S503, the control unit 101 calculates the invalid region of the equirectangular image for all frames. Dispense. The process in step S503 will be explained in detail with reference to the flowchart in Figure 7.

[0057] In step S504, the control unit 101 generates an equirectangular image for each frame and outputs the generated equirectangular image. The process in step S504 will be explained in detail with reference to the flowchart in Figure 8.

[0058] (Processing in step S502) The process for calculating the blur correction value will be explained with reference to the flowchart in Figure 6. In the following, the control unit 101 calculates the blur correction value for each frame. By using the blur correction value when generating an equirectangular image from a circular fisheye image, it is possible to correct the effect of blur (hand shake; camera shake) in the equirectangular image for each frame.

[0059] In step S601, the control unit 101 sets both the variable i, which represents the frame number of the image data to be read, and the variable n, which represents the frame number of the image data for which the blur correction value is calculated, to 0.

[0060] In step S602, the control unit 101 determines whether it has read all the image data (circumferential fisheye image data) for all frames stored in the video file. If it is determined that all the image data for all frames has been read, the process proceeds to step S612. If it is determined that at least one frame's image data has not been read, the process proceeds to step S603.

[0061] In step S603, the control unit 101 reads the circular fisheye image data of frame number i (=frame i) from the video sampling area 3031 of the video file. Here, if the image data stored in the video sampling area 3031 is compressed, the control unit 101 decodes the image data to obtain RGB format image data.

[0062] In step S604, the control unit 101 analyzes the read image data (performs image processing on the image data) and extracts feature points from the circular fisheye image.

[0063] In step S605, the control unit 101 determines whether or not sensor data for frame i is stored in the timed metadata sampling area 3033 of the video file. If it is determined that sensor data for frame i is stored, the process proceeds to step S606. If it is determined that sensor data for frame i is not stored, the process proceeds to step S607.

[0064] In step S606, the control unit 101 reads sensor data for frame i (sensor data relating to the camera's movement when capturing the circular fisheye image of frame i) from the timed metadata sampling area 3033.

[0065] In step S607, the control unit 101 adds 1 to the variable i, which represents the read frame number.

[0066] In step S608, the control unit 101 determines whether it has read the image data of the number of frames necessary to calculate the amount of blur for frame number n (=frame n). For example, the amount of blur is the "amount of camera movement, or the amount of movement of the subject in the image data" caused by camera shake during a specific period centered on the time of one frame. For example, in order to calculate the amount of blur for one frame, the frames for one second before and after that frame are read. Image data is required. If it is determined that the required number of frames of image data have been loaded, proceed to step S609. If it is determined that the required number of frames of image data have not been loaded, return to step S602.

[0067] In step S609, the control unit 101 calculates the amount of blur in frame n. For example, the control unit 101 calculates the amount of blur in frame n based on the "feature point information and sensor data" of frame n and the frames before and after it. At this time, the control unit 101 may calculate the amount of blur based only on the feature point information, or based only on the sensor data.

[0068] Camera movement caused by unintended camera shake differs from camera movement intended for camera work. For example, when camera shake occurs, the camera moves back and forth, such as left and right or up and down, at short intervals. Therefore, the control unit 101 can calculate the amount of shake based on the movement pattern of feature points between multiple frames, or the values ​​of the gyro sensor mounted on the camera.

[0069] In step S610, the control unit 101 calculates a shake correction value to correct the shake amount of frame n calculated in step S609.

[0070] In step S611, the control unit 101 adds 1 to the variable n.

[0071] As mentioned above, in order to calculate the amount of image stabilization for a single frame, information from the frames before and after that frame is also necessary. Therefore, when all frames have been read, the image stabilization values ​​for all frames have not been calculated, and there are still several frames for which the image stabilization values ​​have not been calculated. Thus, in steps S612 and beyond, the image stabilization values ​​for these remaining frames are calculated.

[0072] In step S612, the control unit 101 determines whether or not it has calculated the image stabilization values ​​for all frames. If it is determined that the image stabilization values ​​have been calculated for all frames, the process in this flowchart ends. If it is determined that the image stabilization values ​​have not been calculated for all frames, the process proceeds to step S613.

[0073] In step S613, the control unit 101 calculates the amount of blur in frame n. The process in step S613 is the same as the process in step S609.

[0074] In step S614, the control unit 101 calculates a blur correction value to compensate for the blur amount of frame n calculated in step S613. The process in step S614 is the same as the process in step S610.

[0075] In step S615, the control unit 101 adds 1 to the variable n.

[0076] (Processing in step S503) Referring to the flowchart in Figure 7, the process for calculating invalid regions (invalid regions common to all frames) in equirectangular images will be explained.

[0077] In step S701, the control unit 101 sets the variable i, which represents the frame number, to 0.

[0078] In step S702, the control unit 101 determines whether the calculation of invalid pixels in the equirectangular images of all frames has been completed. Here, an invalid pixel is a pixel to which an invalid pixel value (a specific pixel value) will be set in a later process. If it is determined that the calculation of invalid pixels in the equirectangular images of all frames has been completed, the processing in this flowchart ends. If it is determined that the calculation of at least one invalid pixel in the equirectangular image of the frame has not been completed, the process proceeds to step S703.

[0079] In step S703, the control unit 101 acquires the blur correction value for frame i. The blur correction value is used to correct the position of the target region for equirectangular transformation in the circular fisheye image 405 shown in Figure 4C.

[0080] In step S704, the control unit 101 sets the coordinates (x,y) of the pixels to be processed in the equirectangular image of frame i to (0,0). Here, the control unit 101 processes both monocular images (both left eye image and right eye image) in the equirectangular image in one loop (steps S705 to S710). In the monocular images of the VR image, the width and height are equal to each other. Therefore, if we set the number of pixels in the monocular image to j and the height of the monocular image to j rows, the coordinates of the monocular image are included in the range from the top left (0,0) to the bottom right (j-1,j-1) of the monocular image.

[0081] In step S705, the control unit 101 determines whether the processes in steps S706 to S710 have been performed for all pixels of the equirectangular image in one frame. If it is determined that the processes in steps S706 to S710 have been performed for all pixels, the process proceeds to step S711. If it is determined that the processes in steps S706 to S710 have not been performed for at least one of the pixels, the process proceeds to step S706.

[0082] In step S706, the control unit 101 determines the coordinates (xL, yL) of the circular fisheye image corresponding to the coordinates (x, y) of the pixels to be processed in the left eye image. The coordinates of the circular fisheye image corresponding to the coordinates of the equirectangular image can be calculated by performing the inverse transformation of the equirectangular transformation. At this time, the control unit 101 determines the coordinates (xL, yL) of the circular fisheye image based on the common metadata read in step S501 and the manual correction parameter values ​​set by the user. Furthermore, the control unit 101 determines the coordinates (xL, yL) of the circular fisheye image based on the blur correction value obtained in step S703, taking into account blur correction in the equirectangular image.

[0083] In step S707, the control unit 101 determines the coordinates (xR, yR) of the circular fisheye image that correspond to the coordinates (x, y) of the pixels to be processed in the right eye image. This is the same process as in step S706.

[0084] In step S708, the control unit 101 determines whether both coordinates (xL, yL) and (xR, yR) belong within the effective area. In Embodiment 1, the effective area is the image circle (shooting area). If it is determined that both coordinates are within the image circle, the process proceeds to step S710. If it is determined that at least one of the coordinates is not within the image circle, the process proceeds to step S709.

[0085] In step S709, the control unit 101 determines that a pixel in the equirectangular image to be processed is an invalid pixel and stores the coordinates (x,y) of that pixel in the invalid coordinate table. The invalid coordinate table is used in common throughout all frames of the VR video. Therefore, the invalid coordinate table contains the coordinates of pixels that have been determined to be invalid pixels in at least one frame. Thus, by controlling the invalid coordinate table in this way, the control unit 101 can set a set of coordinates (invalid coordinates) of pixels that have been determined to be invalid pixels in at least one frame as an invalid region.

[0086] In step S710, the control unit 101 updates the coordinate (x,y) to the coordinate of the next pixel. The coordinate of the next pixel is the X coordinate, which is x, plus 1. However, if the X coordinate obtained by adding 1 to x reaches the width j, then the Y coordinate, which is y, is added to return the X coordinate, which is x, to 0.

[0087] In step S711, the control unit 101 adds 1 to the variable i, which represents the frame number.

[0088] (Processing in step S504) Referring to the flowchart in Figure 8, the process of generating an equirectangular image from a circular fisheye image for all frames will be explained.

[0089] In step S801, the control unit 101 sets the variable i, which represents the frame number, to 0.

[0090] In step S802, the control unit 101 determines whether or not it has output equirectangular images for all frames that make up the VR video. If it is determined that equirectangular images for all frames have been output, the process of this flowchart ends. If it is determined that at least one equirectangular image of any of the frames has not been output, the process proceeds to step S803.

[0091] In step S803, the control unit 101 reads the image data of the circular fisheye image of frame i from the video sampling area 3031 through the input unit. If the image data stored in the video sampling area 3031 is compressed, the control unit 101 decodes the image data to convert it to RGB format.

[0092] In step S804, the control unit 101 obtains the blur correction value for frame i. The process in step S804 is the same as the process in step S703.

[0093] In step S805, the control unit 101 sets the coordinates (x,y) of the pixels to be processed in the equirectangular image to (0,0). Here, as with the flowchart in Figure 7, both the left eye image and the right eye image are processed in a single loop (steps S806 to S812). For this reason, the coordinates of one eye image are taken from the top left (0,0) to the bottom right (j-1,j-1), assuming the width of the one eye image is j pixels and the height is j rows.

[0094] In step S806, the control unit 101 determines whether the pixel values ​​of all pixels in the equirectangular image of frame i have been set. If it is determined that the pixel values ​​of all pixels in the equirectangular image of frame i have been set, the process proceeds to step S813. If it is determined that the pixel values ​​of at least one of the pixels in the equirectangular image of frame i have not been set, the process proceeds to step S807.

[0095] In step S807, the control unit 101 determines whether the coordinates (x,y) of the pixel to be processed are already registered in the invalid coordinate table (i.e., whether the pixel with coordinates (x,y) is included in the invalid region). If it is determined that the coordinates (x,y) are already registered in the invalid coordinate table, the process proceeds to step S808. If it is determined that the coordinates (x,y) are not already registered in the invalid coordinate table, the process proceeds to step S810.

[0096] In step S808, the control unit 101 sets an invalid pixel value (invalid pixel value) for the pixel value of coordinate (x,y) of the left eye image in the equirectangular image. For example, an invalid pixel value is a pixel value of a specific value, such as a pixel value representing black, a pixel value representing white, or a pixel value representing blue.

[0097] In step S809, the control unit 101 sets an invalid pixel value for the coordinate (x,y) of the right eye image in the equirectangular image. According to steps S808 and S809, invalid pixel values ​​are set for invalid regions of the same position and shape in both the left eye image and the right eye image in the equirectangular image.

[0098] In step S810, the control unit 101 sets the pixel value of the coordinate (x,y) of the left eye image in the equirectangular image based on the blur correction value. In step S811, the control unit 101 sets the pixel value of the coordinate (x,y) of the right eye image in the equirectangular image based on the blur correction value. In this way, by setting the pixel values ​​of the pixels in the equirectangular image based on the blur correction value, it becomes possible to generate an equirectangular image with reduced effects of camera shake. Therefore, even when the subject is not moving but camera shake is occurring, the position of the subject in the equirectangular image can be maintained at a constant position in multiple consecutive frames. Details of the processing in steps S810 and S811 will be described later with reference to the flowchart in Figure 9.

[0099] In step S812, the control unit 101 updates the coordinate (x,y) to the coordinate of the next pixel. The coordinate of the next pixel is the X coordinate, x, plus 1. However, if the X coordinate obtained by adding 1 to x reaches the width j, the Y coordinate, y, is added to return the X coordinate, x, to 0. The processing in steps S807 to S812 is repeated for each pixel of the equirectangular image of one frame, thereby setting the pixel value of each pixel of the equirectangular image of that frame, and as a result the equirectangular image of that frame is generated.

[0100] In step S813, the control unit 101 outputs the equirectangular image of frame number i, which was generated in steps S803 to S812. If the equirectangular image is to be saved to a file, it is output to the external storage device 104. If the equirectangular image is to be displayed, it is output to the display unit 106.

[0101] In step S814, the control unit 101 adds 1 to the variable i, which represents the frame number.

[0102] Referring to the flowchart in Figure 9, the process of setting the pixel value of one pixel in an equirectangular image will be explained.

[0103] In step S901, the control unit 101 calculates the coordinates (X,Y) of the circular fisheye image corresponding to the coordinates (x,y) of the equirectangular image, based on the blur correction value and the like. The process in step S901 is the same as the processes in steps S706 and S707 in Figure 7.

[0104] In step S902, the control unit 101 obtains the pixel value of coordinate (X,Y) in the circular fisheye image. Alternatively, the control unit 101 obtains the pixel value of coordinate (X,Y) as an interpolated pixel value based on the number of pixels surrounding the coordinate (X,Y) obtained in step S901, using a method such as bicubic interpolation.

[0105] In step S903, the control unit 101 sets the pixel values ​​acquired in step S902 as the pixel values ​​of the coordinates (x,y) in the equirectangular image.

[0106] According to Embodiment 1, even when different image stabilization is performed for each frame, and an equirectangular image is generated from a circular fisheye image, the position and shape of the invalid region in the equirectangular image can be kept the same throughout all frames. This makes it possible to suppress the occurrence of flicker caused by changes in the shape and position of the invalid region when playing back VR video (equistrectangular image).

[0107] <Embodiment 2> Figure 10A shows an example of a binocular fisheye lens according to Embodiment 2. Figure 10B shows an example of a circular fisheye image acquired by a camera equipped with a binocular fisheye lens.

[0108] In Figure 10A, the binocular fisheye lens 1010 is shown as a binocular fisheye lens viewed from above. The binocular fisheye lens 1010 is used to acquire two circular fisheye images. The binocular fisheye lens 1010 has a glass section 1011, a support 1012, a glass section 1013, a support 1014, and a support 1015.

[0109] The glass portion 1011 is the glass portion of the left-eye fisheye lens. The support 1012 is the support for the glass portion 1011 of the left-eye fisheye lens.

[0110] The glass portion 1013 is the glass portion of the fisheye lens for the right eye. The support 1014 is the support for the glass portion 1013 of the fisheye lens for the right eye. The support 1015 is the support for the two fisheye lenses, one for the left and one for the right eye.

[0111] The angle of view 1016, formed by connecting line segments AB and BC, is the angle of view of the left-eye fisheye lens. For the left-eye fisheye lens, the angle of view 1016 is 180 degrees. The angle of view 1017, formed by connecting line segments DE and EF, is the angle of view of the right-eye fisheye lens. Similar to the angle of view 1016, the angle of view 1017 is 180 degrees.

[0112] Circumferential fisheye image 1020 is an example of a circular fisheye image acquired by a camera equipped with a twin-lens fisheye lens 1010.

[0113] Image circle 1021 is the image circle corresponding to glass part 1011. Image circle 1022 is the image circle corresponding to glass part 1013.

[0114] In cases like the 1010 twin-lens fisheye lens, where the two lenses are close together and the field of view of each lens is close to 180 degrees, the field of view of one lens will overlap with the field of view of the other lens.

[0115] Image 1023 is the image of glass part 1013 reflected in glass part 1011. Similarly, image 1024 is the image of glass part 1011 reflected in glass part 1013. Since these images are within the image circle, they can be subject to equirectangular transformation. Also, due to the parameter values ​​for image stabilization (parameter values ​​for equirectangular transformation), the reflection positions of these images in the equirectangular image are not constant. On the other hand, in the case of a twin-lens fisheye lens 1010, where the two fisheye lenses are in a fixed positional relationship, the reflection positions of these images in the circular fisheye image are constant.

[0116] Referring to Figure 11, the process for calculating the invalid region of the equirectangular images for all frames in Embodiment 2 will be described. This process corresponds to step S503 in the flowchart of Figure 5 relating to Embodiment 1. The processes other than step S503 are the same as the processes in each step of the flowchart of Figure 5. Also, steps S701 to S711 in the flowchart of Figure 11 are the same as the steps of the same name in the flowchart of Figure 7.

[0117] In step S1101, the control unit 101 acquires information about a pre-specified exclusion region within the image circle of the circular fisheye image. The information about the exclusion region is, for example, information about the coordinates of images (images 1023 and 1024) that are not needed as a VR image within the image circle of the circular fisheye image 1120, as shown in Figure 10B. Since the position of the lens reflection region is constant in the circular fisheye image, it is possible to calculate the coordinates of this position in advance and save them in advance in the form of a coordinate table or mask image.

[0118] In step S1110, the control unit 101 determines whether the two coordinates (xL, yL) and (xR, yR) of the circular fisheye image obtained in steps S706 and S707 are outside the exclusion area. If both coordinates are determined to be outside the exclusion area, the process proceeds to step S710. If at least one of the coordinates is determined to be within the exclusion area, the process proceeds to step S709.

[0119] According to Embodiment 2, the coordinates of the excluded region within the image circle of the circular fisheye image are treated as non-active regions, just like the region outside the image circle. In other words, the region in the equirectangular image corresponding to the excluded region is also treated as an invalid region. Therefore, unwanted images corresponding to the excluded region are not displayed in the equirectangular image, and the shape of the invalid region does not change in each frame. This suppresses flickering in VR videos and improves the user's immersion in VR videos.

[0120] <Embodiment 3> In embodiments 1 and 2, calculations are performed in the reverse direction of the equirectangular transformation to calculate the invalid region. These processes involve a large amount of computation and are likely to take a long time to process. However, in some cases, it is desirable to calculate the invalid region of an equirectangular image in a short processing time, even if the accuracy of the invalid region is low. Therefore, embodiment 3 describes an editing device 100 that can calculate the invalid region of an equirectangular image in a short processing time by calculating the invalid region according to the camera shake in four directions during shooting.

[0121] Referring to the flowchart in Figure 12, the process for calculating the blur correction value in Embodiment 3 (corresponding to step S502 in the flowchart of Figure 5 relating to Embodiment 1) will be explained.

[0122] In the flowchart of Figure 12, the processes in steps S601, S602, and S606-S615 are the same as the processes with the same names in the flowchart of Figure 6. On the other hand, in the flowchart of Figure 12, if it is determined in step S602 that no image data from any of the frames has been read, the process proceeds to step S606. In other words, the processes in steps S603-S605 (more specifically, the feature point extraction process in step S604) are not performed, thus reducing the number of processes performed by the editing device 100.

[0123] Referring to the flowchart in Figure 13, the process for calculating the invalid region of the equirectangular image for all frames (processing in step S503) in Embodiment 3 will be described. The processes in steps S701 to S703 and S711 in the flowchart of Figure 13 are the same as the processes in the steps of the same name in the flowchart of Figure 7.

[0124] In step S1305, the control unit 101 obtains the maximum image stabilization value for each of the four directions from the image stabilization values ​​acquired in step S703. The maximum image stabilization values ​​for the four directions are the maximum image stabilization value in the positive direction of the X-axis, the maximum image stabilization value in the negative direction of the X-axis, the maximum image stabilization value in the positive direction of the Y-axis, and the maximum image stabilization value in the negative direction of the Y-axis. The maximum image stabilization values ​​for the four directions are values ​​corresponding to the magnitude of camera shake in the four directions caused by hand shake.

[0125] In step S1306, the control unit 101 determines an invalid area corresponding to the maximum blur correction value in each direction, which was determined in step S1305. This can be achieved, for example, by preparing several mask images of invalid areas corresponding to the blur correction value in each direction in advance, and then selecting a mask image of an invalid area corresponding to a blur correction value equal to or greater than the maximum blur correction value in that direction.

[0126] For example, Figure 14 shows four types of mask images of invalid regions, which are prepared in advance according to the positive X-axis shake correction value. Mask image 1401 is the case when the positive X-axis shake correction value is 0. These are mask images. Mask image 1402 is the mask image when the positive X-axis shake correction value is small. Mask image 1403 is the mask image when the positive X-axis shake correction value is medium. Mask image 1404 is the mask image when the positive X-axis shake correction value is large. In each mask image, the areas shown in black are invalid areas.

[0127] Thus, in a mask image corresponding to the positive X-axis blur correction value, the area representing the invalid region appears on the right side of the image. Conversely, in a mask image corresponding to the negative X-axis blur correction value, for example, the area representing the invalid region appears on the left side of the image.

[0128] Similarly, in a mask image corresponding to a positive Y-axis blur correction value, an area representing the invalid region appears at the bottom of the image, while in a mask image corresponding to a negative Y-axis blur correction value, an area representing the invalid region appears at the top of the image.

[0129] In step S1307, the control unit 101 obtains the logical OR of the mask images of the invalid regions in the four directions determined in step S1306 as a common invalid region for the VR video (all frames).

[0130] For example, consider a case where a mask image is selected for a small positive X-axis shake correction value, a large negative X-axis shake correction value, a medium positive Y-axis shake correction value, and a mask image for a zero negative Y-axis shake correction value. In this case, the control unit 101 acquires a mask image 1500 that represents an invalid region (the region shown in black), which is the logical OR of the four mask images 1500, as shown in Figure 15.

[0131] According to Embodiment 3, since there is no pixel-by-pixel loop processing in steps S705 to S710 of the flowchart in Figure 7, the invalid region of the equirectangular image can be calculated with less processing and a shorter processing time.

[0132] <Embodiment 4> Information on invalid regions in equirectangular images, calculated in the past, can be reused if certain conditions are met. These conditions include, for example, the same method for calculating invalid regions and the same manual correction parameters.

[0133] The process for generating a VR video according to Embodiment 4 will be explained with reference to the flowchart in Figure 16. Note that steps S501 to S504 in the flowchart of Figure 16 perform the same processing as the steps with the same names in the flowchart of Figure 5.

[0134] In step S1603, the control unit 101 determines whether invalid region information (information including invalid regions and settings for calculating invalid regions) of the equirectangular image is stored in a file. If it is determined that invalid region information is stored in a file (i.e., information regarding invalid regions calculated during past image conversions is stored in a file), the process proceeds to step S1604. If it is determined that invalid region information is not stored in a file, the process proceeds to step S503. Here, the file storing the invalid region information of the equirectangular image may be the video file itself, or it may be a separate file from the video file.

[0135] In step S1604, the control unit 101 determines whether the invalid region of the equirectangular image stored in the file is calculated using the same settings as the current settings. Here, "settings" refers to, for example, the method for calculating the invalid region or the manual correction parameter values. If it is determined that the invalid region is calculated using the same settings as the current settings, the process proceeds to step S1605. If it is determined that the invalid region is not calculated using the same settings as the current settings, the process proceeds to step S503.

[0136] In step S1605, the control unit 101 reads the invalid region information of the equirectangular image stored in the file. This process then generates an equirectangular image (VR video) based on the invalid region information in the subsequent step S504.

[0137] In step S1607, the control unit 101 saves the invalid region of the equirectangular image of the video file calculated in step S503, the settings for calculating the invalid region, and the manual correction parameters as invalid region information to a file. Here, the file to which the information is saved may be the video file itself or a separate file from the video file.

[0138] According to Embodiment 4, if appropriate invalid region information is stored in advance, the invalid region calculation process can be omitted, making it possible to generate an equirectangular image at a faster speed.

[0139] <Embodiment 5> Multiple video files may be played back consecutively as a single video. Therefore, in Embodiment 5, the editing device 100 controls the playback unit (multiple video files), rather than on a per-video-file basis, to ensure that the position and shape of the invalid area do not change.

[0140] Referring to the flowchart in Figure 17, a method for generating an equirectangular image from a circular fisheye image and creating a VR video in Embodiment 5 will be described. Steps S501 to S504 in the flowchart of Figure 17 are the same as steps S501 to S504 in the flowchart of Figure 5. In the flowchart of Figure 17, steps S501 to S504 are performed for the video file with file number n.

[0141] In step S503, the same invalid coordinate table is used for all video files that are played consecutively (all video files that can be played consecutively). As a result, the coordinates of pixels that are determined to be invalid pixels in any frame of any video file are registered in the invalid coordinate table. In step S504, the same invalid coordinate table is used for all video files to output (generate) an equirectangular image.

[0142] In step S1701, the control unit 101 sets the variable n, which represents the file number of the video file, to 0.

[0143] In step S1702, the control unit 101 determines whether or not the processes in steps S501 to S503 have been performed for all video files. If it is determined that the processes in steps S501 to S503 have been performed for all video files, the process proceeds to step S1707. If it is determined that the processes in steps S501 to S503 have not been performed for at least one of the video files, the process proceeds to step S501.

[0144] In step S1706, the control unit 101 adds 1 to the variable n, which represents the file number.

[0145] In step S1707, the control unit 101 sets the variable n, which represents the file number, to 0 again.

[0146] In step S1708, the control unit 101 determines whether or not the process in step S504 has been performed for all video files. If it is determined that the process in step S504 has been performed for all video files, the process in this flowchart is completed. If it is determined that the process in step S504 has not been performed for at least one of the video files, the process proceeds to step S504.

[0147] In step S1710, the control unit 101 adds 1 to the variable n, which represents the file number.

[0148] In Embodiment 5, the position and shape of the invalid region in the equirectangular image can be made the same throughout all frames of multiple video files. This makes it possible to suppress flickering caused by changes in the shape of the invalid region when playing back equirectangular images from multiple video files.

[0149] <Embodiment 6> In Embodiment 6, the editing device 100 calculates an equirectangular image from a circular fisheye image based on image correction information such as blur correction values ​​(correction values ​​for camera shake correction). The editing device 100 also generates a mask image (image mask; mask) that covers a portion of the equirectangular image based on the image correction information. The circular fisheye image in Embodiment 6 has two image circles (shooting areas).

[0150] Figure 18 shows the configuration of the control unit 101 in Embodiment 6. The control unit 101 includes a correction information generation unit 109, a coordinate information generation unit 110, and a mask generation unit 111. Each component of the control unit 101 will be explained with reference to the flowchart in Figure 19. The flowchart in Figure 19 is a diagram illustrating image generation according to Embodiment 6.

[0151] In step S501, the control unit 101 reads common metadata stored in the file metadata area 302 of the video file through the input unit. The common metadata is metadata related to the video shooting state. For example, the common metadata includes the number of pixels in the video file, the design values ​​of the optical system, information on the size of the image circle and its position on the sensor, information on manufacturing errors of the optical system, time-series information on the orientation of the optical system during video shooting, and information on the sensor output.

[0152] In step S502, the correction information generation unit 109 generates blur correction values ​​for each frame as image correction information (geometric image correction information), for example, according to the flowchart in Figure 6. By using the blur correction values ​​when generating an equirectangular image from a circular fisheye image (when generating the equirectangular image), it is possible to correct the effect of camera shake (hand shake; camera shake) in the equirectangular image for each frame.

[0153] In Embodiment 6, in step S609 of the flowchart in Figure 6, the correction information generation unit 109 (control unit 101) generates blur correction values ​​for the area of ​​the circular fisheye image based on metadata related to the shooting state of the video. Also, in step S602, whether or not image data for all frames has been read may be determined by referring to "information on the number of frames the video has" included in the metadata related to the shooting state of the video.

[0154] In step S1902, the mask generation unit 111 generates a mask image (such as the mask image 2200 shown in Figure 22) based on the blur correction value, which is image correction information. The mask image includes at least a "mask region that covers the equirectangular image" and a "maskless region that does not cover the equirectangular image but allows the equirectangular image to be displayed transparently." The mask image generation process will be described later using the flowchart in Figure 20.

[0155] In step S1903, the coordinate information generation unit 110 generates an equirectangular image by determining the coordinate values ​​of the region of the circular fisheye image corresponding to the equirectangular image for each frame.

[0156] Figure 21 shows a flowchart illustrating the process of generating an equirectangular image from a circular fisheye image. The flowchart in Figure 21 follows steps S807 to S809 from the flowchart in Figure 8. This is a flowchart with the following steps removed. Specifically, if it is determined in step S806 that the pixel value of at least one pixel in the equirectangular image of frame i has not been set, the process in step S807 is skipped and the process proceeds to step S810. In other words, in embodiment 6, an invalid coordinate table is not used, and the calculation of invalid regions in the equirectangular image is not performed.

[0157] In step S1904, the coordinate information generation unit 110 generates a new equirectangular image by applying (superimposing; combining) the mask image generated in step S1902 to the equirectangular image generated in step S1903.

[0158] The coordinate information generation unit 110 superimposes a mask image onto an equirectangular image, for example. In this case, the coordinate information generation unit 110 ensures that the areas in the equirectangular image that overlap with the mask area are hidden by the mask area, while areas that overlap with areas without a mask in the equirectangular image are not hidden. For example, the coordinate information generation unit 110 generates a new equirectangular image (an equirectangular image with a mask image) by selecting the darker pixel (the one with the lower brightness value) for each pixel at the same coordinate in the mask image and the equirectangular image. In this way, the coordinate information generation unit 110 may generate a new equirectangular image.

[0159] In Embodiment 6, metadata is used, so the mask image takes into account the results of image stabilization processing. Therefore, even if an area unintended by the photographer (an area outside the image circle) appears in the equirectangular image, that area can be hidden by the mask image. Thus, the user's sense of immersion in the VR video is improved.

[0160] (Mask image generation process; S1902) The process of generating the mask image in step S1902 will be explained with reference to the flowchart in Figure 20.

[0161] In step S2001, the mask generation unit 111 sets the variable i, which indicates the position of a pixel, to 0. Hereafter, the i-th pixel refers to one pixel in the mask image. The processing from step S2001 onward may be carried out sequentially in the X direction of the (X,Y) coordinate system, or sequentially in the Y direction.

[0162] In step S2002, the mask generation unit 111 determines whether or not it has calculated the pixel values ​​for all pixels of the mask image. If it is determined that the pixel value for any of the pixels of the mask image has not been calculated, the process proceeds to step S2003. If it is determined that the pixel values ​​for all pixels of the mask image have been calculated, the process in this flowchart ends.

[0163] In step S2003, the mask generation unit 111 determines whether the i-th pixel of the mask image is included in the mask region. In Embodiment 6, metadata related to the image circle 1021 and the blur correction value (geometric image correction information) calculated in step S502 are used to determine whether a certain pixel is included in the mask region.

[0164] For example, the coordinate information generation unit 110 calculates the coordinates of the circular fisheye image (see Figures 4A to 4C) that correspond to the coordinates of the equirectangular image. The mask generation unit 111 then refers to metadata about the image circle 1021 in the region of the circular fisheye image to determine whether a certain pixel in the equirectangular image corresponds to a pixel included in the image circle 1021. The mask generation unit 111 then determines the region with the same coordinates as the "region in the equirectangular image that does not correspond to a pixel within the image circle 1021" (the region of the mask image) as the mask region when no geometric image correction such as image stabilization is performed.

[0165] Subsequently, the mask generation unit 111 corrects (adjusts) the mask area (position and size of the mask area) based on the blur correction value (geometric image correction information) calculated in step S502. For example, the mask generation unit 111 refers to the blur correction value for each frame and widens the mask area in cases where no image stabilization is performed by the amount of the blur correction value. In this way, the mask generation unit 111 calculates a mask area (position and size of the mask area) that takes image stabilization into account.

[0166] Referring to the mask region calculated in this way, in step S2003, the mask generation unit 111 determines whether or not the i-th pixel is included in the mask region. If it is determined that the i-th pixel is included in the mask region, the process proceeds to step S2005. If it is determined that the i-th pixel is not included in the mask region, the process proceeds to step S2004.

[0167] In step S2004, the mask generation unit 111 sets the i-th pixel in the mask image to a white pixel (the pixel value of the i-th pixel is set to a pixel value indicating white). When the area set to a white pixel in the mask image is superimposed on a certain area of ​​the equirectangular image, it is displayed transparently (displayed without obstruction) the said area.

[0168] In step S2005, the mask generation unit 111 sets the i-th pixel in the mask image to a black pixel (the pixel value of the i-th pixel is set to a pixel value indicating black). When the area set to a black pixel in the mask image is superimposed on a certain area of ​​the equirectangular image, it covers that area by applying black to it.

[0169] In step S2006, the mask generation unit 111 adds 1 to the variable i. This causes the processing in steps S2002 to S2005 to be performed for the next pixel of the mask image.

[0170] Thus, in Embodiment 6, pixels in the mask region are set to black pixels, and pixels in the unmasked region are set to white pixels. The mask generation unit 111 performs this processing to generate a mask image 2200, which is a distribution of white and black pixels, as shown in Figures 22 and 23A. Note that the distribution is not limited to white and black pixels, as long as the mask region and the unmasked region can be distinguished.

[0171] As shown in Figures 22 and 23A, the mask image 2200 may be generated individually for each frame using the blur correction value (image stabilization correction value) in the corresponding equirectangular image 402. The mask image 2200 may also be generated based on the maximum correction value in all frames. In this case, the mask image 2200 may be applied to all frames of the equirectangular image. In this case, for example, a mask image is generated that prevents invalid areas from being captured in all frames of the equirectangular image. Furthermore, the mask image 2200 applied to all frames of the equirectangular image is the same. Even if the blur correction value in the image of each frame changes, the shape of the masked peripheral area does not change. Therefore, it is possible to suppress flickering in VR videos and improve the user's immersion in VR videos. In addition, a mask image may be generated in which a predetermined area is masked so as to prevent invalid areas from being captured even if blur correction is performed without using a blur correction value.

[0172] Figure 22 shows an example of providing a gradient region in the mask image 2200. Figure 22 is an excerpt of the edge of the mask image 2200. The mask image 2200 has an unmasked region 2201. The mask image 2200 has a gradient region 2202 and a masked region 2203 as mask regions. The unmasked region 2201 is a region of white pixels. The masked region 2203 is a region of black pixels. The gradient region 2202 has a brightness value intermediate between the unmasked region 2201 and the masked region 2203. In Embodiment 6, The luminance value of gradient region 2202 is determined by linear interpolation between the luminance value of unmasked region 2201 and the luminance value of masked region 2203. As a result, the luminance value of gradient region 2202 changes stepwise (continuously) from unmasked region 2201 to masked region 2203.

[0173] For example, when the gradient region 2202 is superimposed on a certain region of an equirectangular image, that region will be colored by the gradient region 2202. Alternatively, when the gradient region 2202 is superimposed on a certain region of an equirectangular image, the gradient region 2202 may display the region transparently, with higher transparency the higher its own luminance value. By applying a mask image having the gradient region 2202, the luminance changes in the peripheral areas of a VR video can be smoothed out. Therefore, the sense of immersion in the VR video can be improved.

[0174] The gradient region 2202 smooths out the change in brightness at the boundary between the unmasked region 2201 and the masked region 2203. In Embodiment 6, the starting position of the gradient region 2202 (= the boundary position between the gradient region 2202 and the unmasked region 2201) is determined based on the blur correction value. A mask image having the gradient region 2202 may be generated frame by frame based on the blur correction value of each frame. The starting position of the gradient region 2202 may be determined based on the maximum value of the blur correction value for all frames.

[0175] Referring to Figures 23A and 23B, the starting positions of the gradient region 2202 and the masked region 2203 will be explained.

[0176] Figure 23A is an excerpt of pixels corresponding to one circular fisheye region from the mask image 2200. Figure 23B is an enlarged view of a portion of the mask image 2200 in Figure 23A. The mask image 2200 has an unmasked region 2201, a gradient region 2202, and a masked region 2203. The gradient start position 2301 indicates the starting position of the gradient region 2202 when no correction is performed by image stabilization. In Figure 23B, the gradient start position 2301 is the starting position of the gradient region 2202 closer to the center of the mask image 2200. The start position 2302 is the starting position of the masked region 2203 closer to the center of the mask image 2200. The gradient start position 2304 is the starting position of the gradient region 2202 when correction is performed by image stabilization.

[0177] The starting position variation amount 2303, which is the difference between the gradient starting position 2301 and the gradient starting position 2304, corresponds to the amount of relative positional shift between the image circle 4011 and the region 4012 due to image stabilization. In other words, the gradient starting position 2304 is shifted toward the center of the mask image 2200 (the direction in which the unmasked region 2201 exists) from the gradient starting position 2301 by the number of pixels corresponding to the blur correction value. In this way, by taking the blur correction value information into account for the starting position of the gradient region 2202, the brightness change in the peripheral area can be made smoother. This makes it possible to more naturally hide reflections and other unintended elements with the mask image. The gradient region is determined, for example, based on the white and black pixels set by the mask image generation process in step S1902 (flowchart in Figure 20). Only a region of pixels of a predetermined width within the black pixel region may be determined as the gradient region. Alternatively, only a region of pixels of a predetermined width within the white pixel region may be determined as the gradient region. Alternatively, a region of pixels of a predetermined width that includes both white and black pixels may be determined as a gradient region.

[0178] Figure 24 shows an example of mask image 2200. Mask image 2200 corresponds to a circular fisheye image 403 (see Figure 4B) having two image circles. The unmasked region 2201 is the region of white pixels. The masked region 2203 is the region of black pixels. Unmasked region 2 A gradient region 2202 is set between 201 and the masked region 2203. The masked image 2200 has two unmasked regions 2201 corresponding to the regions of the circular fisheye image. The masked region 2203 is positioned to surround the periphery of the unmasked region 2201.

[0179] The mask image 2200 shown in Figure 24 contains two unmasked regions 2201 separated by a masked region 2203. The two unmasked regions 2201 correspond to the two image circles 1021 (the regions of the two circular fisheye images) in the circular fisheye image 403. As shown in Figure 24, the mask image 2200 has a first masked region 2204 and a second masked region 2205. The first masked region 2204 includes a range corresponding to the invalid region which is outside the image circle.

[0180] By the way, when shooting a video containing two image circles, for example, when using a fisheye lens with a field of view exceeding 180°, the optical system of one optical system may be reflected in the video shot by the other optical system, as explained using Figures 10A and 10B. There is a concern that this unintended reflection by the photographer may impair the viewer's sense of immersion when watching the VR video. Therefore, in Embodiment 6, the mask image 2200 has a second mask region 2205. The second mask region 2205 includes the range corresponding to images 1023 and 1024, which are the reflection regions of the optical system.

[0181] When a user views a VR video, they view the video, captured by two optical systems, with each eye using an HMD (Head-Mounted Display). If the shape of the mask region differs between the left and right eyes, there is a concern that areas will be visible to only one eye, causing discomfort in the video. In Embodiment 6, in order to make the shape of the mask region the same on both sides, the second mask region 2205 is configured to have a symmetrical shape in each of the left and right videos.

[0182] As described above, Embodiment 6 demonstrates an example in which image correction values ​​are used as image correction information for generating a mask image. In VR videos, it is important that the parallax between the left-eye image and the right-eye image is appropriately set in order to experience a proper sense of depth. For this reason, parallax correction is sometimes performed to adjust the positions of the left and right images. In addition, if there is an unintended tilt in the video in a VR video, there is a concern that it will impair the viewer's sense of immersion. To prevent this, horizontal correction is sometimes performed to adjust the tilt of the video. Therefore, the geometric correction for the equirectangular image is not limited to image stabilization, but may also be horizontal correction or parallax correction. The image correction information may be a correction value for either horizontal correction or parallax correction.

[0183] Furthermore, with regard to image stabilization, image correction information may be calculated using the output of sensors (sensor information), such as an accelerometer mounted on the imaging device, as metadata. Alternatively, information on the feature points of each detected frame may be used as metadata. The metadata may also be a combination of sensor output and feature point information. With regard to horizontal correction, for example, the output of a sensor on the imaging device, such as a gyro sensor, may be used as metadata. With regard to parallax correction, information regarding the deviation of the actual value of the optical system from the design value may be used as metadata. Additionally, information regarding the relative positions of two image circles calculated from the image may be used as metadata. Note that in Embodiment 6, the mask image was generated to be applied to an equirectangular image, but this is not limited to this. For example, the mask image may be generated to be applied to a circular fisheye image, and the equirectangular image may be obtained after applying the mask image to the circular fisheye image.

[0184] <Embodiment 7> In Embodiment 7, when obtaining an equirectangular image from a fisheye circular image, the error region (compared to the effective region) is used. The process for handling cases where, despite the response, a region occurs where a pixel value corresponding to an invalid region is set is explained using Figure 25. The flowchart in Figure 25 shows the process in Embodiment 7. Steps in which the same process as in the flowchart of Figure 5 or the flowchart of Figure 19 is performed are given the same numbers and their explanations are omitted.

[0185] In step S2501, the control unit 101 generates an equirectangular image for each frame and outputs the generated equirectangular image.

[0186] In step S2502, the mask image generated in step S1902 is applied to the equirectangular image generated in step S504 to generate a new equirectangular image.

[0187] Here, in step S2501 described above, the flowchart in Figure 9 is processed, similar to step S504.

[0188] In step S902 of the flowchart in Figure 9, the control unit 101 acquires the pixel values ​​for each coordinate (X,Y) in the region of the circular fisheye image. At this time, the control unit 101, for example, determines the coordinates of the circular fisheye image corresponding to each pixel in the equirectangular image and acquires the pixel value for that pixel. Here, in order to reduce the calculation processing in the control unit 101, interpolation processing may be performed when calculating the pixels of the circular fisheye video corresponding to the pixels of the equirectangular image.

[0189] Refer to Figures 26 and 27 to explain the error region caused by interpolation (a region where a pixel value corresponding to an invalid region is set, even though it corresponds to an effective region). Figure 26 is an enlarged view of a portion of the equirectangular image 406 when no interpolation is performed. One rectangular region in Figure 26 represents one pixel in the equirectangular image 406.

[0190] The coordinate information generation unit 110 calculates the coordinates of a pixel in the region of the circular fisheye image for a given pixel in the equirectangular image 406. The invalid region boundary line 2601 is a line projected onto the region of the equirectangular image from a line indicating the boundary between the valid and invalid regions in the region of the circular fisheye image. In other words, in Figure 26, pixels located inside (lower right) of the invalid region boundary line 2601 are input with the pixel value of the corresponding pixel in the region of the circular fisheye image (valid region) and are treated as valid pixels 2605. On the other hand, pixels outside (upper left) of the invalid region boundary line 2601 are determined to have no corresponding pixel (corresponding to the invalid region) and are input with the value of a black pixel 2604.

[0191] Figure 27 shows a portion of the same equirectangular image as Figure 26, but it shows a portion of the equirectangular image when interpolation is performed. When interpolation is performed, the pixels in the circular fisheye image region corresponding to each pixel are not determined individually. In this case, a certain block (interpolation block 2602) is treated as one unit, and the "coordinates in the circular fisheye image region" corresponding to the center 2603 of the interpolation block 2602 are determined. In Embodiment 7, the interpolation block 2602 is a 4x4 square. In this case, a method using linear interpolation can be considered for determining the "coordinates in the circular fisheye image region" corresponding to each pixel. The "X coordinate in the circular fisheye image region" corresponding to each pixel can be calculated by linear interpolation using the coordinates of two adjacent centers 2603 in the X direction. The "Y coordinate in the circular fisheye image region" corresponding to each pixel can be calculated by linear interpolation using the coordinates of two adjacent centers 2603 in the Y direction. Note that here, the interpolation block is assumed to be a 4x4 square, but it is not limited to this. The interpolation block may be assumed to be a 3x3 square, and the coordinates in the region of the circular fisheye image corresponding to the pixel at the center of the interpolation block may be used. Note that the shape of the interpolation block is not limited to a square, but may also be a rectangle made up of multiple pixels.

[0192] Furthermore, if center 2603 is determined to correspond to an invalid region in the circular fisheye image, the corresponding coordinates of that center 2603 cannot be determined. Also, the "coordinates in the circular fisheye image region" corresponding to each pixel are determined by linear interpolation using the coordinates of two adjacent center 2603s. For this reason, pixels located between "center 2603 determined to correspond to an invalid region" and "the adjacent center 2603" are input as black pixels 2604, which are invalid pixels.

[0193] In Figure 27, compared to Figure 26, the number of pixels to which black pixels 2604 are input has increased due to the interpolation process. Also, in Figure 27, the shape of the boundary between black pixels 2604 and effective pixels 2605 is rough, which hinders the viewer's sense of immersion when watching VR video. Therefore, in Embodiment 7, the masked region 2203 is generated in such a way that it can cover the error region 2606, which is a region in which invalid pixel values ​​are input despite corresponding to effective pixels. More specifically, the mask generation unit 111 pre-calculates the error region 2606 caused by the interpolation process and generates the masked region 2203 wider by the width of the error region 2606 when generating the mask image 2200.

[0194] Therefore, in Embodiment 7, the area caused by calculation errors in the equirectangular image can be hidden by the mask image, thus preventing any loss of immersion for the viewer when watching the VR video.

[0195] In Embodiment 7, an example was shown in which an equirectangular image was generated from the results of calculating the invalid region, and then a mask image was generated. The generation order may be such that the mask image is generated first, or the generation of the equirectangular image and the mask image may be processed simultaneously.

[0196] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" may be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, "greater than or equal to A" may be rephrased as "greater than (higher; longer; more) than A," and "less than or equal to A" may be rephrased as "less than (lower; shorter; fewer) than A." And "greater than (higher; longer; more) than A" may be rephrased as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be rephrased as "less than or equal to A."

[0197] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0198] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0199] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0200] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0201] The above-disclosed embodiments include the following configurations, methods, and programs. (Composition 1) An electronic device that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, The means for acquiring the first video, Setting means for setting invalid areas of the same shape at the same position in the plurality of frames of the second video, A generation means for generating the second video by converting the first video such that for each of the plurality of frames of the second video, a pixel value indicating invalidity is set in the invalid region, An electronic device characterized by having the following features. (Configuration 2) The setting means is, For each frame, invalid coordinates in the second video that do not correspond to the valid area of ​​the first video are determined. The set of coordinates determined to be invalid in at least one of the plurality of frames is set as the invalid region. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) The effective area is the imaging area. The electronic device according to configuration 2, characterized by the features described above. (Composition 4) The effective area is the area within the imaging area that does not include the area specified in advance. The electronic device according to configuration 3, characterized by the features described above. (Composition 5) The setting means sets the region corresponding to the shaking of the imaging device while shooting the first video as the invalid region. The electronic device according to configuration 1, characterized by the features described above. (Composition 6) The setting means sets the areas corresponding to the four-directional shake of the imaging device while shooting the first video as the invalid areas. The electronic device according to configuration 5, characterized by the features described herein. (Composition 7) The generation means generates the second video which has been corrected to reduce the effect of camera shake during shooting by the imaging device. An electronic device according to any one of configurations 1 to 6, characterized by the features described herein. (Composition 8) The generation means generates the second video to which the correction has been applied based on the result of image processing on the first video. The electronic device according to configuration 7, characterized by the features described above. (Composition 9) The acquisition means acquires sensor data relating to the imaging device while the first video is being filmed, along with the first video. The generation means generates the corrected second video based on the sensor data. The electronic device according to configuration 7 or 8, characterized by the above. (Composition 10) The pixel value indicating invalidity is the pixel value representing black. An electronic device according to any one of configurations 1 to 9, characterized by the above. (Composition 11) The generation means generates the second video based on the invalid area information stored in the specific file if the invalid area information stored in the specific file is stored in a specific file and the invalid area information stored in the specific file is calculated according to the same settings as the current settings. An electronic device according to any one of configurations 1 to 10, characterized by the above. (Composition 12) The aforementioned first video includes multiple video files, An electronic device according to any one of configurations 1 to 11, characterized by the features described herein. (Composition 13) An electronic device that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, An acquisition means for acquiring the first video and metadata relating to the shooting status of the first video, Information generation means that generates correction information relating to geometric correction performed when the second video is generated, based on the metadata, A mask generation means for generating a mask image that covers a portion of the second video, A generation means that applies the mask image to each frame and generates the second video converted from the first video based on the correction information, An electronic device characterized by having the following features. (Composition 14) The correction information is the maximum value of the correction for each frame of the first video. The generation means applies the same mask image to all frames of the second video. The electronic device according to configuration 13, characterized by the features described above. (Composition 15) The mask image has a first region that covers the second video by assigning a specific color to the second video, and a second region that displays the second video without covering it. The electronic device according to configuration 13 or 14, characterized by the features described herein. (Composition 16) The mask image has a third region between the first region and the second region, from the first region to the second region, in which the brightness of the color applied to the second video changes continuously. The electronic device according to configuration 15, characterized by the features described herein. (Composition 17) The position of the boundary between the third region and the second region is the position based on the correction information. , The electronic device according to configuration 16, characterized by the features described above. (Composition 18) If the second video includes a specific region that corresponds to the aforementioned shooting region but does not have pixel values ​​corresponding to the aforementioned shooting region, the mask image has a region that covers the specific region. An electronic device according to any one of configurations 13 to 17, characterized by the features described herein. (Composition 19) The first video described above has at least two circular fisheye image regions, The second video has at least two circular fisheye images and corresponding equirectangular projection images. The mask image has a region that covers the peripheral part of the equirectangular projection image. An electronic device according to any one of configurations 13 to 18, characterized by the features described herein. (Composition 20) The first video has an area in which the optical system that acquires the shooting area is reflected, The mask image has a region that covers the region in the second video that corresponds to the region in which the optical system is reflected. An electronic device according to any one of configurations 13 to 19, characterized by the features described herein. (Composition 21) The aforementioned correction includes any of the following: image stabilization, horizontal correction, and parallax correction. An electronic device according to any one of configurations 13 to 20, characterized by the above. (Composition 22) The metadata includes at least one of the following: information regarding the design value of the optical system that acquires the imaging area; information regarding the deviation of the actual value of the optical system from the design value; time-series information regarding the orientation of the optical system; and information regarding the relative positions of the at least two image circles when the imaging area has at least two image circles. An electronic device according to any one of configurations 13 to 21, characterized by the features described herein. (Composition 23) The generation means generates the second video by obtaining the pixel values ​​of the coordinates of the first video corresponding to each coordinate of the second video based on the correction information. An electronic device according to any one of configurations 13 to 22, characterized by the features described herein. (Composition 24) The mask generation means generates the mask image that covers a portion of the second video based on the correction information. An electronic device according to any one of configurations 13 to 23, characterized by the features described herein. (Composition 25) The generation means generates the second video by converting the first video based on the correction information, and then applies the mask image to each frame of the second video to cover a portion of the second video. An electronic device according to any one of configurations 13 to 24, characterized by the features described herein. (Method 1) A control method for electronic equipment that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, The acquisition step of obtaining the first video, A setting step of setting invalid areas of the same shape at the same position in the multiple frames of the second video, A generation step of generating the second video by converting the first video such that, for each of the plurality of frames of the second video, a pixel value indicating invalidity is set in the invalid region, A method for controlling electronic equipment, characterized by having the following features. (Method 2) A control method for electronic equipment that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, An acquisition step of acquiring the first video and metadata relating to the shooting status of the first video, An information generation step that generates correction information related to geometric correction performed when the second video is generated, based on the metadata, A mask generation step of generating a mask image that covers a portion of the second video, A generation step in which the mask image is applied to each frame and a second video is generated by converting the first video based on the correction information, A method for controlling electronic equipment, characterized by having the following features. (program) A program for causing a computer to function as one of the means of the electronic equipment described in any of configurations 1 to 25. [Explanation of Symbols]

[0202] 100: Editing device (electronic equipment), 101: Control unit

Claims

1. An electronic device that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, The acquisition means for acquiring the first video, Setting means for setting invalid areas of the same shape at the same position in the plurality of frames of the second video, A generation means for generating the second video by converting the first video such that for each of the plurality of frames of the second video, a pixel value indicating invalidity is set in the invalid region, An electronic device characterized by having the following features.

2. The setting means is, For each frame, invalid coordinates in the second video that do not correspond to the valid area of ​​the first video are determined. The set of coordinates determined to be invalid in at least one of the plurality of frames is set as the invalid region. The electronic device according to feature 1.

3. The effective area is the imaging area. The electronic device according to feature 2.

4. The effective area is the area within the imaging area that does not include the area specified in advance. The electronic device according to feature 3.

5. The setting means sets the region corresponding to the shaking of the imaging device while shooting the first video as the invalid region. The electronic device according to feature 1.

6. The setting means sets the regions corresponding to the four-directional shake of the imaging device while shooting the first video as the invalid regions. The electronic device according to feature 5.

7. The generation means generates the second video which has been corrected to reduce the effect of camera shake during shooting by the imaging device. The electronic device according to feature 1.

8. The generation means generates the second video to which the correction has been applied based on the result of image processing on the first video. The electronic device according to feature 7.

9. The acquisition means acquires sensor data relating to the imaging device while the first video is being filmed, along with the first video. The generation means generates the corrected second video based on the sensor data. The electronic device according to feature 7.

10. The pixel value indicating invalidity is the pixel value representing black. The electronic device according to feature 1.

11. The generation means generates the second video based on the invalid area information stored in the specific file if the invalid area information stored in the specific file is stored in a specific file and the invalid area information stored in the specific file is calculated according to the same settings as the current settings. The electronic device according to feature 1.

12. The aforementioned first video includes multiple video files, The electronic device according to feature 1.

13. An electronic device that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, An acquisition means for acquiring the first video and metadata relating to the shooting status of the first video, Information generation means that generates correction information relating to geometric correction performed when the second video is generated, based on the metadata, A mask generation means for generating a mask image that covers a portion of the second video, A generation means that applies the mask image to each frame and generates the second video converted from the first video based on the correction information, An electronic device characterized by having the following features.

14. The correction information is the maximum value of the correction for each frame of the first video. The generation means applies the same mask image to all frames of the second video. The electronic device according to feature 13.

15. The mask image has a first region that covers the second video by assigning a specific color to the second video, and a second region that displays the second video without covering it. The electronic device according to feature 13.

16. The mask image has a third region between the first region and the second region, from the first region to the second region, in which the brightness of the color applied to the second video changes continuously. The electronic device according to feature 15.

17. The position of the boundary between the third region and the second region is the position based on the correction information. The electronic device according to feature 16.

18. If the second video includes a specific region that corresponds to the aforementioned shooting region but does not have pixel values ​​corresponding to the aforementioned shooting region, the mask image has a region that covers the specific region. The electronic device according to feature 13.

19. The first video has at least two circular fisheye image regions, The second video has an equirectangular projection image corresponding to each of the at least two circular fisheye images, The mask image has a region that covers the peripheral part of the equirectangular projection image. The electronic device according to feature 13.

20. The first video has an area in which the optical system that acquires the shooting area is reflected, The mask image has a region that covers the region in the second video that corresponds to the region in which the optical system is reflected. The electronic device according to feature 13.

21. The aforementioned correction includes any of the following: image stabilization, horizontal correction, and parallax correction. The electronic device according to feature 13.

22. The metadata includes at least one of the following: information regarding the design value of the optical system that acquires the imaging area; information regarding the deviation of the actual value of the optical system from the design value; time-series information regarding the orientation of the optical system; and information regarding the relative positions of the at least two image circles when the imaging area has at least two image circles. The electronic device according to feature 13.

23. The generation means generates the second video by obtaining the pixel values ​​of the coordinates of the first video corresponding to each coordinate of the second video based on the correction information. The electronic device according to feature 13.

24. The mask generation means generates the mask image that covers a portion of the second video based on the correction information. The electronic device according to feature 13.

25. The generation means generates the second video by converting the first video based on the correction information, and then applies the mask image to each frame of the second video to cover a portion of the second video. The electronic device according to feature 13.

26. A control method for electronic equipment that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, The acquisition step of obtaining the first video, A setting step of setting invalid areas of the same shape at the same position in the plurality of frames of the second video, A generation step of generating the second video by converting the first video such that for each of the plurality of frames of the second video, a pixel value indicating invalidity is set in the invalid area, A method for controlling electronic equipment, characterized by having the following features.

27. A control method for electronic equipment that converts a first video of a first display method, which includes a shooting area representing the result of shooting by an imaging device, into a second video of a second display method having multiple frames, An acquisition step of acquiring the first video and metadata relating to the shooting status of the first video, An information generation step that generates correction information related to geometric correction performed when the second video is generated, based on the metadata, A mask generation step of generating a mask image that covers a portion of the second video, A generation step in which the mask image is applied to each frame and a second video is generated by converting the first video based on the correction information, A method for controlling electronic equipment, characterized by having the following features.

28. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 25.

Images