Image processing apparatus, image processing method, computer program product, and computer-readable storage medium

Abstract

An image processing apparatus, an image processing method, a computer program product, and a computer-readable storage medium are provided. The image processing apparatus according to the present disclosure includes: a moving image acquisition unit configured to acquire a photographed moving image; an acquisition unit configured to acquire a blur correction amount for reducing blur of the moving image; a processing unit configured to perform geometric transformation on images of frames of the moving image, wherein the processing unit acquires an image in which blur is reduced as a post-geometric transformation image based on the blur correction amount; a generation unit configured to generate a mask based on a temporal change in the blur correction amount; and a superimposition unit configured to superimpose the mask on the post-geometric transformation image, wherein the generation unit generates the mask so that a region in the post-geometric transformation image that is not covered by the mask has a size specified by a user and moves by an amount of movement based on the size and the temporal change in the blur correction amount.

Description

Technical Field

[0001] This disclosure relates to an image processing apparatus for reducing unwanted regions from an image after geometric transformation. Background Technology

[0002] Typically, VR images (VR motion images) consist of two images (a left-lens image and a right-lens image), which users can view in stereoscopic form using parallax when played on a head-mounted display (HMD). VR images may include, for example, two isometric cylindrical projection images, which are the result of isometric cylindrical transformations of two circular fisheye images taken through two circular fisheye lenses.

[0003] Without properly fitting two circular fisheye lenses, the parallax between the two circular fisheye images may not be suitable for stereoscopic vision. In this case, a transformation can be performed by adjusting the parameters of the isometric cylindrical transformation used to transform the circular fisheye image into an isometric cylindrical projection image, thereby generating an isometric cylindrical projection image with parallax suitable for stereoscopic vision. However, as a result of parameter adjustment, by referencing the out-of-focus area and vignetting area of ​​the circular fisheye image, the dark areas may be set as part of the isometric cylindrical projection image.

[0004] Japanese Patent Application Publication No. 2022-184139 discloses a technique for setting black regions during equidistant cylindrical transformation of VR images so that the shapes of the black regions in the left and right images are consistent with each other.

[0005] However, according to the technology disclosed in Japanese Patent Application Publication No. 2022-184139, it is not possible to reduce unnecessary areas such as black areas from images after geometric transformations such as equidistant cylindrical projection images. Summary of the Invention

[0006] This disclosure provides techniques for effectively reducing unwanted regions, such as black areas, from geometrically transformed images, such as isometric cylindrical projection images.

[0007] The image processing apparatus according to this disclosure includes: a motion image acquisition unit configured to acquire photographic motion images; an acquisition unit configured to acquire a blur correction amount for reducing blur in the motion images; a processing unit configured to perform geometric transformation on each frame of the motion images, wherein the processing unit acquires a blurred image as a geometrically transformed image based on the blur correction amount; a generation unit configured to generate a mask based on a time variation of the blur correction amount; and an overlay unit configured to overlay the mask onto the geometrically transformed image, wherein the generation unit generates the mask such that regions in the geometrically transformed image not covered by the mask have a size specified by a user, and move by a movement amount based on the size and a time variation of the blur correction amount.

[0008] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is given by way of example. Attached Figure Description

[0009] Figure 1 This is a block diagram of an editing device according to the first embodiment.

[0010] Figure 2 This is a flowchart of motion image photography processing according to the first embodiment.

[0011] Figure 3 This is a schematic diagram of a motion image file according to the first embodiment.

[0012] Figures 4A to 4C This is a schematic diagram used to illustrate the problem to be solved in the first embodiment.

[0013] Figure 5 This is a flowchart of the VR motion image generation process according to the first embodiment.

[0014] Figure 6 This is a flowchart of the fuzzy correction value calculation process based on the first embodiment.

[0015] Figure 7 This is a flowchart of the mask parameter generation process according to the first embodiment.

[0016] Figure 8 It is a graph showing the time variation of the fuzzy correction value according to the first embodiment.

[0017] Figure 9 This is a flowchart of the equidistant cylindrical image generation process according to the first embodiment.

[0018] Figure 10 This is a flowchart of the coordinate calculation process based on the first embodiment.

[0019] Figure 11 This is a flowchart of the pixel generation process according to the first embodiment.

[0020] Figure 12 This is a schematic diagram of a circular fisheye image according to the first embodiment.

[0021] Figure 13A This is a schematic diagram of a circular fisheye image according to the first embodiment.

[0022] Figure 13B and Figure 13C This is a schematic diagram of an equidistant cylindrical image according to the first embodiment.

[0023] Figure 14 This is a block diagram of an editing device according to the second embodiment.

[0024] Figure 15 This is a flowchart of the mask parameter generation process according to the second embodiment.

[0025] Figure 16 This is a block diagram of a photographic apparatus according to the third embodiment.

[0026] Figure 17 This is a schematic diagram of a photographic image according to the third embodiment. Detailed Implementation

[0027] Embodiments of this disclosure will be described. In the case of generating motion image files for VR images, motion blur caused by camera shake can be reduced by adjusting the parameters used for isometric cylindrical transformation. However, as a result of this parameter adjustment, black areas can be set as part of the isometric cylindrical projection image. Furthermore, temporal changes in the parameters cause the black areas to shift, resulting in flickering around the periphery of the isometric cylindrical projection image. Unnecessary areas can be eliminated (covered) by superimposing a mask on the isometric cylindrical projection image that exposes the center portion of the image, but this reduces the effective area of ​​the video. In the following embodiments, an image with small unnecessary areas (such as black areas) and a wide effective area is obtained as a geometrically transformed image such as an isometric cylindrical projection image. First Embodiment

[0028] The first embodiment of this disclosure will be described below. Figure 1This is a block diagram illustrating the configuration of an editing device 100, which is an example of an image processing apparatus according to the first embodiment. The editing device 100 is, for example, an electronic device such as a computer, a smartphone, or a photographic device (such as a digital camera). The editing device 100 includes a control unit 101, a ROM 102, a RAM 103, a storage medium 104, an operation unit 105, a display unit 106, a communication unit 107, and a system bus 108.

[0029] The control unit 101 controls the entire editing device 100. The control unit 101 is, for example, a central processing unit (CPU). The control unit 101 can also perform image generation, setting processing, and calculation processing.

[0030] ROM 102 is a read-only memory (ROM) that stores information (programs and parameters) that does not need to be changed.

[0031] RAM 103 is a random access memory (RAM) that temporarily stores information (programs and data) supplied from external devices, etc.

[0032] Storage medium 104 is fixedly installed in editing device 100. Storage medium 104 is a hard disk or flash memory. Storage medium 104 is removable from editing device 100. Storage medium 104 can be an optical disk such as a floppy disk (FD) or a compact disk (CD), or it can be a magnetic card, optical card, IC card, or memory card.

[0033] The operation unit 105 receives user operations (instructions from the user). The operation unit 105 includes operating components (such as buttons or touch panels) that can be operated by the user.

[0034] The display unit 106 displays the data stored in the editing device 100 and the data supplied to the editing device 100.

[0035] The communication unit 107 communicates with external devices such as cameras (photographic devices; digital cameras).

[0036] The system bus 108 connects multiple components (units) communicatively to each other.

[0037] Moving image files, including image data and metadata, are stored (recorded) in storage medium 104. The moving image files can be read from storage medium 104. Communication unit 107 can also receive moving image files from external devices such as cameras. There are no particular limitations on the method of acquiring moving image files (moving image acquisition).

[0038] Figure 2 This is a flowchart illustrating motion image photography processing performed by a camera (photographic device) that can communicate with the editing device 100.

[0039] In step S201, the camera determines whether sensor data can be stored (recorded) in a motion picture file. Sensor data, for example, is inertial data representing acceleration measured by a gyroscope sensor. The camera can record its movement (pose change) during motion picture capture by periodically storing the inertial data from the gyroscope sensor in a motion picture file. If the camera determines that the sensor data can be stored in a motion picture file, the process proceeds to step S202. If the camera determines that the sensor data cannot be stored in a motion picture file, the process proceeds to step S203.

[0040] In step S202, the camera stores the sensor data as timing metadata (metadata as time series information) in the motion picture file.

[0041] In step S203, the camera determines whether to store audio data (sound data) in the moving image file. Depending on the shooting mode, the camera does not store audio data in the moving image file. If the camera determines that audio data should be stored in the moving image file, the process proceeds to step S204. If the camera determines that audio data should not be stored in the moving image file, the process proceeds to step S205.

[0042] In step S204, the camera stores the audio data (sound data) as audio sampling data in the motion picture file.

[0043] In step S205, the camera stores the image data of one frame as video sampling data in a moving image file.

[0044] In step S206, the camera determines whether the end of motion image photography has been indicated. If the camera determines that the end of motion image photography has been indicated, the motion image photography process ends. If the camera determines that the end of motion image photography has not been indicated, the process proceeds to step S201.

[0045] Figure 3 It shows through Figure 2 The diagram illustrates the structure of a motion image file acquired by a camera during motion image photography processing. The motion image file includes a motion image header region 301, a file metadata region 302, and a sampled data region 303.

[0046] The motion picture header region 301 stores information related to the entire motion picture (hereinafter referred to as "motion picture related information"). For example, motion picture related information includes at least one of the following: the image size (width and height) of the motion picture, the frame rate of the motion picture, the sampling frequency of the audio (sound), the sample size of the audio, and the number of audio channels. Motion picture related information may include information such as the temporal position (playback position; playback time) and size of samples (a frame of image in the case of video) of various media (video, audio, and timing metadata, etc.).

[0047] File metadata area 302 stores metadata related to the file itself or metadata related to the entire moving image. For example, in file metadata area 302, information related to the lens during photography, design values ​​(such as the center position or radius of the image circle), or measurement values ​​measured through calibration are stored as metadata.

[0048] The sampling data area 303 stores the sampling data of each media. The sampling data area 303 includes a video sampling area 3031, an audio sampling area 3032, and a timing metadata sampling area 3033.

[0049] Video sampling area 3031 stores the sampling data of the video media. The sampling data of the video media is... Figure 2 The image data stored in step S205. For example, when two fisheye lenses are attached to a camera, two circular fisheye images corresponding to the two fisheye lenses (multiple images corresponding to multiple optical systems respectively) are stored as a video sample in the video sampling area 3031. A single image having two circular fisheye image areas can be stored as a video sample.

[0050] Audio sampling area 3032 stores the sampling data of the audio media. The sampling data of the audio media is... Figure 2 The audio data stored in step S204. If no audio data is recorded, the audio sampling area 3032 may not exist.

[0051] The timing metadata sampling area 3033 stores the sampled data of the timing metadata media. The sampled data of the timing metadata media, for example, is... Figure 2 The sensor data stored in step S202. If timing metadata is not recorded, the timing metadata sampling area 3033 may not exist.

[0052] The editing device 100 generates VR motion images (VR180 motion images) by performing an isometric cylindrical transformation on a circular fisheye image recorded by a camera. Figures 4A to 4CThis is a schematic diagram showing a circular fisheye image recorded by a camera and an isometric cylindrical projection image (hereinafter referred to as an "isometric cylindrical image") obtained by the editing device 100. The isometric cylindrical image is an image obtained by performing an isometric cylindrical transformation on the circular fisheye image. Note that, for simplicity, a circular fisheye image recorded by a camera equipped with one fisheye lens and an isometric cylindrical image generated from that circular fisheye image will be described. However, the same description applies to two circular fisheye images recorded by a camera equipped with two fisheye lenses and two isometric cylindrical images generated from those two circular fisheye images.

[0053] exist Figure 4A In the image, circular fisheye image 401 is a circular fisheye image recorded by a camera equipped with a fisheye lens. The imaging circle 4011 (the area inside the imaging circle 4011) is the area photographed by the camera through the fisheye lens (the photographing area) and is the valid area containing video. The subject is photographed inside the imaging circle 4011. The area outside the imaging circle 4011 is an invalid area without video. The color of the invalid area is not particularly limited and is, for example, black or a near-black color. The invalid area is an area independent of the subject. Information related to the imaging circle 4011 in the circular fisheye image 401 (such as the center position and radius of the imaging circle 4011) is stored as file metadata in the file metadata area 302. Typically, since the position of the lens (the relative position of the lens to the imaging element) does not change during moving image photography, the position of the imaging circle 4011 is constant.

[0054] Region 4012 is the region to be transformed into an isometric cylindrical image. The inner part of region 4012 is transformed into an isometric cylindrical image. Information related to region 4012 is also stored as file metadata in file metadata region 302. The center position of region 4012 is the same as the center position of imaging circle 4011 by default, but it can be changed.

[0055] The isometric cylindrical image 402 is obtained by performing an isometric cylindrical transformation on region 4012. The subject is positioned at the center of the isometric cylindrical image 402.

[0056] exist Figure 4B Similar to circular fisheye image 401, circular fisheye image 403 is a circular fisheye image recorded by the camera. Similar to imaging circle 4011, imaging circle 4031 is the photographic area and the effective area. Similar to the area outside imaging circle 4011, the area outside imaging circle 4031 is the ineffective area. Similar to area 4012, area 4032 is the area to be subjected to isometric cylindrical transformation.

[0057] In the circular fisheye image 403, the camera is slightly pointed to the left from the position when recording the circular fisheye image 401, so that the subject is positioned slightly to the right of the center of the imaging circle 4031.

[0058] The isometric cylindrical image 404 is obtained by performing an isometric cylindrical transformation on region 4032. The relative position of region 4032 with respect to the imaging circle 4031 in the circular fisheye image 403 is the same as the relative position of region 4012 with respect to the imaging circle 4011 in the circular fisheye image 401. Therefore, the subject is positioned slightly to the right of the center of the isometric cylindrical image 404.

[0059] exist Figure 4C Similar to circular fisheye images 401 and 403, circular fisheye image 405 is a circular fisheye image recorded by the camera. Similar to imaging circles 4011 and 4031, imaging circle 4051 is the photographic area and the effective area. Similar to the area outside imaging circle 4011 and the area outside imaging circle 4031, the area outside imaging circle 4051 is the ineffective area. Similar to areas 4012 and 4032, area 4052 is the area to be subjected to isometric cylindrical transformation.

[0060] In circular fisheye image 405, similar to circular fisheye image 403, the subject is positioned slightly to the right of the center of imaging circle 4051. Furthermore, region 4052 is positioned slightly to the right of both region 4012 and region 4032.

[0061] The isometric cylindrical image 406 is obtained by performing an isometric cylindrical transformation on region 4052. By arranging region 4052 at a position slightly offset to the right from the positions of region 4012 and region 4032, the subject is positioned at the center of the isometric cylindrical image 406. As described above, in this embodiment, when generating an isometric cylindrical image from a circular fisheye image, the region in the circular fisheye image to undergo the isometric cylindrical transformation is changed. As a result, blurring of the moving image caused by camera shake can be reduced.

[0062] However, since a portion (the right-hand portion) of region 4052 protrudes from the imaging ring 4051, an unnecessary region corresponding to the invalid region of the circular fisheye image 405 is set as a portion (the right-hand portion) of the equidistant cylindrical image 406. This unnecessary region moves and flickers in the periphery of the equidistant cylindrical image as camera shake continues during motion image capture. Furthermore, when a user views VR motion images obtained through equidistant cylindrical transformation using a head-mounted display (HMD), the movement of this unnecessary region may not only cause discomfort but also VR sickness.

[0063] Figure 5 This is a flowchart illustrating the VR motion image generation process for generating VR motion images. Figure 5 The VR motion image generation process is implemented by the control unit 101 loading the program recorded in ROM 102 into RAM 103 and executing the program. For example, when the user instructs the editing device 100 to perform the function of generating VR motion images by performing an isometric cylindrical transformation to transform a circular fisheye image into an isometric cylindrical image, Figure 5 The operation begins.

[0064] In step S501, the control unit 101 reads common metadata (metadata common to all frames) stored in the file metadata area 302 of the motion image file. For example, the common metadata includes information such as the center position of the imaging circle, the radius of the imaging circle, and the radius of the region to be subjected to the isometric cylindrical transformation. The control unit 101 also reads parameter values ​​(manually corrected parameter values) set by the user using the operation unit 105 as metadata. This metadata is used in a later step to generate an isometric cylindrical image from the circular fisheye image.

[0065] In step S502, the control unit 101 performs a process of calculating the blur correction value (blur correction amount) for each frame (blur correction value calculation process). This will be discussed later. Figure 6 Describe the details of the blur correction value calculation process. The blur correction value (blur correction amount) is a correction value (correction amount) that reduces the blur of moving images caused by camera shake.

[0066] In step S503, the control unit 101 performs the process of generating mask parameters for each frame (mask parameter generation process). This will be discussed later. Figure 7 Describe the details of the mask parameter generation process. Mask parameters are parameters related to the mask to be superimposed on the equidistant cylindrical image. The mask parameter generation process can be interpreted as the process of generating the mask.

[0067] In step S504, the control unit 101 performs the process of generating equidistant cylindrical images for each frame (equidistant cylindrical image generation process). This will be referred to later. Figure 9 Describe the details of the process for generating images of equidistant cylinders.

[0068] Figure 6 It is shown in Figure 5 The flowchart shows the fuzzy correction value calculation process performed in step S502.

[0069] In step S601, the control unit 101 sets each of the variable i, which represents the frame number of the image data to be read, and the variable n, which represents the frame number of the image data for calculating the blur correction value, to 0.

[0070] In step S602, the control unit 101 determines, based on variable i, whether all frame image data (image data of the circular fisheye image) stored in the motion picture file has been read. If the control unit 101 determines that all frame image data has been read, the process proceeds to step S612. If the control unit 101 determines that at least one frame image data has not been read, the process proceeds to step S603.

[0071] In step S603, the control unit 101 reads the image data of the circular fisheye image of frame i (=frame i) from the video sampling area 3031 of the moving image file. Here, if the image data stored in the video sampling area 3031 is compressed, the control unit 101 decodes the image data. The format of the decoded image data is not particularly limited, and is, for example, RGB format image data.

[0072] In step S604, the control unit 101 analyzes the read image data (performs image processing on the image data) and extracts feature points from the circular fisheye image.

[0073] In step S605, the control unit 101 determines whether the sensor data of frame i is stored in the timing metadata sampling area 3033 of the motion image file. If the control unit 101 determines that the sensor data of frame i is stored, the process proceeds to step S606. If the control unit 101 determines that the sensor data of frame i is not stored, the process proceeds to step S607.

[0074] In step S606, the control unit 101 reads the sensor data of frame i (sensor data related to the movement of the camera during the capture of the circular fisheye image of frame i) from the timing metadata sampling area 3033.

[0075] In step S607, the control unit 101 updates variable i by incrementing variable i, which represents the frame number read, by 1.

[0076] In step S608, the control unit 101 determines whether the image data of the frame required to calculate the blur amount of the frame with frame number n (=frame n) has been read. For example, the blur amount is the magnitude of camera shake during a specific time period centered on the time of one frame, the amount of camera movement caused by camera shake, or the amount of movement of the subject in the image caused by camera shake. Therefore, for example, to calculate the blur amount of one frame, image data of frames one second before and after this frame are used. The required frame can vary depending on the variable n (frame n). For example, if there are no frames before frame n, the required frame may only include frames after frame n. If there are no frames after frame n, the required frame may only include frames before frame n. The time of frame n in the time period of the required frame may not be the center of that time period. If the control unit 101 determines that the image data of the required frame has been read, the process proceeds to step S609. If the control unit 101 determines that the image data of the required frame has not been read, the process proceeds to step S602.

[0077] In step S609, the control unit 101 calculates the blur amount of frame n. For example, the control unit 101 calculates the blur amount of frame n based on feature points and sensor data of frame n and the frames before and after it. Note that the control unit 101 may calculate the blur amount based solely on feature points or solely on sensor data.

[0078] The camera movement caused by unintended camera shake from the user differs from the movement of the camera as a photographic process. For example, in the event of camera shake, the camera moves back and forth, such as left and right or up and down, in a short period. Therefore, the control unit 101 can calculate the amount of blur based on the movement pattern of feature points during a period of time spanning multiple frames or the change pattern of values ​​obtained from a gyroscope sensor located in the camera during that period.

[0079] In step S610, the control unit 101 calculates a blur correction value for frame n based on the blur amount of frame n calculated in step S609, which is used to reduce (eliminate) the blur of the moving image corresponding to the blur amount. Various methods, such as those used for calculating camera shake correction, can be used to calculate the blur correction value. The control unit 101 calculates the inverse vector of the blur amount vector as the vector of the blur correction value.

[0080] In step S611, the control unit 101 updates the variable n by incrementing the variable n by 1.

[0081] As described above, in order to calculate the blur correction value of a frame, information related to the frames before and after that frame is also necessary. Therefore, even after all frames have been read, there are still several frames whose blur correction values ​​have not yet been calculated. Therefore, the following processing steps S612 to S615 are performed.

[0082] In step S612, the control unit 101 determines whether the blur correction values ​​for all frames have been calculated based on the variable n. If the control unit 101 determines that the blur correction values ​​for all frames have been calculated, the blur correction value calculation process ends. If the control unit 101 determines that the blur correction values ​​for at least one frame have not been calculated, the process proceeds to step S613.

[0083] In step S613, the control unit 101 calculates the blur amount of frame n in the same manner as in step S609.

[0084] In step S614, similar to step S610, the control unit 101 calculates a blur correction value for reducing (eliminating) the blur of the motion image corresponding to the blur amount based on the blur amount of frame n calculated in step S613.

[0085] In step S615, the control unit 101 updates the variable n by incrementing the variable n by 1.

[0086] Note that an example of the editing device 100 (control unit 101) calculating a blur correction value based on moving images and sensor data has been described, but the method of obtaining the blur correction value is not limited to this. For example, a camera may calculate the blur correction value during moving image photography, and the calculated blur correction value may be stored in a moving image file. The editing device 100 (control unit 101) can then read the blur correction value from the moving image file.

[0087] Figure 7 It is shown in Figure 5 The flowchart below shows the mask parameter generation process performed in step S503. In this process, a mask (mask parameters) is generated based on the time variation of the blur correction value. In the first embodiment, the mask parameters include the mask movement amount and the mask size.

[0088] In step S701, the control unit 101 reads the blur correction values ​​for all frames calculated in step S502. In step S502, it is assumed that multiple blur correction values ​​corresponding to multiple directions are calculated for each frame. For example, the pan, tilt, and yaw axes are determined based on a predetermined camera pose. Then, in step S502, three blur correction values ​​are calculated, corresponding to the pan correction value, tilt correction value, and yaw correction value respectively, about the rotation direction around the pan axis, the rotation direction around the tilt axis, and the rotation direction around the yaw axis. Similarly, multiple correction values ​​corresponding to multiple directions are used. An example using one type of correction value will be described below for simplicity.

[0089] In step S702, the control unit 101 uses the blur correction values ​​of all frames read in step S701 to calculate the mask movement amount for each frame. The mask movement amount is the amount of movement of the area (opening area) in the equidistant cylindrical image that is not covered by the mask, and for example, the amount of movement of the center position of the opening area relative to the center position of the equidistant cylindrical image. For example, the control unit 101 calculates the mask movement amount as the moving average of the blur correction values. Figure 8 In the diagram, the horizontal axis represents time, and the vertical axis represents the blur correction value. The solid line 801 represents the time variation of the blur correction value calculated in step S502, and the dashed line 802 represents the time variation of the moving average (mask movement) of the blur correction value. If the moving average period used to average the blur correction value is long, the time variation of the moving average (mask movement) becomes gradual; conversely, if the moving average period is short, the time variation of the moving average (mask movement) becomes abrupt. There is no particular limitation on the moving average period, and it is, for example, a period of 50 frames. Preferably, the moving average period is set such that no discomfort occurs during movement in the opening area.

[0090] Note that the mask shift can be a simple moving average calculated without weighting, or a weighted moving average calculated with weighting. Other values ​​corresponding to the moving average can be obtained as the mask shift. Frequency filters, such as low-pass filters, can be used to obtain a value representing the time variation that smooths out the time variation of the blur correction value as the mask shift. There are no particular restrictions on the method for determining the mask shift, as long as the aperture region moves more slowly than the region undergoing the equidistant cylindrical transformation.

[0091] In step S703, the control unit 101 sets the variable j, which represents the absolute value of the difference between the calculated blur correction value and the mask movement amount (moving average value), to 0.

[0092] In step S704, the control unit 101 determines, based on variable j, whether the difference between the mask movement amount (movement average) calculated in step S702 and the blur correction value calculated in step S502 has been calculated for all frames. If the control unit 101 determines that the difference has been calculated for all frames, the mask parameter generation process ends. If the control unit 101 determines that the difference has not been calculated for at least one frame, the process proceeds to step S705.

[0093] In step S705, the control unit 101 calculates the difference (deviation) between the mask movement amount (moving average value) and the blur correction value of the frame with frame number j (= frame j).

[0094] In step S706, the control unit 101 determines whether the difference calculated in step S705 is greater than the maximum value of the differences calculated up to the previous time. If the control unit 101 determines that the difference is greater than the maximum value, the process proceeds to step S707. Furthermore, if the control unit 101 calculates the difference for the first time, the process proceeds to step S707. If the control unit 101 determines that the difference is equal to or less than the maximum value, the process proceeds to step S709.

[0095] In step S707, the control unit 101 determines the difference (maximum value) calculated in step S705 as the mask size. The mask size is the size of the aforementioned opening region.

[0096] In step S708, the control unit 101 updates variable j by incrementing variable j by 1.

[0097] In step S709, similarly, the control unit 101 updates variable j by incrementing variable j by 1.

[0098] By performing the processing steps S704 to S709 for all frames, the maximum value of the difference calculated for all frames (the difference between the mask shift amount (moving average) and the blur correction value) is determined as the mask size. Note that other values ​​corresponding to the maximum value of the difference between the mask shift amount (moving average) and the blur correction value can also be determined as the mask size.

[0099] exist Figure 8 In the example, after determining difference 803 as the mask size, the mask size is updated to difference 804, and then to difference 808. Finally, difference 808 is determined as the mask size.

[0100] In the first embodiment, since the mask movement amount is determined for each frame, it can be changed; however, since only one mask size is determined for a moving image, it remains unchanged. Because the mask size does not change, the field of view does not narrow or widen midway through the moving image. However, the method for determining the mask movement amount and mask size is not limited to the method described above, and the mask movement amount and mask size can be determined such that the mask size changes according to the change in the mask movement amount (movement of the opening region).

[0101] Figure 9 It is shown in Figure 5 The flowchart shows the process of generating an equidistant cylindrical image in step S504.

[0102] In the process of generating isometric cylindrical images, an image with reduced (eliminated) blur from the moving image is obtained based on a blur correction value. In VR motion images using isometric cylindrical images with reduced (eliminated) blur, for example, the relative position of the stationary subject with respect to the isometric cylindrical image remains (essentially) unchanged. Therefore, when viewing VR motion images using an HMD, the user can view the stationary subject in a certain direction.

[0103] Furthermore, in the equidistant cylindrical image generation process, a mask is superimposed on the equidistant cylindrical image based on the mask parameters generated in step S503. By superimposing the mask on the region including unnecessary areas to hide the unnecessary areas (movement) corresponding to the invalid areas (areas outside the imaging circle) of the circular fisheye image, flickering in the periphery of the equidistant cylindrical image can be suppressed. As a result, user discomfort and VR motion sickness can be suppressed when the user views VR motion images using an HMD.

[0104] In the first embodiment, because the opening region of the mask moves, a wider opening region can be set compared to the case where the opening region is fixed. Therefore, an image with a small unnecessary region and a wide effective region can be obtained as an equidistant cylindrical image after mask stacking.

[0105] In step S901, the control unit 101 sets the variable k, which represents the frame number of the generated equidistant cylindrical image, to 0.

[0106] In step S902, the control unit 101 determines, based on variable k, whether equidistant cylindrical images have been generated for all frames. If the control unit 101 determines that equidistant cylindrical images have been generated for all frames, the equidistant cylindrical image generation process ends. If the control unit 101 determines that equidistant cylindrical images have not been generated for at least one frame, the process proceeds to step S903.

[0107] In step S903, the control unit 101 obtains the fuzz correction value of the frame (= frame k) with frame number k calculated in step S502.

[0108] In step S904, the control unit 101 acquires the mask parameters (mask movement amount and mask size of frame k) calculated in step S503. The processing in steps S902 to S910 is performed for each frame, but information related to the mask size can be acquired only once.

[0109] In step S905, the control unit 101 sets the coordinates of the processing object (the coordinates of the pixels of the equidistant cylindrical image of frame k) (X, Y) to the initial coordinates (0, 0).

[0110] In step S906, the control unit 101 determines whether all pixels of frame k have been generated. If the control unit 101 determines that all pixels have been generated, the process proceeds to step S910; if the control unit 101 determines that at least one pixel has not been generated, the process proceeds to step S907.

[0111] In step S907, the control unit 101 performs a process (coordinate calculation processing) to calculate the coordinates of the circular fisheye image corresponding to the coordinates of the equidistant cylindrical image. This will be discussed later. Figure 10 The details of the coordinate calculation process are described. Typically, in a circular fisheye image, when the focal length is defined as f and the viewing angle as θ, the imaging position is represented by f × θ. This projection method is a lens projection method, such as a so-called equidistant projection lens. In an equidistant cylindrical image, the azimuth angle λ and elevation angle ψ in three-dimensional space are drawn in the X and Y directions, respectively. Based on these, the xy coordinates in the circular fisheye image can be calculated from the XY coordinates in the equidistant cylindrical image. Furthermore, by using the blur correction value obtained in step S903, the xy coordinates of the circular fisheye image corresponding to the XY coordinates of the blurred equidistant cylindrical image (reducing the blur of the moving image) can be calculated.

[0112] In step S908, the control unit 101 performs pixel processing (pixel generation processing) on ​​the equidistant cylindrical image (the equidistant cylindrical image after blur correction and mask overlay) of frame k. (See later for further details.) Figure 11 Describe the details of the pixel generation process.

[0113] In step S909, the control unit 101 updates the coordinates (X, Y) of the object to be processed to the next coordinate.

[0114] In step S910, the control unit 101 updates variable k by incrementing variable k by 1.

[0115] Figure 10 It is shown in Figure 9 The flowchart shows the coordinate calculation process performed in step S907.

[0116] In step S1001, the control unit 101 acquires the set coordinates (X, Y) of the processing object.

[0117] In step S1002, the control unit 101 calculates the coordinates (x1, y1, z1) in three-dimensional space corresponding to the coordinates (X, Y) obtained in step S1001. The coordinates (x1, y1, z1) can be calculated based on the relationship between the azimuth angle λ and the elevation angle ψ in three-dimensional space, respectively, drawn in the X and Y directions.

[0118] In step S1003, the control unit 101 corrects the coordinates (x1, y1, z1) to coordinates (x2, y2, z2) based on the fuzz correction value obtained in step S903.

[0119] In step S1004, the control unit 101 calculates the coordinates (x, y) of the circular fisheye image corresponding to the coordinates (X, Y) of the equidistant cylindrical image obtained in step S1003 based on the corrected coordinates (x2, y2, z2).

[0120] Figure 11 It is shown in Figure 9 The flowchart shows the pixel generation process performed in step S908.

[0121] In step S1101, the control unit 101 acquires the set coordinates (X, Y) of the processing object (coordinates (X, Y) of the equidistant cylindrical image after blur correction).

[0122] In step S1102, the control unit 101 acquires the coordinates (x, y) of the circular fisheye image calculated in step S907.

[0123] In step S1103, the control unit 101 determines whether the coordinates (X, Y) obtained in step S1101 are within the mask region based on the mask parameters obtained in step S904. If the control unit 101 determines that the coordinates (X, Y) are within the mask region, the process proceeds to step S1104. If the control unit 101 determines that the coordinates (X, Y) are outside the mask region (within the opening region), the process proceeds to step S1105.

[0124] In step S1104, the control unit 101 sets the brightness coefficient to 0.

[0125] In step S1105, the control unit 101 sets the brightness coefficient to 1.

[0126] In step S1106, the control unit 101 multiplies the brightness value of the pixel at coordinates (x, y) obtained in step S1102 in the circular fisheye image of frame k by the brightness coefficient set in step S1104 or S1105. If the coordinates (X, Y) are within the mask area, a black pixel value (the pixel value of the mask) is obtained. As a result, the pixel value at coordinates (X, Y) of the equidistant cylindrical image after blur correction and mask overlay is obtained. Note that the mask color is not limited to black.

[0127] Note that although examples of binary values ​​of 0 or 1 for the luminance coefficient have been described, the luminance coefficient can be changed incrementally or continuously, allowing the luminance value to change smoothly at the boundary between the inner and outer sides of the mask area. For example, the luminance coefficient can be determined based on the distance from the coordinate corresponding to the center of the lens to the coordinate (X, Y), or the luminance coefficient can be determined based on the distance from the boundary between the inner and outer sides of the mask area to the coordinate (X, Y).

[0128] The mask generation method according to the first embodiment will be described.

[0129] Figure 12 This is a schematic diagram illustrating a circular fisheye image. Figure 12 In this image, a mask with an opening region of radius R (mask radius R) is superimposed on a circular fisheye image. The black area is the mask region, and the white area is the opening region. To obtain a wide effective area, it is preferable to generate a mask that exposes the entire imaging circle. Ghosting and glare may appear in the area outside the imaging circle (ineffective area). To hide ghosting and glare, it is preferable to generate a mask that covers the entire ineffective area. Considering these factors, it is preferable to generate a mask such that the opening region coincides with the area of ​​the imaging circle. However, when generating such a mask, as mentioned above, when performing blur correction, the unnecessary region corresponding to the ineffective region is set as part of the equidistant cylindrical image, and the unnecessary region moves according to the change in blur correction value, causing flickering in the periphery of the equidistant cylindrical image.

[0130] Consider the case where the distance from the center of the circular fisheye image is Δmax, corresponding to the maximum absolute value of the blur correction value. In this case, unwanted areas can be hidden by setting a value of Δmax that is smaller than the imaging circle radius imgR. Figure 8 In this context, the absolute value of the blur correction value 805 is the maximum value. Figure 8 In the diagram, the blur correction value 805 is represented by an angle, but the distance Δmax can be derived from the projection formula f×θ. However, when a value smaller than the imaging circle radius imgR is set as the mask radius R, the mask radius R becomes extremely small, and the effective area becomes extremely small.

[0131] Therefore, in the first embodiment, the opening region of the mask is moved by a mask movement amount such that the opening region moves more smoothly compared to the movement of the object region under equidistant cylindrical transformation, where the mask movement amount is a moving average of the blur correction value. Then, the maximum value Δθmax, which is smaller than the imaging circle radius imgR than the difference between the moving average and the blur correction value, is set as the mask radius R (mask size). The value obtained by adding the mask movement amount and the mask size is consistent with the maximum value of the blur correction amount. This allows unnecessary regions to be hidden while suppressing the effective region from becoming too small.

[0132] exist Figure 8 In the diagram, the dashed line 802 represents the time variation of the moving average (mask movement amount) of the blur correction value, and the difference 808 between the moving average (mask movement amount) and the blur correction value is the mask size. The dashed line 806a represents the time variation of the value obtained by offsetting the mask movement amount by the mask size in the negative direction, and the dashed line 806b represents the time variation of the value obtained by offsetting the mask movement amount by the mask size in the positive direction. The dashed lines 806a and 806b correspond to the case of using the mask according to the first embodiment. The double-dash line 807a represents the value obtained by adding a negative sign to the absolute value (maximum value) of the blur correction value 805, and the double-dash line 807b represents the value obtained by adding a positive sign to the absolute value (maximum value) of the blur correction value 805. The double-dash lines 807a and 807b correspond to the case where both the opening area and the mask size are fixed (comparative example). In the first embodiment, since the blur correction value is not lower than the dashed line 806a and not higher than the dashed line 806b, unnecessary areas can be completely hidden. Similarly, in the comparative example, since the blur correction value is not lower than the double-dotted line 807a and not higher than the double-dotted line 807b, unnecessary areas can be completely hidden. As described above, in both the first embodiment and the comparative example, unnecessary areas can be completely hidden. However, in the comparative example, as the absolute value (maximum value) of the blur correction value 805 increases, the mask size decreases, and the absolute value (maximum value) of the blur correction value 805 is greater than the difference 808. Therefore, in the first embodiment, the opening area (effective area) can be larger than the opening area in the comparative example.

[0133] The images and regions according to the first embodiment will be described.

[0134] Figure 13A This is a schematic diagram illustrating a circular fisheye image. Region 1301 is the region inside the imaging circle (the region inside the imaging circle). Region 1303 is the region outside the imaging circle (the region outside the imaging circle). The dashed line 1302 is the boundary line between the region inside the imaging circle 1301 and the region outside the imaging circle 1303.

[0135] Figure 13B This is a schematic diagram showing an equidistant cylindrical image. Region 1301e is the effective region and the opening region of the mask. Position 1305e is the center position of the opening region 1301e (effective region). Region 1304e is the region covered by the mask (mask region). Solid line 1303e is the boundary line between the opening region 1301e (effective region) and the mask region 1304e. Dashed line 1302e corresponds to the boundary line 1302 between the region 1301 inside the imaging circle and the region 1303 outside the imaging circle. As described above, the boundary line 1302e is moved according to blur correction. The mask is superimposed such that the boundary line 1302e is not located inside the opening region 1301e (inside the boundary line 1303e between the opening region 1301e and the mask region 1304e). In this way, flickering and other issues in the periphery of the equidistant cylindrical image can be suppressed.

[0136] Figure 13C It shows the connection. Figure 13B A schematic diagram of an equidistant cylindrical image of a frame following the one in the diagram. The center position 1305e of the opening region 1301e moves smoothly according to the amount of mask movement. Therefore, even when a relatively large opening region 1301e is set, the boundary line 1302e can be prevented from being located inside the opening region 1301e.

[0137] As described above, according to the first embodiment, a mask is generated such that the regions in the equidistant cylindrical image not covered by the mask are moved by a movement amount based on a time-varying change in the blur correction amount. In this way, unwanted regions such as black areas can be effectively reduced from the equidistant cylindrical image. An image with small unwanted regions and a wide effective region can be obtained as the equidistant cylindrical image after mask stacking.

[0138] Note that although an example of transforming a circular fisheye image into an isometric cylindrical image using an isometric cylindrical transformation has been described, the image before transformation is not limited to a circular fisheye image, and the image after transformation is not limited to an isometric cylindrical image. The transformation is not limited to the isometric cylindrical transformation and can be other geometric transformations. Furthermore, when multiple images corresponding to multiple optical systems are obtained as images before transformation, geometric transformations, mask generation, and mask overlay can be performed on each of these multiple images. Second Embodiment

[0139] A second embodiment of this disclosure will be described. In the following, descriptions of points identical to those in the first embodiment (configuration and processing, etc.) will be appropriately omitted. In the first embodiment, the mask size is determined by the editing device 100. In the second embodiment, the mask size is specified by the user. The amount of mask movement is then determined based on the time-varying changes in the mask size and the blur correction amount.

[0140] Figure 14 This is a block diagram illustrating the configuration of an editing device 100, which is an example of an image processing apparatus according to a second embodiment. Figure 1 Similarly, the editing device 100 (the editing device 100 according to the first embodiment) in the middle, Figure 14 The editing device 100 includes a control unit 101, a ROM 102, a RAM 103, a storage medium 104, an operation unit 105, a display unit 106, a communication unit 107, and a system bus 108. Furthermore, Figure 14 The editing device 100 includes a mask designation unit 1409. The mask designation unit 1409 receives user operations (instructions from the user) for specifying the mask size.

[0141] Figure 15 This is a flowchart illustrating the mask parameter generation process according to the second embodiment. Figure 15 In the generation process Figure 5 The process is carried out in step S503.

[0142] In step S1501, the control unit 101 reads the blur correction values ​​of all frames calculated in step S502.

[0143] In step S1502, the control unit 101 obtains information related to the mask size specified by the user from the mask designation unit 1409. The user specifies a small mask size when they want to reduce the movement of the opening area of ​​the mask, and a large mask size when they want to expand the opening area.

[0144] In step S1503, the control unit 101 sets the moving average time period P to 2. This moving average time period P is the time period used to average the fuzz correction values. Moving average time period P=2 means that the moving average time period is the time period of two frames that include the frame from which the moving average of the fuzz correction values ​​is calculated.

[0145] In step S1504, the control unit 101 uses the blur correction values ​​of all frames read in step S1501 and the set moving average time period P to calculate the moving average of the blur correction values ​​of each frame.

[0146] In step S1505, the control unit 101 calculates the maximum value of the difference (absolute value) between the fuzzy correction value and the moving average value, as in the first embodiment.

[0147] In step S1506, the control unit 101 determines whether the maximum value calculated in step S1505 is greater than the mask size specified by the user. If the control unit 101 determines that the maximum value is greater than the mask size, the process proceeds to step S1508. If the control unit 101 determines that the maximum value is less than or equal to the mask size, the process proceeds to step S1507.

[0148] In step S1507, the control unit 101 updates the moving average period P by incrementing the moving average period P by 1.

[0149] In step S1508, the control unit 101 updates the moving average period P by subtracting 1 from the moving average period P.

[0150] In step S1509, the control unit 101 uses the blur correction values ​​of all frames read in step S1501 and the set moving average time period P to calculate the moving average of the blur correction values ​​of each frame as the mask movement amount of each frame.

[0151] As described above, according to the second embodiment, in a user-specified mask size configuration, the same effect can be obtained through the same operation as in the first embodiment. Furthermore, since the moving average period P is set to maximize the difference (absolute value) between the blur correction value and the moving average within a range not exceeding the mask size, the movement of the mask's opening region can be minimized. Third Embodiment

[0152] A third embodiment of this disclosure will be described. In the following, descriptions of points identical to those in the first embodiment (configuration and processing, etc.) will be appropriately omitted.

[0153] Figure 16 This is a block diagram illustrating the configuration of a photographic apparatus 1600, which is an example of an image processing device according to a third embodiment. The photographic apparatus 1600 includes a camera body 1610 and an interchangeable lens 1620.

[0154] The interchangeable lens 1620 is detachable from the camera body 1610 and includes a right optical system 1621R, a left optical system 1621L, a lens mount 1622, and a lens system control unit 1623. The lens system control unit 1623 performs drive control of the right optical system 1621R and the left optical system 1621L, etc. The right optical system 1621R and the left optical system 1621L are arranged to be spaced apart from each other by a predetermined distance (baseline length) in the horizontal direction. In the imaging element 1611 included in the camera body 1610, two imaging rings corresponding to the right optical system 1621R and the left optical system 1621L are formed. When viewed from the imaging element 1611 side, the right optical system 1621R is positioned on the right and the left optical system 1621L is positioned on the left. Therefore, moving or still images captured by the right optical system 1621R can be recorded as images for the right eye, and moving or still images captured by the left optical system 1621L can be recorded as images for the left eye. Using a 3D display or VR goggles, the images for the right and left eyes are played, so that video based on the right-eye image is presented to the user's right eye, and video based on the left-eye image is presented to the user's left eye. In this way, the user can view the video in stereoscopic form.

[0155] The camera body 1610 includes an image sensor 1611, an A / D converter 1612, an image processing unit 1613, a display unit 1614, an operation unit 1615, a recording unit 1616, a main system control unit 1617, a storage unit 1618, and a camera mount 1619. When the lens mount 1622 of the interchangeable lens 1620 is mounted on the camera mount 1619 of the camera body 1610, the main system control unit 1617 and the lens system control unit 1623 are electrically connected.

[0156] The imaging element 1611 converts the formed image of the subject (optical signal) into an analog electrical signal. The A / D converter 1612 converts the analog electrical signal output from the imaging element 1611 into a digital electrical signal (image signal). The image processing unit 1613 performs various types of image processing on the digital electrical signal (image signal) output from the A / D converter 1612.

[0157] Display unit 1614 displays various types of information. Display unit 1614 may be, for example, an electronic viewfinder or a rear display unit, and includes a liquid crystal panel or an organic EL panel as the display panel. Operation unit 1615 is a user interface that receives user operations on the photographic device 1600. Note that if display unit 1614 includes a touch panel, the touch panel is also one of the operation units 1615.

[0158] Recording unit 1616 records various types of data, such as image data processed by image processing unit 1613, in storage unit 1618, etc. Storage unit 1618 stores individual identification information, such as individual information of photographic device 1600 and manufacturing error information. Typically, the interchangeable lens 1620 generates errors (manufacturing errors) during its manufacturing process. Due to this manufacturing error, for example, when the interchangeable lens 1620 is attached to camera body 1610, tilt or displacement of the right optical system 1621R relative to the imaging element 1611 occurs. Similarly, tilt or displacement of the left optical system 1621L relative to the imaging element 1611 occurs. In the third embodiment, body system control unit 1617 or image processing unit 1613 acquires and uses information related to manufacturing errors to convert a circular fisheye image into an equidistant cylindrical image that mitigates the effects of manufacturing errors.

[0159] The main system control unit 1617 is, for example, a CPU, and controls the entire camera device 1600 as a whole.

[0160] Figure 17 These are images captured by photographic device 1600. Figure 17 The image has two regions, representing a circular fisheye image captured by two optical systems: a right optical system 1621R and a left optical system 1621L. Here, it is assumed that the viewing angles of each optical system in the right optical system 1621R and the left optical system 1621L are very wide, 180 degrees or greater. Therefore, a photographic device (e.g., another optical system, etc.) is captured within the circular fisheye image of one optical system. Figure 17 In the image, regions 1701 and 1702 are areas where adjacent optical systems are captured. Here, the circular fisheye image corresponding to the right optical system 1621R is arranged on the left, and the circular fisheye image corresponding to the left optical system 1621L is arranged on the right. The reason for this arrangement is that... Figure 17 The image is obtained by rotating the image by 180 degrees, which is achieved by vertically and horizontally reversing the optical image through the right optical system 1621R to the right and vertically and horizontally reversing the optical image through the left optical system 1621L to the left. Therefore, regions 1701 and 1702 are arranged on the outside.

[0161] Although the same subject as the subject in region 1704 should be photographed in region 1701, an adjacent optical system (left optical system 1621L) is used in region 1701, resulting in different subjects being photographed in regions 1701 and 1704. Similarly, although the same subject as the subject in region 1703 should be photographed in region 1702, an adjacent optical system (right optical system 1621R) is used in region 1702, resulting in different subjects being photographed in regions 1702 and 1703. As described above, when different subjects appear in two regions where similar subjects should appear, stereoscopic vision cannot be achieved, or uncomfortable stereoscopic vision is achieved.

[0162] Therefore, in the third embodiment, the area where the imaging device 1600 is located is designated as an unnecessary area, and a mask is generated to cover this unnecessary area. By covering not only areas 1701 and 1702 but also areas 1703 and 1704 with the mask, discomfort in stereoscopic vision can be further reduced. In the third embodiment, isometric cylindrical transformation, mask generation, and mask superposition are performed on each optical system in the right optical system 1621R and the left optical system 1621L. The mask can be generated in a manner similar to the first and second embodiments, and the isometric cylindrical image superimposed with the mask is generated by the main system control unit 1617 or the image processing unit 1613. However, for the right optical system 1621R, the mask movement amount and mask size are further considered based on the distance X1 from the center of the imaging circle to areas 1701 (and 1703), such that the opening area of ​​the mask does not include areas 1701 (and 1703). For example, a mask size smaller than the mask size of the first embodiment by a distance X1 is set. Similarly, for the left optical system 1621L, the mask movement amount and mask size are further set by considering the distance X2 from the center of the imaging circle to region 1702 (and region 1704), such that the opening region of the mask does not include region 1702 (and region 1704).

[0163] Furthermore, as described above, in the third embodiment, the circular fisheye image is transformed into an equidistant cylindrical image that mitigates the impact of manufacturing errors by using information related to manufacturing errors. Therefore, the main system control unit 1617 or the image processing unit 1613 generates a mask that mitigates the impact of manufacturing errors based on the information related to manufacturing errors.

[0164] As described above, according to the third embodiment, the area where the photographic device was photographed can be effectively reduced from the geometrically transformed image. Furthermore, by taking manufacturing errors into account during geometric transformation and mask generation, a suitable image that mitigates the effects of manufacturing errors can be obtained as the image after geometric transformation and mask overlay.

[0165] Note that the various types of control described above can be performed by a single piece of hardware (e.g., a processor or circuitry) or by processing in other ways. Processing can be distributed among multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits), thereby performing control over the entire device.

[0166] Furthermore, the processors mentioned above are processors in a broad sense, encompassing both general-purpose and special-purpose processors. Examples of general-purpose processors include central processing units (CPUs), microprocessor units (MPUs), and digital signal processors (DSPs). Examples of special-purpose processors include graphics processing units (GPUs), application-specific integrated circuits (ASICs), and programmable logic devices (PLDs). Examples of PLDs include field-programmable gate arrays (FPGAs) and complex programmable logic devices (CPLDs).

[0167] The above embodiments (including variations) are merely examples. Any configurations obtained by appropriately modifying or changing some configurations of the embodiments within the scope of this disclosure are also included in this disclosure. This disclosure also includes other configurations obtained by appropriately combining various features of the embodiments.

[0168] According to this disclosure, unwanted areas such as black areas can be effectively reduced from geometrically transformed images such as isometric cylindrical projection images. Other embodiments

[0169] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0170] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: A motion image acquisition unit, configured to acquire photographic motion images; An acquisition unit is configured to acquire a blur correction amount for reducing the blur of the moving image; A processing unit configured to perform geometric transformations on images of each frame of the motion image, wherein the processing unit obtains an image with reduced blur as a geometrically transformed image based on the blur correction amount; A generation unit is configured to generate a mask based on the time variation of the blur correction amount; as well as An overlay unit is configured to overlay the mask onto the geometrically transformed image. The generation unit generates the mask such that the region in the geometrically transformed image not covered by the mask has a size specified by the user, and the movement is based on the time variation of the size and the blur correction amount.

2. The image processing apparatus according to claim 1, wherein, The generation unit generates the mask such that the region not covered by the mask moves by an amount corresponding to the moving average of the blur correction amount over a time period based on the size.

3. The image processing apparatus according to claim 2, wherein, As the time period in which the moving average can be obtained, the generation unit determines the time period in which the difference between the fuzz correction amount and the moving amount becomes the largest within a range not exceeding the size.

4. The image processing apparatus according to any one of claims 1 to 3, wherein, The generation unit generates the mask such that the size of the area not covered by the mask remains unchanged.

5. The image processing apparatus according to any one of claims 1 to 3, wherein, Each frame of the moving image has a region inside the imaging circle and a region outside the imaging circle, and The mask covers the area outside the imaging circle.

6. The image processing apparatus according to any one of claims 1 to 3, wherein, The images of each frame of the moving image are images captured by a photographic device that takes pictures of the moving image, and The mask is a mask used to cover the area where the photographic device is being photographed.

7. The image processing apparatus according to any one of claims 1 to 3, further comprising an information acquisition unit configured to acquire information relating to manufacturing errors of the optical system for photographing the moving image, wherein, The processing unit, based on information related to the manufacturing error, acquires an image that mitigates the impact of the manufacturing error as the geometrically transformed image. The generation unit generates the mask that mitigates the impact of the manufacturing error based on information related to the manufacturing error.

8. The image processing apparatus according to any one of claims 1 to 3, wherein, The images in each frame of the motion picture include multiple images corresponding to multiple optical systems. The processing unit performs the geometric transformation on each of the plurality of images, and The generation unit generates the mask for each of the plurality of images.

9. The image processing apparatus according to any one of claims 1 to 3, wherein, The images in each frame of the motion picture include circular fisheye images, which have an area inside the imaging circle and an area outside the imaging circle, and The processing unit performs an isometric cylindrical transformation on the circular fisheye image as a geometric transformation.

10. An image processing method, comprising: The motion image acquisition step is used to acquire photographic motion images; The acquisition step is used to acquire a blur correction amount for reducing the blur of the moving image; The processing step is used to perform geometric transformation on the images of each frame of the motion image, wherein in the processing step, an image with reduced blur is obtained based on the blur correction amount as the geometrically transformed image; A generation step is used to generate a mask based on the time variation of the blur correction amount; as well as The overlay step is used to overlay the mask onto the geometrically transformed image. In the generation step, the mask is generated such that the regions in the geometrically transformed image not covered by the mask have a size specified by the user, and the movement is based on the time variation of the size and the blur correction amount.

11. A computer program product comprising a program that causes a computer to perform the steps of the image processing method according to claim 10.

12. A computer-readable storage medium storing a program that causes a computer to perform the steps of the image processing method according to claim 10.

Images