Image processing apparatus, image processing method, and program

Abstract

To obtain the effect of cyclic noise reduction processing for the first frame of a moving image.SOLUTION: An image processing device includes means for acquiring a moving image and means for applying cyclic noise reduction processing to the first frame using one or more frames after the first frame on playback when the moving image is played back.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Shooting can be performed using a camera capable of shooting a RAW format video (hereinafter referred to as a RAW video), and various edits can be performed on the obtained RAW video. Information regarding the image signal generated by the image sensor of the camera is recorded as it is in each frame constituting the RAW video. The RAW video can have its image quality freely adjusted by performing development processing using a camera, a personal computer, or the like.

[0003] On the other hand, a technique for reducing noise from each frame of a video using circular noise reduction processing is used. Circular noise reduction processing is a technique for reducing noise by utilizing the correlation between a developed frame (image) and a frame (image) stored in memory that is more than one frame period before.

[0004] As related techniques, the techniques of Patent Documents 1 and 2 have been proposed. The technique of Patent Document 1 performs frame noise reduction based on frame correlation. The technique of Patent Document 2 applies a scene change effect to the subsequent video of the first video and the preceding video of the second video only for a period determined that the scene continues when the first video and the second video are played back continuously.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] As described above, the cyclic noise reduction process is a technique for reducing noise by using the correlation with frames more than one frame period before. Therefore, since there is no frame before the first frame among the frames constituting the video, the cyclic noise reduction process cannot be performed on the first frame. For this reason, there is a problem that the image quality of the first frame deteriorates. The technique of Patent Document 1 does not perform noise reduction on the first frame. Further, Patent Document 2 is not related to the cyclic noise reduction process. Such a problem can also occur in videos other than RAW videos.

[0007] Therefore, an object of the present invention is to enable the effect of the cyclic noise reduction process to be obtained for the first frame of a video.

Means for Solving the Problems

[0008] To achieve the above object, an image processing apparatus according to the present invention includes means for acquiring a video, and when reproducing the video, using one or more frames after the first frame at the time of reproduction, for the first frame te no is reduction processing and performs cyclic noise reduction processing on frames other than the head frame using one or more frames before the frame and means for performing such processing.

Effects of the Invention

[0009] According to the present invention, it is possible to obtain the effect of the cyclic noise reduction process for the first frame of a video.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0011] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment.

[0012] <First Embodiment> FIG. 1 is a block diagram showing an example of the configuration of the imaging device 100. The imaging device 100 includes a lens 101, a lens driving unit 102, a mechanical shutter 103 (mechanical shutter), a diaphragm 104, and a shutter / diaphragm driving unit 105 (mechanical shutter / diaphragm driving unit). The imaging device 100 includes an imaging element 106, an image signal processing circuit 107, a first memory unit 108, a control unit 109, and a recording medium control I / F unit 110. The imaging device 100 includes a display unit 111, a recording medium 112, an external I / F unit 113, a second memory unit 114, and an operation unit 115. "I / F" represents an interface.

[0013] The lens 101 is driven and controlled for zooming, focusing, etc. by the lens driving unit 102. The subject image from the lens 101 is adjusted to an appropriate light amount by the diaphragm 104 and forms an image on the imaging surface of the imaging element 106. The subject image formed on the imaging surface of the imaging element 106 is photoelectrically converted. The photoelectrically converted subject image is subjected to gain adjustment processing, A / D conversion processing for converting an analog signal to a digital signal, etc. The digital signal converted from the analog signal is sent to the image signal processing circuit 107 as, for example, R, Gr, Gb, B image signals. The image signal processing circuit 107 performs various image signal processes such as development processing, low-pass filter processing for reducing noise, shading processing, white balance (WB) processing, and cyclic noise reduction processing. The image signal processing circuit 107 further performs various corrections and compression of image data, etc.

[0014] The mechanical shutter 103 and the aperture 104 are driven and controlled by the shutter - aperture drive unit 105. The control unit 109 performs overall control and various calculations of the imaging device 100. The first memory unit 108 stores image data. The recording medium control I / F unit 110 records and reads out image data to and from the recording medium 112. The display unit 111 displays image data and various information. The recording medium 112 is a removable recording medium such as a semiconductor memory and records image data and various information. The external I / F unit 113 is an interface for communicating with an external device such as an external computer. The second memory unit 114 stores the calculation results in the control unit 109.

[0015] The control unit 109 is realized by, for example, a CPU or the like. By the CPU that performs the functions of the control unit 109 executing a predetermined program, the control of each embodiment is realized. The functions of the control unit 109 may be realized by a predetermined circuit. The control unit 109 functions as an image processing device. The first memory unit 108 and the second memory unit 114 may be separate memories as shown in FIG. 1, or may be one memory.

[0016] The operation unit 115 is a member that can be operated by the user. Information regarding the driving conditions of the imaging device 100 instructed by the user via the operation unit 115 is sent to the control unit 109. The control unit 109 performs overall control of the imaging device 100 based on the sent information. The operation unit 115 and the display unit 111 may be an integrated touch - panel display. The control unit 109 includes a cyclic frame holding control unit 116 and a cyclic frame determination unit 117. The cyclic frame holding control unit 116 causes the first memory unit 108 to hold the image data of the input frame used for the cyclic noise reduction process. Hereinafter, the cyclic noise reduction process may be referred to as the cyclic NR process. The cyclic frame determination unit 117 determines the image data of the frame used for the cyclic NR process based on the frame number. The image data of the frame used for the cyclic NR process may be determined based on other than the frame number.

[0017] Figure 2 is a diagram showing an example of the configuration of a video file. The video file includes various boxes. The ftyp box 200 indicates file format compatibility. The moov box 201 is a box in which management information necessary for playback and thumbnail images are stored. The XMP box 202 is a box in which XMP (Extensibile Metadata Platform) is stored, and arbitrary metadata can be set. The uuid box 203 is a box to which arbitrary information can be added. In the case of a RAW video, a preview image 220 is stored in the uuid box 203. Circulation RAW data can also be stored in the uuid box 203. Encoded video data, audio data, time code data, and frame unit metadata are stored in the mdat box 204.

[0018] The configuration of the moov box 201 will be described. Arbitrary information such as a thumbnail image 218 to be displayed during video playback and management information 219 to be used during playback can be added and stored in the uuid box 213 included in the moov box 201. The track boxes 205 to 208 are boxes in which management information regarding video data, audio data, time code data, frame unit metadata, etc. is stored. The stsz boxes 209 to 212 are boxes that store the data size for each encoding unit of video data, audio data, time code data, frame unit metadata, etc. Information indicating the storage position of the mdat 204 of the video data, audio data, time code data, and frame unit metadata stored in the track boxes 205 to 208 is stored in the stco boxes 214 to 217. Each data is stored in the mdat box 204 in a unit called a chunk, which is composed of one or more encoding units.

[0019] The configuration of the mdat box 204 will be described. The metadata 230 to 241 are data such as video data, audio data, time code data, and data in units of frames stored in the mdat box 204. Each data can be accessed in units of chunks based on the values described in the stco box. For example, the metadata 230 (CV1) can be traced from CV1 of the stco box 214. The configuration of the video file is not limited to the example in FIG. 2.

[0020] FIG. 3 is a flowchart showing an example of the flow of RAW video playback processing in the first embodiment. The processing of the flowchart shown in each embodiment is realized by the control unit 109 executing a predetermined program. The processing of the flowchart in FIG. 3 starts, for example, when the user uses the operation unit 115 to select a RAW video and instructs the start of playback. At this time, the control unit 109 controls the recording medium control I / F unit 110 to start reading the selected RAW video file from the recording medium 112 and acquires the RAW video file. Then, the control unit 109 executes the processing of the flowchart in FIG. 3 for each frame of the read RAW video file.

[0021] Although the example shows the processing of the flowchart in FIG. 3 being executed by the control unit ********** of the imaging device 100, the execution subject of the processing of the flowchart in FIG. 3 may be a device other than the imaging device 100. For example, an external device (e.g., a personal computer) acquires a RAW video file via the recording medium 112 or the external I / F unit 113. Then, the external device may execute the processing of the flowchart in FIG. 3 for the acquired RAW video file. In this case, the external device has the function as the control unit 109 (image processing device).

[0022] It should be noted that there is an unclear part in the original text where "control unit 109" has some asterisks in the description in ID=8. I have translated it as accurately as possible based on the available context. If there are specific clarifications for that part, the translation can be further refined.In S301, the loop frame determination unit 117 determines whether the current frame among the plurality of frames constituting the video to be played is the first frame N (N is a natural number). When the control unit 109 determines Yes in S301 (when it is the first frame), the process proceeds to S302. When the control unit 109 determines No in S301 (when it is not the first frame), the process proceeds to S308.

[0023] In S302, the control unit 109 determines whether the strength setting of the loop NR is "strong". When the control unit 109 determines Yes in S302 (when the strength setting of the loop NR is "strong"), the process proceeds to S303. In S303, the control unit 109 reads the frame (N + 2) two frames after the first frame N from the RAW video file, and causes the read frame to be developed by the image signal processing circuit 107. In S304, the control unit 109 causes the frame (frame (N + 2)) developed in S303 to be held in the first memory unit 108 via the loop frame holding control unit 116.

[0024] In S305, the control unit 109 reads the frame (N + 1) one frame after the first frame N from the video file, and causes the read frame to be developed by the image signal processing circuit 107. In S306, the control unit 109 performs loop NR processing on the frame (N + 1) developed in S305 using the frame (N + 2) held in the first memory unit 108 in S304. By performing the loop NR processing using two consecutive frames, various noises in the time direction can be reduced. In S307, the control unit 109 causes the frame (N + 1) subjected to the loop NR processing to be held in the first memory unit 108 via the loop frame holding control unit 116.

[0025] In S308, the control unit 109 causes the image signal processing circuit 107 to perform development processing on the current frame N. In S309, the control unit 109 performs circular NR processing on the current frame N that has been developed in S308 using the frame (N + 1) held in the first memory unit 108. In S310, the control unit 109 causes the display unit 111 to display the image data of the frame N on which the circular NR processing has been performed. In S311, the circular frame holding control unit 116 of the control unit 109 causes the image data on which the circular NR processing has been performed on the current frame N to be held in the first memory unit 108.

[0026] When the control unit 109 determines No in S302 (when the intensity setting of circular NR is not "strong"), the process proceeds to S312. In S312, the control unit 109 reads out the frame (N + 1) that is one frame after the current frame N (the first frame) from the RAW video file, and causes the read frame to be developed by the image signal processing circuit 107. In S313, the control unit 109 causes the developed frame (N + 1) to be held in the first memory unit 108 via the circular frame holding control unit 116. Then, the control unit 109 proceeds with the process to S308. Then, the control unit 109 performs the processes after S308.

[0027] As described above, the control unit 109 performs circular NR processing on the first frame N (the current frame N) using the reverse-played frame. Circular NR processing is a method of reducing noise by utilizing the frame correlation between the current frame and the immediately preceding frame. Therefore, when performing circular NR processing on the first frame, a frame corresponding to the immediately preceding frame is required. However, when playing back an RA video, a frame before the first frame does not exist in the RAW video file. Therefore, the control unit 109 performs circular NR processing using one or more frames after the first frame N. As a result, when playing back the video, circular NR processing can be performed from the first frame, improving the image quality of the first frame. Accordingly, the image quality of the entire video also improves.

[0028] FIG. 4 is a diagram showing an example when a RAW video file is played from the beginning. In FIG. 4, the playback frame (the first frame) of the RAW video file is frame 8 (N = 8). When the intensity setting of the cyclic NR is "weak", the control unit 109 reads frame 9, which is the frame next to the first frame, and performs development processing on frame 9 as represented by arrow 401. The image data of frame 9 is held in the first memory unit 108.

[0029] Next, the control unit 109 reads the image data corresponding to the first frame 8 and performs development processing on it as represented by arrow 402. That is, the development processing of frame 9 read from the RAW video file is performed before the first frame 8. The control unit 109 performs cyclic NR processing on the first frame 8 on which the development processing has been performed, using frame 9 held in the first memory unit 108, as represented by arrow 403. Thereby, the image data of frame 8 on which the cyclic NR processing has been performed is generated. The generated image data of frame 8 is displayed on the display unit 111 as a display image, as represented by arrow 404. Also, the image data of frame 8 on which the cyclic NR processing has been performed is held in the first memory unit 108.

[0030] For the frames after the frame next to the first frame, the control unit 109 performs cyclic NR processing on the image of the current frame for each frame, using the image of the frame on which the previous cyclic NR processing has been performed, as represented by arrow 405. In the example of FIG. 4, for the second frame (frame 9), the control unit 109 performs cyclic NR processing using the current frame (frame 9) and the previous frame (frame 8). Thereby, the cyclic NR processing can be sequentially performed on each frame following the first frame.

[0031] FIG. 5 is a diagram showing an example when a RAW video file is played back from the middle. In the example of FIG. 5, the RAW video is played back from frame 20 in the middle of the RAW video file. When the intensity setting of the cyclic NR is "weak", the control unit 109 reads frame 19, which is the frame immediately before frame 20, and performs development processing on frame 19 as represented by arrow 501. The image data of frame 19 is held in the first memory unit 108.

[0032] Also, the control unit 109 reads frame 20 as the playback frame 1, and performs development processing on frame 20 as represented by arrow 502. The control unit 109 performs cyclic NR processing on the developed frame 20 using the frame 19 held in the first memory unit 108 as represented by arrow 503. Thereby, the image data of frame 20 subjected to the cyclic NR processing is generated. The image data of frame 20 subjected to the cyclic NR processing is displayed on the display unit 111 as a display image as represented by arrow 504. Also, the image data of frame 20 subjected to the cyclic NR processing is held in the first memory unit 108.

[0033] For each frame after frame 20, the same processing as in the example of FIG. 4 is performed. The control unit 109 performs cyclic NR processing on the currently developed frame for each frame using the frame subjected to the previous cyclic NR processing as represented by arrow 505. As described above, even when the RAW video file is played back from the middle, the cyclic NR processing can be sequentially performed with the played-back frame in the middle as the leading frame.

[0034] FIG. 6 is a diagram showing an example when a RAW video file recorded with a long-exposure is played back. In the example of FIG. 6, it is assumed that the same image data for four frames is recorded in the RAW video file with a long-exposure. When the intensity setting of the cyclic NR is "weak", the control unit 109 reads out frame 12, which is several frames after frame 8, as the frame immediately before frame 8, where the frames are accumulated with a longer exposure time than frame 8. That is, the control unit 109 uses one or more frames after the number of frames accumulated according to the exposure time for the cyclic NR process. As represented by arrow 601, the control unit 109 performs development processing on frame 12. The image data of frame 12 subjected to the cyclic NR process is held in the first memory unit 108.

[0035] Also, the control unit 109 reads out frame 8 and performs development processing as represented by arrow 602. At this time, the control unit 109 sets the read frame 8 as the playback frame 1. As represented by arrow 603, the control unit 109 performs cyclic NR processing on the developed frame 8 using frame 12 held in the first memory unit 108. Thereby, the image data of frame 8 subjected to the cyclic NR process is generated. The image data of frame 8 subjected to the cyclic NR process is displayed on the display unit 111 as a display image as represented by arrow 404. The image data of frame 8 subjected to the cyclic NR process is held in the first memory unit 108.

[0036] The cyclic NR process for each frame after frame 9 is the same as in the examples of FIGS. 4 and 5. As described above, even when playing back a RAW video file recorded with a long-exposure, the cyclic NR process can be sequentially performed on each frame from the first frame.

[0037] The examples of FIGS. 4 to 6 have described the case where the intensity setting of the cyclic NR is "weak", but the examples of FIGS. 4 to 6 can also be applied when the intensity setting of the cyclic NR is "strong". Further, in the example of the flowchart of FIG. 3, when the intensity setting of the cyclic NR is "strong", the number of frames to be played back in reverse is two, but the number of frames to be played back in reverse (the number of frames to which the cyclic NR process is applied) may be three or more. In this case, the processes of S303 to S307 are repeatedly performed according to the number of frames to be played back in reverse. If the number of frames to be played back in reverse becomes too large, the time until video playback starts becomes long. Therefore, a certain limit may be provided for the number of frames to be played back in reverse.

[0038] The control unit 109 may cause the display unit 111 to display a video selection screen. FIG. 18 is a diagram showing an example of the video selection screen. The video selection screen is a screen for selecting a video file to be played back from a plurality of video files. The video selection screen 1801 in FIG. 18(A) includes a cursor 1802, a cursor movement unit 1803, an OK button 1804, and a cancel button 1805. The user can select a video to be played back using the operation unit 115. The cursor 1802 indicates the video being selected among the plurality of videos. The user can change the video to be played back by operating the cursor movement unit 1803 up and down using the operation unit 115. The OK button 1804 is a button for confirming the selection of the video to be played back. The selection of the video to be played back is not confirmed until the OK button 1804 is pressed. The cancel button 1805 is a button for causing a transition from the video selection screen 1801 to another screen.

[0039] The video selection screen 1851 in FIG. 18(B) is a screen for selecting a video to be played from a list of thumbnails. The video selection screen 1851 in FIG. 18(B) includes a thumbnail area 1852, an OK button 1853, and a cancel button 1854. The thumbnail area 1852 is an area for displaying thumbnails of a plurality of videos. The OK button 1853 is the same as the OK button 1804 in FIG. 18(A). The cancel button 1854 is the same as the cancel button 1805 in FIG. 18(A). The user can select a video to be played from among a plurality of thumbnails using the operation unit 115. In the example of FIG. 18(B), video 2 is selected.

[0040] After a video is selected on the video selection screen 1801 in FIG. 18(A), when the OK button 1804 is pressed, the selection of the selected video is confirmed. Before the OK button 1804 is pressed, the control unit 109 may perform development processing on one or more frames after the head frame as described above for the head frame of the selected video. For example, in the example of FIG. 18(A), video 2 is selected, but the selection has not been confirmed. At this time, the control unit 109 performs development processing on the frame next to the head frame of the selected video 2. As a result, an image of the frame next to the head frame of video 2 is generated. When the OK button 1804 is pressed and the selection is confirmed, since the frame next to the head frame of video 2 has been developed, the time required for the cycle NR process can be shortened. That is, when the selection of the video is confirmed, the playback of the video starts quickly.

[0041] Also, assume that before the confirmation button 1804 is pressed, the selected video is changed from video 2 to video 3 by an operation using the operation unit 115. In this case, the control unit 109 performs the same processing on video 3 as the processing performed on video 2 described above. Thereby, when an arbitrary video is selected, the reproduction of the video subjected to the cyclic NR process on the leading frame can be quickly performed. The above also applies to the video selection screen 1851 in FIG. 18(B). The control unit 109 may change whether to perform the development process for performing the cyclic NR process on the leading frame of the selected video according to the strength setting of the cyclic NR, that is, the number of frames to be played back in reverse.

[0042] As described above, when the control unit 109 performs the cyclic NR process on the leading frame during video playback, one or more frames after the leading frame are used. Thereby, among the frames constituting the video, the cyclic NR process can also be performed on the leading frame, so that the effect of the noise reduction process can be obtained for the leading frame as well. Here, a proxy video (a video having a smaller number of pixels and size than the RAW video) is often recorded as an editing video at the same time as the RAW video. If the cyclic NR process is not performed on the leading frame of the RAW video, the noise reduction effect becomes lower than that of the proxy video recorded at the same time as the RAW video. However, as described above, the control unit 109 performs the cyclic NR process on the leading frame using one or more frames after the leading frame. Therefore, it is possible to realize the reproduction of the RAW video subjected to the cyclic NR process comparable to that of the proxy video.

[0043] The above example described an example where a RAW video is the processing target, but the processing target may be a video other than a RAW video. For example, when playing an image or a video file recorded in an image format (such as RAW or YUV) without cyclic NR processing, it can also be applied when performing cyclic NR processing on the first frame. Also, the processing by the control unit 109 described above may be applied only when performing a predetermined editing process on the video file. Examples of the predetermined editing process include splitting and combining video files, transcoding to convert the recording format, resolution, or frame rate of the video file and recording it, and extracting a still image by cutting out one frame of a RAW video. In this case, the processing of the flowchart in FIG. 3 is executed only when performing an editing process. The above points are common to each embodiment.

[0044] <Second Embodiment> The second embodiment will be described. In the second embodiment, the control unit 109 combines a plurality of video files and plays the combined video file. FIG. 7 is a diagram showing an example of combining videos and playing the combined video. In the example of FIG. 7, the videos are combined in the order of Video 1, Video 2, and Video 3. In this embodiment, the control unit 109 performs the same processing on the combined video as in the first embodiment.

[0045] In order to perform cyclic NR processing on the first frame 1-1 of Video 1, the control unit 109 first performs the development process on the next frame 1-2 of Video 1. Similarly, in order to perform cyclic NR processing on the first frame 2-1 of Video 2, the control unit 109 first performs the development process on the next frame 2-2 of Video 2. Also, in order to perform cyclic NR processing on the first frame 3-1 of Video 3, the control unit 109 first performs the development process on the next frame 3-2 of Video 3. The frames 1-2, 2-2, and 3-2 used for performing cyclic NR processing are not displayed on the display unit 111.

[0046] Here, assume a case where when the video has been played up to the junction point between Video 1 and Video 2, development processing for the next frame 2-2 is performed in order to perform cyclic NR processing on the first frame 2-1 of Video 2. In this case, due to performing the development processing on Frame 2-2, the playback of the video temporarily stops. In the second embodiment, before Frames 2-2 and 3-2 of the combined video are played, the control unit 109 completes the development processing to be performed on the combined Frames 2-2 and 3-2. Thereby, even when the combined video is played, control is performed to obtain the same effect as in the first embodiment without temporary stop.

[0047] FIG. 8 is a flowchart showing an example of the processing flow when playing the combined video of the second embodiment. In S801, the control unit 109 determines whether the RAW video to be played is a combined video. If the control unit 109 determines Yes in S801 (when the video to be played is a combined video), the process proceeds to S802. In S802, the control unit 109 determines whether it is possible to execute the development processing for each frame of the combined video at a speed equal to or less than half of the playback period of the combined video. If the control unit 109 determines Yes in S802 (when it is possible to execute the development processing at a speed equal to or less than half of the playback period of the combined video), the process proceeds to S803.

[0048] In S803, when the cyclic NR processing for the first frame is completed, the control unit 109 starts playing the combined video. The above-described cyclic NR processing is performed on the first frame of the first video in the combined video. Then, the combined video is played. In S804, while playing the combined video, the control unit 109 performs development processing on the frame next to the frame at the junction point at twice the playback speed of the combined video to perform cyclic NR processing on the frame at the junction point. That is, during the idle time when the combined video is being played, the control unit 109 performs cyclic NR processing on the frame at the junction point (the first frame of the combined video) in advance to complete the preparation for the cyclic NR processing.

[0049] Here, the process of S804 is executed when it is determined as Yes in S802. Therefore, even if the first video to be played is composed of two frames, when the leading frame of the next video (the frame at the joining position) is played, the preparation for the cyclic NR process is completed. Thus, when the frame at the joining position is played, since the preparation for the cyclic NR process of the frame is completed, the combined video being played will not pause. After the process of S804, the control unit 109 ends the process of the flowchart in FIG. 8.

[0050] When the control unit 109 determines as No in S802 (when the development process cannot be executed at a speed equal to or less than half of the video playback cycle), the process proceeds to S805. In S805, the control unit 109 completes the cyclic NR process for the leading frame and all frames at the joining positions. By performing the process of S805, before the playback of the combined video starts, the preparation for the cyclic NR process for the leading frame and each frame at the joining position is completed. In S806, the control unit 109 starts the playback of the combined video. When the frame at the joining position in the combined video is played, since the preparation for the cyclic NR process of the frame is completed, the combined video being played will not pause. After the process of S806, the control unit 109 ends the process of the flowchart in FIG. 8.

[0051] When the control unit 109 determines as No in S801 (when the video to be played is not a combined video), the process proceeds to S807. In this case, since the video is not combined, there is no need to consider the cyclic NR process for the frames at the joining positions. In S807, the control unit 109 performs the cyclic NR process on the leading frame of the video to be played. The process of S807 is the same as the process described in the first embodiment. Thereby, the preparation for the cyclic NR process for the leading frame is completed. In S808, the playback of the video starts. After the process of S808, the control unit 109 ends the process of the flowchart in FIG. 8.

[0052] In the above example, in S804, the control unit 109 pre-performs the development process of the frame next to the frame at the connection point (the first frame of the next video), and generates an image (image after the cyclic NR process) subjected to the cyclic NR process. In this regard, the control unit 109 may pre-perform the development process of the frame next to the frame at the connection point, and may perform the development process and the cyclic NR process of the frame at the connection point in real time.

[0053] <Third Embodiment> Next, the third embodiment will be described. In the third embodiment, the intensity of the cyclic NR process applied to the first frame when the generated RAW video is generated is weakened to reduce the afterimage phenomenon occurring in the first frame.

[0054] FIG. 9 is a diagram showing an example of a proxy video file when a pendulum is photographed and a RAW video file at the time of reproduction. The 7th to 9th frames of the proxy video file are frames 904 to 906. Regarding the RAW video file, it is assumed that the 8th frame is the first frame 902, and the 9th frame is the next frame 903 of the first frame 902. By sequentially reproducing the frames 904 to 906 of the proxy video file, a video in which the pendulum moves greatly from left to right is displayed.

[0055] Here, assume a case where the control unit 109 develops the next frame 903 first as the frame 901 with respect to the first frame 902 of the RAW video file as described in the first embodiment, and performs the cyclic NR process using the frame 901. The frame 901 is the same image as the frame 903 and is used for the cyclic NR process but is not displayed. Here, in the frame 901, the sphere of the pendulum swings greatly in the right direction (direction B). When the cyclic NR process is performed on the first frame 902 using the frame 901, an afterimage of the sphere of the pendulum is generated on the B side. The afterimage is a phenomenon caused by taking the difference between the area of the sphere in the frame 901 and the same area of the first frame 902 by the inter-frame process.

[0056] On the other hand, for the proxy video file, for frame 905, circular NR processing is performed using frame 904. In frame 904, the sphere swings greatly in the left direction (direction A). Therefore, when circular NR processing is performed on frame 905 using frame 904, an afterimage of the sphere is generated on the A side. That is, in the RAW video file and the proxy video file, the directions in which the afterimage of the sphere is generated are opposite. As shown in FIG. 9, when the directions in which the afterimage is generated are different between the proxy video file and the RAW video file, the visibility decreases due to the sense of discomfort.

[0057] The control unit 109 of the third embodiment changes the setting of the circular NR processing applied to the first frame when playing the RAW video, and performs processing to reduce the afterimage phenomenon occurring in the first frame. FIG. 10 is a diagram showing an example of the configuration of the control unit 109 in the third embodiment. Since the other configurations in the imaging device 100 other than the control unit 109 are the same as those in FIG. 1, the description thereof is omitted. The control unit 109 includes a video analysis unit 1001, a reference frame determination unit 1002, and a noise reduction setting control unit 1003 in addition to a circular frame holding control unit 116 and a circular frame determination unit 117.

[0058] The video analysis unit 1001 analyzes the video of at least two frames among the frames of the RAW video, and analyzes the amount of movement of the video or the amount of movement of the subject in the video. The reference frame determination unit 1002 determines whether to use the result (image data) obtained by developing a frame after the first frame as the reference frame used for circular NR based on the analyzed amount of movement. The noise reduction setting control unit 1003 changes the setting of the circular NR processing performed by the image signal processing circuit 107 based on the determination result of the reference frame determination unit 1002.

[0059] FIG. 11 is a flowchart showing an example of the flow of RAW video playback processing in the third embodiment. Hereinafter, in the third embodiment, an example will be described in which the intensity setting of the cyclic NR process is "weak" and the cyclic NR process is applied to the first frame. However, the third embodiment is also applicable when the intensity setting of the cyclic NR process is "strong". The processing of the flowchart in FIG. 11 starts, for example, when the user instructs the playback of a RAW video using the operation unit 115.

[0060] In S1101, the video analysis unit 1001 reads out frame 903 as frame 901 which is the frame to be subjected to the cyclic NR process for the first frame 902 of the RAW video, and the first frame 902, and performs video analysis on the two frames. At this time, as video analysis, the video analysis unit 1001 analyzes the amount of movement of the video of the two frames. For example, the video analysis unit 1001 divides the images of the two frames by a predetermined number of mesh frames, and based on the similarity of the luminance value distribution of the pixels inside the mesh frames, analyzes the amount of frame movement between the two frames as horizontal and vertical pixel values as movement amount information. The number of divisions of the mesh frames may be an arbitrary value such as "255×255".

[0061] In S1102, the video analysis unit 1001 calculates an evaluation value (frame evaluation value) indicating whether an afterimage is likely to occur based on the analysis result of S1101. The frame evaluation value represents an index of the possibility of an afterimage occurring in a frame. The video analysis unit 1001 sets the maximum value of the movement amount of each of the above-described mesh frames as the frame evaluation value, and increases the value of the frame evaluation value as the movement amount increases. The video analysis unit 1001 may calculate the frame evaluation value using any information for detecting camera panning, tilting, and the movement of the subject to determine whether it is a scene where an afterimage is likely to occur. For example, the video analysis unit 1001 performs subject recognition, and in the case of a frame including a subject that is likely to be a moving object (such as a moving animal or a sports scene), the frame evaluation value may be set higher than when the subject is not a moving object. The video analysis unit 1001 may calculate the frame evaluation value based on metadata recorded in association with the video when the video is shot and recorded.

[0062] In S1103, the reference frame determination unit 1002 determines whether the frame evaluation value calculated by the video analysis unit 1001 has reached a predetermined threshold value. If the reference frame determination unit 1002 determines Yes in S1103 (when the frame evaluation value has not reached the predetermined threshold value), the process proceeds to S1104. In this case, it is allowed to use a frame after the head frame 902 as the frame to which the cyclic NR process is applied to the head frame 902 of the RAW video file.

[0063] In S1104, the noise reduction setting control unit 1003 sets the cyclic NR process to the first NR setting. The first NR setting is a setting applied to perform the cyclic NR process of the first embodiment and the second embodiment described above. Also, the first NR setting may be the same as the setting of the cyclic NR process applied to the proxy video. When it is determined Yes in S1103, since the amount of movement between frames is not so large, it is assumed that the influence on visibility due to afterimages is somewhat low.

[0064] If the reference frame determination unit 1002 determines No in S1103 (when the frame evaluation value is equal to or greater than the predetermined threshold value), the process proceeds to S1105. In this case, it is not allowed to use a frame after the head frame 902 as the frame to which the cyclic NR process is applied to the head frame 902 of the RAW video file. In S1105, the noise reduction setting control unit 1003 sets the cyclic NR process to the second NR setting. The second NR setting is a setting for suppressing afterimages.

[0065] The intensity of the cyclic NR process is represented by a cyclic coefficient. The cyclic coefficient is a numerical value indicating the degree of removing the influence of the difference between frames, and the higher the numerical value, the higher the noise reduction effect. Also, in the circuit of the cyclic NR process, when it is determined that there is a moving object, a process for reducing afterimages is performed by performing a cyclic type noise reduction process with a reduced cyclic coefficient. In S1105, the noise reduction setting control unit 1003 may change to a setting for reducing afterimages, for example, by lowering the threshold value of the moving object determination of the cyclic NR process circuit.

[0066] In S1106, the control unit 109 performs cyclic NR setting on the leading frame 902 with the cyclic NR process setting set in either S1104 or S1105. In S1104, the control unit 109 performs the same cyclic NR process as in the first embodiment or the second embodiment using a frame after the leading frame 902 as the frame for performing the cyclic NR process on the leading frame 902 of the RAW video file. On the other hand, in S1105, the control unit 109 does not use a frame after the leading frame 902 as the frame for performing the cyclic NR process on the leading frame 902 of the RAW video file. Therefore, when the cyclic NR process set in S1105 is performed, the noise reduction effect becomes lower, but by performing the process of S1105, afterimage reduction is achieved.

[0067] Here, assume a case where the cyclic NR process is performed on the leading frame 902 of the RAW video file based on the setting of S1105, and the cyclic NR process is performed on the frame 903 next to the leading frame 902 based on the setting of S1104. In this case, a large difference in image quality will occur between the leading frame 902 and the next frame 903. In such a case, for example, the noise that appeared when the leading frame 902 was played suddenly disappears when the next frame is played. Therefore, due to the large difference in image quality between the two frames, the visibility decreases.

[0068] Therefore, the control unit 109 may reduce the change amount of the cyclic NR process setting applied to the leading frame in accordance with the progress of the playback frame with a predetermined time constant to reduce the change in the image quality difference. The control unit 109 may increase the time constant as the change amount of the cyclic NR setting applied to the leading frame increases, and gradually change the change amount of the NR setting.

[0069] The video analysis unit 1001 analyzes the frames of the RAW video file, but may also analyze the proxy video file recorded simultaneously with the RAW video file. In this case, based on the video analysis results of the proxy video file, the same processing as described above is performed. Further, the video analysis unit 1001 may compare the frame 904 of the proxy video file with the video analysis results of the developed files of the frames 901 and 902 of the RAW video file. In this case, based on the result of the above comparison and the estimated result of the difference in the afterimage generation direction, the setting of the cyclic NR process may be changed.

[0070] <Fourth Embodiment> Next, the fourth embodiment will be described. In the third embodiment, by changing the setting of the cyclic NR process, a process for reducing afterimages is performed. The cyclic NR process is a process for reducing noise using frame correlation. In the fourth embodiment, depending on whether the frame evaluation value has reached a predetermined threshold, a process for reducing afterimages using a method that does not use frame correlation (a noise reduction process that is not cyclic) is performed. In the fourth embodiment, for example, a spatial NR process such as an epsilon filter is applied. In the spatial NR process, only the information within one frame is used, so afterimages do not occur.

[0071] In the fourth embodiment, the cyclic NR process is applied to the first frame of the proxy video file. To the first frame of the RAW video file, depending on whether an afterimage is likely to occur, an NR process using an epsilon filter, which is a spatial NR process, is applied. The strength of the epsilon filter is increased by increasing the surrounding pixel region and the weighting ratio of the surrounding pixels for the pixel of interest. Hereinafter, a case where an example of enhancing the noise reduction effect by expanding the reference pixels, such as from 3×3 pixels to 5×5 pixels, is applied will be described.

[0072] FIG. 12 is a diagram showing an example of the configuration of the image signal processing circuit 107 and the control unit 109 in the fourth embodiment. Each part of the control unit 109 is the same as that in FIG. 10. The video analysis unit 1201 can also analyze metadata recorded in association with the moving image. The image signal processing circuit 107 includes a first noise reduction circuit 1204 that performs epsilon filter processing, which is spatial NR processing, and a second noise reduction circuit 1205 that performs cyclic NR processing.

[0073] FIG. 13 is a flowchart showing an example of the flow of RAW video playback processing in the fourth embodiment. The processing of each step other than S1305 is the same as that in FIG. 11. In S1305, the noise reduction setting control unit 1203 sets the setting of the NR processing to epsilon-type NR. At this time, the noise reduction setting control unit 1203 analyzes the cyclic NR setting value set when recording the proxy video file from the metadata, and performs conversion to an epsilon filter setting in which the noise reduction effect becomes substantially constant from the cyclic NR setting value. The conversion of the setting value is performed based on, for example, conversion setting table data in which the noise reduction effects are close and the adverse effects are acceptable. The conversion setting table data is held in, for example, the second memory unit 114.

[0074] The cyclic NR processing causes an afterimage of the moving image to be an adverse effect, and the spatial filter type noise reduction causes a decrease in resolution to be an adverse effect. Therefore, the conversion setting table data prepared is adjusted to appropriate values in advance. In the conversion setting table data, the settings from the cyclic NR setting to the epsilon filter setting are made in the order of R1 (weak: cyclic coefficient 2), P1 (weak: 3×3 pixels), R2 (strong: cyclic coefficient 4), and P2 (strong: 5×5 pixels). That is, as the noise reduction effect of the cyclic NR setting increases, the setting in which the noise reduction effect of the epsilon filter processing that is not cyclic increases is made in the conversion setting table data. When S1305 is executed, in S1306, epsilon-type NR processing is performed.

[0075] As described above, in the fourth embodiment, when afterimages are likely to occur, epsilon-type NR processing is performed. As a result, when the RAW video file is played back, it is possible to suppress the degree of afterimages that appear in a direction different from that of the proxy image file, and the noise reduction effect can be made substantially constant.

[0076] <Fifth Embodiment> Next, the fifth embodiment will be described. FIG. 14 is a diagram showing an example of the configuration of the control unit 109 in the fourth embodiment. The control unit 109 in FIG. 14 has a metadata addition unit 1404 and a metadata analysis unit 1405 added to the control unit 109 in FIG. 10. Other configurations are the same as those of the control unit 109 in FIG. 10. The video analysis unit 1001 stores information indicating the calculated amount of movement and the result determined by the reference frame determination unit 1002 in the second memory unit 114. The metadata addition unit 1404 records the calculation result stored in the second memory unit 114 as metadata on the recording medium 112 together with the image data. The metadata analysis unit 1405 stores the image data and metadata read from the recording medium 112 in the second memory unit 114 and analyzes the metadata. The analysis result of the metadata can be referred to by the reference frame determination unit 1002.

[0077] FIG. 15 is a diagram showing an example of a flowchart of RAW video recording processing in the fifth embodiment. Hereinafter, the RAW video will be described as being recorded at a frame rate of 60 fps, but the frame rate may be any value. The processing of the flowchart in FIG. 15 is executed for each frame, for example, after the user instructs video recording using the operation unit 115. It is assumed that the recorded RAW video is subjected to cyclic NR processing with respect to the first frame when it is played back.

[0078] In S1501, the video analysis unit 1001 analyzes at least two frames (the current frame and the previous frame) of the video, and calculates the amount of movement based on the video analysis result. In S1502, the reference frame determination unit 1002 determines whether to use the development result of frames after N + 1 as the reference frame for the cyclic NR process. The determination in S1502 is made based on whether the amount of movement calculated in S1501 is equal to or greater than a predetermined amount. The predetermined amount can be set to any value. For example, the control unit 109 may make the same determination as S1103 in FIG. 11 in S1502.

[0079] When the reference frame determination unit 1002 determines Yes in S1502 (when using the development result of frames after N + 1), the process proceeds to S1503. On the other hand, when the reference frame determination unit 1002 determines No in S1502 (when not using the development result of frames after N + 1), the process proceeds to S1504. In S1503, the metadata addition unit 1404 sets information indicating that application to cyclic NR is allowed in the metadata. In S1504, the metadata addition unit 1404 sets information indicating that application to cyclic NR is not allowed in the metadata.

[0080] In S1505, the metadata addition unit 1404 sets the amount of movement calculated in S1501 in the metadata. In S1506, the control unit 109 records the above-described metadata together with the RAW video file on the recording medium 112.

[0081] FIG. 16 is a diagram showing a first example of the RAW video recording process. The first example is a process performed when shooting a RAW video in a normal scene where afterimages are difficult to see. In the example of FIG. 16, the frame rate at which the video is shot is 600 fps. The control unit 109 reads out the subject image stored in the image sensor 106 and causes the image signal processing circuit 107 to perform image processing. The frames of the image-processed image are recorded on the recording medium 112 as RAW recorded images.

[0082] The values calculated by the image analysis unit 1001 and the cycle frame determination unit 117 are stored in the second memory unit 114 together with the above-described metadata. Since the first example is performed when shooting a RAW video in a normal scene where afterimages are hardly visible, in S1503, information indicating that it can be applied to the cycle NR process is added by the metadata addition unit 1404 to the metadata. In the example of FIG. 16, information indicating that it can be applied to the cycle NR process is recorded as the value "1" in the cycle NR use presence / absence.

[0083] FIG. 17 is a diagram showing a second example of the RAW video recording process. The second example is a process performed when shooting a RAW video in a scene where afterimages are easily visible. As shown in the example of FIG. 17, due to the large amount of movement, afterimages are easily visible in the video of each frame of the RAW video. Here, in S1504, information indicating that it cannot be applied to the cycle NR process is added by the metadata addition unit 1404 to the metadata. In the example of FIG. 17, information indicating that it cannot be applied to the cycle NR process is recorded as the value "0" in the cycle NR use presence / absence.

[0084] In the fifth embodiment, when the RAW video recording process is performed, it becomes possible to record metadata regarding the cycle NR process together with the RAW video. The fifth embodiment is effective when applied when the control of the third and fourth embodiments is performed.

[0085] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist. The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors of a computer of the system or device read and execute the program. Further, the present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of Reference Numerals

[0086] 100 Imaging device 106 imaging element 109 control unit 1001 video analysis unit 1002 reference frame determination unit 1003 noise reduction setting control unit 1204 first noise reduction circuit 1205 second noise reduction circuit

Claims

1. Means for acquiring a video, When playing the video, noise reduction processing is performed on the leading frame using one or more frames after the leading frame at the time of playback, and circular noise reduction processing is performed on frames other than the leading frame using one or more frames before that frame, and an image processing apparatus comprising the same.

2. The number of frames used for the noise reduction processing is changed according to the intensity setting of the noise reduction processing, and the image processing apparatus according to claim 1, characterized in that.

3. A certain limit is provided for the number of frames used for the noise reduction processing, and the image processing apparatus according to claim 2, characterized in that.

4. One or more frames after the number of frames accumulated according to the exposure time from the leading frame are used for the noise reduction processing, and the image processing apparatus according to claim 1 or 2, characterized in that.

5. Means for displaying a screen for selecting one of the plurality of videos, Means for performing development processing on one or more frames used when applying the noise reduction processing to the leading frame of the video selected when the screen is displayed, and further comprising the image processing apparatus according to any one of claims 1 to 4.

6. When the video being selected on the screen is changed, development processing of one or more frames used when applying the noise reduction processing to the leading frame of the changed video is performed, and the image processing apparatus according to claim 5, characterized in that.

7. In response to the selection of the video on the screen being confirmed, the noise reduction processing is performed on the leading frame using one or more frames on which the development processing has been performed, and the image processing apparatus according to claim 5 or 6, characterized in that.

8. The one or more frames applied to the noise reduction processing performed on the leading frame are not displayed, and the image processing apparatus according to any one of claims 1 to 7, characterized in that.

9. When playing a combined video obtained by combining a plurality of the videos, the frame at the combination point is used as the head frame, and noise reduction processing is performed on the head frame. The image processing apparatus according to any one of claims 1 to 8, characterized in that.

10. Depending on the speed of performing the development process of the frame, while playing the combined video, noise reduction processing is performed on the frame at the combination point, or before playing the combined video, noise reduction processing is performed on the frame at the combination point. The image processing apparatus according to claim 9, characterized in that control is performed.

11. When the development process of the frame can be executed in half or less of the reproduction cycle of the combined video, noise reduction processing is performed on the frame at the combination point while playing the combined video. When the development process of the frame cannot be executed in half or less of the reproduction cycle of the combined video, noise reduction processing is performed on the frame at the combination point before playing the combined video. The image processing apparatus according to claim 10, characterized in that.

12. Based on whether an evaluation value representing an index of the possibility of an afterimage occurring in a frame has reached a predetermined threshold value, it is controlled whether to perform noise reduction processing on the head frame using one or more frames after the head frame. The image processing apparatus according to any one of claims 1 to 11, characterized in that.

13. The evaluation value is evaluated based on the video of the frame or the analysis result of the amount of movement of the subject in the video. The image processing apparatus according to claim 12.

14. Information indicating whether it is allowed to use one or more frames after the head frame for noise reduction processing on the head frame is added to each frame of the video. The image processing apparatus according to any one of claims 1 to 13, characterized in that.

15. The means for acquiring the video acquires a RAW video. The image processing apparatus according to any one of claims 1 to 14, characterized in that.

16. A step of acquiring a video, When playing the video, noise reduction processing is performed on the leading frame using one or more frames after the leading frame at the time of playback, and circular noise reduction processing is performed on frames other than the leading frame using one or more frames before the frame. An image processing method comprising the steps of:

17. A program for causing a computer to execute each means of the image processing apparatus according to any one of claims 1 to 15.

Images