Image processing device, image processing method, and program

Abstract

This invention enables combination to be performed for a plurality of pieces of image data at the stage of RAW images before the implementation of a developing process. This image processing device comprises a combination processing unit that combines a plurality of pieces of RAW image data each of which constitutes one frame of image, thereby generating combined RAW image data.

Description

[Technical Field]

[0001] The present technology relates to an image processing device, an image processing method, and a program, and to a technology for image synthesis processing. [Background technology]

[0002] An image before it is created using an imaging device is sometimes called a RAW image. It is common to record a RAW image as the image resulting from the image capture. This is because recording a RAW image has the advantage of increasing the degree of freedom in post-capture processing, such as color reproduction.

[0003] Patent Document 1 listed below discloses a technique that makes it possible to easily achieve image effects such as front curtain synchronization, rear curtain synchronization, and multi-flash using long exposure without requiring skill. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-232382 Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, a photographer sets the shutter speed when capturing an image with an imaging device. Some imaging devices automatically set the shutter speed. In either case, the shutter speed is set when capturing an image. Therefore, even if RAW image data is recorded, it is not possible to adjust the shutter speed after capturing an image, which can lead to a failed capture due to an inappropriate shutter speed setting before capturing the image. Furthermore, RAW image data undergoes development processing to generate image data in a specified format, but when the developed image data is combined as in Patent Document 1, the development processing is not necessarily suitable for the combined image.

[0006] The present disclosure focuses on these points and aims to propose a technique for more effectively utilizing RAW images. [Means for solving the problem]

[0007] The image processing device according to the present technology includes a synthesis processing unit that synthesizes a plurality of RAW image data, each of which constitutes an image of one frame, to generate synthesized RAW image data. The multiple RAW image data to be synthesized by the synthesis processing unit are multiple RAW image data that have been captured consecutively in time with a non-exposure period interposed therebetween by a single image sensor and have been processed so that the exposure periods of each are consecutive in time. A group of RAW image data obtained by capturing images is synthesized, i.e., synthesized at a stage before development processing.

[0008] The image processing device according to the present technology described above may further include a development processing unit that performs development processing on the composite RAW image data generated by the synthesis processing unit to generate image data in a predetermined format. That is, after the RAW image data is synthesized, the synthesized RAW image data is subjected to development processing.

[0009] In the image processing device according to the present technology described above, it is conceivable that the synthesis processing unit generates, through synthesis processing, synthesized RAW image data corresponding to an image captured at a shutter speed slower than the shutter speed at which the image was captured. By combining multiple pieces of RAW image data, an image can be obtained with a shutter speed that has a longer exposure period than the shutter speed at which each individual RAW image was captured.

[0010] In the image processing device according to the present technology described above, it is conceivable that the plurality of RAW image data to be combined by the combining processing unit are RAW image data of a plurality of frames that are continuous in time. For example, RAW image data of a plurality of frames obtained by capturing images continuously in time, such as by continuous shooting or video shooting, is used as the object of synthesis.

[0011] In the image processing device according to the present technology described above, the plurality of RAW image data to be combined by the combining processing unit may be RAW image data of a plurality of frames that are temporally continuous at a constant shutter speed. For example, RAW image data of a plurality of frames obtained by capturing images continuously in time, such as by capturing continuous images or moving images at a constant shutter speed, is used as the object to be synthesized.

[0012] In the image processing device according to the present technology described above, the multiple RAW image data to be synthesized by the synthesis processing unit may be multiple RAW image data that are adjacent in time and whose exposure periods are consecutive in time. For example, the temporally continuous RAW image data is a series of RAW image data that does not include a non-exposure period.

[0013] In the image processing device according to the present technology described above, the plurality of RAW image data to be synthesized by the synthesis processing unit are The aforementioned By performing additional processing to add an exposure amount corresponding to a non-exposure period of the image sensor to an exposure amount of the image sensor during an exposure period of the image sensor, Each other It is conceivable that the exposure periods are continuous in time. In imaging operations using an image sensor, for example, exposure and readout periods are performed in a time-division manner, and the readout period is a non-exposure period where exposure is interrupted. Interpolation is performed during such non-exposure periods to correspond to the exposure amount (amount of charge accumulated by exposure).

[0014] In the image processing device according to the present technology described above, it is considered that the multiple RAW image data to be synthesized by the synthesis processing unit are read out alternately and sequentially from multiple image sensors, so that the exposure periods are continuous in time. By using a plurality of image pickup elements, it is possible to perform exposure at one image pickup element during the readout period at the other image pickup element.

[0015] In the image processing device according to the present technology described above, the synthesis processing unit may select, based on a specified shutter speed, RAW image data to be synthesized from multiple RAW image data that are consecutive in time. For example, when performing composition processing, the user can specify the shutter speed.

[0016] In the image processing device according to the present technology described above, the synthesis processing unit may select RAW image data to be synthesized from multiple RAW image data that are consecutive in time based on the specified number of images. For example, when performing a synthesis process, the user can specify the number of images.

[0017] In the image processing device according to the present technology described above, the RAW image data is considered to be image data having the same color arrangement as the color arrangement of the imaging element.

[0018] In the image processing device according to the present technology described above, the RAW image data is image data in which pixel values ​​read from an image sensor are converted into the form of luminance values ​​and chroma values, and is considered to be image data to which no color reproduction / sharpness processing has been applied. This is image data known as YC-RAW.

[0019] The image processing method according to the present technology is an image processing method in which an image processing device executes a synthesis process to synthesize a plurality of RAW image data, each of which constitutes an image of one frame, to generate synthesized RAW image data. The multiple RAW image data to be combined in the combination process are multiple RAW image data that have been captured consecutively in time with a non-exposure period interposed therebetween using a single image sensor, and have been processed so that the exposure periods of each are consecutive in time. This allows for effective use of RAW image data. The program according to the present technology is a program that causes an information processing device to execute this image processing, thereby making it possible to easily realize the image processing device. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is an explanatory diagram of a synthesis process according to an embodiment of the present technology; [Figure 2] 1 is a block diagram of an imaging apparatus according to an embodiment; [Figure 3] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 4] FIG. 1 is an explanatory diagram of RAW image data and development processing. [Figure 5] 1A and 1B are explanatory diagrams of a combination process and a development process according to an embodiment and a comparative example. [Figure 6] 10 is a flowchart of a process during imaging according to an embodiment. [Figure 7] 10 is a flowchart including a synthesis process and a development process according to an embodiment. [Figure 8] 10A and 10B are explanatory diagrams of additional processing for making exposure periods continuous in the embodiment. [Figure 9] 10A to 10C are explanatory diagrams illustrating the imaging of RAW image data with continuous exposure periods in an embodiment. [Figure 10] FIG. 1 is an explanatory diagram of YC-RAW image data and development processing. [Figure 11] 3A to 3C are explanatory diagrams of a composition process and a development process according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] The embodiments will be described below in the following order. <1. Overview> 2. Configuration of the imaging device <3. Configuration of information processing device> <4. Compositing and developing RAW image data> <5. Composition and development of YC-RAW image data> <6. Summary and Variations>

[0022] In this disclosure, a RAW image refers to an image before some or all of the development process is performed by an imaging device, etc. Image data constituting one image (one frame) as a RAW image is also referred to as RAW image data. Although there are various types of image data that are specifically called RAW image data, in this embodiment, image data that has the same color array as the color array of an image sensor is used as an example of RAW image data. Note that image data that has undergone the same defect correction and thus has the same color array as the image sensor is also included in RAW image data. For example, if an image sensor outputs R (red), G (green), and B (blue) pixel signals, this RAW image data will be in the R, G, B format. Note that RAW image data can also include pixel values ​​for W (white) in addition to R, G, and B. Furthermore, in the case of a Bayer array image sensor, RAW image data includes pixel values ​​for G1 (green 1), G2 (green 2), R (red), and B (blue). There is also an image data format in which the pixel values ​​of G1, G2, R, and B are grouped together as four separate channels. For example, this is image data in which one frame is composed of only G1, G2, R, and B, and then each channel is compressed. In other words, this is image data in the form of G1 image data, G2 image data, R image data, and B image data. This is also included as an example of RAW image data.

[0023] Furthermore, image data read from an imaging element in the form of luminance values ​​and chroma values, without undergoing color reproduction / sharpness processing, is also sometimes called RAW image data. Although this is also a type of RAW image data in this disclosure, for the sake of distinction, it will be referred to as "YC-RAW image data."

[0024] <1. Overview> The image processing device according to the embodiment is expected to be installed as an image processing unit in an imaging device (camera) or an information processing device that performs image editing, etc. Furthermore, the imaging device or information processing device equipped with such an image processing unit can also be considered as an image processing device.

[0025] Such an image processing device performs processing to synthesize multiple pieces of RAW image data to generate synthesized RAW image data. 1 shows a schematic diagram of generating one piece of composite RAW image data by combining a plurality of RAW image data R#1 to R#n. The combining process in this embodiment may be a process of superimposing a plurality of image data by aligning the positions of the frames of the plurality of image data, or a process of superimposing a plurality of image data by aligning the positions of specific subjects of the plurality of image data.

[0026] In this case, the multiple RAW image data (R#1 to R#n) to be subjected to the synthesis process can be considered as, for example, a group of RAW image data that are captured consecutively and have a relationship as temporally consecutive frames. However, it is also possible to combine multiple sets of RAW image data that are completely unrelated in terms of time, and it is also possible to combine multiple sets of RAW image data that are images that are not particularly correlated as subjects.

[0027] Furthermore, one piece of composite RAW image data may be generated from a plurality of pieces of RAW image data, or a smaller number of pieces of composite RAW image data may be generated from the plurality of pieces of RAW image data. For example, if there are 10 pieces of RAW image data that are consecutive in time, it is possible to selectively use the RAW image data to be combined, such as RAW image data that combines all 10 pieces or RAW image data that combines seven pieces.

[0028] The composite RAW image data is then subjected to a development process, which allows image data to be obtained in a predetermined format, such as JPEG (Joint Photographic Experts Group) image data. Of course, the predetermined format is not limited to JPEG image data; other formats such as HEIF (High Efficiency Image File Format), YUV422, and YUV420 are also acceptable.

[0029] In the following description of the embodiment, an example will be mainly described in which multiple pieces of chronologically consecutive RAW image data are combined to generate combined RAW image data. For example, multiple consecutively recorded RAW image data can be combined to generate a single composite RAW image data, which achieves the same effect as a long exposure. Furthermore, by setting the number of RAW image data to be combined, the composite RAW image data can be created at any shutter speed slower than the shutter speed at which each RAW image data was captured. In addition, the composite RAW image data obtained by the synthesis process undergoes lens correction, NR (noise reduction), demosaic, color reproduction, sharpness processing, etc., just like in normal development processes.

[0030] 2. Configuration of the imaging device An example of the configuration of the imaging device 1 will be described with reference to FIG. This imaging device 1 is equipped with an image processing unit 20 that performs synthesis processing of RAW image data, and this image processing unit 20, or the imaging device 1 equipped with the image processing unit 20, can be considered an example of an image processing device of the present disclosure.

[0031] The imaging device 1 has, for example, a lens system 11, an imaging element unit 12, a recording control unit 14, a display unit 15, a communication unit 16, an operation unit 17, a camera control unit 18, a memory unit 19, an image processing unit 20, a buffer memory 21, a driver unit 22, and a sensor unit 23.

[0032] The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. The lens system 11 guides light (incident light) from a subject and focuses the light on the imaging element unit 12.

[0033] The imaging element unit 12 includes an image sensor 12a (imaging element) such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type. The image sensor 12 photoelectrically converts the light received by the image sensor 12a into an electrical signal, which is then subjected to processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control), and then A / D (Analog / Digital) conversion, and the resulting digital image signal is output to the downstream image processor 20 and camera controller 18.

[0034] The image processing unit 20 is configured as an image processor, for example, using a DSP (Digital Signal Processor). The image processing unit 20 performs various types of signal processing on the digital signal (captured image signal) from the image sensor unit 12, that is, the RAW image data. Note that RAW image data may refer to image data that has undergone some processing of the digital signal from the imaging element unit 12, such as the above-mentioned YC-RAW image data. Also, even if the format does not reach that of YC-RAW image data, RAW image data may be obtained by performing preprocessing, such as clamping the R, G, and B black levels to a predetermined level, or correction processing between the R, G, and B color channels, on the captured image signal from the imaging element unit 12. RAW image data may also include image data that has undergone lens correction and noise reduction.

[0035] In this embodiment, the image processing unit 20 has signal processing functions as a synthesis processing unit 31 and a development processing unit 32 . As will be described later, the synthesis processing unit 31 performs synthesis processing to generate synthesized RAW image data by synthesizing a plurality of RAW image data, each of which constitutes an image of one frame. The development processing unit 32 performs development processing to generate image data in a predetermined format by performing development processing on RAW image data and composite RAW image data generated by the synthesis processing unit 31. For example, the development processing unit 32 performs lens correction, noise reduction, synchronization processing, YC generation processing, color reproduction / sharpness processing, etc.

[0036] In the synchronization process, a color separation process is performed so that the image data for each pixel contains all the color components R, G, and B. For example, in the case of an image sensor that uses a Bayer color filter, a demosaic process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, and B image data. Color reproduction / sharpness processing involves adjusting gradation, saturation, tone, contrast, etc., which are used to create images.

[0037] These processes performed by the development processing unit 32 are development processes in the broad sense, but the color reproduction / sharpness processes in particular are called development processes in the narrow sense. Image data that has undergone development processes in the narrow sense loses some of the information in the original RAW image data, which reduces the degree of freedom in subsequent image editing. Furthermore, image data that has not undergone development processing in the narrow sense can be said to fall within the category of RAW image data and YC-RAW image data as referred to in this disclosure.

[0038] In this way, the image processing unit 20 performs development processing in a broad sense using the development processing unit 32 to generate image data in a predetermined format. In this case, resolution conversion and file creation processing may be performed. In the file creation processing, image data is subjected to, for example, compression encoding for recording or communication, formatting, and generation and addition of metadata to create files for recording or communication. For example, still image files can be generated in formats such as JPEG, TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), HEIF, YUV422, YUV420, etc. It is also possible to generate image files in the MP4 format used for recording MPEG-4 compliant video and audio. Note that there are also cases where an image file is generated using RAW image data that has not undergone development processing.

[0039] The buffer memory 21 is formed by, for example, a DRAM (Dynamic Random Access Memory), and is used for temporary storage of image data in the image processing unit 20 during the above-mentioned synthesis processing and development processing.

[0040] The recording control unit 14 performs recording and reproduction on a recording medium such as a nonvolatile memory, and performs processing to record image files such as moving image data and still image data on the recording medium. The recording control unit 14 may take a variety of actual forms. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and its write / read circuit. The recording control unit 14 may also take the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be attached to or detached from the imaging device 1, such as a memory card (such as a portable flash memory). The recording control unit 14 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0041] The display unit 15 is a display unit that displays various information to the user, and is, for example, a display panel or viewfinder using a display device such as a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL (Electro-Luminescence) display that is arranged on the housing of the imaging device 1. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18 . For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14. Furthermore, image data of the captured image that has been resolution converted for display by the image processing unit 20 is supplied to the display unit 15, and the display unit 15 may display based on the image data of the captured image in response to an instruction from the camera control unit 18. This allows the display of a so-called through image (monitoring image of the subject), which is an image captured while checking the composition or recording a video. Furthermore, based on instructions from the camera control unit 18, the display unit 15 displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface), on the screen.

[0042] The communication unit 16 performs wired or wireless data communication and network communication with external devices, such as transmitting and outputting still image files and video files containing captured image data and metadata to external information processing devices, display devices, recording devices, playback devices, etc. The communication unit 16 also serves as a network communication unit, and can communicate over various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data to and from servers, terminals, and the like on the network. The imaging device 1 may also be capable of mutual information communication with, for example, a PC, a smartphone, a tablet terminal, etc., via the communication unit 16, for example, by short-range wireless communication such as Bluetooth (registered trademark), Wi-Fi (registered trademark), or NFC (Near Field Communication), infrared communication, etc. The imaging device 1 may also be capable of mutual communication with other devices via wired connection communication. Therefore, the imaging device 1 can transmit the captured image and metadata via the communication unit 16 to an information processing device 70, which will be described later.

[0043] The operation unit 17 collectively refers to input devices that allow the user to input various operations. Specifically, the operation unit 17 refers to various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. The operation unit 17 detects the user's operation, and a signal corresponding to the input operation is sent to the camera control unit 18 .

[0044] The camera control unit 18 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 18. The illustrated memory unit 19 comprehensively represents, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The memory unit 19 may be a memory area built into the microcomputer chip that serves as the camera control unit 18, or may be configured as a separate memory chip. The camera control unit 18 controls the entire imaging device 1 by executing a program stored in the ROM or flash memory of the memory unit 19 . For example, the camera control unit 18 controls the operation of each necessary unit, such as controlling the shutter speed of the image sensor unit 12, issuing instructions for various signal processing in the image processing unit 20, imaging and recording operations in response to user operations, playback of recorded image files, operations of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, and user interface operations.

[0045] The RAM in the memory unit 19 is used as a work area for the CPU of the camera control unit 18 to process various data, and is used to temporarily store data, programs, and the like. The ROM and flash memory (non-volatile memory) in memory unit 19 are used to store the OS (Operating System) that the CPU uses to control each part, content files such as image files, application programs for various operations, firmware, various setting information, etc.

[0046] The driver section 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 18, and cause the focus lens and zoom lens to move, and the aperture blades of the aperture mechanism to open and close, etc.

[0047] The sensor unit 23 collectively represents various sensors mounted on the imaging device. When an IMU (inertial measurement unit), for example, is installed as the sensor unit 23, angular velocity can be detected using a three-axis angular velocity (gyro) sensor of pitch, yaw, and roll, and acceleration can be detected using an acceleration sensor. The sensor unit 23 may also include, for example, a position information sensor, an illuminance sensor, a distance measurement sensor, and the like. Various information detected by the sensor unit 23, such as position information, distance information, illuminance information, IMU data, etc., is added to the captured image as metadata together with date and time information managed by the camera control unit 18.

[0048] <3. Configuration of information processing device> Next, an example of the configuration of the information processing device 70 will be described with reference to FIG. The information processing device 70 is a device such as a computer that is capable of information processing, particularly image processing. Specific examples of the information processing device 70 include personal computers (PCs), mobile terminal devices such as smartphones and tablets, mobile phones, video editing devices, and video playback devices. The information processing device 70 may also be a computer configured as a server device or a computing device in cloud computing. This information processing device 70 is equipped with an image processing unit 20 that performs synthesis processing of RAW image data, and this image processing unit 20, or the information processing device 70 equipped with the image processing unit 20, can be considered an example of an image processing device of the present disclosure.

[0049] The CPU 71 of the information processing device 70 executes various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate.

[0050] The image processing unit 20 has the functions of the synthesis processing unit 31 and the development processing unit 32 described in the imaging device 1 above.

[0051] The synthesis processing unit 31 and development processing unit 32 as the image processing unit 20 may be provided as functions within the CPU 71. The image processing unit 20 may also be realized by a CPU separate from the CPU 71, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, or the like.

[0052] The CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74, and image processing unit 20 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0053] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0054] Furthermore, the input / output interface 75 is connected integrally or separately to a display unit 77 made up of an LCD or organic EL panel or the like, and an audio output unit 78 made up of a speaker or the like. The display unit 77 is a display unit that displays various types of information, and is configured by, for example, a display device provided in the housing of the information processing device 70, or a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface), based on instructions from the CPU 71.

[0055] The input / output interface 75 may be connected to a storage unit 79 configured with a HDD or solid-state memory, or a communication unit 80 configured with a modem or the like.

[0056] The storage unit 79 can store data to be processed and various programs. When the information processing device 70 functions as the image processing device of the present disclosure, it is expected that the memory unit 79 will store image data to be processed (e.g., RAW image data), composite RAW image data, or JPEG image data developed from composite RAW image data, etc. The storage unit 79 also stores programs for synthesis processing and development processing.

[0057] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like. Communication with the imaging device 1, particularly reception of captured images and the like, is performed by a communication unit 80.

[0058] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. Drive 81 allows data files such as image files and various computer programs to be read from removable recording medium 82. The read data files are stored in storage unit 79, and images and sounds contained in the data files are output on display unit 77 and audio output unit 78. Furthermore, computer programs and the like read from removable recording medium 82 are installed in storage unit 79 as needed.

[0059] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0060] <4. Compositing and developing RAW image data> The following describes the RAW image data synthesis process and development process performed by the image processing unit 20 in the imaging device 1 and the information processing device 70.

[0061] For example, the image processing unit 20 performs processing to synthesize multiple pieces of RAW image data and generate one or more pieces of synthesized RAW image data for the purpose of editing the shutter speed after capturing an image.

[0062] First, FIG. 4 shows a schematic flow of the development process in the image processing unit 20. As shown in FIG. For example, image data input from the image sensor unit 12 becomes RAW image data either as it is or after undergoing pre-processing (not shown). In Figure 4, the image data input from the image sensor unit 12 is shown as RAW image data, assuming that the image capturing device 1 is used. However, in the case of the information processing device 70, it can be considered as RAW image data read out from the memory unit 79, for example.

[0063] Development processing in a broad sense includes, for example, lens correction in step ST1, noise reduction (NR) in step ST2, demosaicing in step ST3, and color reproduction / sharpness processing in step ST4, which is development processing in a narrow sense. Then, for example, JPEG image data is generated as image data after such development processing.

[0064] As an example of the relationship between the development process and the composition process, a comparative example and an embodiment are shown in FIGS. 5A and 5B. Figure 5A is shown as a comparative example to the embodiment, and shows multiple JPEG image data (J#1 to J#n), each of which has been generated through development processing (steps ST1, ST2, ST3, ST4) on RAW image data. The plurality of JPEG image data (J#1 to J#n) are subjected to a synthesis process in step ST20 to generate synthesized JPEG image data.

[0065] As a process for combining a plurality of images, it is assumed that a plurality of JPEG image data (J#1 to J#n) after development processing is combined, as in this comparative example. However, in this case, the following problems can be pointed out: First, a broad development process (steps ST1, ST2, ST3, ST4) is required for each of the multiple image data used in compositing. For example, it would be fine if each of the individual JPEG image data (J#1 to J#n) was also to be saved, but if the goal is only to create a composite image, the processing load will increase.

[0066] Furthermore, the development process is performed using signal processing optimized for each image data, which is not necessarily optimal for the combined image data. Furthermore, since the composite image has been developed, its editability is limited, especially since some of the information that existed in the RAW image data has been lost due to thinning of color information during the color reproduction / sharpness processing in step ST4.

[0067] Therefore, in this embodiment, a synthesis process as shown in FIG. 5B is performed. FIG. 5B shows a plurality of RAW image data (R#1 to R#n) as image data to be subjected to the synthesis process. For example, the multiple RAW image data (R#1 to R#n) are subjected to the synthesis process in step ST10, thereby obtaining synthesized RAW image data. The composite RAW image data is then subjected to development processing in steps ST1, ST2, ST3, and ST4, thereby generating JPEG image data as a composite image.

[0068] In this way, development processing only needs to be performed once after compositing, reducing the signal processing load. Furthermore, since the development processing can be performed using parameters optimized for the composite RAW image data after compositing, the image quality of the generated developed image data, for example, the above-mentioned JPEG image data, can be improved. Furthermore, since the composite RAW image data has not undergone image processing such as color reproduction / sharpness processing, there is no loss of information due to image processing, and therefore it has high editability in terms of color reproduction, for example. In this way, by combining a plurality of pieces of RAW image data, it becomes possible to perform signal processing that is optimized for the combined image and is also efficient.

[0069] Here, consider a case where a plurality of RAW image data (R#1 to R#n) are image data recorded consecutively in time. This will generate a composite image with a shutter speed equal to the shutter speed at which one piece of RAW image data was captured multiplied by the number of composite images. For example, if the shutter speed is 1 / 100 seconds and 100 frames of RAW image data are continuously captured and combined, a composite image with a shutter speed of 1 second can be obtained.

[0070] With this concept, you can edit the shutter speed after capturing an image. An example of the process for this purpose will be explained with reference to FIGS.

[0071] First, FIG. 6 shows an example of processing performed by the imaging device 1 when capturing an image. The user can select the RAW blending mode in the imaging device 1. When the user performs an operation to select the RAW blending mode, the image processing unit 20 performs the illustrated process under the control of the camera control unit 18.

[0072] In step S101, the image processing unit 20 sets the RAW synthesis mode based on an instruction from the camera control unit 18. This is the setting of the operation mode in which the processing from step S103 onwards is performed.

[0073] In step S102, the image processing unit 20 waits for a release operation. The user performs a release operation, for example, by continuously pressing the shutter button, in the same manner as for continuous shooting. When the camera control unit 18 detects a release operation by the user, the image processing unit 20 proceeds from step S102 to step S103 in response to the release instruction from the camera control unit 18, and performs the subsequent processes.

[0074] When a release operation is performed, a continuous image capturing operation is performed in the image sensor unit 12. For example, image data is sequentially sent to the image processing unit 20 by an image capturing operation at a constant shutter speed. In step S103, the image processing unit 20 performs buffering of the RAW image data, that is, performs processing to store one frame of RAW image data input from the image sensor unit 12 in the buffer memory 21. The image processing unit 20 repeats the process of step S103 while monitoring the notification from the camera control unit 18 in step S104 that the release operation has been completed. As a result, multiple RAW image data are buffered until the release operation is completed.

[0075] When the release operation is completed, the image processing unit 20 proceeds to step S105 and performs additional processing on each of the multiple RAW image data temporarily stored in the buffer memory 21. This additional processing is processing to make the exposure periods of the multiple RAW image data continuous in time, that is, to ensure that there is no interruption in the exposure period. Specifically, for each of the multiple RAW image data, the processing adds an exposure amount corresponding to the non-exposure period of the image sensor 12a to the exposure amount during the exposure period of the image sensor 12a. The additional process of step S105 does not necessarily have to be started after the release operation is completed, but may be started while continuous shooting is being performed. 8, a specific example of the additional processing in step S105 will be described below, but there are cases where this additional processing itself is not performed. In other words, a processing example in which step S105 does not exist in FIG. 6 is also possible.

[0076] 8 shows an exposure period Ta and a readout period Tb of the charge due to exposure as the imaging operation of the image sensor 12a. In reality, there are periods when neither exposure nor readout is performed, as well as reset periods, but for the sake of simplicity of illustration and explanation, these are included in the readout period Tb.

[0077] One frame of image data is obtained through the operations of the exposure period Ta and the readout period Tb, and when continuous imaging is performed, the exposure period Ta and the readout period Tb are repeated. However, during the readout period Tb, the image sensor 12a is not exposed to light, so even if multiple pieces of RAW image data are combined as they are, the result will not be exactly the same as if the shutter speed were simply slowed down.

[0078] Of course, the compositing process in the sense of editing the shutter speed does not necessarily have to be in the strict sense, so it is possible to edit the shutter speed in a pseudo sense by compositing multiple RAW image data sets with gaps in the exposure period Ta. In this case, the additional processing of step S105 may be unnecessary.

[0079] On the other hand, to edit the shutter speed in a precise sense, it is appropriate to combine a plurality of RAW image data sets without any break in the exposure period Ta. In reality, the exposure period Ta is interrupted due to the need for the readout period Tb, but by performing a process of adding the exposure amount as pixel information for that readout period to the pixel value, image data without any interruption in the exposure period Ta is generated. As an example, the pixel values ​​of each frame are interpolated to calculate the pixel values ​​as if exposure had also been performed during the readout period Tb, taking into account the ratio of the length of the exposure period Ta to the readout period Tb and the amount of change from the pixel values ​​of the previous and next frames before additional processing. Then, additional processing is performed on each of these pixel values ​​to obtain RAW image data R#1, R#2, R#3, etc. As a result, the RAW image data R#1, R#2, R#3, . . . can be a plurality of RAW image data without any break in the exposure period Ta.

[0080] Incidentally, by providing a plurality of image sensors 12a1 and 12a2 in the imaging element unit 12, it is possible to obtain a plurality of RAW image data without interruption during the exposure period Ta without performing additional processing. 9, for example, RAW image data R#1 is obtained from the output of image sensor 12a1, and then RAW image data R#2 is obtained from the output of image sensor 12a2. At this time, image sensors 12a1 and 12a2 are operated in synchronization with a difference in exposure timing. That is, as shown in the figure, exposure period Ta of image sensor 12a2 starts at the end timing of exposure period Ta of image sensor 12a1, and exposure period Ta of image sensor 12a1 starts at the end timing of exposure period Ta of image sensor 12a2. In this way, the RAW image data R#1, R#2, R#3, R#4, . . . obtained alternately from the image sensors 12a1 and 12a2 become image data with continuous exposure periods.

[0081] Furthermore, even when one image sensor 12a is used, additional processing may not be performed if the image sensor 12a is one in which interruptions in the exposure period due to the readout period Tb can be almost ignored, or if the image to be ultimately obtained does not include a moving subject, etc. In other words, additional processing may be enabled for imaging devices or imaging scenes in which interruptions in the exposure period have a significant effect.

[0082] After the additional processing is performed in step S105 of Fig. 6, or after it is determined in step S104 that the release operation has ended if the additional processing of step S105 is not performed as part of the processing in Fig. 6, the image processing unit 20 packs and records the multiple RAW image data R#1, R#2, ... as a group of image data in step S106. Packing can be performed using HEIF, for example. For example, the image processing unit 20 may transfer a plurality of RAW image data to the recording control unit 14, which then records the grouped image data on a recording medium. Alternatively, the communication unit 16 may transmit the plurality of RAW image data as a grouped image data to an external device, such as the information processing device 70.

[0083] For the plurality of RAW image data packed as described above, post-capture shutter speed editing can be performed in the image capturing device 1 or the information processing device 70 by the process of FIG. In the case of the imaging device 1, the target is a group of RAW image data recorded on a recording medium. In addition, in the case of the information processing device 70, the processing shown in Figure 7 can be performed on a group of RAW image data stored in the memory unit 79, which has been received from the imaging device 1 or imported via a removable recording medium 82.

[0084] In step S150, the image processing unit 20 sets a designated shutter speed or a designated number of images to be combined. For example, the user can use the operation unit 17 (or the input unit 76) via a user interface to specify an arbitrary shutter speed and an arbitrary number of images to be combined. When the camera control unit 18 (or CPU 71) detects a user operation, it notifies the image processing unit 20. In response to this, the image processing unit 20 sets the shutter speed or the number of images to be combined.

[0085] Setting the shutter speed means dividing the shutter speed at the time of image capture by the determined shutter speed to calculate the number of RAW image data to be subjected to the synthesis process. Also, setting the number of images directly means determining the number of RAW image data to be subjected to the synthesis process.

[0086] In step S151, the image processing unit 20 sets the range of RAW image data to be combined. For example, assume that 50 pieces of RAW image data, from RAW image data R#1 to combined RAW image data R#50, are packed as temporally consecutive image data. In this case, the range from the start image, which is the first in time, to the end image, which is the last in time, among the 50 pieces of RAW image data, is set. For example, the range from composite RAW image data R#10 to RAW image data R#30 is set as the range to be composited. In response to the user's operation to specify a start image, an end image is determined from the start image at a specified shutter speed (or specified number of images), and the range of the object to be combined is determined.

[0087] In this processing example, the shutter speed or number of images is set in step S150, so if the user specifies either the start image or the end image in step S151, the range of image data to be synthesized can be set. Alternatively, the user may specify a start image and an end image as desired, without specifying the shutter speed or the number of images, so that a composite image with a desired shutter speed can be obtained.

[0088] In step S152, the image processing unit 20 performs a synthesis process on the plurality of RAW image data within the range set in step S151. At this time, a simple compositing process may be performed in which pixel values ​​are combined (for example, averaged) for each pixel, or a compositing process may be performed in which multiple images are combined with different weightings.Furthermore, a compositing process may be performed in which pixel positions in a composite RAW image of a specific part of the subject are fixed and combined to create an image in which the specific part is not blurred.

[0089] The synthesis process results in synthesized RAW image data. In step S153, the image processing unit 20 performs development processing on the combined RAW image data, thereby generating, for example, JPEG image data. In step S154, the image processing unit 20 outputs image data to be recorded. For example, in the case of the imaging device 1, the JPEG image data and composite RAW image data are transferred to the recording control unit 14, which then records them on a recording medium. In the case of the information processing device 70, the image processing unit 20 transfers the JPEG image data and composite RAW image data to the storage unit 79, which then records them. The image data to be recorded may be only JPEG image data that has undergone development processing, or may be only composite RAW image data. If you record the composite RAW image data, you can then perform image editing and development processes on the composite image with a high degree of freedom.

[0090] The JPEG image data and composite RAW image data recorded in this manner can be said to be image data that has been edited to an arbitrary shutter speed after being captured.

[0091] <5. Composition and development of YC-RAW image data> In the above example, the compositing process is performed on RAW image data obtained by the image sensor unit 12. However, the same concept can be applied to the compositing of YC-RAW image data in the form of luminance values ​​and chroma values. The case of YC-RAW image data will be described below. For the sake of convenience, hereinafter, RAW image data in R, G, B format obtained by the image sensor unit 12 will be referred to as "RGB-RAW image data" to distinguish it from YC-RAW image data. Furthermore, composite RAW image data made from RGB-RAW image data will be referred to as "composite RGB-RAW image data."

[0092] FIG. 10 shows the flow of processing by the image processing unit 20 when obtaining YC-RAW image data. The RGB-RAW image data input from the image sensor unit 12 undergoes broad development processing, such as lens correction in step ST1, NR in step ST2, and demosaic in step ST3, and then color reproduction / sharpness processing in step ST4, which is narrow development processing, to generate, for example, JPEG image data. In this case, YC conversion is performed in step ST5 after demosaicing, thereby obtaining YC-RAW image data. It should be noted that when generating YC-RAW image data, the lens correction in step ST1 and the NR in step ST2 do not have to be performed.

[0093] By packing and recording multiple temporally consecutive images as a group as this type of YC-RAW image data, it becomes possible to perform synthesis processing using the multiple YC-RAW image data at a later point in time. In particular, with YC-RAW image data, since it is in a state before color reproduction / sharpness processing is applied, the information at the RAW image data stage is not lost, and so the same advantages as those obtained when combining RAW image data as described above are obtained.

[0094] 11A, 11B, and 11C show examples of synthesis processing using YC-RAW image data. 11A shows an example in which multiple YC-RAW image data (YCR#1 to YCR#n) are subjected to a synthesis process in step ST11 to generate synthesized YC-RAW image data. In addition, the synthesized YC-RAW image data can be subjected to color reproduction / sharpness processing in step ST4 to generate developed image data such as JPEG image data as a synthesized image.

[0095] Figure 11B shows an example in which the synthesis process of step ST10 is performed on multiple RGB-RAW image data (R#1 to R#n) to generate synthetic RGB-RAW image data, and the processes of steps ST1, ST2, ST3, and ST5 are performed to generate synthetic YC-RAW image data.

[0096] 11C shows an example in which multiple pieces of YC-RAW image data are restored to RGB-RAW image data (R#1 to R#n) and then combined. The multiple pieces of RGB-RAW image data (R#1 to R#n) restored from the multiple pieces of YC-RAW image data are subjected to the combining process of step ST10 to generate combined RGB-RAW image data, and steps ST1, ST2, ST3, and ST4 are performed to generate, for example, JPEG image data as a combined image.

[0097] As in the above examples, various examples of synthesis processing using YC-RAW image data are possible.

[0098] <6. Summary and Variations> According to the above embodiment, the following effects can be obtained. The image processing device of the embodiment, i.e., the image processing unit 20 (or the imaging device 1 or information processing device 70 equipped with the image processing unit 20), is equipped with a synthesis processing unit 31 that synthesizes multiple RAW image data, each of which constitutes one frame of image, to generate synthetic RAW image data. Performing the compositing process to combine multiple image data at the RAW image data stage means that the multiple image data are combined before development processing is performed on them, and then development processing is performed. Then, performing development processing on the composite RAW image data to generate JPEG image data or the like means that development processing appropriate for that composite RAW image data can be performed. The multiple RAW image data to be combined may be captured as temporally consecutive still images, or may be RAW image data of each frame captured as a moving image. Furthermore, the multiple RAW image data to be combined may be multiple RAW image data that are unrelated to each other, such as having no relationship in terms of capture time or subject. In other words, it is also possible to combine multiple arbitrary RAW image data selected by the user.

[0099] The image processing unit 20 according to the embodiment further includes a development processing unit 32 that performs development processing on the composite RAW image data generated by the synthesis processing unit 31 to generate image data in a predetermined format. By doing this, development processing is performed using various parameters suitable for the composite RAW image data, and image data in a predetermined format after synthesis, such as JPEG image data, is obtained, thereby improving the image quality of the generated composite image (for example, an image based on JPEG image data). Furthermore, since it is not necessary to perform development processing on each of the multiple pieces of RAW image data before combining, the efficiency of development processing can be improved. In addition, composite RAW image data is created by combining RAW image data before development processing, including compression processing, and since much of the information from the original image remains, it also has the advantage of giving users greater freedom when creating images. In addition, by saving the composite RAW image data, it is also suitable for developing in the future when development processing technology advances.

[0100] The example in which the synthesis processing unit 31 of the embodiment generates, through synthesis processing, synthesized RAW image data corresponding to an image captured at a shutter speed slower than the shutter speed at the time of image capture has been given. For example, by combining multiple RAW image data related in terms of subject content, it is possible to obtain composite RAW image data corresponding to an image captured at a shutter speed slower than the shutter speed at which each of the RAW image data was captured.

[0101] In the embodiment, an example has been given in which the multiple RAW image data combined by the combining processing unit 31 are RAW image data of multiple frames that are continuous in time. This makes it possible to generate an image after capture that looks as if it was captured at a shutter speed slower than the shutter speed at which the image was captured, which means that the shutter speed can be edited after capture. This also makes it possible to prevent mistakes due to incorrect shutter speed settings at precious photo opportunities, for example. You will no longer miss a photo opportunity because you have to adjust the shutter speed. Furthermore, since the shutter speed during image capture is not changed, there is no need to adjust the amount of light, and a composite image at any shutter speed can be obtained from images captured with an appropriate amount of light. Furthermore, fine-tuning the shutter speed after capturing the image can lead to new image expressions. In addition, development processing can be performed on the combined RAW image data after editing the shutter speed.

[0102] In the embodiment, an example has been given in which the plurality of RAW image data to be combined by the combining processing unit 31 is RAW image data of a plurality of frames that are temporally continuous at a constant shutter speed. The multiple RAW image data obtained by capturing images are image data of the same shutter speed, so by simply setting the range of RAW image data to be combined, you can obtain a combined image of any shutter speed.

[0103] In the embodiment, an example has been given in which the multiple RAW image data to be synthesized by the synthesis processing unit 31 are RAW image data that are adjacent in time and have mutually continuous exposure periods (see Figures 8 and 9). By making the multiple RAW image data into a continuous state with no interruption in the exposure period, the composite RAW image data becomes image data that can be treated as equivalent to an image captured at a shutter speed slower than the actual shutter speed at the time of capture. In other words, it becomes possible to edit the shutter speed in the original sense of the word, i.e., to lengthen or shorten the continuous exposure period. This makes it possible to change the shutter speed after shooting, something that normally cannot be done after shooting.

[0104] In the embodiment, an example has been given in which the multiple RAW image data to be synthesized by the synthesis processing unit 31 are subjected to additional processing in which an exposure amount corresponding to a non-exposure period at the image sensor is added to the exposure amount during the exposure period at the image sensor, thereby making the exposure periods continuous in time (see Figure 8). As a result, even if multiple RAW image data have non-exposure periods such as readout periods and charge reset periods, they can be considered as data obtained without these non-exposure periods. Therefore, a series of RAW image data can be processed as multiple RAW image data with temporally continuous exposure periods.

[0105] In the embodiment, an example has also been given in which multiple RAW image data to be synthesized by the synthesis processing unit 31 are read out alternately and sequentially from multiple image sensors 12a1, 12a2, thereby making the exposure periods continuous in time (see Figure 9). By using multiple image sensors 12a1 and 12a2 and setting the exposure timing of the image sensors 12a1 and 12a2 so that one is exposed during the non-exposure period of the other, and by reading out RAW image data alternately and sequentially from the image sensors 12a1 and 12a2, the series of RAW image data becomes multiple RAW image data whose exposure periods are continuous in time.

[0106] In the embodiment, an example has been given in which the combining processing unit 31 selects RAW image data to be combined from a plurality of RAW image data that are consecutive in time based on a specified shutter speed (see FIG. 7). This allows for composition processing according to the user's shutter speed designation operation, and the composite RAW image data becomes an image that looks as if it was captured at the specified shutter speed, making it easy to use in terms of shutter speed editing.

[0107] In the embodiment, an example has also been given in which the synthesis processing unit 31 selects RAW image data to be subjected to synthesis processing from multiple pieces of RAW image data that are consecutive in time based on the specified number of images (see FIG. 7). This allows the compositing process to be performed according to the user's designation of the number of images, and the composite RAW image data becomes a composite image of the designated number of images. This provides the user with a designation method different from designation by shutter speed, which may be easier to use depending on the user. By allowing the user to switch between an operation mode in which the shutter speed is specified and an operation mode in which the number of shots is specified, usability that meets the user's preferences can be realized.

[0108] In the embodiment, an example has been given in which the RAW image data is RGB-RAW image data having the same color arrangement as the color arrangement of the imaging element. For example, this is image data in R, G, B format. It may be composed of pixel values ​​of G1, G2, R, and B pixels in a Bayer array, or image data that has undergone some of the processing described above. RAW image data composed of such pixel values ​​is the original image information read from the image sensor unit 12, and offers a high degree of freedom in post-combination development processing.

[0109] In addition, in the embodiment, an example of RAW image data is given as YC-RAW image data, which is image data in the form of brightness values ​​and chroma values ​​read out from the image sensor and has not been subjected to color reproduction / sharpness processing. For YC-RAW image data, the efficiency of development processing can be improved by combining the images and performing the compositing process. Furthermore, YC-RAW image data can be restored to RAW image data, so that compositing and subsequent development can be performed after the image data has been restored to RAW image data.

[0110] The program according to the embodiment is a program that causes, for example, a CPU, a DSP, a GPU, a GPGPU, an AI processor, or a device including any of these, to execute the processes shown in FIGS. That is, the program according to the embodiment is a program that causes an information processing apparatus to execute a synthesis process that synthesizes a plurality of RAW image data, each of which constitutes an image of one frame, to generate synthesized RAW image data. Such a program allows the image processing device referred to in the present disclosure to be realized by various computer devices.

[0111] These programs can be recorded in advance on a HDD as a recording medium built into a device such as a computer, or on a ROM in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0112] Furthermore, such a program is suitable for providing the image processing device of the present disclosure to a wide range of devices. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the image processing device of the present disclosure.

[0113] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0114] The present technology can also be configured as follows. (1) A synthesis processing unit is provided that synthesizes multiple pieces of RAW image data, each of which constitutes one frame of image, to generate synthesized RAW image data. Image processing device. (2) The image processing device further includes a development processing unit that performs development processing on the composite RAW image data generated by the synthesis processing unit to generate image data in a predetermined format. The image processing device according to (1) above. (3) The synthesis processing unit generates, through synthesis processing, synthesized RAW image data corresponding to an image captured at a shutter speed slower than the shutter speed at the time of image capture. The image processing device according to (1) or (2) above. (4) The plurality of RAW image data to be synthesized by the synthesis processing unit are RAW image data of a plurality of frames that are consecutive in time. The image processing device according to any one of (1) to (3) above. (5) The plurality of RAW image data to be synthesized by the synthesis processing unit are RAW image data of a plurality of frames that are temporally continuous at a constant shutter speed. The image processing device according to any one of (1) to (4) above. (6) The image processing device described in any one of (1) to (5) above, wherein the multiple RAW image data to be synthesized by the synthesis processing unit are multiple RAW image data that are adjacent in time and whose exposure periods are consecutive in time. (7) The plurality of RAW image data to be synthesized by the synthesis processing unit are subjected to additional processing of adding an exposure amount corresponding to a non-exposure period of the image sensor to the exposure amount during the exposure period of the image sensor, so that the exposure periods are continuous in time. The image processing device according to (6) above. (8) The plurality of RAW image data to be synthesized by the synthesis processing unit are read out alternately and sequentially from the plurality of image sensors, so that the exposure periods are continuous in time. The image processing device according to (6) above. (9) The synthesis processing unit Based on the specified shutter speed, the RAW image data to be combined is selected from multiple RAW image data that are consecutive in time. An image processing device according to any one of (1) to (8) above. (10) The synthesis processing unit Selects RAW image data to be combined from multiple chronologically consecutive RAW image data based on the specified number of images. The image processing device according to any one of (1) to (9) above. (11) The image processing device according to any one of (1) to (10) above, wherein the RAW image data is image data having the same color arrangement as the color arrangement of an imaging element. (12) The RAW image data is image data in which pixel values ​​read from an image sensor are converted into luminance and chroma values, and is image data that has not been subjected to color reproduction / sharpness processing. The image processing device according to any one of (1) to (10) above. (13) A synthesis process that synthesizes multiple RAW image data, each of which makes up one frame of image, to generate composite RAW image data. An image processing method executed by an image processing device. (14) A synthesis process that synthesizes multiple RAW image data, each of which makes up one frame of image, to generate composite RAW image data. A program to be executed by an information processing device. [Explanation of symbols]

[0115] 1. Imaging device 12 Image sensor section 12a, 12a1, 12a2 Image sensors 14 Recording control section 15 Display section 16 Communications Department 17 Control section 18 Camera control unit 19 Memory section 20 Image processing section 21 Buffer memory 31 Image synthesis unit 32 Development Processing Section 70 Information processing device, 71 CPU 79 Memory section 80 Communications Department

Claims

1. a synthesis processing unit that synthesizes a plurality of pieces of RAW image data, each of which constitutes an image of one frame, to generate synthesized RAW image data; The plurality of RAW image data to be synthesized by the synthesis processing unit are RAW image data that are captured successively in time with a single image sensor with a non-exposure period interposed therebetween and that have been subjected to processing to make the exposure periods of the respective RAW image data successive in time. Image processing device.

2. The plurality of RAW image data to be combined by the combining processing unit are subjected to additional processing of adding an exposure amount corresponding to a non-exposure period of the image sensor to the exposure amount during the exposure period of the image sensor, so that the exposure periods of the respective sets of RAW image data are temporally continuous. The image processing device according to claim 1 .

3. The image processing device further includes a development processing unit that performs development processing on the composite RAW image data generated by the synthesis processing unit to generate image data in a predetermined format.

3. The image processing device according to claim 1.

4. The synthesis processing unit generates, through synthesis processing, synthesized RAW image data corresponding to an image captured at a shutter speed slower than the shutter speed at which the image was captured.

3. The image processing device according to claim 1.

5. The plurality of RAW image data to be synthesized by the synthesis processing unit are RAW image data of a plurality of frames that are temporally continuous at a constant shutter speed.

3. The image processing device according to claim 1.

6. The synthesis processing unit Based on the specified shutter speed, RAW image data to be subjected to compositing processing is selected from a plurality of RAW image data that are consecutive in time.

3. The image processing device according to claim 1.

7. The synthesis processing unit Based on the specified number of images, RAW image data to be subjected to compositing processing is selected from a plurality of RAW image data that are consecutive in time.

3. The image processing device according to claim 1.

8. The RAW image data is image data having the same color arrangement as the color arrangement of the image sensor. The image processing device according to claim 1 .

9. The RAW image data is image data in which pixel values ​​read from an image sensor are converted into luminance and chroma values, and is image data that has not been subjected to color reproduction / sharpness processing.

3. The image processing device according to claim 1.

10. An image processing method in which an image processing device executes a synthesis process to generate synthesized RAW image data by synthesizing a plurality of RAW image data, each of which constitutes an image of one frame, The plurality of RAW image data to be combined in the combining process are RAW image data that are captured successively in time with a single image sensor with a non-exposure period interposed therebetween, and are processed so that the exposure periods of the RAW image data are successive in time. Image processing methods.

11. a program that causes an information processing device to execute a synthesis process that synthesizes a plurality of RAW image data, each of which constitutes an image of one frame, to generate synthesized RAW image data; The plurality of RAW image data to be combined in the combining process are RAW image data that are captured successively in time with a single image sensor with a non-exposure period interposed therebetween, and are processed so that the exposure periods of the RAW image data are successive in time. program.

Images