Control device, control method, and program

Abstract

Provided is a suitable user interface for synthesis processing of a plurality of temporally consecutive items of image data. A control device is provided with a user interface control unit which, from among a group of images from a plurality of temporally consecutive items of image data, simultaneously displays some of the images, which include an image which is temporally at the start of a synthesis range that is to undergo synthesis processing and an image which is temporally at the end of said synthesis range.

Description

[Technical field]

[0001] The present technology relates to a control device, a control method, and a program, and in particular to a user interface for image synthesis processing. [Background technology]

[0002] It is known that image synthesis processing is performed to synthesize a plurality of images to produce a variety of images. The following Patent Document 1 discloses a technology that enables easy realization of image effects such as front-curtain sync / rear-curtain sync / multiple flash using long exposure by performing image synthesis without requiring any expertise, and a user interface for this purpose. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-232385 A Summary of the Invention [Problem to be solved by the invention]

[0004] When performing a compositing process, a user must select multiple pieces of image data to be subjected to the compositing process. However, as the amount of image data to be composited increases, the user interface screen may become cluttered and the visibility and operability of each image may decrease.

[0005] In view of the above, an object of the present disclosure is to propose a user interface that improves visibility and operability in operations for image synthesis. [Means for solving the problem]

[0006] The control device related to the present technology includes a user interface control unit that controls the simultaneous display of a portion of images, including a first image that is the first in time and a last image that is the last in time in a synthesis range to be subjected to synthesis processing, from a group of images made up of multiple image data that are consecutive in time. At least the first image and the last image of the time range to be combined are displayed simultaneously so that the user can confirm them. The user interface control unit then controls the display of the first image or the last image to display a fixed area or a candidate for a fixed area that fixes the position within the frame between each image in the synthesis process, and also controls the display to indicate its suitability as a fixed area. For example, the suitability of a fixed region or a candidate for a fixed region may be presented to the user by differences in numerical values ​​or display modes.

[0007] In the control device related to the above-mentioned present technology, it is considered that the user interface control unit performs control so that the first image and the last image are displayed in an arrangement in which they are displayed side by side on one screen without performing a screen transition. For example, the first and last images can be displayed simultaneously and compared without scrolling the screen.

[0008] In the control device related to the above-mentioned present technology, it is considered that the user interface control unit performs display control of the shutter speed for a composite image obtained by a synthesis process of multiple images included in a synthesis range from the first image to the last image. By synthesizing images that are consecutive in time, the resulting composite image will have a different shutter speed from the individual images within the composite range. The shutter speed is displayed.

[0009] In the control device according to the present technology described above, it is conceivable that the user interface control unit changes the setting of the leading image or the trailing image in response to a change operation of the leading image or the trailing image. To allow a user to arbitrarily select either or both of a leading image and a trailing image.

[0010] In the control device related to the above-mentioned present technology, the user interface control unit may execute a timeline display for the group of images, and the operation on the timeline display may be an operation to change the first image or the last image. For example, a slide bar representing the time axis of the image group may be displayed and operated by the user.

[0011] In the control device according to the present technology described above, it is considered that the user interface control unit changes the image to be displayed as the leading image or the trailing image in response to an operation. For example, the first image and the last image are switched by a playback operation or a frame-by-frame operation.

[0012] In the control device according to the present technology described above, it is considered that the user interface control unit sets the first image or the last image in response to an input operation of a shutter speed. For example, a user interface screen is provided that allows the shutter speed to be input.

[0013] In the control device related to the above-mentioned present technology, it is considered that the user interface control unit performs control to display a fixed area on the display of the first image or the last image, which fixes the position within the frame between each image during the synthesis process. When combining multiple images, a specific subject area in a captured image is set as a fixed area, which is an area (pixel position) in a frame of each image to be combined that is fixed to the same pixel position.

[0014] In the control device related to the above-mentioned technology, it is considered that the user interface control unit performs processing to set a fixed area within the frame of the first image or the last image in response to an operation to specify a fixed area that fixes the position within the frame between each image in the synthesis processing. That is, a specific subject area in a captured image can be set as a fixed area in response to a user operation.

[0015] In the control device related to the above-mentioned technology, the user interface control unit may perform control to display candidates for fixed areas that fix the position within the frame between each image in the synthesis process on the display of the first image or the last image, and may perform processing to set a fixed area within the frame of the first image or the last image in response to an operation specifying the candidate fixed area. For example, candidates for fixed regions are presented on an image so that the user can specify them.

[0017] In the control device related to the above-mentioned present technology, the user interface control unit performs user interface control that allows the processing method of the compositing processing to be selected, and is considered to set the processing method of the compositing processing in response to a selection operation of the processing method of the compositing processing. As a synthesis process for a plurality of images, a plurality of synthesis process methods are conceivable, such as an averaging method, a comparative brightness method, a weighting method, etc. A user interface is provided that allows these to be selected.

[0018] In the control device according to the present technology described above, it is considered that the user interface control unit performs control to simultaneously display a composite image together with the leading image and the trailing image. In addition to the leading image and the trailing image, a composite image generated by a synthesis process of the synthesis range indicated by the leading image and the trailing image is also displayed at the same time.

[0019] In the control device according to the present technology described above, it is considered that the user interface control unit performs control to simultaneously display an operator for instructing a synthesis process together with the leading image and the trailing image. It is possible to instruct the actual synthesis process on the screen that displays the first image and the last image.

[0020] In the control device according to the present technology described above, it is considered that each image data of the image group is image data in which the exposure periods are temporally continuous when the images are captured. For example, image data that is continuous in time is a series of image data that does not include a non-exposure period.

[0021] In the control device according to the present technology described above, it is considered that each image data of the image group is RAW image data. A group of RAW image data obtained by capturing images is synthesized, that is, synthesized at a stage before development processing.

[0022] In the control device according to the present technology described above, it is considered that each image data of the image group is image data that is temporally continuous by continuous still image capturing or video capturing. That is, it is image data of a plurality of frames as a series of still images obtained by one continuous shooting, or image data of each frame constituting a moving image obtained by shooting a moving image.

[0023] The control device according to the present technology described above may further include a synthesis processing unit that synthesizes a plurality of images included in a synthesis range from the first image to the last image set by the user interface control unit. For example, the synthesis process is performed within a range set on the user interface screen.

[0024] The control method related to the present technology is a control method in which a control device performs user interface control to simultaneously display a portion of images, including a first image that is the first in time and an end image that is the last in time in a synthesis range to be subjected to synthesis processing, from a group of images made up of multiple image data that are consecutive in time. The user interface control displays a fixed area or a candidate for a fixed area that fixes the position within the frame between each image in the synthesis process on the display of the first image or the last image, and also controls to display a display indicating the suitability of the area as a fixed area. This provides a user interface for the synthesis process. The program according to the present technology is a program for causing a processor to execute this user interface control, thereby enabling the above-mentioned control device to be realized. [Brief description of the drawings]

[0025] [Figure 1]1 is an explanatory diagram of a synthesis process and a user interface according to an embodiment of the present technology; [Diagram 2] 1 is a block diagram of an imaging apparatus according to an embodiment; [Diagram 3] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 4] FIG. 4 is an explanatory diagram of a development process of image data. [Diagram 5] 4A to 4C are explanatory diagrams of a synthesis process and a development process according to an embodiment of the present invention. [Figure 6] 4A to 4C are explanatory diagrams of a synthesis process and a development process according to an embodiment of the present invention. [Figure 7] 11A and 11B are explanatory diagrams of additional processing for making exposure periods continuous in the embodiment. [Figure 8] 10A to 10C are explanatory diagrams of capturing RAW image data with continuous exposure periods in an embodiment. [Figure 9] FIG. 2 is an explanatory diagram of a screen configuration of a UI screen according to an embodiment. [Figure 10] FIG. 4 is an explanatory diagram of an example of a UI screen according to the embodiment. [Figure 11] FIG. 11 is an explanatory diagram of setting a fixed area on a UI screen according to an embodiment. [Figure 12] 11 is a flowchart of a UI control process according to the first embodiment. [Figure 13] 13 is a flowchart of a UI control process according to the second embodiment. [Figure 14] 13 is an explanatory diagram of displaying fixed region candidates and suitability according to the second embodiment; FIG. [Figure 15] FIG. 11 is an explanatory diagram of a display of suitability according to the second embodiment. [Figure 16] FIG. 11 is an explanatory diagram of a change in a synthesis range by specifying a fixed region according to the second embodiment. [Figure 17] 11A and 11B are diagrams illustrating examples of synthesis modes according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The embodiments will be described below in the following order. <1. Overview> 2. Configuration of the imaging device <3. Configuration of information processing device> <4. Compositing and development processing> <5.UI processing example> <6. Example of UI Control in First Embodiment> <7. Example of UI Control in the Second Embodiment> <8. Examples of blending modes> <9. Summary and Variations>

[0027] In this disclosure, the term "synthesis range" is used in relation to image synthesis processing, and this "synthesis range" refers to a time range. Synthesis processing in this disclosure is processing for synthesizing image data of multiple frames that are consecutive in time. In this case, the range of image data of each of the multiple frames to be synthesized is called the synthesis range. Furthermore, within that synthesis range, the image of image data that is the first in time is called the "leading image," and the image of image data that is the last in time is called the "last image." The synthesis range is set by a user operation, or may be set automatically. The term "image group" refers to image data of multiple frames that are consecutive in time and for which a synthesis range can be set. The entire or partial section of this image group becomes the above-mentioned "synthesis range."

[0028] In addition, in this disclosure, the meaning of "continuous in time" with respect to multiple image data may mean that the data is completely continuous in time, but is not limited to this and broadly refers to data that is continuous in time even if it is intermittent. For example, each piece of image data that becomes a plurality of frames obtained by continuous shooting of still images or general video recording is also "multiple image data that are continuous in time." In addition, each piece of image data as multiple still images when a user of an imaging device takes a single shot (taking a single still image) several times instead of continuous shooting can also be considered as "multiple image data that are continuous in time" because the context of each image data can be identified. In other words, a plurality of image data items that are temporally related correspond to "a plurality of image data items that are temporally consecutive."

[0029] Although the image data to be synthesized will be described mainly as an example of synthesizing a RAW image, the image data is not limited to this and may be image data in various formats such as JPEG (Joint Photographic Experts Group) format. A RAW image refers to an image before some or all of the development process is performed by an imaging device, etc. Image data constituting one image (one frame) as a RAW image is called RAW image data. Specifically, there are various types of image data called RAW image data, but in this embodiment, image data having the same color arrangement as that of an image sensor is taken as an example of RAW image data. Note that image data that has been subjected to the same defect correction and has the same color arrangement as that of an image sensor is also included in RAW image data. For example, when an image sensor outputs pixel signals of R (red), G (green), and B (blue), this RAW image data is image data in the R, G, B format. Note that RAW image data also includes image data containing pixel values ​​of W (white) as well as R, G, and B. Furthermore, in the case of an image sensor with a Bayer array, image data containing pixel values ​​of G1 (green 1), G2 (green 2), R (red), and B (blue) is RAW image data. There is also an image data format in which the pixel values ​​of G1, G2, R, and B are grouped together as four separate channels. For example, this is image data in which one frame is composed of only G1, G2, R, and B, and then each channel is compressed. In other words, this is image data as G1 image data, G2 image data, R image data, and B image data. This is also included in the example of RAW image data. In addition, image data read from an imaging element in the form of luminance values ​​and chroma values, and image data that has not been subjected to color reproduction / sharpness processing, is also sometimes called RAW image data. Although this is also a type of RAW image data in this disclosure, for the sake of distinction, it will be called "YC-RAW image data."

[0030] <1. Overview> The control device of the embodiment is assumed to be mounted as a control unit in an imaging device (camera) or an information processing device that performs image editing, etc. In addition, the imaging device or information processing device that mounts these control units can be considered as the control device of the present disclosure.

[0031] Such a control device performs control to provide a user interface (also referred to as "UI") screen for synthesis processing of a group of images based on multiple pieces of image data that are successive in time. FIG. 1 shows a schematic diagram of a process in which a plurality of RAW image data, from RAW image data R#1 to RAW image data R#n, are input through an imaging operation and then subjected to a synthesis process. In addition, the compositing process in this embodiment may be a compositing process in which multiple image data are superimposed by aligning the frame positions of each other, or a compositing process in which the positions of specific subjects in multiple image data are align and superimposed.

[0032] For example, the multiple RAW image data (R#1 to R#n) are image data of still images captured continuously by continuous shooting, image data of each frame captured as a video, etc. These multiple image data form an image group having a relationship as temporally consecutive frames. The control device provides a UI screen 100 to a user who wishes to perform a synthesis process on such multiple pieces of RAW image data (R#1 to R#n). The user can check the image contents on the UI screen 100 and set the synthesis range by determining the leading and trailing images.

[0033] The synthesis process is carried out based on the settings made on the UI screen 100. As a result, for example, composite RAW image data is generated. One piece of composite RAW image data may be generated from a plurality of pieces of RAW image data, or a plurality of pieces of composite RAW image data may be generated in a number less than the number of pieces of RAW image data. For example, when RAW image data of 10 consecutive images (10 frames) is input as an image group, it is possible to selectively use the RAW image data to be synthesized, such as RAW image data synthesized from all 10 images, or RAW image data synthesized from seven of the 10 images.

[0034] Also, development processing may be performed on the composite RAW image data. This allows image data in a predetermined format, such as JPEG image data, to be obtained. Of course, the predetermined format of image data is not limited to JPEG format. For example, HEIF (High Efficiency Image File Format), YUV422, YUV420, etc. may be used.

[0035] In the following description of the embodiment, a case will be mainly described as an example in which a plurality of pieces of chronologically consecutive RAW image data are synthesized to generate synthesized RAW image data. For example, multiple RAW image data that have been recorded consecutively can be synthesized to generate one synthetic RAW image data. This achieves the same effect as a long exposure. In addition, by setting the number of RAW image data to be synthesized, the synthetic RAW image data can be created at any shutter speed different from the shutter speed at which each RAW image data was captured. In addition, the composite RAW image data obtained by the synthesis process can also be subjected to lens correction, NR (noise reduction), demosaic, color reproduction, sharpness processing, etc., just like in normal development processing.

[0036] 2. Configuration of the imaging device An example of the configuration of the imaging device 1 will be described with reference to FIG. This imaging device 1 is equipped with a UI control unit 30 that performs UI control for synthesis, and the imaging device 1 equipped with this UI control unit 30, or a processor (camera control unit 18) equipped with the UI control unit 30, can be considered as an example of a control device of the present disclosure.

[0037] The imaging device 1 has, for example, a lens system 11, an imaging element unit 12, a recording control unit 14, a display unit 15, a communication unit 16, an operation unit 17, a camera control unit 18, a memory unit 19, an image processing unit 20, a buffer memory 21, a driver unit 22, and a sensor unit 23.

[0038] The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. The lens system 11 guides light (incident light) from a subject and collects the light on the imaging element unit 12.

[0039] The imaging element unit 12 includes an image sensor 12a (imaging element) such as a complementary metal oxide semiconductor (CMOS) type or a charge coupled device (CCD) type. The image sensor unit 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc., on the electric signal obtained by photoelectrically converting the light received by the image sensor 12a, and further performs A / D (Analog / Digital) conversion processing. Then, the image signal as digital data is output to the downstream image processing unit 20 and the camera control unit 18.

[0040] The image processing unit 20 is configured as an image processor using, for example, a DSP (Digital Signal Processor). The image processing unit 20 performs various types of signal processing on the digital signal (captured image signal) from the image sensor unit 12, that is, the RAW image data. Note that RAW image data may refer to image data that has undergone some processing of the digital signal from the imaging element unit 12, such as the above-mentioned YC-RAW image data. Even if the format does not reach the YC-RAW image data format, RAW image data may be obtained by performing preprocessing such as clamping the R, G, and B black levels to a predetermined level and correction processing between the R, G, and B color channels on the captured image signal from the imaging element unit 12. RAW image data may also include image data that has undergone lens correction and noise reduction.

[0041] In this embodiment, the image processing unit 20 has a signal processing function as a synthesis processing unit 31 and a development processing unit 32 . As described later, the synthesis processing unit 31 performs synthesis processing for generating synthetic RAW image data by synthesizing a plurality of RAW image data, each of which constitutes an image of one frame. As will be described next, the image data developed by the development processing section 32 can be subjected to a synthesis process by the synthesis processing section 31.

[0042] The development processing unit 32 performs development processing on RAW image data and composite RAW image data generated by the synthesis processing unit 31 to generate image data in a predetermined format. For example, the development processing unit 32 performs lens correction, noise reduction, synchronization processing, YC generation processing, color reproduction / sharpness processing, etc.

[0043] In the synchronization process, a color separation process is performed so that the image data for each pixel has all the color components R, G, and B. For example, in the case of an image sensor that uses a Bayer array color filter, a demosaic process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, B image data. Color reproduction / sharpness processing involves adjusting gradation, saturation, tone, contrast, and other aspects of image creation.

[0044] These processes performed by the development processing unit 32 are development processes in the broad sense, but the color reproduction / sharpness processes in particular are called development processes in the narrow sense. Image data that has been subjected to development processes in the narrow sense loses part of the information of the original RAW image data, so the degree of freedom in subsequent image editing is limited. Furthermore, image data that has not been subjected to development processing in the narrow sense can be said to fall within the scope of RAW image data and YC-RAW image data as referred to in this disclosure.

[0045] In this manner, the image processing unit 20 performs development processing in a broad sense using the development processing unit 32 to generate image data in a predetermined format. In this case, resolution conversion and file creation processing may be performed. In the file creation processing, image data is subjected to, for example, compression encoding for recording or communication, formatting, generation and addition of metadata, and the like to create files for recording or communication. For example, still image files may be generated in formats such as JPEG, TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), HEIF, YUV422, YUV420, etc. It is also possible to generate image files in the MP4 format used for recording video and audio conforming to MPEG-4. Note that image files may also be generated based on RAW image data that has not been subjected to development processing.

[0046] The buffer memory 21 is formed of, for example, a dynamic random access memory (DRAM). The buffer memory 21 is used for temporary storage of image data in the image processing unit 20 during the above-mentioned synthesis process and development process.

[0047] The recording control unit 14 performs recording and reproduction on a recording medium, such as a non-volatile memory, etc. The recording control unit 14 performs processing for recording image files, such as moving image data and still image data, on the recording medium, for example. The actual form of the recording control unit 14 may be various. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and a write / read circuit for the memory. The recording control unit 14 may also be in the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be detached from the imaging device 1, such as a memory card (such as a portable flash memory). The recording control unit 14 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0048] The display unit 15 is a display unit that displays various information to the user, and is, for example, a display panel or viewfinder using a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display disposed on the housing of the imaging device 1. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18 . For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14. Further, image data of the captured image that has been resolution-converted for display by the image processing unit 20 is supplied to the display unit 15, and the display unit 15 may perform display based on the image data of the captured image in response to an instruction from the camera control unit 18. This causes a so-called through image (monitoring image of the subject), which is an image captured while checking the composition or recording a video, to be displayed. Further, based on instructions from the camera control unit 18, the display unit 15 executes display of various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface) on the screen.

[0049] The communication unit 16 performs data communication and network communication with external devices via wired or wireless communication, for example, transmitting and outputting still image files and video files including captured image data and metadata to external information processing devices, display devices, recording devices, playback devices, etc. The communication unit 16 also serves as a network communication unit, and can communicate via various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data between servers, terminals, and the like on the network. The imaging device 1 may also be capable of communicating information with, for example, a PC, a smartphone, a tablet terminal, etc., by means of the communication unit 16, for example, by short-range wireless communication such as Bluetooth (registered trademark), Wi-Fi (registered trademark), or NFC (Near field communication), infrared communication, etc. The imaging device 1 may also be capable of communicating with other devices by wired connection communication. Therefore, the imaging device 1 can transmit captured images and metadata via the communication unit 16 to an information processing device 70, which will be described later.

[0050] The operation unit 17 collectively refers to input devices for a user to perform various operational inputs. Specifically, the operation unit 17 refers to various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. An operation performed by the user is detected by the operation unit 17, and a signal corresponding to the input operation is sent to the camera control unit .

[0051] The camera control unit 18 is composed of a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 18. The illustrated memory unit 19 collectively represents, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), a flash memory, and the like. The memory unit 19 may be a memory area built into the microcomputer chip serving as the camera control unit 18, or may be configured by a separate memory chip. The camera control unit 18 executes a program stored in the ROM, flash memory, or the like of the memory unit 19 to control the entire imaging device 1 . For example, the camera control unit 18 controls the operation of each necessary unit with regard to controlling the shutter speed of the image sensor unit 12, instructing various signal processing in the image processing unit 20, imaging operations and recording operations in response to user operations, playback operations of recorded image files, operations of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, user interface operations, etc.

[0052] The camera control unit 18 realizes the function of the UI control unit 30 by, for example, software. The UI control unit 30 controls the display unit 15, an external display device connected to the imaging device 1, or an external display device capable of communicating with the imaging device to display a UI screen 100, and also performs processing to detect user operations on the UI screen 100. As will be described in detail later, the user can use the UI screen 100 to check images for synthesis processing and perform various setting operations. The UI control unit 30 (camera control unit 18) controls the compositing process by the compositing processing unit 31 based on the setting of the compositing process performed via the UI screen 100. The function of the image processing unit 20 described above may be realized by software in the camera control unit 18.

[0053] The RAM in the memory unit 19 is used as a working area for various data processing by the CPU of the camera control unit 18, for temporarily storing data, programs, and the like. The ROM and flash memory (non-volatile memory) in memory unit 19 are used to store the OS (Operating System) that the CPU uses to control each part, content files such as image files, application programs for various operations, firmware, various setting information, etc.

[0054] The driver unit 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 18, thereby moving the focus lens and zoom lens, opening and closing the aperture blades of the aperture mechanism, and so on.

[0055] The sensor unit 23 collectively represents various sensors mounted on the imaging device. When an IMU (inertial measurement unit), for example, is installed as the sensor unit 23, angular velocity can be detected using a three-axis angular velocity (gyro) sensor of pitch, yaw, and roll, and acceleration can be detected using an acceleration sensor. The sensor unit 23 may also include, for example, a position information sensor, an illuminance sensor, a distance measurement sensor, and the like. Various information detected by the sensor unit 23, such as position information, distance information, illuminance information, IMU data, etc., is added as metadata to the captured image together with date and time information managed by the camera control unit 18.

[0056] <3. Configuration of information processing device> Next, an example of the configuration of the information processing device 70 will be described with reference to FIG. The information processing device 70 is a device capable of information processing, particularly image processing, such as a computer device. Specific examples of the information processing device 70 include personal computers (PCs), mobile terminal devices such as smartphones and tablets, mobile phones, video editing devices, video playback devices, etc. The information processing device 70 may also be a computer device configured as a server device or a computing device in cloud computing. This information processing device 70 is equipped with a UI control unit 30 that performs UI control for the synthesis processing, and the information processing device 70 equipped with this UI control unit 30, or a processor (CPU 71) equipped with the UI control unit 30, can be considered as an example of a control device of the present disclosure.

[0057] A CPU 71 of an information processing device 70 executes various processes according to programs stored in a ROM 72 or a non-volatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes, as appropriate.

[0058] The image processing unit 20 has the functions of the synthesis processing unit 31 and the development processing unit 32 described in the imaging device 1 above. The synthesis processing section 31 and the development processing section 32 constituting the image processing section 20 may be provided as functions within the CPU 71. Furthermore, the image processing unit 20 may be realized by a CPU separate from the CPU 71, a graphics processing unit (GPU), general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, or the like.

[0059] The CPU 71 realizes the function of the UI control unit 30 by, for example, software. The UI control unit 30 controls the display unit 77, an external or remote display device capable of communicating with the information processing device 70, and the like to display a UI screen 100, and also performs processing to detect user operations on the UI screen 100. The user can use the UI screen 100 to check images for synthesis processing and perform various setting operations. The UI control unit 30 (CPU 71 ) controls the compositing process by the compositing processing unit 31 based on the setting of the compositing process performed via the UI screen 100 .

[0060] The CPU 71, ROM 72, RAM 73, non-volatile memory unit 74, and image processing unit 20 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0061] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, a remote controller, or the like. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0062] Further, a display unit 77 formed of an LCD or an organic EL panel, etc., and an audio output unit 78 formed of a speaker, etc. are connected to the input / output interface 75 either integrally or separately. The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like. The display unit 77 executes display of images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 performs display of various operation menus, icons, messages, etc., that is, GUI, based on instructions from the CPU 71.

[0063] The input / output interface 75 may be connected to a storage unit 79 configured with a HDD or solid-state memory, or a communication unit 80 configured with a modem or the like.

[0064] The storage unit 79 can store data to be processed and various programs. When the information processing device 70 functions as the control device of the present disclosure, it is expected that the memory unit 79 will store image data (e.g., RAW image data) that can be the subject of synthesis processing, or that it will store synthetic RAW image data, or JPEG image data that has been developed from synthetic RAW image data, etc. The storage unit 79 also stores programs for UI control processing, synthesis processing, development processing, and the like.

[0065] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like. Communication with the imaging device 1, particularly reception of captured images and the like, is performed by a communication unit 80.

[0066] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 81 allows data files such as image files and various computer programs to be read from the removable recording medium 82. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the sound output unit 78. Furthermore, the computer programs and the like read from the removable recording medium 82 are installed in the storage unit 79 as necessary.

[0067] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0068] <4. Compositing and development processing> Here, the synthesis process and development process of image data carried out by the image processing unit 20 in the imaging device 1 and the information processing device 70 will be described.

[0069] For example, the image processing unit 20 can perform processing for synthesizing a plurality of RAW image data to generate one or a plurality of pieces of synthesized RAW image data for the purpose of editing the shutter speed after capturing an image.

[0070] First, FIG. 4 shows a schematic flow of the development process executed by the development processor 32 in the image processor 20. As shown in FIG. For example, image data input from the image sensor unit 12 becomes RAW image data either by itself or after undergoing pre-processing (not shown). In FIG. 4, the image data input from the image sensor unit 12 is shown as RAW image data assuming the imaging device 1, but in the case of the information processing device 70, it may be considered as RAW image data read out from the memory unit 79, for example.

[0071] Development processing in the broad sense includes, for example, lens correction in step ST1, NR (noise reduction) in step ST2, demosaic in step ST3, and color reproduction / sharpness processing in step ST4, which is development processing in the narrow sense. Then, as image data after such development processing, for example, JPEG image data is generated. Furthermore, after demosaic, YC conversion is performed in step ST5 to obtain YC-RAW image data.

[0072] Examples of the relationship between the development processing carried out in the development processing unit 32 and the synthesis processing carried out in the synthesis processing unit 31 are shown in FIGS. 5A, 5B, 6A, 6B, and 6C. The synthesis process example I in Fig. 5A is an example of synthesizing JPEG image data. The figure shows a number of JPEG image data (J#1 to J#n) each of which has been generated through development processing (steps ST1, ST2, ST3, ST4) for RAW image data. The synthesis process is performed on these JPEG image data (J#1 to J#n) in step ST10 to generate synthesized JPEG image data.

[0073] The synthesis process example II in FIG. 5B is an example of synthesizing RAW image data. In the figure, multiple pieces of RAW image data (R#1 to R#n) are shown as image data to be subjected to the synthesis process. For example, the synthesis process is performed in step ST10 on these multiple pieces of RAW image data (R#1 to R#n). As a result, synthesized RAW image data is obtained. The composite RAW image data is then subjected to development processing in steps ST1, ST2, ST3, and ST4, thereby generating JPEG image data as a composite image.

[0074] Combining RAW image data as shown in FIG. 5B has the following advantages. For example, when multiple JPEG image data (J#1 to J#n) after development processing are synthesized as shown in FIG. 5A, the following can be pointed out. First, a broad-sense development process (steps ST1, ST2, ST3, and ST4) is required for each of the multiple image data used for compositing. For example, if each of the individual JPEG image data (J#1 to J#n) is also to be saved, this is fine, but if the only purpose is to create a composite image, the processing load will increase.

[0075] Furthermore, the development process is performed using signal processing optimized for each image data, which is not necessarily optimal for the combined image data. In addition, since the composite image is a post-development image, its editability is limited, particularly because some of the information that existed in the RAW image data is lost due to thinning of color information, etc., in the color reproduction / sharpness processing in step ST4.

[0076] On the other hand, if RAW image data is combined as shown in Fig. 5B, development processing needs to be performed only once after combination, reducing the signal processing load. In addition, the development processing can be performed using parameters optimized for the combined RAW image data after combination, which can improve the image quality of the generated developed image data, for example, the above-mentioned JPEG image data. In addition, since the composite RAW image data has not undergone image creation such as color reproduction / sharpness processing, there is no loss of information due to image creation, and therefore the image has high editability in terms of color reproduction, for example. In this way, by combining a plurality of pieces of RAW image data, it is possible to perform efficient signal processing that is optimized for the combined image.

[0077] Next, a compositing process example III in FIG. 6A is an example of compositing process using YC-RAW image data. 6A, 6B, and 6C, for the sake of explanation, RAW image data in R, G, B format obtained by the imaging element unit 12 is referred to as "RGB-RAW image data" to distinguish it from YC-RAW image data. Also, composite RAW image data made of RGB-RAW image data is referred to as "composite RGB-RAW image data." 6A shows a plurality of YC-RAW image data (YCR#1 to YCR#n), that is, image data that has been subjected to the processes of steps ST1, ST2, ST3, and ST5 for the RGB-RAW image data. The YC-RAW image data (YCR#1 to YCR#n) are subjected to the synthesis process of step ST10 to generate synthetic YC-RAW image data. The synthetic YC-RAW image data is also subjected to color reproduction / sharpness process of step ST4 to generate, for example, JPEG image data as a synthetic image.

[0078] The compositing process example IV in FIG. 6B is an example in which the compositing process of step ST10 is performed on multiple RGB-RAW image data (R#1 to R#n) to generate composite RGB-RAW image data, and the processes of steps ST1, ST2, ST3, and ST5 are performed to generate composite YC-RAW image data.

[0079] 6C is an example in which multiple YC-RAW image data are restored to RGB-RAW image data (R#1 to R#n) and then synthesized. The multiple RGB-RAW image data (R#1 to R#n) restored from the multiple YC-RAW image data are subjected to the synthesis process of step ST10 to generate synthetic RGB-RAW image data, and the processes of steps ST1, ST2, ST3, and ST4 are performed to generate JPEG image data as a synthetic image.

[0080] As in the above examples, there may be cases where the YC-RAW image data is used for the synthesis process. As the YC-RAW image data is in a state before color reproduction / sharpness processing is performed, the information at the RAW image data stage is not lost, and therefore the same advantages as in the case of synthesizing the RAW image data in Figures 5A and 5B described above can be obtained.

[0081] 5A, 5B, 6A, 6B, and 6C, a UI process is performed in step ST15 for the compositing process in step ST10. This UI process is performed by the UI control unit 30, and includes display control of the UI screen 100, detection of a user operation using the UI screen 100, and settings related to the compositing process based on the same. The UI process allows the user to set the desired settings for each instance of image composition.

[0082] Here, the concept of shutter speed editing realized by the above-described composition process will be explained.

[0083] Consider the case where the image data to be synthesized in each of the above examples, such as multiple RGB RAW image data (R#1 to R#n), multiple JPEG image data (J#1 to J#n), multiple YC RAW image data (YCR#1 to YCR#n), etc., is image data as a group of images recorded consecutively in time.

[0084] This allows the generation of a composite image with a shutter speed equal to the shutter speed at which one piece (one frame) of image data was captured multiplied by the number of composite images. For example, if the shutter speed is 1 / 100 seconds and 100 RAW image data are continuously shot and combined, a composite image with a shutter speed of 1 second can be obtained. With this concept, you can edit the shutter speed after capturing an image.

[0085] Note that multiple pieces of image data that are consecutive in time are not necessarily captured without any gaps in the exposure time. For example, the imaging operation by the image sensor 12a includes an exposure period and a readout period of the exposed electric charge, so that even in each frame of a moving image capture, the exposure period is not strictly consecutive. Therefore, it cannot be said to be editing of shutter speed in the strict sense, but since the compositing process in the sense of editing of shutter speed does not necessarily have to be in the strict sense, it is possible to edit shutter speed in a pseudo sense by compositing multiple RAW image data with interrupted exposure periods in this way.

[0086] However, it is possible to get closer to shutter speed editing in the strict sense, or even to achieve shutter speed editing in the strict sense.

[0087] As a method to approach shutter speed editing in the strict sense, additional processing can be considered. This additional processing is processing to make the exposure periods of the multiple RAW image data continuous in time, that is, to make the exposure periods continuous. Specifically, this is processing to add an exposure amount corresponding to the non-exposure period of the image sensor 12a to the exposure amount in the exposure period of the image sensor 12a for each of the multiple RAW image data.

[0088] This will be explained with reference to Fig. 7. Fig. 7 shows an exposure period Ta and a readout period Tb of the charge due to exposure as the imaging operation of the image sensor 12a. Note that, in reality, there are periods in which neither exposure nor readout is performed, as well as reset periods, but to simplify the illustration and explanation, these are included in the readout period Tb.

[0089] One frame of image data is obtained by the operation of the exposure period Ta and the readout period Tb, and when continuous shooting or video shooting is performed, the exposure period Ta and the readout period Tb are repeated. However, during the readout period Tb, no exposure is performed on the image sensor 12a. In this case, even if multiple pieces of RAW image data are combined as is, the result is not exactly the same as if the shutter speed was simply slowed down, i.e., the image was exposed for a long time.

[0090] Therefore, by performing a process of adding the exposure amount as pixel information in the readout period to the pixel value, image data without interruption in the exposure period Ta is generated. As an example, the pixel values ​​of each frame are calculated as if exposure had been performed during the readout period Tb by performing an interpolation operation while taking into consideration the ratio of the length of the exposure period Ta to the readout period Tb and the amount of change from the pixel values ​​of the previous and next frames before additional processing. Then, additional processing is performed on each pixel value in this way to obtain RAW image data R#1, R#2, R#3, and so on. As a result, the RAW image data R#1, R#2, R#3, . . . can be a plurality of RAW image data without interruption of the exposure period Ta. The JPEG image data and YC-RAW image data generated from such RAW image data R#1, R#2, R#3, . . . can also be made into continuous image data without any interruption in the exposure period Ta.

[0091] In order to obtain image data that can be subjected to shutter speed editing in the strict sense, it is considered that a plurality of image sensors 12a1, 12a2 are provided in the imaging element unit 12. In that case, a plurality of RAW image data can be obtained without interruption of the exposure period Ta without performing the above-mentioned additional processing.

[0092] 8, for example, RAW image data R#1 is obtained from the output of the image sensor 12a1, and then RAW image data R#2 is obtained from the output of the image sensor 12a2. At this time, image sensors 12a1 and 12a2 are operated to be synchronized with a difference in exposure timing. That is, as shown in the figure, the exposure period Ta of image sensor 12a2 starts at the end timing of the exposure period Ta of image sensor 12a1, and the exposure period Ta of image sensor 12a1 starts at the end timing of the exposure period Ta of image sensor 12a2. In this way, the RAW image data R#1, R#2, R#3, R#4, etc. alternately obtained from the image sensors 12a1, 12a2 become image data with continuous exposure periods. The JPEG image data and YC-RAW image data generated from such RAW image data R#1, R#2, R#3, etc. can also be continuous image data with no break in the exposure period Ta.

[0093] Furthermore, even when one image sensor 12a is used, in cases where the image sensor 12a is one in which the interruption of the exposure period due to the readout period Tb can be almost ignored, or where the final image to be obtained does not include a moving subject, etc., it is possible to realize shutter speed editing that is sufficient for practical use by the composition process without performing the above-mentioned additional process. In other words, the additional process may be performed in the case of an imaging device or imaging scene in which the impact of the interruption of the exposure period is significant.

[0094] <5.UI processing example> An example of UI processing will be described below. The UI processing example of the embodiment provides the user with excellent operability for image synthesis and editing.

[0095] An example of the screen configuration of the UI screen 100 is shown in FIG. The UI screen 100 has, for example, an input selection area 2, an adjustment area 3, a leading image area 4, an ending image area 5, and a composite image area 6. Note that this is just one example, and other areas may be provided, or some areas may not be provided. Each area is displayed simultaneously as one screen, for example. It is particularly desirable to display each area without scrolling the screen or turning pages.

[0096] The input selection area 2 is an area for selecting an editing target, that is, for selecting a group of images to be subjected to synthesis processing as image editing. The adjustment area 3 is an area for setting up a fixed area pool, specifying a fixed area, and setting development settings, which will be described later. The leading image area 4 is an area for displaying an image to be used as the leading image in the synthesis process, selecting the leading image, specifying a fixed area, and so on. The end image area 5 is an area for displaying an image to be the end image of the synthesis process, selecting the end image, specifying a fixed area, and so on. The composite image area 6 is an area for selecting a composite mode (composite processing method), setting development after composite, displaying the composite and developed image, setting output, and performing output operations.

[0097] A user interface using such a UI screen 100 has the following features (a) to (f).

[0098] (a) It has an input interface for a group of images as multiple images that are consecutive in time. For example, by displaying or operating the input selection area 2, the user can select an image group of RAW image data or an image group of JPEG image data, and set them as the target of the setting operation on the UI screen 100. Furthermore, the image group is not limited to a group of a plurality of images packed together by continuous shooting or the like at the time of image capture, but may be a group of images selected by the user one by one. For this purpose, the input selection area 2 displays selectable image groups or individual images for the user to select.

[0099] (b) The UI screen has a display frame for the image to be processed. For example, to show the synthesis range, two or more image display areas are provided so that the leading image and the trailing image can be compared and checked. The leading image area 4 and the trailing image area 5 correspond to these display areas. In addition, RAW images may be selected as images to be displayed in the leading image area 4 and the trailing image area 5, but the visibility of the image is poor if the RAW image data is displayed as is. Therefore, when the image data for image display is RAW image data, it is desirable to perform processing to display the image in an easily visible manner. For example, RAW image data is automatically developed using development settings that make it easy for the user to edit, and then displayed. Alternatively, the developed image data associated with the RAW image data is displayed. For example, if the developed JPEG image data or thumbnail image data is recorded together with the RAW image data when capturing an image, even if the RAW image data is selected as the object to be edited, the JPEG image data or thumbnail image data corresponding to the RAW image data is used for image display in the first image area 4 or the last image area 5.

[0100] (c) Composite editing is possible. The leading image can be selected or changed in the leading image area 4. The trailing image can be selected or changed in the trailing image area 5. These are equivalent to editing the synthesis range. In addition, it is possible to specify a fixed area on the first or last image. A fixed area is an area whose position within a frame is fixed between each image for multiple image data to be synthesized. For example, it is specified as an area of ​​a specific subject. In addition, it is possible to select a synthesis mode on the UI screen 100, which indicates the synthesis method to be used for synthesis processing.

[0101] (d) A display frame for the synthesis results. A composite image area 6 is provided to allow the user to check what kind of composite result will be obtained with the current composite settings.

[0102] (e) Development settings can be set for composite images. For example, development settings can be made through a user interface using the composite image area 6. The setting results may be displayed in the composite image area 6 or in a separate frame.

[0103] (f) It is possible to select the output format of the synthesis results and to instruct the output of the synthesis results. For example, it is possible to select whether to output the composite result as RAW image data or as an image developed according to the development settings.

[0104] A specific example of a UI screen 100 having these characteristics is shown in Fig. 10. As shown in the example, on the UI screen 100, at least a leading image 50 and a trailing image 60 are simultaneously displayed, and various displays for image synthesis are also performed.

[0105] 10, a menu section 40 is provided at the top left of the screen, with controls such as "File," "Edit," "Output," "Auto," and "Help" being provided. This allows the selection and saving of image files, various editing operations, output settings, automatic processing settings, help display, and the like to be performed from the menu section 40.

[0106] In the input selection area 2, an import button 41 and a list section 42 are provided. The import button 41 is an operator for loading materials (image group) to be edited and adding them to the list section . The imported image groups are displayed in a list in the list section 42. Each image group is displayed, for example, by a thumbnail image of a representative image, or may be displayed by the file name of the image group.

[0107] In the adjustment area 3, an add button 43, a fixed area type 44, a fixed area name display section 45, and a fixed area designation section 46 are provided. The add button 43 is an operator for adding a selected fixed region to the region pool, which is a storage area for storing fixed regions set for a group of images to be processed.

[0108] The fixed regions stored in the region pool are then listed in adjustment area 3. The fixed area type 44 displays either “start” or “end” to indicate whether the pooled fixed area is specified in the leading image 50 or the trailing image 60. The fixed area name display section 45 displays the name of the pooled fixed area. In the fixed area designation section 46, the fixed areas to be used in the synthesis process can be designated by check boxes from among the pooled fixed areas. One or more fixed areas can be designated as those to be used.

[0109] The first image area 4 has a fixed area setting image display section 47, a fixed area designation mode operation section 48, an AI support mode display section 49, a first image 50, a shutter speed display section 51, a bar / playback control display section 52, a frame number display section 53, and an edit button 54 arranged therein.

[0110] In the fixed area setting image display section 47, "start" or "end" is displayed to indicate whether the fixed area is set in the first image 50 or the last image 60. In AI support mode display section 49, a check box can be used to specify whether the AI ​​support mode is enabled or disabled when a fixed region is selected. The AI ​​support mode will be described in the second embodiment. The shutter speed display section 51 displays the converted shutter speed when a composition process is performed on the current composition range, that is, the range from the first image 50 to the last image 60. The fixed region setting image display section 47 , the AI ​​support mode display section 49 , and the shutter speed display section 51 may be displayed in a location other than the top image area 4 .

[0111] The fixed area designation mode operation unit 48 makes it possible to switch the fixed area designation mode in the leading image area 4. As the fixed area designation mode, for example, a rectangle designation for selecting a rectangular range and a segment designation for selecting a subject area can be selected by the user. Of course, other selectable fixed area designation modes may be provided.

[0112] The leading image 50 is an image that is set at the beginning of the synthesis range. Since the leading image 50 can be changed by a user operation or the like, the current leading image 50 is displayed in the leading image area 4.

[0113] Below the first image 50, a bar / playback operator display section 52 is displayed, which includes a slide bar and a playback operator. The slide bar represents the time axis of the group of images to be processed, and the user can select the top image 50 by performing an operation such as moving the pointer on the slide bar. The playback controls include playback, stop, frame forward, and frame back operations. These operations change the displayed top image 50. The user can select the top image 50 by operating these playback controls and moving forward or backward through the image group.

[0114] The frame number of the image currently set as the leading image 50, for example the frame number within an image group, is displayed as the frame number display section 53. For example, if the image group includes 30 images, one of the frame numbers "1" to "30" is displayed. Also, the user can input a number into the frame number display section 53 to select the top image 50 by frame number.

[0115] The edit button 54 is an operator that is operated when editing in the lead image area 4. That is, when the user wishes to select the lead image 50 or to specify a fixed area on the lead image 50, the user operates the edit button 54. This enables editing using the lead image area 4.

[0116] In the end image area 5, a fixed area designation mode display section 55, an end image 60, a bar / playback operator display section 52, a frame number display section 58, and an edit button 57 are arranged.

[0117] The fixed area designation mode operation unit 48 makes it possible to switch the fixed area designation mode in the terminal image area 5. As in the case of the head image area 4 described above, the user can select, for example, rectangle designation or segment designation.

[0118] The end image 60 is an image that is set at the end of the synthesis range. Since the end image 60 can be changed by a user operation or the like, the current end image 60 is displayed in the end image area 5.

[0119] At the bottom of the end image 60, a slide bar and a playback operator are displayed as the bar / playback operator display section 56. The slide bar represents the time axis of the group of images to be processed, and the user can select the last image 60 by performing an operation such as moving the pointer on the slide bar. The playback controls include playback, stop, frame forward, and frame back operations. These operations change the displayed end image 60. The user can select the end image 60 by operating these playback controls and moving forward or backward through the image group.

[0120] The frame number of the image currently set as the terminal image 60, for example the frame number within an image group, is displayed as the frame number display section 58. The user can also select the terminal image 60 by frame number by inputting a number in the box of the frame number display section 58.

[0121] The edit button 57 is an operator that is operated when editing in the terminal image area 5. That is, when the user wants to select the terminal image 60 or specify a fixed area on the terminal image 60, the edit button 57 is operated. This makes it possible to edit using the terminal image area 5.

[0122] The composite image area 6 is provided with a composite mode setting section 61, a composite execution button 62, a save button 63, a composite progress display section 64, a composite image 65, and a history display section 66.

[0123] A blending mode setting section 61 is provided to allow the user to select a blending mode, for example, an averaging mode, a comparatively bright mode, a weighting mode, etc., can be selected in the form of a pull-down menu, etc. The averaging mode is a synthesis processing mode in which corresponding pixel values ​​in a plurality of pieces of image data within a synthesis range are averaged to obtain pixel values ​​of a synthesis image. The relatively bright mode is a mode in which only pixel values ​​having a luminance value equal to or greater than a predetermined value are used as valid for synthesis. The weighting mode is a synthesis processing mode in which the weighting is changed for some of the image data in the synthesis range and the image is synthesized. The comparatively bright mode and the weighting mode will be described later.

[0124] The synthesis execution button 62 is an operator for executing synthesis processing. The save button 63 is an operator for instructing the saving of the composite image data. Note that the format of the image data of the composite image to be saved may be made selectable.

[0125] As the composite image 65, an image obtained by a composite process of a plurality of image data from the first image 50 to the last image 60 set at that time is displayed. This composite image may be a composite of a plurality of images from the first image 50 to the last image 60 using a specific composite processing setting, or may be an image obtained by a composite processing that reflects the composite mode setting at that time. Furthermore, if a development setting is set at that time, an image that has already been developed may be displayed.

[0126] A synthesis progress display section 64 is provided below the synthesis image 65, and the progress of the synthesis process is expressed by, for example, the length of a bar. Furthermore, the history display section 66 displays the history of synthesis processing and storage.

[0127] Note that the above is a representative example of the contents of the UI screen 100, and it is assumed that other displays and controls are included. For example, in the example of FIG. 10, the user interface necessary for development settings is not shown, but it is assumed that such an area will be prepared in the adjustment area 3, the composite image area 6, or the like.

[0128] The composition process setting by the user using such a UI screen 100 will now be described. The user performs an operation to select any image group from the imported image group displayed in the list section 42. Then, the image group is set as a target for the composition process setting, and the display of the image group begins.

[0129] That is, the adjustment area 3 displays the fixed regions that have been pooled in the past for that image group. As the leading image area 4 and the trailing image area 5, a leading image 50 and a trailing image 60 set as defaults are displayed. For example, a default setting could be to make the first image 50 and the last image both the first frame image in the image group. Alternatively, as a default setting, the leading image 50 may be the image of the first frame of the image group, and the trailing image 60 may be the image of the second frame of the image group. Alternatively, as a default setting, the leading image 50 may be the first frame image of the image group, and the trailing image 60 may be the last frame image of the image group. Furthermore, as a default setting, the first image 50 and the last image 60 set for the image group during a past editing process for the image group may be displayed. Also, a process may be performed in which the main subject within the time range is determined by image analysis of each image in the image group, and the first frame in which the main subject is captured is designated as the leading image 50, and the last frame in which the main subject is captured is designated as the ending image 60.

[0130] Thereafter, the user can arbitrarily select a leading image 50 and a trailing image 60 using the leading image area 4 and the trailing image area 5 . For example, the first image 50 or the last image 60 is selected using the slide bar or playback operator of the bar / playback operator display sections 52 and 56 .

[0131] In addition, the user may input a shutter speed value into the shutter speed display section 51, so that a synthesis range according to the input shutter speed is set based on either the first image 50 or the last image 60. Alternatively, the synthesis range may be set according to the input shutter speed, with the central frame in the range from the leading image 50 to the trailing image 60 at that time as the reference.

[0132] Moreover, by pressing a synthesis execution button 62, synthesis processing can be executed for the synthesis range selected at that time from the leading image 50 to the trailing image 60, and the synthesized image 65 can be confirmed. Note that even if the user does not operate the synthesis execution button 62, the synthesis process may be performed every time either the leading image 50 or the trailing image 60 is changed, and the synthesized image 65 may be updated.

[0133] The shutter speed display unit 51 displays the shutter speed when a compositing process is performed on the compositing range from the first image 50 to the last image 60, allowing the user to work while understanding what shutter speed the image will have for the current compositing range. In addition, the user can select the frames to be the leading image 50 and the trailing image 60 while checking the image contents, making it easy to select the range to be combined. In addition, since the user can work while simultaneously viewing the first image 50 through the last image 60 on one screen, the user can easily grasp the synthesis range. Since the composite image 65 can be viewed at the same time, the user can select the first image 50 or the last image 60 while getting an idea of ​​what the composite image will look like.

[0134] The user can also specify a fixed region for the compositing process. FIG. 11 shows a state in which a fixed area frame 59 indicating the fixed area is displayed. Using the adjustment area 3, the user can specify a pooled fixed area, as well as a new fixed area on the image.

[0135] For example, the user can select either the first image 50 or the last image 60 and specify a fixed area on that image. For example, the user can drag the fixed area frame 59 to enlarge / reduce it or move it to position the fixed area frame 59 within the range of the desired subject. As a result, the inside of that frame becomes the fixed area.

[0136] Alternatively, the user may perform an operation to specify the range freehand, so that an indefinite fixed area frame 59 is drawn on the image, thereby specifying the fixed area. Also, when the user specifies a number of points on the image, a rectangular range including those points may be set as a fixed area. In addition, when a user specifies a certain subject, the range of the subject, for example, the range of a face, a body, a specific object, etc., may be automatically determined and the area may be set as a fixed area. In this case, in order to determine the subject range, it is considered to apply object recognition processing such as semantic segmentation.

[0137] When a fixed area has been set, the fixed area can be pooled by operating the add button 43. By pooling the settings of a newly specified fixed area, the settings of the fixed area can be used thereafter.

[0138] The designated fixed area is displayed in a fixed area frame 59 as shown in FIG. When a fixed area frame 59 is displayed on the leading image 50, in all images in the synthesis range, synthesis is performed with the in-frame position of the subject image corresponding to that fixed area aligned with the in-frame position of the subject in the leading image 50. Therefore, the object in the fixed area is not blurred in the composite image 65 .

[0139] For example, in Fig. 11, it is assumed that a group of images was taken by a user taking continuous shots of a moving train. For example, the user takes continuous shots while swinging the imaging device 1 from left to right in time with the movement of the train. Therefore, the imaging direction of each image in the group of images changes slightly. In this case, by setting the front part of the train as a fixed region, during the synthesis process, the position within the frame of the front part of the train for each image data is aligned with the position within the frame of the front part of the train in the leading image 50 and synthesized. Therefore, as in the illustrated synthesized image 65, the front part of the train becomes a clear image and the background becomes a blurred image.

[0140] The fixed region may be specified in the terminal image 60 . In this case, all image data in the synthesis range is synthesized in a state in which the position within the frame of the subject that is set as a fixed region is aligned with the position within the frame of that subject in the terminal image 60 . Therefore, even if the subject area is the same, the composite image will be different if the first image 50 is set as a fixed area and if the last image 60 is set as a fixed area.

[0141] When a plurality of fixed regions are set, the synthesis process is performed with the plurality of fixed regions being aligned within the frame in all images in the synthesis range.

[0142] It is also possible not to specify a fixed area. If you want to turn off the fixed area setting after validating the pooled fixed area, you can do so by unchecking the fixed area specification section 46. Furthermore, after the user sets the fixed area frame 59 on the image, the user can perform an operation to erase the fixed area frame 59, thereby canceling the designation of the fixed area.

[0143] <6. Example of UI Control in First Embodiment> An example of the processing performed by the UI control unit 30 while displaying the above-described UI screen 100 will be described with reference to FIG. FIG. 12 shows an example of a control process in a state where the UI control unit 30 has caused the UI screen 100 to be displayed on the display unit 15 (or the display unit 77, or another display device).

[0144] In step S101, the UI control unit 30 determines whether or not the UI processing is to be ended. If the user performs an operation such as closing the UI screen 100 and it is determined that the UI processing is to be ended, the UI control unit 30 proceeds from step S101 to step S140, performs an ending process to close the UI screen 100, and ends the process of FIG.

[0145] After the UI screen 100 is opened, the UI control unit 30 monitors the image group selection operation in step S102. Although not shown in the flowchart, the UI screen 100 also sequentially performs processes corresponding to operations of the menu unit 40 and the import button 41.

[0146] When a selection operation is performed on the image group displayed in the input selection area 2, the UI control unit 30 proceeds from step S102 to step S103 and performs image area display processing. This is processing to display the default leading image 50 in the leading image area 4 and the default trailing image 60 in the trailing image area 5, and also to display the fixed area pooled for the image group in the adjustment area 3. The UI control unit 30 may set the default first image 50 and last image 60 to specific frames of the selected image group as described above, or may set frames determined by image analysis, or may set frames reflecting settings from the previous processing.

[0147] While the first image 50 and the last image 60 are displayed, the UI control unit 30 repeatedly monitors the operation of the synthesis range in step S120, the synthesis mode operation in step S122, the fixed area designation operation in step S124, the synthesis operation in step S126, and the output operation in step S128. In addition, the UI control unit 30 monitors various operations and processes corresponding thereto, but this is omitted to avoid complicating the explanation.

[0148] When a user operation on the synthesis range is detected, the UI control unit 30 proceeds from step S120 to step S121, and changes the setting of the synthesis range and changes the display to reflect the change. For example, this is an operation to change the leading image 50 by the bar / playback operator display sections 52 and 56, or an operation to change the trailing image 60.

[0149] When the user performs an operation to change the first image 50 using the bar / playback operator display section 52, the UI control section 30 changes the setting of the first image 50 and switches the displayed first image 50. In addition, the UI control section 30 changes the values ​​in the frame number display section 53 and the shutter speed display section 51 accordingly. When the user performs an operation to change the end image 60 using the bar / playback operator display unit 56, the UI control unit 30 changes the setting of the end image 60 and switches the displayed end image 60. In addition, the UI control unit 30 changes the values ​​in the frame number display unit 58 and the shutter speed display unit 51 accordingly.

[0150] There may be a case where the user performs an operation of inputting a shutter speed value in the shutter speed display section 51. Alternatively, although not shown in Fig. 10, the user may be allowed to specify the number of images to be combined. These operations also constitute operations on the combination range.

[0151] When the user inputs an arbitrary shutter speed value to the shutter speed display unit 51, the UI control unit 30 calculates a synthesis range for realizing that shutter speed from the leading image 50 at that time, for example, and sets the frame that is the end of the synthesis range to the leading image 60. Alternatively, the UI control unit 30 calculates a synthesis range for realizing that shutter speed based on the leading image 60 at that time, and sets the frame that is the start of the synthesis range to the leading image 50. The user may select whether the first image 50 or the last image 60 is to be used as the reference, or, for example, the image for which a fixed area is set at that point in time may be used as the reference.

[0152] In addition, the synthesis range for achieving the input shutter speed may be calculated using a frame that is temporally intermediate in the section from the first image 50 to the last image 60 at that time as a reference, and the first and last frames of the synthesis range may be set to the first image 50 and the last image 60. Also, the user may specify a reference frame within the synthesis range. The above method can also be used when allowing the user to input the number of images to be combined.

[0153] When the UI control unit 30 detects an operation of the composition mode setting unit 61 by the user, the UI control unit 30 proceeds from step S122 to step S123, and changes the composition mode setting and changes the display to reflect the change. For example, the composition mode setting is changed to an additive average mode, a comparatively bright mode, a weighting mode, etc. Also, the display state of the composition mode setting unit 61 is changed to indicate the mode.

[0154] When a user's operation to specify a fixed area is detected, the UI control unit 30 proceeds from step S124 to step S125, where it sets a fixed area and updates the display accordingly. For example, the display, deformation, and position change of the fixed area frame 59 are performed in response to a user operation. When an operation is performed in the adjustment area 3, the UI control unit 30 also performs fixed area setting in response to the operation, displays the fixed area frame 59, and the like in step S125.

[0155] When it is detected that the user has operated the merge execution button 62, the UI control unit 30 advances from step S126 to step S127, and performs a merge process and updates the display accordingly. For example, the UI control unit 30 causes the image processing unit 20 to execute a compositing process for the compositing range at that time. It is also possible to instruct a compositing process that reflects the compositing mode and fixed area at that time. It is also possible to instruct a development process that reflects the development settings at that time. Then, a composite image 65 obtained as a result of the composition process is displayed in the composite image area 6.

[0156] When the user's operation of the save button 63 is detected, the UI control unit 30 proceeds from step S128 to step S129, and performs a recording process of the composite image at that time point. That is, the composite image data at that time point is output to the recording control unit 14 or the storage unit 79, and is stored in a recording medium. In this case, the image data format may be specified, and the image data may be recorded as RAW image data or may be recorded after development.

[0157] By the above-described process of FIG. 12, the user can carry out editing operations relating to image synthesis while viewing the UI screen 100.

[0158] <7. Example of UI Control in the Second Embodiment> As a second embodiment, an example in which the AI ​​support mode is applied to fixed region setting will be described. The AI ​​support mode is a mode that assists the user in specifying a fixed region, and presents candidates for fixed regions and their suitability as fixed regions.

[0159] For example, when the user checks a check box in the AI ​​support mode display section 49 on the UI screen 100, the AI ​​support mode is enabled. In the AI ​​support mode, the degree of suitability for compositing is determined by AI processing depending on the compositing mode selected by the compositing mode setting unit 61, for example, the additive average mode, the comparatively bright mode, the weighting mode, etc., and the fixed area candidates and the suitability of the fixed area are presented according to the determination result. The suitability is an evaluation value indicating the degree to which the composite image will be a good image when the area is used as the fixed area.

[0160] One of the evaluation criteria for suitability is the degree to which images are displaced within the synthesis range when a fixed region is set. In addition, it is also possible to determine whether a desired image will be produced according to the image content (subject type), the composition range, and the composition mode, based on learning data of desirable and undesirable composite images, and to obtain the degree of suitability. For example, one of the evaluation criteria is whether a composite image will be generally popular.

[0161] Fig. 13 shows an example of processing by the UI control unit 30. This is the same as the processing in Fig. 12 except that steps S130 to S133 have been added, and the rest of the processing is the same as in Fig. 12, so a description thereof will be omitted.

[0162] If the AI ​​support mode is enabled, UI control unit 30 proceeds from step S130 to step S131, and branches the process depending on whether or not a fixed region has been specified at that point.

[0163] If a fixed area has already been specified, the UI control unit 30 proceeds to step S132, where it calculates the suitability of the fixed area and executes a display to present it. That is, the UI control unit 30 calculates a value indicating the degree to which a composite image will be desirable when each image in the composite range from the current top image 50 to the last image 60 is aligned and composited in the specified fixed area in the composite process of the currently selected composite mode. The UI control unit 30 then performs a display according to the suitability, for example, a numerical display, or a control of the display mode according to the suitability.

[0164] If a fixed region is not specified, the UI control unit 30 proceeds from step S131 to step S133, and searches for candidate fixed region areas in the leading image 50 and the trailing image 60. The UI control unit 30 then performs display control to display the candidate fixed region and to present the suitability of the region as the fixed region. For example, the UI control unit 30 performs object recognition on the first image 50 and the last image 60 using semantic segmentation processing or the like. Then, for each recognized object, the suitability of the object to be synthesized in the current synthesis mode and synthesis range, assuming that the object is to be made into a fixed region, is calculated as described above. Then, the subject region with a suitability equal to or greater than a threshold value is set as a candidate for the fixed region. After the UI control unit 30 has selected candidates for the fixed area in this manner, it presents the candidate area on the leading image 50 or the trailing image 60, and also performs display according to its suitability.

[0165] By performing the AI ​​support mode processing as described above, the user can specify a fixed region while checking fixed region candidates. After the fixed region is specified, the user can instruct the execution of synthesis or re-specify the fixed region based on the suitability of the specified region.

[0166] Here are some specific examples. FIG. 14 shows an example in which candidates for fixed regions are presented before a fixed region is specified. For example, when the UI control unit 30 finds a candidate for a fixed area in the terminal image 60, as shown in the figure, it displays a candidate frame 59A and presents the area selected as the candidate for the fixed area to the user. In this case, two candidate areas are displayed by two candidate frames 59A. At the same time, a numerical value is displayed as a suitability indicator 67. This allows the user to understand that the train in the foreground is more suitable as a fixed area.

[0167] In this example, the candidate frame 59A is shown in the end image 60, but it may also be shown on the leading image 50 side. Also, the candidate frame 59A may also be shown on both the leading image 50 and the trailing image 60. Areas of different subjects may be set as candidates, and even if it is the same subject, the suitability may differ between when it is specified on the leading image 50 side and when it is specified on the trailing image 60 side, so it is preferable to display each. For example, when the front of the train on the foreground side as shown in the figure is set as a fixed area, the area specified on the trailing image 60 side (i.e., fixed at a position within the frame on the trailing image 60 side) may be more desirable for the composition of the composite image.

[0168] 15 shows an example of a display mode of suitability. In this case, an example is shown in which images with different gradations are superimposed as suitability display 67 instead of using numerical values. For example, the suitability is calculated for each object region and converted into a luminance gradation.Then, the luminance gradation is superimposed. 15, in the lead image 50, the front part of the train in front has a high brightness, while the area of ​​the train in the rear has a relatively low brightness, and the other parts are darker.

[0169] This allows the user to intuitively grasp the suitability of each part of the subject in the image. In addition, since the suitability is presented for the entire image, it is also suitable for cases where you want to fix any area, not just the subject detected by object recognition. In the case of the example of FIG. 15, since the suitability is shown on the entire screen, it can be said that the candidate frame 59A does not need to be displayed.

[0170] The above examples of FIGS. 14 and 15 are examples of display control executed by the UI control unit 30 in the process of step S133 in FIG. As an example of display control performed in step S132 after the fixed area is specified, as shown in Fig. 16A, a fixed area frame 59 and an aptitude display 67 are displayed. In this example, two fixed areas are shown by two fixed area frames 59, and the respective aptitude displays 67 are performed by numerical values.

[0171] The suitability display 67 of this specified fixed area is changed in response to a subsequent change in the synthesis range or synthesis mode.

[0172] For example, suppose that the user changes the designation of the end image 60 from the state of Fig. 16A to that of Fig. 16B. In Fig. 16A, the frame number display section 58 is "12", but in Fig. 16B, the user has changed the end image 60 to an image at a later time, so the frame number display section 58 is now "29". In the case of Fig. 16B, the composition range is also wider, so the shutter speed display section 51 is also updated from "1 / 48" in Fig. 16A to "9 / 160". When the user performs an operation to change the synthesis range in this way, the suitability of the fixed region is also recalculated, so that the value of the suitability display 67 changes as shown in the figure.

[0173] In the processing of FIG. 13, while the AI ​​support mode is active, processing of step S132 or step S133 is always performed. Therefore, after the user specifies a fixed area, the UI control unit 30 will recalculate the suitability in step S132 and update the display each time the synthesis range or synthesis mode is changed.

[0174] As described above, the AI ​​support mode process involves presenting candidates for fixed areas using candidate frame 59A and displaying suitability 67 for the candidate areas and fixed areas. In particular, when a user captures an image while moving imaging device 1 as if following a subject, that is, when performing so-called panning, the selection of a fixed area becomes important. This is because the imaging direction of each image within the synthesis range changes. For this reason, it is useful to be able to assist in the designation of a fixed area in the AI ​​support mode.

[0175] It should be noted that if the image group that is the subject of the composition process is a panning shot, the AI ​​support mode may be automatically activated. For example, when an analysis of each image in the image group shows that the position of a specific subject within the frame is gradually moving, the UI control unit 30 may determine that a panning shot has been taken, and may activate the AI ​​support mode. Furthermore, if the metadata added to the image data includes information about panning, the UI control unit 30 can determine that the images are panning shots from the metadata. For example, the imaging device 1 may determine panning shots from the detection value of the IMU when capturing images, and add the information to the metadata. Alternatively, the user may perform a predetermined operation when performing panning shots, and the information about panning shots may be added to the metadata. Furthermore, if IMU data is recorded in the metadata for each image (frame), the UI control unit 30 can analyze the IMU data during the synthesis process to determine whether the images are the result of panning.

[0176] In addition, the image stabilization effect can be obtained not only for panning but also for the area specified as a fixed area. In that sense, it is also effective to execute the AI ​​support mode not only for panning but also for other purposes.

[0177] <8. Examples of blending modes> In the above explanation, the averaging mode, the comparatively bright mode, and the weighting mode have been given as examples of the types of blending modes. The averaging mode is used to average the pixel values ​​of each image, but here we will explain the comparatively bright mode and the weighting mode.

[0178] The relatively bright mode is a mode in which only pixel values ​​having a luminance value equal to or greater than a predetermined value are used for compositing, and can be said to be a compositing mode when it is desired to make the most of a moving bright subject. For example, in the case of a subject that has a momentary high brightness in a dark background, such as a night view, starry sky, or fireworks, if the image is synthesized in additive average mode, the momentary high brightness part will be dark in the synthesized image. This is because the brightness value of the corresponding pixel will be lowered by averaging because the part is dark in the other images.

[0179] Consider the images shown in Figure 17. These are a series of images taken in which people hold light sources such as lights or fireworks in their hands and wave them as if drawing letters. In each image, as shown as the first image 50 and the last image 60 in Figure 17, the area lit up by the light emitters held by each person is only a part of the image, but by combining these successively captured images, an image can be generated in which letters are drawn with light, as shown in composite image 65. When considering the composition of such images, in the additive average mode, the brightness of the light-based character portion decreases, but if the images are composed in the comparatively bright mode, the light-based character appears in a high-brightness state.

[0180] In other words, a desirable image can be generated with the shutter speed changed. Furthermore, the content and atmosphere of the composite image 65 will change significantly depending on the composition range when combining such a group of images. Therefore, even in the comparatively bright mode composition process, being able to freely set the composition range on the UI screen 100 is useful for the user, as it allows them to express themselves as they like.

[0181] Next, the weighting mode will be described. The weighting mode is a mode in which the weighting of each image data within the synthesis range is changed and synthesis is performed. It is also possible to change the weighting of each image data in a specific region within the frame.

[0182] First, an example of changing the weighting for each image data within the synthesis range will be given. By increasing the weight of the leading image 50 or the trailing image 60, the effect of front curtain synchronization or rear curtain synchronization can be obtained in the composite image 65. It is also possible to increase the weighting of any frame or images of multiple frames specified by the user. This allows a composite image 65 to be obtained in which the images of specific frames are emphasized.

[0183] It is also possible to set the weight of some frames to zero within the composite range. For example, if there is a single frame that you do not want to add to the composite, you can exclude that image from the composite.

[0184] For such weighting, a weight can be specified numerically for each image (frame) on the UI screen 100. For example, the initial value can be set to "1.0" for all frames, and the user can change the value from "0.0" to "65535.0". It is also possible to specify frames that are not to be used in synthesis, that is, frames whose weight is set to 0.0, and to specify frames that are to be emphasized (whose weight is to be increased).

[0185] It is also possible to vary the weights within a frame of an image. For example, if an unnecessary subject is captured in an image, the weight of that area can be set to zero, thereby making it possible to obtain a composite image 65 that excludes that subject. Conversely, the weight of the area of ​​the subject that is to be emphasized can be increased.

[0186] In order to change the weight within the image frame, the UI screen 100 provides a user interface that allows a user to specify an area and set the weight, similar to the specification of the fixed area described above. For example, the area can be specified by a rectangular frame, by freehand, or by specifying multiple points to specify an area that includes the area. Also, the area recognized by the semantic segmentation process may be presented, and the user may specify the area and change the weight.

[0187] <9. Summary and Variations> According to the above embodiment, the following effects can be obtained. The control device according to the embodiment includes a UI control unit 30. For example, the imaging device 1, the camera control unit 18, the information processing device 70, and the CPU 71 correspond to the control device including the UI control unit 30. The UI control unit 30 then performs control to simultaneously display images of a portion of image data including image data of a leading image 50 that is the first image in time and an ending image 60 that is the last image in time in the synthesis range to be subjected to the synthesis process, out of a group of images made up of multiple image data that are consecutive in time. When performing compositing processing of RAW images of multiple frames that are consecutive in time, by simultaneously displaying at least the first image 50 and the last image 60 in the compositing range, which is the time range to be composited, the user can check the two images at the start and end of the compositing, making it easier to grasp the time range to be composited. This also makes it easier to set the compositing range. Furthermore, since not all images in the synthesis range are displayed, each image does not become too small on one screen, and a UI screen that allows the user to easily view the images can be provided.

[0188] The UI control unit 30 of the embodiment performs control so that the first image 50 and the last image 60 are displayed side by side without screen transition within one screen of the UI screen 100. For example, as described in Fig. 9 and Fig. 10, the first image 50 and the last image 60 are displayed simultaneously and can be compared without scrolling the screen. This allows the user to compare the leading image 50 and the trailing image 60 without performing operations such as screen transitions or moving the display range. In other words, a UI screen can be provided that allows the user to easily grasp the composite range at a glance and allows the user to change the composite range while checking the composite range.

[0189] In the embodiment, the UI control unit 30 performs an example of controlling the display of the shutter speed for a composite image obtained by a synthesis process of a plurality of images included in a synthesis range from the first image 50 to the last image 60. The display is performed by the shutter speed display unit 51. This allows the user to know what shutter speed the composite image will have, and also serves as a guide for setting the composite range when the user wishes to generate a composite image with a slower shutter speed.

[0190] The UI control unit 30 in the embodiment performs control to change the setting of the leading image 50 or the trailing image 60 in response to an operation to change the leading image 50 or the trailing image 60. This allows the user to arbitrarily specify the first image 50 and the last image 60 in the image group, thereby enabling the user to generate a composite image with a desired shutter speed.

[0191] The UI control unit 30 of the embodiment executes a time axis display for the image group, and an operation on the time axis display is an operation for changing the first image or the last image. For example, a slide bar representing the time axis of the image group is illustrated. The user can operate the slide bars of the bar / playback operator display sections 52, 56 to select the leading image 50 or the trailing image 60. In this case, the user can intuitively select the synthesis range from within the image group.

[0192] The UI control unit 30 in the embodiment changes the image to be displayed as the leading image 50 or the trailing image 60 in response to an operation. This allows the user to select the leading image 50 or the ending image 60 while switching the image displayed as the leading image or the ending image by performing a play operation, a stop operation, a frame forward operation, or a frame back operation using the operators on the bar / playback operator display sections 52, 56. In other words, the leading image 50 and the ending image 60 can be specified while checking the images. This makes the UI screen 100 a screen suitable for setting the leading image 50 and the ending image 60 according to the user's wishes.

[0193] It has also been described that the UI control unit 30 of the embodiment sets the first image 50 or the last image 60 in response to an input operation that specifies the shutter speed. For example, the user can input an arbitrary shutter speed value to the shutter speed display unit 51. In response to this, the UI control unit 30 calculates a synthesis range for realizing the input shutter speed from, for example, the first image 50, and sets the image at the end of the synthesis range to the last image 60. Alternatively, the UI control unit 30 calculates a synthesis range for realizing the input shutter speed using the last image 60 as a reference, and sets the image at the start of the synthesis range to the first image 50. In this way, a convenient editing function can be realized for a user who wants to realize a synthetic image with a desired shutter speed. In addition, whether the first image 50 or the last image 60 is used as the reference may be fixed to one of them, or it may be the image for which the fixed area is set by the fixed area frame 59 at that point in time. Also, an image other than the first image 50 and the last image 60 may be used as the reference.

[0194] The UI control unit 30 of the embodiment performs control to display a fixed area that fixes the position in the frame between each image in the synthesis process on the display of the first image 50 or the last image 60. For example, it is displayed by a fixed area frame 59. Since the fixed area is the same area (pixel position) in the frame for each image to be synthesized, the subject will be clear and not blurred in the synthesized image. In other words, a synthesized image that clearly expresses a specific subject can be obtained. In this case, the fixed area is displayed on the UI screen 100, so that the user can recognize which subject will be clearly imaged.

[0195] The UI control unit 30 in the embodiment performs processing for setting a fixed area within the frame of the leading image 50 or the trailing image 60 in response to an operation for designating a fixed area. This allows the user to specify any area or subject as a fixed area in the leading image 50 or the trailing image 60. Therefore, a synthesis process can be executed as image editing desired by the user. In particular, the fixed area aligns the subject in the image of the composition range, so it is easy to realize image expressions that require technique, such as panning. Furthermore, the fixed area setting can provide a correction effect corresponding to subject blur.

[0196] In the embodiment, the UI control unit 30 performs control to display candidates for fixed areas, for example using a candidate frame 59A, on the display of the first image 50 or the last image 60, and performs processing to set a fixed area within the frame of the first image 50 or the last image 60 in response to an operation to specify a candidate fixed area. Presenting candidates for fixed regions serves as a guide for the user in specifying a fixed region, and also makes it easier to specify a fixed region. Furthermore, by allowing fixed area designation for either the leading image 50 or the trailing image 60, a composition closer to the final image can be achieved.

[0197] In the embodiment, the UI control unit 30 displays a fixed area or a candidate for a fixed area using a fixed area frame 59 or a candidate frame 59A on the display of the first image 50 or the last image 60, and also performs control to execute a suitability display 67 that indicates the suitability as a fixed area. By displaying the suitability of fixed regions and candidates for fixed regions, the user can determine whether the current settings are appropriate for compositing processing, or whether better settings are possible. Therefore, the display of suitability is an effective aid for operating the UI screen for compositing processing settings.

[0198] The UI control unit 30 of the embodiment controls a user interface that allows the selection of a processing method for the compositing process, and sets the processing method for the compositing process in response to a selection operation of the processing method for the compositing process. For example, an additive average mode, a comparatively bright mode, a weighting mode, etc. can be selected as the compositing mode. By allowing the user to select any desired composition mode, the user can edit the composite image with a higher degree of freedom. Also, by setting the composition mode on the user interface screen, composition editing with a higher degree of freedom is provided. The synthesis modes are not limited to the above three types, and other synthesis processing methods may also be implemented.

[0199] In the embodiment, the UI control unit 30 performs control to simultaneously display the leading image 50, the trailing image 60, and the composite image 65 (see FIGS. 9 and 10). This enables the user to visually compare a composite image 65 generated by a compositing process of the compositing range indicated by the start image 50 and the end image 60. Therefore, while checking the compositing result on the UI screen 100, the user can set the start image 50 and the end image 60 and specify a fixed area, etc., which improves usability. Moreover, by displaying a composite image 65 that reflects the development processing, it is possible to carry out editing processing for composition while checking the final image.

[0200] The UI control unit 30 in the embodiment performs control to simultaneously display the leading image 50, the trailing image 60, and an operator for instructing a synthesis process. That is, a synthesis execution button 62 is provided on the UI screen 100. This allows the user to instruct synthesis processing on the user interface screen where the start image 50 and the end image 60 can be compared visually, which is extremely convenient.

[0201] In the embodiment, an example has been given in which each image data of an image group is image data in which the exposure periods are temporally continuous when the images are captured (see FIGS. 7 and 8). For example, by making multiple RAW image data into a continuous state with no interruption in the exposure period, the composite RAW image data becomes image data that can be treated as equivalent to an image captured at a shutter speed slower than the shutter speed at which the image was actually captured. In other words, it becomes possible to edit the shutter speed in the original sense of the word, that is, the length of the continuous exposure period. This makes it possible to change the shutter speed after shooting, something that normally cannot be done after shooting.

[0202] In the above example, the multiple RAW image data to be synthesized by the synthesis processing unit 31 are processed by adding an exposure amount corresponding to a non-exposure period in the image sensor 12a to the exposure amount in the exposure period in the image sensor 12a, so that the exposure periods are made continuous in time (see FIG. 7). As a result, even if there is a non-exposure period such as a readout period or a charge reset period, the multiple RAW image data can be considered as data obtained in a state in which the non-exposure period does not exist. Therefore, the synthesis process can be performed by treating a series of RAW image data as multiple RAW image data whose exposure periods are made continuous in time. Also, an example has been given in which the multiple RAW image data to be synthesized by the synthesis processing unit 31 are read out alternately and sequentially from the multiple image sensors 12a1, 12a2, so that the exposure periods are continuous in time (see FIG. 8). By using the multiple image sensors 12a1, 12a2, setting the exposure timing of the image sensors 12a1, 12a2 so that one is exposed during a non-exposure period of the other, and reading out the RAW image data alternately and sequentially from the image sensors 12a1, 12a2, the series of RAW image data becomes multiple RAW image data whose exposure periods are continuous in time.

[0203] In the embodiment, an example has been given in which each image data of the image group to be combined is RAW image data. Performing the compositing process of combining multiple image data at the RAW image data stage means that the multiple image data are combined at a stage before development processing is performed on them, and then development processing is performed. Then, performing development processing on the composite RAW image data to generate JPEG image data or the like means that development processing suitable for that composite RAW image data can be performed. The multiple RAW image data to be synthesized may be captured as still images that are continuous in time, or may be RAW image data of each frame captured as a moving image. Furthermore, the multiple RAW image data to be synthesized may be multiple RAW image data that are unrelated to each other, such as those that have no relationship in terms of the capture time or the subject. In other words, it is also possible to synthesize multiple arbitrary RAW image data selected by the user. In addition, by developing the composite RAW image data after combining the RAW image data, development processing is performed with various parameters suitable for the composite RAW image data, and image data in a predetermined format after combination, such as JPEG image data, is obtained. This makes it possible to improve the image quality of the generated composite image (for example, an image based on JPEG image data). Furthermore, since it is not necessary to carry out development processing on each of the multiple pieces of RAW image data before combining, it is possible to realize more efficient development processing. In addition, composite RAW image data is created by combining RAW image data before development processes, including compression processes, and since a lot of the information from the original image remains, it has the advantage of giving users greater freedom when creating images. In addition, by storing composite RAW image data, it will be possible to develop the image in the future when development processing technology advances.

[0204] The RAW image data is an example of image data having the same color arrangement as the color arrangement of the imaging element. For example, image data in R, G, B format. It includes image data composed of pixel values ​​of G1, G2, R, and B pixels in the Bayer array, and image data that has been subjected to some of the above-mentioned processing. RAW image data composed of such pixel values ​​is the original image information read out from the imaging element unit 12, and has a high degree of freedom in development processing after synthesis. As an example of RAW image data, we have given YC-RAW image data, which is image data in which pixel values ​​read from the image sensor are converted into brightness and chroma values ​​and which has not been subjected to color reproduction / sharpness processing. The efficiency of development processing can also be improved by combining YC-RAW image data and performing a composition process. YC-RAW image data can also be restored to RAW image data, making it possible to perform composition processing and subsequent development processing after returning it to RAW image data.

[0205] In the embodiment, an example has been given in which each image data of an image group is image data that is temporally continuous by continuous still image capturing or moving image capturing. In the case of a group of images captured by continuous still image shooting or video shooting, the image data in the synthesis range are continuous in time, which is suitable from the viewpoint of editing the shutter speed. In addition, the UI screen of the embodiment can be suitably used for synthesis processing using a group of images captured by normal continuous image shooting or video shooting as material.

[0206] The control device (e.g., the imaging device 1, the information processing device 70) in the embodiment is exemplified as further including a synthesis processing unit 31 that synthesizes a plurality of images included in a synthesis range from the first image 50 to the last image 60 set by the UI control unit 30. It has also been described that the camera control unit 18 and the CPU 71 may include the synthesis processing unit 31 together with the UI control unit 30. This allows multiple image data to be synthesized to obtain synthesized image data with a shutter speed slower than the shutter speed at which each image data was captured. For example, in the embodiment, the multiple RAW image data synthesized by the synthesis processing unit 31 is RAW image data captured as temporally continuous frames. This allows an image to be generated after capture as if it was captured at a shutter speed slower than the shutter speed at which the image was captured. In other words, it becomes possible to edit the shutter speed after shooting. This also makes it possible to prevent failure due to incorrect shutter speed settings at a precious photo opportunity, for example. You will also no longer miss the perfect shot because of the time it takes to adjust the shutter speed. Furthermore, since the shutter speed during image capture is not changed, there is no need to adjust the amount of light, and a composite image at any shutter speed can be obtained from images captured with an appropriate amount of light. Furthermore, by fine-tuning the shutter speed after shooting, new image expressions can be achieved. Also, after editing the shutter speed, the composite RAW image data can be developed.

[0207] The program of the embodiment is a program that causes, for example, a CPU, a DSP, a GPU, a GPGPU, an AI processor, or a device including these, to execute the processes as shown in FIGS. In other words, the program of the embodiment is a program that causes a processing device to execute UI control to simultaneously display a portion of images, including a leading image 50 that is the first image in time and a trailing image 60 that is the last image in time, in a synthesis range to be subjected to synthesis processing, from a group of images made up of multiple image data that are consecutive in time. Such a program allows the control device referred to in the present disclosure to be realized by various computer devices.

[0208] These programs can be pre-recorded on a HDD as a recording medium built into a device such as a computer device, or on a ROM in a microcomputer having a CPU. Alternatively, they can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0209] In addition, such a program is suitable for providing the control device of the present disclosure in a wide range of applications. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the control device of the present disclosure.

[0210] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0211] The present technology can also be configured as follows. (1) The image processing device includes a user interface control unit that controls simultaneous display of a portion of images including a first image that is the first in time and a last image that is the last in time in a synthesis range to be subjected to synthesis processing, among a group of images formed by a plurality of image data that are consecutive in time. Control device. (2) The user interface control unit includes: The first image and the last image are displayed in a layout in which they are displayed side by side on one screen without screen transition. The control device described in (1) above. (3) The user interface control unit includes: The display control unit controls the shutter speed for a composite image obtained by a synthesis process of a plurality of images included in a synthesis range from the first image to the last image. The control device according to (1) or (2) above. (4) The user interface control unit includes: The setting of the leading image or the trailing image is changed in response to a change operation of the leading image or the trailing image. A control device according to any one of (1) to (3) above. (5) The user interface control unit includes: A time axis display of the image group is executed, and an operation on the time axis display is a change operation of the first image or the last image. The control device described in (4) above. (6) The user interface control unit includes: The image to be displayed as the first image or the last image is changed according to an operation. The control device according to (4) or (5) above. (7) The user interface control unit includes: The first image or the last image is set in response to an input operation of the shutter speed. A control device according to any one of (4) to (6) above. (8) The user interface control unit includes: A fixed area for fixing an in-frame position between each image in a synthesis process is displayed on the display of the first image or the last image. A control device according to any one of (1) to (7) above. (9) The user interface control unit includes: In the synthesis process, a process of setting a fixed area within the frame of the first image or the last image is performed in response to an operation of designating a fixed area for fixing an in-frame position between each image. A control device according to any one of (1) to (8) above. (10) The user interface control unit includes: performing control to display, on the display of the first image or the last image, a candidate for a fixed area for fixing an in-frame position between each image in a synthesis process; A process of setting a fixed area within the frame of the first image or the last image in response to an operation of designating a candidate for a fixed area is performed. A control device according to any one of (1) to (9) above. (11) The user interface control unit includes: A fixed area or a candidate for a fixed area for fixing an in-frame position between each image in a synthesis process is displayed on the display of the first image or the last image, and a display showing the suitability as a fixed area is performed. A control device according to any one of (1) to (10) above. (12) The user interface control unit includes: A user interface control is provided that allows the user to select the synthesis processing method, Select the processing method for the compositing process according to the operation. A control device according to any one of (1) to (11) above. (13) The user interface control unit includes: Control is performed to simultaneously display the top image, the bottom image, and a composite image. A control device according to any one of (1) to (12) above. (14) The user interface control unit includes: Control is performed to simultaneously display the leading image, the trailing image, and an operator for instructing a synthesis process. A control device according to any one of (1) to (13) above. (15) The image data of each of the image groups is image data in which the exposure periods are continuous in time when the images are captured. A control device according to any one of (1) to (14) above. (16) Each image data of the image group is RAW image data. A control device according to any one of (1) to (15) above. (17) The image data of each of the image groups is image data that is temporally continuous by continuous still image capture or video capture. A control device according to any one of (1) to (16) above. (18) The image processing device further includes a synthesis processing unit that synthesizes a plurality of images included in a synthesis range from the first image to the last image set by the user interface control unit. A control device according to any one of (1) to (17) above. (19) The control device, A user interface control is performed to simultaneously display a part of images including a first image that is the first in time and a last image that is the last in time in a synthesis range to be subjected to synthesis processing among a group of images formed by a plurality of image data that are consecutive in time. Control methods. (20) A user interface control for simultaneously displaying a portion of images including a leading image that is the first image in time and a trailing image that is the last image in time in a synthesis range to be subjected to synthesis processing among a group of images formed by a plurality of image data that are consecutive in time, A program to be executed by a processor. [Explanation of symbols]

[0212] 1. Imaging device 2 Input selection area 3 Adjustment Area 4 Top image area 5 End Image Area 6 Composite Image Area 12 Image sensor section 12a, 12a1, 12a2 Image sensor 18 Camera control unit 20 Image processing section 30 UI control section 31 Synthesis Processing Unit 32 Development Processing Section 50 First Image 51 Shutter speed display 52,56 Bar / Playback Control Display 59 Fixed Area Frame 59A Candidate slot 60 End Image 61 Blending mode setting section 62 Synthesis execution button 65 Composite Images 67 Suitability indicator 70 Information processing device, 71 CPU 100 UI screens

Claims

1. a user interface control unit that performs control to simultaneously display a portion of images including a leading image that is the first image in time and a trailing image that is the last image in time in a synthesis range to be subjected to synthesis processing, among a group of images formed by a plurality of image data that are consecutive in time, The user interface control unit includes: A fixed area or a candidate for a fixed area for fixing an in-frame position between each image in a synthesis process is displayed on the display of the first image or the last image, and a display showing the suitability as a fixed area is performed. Control device.

2. The user interface control unit includes: The first image and the last image are displayed in a layout in which they are displayed side by side on one screen without screen transition. The control device according to claim 1 .

3. The user interface control unit includes: The display control unit controls the shutter speed for a composite image obtained by a synthesis process of a plurality of images included in a synthesis range from the first image to the last image. The control device according to claim 1 .

4. The user interface control unit includes: The setting of the leading image or the trailing image is changed in response to a change operation of the leading image or the trailing image. The control device according to claim 1 .

5. The user interface control unit includes: A time axis display of the image group is executed, and an operation on the time axis display is a change operation of the first image or the last image. The control device according to claim 4.

6. The user interface control unit includes: The image to be displayed as the first image or the last image is changed according to an operation. The control device according to claim 4.

7. The user interface control unit includes: The first image or the last image is set in response to an input operation of the shutter speed. The control device according to claim 4.

8. The user interface control unit includes: A fixed area for fixing an in-frame position between each image in a synthesis process is displayed on the display of the first image or the last image. The control device according to claim 1 .

9. The user interface control unit includes: In the synthesis process, a process of setting a fixed area within the frame of the first image or the last image is performed in response to an operation of designating a fixed area for fixing an in-frame position between each image. The control device according to claim 1 .

10. The user interface control unit includes: performing control to display, on the display of the first image or the last image, a candidate for a fixed area for fixing an in-frame position between each image in a synthesis process; A process of setting a fixed area within the frame of the first image or the last image in response to an operation of designating a candidate for a fixed area is performed. The control device according to claim 1 .

11. The user interface control unit includes: A user interface control is provided that allows the user to select the synthesis processing method. Select the processing method for the compositing process according to the operation. The control device according to claim 1 .

12. The user interface control unit includes: Control is performed to simultaneously display the top image, the bottom image, and a composite image. The control device according to claim 1 .

13. The user interface control unit includes: Control is performed to simultaneously display the leading image, the trailing image, and an operator for instructing a synthesis process. The control device according to claim 1 .

14. The image data of each of the image groups is image data in which the exposure periods are continuous in time when the images are captured. The control device according to claim 1 .

15. Each image data of the image group is RAW image data. The control device according to claim 1 .

16. The image data of each of the image groups is image data that is temporally continuous by continuous still image capture or video capture. The control device according to claim 1 .

17. The image processing device further includes a synthesis processing unit that synthesizes a plurality of images included in a synthesis range from the first image to the last image set by the user interface control unit. The control device according to claim 1 .

18. The control device, A user interface control is performed to simultaneously display a part of images including a leading image that is the first image in time and a trailing image that is the last image in time in a synthesis range to be subjected to synthesis processing, among a group of images formed by a plurality of image data that are consecutive in time; In the user interface control, A fixed area or a candidate for a fixed area for fixing an in-frame position between each image in a synthesis process is displayed on the display of the first image or the last image, and a display showing the suitability as a fixed area is performed. Control methods.

19. a user interface control is executed by the arithmetic processing device to simultaneously display a part of images including a leading image that is the first image in time and a trailing image that is the last image in time in a synthesis range to be subjected to synthesis processing, among a group of images based on a plurality of image data that are successive in time; In the user interface control, The arithmetic processing device is caused to execute control to display a fixed area or a candidate for a fixed area that fixes an in-frame position between each image in a synthesis process on the display of the first image or the last image, and to execute control to display a suitability as a fixed area. program.

Images