Imaging apparatus, imaging apparatus control method, and computer program product

Abstract

The imaging device comprises at least one camera module, an imaging control unit, an image processing unit and a display control unit. The imaging control unit obtains a first shot image through the at least one camera module in response to a user shooting instruction, and obtains at least one second shot image through the at least one camera module after obtaining the first shot image. The image processing unit identifies a first region corresponding to a photographing range of the latest second photographed image, and a second region which is a part of a photographing range of the first photographed image and which does not partially overlap a photographing range of at least one second photographed image. The display control unit outputs auxiliary information indicating the first area and the second area.

Description

Technical Field

[0001] The present application relates to an imaging device, an imaging device control method, and a computer program product. Background Art

[0002] In imaging devices such as smartphones, technologies for obtaining high-quality images by fusing multiple captured images are known, including panoramic synthesis, high dynamic range (HDR) synthesis, noise reduction (NR), etc.

[0003] Prior art literature:

[0004] [Patent Document 1] U.S. Application No. 6075905;

[0005] [Patent Document 2] Japanese Patent No.: 6450589;

[0006] [Patent Document 3] Japanese Patent No.: 4146938;

[0007] [Patent Document 4] Japanese Application No.: 5092722;

[0008] [Patent Document 5] U.S. Patent Application Publication No.: 2008 / 0030592.

[0009] Non-patent literature:

[0010] T. Wang, J. Xie, W. Sun, Q. Yan, and Q. Chen, "Dual-camera super-resolution based on aligned attention module," in 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp. 1981–1990, doi: 10.1109 / ICCV48922.2021.00201. Summary of the Invention

[0011] In order to obtain multiple images with different shooting ranges or shooting directions, some user operations may be required, such as moving the imaging device. Therefore, shooting assistance is needed to present the necessary operations to the user regarding which shooting range or direction the image should be shot.

[0012] The problem to be solved by the embodiments of the present application is to provide appropriate assistance for obtaining multiple captured images for image synthesis.

[0013] The imaging device of this embodiment includes: at least one camera module, an imaging control unit, an image processing unit, and a display control unit. The imaging control unit, in response to a user's shooting instruction, acquires a first captured image through the at least one camera module, and after acquiring the first captured image, acquires at least one second captured image through the at least one camera module. The image processing unit identifies a first area corresponding to the shooting range of the most recent second captured image, and a second area that is a portion of the shooting range of the first captured image and does not overlap with the shooting range of at least one second captured image. The display control unit outputs auxiliary information indicating the first area and the second area.

[0014] The imaging device control method of this embodiment is performed by an imaging device including at least one camera module. The method includes: in response to a user's shooting instruction, acquiring a first captured image through at least one of the camera modules; after acquiring the first captured image, acquiring at least one second captured image through at least one of the camera modules; identifying a first area corresponding to the shooting range of the most recent of the second captured images, and a second area that is a portion of the shooting range of the first captured image and does not overlap with the shooting range of at least one of the second captured images; and outputting auxiliary information indicating the first and second areas.

[0015] The computer program product of this embodiment stores a program that is executed by a computer in an imaging device including at least one camera module. The program causes the computer to perform the following operations: in response to a user's shooting instruction, acquire a first captured image using at least one of the camera modules; after acquiring the first captured image, acquire at least one second captured image using at least one of the camera modules; identify a first area corresponding to the capturing range of the most recent second captured image, and a second area that is a portion of the capturing range of the first captured image and does not overlap with the capturing range of at least one of the second captured images; and output auxiliary information indicating the first and second areas.

[0016] According to the embodiments of the present application, appropriate assistance can be provided for acquiring multiple captured images for image synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 FIG1 is a diagram showing an example of a hardware configuration of an imaging device according to the first embodiment;

[0018] Figure 2 is a diagram illustrating an example of a functional configuration of an imaging device according to the first embodiment;

[0019] Figure 3 FIG1 is a flowchart illustrating an example of a shooting assistance process according to the first embodiment;

[0020] Figure 4 is a diagram showing an example of a display screen of a shooting assistance flow according to the first embodiment;

[0021] Figure 5 is a schematic diagram explaining alignment processing according to the first embodiment;

[0022] Figure 6 is a schematic diagram explaining REF fusion processing according to the first embodiment;

[0023] Figure 7 FIG2 is an exemplary diagram of a display screen of a shooting assistance process according to the second embodiment;

[0024] Figure 8 FIG. 1 is a diagram showing an example of a hardware configuration of an imaging system according to a third embodiment. DETAILED DESCRIPTION

[0025] The following will describe in detail an imaging device, an imaging system, an imaging device control method, a program, and a computer program product according to various embodiments with reference to the accompanying drawings. It should be noted that the present application is not limited to these embodiments.

[0026] When describing this embodiment, components that have the same or substantially the same functions as those described in the previous drawings will be given the same reference numerals, and their descriptions may be omitted as appropriate. Furthermore, even if identical or substantially the same components are shown in the drawings, the dimensions and proportions of those components may vary between drawings. Furthermore, to ensure the visibility of the drawings, for example, only essential components may be labeled with reference numerals in each drawing, and components that have the same or substantially the same functions as those described in the previous drawings may not be labeled with reference numerals.

[0027] In imaging devices such as smartphones, technologies for acquiring high-quality images by fusing multiple captured images are known, including panoramic synthesis, high dynamic range (HDR) synthesis, and noise reduction (NR).

[0028] In order to obtain multiple images with different shooting ranges or shooting directions, the user may need to perform some operations, such as moving the imaging device. Therefore, it is necessary to provide shooting assistance to present the user with the necessary operations regarding which shooting range or direction to use for shooting images.

[0029] The embodiments of the present application will describe an imaging device, an imaging system, an imaging device control method, a program, and a computer program product, which can provide appropriate assistance for acquiring multiple captured images for image synthesis.

[0030] First embodiment

[0031] Figure 1 1 is an example diagram of the hardware configuration of the imaging device 1 according to the embodiment. The imaging device 1 is an information terminal device including a digital camera function, such as a smart phone or a tablet terminal. The imaging device 1 can also be other digital cameras. Figure 1 As shown, the imaging device 1 includes an imaging unit 10, an image signal processing circuit (image signal processor: ISP) 12, an input interface 13, an inertial measurement unit (IMU) 14, a processing circuit 15, a memory 16, and a display 17. The imaging unit 10 is electrically connected to the ISP 12. The ISP 12, input interface 13, IMU 14, processing circuit 15, memory 16, and display 17 are connected via signal lines (e.g., bus 19) to enable mutual communication.

[0032] The imaging unit 10 includes at least two camera modules. Figure 1 The imaging unit 10 is shown as comprising n camera modules 111 to 11n (n is a natural number). It should be noted that if there is no difference between the n camera modules 111 to 11n, they can be simply referred to as the camera module 11.

[0033] The camera module 11 acquires an image of the object area to generate a captured image (image data). The camera module 11 includes an optical system and an image sensor. The camera module 11 is fixedly mounted on the imaging device 1, and its shooting direction is fixed.

[0034] The optical system includes the optical elements that converge the light beam from the subject onto the imaging surface of the image sensor. Any optical system can be used as long as it has at least one optical element with optical power to achieve the desired imaging performance. In other words, the optical system can be composed of a compound lens including at least one single lens, or a combination of a lens system and a reflective system.

[0035] The image sensor captures an image of the subject area to generate an image signal. Suitable image sensors include solid-state imaging devices such as CCDs (charge-coupled devices) and CMOSs (complementary metal oxide semiconductors). Furthermore, the image sensor performs analog processing on the generated image signal, such as noise reduction, amplification, or analog-to-digital conversion, and outputs digital image data (captured image). It should be noted that some or all of the analog processing may be implemented by at least one other circuit located downstream of the image sensor, serving as an analog-to-digital front end (AFE).

[0036] It should be noted that the camera module 11 is configured to change the focus position. Here, "changeable focus position" means that for each of the object points existing at at least two different positions in the optical axis direction of the optical system, the size of the diffuse spot imaged on the imaging surface may be smaller than the allowable circle of confusion diameter. The definition of the allowable circle of confusion diameter may be based on the pixel pitch of the image sensor or the imaging performance of the optical system. In other words, the imaging unit 10 may be configured to enable any photographed object to be focused or blurred (background blurred). Specifically, the camera module 11 may be configured to be able to move at least one of the following along the optical axis direction of the optical system: the image side focus position, the object side focus position of the optical system, and the imaging surface of the image sensor.

[0037] It should be noted that, for simplicity of description, this embodiment uses the imaging unit 10 as an example, comprising two camera modules 111 and 112. Camera module 112 is configured to be able to perform imaging of finer textures than camera module 111. It should be noted that the imaging device 1 may also include two or more camera modules configured to be able to perform imaging of finer textures than camera module 111.

[0038] Exemplarily, camera module 111 is configured to perform imaging operations at a first viewing angle. Camera module 112 is configured to perform imaging operations at a second viewing angle that is smaller than the first viewing angle. In other words, the optical system focal length of camera module 112 is longer than the optical system focal length of camera module 111.

[0039] For example, camera module 111 is configured to be capable of performing imaging at a first resolution (a first number of pixels). Camera module 112 is configured to be capable of performing imaging at a second resolution (a second number of pixels) that is higher than the first resolution. In other words, the focal length of the optical system of camera module 112 is longer than the focal length of the optical system of camera module 111.

[0040] In this embodiment, the captured image acquired by the camera module 111 is referred to as a low-resolution (LR) image. It should be noted that the LR image may also be referred to as a low-quality (LQ) image. The captured image acquired by the camera module 112 is referred to as a reference (REF) image. It should be noted that the LR image in this embodiment is an example of the first captured image, while the REF image is an example of the second captured image.

[0041] As described below, the imaging device 1 of this embodiment is configured to acquire a high-resolution (HR) image by fusing two or more REF images into a single LR image captured by the camera module 111. An HR image may also be referred to as a high-quality (HQ) image. It should be noted that the HR image in this embodiment is an example of a fused image, which corresponds to the captured range of the LR image.

[0042] The ISP 12 is located downstream of the imaging unit 10 and performs various image processing required for displaying and recording the images from the camera module 11. Such image processing includes, but is not limited to: optical black (OB) subtraction processing, white balance (WB) correction processing, demosaicing processing, color conversion processing, gamma conversion processing, noise reduction processing, scaling processing, compression processing, etc. The ISP 12 controls the various components of the imaging unit 10 according to the program stored in the internal memory or the memory 16. As hardware resources, the ISP 12 includes processing circuits and memories. The processing circuit can appropriately adopt various types of processors, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA). The memory can appropriately adopt various types of memories, such as read-only memory (ROM), flash memory, and random access memory (RAM). It should be noted that the ISP 12 can adopt a microcontroller, and some or all of the functions of the ISP 12 can be implemented by the image sensor of the camera module 11.

[0043] The input interface 13 receives user input operations on the imaging device 1 through various input devices, such as a touch screen, switches, buttons, a keyboard, and a microphone capable of receiving user voice input. For example, the input interface 13 receives a user's shutter operation (shooting instruction) for shooting via the touch screen provided on the display interface of the display 17 and the operation buttons on the housing of the imaging device 1.

[0044] The inertial measurement unit (IMU) 14 is capable of detecting the three-dimensional inertial motion of the imaging device 1. The IMU 14 can detect inertial forces generated by changes in the position and / or orientation of the imaging device 1 and output the detected inertial forces as electrical signals in the form of acceleration and / or angular velocity. For example, the IMU 14 includes an accelerometer for detecting translational motion of the imaging device 1. For example, the IMU 14 includes a gyroscope for detecting rotational motion of the imaging device 1.

[0045] The processing circuit 15 controls the various components of the imaging device 1 according to a program stored in the internal memory or the memory 16. Processing circuit 15 can be implemented as a CPU, DSP, ASIC, FPGA, or other suitable processor. Furthermore, the internal memory of processing circuit 15 can be implemented as a ROM, flash memory, RAM, or other suitable memory. It should be noted that processing circuit 15 can also be implemented as a microcomputer.

[0046] Memory 16 stores the programs required for the operation of imaging device 1. It also stores information required for various device processes, including various parameters and thresholds. Memory 16 also temporarily stores images output by ISP 12, processed data from processing circuit 15, and captured image data used for image synthesis. Hardware resources include nonvolatile memory such as ROM and flash memory, as well as volatile memory such as dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and static random access memory (SRAM).

[0047] Display 17 displays images captured by imaging unit 10 and various operation interfaces. During the acquisition of multiple images for image synthesis, display 17 presents a display screen containing auxiliary information to assist the user in operating imaging device 1. Display 17 can be implemented as a liquid crystal display (LCD), an organic electroluminescent (EL) display, a projector, or other device. Display 17 can be configured as a touchscreen display, wherein the display screen includes a touch panel for receiving user input operations.

[0048] Figure 2 1 is a diagram illustrating an example of the functional configuration of the imaging device 1 according to this embodiment. The imaging device 1 implements the functions of the imaging control unit 101, image processing unit 102, storage control unit 103, and display control unit 104 by executing programs loaded into the internal memory of the ISP 12, the internal memory of the processing circuit 15, or the RAM of the storage 16.

[0049] It should be noted that the imaging control unit 101, image processing unit 102, storage control unit 103, and display control unit 104 may be implemented by a single processor or by a combination of two or more independent processors. In addition, each of the imaging control unit 101, image processing unit 102, storage control unit 103, and display control unit 104 may be implemented by being integrated or distributed across multiple processors.

[0050] The imaging control unit 101 performs automatic exposure (AE) processing, setting imaging conditions including the aperture value and shutter speed value based on an AE evaluation value representing the brightness of the subject in the captured image. For example, the imaging control unit 101 performs AE processing based on the user's first shutter operation as a trigger.

[0051] Furthermore, the imaging control unit 101 performs autofocus (AF) adjustment processing, controlling at least one of a focus lens and an image sensor included in the optical system based on focus information acquired from an image or the like. The focus information may be, for example, an AF evaluation value (contrast value) calculated based on a captured image. Furthermore, when the image sensor is configured with focus detection pixels, the focus information may be a defocus amount calculated based on the output of the focus detection pixels.

[0052] Furthermore, the imaging control unit 101 controls the imaging unit 10 to capture images. For example, the imaging control unit initiates the capture process based on the user's second shutter operation (capture command). The first and second shutter operations include pressing an operation button on the imaging device 1 housing and tapping any subject on the touchscreen display (display 17) during preview display. Note that preview display can also be referred to as live view display.

[0053] Illustratively, the imaging control unit 101 acquires LR images in response to a user shooting instruction.

[0054] Exemplarily, the imaging control unit 101 acquires at least two REF images for fusion into the acquired LR image. More specifically, when acquiring the LR image, the imaging control unit 101 acquires a REF image via the camera module 112, and then acquires at least one more REF image after acquiring the LR image. The at least two REF images may be acquired by the display control unit 104 when displaying the auxiliary information. For example, at least two REF images may be selected from multiple REF images acquired at predetermined time intervals.

[0055] It should be noted that, for example, when the inertial motion of the imaging device 1 exceeds a predetermined threshold, the imaging control unit 101 can acquire a REF image. Alternatively, for example, when there is an area within the shooting range of the LR image where the REF image is not acquired, and the proportion of this area in the shooting range of the camera module 112 exceeds a predetermined threshold, a REF image can also be acquired.

[0056] The image processing unit 102 performs alignment, auxiliary information generation, REF image collection, and REF image fusion. Exemplarily, the image processing unit 102 performs alignment based on the image features of the LR image and the REF image and the output of the IMU 14. During the auxiliary information generation process, each time the database in the memory 16 is updated due to a newly acquired REF image, the image processing unit 102 generates auxiliary information based on the alignment results. In this embodiment, the auxiliary information is used to display a border and mask superimposed on the LR image. The border indicates a first area of the LR image corresponding to the captured range of the most recent REF image, and the mask indicates a second area of the LR image for which a REF image has not yet been acquired. For example, during the REF image collection process, the image processing unit 102 retrieves at least two REF images stored in the database area of the memory 16. During the REF image fusion process, the image processing unit 102 fuses the at least two REF images into the LR image, thereby generating a HR image corresponding to the captured range of the LR image.

[0057] The storage control unit 103 stores the LR image acquired by the camera module 111 in the memory 16. In addition, the storage control unit 103 stores the REF image acquired by the camera module 112 in the database area of the memory 16. Exemplarily, the storage control unit 103 updates the database of the memory 16 with a newly acquired REF image having a more detailed texture than the REF images stored in the database.

[0058] The display control unit 104 outputs the auxiliary information generated by the image processing unit 102. For example, the display control unit 104 outputs the auxiliary information to the display 17, thereby presenting the LR image superimposed with the auxiliary information on the display 17. The auxiliary information superimposed on the LR image may be, for example, a frame indicating a first area corresponding to the captured range of the most recent reference image, and a mask indicating a second area for which no reference image has been captured. The display control unit 104 also displays a message prompting the user to change the shooting direction or shooting position. It should be noted that the display control unit 104 may also display a further message prompting the user to change the position of the subject in the shooting direction.

[0059] An operation example of the imaging device 1 in this embodiment will be described below. Figure 3 FIG. 1 is a flowchart illustrating an example of a shooting assistance process according to an embodiment. Figure 4 FIG. 1 is a diagram illustrating an example of a display screen of a shooting assistance process according to an embodiment.

[0060] In addition, if Figure 4 In the example shown, the display control unit 104 displays a display screen 310 including the LR image 201 on the display 17. For example, Figure 3 The process is performed when the user points the imaging device 1 at a subject of interest. Figure 3 The process shown can also be performed in other situations. Figure 4 In the illustrated example, the display control unit 104 displays a display screen 310 including the LR image 201 displayed in preview on the display 17 .

[0061] The user controls the camera module 111 to acquire LR images by triggering a shooting command such as pressing the operation button 131. The storage control unit 103 stores the acquired LR images in the memory 16 (S1).

[0062] The imaging control unit 101 operates the camera module 112 to acquire the REF image 202 as a candidate for the REF image ( S2 ).

[0063] The image processing unit 102 performs alignment processing. Figure 5 FIG. 1 is a schematic diagram illustrating the alignment process according to this embodiment. Figure 5As shown, the alignment process is a process for roughly spatially positioning the captured REF image 202 relative to the LR image 201. Image processing unit 102 uses the alignment process to identify region 203 in LR image 201 that corresponds to REF image 202. Region 203 is a portion of the range captured by camera module 111 when capturing LR image 201, corresponding to the range captured by camera module 112 when capturing the latest REF image 202. Region 203 in this embodiment is an example of a first region.

[0064] For example, the image processing unit 102 performs alignment processing by extracting image features from each of the LR image 201 and the REF image 202 and performing matching based on the extracted features, such as positioning and / or deformation. Image feature extraction may include edge detection, face detection, or pupil detection. Furthermore, the image processing unit 102 may also identify the direction and magnitude of changes in the captured range based on the output of the IMU 14 to perform alignment processing.

[0065] The storage control unit 103 determines whether the acquired REF image 202 has a more detailed texture than the REF images stored in the database of the memory 16 (step S4 ).

[0066] Illustratively, when this is the first determination, the storage control unit 103 determines that the acquired REF image 202 has a more detailed texture than the REF images stored in the database of the memory 16 , for example, there is no REF image in the database.

[0067] For example, when the region 203 includes the unacquired region 205, the storage control unit 103 determines that the acquired REF image 202 has a more detailed texture than the REF images stored in the database of the memory 16. The storage control unit 103 can, for example, identify the direction and magnitude of the movement of the shooting range based on the output of the IMU 14 to determine whether the unacquired region 205 is included in the region 203.

[0068] For example, when the acquired reference image 202 is an unblurred image, the storage control unit 103 determines that the acquired REF image 202 has a more detailed texture than the REF image stored in the database of the memory 16. A unblurred image herein refers, for example, to an image for which the edge detection evaluation value is greater than a preset threshold. It should be noted that edge detection can be performed on a portion of an image, such as a facial region within the image.

[0069] Here, unacquired region 205 refers to an area in LR image 201 for which no corresponding REF image is stored in the database of memory 16. In other words, unacquired region 205 is an area lacking a REF image for image synthesis. This area corresponds to the shooting range in which camera module 112 should acquire a REF image. In this embodiment, region 205 is an example of a second region.

[0070] Exemplarily, when the resolution of region 203 is higher than the resolution of the corresponding region in the acquired region 209, the storage control unit 103 determines that the acquired REF image 202 has a more detailed texture than the REF images in the database of the memory 16. For example, when a user moves the imaging device 1 in a shooting direction, closer to the subject, to acquire a REF image, the acquired REF image has a more detailed texture than the REF image acquired before the movement. For example, when the imaging device 1 includes three or more camera modules 11, and one of the camera modules 11 is closer to the telephoto end than another camera module 11 that acquires the LR image, or the camera module 11 has a higher resolution, the REF image acquired by the camera module 11 has a more detailed texture than the LR image.

[0071] The acquired region 209 herein refers to a region in the LR image 201 whose corresponding REF image has been stored in the database of the memory 16. In other words, the acquired region 209 corresponds to a portion of the shooting range when the camera module 111 shoots the LR image 201, and this portion corresponds to the shooting range in which the camera module 112 has already captured the REF image.

[0072] If the acquired REF image 202 is determined to have a more detailed texture than the REF image stored in the database (S4: Yes), the storage control unit 103 updates the database with the new REF image. More specifically, the storage control unit 103 stores the REF image 202 acquired in step S2 as the REF image of the region 203 in the database of the memory 16 (S5).

[0073] On the other hand, when it is determined that the acquired REF image 202 does not have a more detailed texture than the REF image stored in the memory 16 (S4: No), Figure 3 The process will enter step S6.

[0074] The image processing unit 102 generates an auxiliary image for displaying the REF area. Figure 4 As shown in the example of , the display control unit 104 presents a display screen 320 including the auxiliary image on the display 17 ( S6 ).

[0075] The auxiliary image showing the REF area includes a frame indicating the area 203 and a mask indicating the unacquired area 205 (in the Figure 4 The image processing unit 102 regards the region in the LR image 201 corresponding to the REF image stored in the updated database as the acquired region 209 and regards the other regions in the LR image 201 except the region 209 as the unacquired region 205.

[0076] For example, an auxiliary image showing the REF region is displayed superimposed on the LR image 201. Furthermore, the display screen 320 includes a message prompting the user to move the imaging device 1 to change the shooting range of the camera module 112, such as a message stating "Move your phone." The display of the REF region and the message are examples of auxiliary information display.

[0077] Note that both the image indicating the region 203 and the mask image can be generated as an auxiliary image showing the REF region. Alternatively, the auxiliary image can be generated by superimposing a frame indicating the region 203 and a mask indicating the unobtained region 205 onto the LR image 201.

[0078] Subsequently, the imaging control unit 101 determines whether the capture process for acquiring the REF image 202 has ended (S7). In this determination, for example, the capture process is determined to have ended when the size of the area containing the REF image or the proportion of that area in the LR image 201 is equal to or greater than a preset threshold. Alternatively, for example, in this determination, the capture process is determined to have ended when a preset time has elapsed since the acquisition of the LR image 201, i.e., a timeout has occurred. Alternatively, for example, in this determination, the capture process is determined to have ended when a user instruction to end the capture is received via the input interface 13.

[0079] When it is determined that the shooting process of acquiring the REF image 202 has not been completed (S7: No), Figure 3 The process will loop through steps S2 to S7 until it is determined in step S7 that the shooting process is finished.

[0080] The REF image 202 and the LR image 201 of the corresponding area 203 on the display screen 320 are acquired almost simultaneously. In other words, the REF image 202 and the LR image 201 are acquired with little or no change in the orientation or position of the imaging device 1. In this case, the camera modules 111 and 112 only need to perform simultaneous or sequential capture in response to the user's capture instructions, and the user does not need to operate the imaging device 1 during the capture process.

[0081] On the other hand, in order to obtain at least two REF images to be fused into the LR image 201, the user needs to change the orientation or position of the imaging device 1, thereby changing the shooting range of the camera module 112 fixed on the imaging device 1. For example, Figure 4 The illustrated display screen 330 shows, by way of example, a case where, after acquiring the REF image 202 corresponding to the area 203 in the display screen 320 , the user pans the imaging device 1 to the left.

[0082] In this case, the newly acquired REF image 202 is identified as region 203, and the auxiliary image is updated to display the REF region based on the identified region 203. More specifically, the display screen 330 displays a frame indicating the region 203 corresponding to the newly stored REF image. Furthermore, the display screen 330 updates the region within the frame on the display screen 320 and displays it as the acquired region 209. The acquired region 209 is an area not covered by the mask. In other words, the display screen 330 displays the region 203 as the area not covered by the mask; before the update, on the display screen 320, the region 203 was part of the mask region 205 corresponding to the newly stored REF image.

[0083] Subsequently, when the user rotates the imaging apparatus 1 clockwise, for example, the display of the REF area is updated according to the area 203 which is updated in sequence, as shown in display screens 340 , 350 , and 360 .

[0084] When determining that the shooting process of acquiring the REF image 202 is completed (S7: Yes), the image processing unit 102 performs REF image collection processing (S8). The REF image collection processing is a process of reading and collecting at least two REF images to be fused into the LR image 201 from the memory 16.

[0085] Subsequently, the image processing unit 102 performs REF fusion processing ( S9 ). Figure 6 FIG. 1 is a schematic diagram of the REF fusion process explained according to the first embodiment. In the REF fusion process, as Figure 6 As shown in the example of , the image processing unit 102 aggregates at least two REF images acquired in the REF collection process to generate a single fused image 210 . Figure 6 An example is shown in which nine REF images REF0 , REF1 , REF2 , REF3 , REF4 , REF5 , REF6 , REF7 , and REF8 are aggregated to generate a single fused image 210 .

[0086] It should be noted that the number of REF images used to generate the fused image 210 is at least two. More specifically, the number of REF images used to generate the fused image 210 is at least one, and does not include a REF image acquired simultaneously with the LR image 201. In other words, the REF images used to generate the fused image 210 include at least one REF image acquired after the user changes the shooting direction or shooting position of the imaging device 1 after acquiring the LR image 201.

[0087] The image processing unit 102 then generates an HR image 220 using the LR image 201 and a fused image 210 obtained by fusing at least two REF images. For example, the image processing unit 102 inputs the LR image 201 and the fused image 210 into the model 161 and uses the output of the model 161 as the HR image 220.

[0088] Model 161 is, for example, a convolutional neural network (CNN) model, whose pre-trained parameters are stored in memory 16. It should be noted that model 161 can be any machine learning model other than a CNN model, or any image generation model other than a machine learning model. Any other image generation model can extract image features from LR image 201 and fused image 210 and perform matching based on these features, such as positioning and / or deformation, to obtain an HR image 220 corresponding to LR image 201. Image feature extraction can include edge detection, face detection, and pupil detection, among others.

[0089] The image processing unit 102 outputs and stores the generated HR image 220 in the memory 16. Furthermore, the display control unit 104 outputs (displays) the generated HR image 220 on the display 17 (S10).

[0090] As described above, the imaging device 1 of this embodiment masks the uncaptured area 205 of the LR image 201 that lacks a corresponding REF image, and displays an auxiliary image with a border indicating the area 203 corresponding to the current capture range of the camera module 112. Furthermore, when a new REF image is stored, the imaging device 1 updates the auxiliary image based on the capture range of the newly stored REF image. More specifically, the imaging device 1 generates an auxiliary image with a border shifted to reflect the capture range of the newly stored REF image and the mask removed.

[0091] With this configuration, the user can easily acquire at least two REF images to be fused into the LR image 201 by moving the imaging device 1, thereby removing the border and mask of the region 203. In other words, the imaging device 1 of this embodiment can provide appropriate assistance for acquiring multiple captured images for image synthesis.

[0092] Second embodiment

[0093] The following mainly describes the differences from the first embodiment, and any duplicate contents are omitted as appropriate. Figure 7 FIG. 1 is a diagram showing an example of a display screen of a shooting assistance process according to the present embodiment.

[0094] The imaging control unit 101 of the second embodiment responds to a user's shooting instruction, acquires an LR image 201 through the camera module 111, and acquires a REF image through the camera module 112. Subsequently, the imaging control unit 101 acquires at least one REF image through the camera module 112, and sequentially acquires LR images 201, for example, at a predetermined frame rate. The LR image acquired in response to the user's shooting instruction is an example of a first captured image. In addition, the REF image acquired through the camera module 112 is an example of a second captured image. In addition, the LR image 201 acquired at a predetermined frame rate is an example of a third captured image.

[0095] The image processing unit 102 identifies a second region based on the photographing range of the LR image 201 acquired in response to the user's photographing instruction. The second region of this embodiment corresponds to a portion where the REF image is not acquired in the photographing range of the LR image 201 acquired in response to the user's photographing instruction.

[0096] In addition, for each LR image 201 acquired sequentially at a predetermined frame rate, the image processing unit 102 identifies a first region. According to this embodiment, region 203 is a portion of the shooting range of each LR image 201 acquired sequentially at a predetermined frame rate, which portion corresponds to the shooting range of the camera module 112 when acquiring the latest REF image 202. For example, the camera module 11 is fixed to the imaging device 1 and its shooting direction is fixed. Therefore, when the shooting range of the LR image 201 changes, the shooting range of the REF image also changes accordingly. Therefore, as shown in display screens 420, 430, and 440, the position of region 203 relative to the LR image 201 is fixed or substantially fixed.

[0097] Furthermore, for each LR image 201 sequentially acquired at a predetermined frame rate, the image processing unit 102 recognizes a region 203 based on the photographing range of the LR image 201 acquired in response to a user's photographing instruction.

[0098] The display control unit 104 displays a display screen 410 including a preview display of the LR image 201 on the display 17. Furthermore, after acquiring the LR image and the REF image in response to a shooting instruction from the user, the display control unit 104 displays a display screen 420 including an auxiliary image indicating the area 203 and the area 205 of the LR image 201 on the display 17.

[0099] Subsequently, the display control unit 104 previews the LR images 201 acquired sequentially at a predetermined frame rate, and simultaneously displays a display screen 430 and a display screen 440 on the display 17 containing auxiliary images indicating areas 203 and 205 corresponding to the previewed LR images 201.

[0100] As described above, the imaging device 1 of this embodiment presents the user with an auxiliary image comprising area 203 and area 205. Area 203 corresponds to the captured range of the most recent REF image, and area 205 represents a portion of the captured range of the LR image 201 acquired in response to the user's capture instruction, for which the REF image has not yet been acquired. This configuration allows the area 205, for which the REF image must be acquired for synthesizing the HR image 220, to be displayed in a distorted manner based on the previewed LR image 201. This configuration achieves the same effects as the previously described embodiment. Furthermore, the previewed LR image 201 changes as the position or orientation of the imaging device 1 changes, allowing the user to intuitively understand how to acquire the area 205 for which the REF image has not yet been acquired.

[0101] Third embodiment

[0102] The following mainly describes the differences from the first embodiment, and any duplicate contents are omitted as appropriate. Figure 8 is a diagram illustrating an example of the hardware configuration of the imaging system 5 according to the present embodiment.

[0103] like Figure 8 As shown, the imaging system 5 of this embodiment includes an imaging device 1 and a server 3. The imaging device 1 also includes a global navigation satellite system (GNSS) interface 21, a communication interface 22, a codec 23, a speaker 24, and a microphone 25. The GNSS interface 21, the communication interface 22, and the codec 23 are connected via signal lines (e.g., a bus 19) so that they can communicate with an image signal processor (ISP) 12, a processing circuit 15, and the like.

[0104] The GNSS interface 21 is a circuit for acquiring information from GNSS satellites, such as global positioning system (GPS) satellites. The communication interface 22 is a circuit for performing wired or wireless communication with another device, such as a server 3. It should be noted that the communication interface 22 can appropriately adopt a communication circuit that supports various communication standards. The codec 23 is a circuit connected to the communication interface 22, the speaker 24 and the microphone 25, and is used to encode and decode various voice data, for example, when the imaging device 1 operates as a smartphone. The speaker 24 outputs the sound data decoded by the codec 23. The microphone 25 receives sound input, such as the user's voice.

[0105] For example, the image processing unit 102 performs an alignment process based on the output of the GNSS interface 21 and identifies the areas 203 and 205. In addition, the storage control unit 103 may identify the direction and magnitude of the shooting range movement based on the output of the GNSS interface 21, for example.

[0106] This configuration improves the accuracy of identifying each area 203 and area 205, or the accuracy of determining the REF image to be stored, thereby providing the user with a more appropriate auxiliary image. It should be noted that the technology in this embodiment is applicable to the above embodiments.

[0107] Fourth embodiment

[0108] The following mainly describes the differences from the first embodiment, and any duplicate content will be omitted as appropriate.

[0109] In the imaging device 1 of this embodiment, the imaging unit 10 includes a camera module 11. In response to a user's shooting instruction, the imaging control unit 101 acquires a LR image 201 via the camera module 111. After acquiring the LR image 201, the imaging control unit 101 acquires at least one REF image via the camera module 111. Furthermore, after acquiring the LR image 201, the display control unit 104 displays a message on the display 17, prompting the user to change their position in the shooting direction to get closer to the subject.

[0110] Then, the storage control unit 103 stores the LR image 201 having a finer texture than the LR image 201 as a REF image in the database of the memory 16. For example, the storage control unit 103 updates the database with the LR image 201 having a smaller field of view (FOV) than the LR image 201 acquired in response to the shooting instruction or the REF image in the database, i.e., the acquired LR image 201. The LR image 201 acquired in response to the shooting instruction is an example of a first captured image. Furthermore, the LR image 201 having a finer texture than the acquired LR image 201, i.e., the REF image, is an example of a second captured image.

[0111] The image processing unit 102 recognizes the shooting range of the latest LR image 201 used as the REF image within the shooting range of the LR image 201 acquired in response to the user's shooting instruction as an area 203. In addition, the image processing unit 102 recognizes the shooting range of the LR image 201 that was not acquired as the REF image within the shooting range of the LR image 201 acquired in response to the user's shooting instruction as an area 205.

[0112] As described above, the imaging device 1 of this embodiment presents an auxiliary image to the user, used to fuse at least one REF image into the LR image 201 acquired in response to the user's capture instruction. The at least one REF image is acquired after the user moves the imaging device 1 close to the subject. Even if the imaging device 1 is equipped with only a single camera module 11, the same effects as those of the above-described embodiments can be achieved. It should be noted that the technical solutions of this embodiment are also applicable to the aforementioned embodiments.

[0113] Other application examples

[0114] It should be noted that while the above embodiments have been described using examples of improving image quality through multi-camera synthesis and by using images closer to the subject for synthesis, the technology is not limited to these processes. The technology according to the above embodiments can also be applied to blur generation processing, high dynamic range (HDR) processing, and noise reduction processing.

[0115] For example, the imaging device 1 presents an auxiliary image to the user to obtain a reference (REF) image, which is then stereoscopically fused with the left and right (LR) images acquired in response to the user's capture instruction. In this case, the imaging device 1 acquires an image captured at a different location than the LR image as a reference (REF) image. Furthermore, the imaging device 1 treats areas where the alignment process failed as areas where no REF image exists.

[0116] For example, the imaging device 1 presents an auxiliary image to the user to obtain a reference image (REF). This REF image is then HDR-fused with the left-right image acquired in response to the user's shooting instruction. In this case, the imaging device 1 acquires an image with a different exposure than the left-right image as the REF image. Furthermore, the imaging device 1 treats areas where alignment failed as areas where no REF image exists.

[0117] As described above, the LR image of the above embodiment refers to any captured image that responds to the user's shooting instruction and is used for synthesis. In addition, the REF image of the above embodiment can be any at least one captured image that is fused with the LR image and has a more detailed texture than the LR image. Therefore, the REF image can appropriately adopt various captured images, including images obtained by the camera module 11 that can obtain higher quality images than the LR image, images obtained closer to the subject than the LR image, images obtained under different shooting conditions, such as different exposure from the LR image, etc., so as to make the image clearer.

[0118] It should be noted that the technical solutions of the above embodiments can be appropriately combined as needed.

[0119] Note that part or all of the processing performed by the imaging device 1 according to the present embodiment may be realized by software.

[0120] According to the present embodiment, the program executed by the computer of the imaging device 1 is recorded and provided in a computer-readable non-volatile storage medium (computer program product), such as a flash memory (semiconductor memory) such as a USB flash drive, a solid-state drive (SSD), or a hard disk drive (HDD), and the like, and the file format is an installable or executable format.

[0121] Furthermore, according to this embodiment, the program executed by the imaging device 1 may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. Furthermore, according to this embodiment, the program executed by the imaging device 1 may be provided or distributed directly via a network such as the Internet.

[0122] Furthermore, according to the present embodiment, the program executed by the imaging device 1 may also be integrated into a ROM or the like in advance and provided.

[0123] According to at least one of the foregoing embodiments, appropriate assistance can be provided for acquiring a plurality of captured images for image synthesis.

[0124] Although certain embodiments have been described, these embodiments are presented only as examples and are not intended to limit the scope of this application. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes in form may be made to the embodiments described herein without departing from the spirit of this application. The accompanying claims and their equivalents are intended to cover such forms or modifications as fall within the scope and spirit of this application.

[0125] Notes

[0126] (1) An imaging device comprising:

[0127] At least one camera module;

[0128] an imaging control unit, configured to respond to a user's shooting instruction, acquire a first captured image through at least one of the camera modules, and after acquiring the first captured image, acquire at least one second captured image through at least one of the camera modules;

[0129] an image processing unit configured to identify a first area corresponding to a shooting range of a latest second captured image, and a second area that is a portion of the shooting range of the first captured image and does not overlap with a shooting range of at least one second captured image; and

[0130] The display control unit is configured to output auxiliary information indicating the first area and the second area.

[0131] (2) According to the imaging device of the above-mentioned item (2), the image processing unit fuses at least one of the second captured images into the first captured image and generates a fused image corresponding to a capturing range of the first captured image.

[0132] (3) According to the imaging device of the above-mentioned item (1) or (2), the display control unit outputs the first captured image on which the auxiliary information is superimposed.

[0133] (4) According to the imaging device of the above-mentioned item (3), the auxiliary information is information for displaying a frame indicating the first area and a mask indicating the second area included in the first captured image in the first captured image.

[0134] (5) According to the imaging device of item (1) or (2) above, the imaging control unit acquires the third captured image at a predetermined frame rate by the camera module that acquires the first captured image;

[0135] The image processing unit recognizes the first area and the second area in the third captured image, and

[0136] The display control unit outputs the third captured image on which the auxiliary information is superimposed.

[0137] (6) According to the imaging device of the above-mentioned item (5), the auxiliary information is information for displaying a frame indicating the first area and a mask indicating the second area included in the third captured image in the third captured image.

[0138] (7) The imaging device according to any one of the above items (1) to (6), further comprising:

[0139] a memory storing a database, wherein the database stores the second captured image; and

[0140] A storage control unit is configured to update the database with a newly acquired second captured image when the newly acquired second captured image by at least one of the camera modules has a texture with more detailed texture than the second captured image stored in the database.

[0141] (8) The imaging device according to the above item (4), wherein the image processing unit updates the first area and the second area every time the database is updated; and

[0142] The display control unit outputs the auxiliary information according to the updated first area and the second area.

[0143] (9) According to any one of items (1) to (8) above, the imaging device, at least one of the camera modules includes a first camera module for acquiring the first captured image and at least one second camera module for acquiring the second captured image,

[0144] The image captured by at least one of the second camera modules has a more detailed texture than the image captured by the first camera module.

[0145] (10) According to the imaging device of any one of items (1) to (8) above, at least one of the camera modules is a single camera module, which is used to obtain the first captured image and the second captured image, and the second captured image has a more detailed texture than the first capture module.

[0146] (11) According to any one of the above-mentioned items (1) to (10), the auxiliary information further includes message information for prompting the user to change the shooting direction or shooting position.

[0147] (12) According to the imaging device of the above item (11), the auxiliary information further includes message information for prompting the user to change the position so as to get closer to the photographic subject in the photographing direction.

[0148] (13) An imaging device control method performed by an imaging device, wherein the imaging device includes at least one camera module, the method comprising:

[0149] In response to a user's shooting instruction, acquiring a first shot image through at least one of the camera modules;

[0150] After acquiring the first captured image, acquiring at least one second captured image through at least one of the camera modules;

[0151] identifying a first area corresponding to a photographing range of the latest second photographed image, and a second area that is a portion of the photographing range of the first photographed image and does not overlap with a photographing range of at least one second photographed image; and

[0152] Auxiliary information indicating the first area and the second area is output.

[0153] (14) A program executed by an imaging device, the imaging device including at least one camera module, the program causing the imaging device to execute:

[0154] In response to a user's shooting instruction, acquiring a first shot image through at least one of the camera modules;

[0155] After acquiring the first captured image, acquiring at least one second captured image through at least one of the camera modules;

[0156] identifying a first area corresponding to a photographing range of the latest second photographed image, and a second area that is a portion of the photographing range of the first photographed image and does not overlap with a photographing range of at least one second photographed image; and

[0157] Auxiliary information indicating the first area and the second area is output.

[0158] (15) A computer program product storing the program of the above-mentioned item (14), wherein the program is executed by a computer of the imaging device.

[0159] Description of reference numerals:

[0160] 1: Imaging device; 3: Server; 5: Imaging system; 10: Imaging unit; 11, 111, 112, … 11n: Camera module; 13: Input interface; 131: Operation button; 14: Inertial measurement unit (IMU); 15: Processing circuit; 16: Memory; 17: Display; 19: Bus; 101: Imaging control unit; 102: Image processing unit; 103: Storage control unit; 104: Display control unit; 21: GNSS interface; 22: Communication interface; 23: Codec; 24: Speaker; 25: Microphone.

Claims

1. An imaging device comprising: At least one camera module; an imaging control unit, configured to respond to a user's shooting instruction, acquire a first captured image through at least one of the camera modules, and after acquiring the first captured image, acquire at least one second captured image through at least one of the camera modules; an image processing unit configured to identify a first area corresponding to a shooting range of a latest second captured image, and a second area that is a portion of the shooting range of the first captured image and does not overlap with a shooting range of at least one second captured image; as well as The display control unit is configured to output auxiliary information indicating the first area and the second area.

2. The imaging device according to claim 1, wherein The image processing unit fuses at least one second captured image into the first captured image and generates a fused image corresponding to a capturing range of the first captured image.

3. The imaging device according to claim 1 or 2, characterized in that The display control unit outputs the first captured image on which the auxiliary information is superimposed.

4. The imaging device according to claim 3, wherein The auxiliary information is information for displaying a frame indicating the first area and a mask indicating the second area included in the first photographed image in the first photographed image.

5. The imaging device according to claim 1 or 2, wherein: The imaging control unit acquires a third captured image at a predetermined frame rate by using the camera module that acquires the first captured image; The image processing unit recognizes the first area and the second area in the third captured image, and The display control unit outputs the third captured image on which the auxiliary information is superimposed.

6. The imaging device according to claim 5, wherein The auxiliary information is information for displaying a frame indicating the first region and a mask indicating the second region included in the third photographed image in the third photographed image.

7. The imaging device according to claim 1 or 2, further comprising: a memory for storing a database, wherein the database stores the second captured image; as well as A storage control unit is configured to update the database with a newly acquired second captured image when the newly acquired second captured image by at least one of the camera modules has a texture with more detailed texture than the second captured image stored in the database.

8. The imaging device according to claim 7, wherein Whenever the database is updated, the image processing unit updates the first area and the second area; and The display control unit outputs the auxiliary information according to the updated first area and the second area.

9. The imaging device according to claim 1 or 2, characterized in that: At least one of the camera modules includes a first camera module for acquiring the first captured image and at least one second camera module for acquiring the second captured image; The image captured by at least one of the second camera modules has a more detailed texture than the image captured by the first camera module.

10. The imaging device according to claim 1 or 2, characterized in that At least one of the camera modules is a single camera module, used to obtain the first captured image and the second captured image, and the second captured image has a more detailed texture than the first capture module.

11. The imaging device according to claim 1 or 2, characterized in that: The auxiliary information also includes message information for prompting the user to change the shooting direction or shooting position.

12. The imaging device according to claim 11, wherein The auxiliary information also includes message information for prompting the user to change the position so as to be closer to the photographed object in the photographing direction.

13. A method for controlling an imaging device, performed by an imaging device, the imaging device comprising at least one camera module, the method comprising: In response to a user's shooting instruction, acquiring a first shot image through at least one of the camera modules; After acquiring the first captured image, acquiring at least one second captured image through at least one of the camera modules; identifying a first area corresponding to a shooting range of a latest second captured image, and a second area that is a portion of the shooting range of the first captured image and does not overlap with a shooting range of at least one second captured image; as well as Auxiliary information indicating the first area and the second area is output.

14. A computer program product storing a program to be executed by a computer of an imaging device including at least one camera module, the program causing the computer to perform the following operations: In response to a user's shooting instruction, acquiring a first shot image through at least one of the camera modules; After acquiring the first captured image, acquiring at least one second captured image through at least one of the camera modules; identifying a first area corresponding to a shooting range of a latest second captured image, and a second area that is a portion of the shooting range of the first captured image and does not overlap with a shooting range of at least one second captured image; as well as Auxiliary information indicating the first area and the second area is output.

Images