Information processing device, information processing method, and information processing program

Abstract

To simplify a framing work.SOLUTION: An information processing device 10 comprises: a determination unit 12 which, on the basis of imaging related information that is related to imaging by a first imaging device, determines alignment information that is used to align a first image captured by the first imaging device and a second image captured by a second imaging device having a wider field angle than the field angle of the first imaging device; and an output unit 13 that outputs the alignment information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] In shooting using a camera, as an example, various finders such as an optical finder and an electronic viewfinder are used to confirm the composition.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above-mentioned finder, trial and error occurs in which the line of sight is alternately moved between the inside and outside of the finder until the composition as intended by the photographer fits within the frame of the camera, so that the framing work becomes complicated.

[0005] Therefore, an object of the present disclosure is to provide an information processing apparatus, an information processing method, and an information processing program that can simplify the framing work.

Means for Solving the Problems

[0006] In order to solve the above problems, an information processing apparatus according to one aspect of the present disclosure includes a determination unit that determines alignment information used for alignment between a first image captured by the first imaging device and a second image captured by a second imaging device having an angle of view wider than the angle of view of the first imaging device, based on imaging-related information related to imaging by the first imaging device, and an output unit that outputs the alignment information.

Brief Description of the Drawings

[0007] [Figure 1] This is a diagram showing an example of the system configuration according to the first embodiment. [Figure 2] This is a block diagram showing an example of the functional configuration of an information processing device according to the first embodiment. [Figure 3] This is a flowchart showing the procedure for information output processing according to the first embodiment. [Figure 4] This is a block diagram showing an example of the functional configuration of an information processing device according to the second embodiment. [Figure 5] This figure shows an example of depth distribution information. [Figure 6] This is a flowchart showing the procedure for information output processing according to the second embodiment. [Figure 7] This is a block diagram showing an example of the functional configuration of an information processing device according to the third embodiment. [Figure 8] This figure shows an example of the correspondence between focal length and angle of view. [Figure 9] This is a flowchart showing the procedure for information output processing according to the third embodiment. [Figure 10] This is a hardware configuration diagram showing an example of a computer. [Modes for carrying out the invention]

[0008] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0009] Furthermore, this disclosure will be explained in the order of the items shown below. 1. First Embodiment 1-1. Example System Configuration 1-2. Examples of Use Cases 1-3. Example of Functional Configuration of the First Imaging Device 1-3-1. Image sensor 1-3-2. Digital Signal Processing Unit 1-3-3.Display section 1-4. Example of Functional Configuration of the Second Imaging Device 1-4-1. Image pickup device 1-4-2. Digital signal processing unit 1-4-3. Alignment unit 1-4-4. Display unit 1-5. One aspect of the problem 1-6. One aspect of the approach to solving the problem 1-7. Functional configuration example of the information processing apparatus 1-7-1. Acquisition unit 1-7-2. Decision unit 1-7-3. Output unit 1-8. Processing procedure of the information processing apparatus 1-9. One aspect of the effect 2. Second embodiment 2-1. Functional configuration example of the information processing apparatus 2-2. Decision unit 2-2-1. Determination unit 2-2-2. Selection unit 2-3. Processing procedure of the information processing apparatus 2-4. One aspect of the effect 3. Third embodiment 3-1. Functional configuration example of the information processing apparatus 3-2. Decision unit 3-2-1. Calculation unit 3-2-2. Generation unit 3-3. Processing procedure of the information processing apparatus 3-4. One aspect of the effect 4. Variation 4-1. Combination between embodiments 4-2. Execution entity of alignment 4-3. Other variations 5. Hardware configuration

[0010] <<1. First embodiment>> <1-1. System configuration example> FIG. 1 is a diagram showing a configuration example of a system according to the first embodiment. The system shown in FIG. 1 provides a shooting support function for displaying a marker M indicating the position of a frame captured by a first imaging device 1 on an image 50 captured by a second imaging device 5 from the aspect of simplifying the framing operation.

[0011] As shown in Figure 1, the system may include a first imaging device 1 and a second imaging device 5. These first imaging device 1 and second imaging device 5 are connected in a communicative manner. As merely an example, the first imaging device 1 and the second imaging device 5 can be connected via a USB (Universal Serial Bus) cable or the like. Furthermore, bidirectional communication is not necessarily required between the first imaging device 1 and the second imaging device 5; they may also be connected via an HDMI (High-Definition Multimedia Interface) cable or the like.

[0012] One aspect of this design is that the first imaging device 1 and the second imaging device 5 do not necessarily have to have the same field of view. For example, the first imaging device 1 may be equipped with a telephoto lens with a longer focal length than the lens used by the second imaging device 5, while the second imaging device 5 may be equipped with a wide-angle lens with a wider field of view than the lens used by the first imaging device 1.

[0013] In other respects, the optical axes of the first imaging device 1 and the second imaging device 5 may or may not coincide. Note that in Figure 1, the first imaging device 1 and the second imaging device 5 are shown as separate devices, but the first imaging device 1 and the second imaging device 5 may also be integrated into two separate devices.

[0014] Although Figure 1 shows an example in which the first imaging device 1 and the second imaging device 5 are connected by a wire, the system is not limited to this, and the first imaging device 1 and the second imaging device 5 can also be connected by short-range wireless communication or wireless communication.

[0015] <1-2. Examples of Use Cases> The first imaging device 1 and the second imaging device 5 do not necessarily have to be electronic devices dedicated solely to imaging. For example, the first imaging device 1 and the second imaging device 5 may have functions other than imaging.

[0016] As just one example, one use case involves overlaying a marker M indicating the position of a frame captured by a digital camera onto an image 50 captured by a smartphone or tablet device. In this case, the first imaging device 1 can be implemented as a digital camera, and the second device 5 can be implemented as a smartphone or tablet device.

[0017] Another example of a use case involves overlaying a marker M indicating the position of a frame captured by a digital camera, such as an interchangeable lens camera with a stabilizer, onto an image 50 captured by AR (Augmented Reality) glasses. In this case, the first imaging device 1 can be implemented as a digital camera, and the second imaging device 5 can be implemented as AR glasses.

[0018] Note that Figure 1 shows, as an example, the marking M of the frame in which the first imaging device 1 captures an image; however, the shape and size of the marking M can be arbitrary. For example, it could be a pointer, figure, symbol, etc., indicating the position of the frame in which the first imaging device 1 captures an image.

[0019] <1-3. Example of Functional Configuration of the First Imaging Device> Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 10 according to the first embodiment. As shown in Figure 2, the first imaging device 1 includes an image sensor 1A, a digital signal processing unit 1B, a display unit 1C, and an information processing device 10.

[0020] Here, Figure 2 shows an example in which the information processing device 10 is incorporated into the first imaging device 1, but this is merely one example. For example, the information processing device 10 may be incorporated into the second imaging device 5, or it may be implemented as a third device different from the first imaging device 1 and the second imaging device 5. The information processing device 10 will be explained after the example functional configurations of the first imaging device 1 and the second imaging device 5 are described.

[0021] <1-3-1. Image sensor 1A> The image sensor 1A can employ image sensors such as CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). As just one example, light collected through the optical system of the first imaging device 1, for example, a telephoto lens, is photoelectrically converted by the image sensor 1A. The RAW image obtained by such photoelectric conversion is input to the digital signal processing unit 1B.

[0022] Here, an image sensor 1A can incorporate an image plane phase difference sensor (not shown). For example, the distance to the subject, i.e., the depth, can be measured from the phase difference signal output from the image plane phase difference sensor. The depth measured for each image plane phase difference sensor in this way can be output as a depth map to the digital signal processing unit 1B.

[0023] While an image plane phase difference sensor is used as an example here, a phase difference sensor may be mounted on the first imaging device 1 instead of an image plane phase difference sensor, or both an image plane phase difference sensor and a phase difference sensor may be mounted. Furthermore, while an image plane phase difference sensor and a phase difference sensor are given as examples here, this does not preclude the first imaging device 1 from being equipped with other depth sensors.

[0024] <1-3-2. Digital Signal Processing Unit 1B> The digital signal processing unit 1B is a functional unit that performs digital signal processing. In one embodiment, the digital signal processing unit 1B can be implemented by hardware such as a DSP (Digital Signal Processor). For example, one example of the above digital signal processing is the process of converting a RAW image to an image in a predetermined format, such as a YC image, known as RAW development, as well as processes such as white balance adjustment and color difference correction.

[0025] In this example, the digital signal processing unit 1B is implemented by hardware, but it may also be implemented by software such as a RAW development engine being executed by a processor.

[0026] <1-3-3.Display section 1C> The display unit 1C is a functional unit that displays various types of information. In one embodiment, the display unit 1C can be implemented by arranging a liquid crystal display or an organic EL (electroluminescence) display on the back of the housing of the first imaging device 1. For example, the display unit 1C displays the YC image in real time each time the YC image is output by the digital signal processing unit 1B. Hereinafter, in order to distinguish the labels of the YC images output by the digital signal processing unit 1B from the YC images output by the digital signal processing unit 5B described later, the former may be referred to as the "telephoto YC image" and the latter as the "wide-angle YC image". This enables a live view function for the telephoto YC image.

[0027] Furthermore, the display unit 1C can be integrated with an input unit (not shown) to be implemented as a touch panel. Also, as mentioned above, since the display unit 5D of the second imaging device 5 indicates the position of the frame in which the first imaging device 1 captures an image, the display unit 1C does not necessarily have to be provided on the first imaging device 1.

[0028] <1-4. Example of Functional Configuration of the Second Imaging Device 5> As shown in Figure 2, the second imaging device 5 includes an image sensor 5A, a digital signal processing unit 5B, an alignment unit 5C, and a display unit 5D.

[0029] <1-4-1. Image sensor 5A> The image sensor 5A can employ an image sensor such as a CCD or CMOS. As just one example, light focused through the optical system of the second imaging device 5, for example, a wide-angle lens, is photoelectrically converted by the image sensor 5A. The RAW image obtained by this photoelectric conversion is input to the digital signal processing unit 5B.

[0030] In this image sensor 5A, as with the image sensor 1A described above, an image plane phase difference sensor (not shown) can be incorporated. The depth measured by each image plane phase difference sensor can be output to the digital signal processing unit 5B as a depth map. Here, an image plane phase difference sensor is given as an example, but a phase difference sensor may be mounted on the second imaging device 5 instead of an image plane phase difference sensor, or both an image plane phase difference sensor and a phase difference sensor may be mounted. Also, although an image plane phase difference sensor and a phase difference sensor are given as examples here, this does not prevent other depth sensors from being provided on the second imaging device 5.

[0031] <1-4-2. Digital Signal Processing Unit 5B> The digital signal processing unit 5B is a functional unit that performs digital signal processing. In one embodiment, the digital signal processing unit 5B is virtually implemented by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). For example, the processor reads programs such as the OS (Operating System) and software such as a RAW development engine from storage (not shown). Then, the processor executes the RAW development engine to deploy a process corresponding to the digital signal processing unit 5B on memory such as RAM (Random Access Memory). As a result, the digital signal processing unit 5B is virtually implemented as a process. Here, a CPU and an MPU are given as examples of processors, but the digital signal processing unit 5B may be implemented by any processor, regardless of whether it is a general-purpose or specialized type. Furthermore, the digital signal processing unit 5B may be implemented by hardwired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0032] The digital signal processing unit 5B can perform the same functions as the digital signal processing unit 1B described above. For example, as an example of the digital signal processing described above, the digital signal processing unit 5B can perform processes such as converting a RAW image to an image in a predetermined format, such as a YC image, also known as RAW development, as well as white balance adjustment and color difference correction.

[0033] In this example, we have shown that the RAW development engine and other software are implemented by a processor, but the digital signal processing unit 1B described above may also be implemented by hardware.

[0034] <1-4-3. Alignment section 5C> The alignment unit 5C is a processing unit that aligns the image captured by the first imaging device 1 with the image captured by the second imaging device 5. In one embodiment, the alignment unit 5C, like the digital signal processing unit 5B, can be virtually implemented by a processor such as a CPU or MPU. For example, the processor reads an image processing program that performs alignment of multiple images. Then, by executing the image processing program, the processor deploys a process corresponding to the alignment unit 5C in memory such as RAM. As a result, the alignment unit 5C is virtually implemented as a process.

[0035] Here, the above-mentioned alignment refers to the general process of registration, which involves applying transformation matrices such as translation, rotation, or deformation to one image to match the other image. One aspect of registration is the search for a transformation matrix that maximizes the similarity between the two images. For example, similarity can be measured using SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), or correlation coefficients. Furthermore, when applying transformation matrices, feature points between the two images can also be matched.

[0036] This registration process enables alignment between the image captured by the first imaging device 1, for example, the telephoto YC image output by the digital signal processing unit 1B, and the image captured by the second imaging device 5, for example, the wide-angle YC image output by the digital signal processing unit 5B. Furthermore, as merely an example, we present here an example in which a transformation matrix for aligning the telephoto YC image with the wide-angle YC image is obtained through the above registration process.

[0037] As merely one example, the alignment unit 5C can superimpose a telephoto YC image onto a wide-angle YC image. In this case, if the optical axes of the first imaging device 1 and the second imaging device 5 are the same, the frame of the telephoto YC image obtained by alignment will remain rectangular, as shown in the example in Figure 1. However, if the optical axes are not the same, the frame of the telephoto YC image obtained by alignment may not be rectangular and may have a distorted shape.

[0038] From this perspective, the alignment unit 5C can superimpose the frame position of the telephoto YC image, for example, the center position, as a marker M, onto the wide-angle YC image. Hereinafter, the wide-angle YC image on which the frame position marker M of the telephoto YC image is superimposed may be referred to as a "wide-angle image with telephoto frame marker". The wide-angle image with telephoto frame marker obtained in this way is output to the display unit 5D.

[0039] <1-4-4.Display section 5D> The display unit 5D is a functional unit that displays various types of information. In one embodiment, the display unit 5D can be implemented using a liquid crystal display or an organic EL display, or it can be implemented as a touch panel by integrating it with an input unit (not shown). For example, the display unit 5D displays the wide-angle image with the telephoto frame indicator in real time each time the wide-angle image with the telephoto frame indicator is output from the alignment unit 5C.

[0040] <1-5. One aspect of the problem> In Embodiment 1 of this disclosure, a wide-angle image with a telephoto frame indicator is displayed on the display unit 5D as the above-mentioned shooting support function. This display allows the user to view the indicator M of the position of the telephoto YC image frame on the wide-angle YC image. Therefore, it eliminates the need for trial and error of alternating between moving the gaze between the inside and outside of the viewfinder, resulting in a simplification of the framing process.

[0041] While there are these advantages, displaying wide-angle images with telephoto frame markings in real time can be bottlenecked by image processing such as aligning the telephoto YC image with the wide-angle YC image. This can cause delays in displaying the frame position of the telephoto YC image superimposed on the wide-angle YC image. When such delays occur, there will be a difference between the displayed frame position of the telephoto YC image and the actual frame position of the telephoto YC image, which may prevent you from capturing the intended composition.

[0042] While various techniques have been proposed for image alignment, they all approach the alignment and registration algorithms themselves. Such algorithmic improvements are merely methods for aligning two images. Therefore, they rely solely on two images for alignment, lacking the perspective of obtaining information that can contribute to alignment during the capture of either image, and thus are not used for alignment.

[0043] <1-6. One aspect of problem-solving approaches> Therefore, the information processing device 10 according to the embodiment of this disclosure provides an information output function that determines and outputs alignment information used for aligning images between the first imaging device 1 and the second imaging device 5 based on imaging-related information related to imaging by the first imaging device 1.

[0044] In other words, the information processing device 10 according to the embodiment of this disclosure outputs alignment information that can contribute to reducing the amount of processing required when aligning with the image captured by the second imaging device 5, from the perspective unique to the imaging performed by the first imaging device 1.

[0045] Therefore, according to the information processing device 10 of the embodiment of this disclosure, the alignment of the telephoto YC image and the wide-angle YC image can be accelerated, thereby improving the real-time display of the wide-angle image with telephoto frame markings.

[0046] <1-7. Example of Functional Configuration of Information Processing Device 10> As shown in Figure 2, the information processing device 10 includes an acquisition unit 11, a determination unit 12, and an output unit 13.

[0047] <1-7-1. Acquisition part 11> The acquisition unit 11 is a processing unit that acquires imaging-related information related to imaging performed by the first imaging device 1 and the second imaging device 5. Here, "imaging-related information" may include not only the images themselves captured by the first imaging device 1 and the second imaging device 5, as illustrated below, but also all information related to imaging. In this example, imaging-related information from both the first imaging device 1 and the second imaging device 5 is acquired, but it is also possible to acquire imaging-related information from either one of them.

[0048] i) Image size and image sensor size of image sensor 1A and image sensor 5A (b) Focal length of the first imaging device 1 and the second imaging device 5 (h) Parameters for imaging of the first imaging device 1 and the second imaging device 5 (ii) Information obtainable from telephoto YC images and wide-angle YC images (e) Information on the bandwidth and stability of data transmission between the first imaging device 1 and the second imaging device 5. (h) Information on the physical positional relationship between the first imaging device 1 and the second imaging device 5, and the amount of optical axis misalignment. (t) Processing capacity of the arithmetic units installed in the first imaging device 1 and the second imaging device 5

[0049] The imaging-related information described in (a) to (g) above can be acquired on a frame-by-frame basis by the first imaging device 1 and the second imaging device 5. For example, to supplement (c) above, the above shooting parameters may include, as an example, exposure, white balance, shutter speed, ISO value, F-number, etc. To supplement (d) above, the information that can be acquired may include, as an example, object recognition result information, depth map, proper exposure, light source information, flicker information, etc.

[0050] Such imaging-related information can be obtained from both the first imaging device 1 and the second imaging device 5. For example, imaging-related information concerning the first imaging device 1 can be obtained from the digital signal processing unit 1B or from the driver IC (Integrated Circuit) that drives the optical system of the first imaging device 1. Similarly, imaging-related information concerning the second imaging device 5 can be obtained from the digital signal processing unit 5B or from the driver IC that drives the optical system of the second imaging device 5.

[0051] <1-7-2. Decision Section 12> The determination unit 12 is a processing unit that determines alignment information based on imaging-related information acquired by the acquisition unit 11. One aspect of this is that the determination unit 12 determines the alignment information that the alignment unit 5C of the second imaging device 5 uses for alignment with the wide-angle YC image. Such alignment information can be determined from the viewpoint of reducing the processing load for alignment with the wide-angle YC image, and consequently reducing the amount of data transmitted from the first imaging device 1 to the second imaging device 5.

[0052] As an example, the determination unit 12 determines at least one of the following, or a combination thereof, as alignment information: depth map, near-focus image, edge information, or YC image. For example, "depth map" refers to a map in which depth is measured by a phase difference sensor, an image plane phase difference sensor, or both of these. Also, "near-focus image" refers to a partial image of the telephoto YC image that corresponds to the vicinity of the subject that has been brought into focus by the focal length adjustment control. For example, the focal length adjustment control may be performed automatically by an AF (AutoFocus) function realized by a phase difference sensor, an image plane phase difference sensor, or both of these, or it may be performed manually by MF (ManualFocus). Also, "edge information" refers to information in which edges have been detected from the telephoto YC image. Also, "telephoto YC image" refers to the telephoto YC image itself output from the digital signal processing unit 1B. These "depth map," "near-focus image," "edge information," and "telephoto YC image" can be changed to the same resolution as the wide-angle YC image when the zoom function of the first imaging device 1 or the second imaging device 5 is in operation.

[0053] <1-7-3. Output section 13> The output unit 13 is a processing unit that outputs the alignment information determined by the determination unit 12. One aspect of this is that the output unit 13 transmits the alignment information determined by the determination unit 12 to the alignment unit 5C of the second imaging device 5.

[0054] <1-8. Processing Procedure of Information Processing Device> Figure 3 is a flowchart showing the information output processing procedure according to the first embodiment. This process can be repeated for each frame in which the first imaging device 1 and the second imaging device 5 capture an image, and this is merely one example.

[0055] As shown in Figure 3, the acquisition unit 11 acquires imaging-related information, such as (a) to (f) above, that is related to imaging performed by the first imaging device 1 and the second imaging device 5 (step S101).

[0056] Next, the determination unit 12 determines the alignment information that the alignment unit 5C of the second imaging device 5 will use to align with the wide-angle YC image, based on the imaging-related information acquired in step S101 (step S102).

[0057] Then, the output unit 13 transmits the alignment information determined in step S102 to the alignment unit 5C of the second imaging device 5 (step S103), and the process ends.

[0058] <1-9. One aspect of the effect> As described above, the information processing device 10 according to the first embodiment determines and outputs alignment information used for aligning images between the first imaging device 1 and the second imaging device 5 based on imaging-related information related to imaging by the first imaging device 1.

[0059] In other words, the information processing device 10 according to the first embodiment outputs alignment information that can contribute to reducing processing load and improving accuracy when aligning with the wide-angle YC image captured by the second imaging device 5, from the perspective unique to the imaging of the first imaging device 1.

[0060] Therefore, according to the information processing device 10 of the first embodiment, the alignment of the telephoto YC image and the wide-angle YC image can be accelerated, making it possible to improve the real-time display of the wide-angle image with telephoto frame markings.

[0061] <<2. Second Embodiment>> In embodiments of this disclosure, an example of an algorithm is given of selecting one or more of the above-mentioned alignment information from "depth map," "near-focus image," "edge information," and "telephoto YC image."

[0062] <2-1. Example of Functional Configuration of Information Processing Device 20> Figure 4 is a block diagram showing an example of the functional configuration of the information processing device 20 according to the second embodiment. As shown in Figure 4, the information processing device 20 differs from the information processing device 10 shown in Figure 2 in that it has a determination unit 21 whose processing content is partially different from that of the determination unit 12.

[0063] <2-2. Decision Section 21> The determination unit 21 shown in Figure 4 differs from the determination unit 12 shown in Figure 2 in that it has a determination unit 21A and a selection unit 21B.

[0064] <2-2-1. Judgment section 21A> The determination unit 21A is a processing unit that determines whether the imaging-related information acquired by the acquisition unit 11 satisfies predetermined conditions.

[0065] In one embodiment, the determination unit 21A acquires depth distribution information for a subject detected from a telephoto YC image acquired from the digital signal processing unit 1B. Here, "subject" refers to any of the following: a subject focused by the AF function or MF function, a subject tracked by the AF function, or a subject obtained as a result of object detection. Then, the determination unit 21A determines whether or not the subject has unevenness based on the depth distribution information of the subject in the telephoto YC image. As merely an example, the determination unit 21A determines whether or not the subject has unevenness as condition 1, based on whether or not the ratio of pixels within a predetermined depth range Th2 to the total number of pixels of the subject is within a predetermined threshold Th1. Such a "depth range Th2" can be set based on the peak of the subject's depth distribution. For example, a certain range before and after the peak of the depth distribution can be set as the depth range Th2. In this case, the range to be included in the depth range Th2 before and after the peak does not necessarily have to be the same and can be changed.

[0066] Figure 5 shows an example of depth distribution information. In the graph shown in Figure 5, the vertical axis represents the number of pixels, and the horizontal axis represents the depth. These vertical and horizontal axes may also be histograms representing the frequency and class of depth. In Figure 5, the area corresponding to the total number of pixels in the subject is shown as a solid white fill, while the area corresponding to the depth range Th2 is shown as hatching. In the example shown in Figure 5, the presence or absence of surface irregularities in the subject can be identified by whether the number of pixels included in the depth range Th2, i.e., the number of pixels in the hatched area, is within a predetermined threshold Th1, for example, 50%, of the total number of pixels in the subject, such as probability or percentage. For example, if it is within the threshold Th1, the subject is identified as having surface irregularities, while if it exceeds the threshold Th1, the subject is identified as not having surface irregularities.

[0067] Then, if the above condition 1 is met, that is, if the ratio of the number of pixels within the depth range Th2 to the total number of pixels of the subject is within the threshold Th1, the determination unit 21A determines the following condition 2. For example, the determination unit 21A determines as condition 2 whether the number of pixels in the telephoto YC image other than the area near the focus point where the focus was achieved by the AF function or MF function exceeds a predetermined threshold Th3 and is within the predetermined threshold Th4. In this case, if the number of pixels in the area other than the area near the focus point exceeds the threshold Th3 and is within the threshold Th4, it is determined that the above condition 2 is met.

[0068] Furthermore, if either condition 1 or condition 2 above is not met, the determination unit 21A determines the following conditions 3 and 4. For example, condition 3 determines whether the transmission bandwidth between the first imaging device 1 and the second imaging device 5 is equal to or greater than a predetermined threshold Th5. Furthermore, if condition 3 is met, i.e., the transmission bandwidth is equal to or greater than the predetermined threshold Th5, the determination unit 21A determines the following condition 4 determines whether the processing capacity of the second imaging device 5, such as the clock frequency of the processor or the number of cores, is equal to or greater than a predetermined threshold Th6.

[0069] <2-2-2. Selection Section 21B> The selection unit 21B is a processing unit that selects one or more of the following as alignment information based on the determination result of the determination unit 21A: "depth map," "image near focus," "edge information," and "telephoto YC image."

[0070] As one aspect, the selection unit 21B selects the depth map as alignment information if the above conditions 1 and 2 are met. Such conditional branching corresponds to the case where the ratio of the number of pixels in the depth range Th2 to the total number of pixels of the subject is within threshold Th1, and the number of pixels in areas other than the vicinity of focus where the amount of blur exceeds threshold Th3 is within threshold Th4.

[0071] The selection of alignment information in this way is based on the following considerations. Specifically, if condition 1 above is met, it is highly likely that sufficient alignment accuracy can be obtained by aligning the telephoto depth map with the wide-angle depth map, without performing alignment between the two YC images, the telephoto YC image and the wide-angle YC image. Furthermore, although the second imaging device 5, which has a shorter focal length than the first imaging device 1, tends to produce less blur compared to the first imaging device 1, if condition 2 above is met, it can be identified that the degree of deviation in blur between the first imaging device 1 and the second imaging device 5 is likely to be sufficient for alignment between depth maps. For these reasons, the depth map is selected as the alignment information. By selecting the depth map as the alignment information in this way, the amount of processing can be reduced compared to when the alignment unit 5C is made to perform alignment between images. Furthermore, since the depth map contains less information than the telephoto YC image or edge information, the transmission delay from the first imaging device 1 to the second imaging device 5 can also be suppressed.

[0072] In other respects, if the selection unit 21B satisfies condition 1 above but does not satisfy condition 2 above, it selects the near-focus image as alignment information. Such conditional branching corresponds to the case where the ratio of the number of pixels in the depth range Th2 to the total number of pixels of the subject is within threshold Th1, and the number of pixels in areas other than the near-focus area where the amount of blur exceeds threshold Th3 is not within threshold Th4.

[0073] The reason for selecting alignment information in this way is that even if condition 1 above is met, if condition 2 above is not met, there is a high probability that the following case will occur. That is, it can be identified that the degree of deviation in the amount of blur between the first imaging device 1 and the second imaging device 5 is likely to exceed the limit that can withstand alignment between depth maps. For this reason, the near-focus image is selected as alignment information. By selecting the near-focus image as alignment information in this way, it is possible to suppress the decrease in alignment accuracy caused by using blurred areas other than the near-focus area for alignment by the alignment unit 5C. Furthermore, since the near-focus image is part of the telephoto YC image, the amount of processing required for alignment can be reduced compared to when the entire telephoto YC image is used for alignment. In addition, since the near-focus image has less information than the telephoto YC image, the transmission delay from the first imaging device 1 to the second imaging device 5 can also be suppressed.

[0074] As a further aspect, the selection unit 21B selects the telephoto YC image and the depth to the subject as alignment information if conditions 3 and 4 above are met. Such conditional branching corresponds to the case where the transmission bandwidth between the first imaging device 1 and the second imaging device 5 is greater than or equal to the threshold Th5, and the processing capacity of the second imaging device 5 is greater than or equal to the threshold Th6.

[0075] The selection of such alignment information is made for the following reasons. Specifically, if both conditions 3 and 4 above are met, transmission delays are less likely to occur when transmitting the telephoto YC image from the first imaging device 1 to the second imaging device 5, and processing delays are less likely to occur when the alignment unit 5C is made to perform alignment of the telephoto YC image and the wide-angle YC image. Furthermore, by including the depth to the subject in the alignment information, the region in which the alignment unit 5C performs alignment with the wide-angle YC image can be narrowed down to the region near that depth, or the weight of the region near that depth can be made greater than the weight of the regions corresponding to other depths, allowing the alignment unit 5C to perform alignment with the wide-angle YC image.

[0076] In other aspects, if the selection unit 21B does not satisfy condition 3 or condition 4 above, it selects edge information and depth to the subject as alignment information. Such conditional branching corresponds to cases where the transmission bandwidth between the first imaging device 1 and the second imaging device 5 is not equal to or greater than the threshold Th5, or where the processing capacity of the second imaging device 5 is not equal to or greater than the threshold Th6.

[0077] The selection of alignment information in this way has the following aspects. Specifically, if either condition 3 or condition 4 above is not met, there is a high probability that a transmission delay will occur when transmitting the telephoto YC image from the first imaging device 1 to the second imaging device 5, or that a processing delay will occur when the alignment unit 5C is made to perform alignment of the telephoto YC image and the wide-angle YC image. In this case, by selecting edge information as alignment information, the amount of processing can be reduced compared to when the alignment unit 5C is made to perform alignment between images. Furthermore, since edge information contains less information than the telephoto YC image, the transmission delay from the first imaging device 1 to the second imaging device 5 can also be suppressed. In addition, by including the depth to the subject in the alignment information, the area in which the alignment unit 5C performs alignment with the wide-angle edge information can be narrowed down to the area near that depth, or the weight of the area near that depth can be made greater than the weight of the area corresponding to other depths, so that the alignment unit 5C performs alignment with the wide-angle edge information.

[0078] Furthermore, if the near-focus image is selected as alignment information, the selection of alignment information can be branched as follows depending on whether conditions 3 and 4 above are met. For example, if conditions 3 and 4 above are met, the near-focus YC image and the depth to the subject can be selected, while if neither condition 3 nor condition 4 above is met, the near-focus edge information and the depth to the subject can be selected. Also, if something other than the depth map is selected as alignment information, for example, if the near-focus image, edge information, or YC image is selected, it corresponds to a case where the subject has little surface irregularity. In this case, depth information with less information than the depth map, such as the depth to the subject, can be selected as further alignment information.

[0079] <2-3. Processing Procedure of Information Processing Device 20> Figure 6 is a flowchart showing the procedure for information output processing according to the second embodiment. This process can be repeated for each frame in which the first imaging device 1 and the second imaging device 5 capture an image, and this is merely one example.

[0080] As shown in Figure 6, the acquisition unit 11 acquires imaging-related information, such as (a) to (f) above, that is related to imaging performed by the first imaging device 1 and the second imaging device 5 (step S101).

[0081] Next, the determination unit 21A determines whether or not the subject has irregularities based on the depth distribution information of the subject detected from the telephoto YC image included in the imaging-related information acquired in step S101, using condition 1. For example, the determination unit 21A determines whether or not the ratio of the number of pixels within a predetermined depth range Th2 to the total number of pixels of the subject is within a predetermined threshold Th1, using condition 1 (step S201).

[0082] At this time, if the above condition 1 is met, that is, if the ratio of the number of pixels in the depth range Th2 to the total number of pixels of the subject is within the threshold Th1 (step S202 Yes), the determination unit 21A determines the following condition 2. That is, the determination unit 21A determines as condition 2 whether the number of pixels in the telephoto YC image other than the area near the focus point where the focus was achieved by the AF function or MF function exceeds a predetermined threshold Th3 is within a predetermined threshold Th4 (step S203).

[0083] Here, if the above condition 2 is met, that is, if the number of pixels in areas other than the vicinity of focus that have a blur amount exceeding the threshold Th3 is within the threshold Th4 (step S203No), the selection unit 21B selects the depth map as alignment information (step S204). On the other hand, if the above condition 2 is not met, that is, if the number of pixels in areas other than the vicinity of focus that have a blur amount exceeding the threshold Th3 is not within the threshold Th4 (step S203Yes), the selection unit 21B selects the image near the focus as alignment information (step S205).

[0084] Furthermore, if either condition 1 or condition 2 above is not met (step S202No or step S203No), the determination unit 21A determines, as condition 3, whether the transmission bandwidth between the first imaging device 1 and the second imaging device 5 is greater than or equal to a predetermined threshold Th5 (step S206).

[0085] Furthermore, if the above condition 3 is met, that is, if the transmission bandwidth is greater than or equal to a predetermined threshold Th5 (step S206Yes), the determination unit 21A determines, as condition 4, whether the processing capacity of the second imaging device 5, for example, the performance values ​​such as the clock frequency and number of cores of the processor, are greater than or equal to a predetermined threshold Th6 (step S207).

[0086] In this case, if the above condition 4 is further satisfied, that is, if the processing capacity of the second imaging device 5 is greater than or equal to the threshold Th6 (step S207Yes), the selection unit 21B selects the telephoto YC image and the depth to the subject as alignment information (step S208). On the other hand, if neither the above condition 3 nor the above condition 4 is satisfied (step S206No or step S207No), edge information and the depth to the subject are selected as alignment information (step S209).

[0087] Here, if the near-focus image is selected as alignment information in step S205, the following branching can be established in the selection of alignment information depending on whether conditions 3 and 4 above are met. For example, if conditions 3 and 4 above are met, the near-focus YC image and the depth to the subject can be selected, while if neither condition 3 nor condition 4 above is met, the near-focus edge information and the depth to the subject can be selected.

[0088] Subsequently, the output unit 13 transmits the alignment information selected in step S204, step S208, or step S209 to the alignment unit 5C of the second imaging device 5 (step S103), and terminates the process.

[0089] <2-4. One aspect of the effect> As described above, the information processing device 20 according to the second embodiment selects one or more of the following as alignment information: depth map, near-focus image, edge information, and telephoto YC image, depending on the presence or absence of unevenness in the subject, the amount of blur, the amount of transmission bandwidth, and the processing capacity of the second imaging device 5. Therefore, it is possible to select alignment information that is suitable for the shooting conditions of the first imaging device 1. Accordingly, the information processing device 20 according to the second embodiment makes it possible to achieve alignment that balances reduction of processing load and maintenance of accuracy.

[0090] <<3. Third Embodiment>> In the embodiments of this disclosure, an example is given in which a telephoto YC image having the same resolution as the wide-angle YC image is generated as alignment information when the zoom function of the first imaging device 1 or the second imaging device 5 is operated.

[0091] The following is merely an example, showing a case where the output from the first imaging device 1 to the second imaging device 5 is a telephoto YC image.

[0092] <3-1. Example of Functional Configuration of Information Processing Device 30> Figure 7 is a block diagram showing an example of the functional configuration of the information processing device 30 according to the third embodiment. As shown in Figure 7, the information processing device 30 differs from the information processing device 10 shown in Figure 2 in that it has a determination unit 31 whose processing content is partially different from that of the determination unit 12.

[0093] <3-2. Decision Section 31> The determination unit 31 shown in Figure 7 differs from the determination unit 12 shown in Figure 2 in that it has a calculation unit 31A and a generation unit 31B.

[0094] <3-2-1. Calculation section 31A> The calculation unit 31A is a processing unit that calculates the number of pixels of the image sensor 5A of the second imaging device 5 that are included in the overlapping region where the field of view from which the second imaging device 5 captures an image and the field of view from which the first imaging device 1 captures an image overlap.

[0095] Figure 8 shows an example of the correspondence between focal length and angle of view. As an example, Figure 8 shows a case where the optical axes of the first imaging device 1 and the second imaging device 2 coincide. As shown in Figure 8, the wide-angle angle of view is larger than the telephoto angle of view. In this case, the number of pixels of the image sensor 5A of the second imaging device 5, which is included in the overlapping region where the wide-angle and telephoto angles of view overlap, can be calculated based on the aspect ratios of image sensors 1A and 5A. For example, the horizontal angle of view on the wide-angle side can be calculated according to equation (1) below. The horizontal angle of view on the telephoto side can be calculated according to equation (2) below. From equations (1) and (2), it can be seen that when either the focal length on the telephoto side or the focal length on the wide-angle side changes, the number of pixels of the image sensor 5A included in the overlapping region, i.e., the resolution of the overlapping region on the wide-angle side, changes.

[0096] Wide-angle horizontal field of view = 2 × arctan[horizontal width of image sensor 5A / 2 / focal length of wide-angle side] ... (1) The horizontal field of view at the telephoto end = 2 × arctan[horizontal width of image sensor 1A / 2 / focal length at the telephoto end] ... (2)

[0097] Based on the above, the calculation unit 31A recalculates the resolution of the overlapping area on the wide-angle side when the zoom function is in operation. For example, the calculation unit 31A calculates the resolution of the overlapping area on the wide-angle side based on the focal length of the telephoto side after zooming and the focal length of the wide-angle side after zooming. By limiting this resolution calculation to when the zoom function is in operation, processing in situations where there is no change in zoom can be omitted.

[0098] <3-2-2. Generation section 31B> The generation unit 31B generates a telephoto YC image as alignment information with a resolution that matches the resolution of the overlapping region on the wide-angle side calculated by the calculation unit 31A. As an example, when the zoom function is in operation, the generation unit 31B changes the resolution of the telephoto YC image output from the digital signal processing unit 1B to a resolution that matches the resolution of the overlapping region on the wide-angle side calculated by the calculation unit 31A. As a result, the telephoto YC image is enlarged or reduced according to the amount of zoom.

[0099] <3-3. Processing Procedure of Information Processing Device 30> Figure 9 is a flowchart showing the procedure for information output processing according to the third embodiment. This process is merely an example and can be repeatedly executed for each frame in which the first imaging device 1 and the second imaging device 5 capture an image. Note that Figure 9 shows an excerpt of the process corresponding to step S102 from the series of processes shown in Figure 3, i.e., the process executed by the determination unit 31.

[0100] As shown in Figure 9, if the zoom function is in operation (step S301 Yes), the calculation unit 31A calculates the resolution of the overlapping region on the wide-angle side based on the telephoto focal length and the wide-angle focal length after zooming (step S302). Subsequently, the generation unit 31B generates a telephoto YC image as alignment information in which the telephoto YC image output from the digital signal processing unit 1B has been changed to a resolution that matches the resolution of the overlapping region on the wide-angle side calculated in step S302 (step S304).

[0101] On the other hand, if the zoom function is not in operation (step S301No), the number of pixels of the image sensor 5A included in the overlapping region, i.e., the resolution of the overlapping region on the wide-angle side, does not change. For this reason, the calculation unit 31A fixes the resolution at the last changed resolution (step S303). Then, the generation unit 31B generates a telephoto YC image as alignment information, in which the telephoto YC image output from the digital signal processing unit 1B has been changed to a resolution that matches the resolution of the overlapping region on the wide-angle side fixed in step S304 (step S304).

[0102] The alignment information generated in step S304 is output to the alignment unit 5C of the second imaging device 5.

[0103] <3-4. One aspect of the effect> As described above, the information processing device 30 according to the third embodiment generates a telephoto YC image as alignment information with a resolution that matches the resolution of the overlapping region on the wide-angle side, calculated based on the telephoto focal length and the wide-angle focal length after zooming. Therefore, even when the zoom function is operating, the alignment unit 5C can perform alignment while the scales of the telephoto YC image and the wide-angle YC image are matched. Accordingly, the information processing device 30 according to the third embodiment makes it possible to narrow the search range of the transformation matrix and improve the accuracy of alignment.

[0104] <<4. Variation>> The following are examples of modifications of the first to third embodiments.

[0105] <4-1. Combinations between embodiments> In the second and third embodiments, examples were given in which they are implemented individually, but the second and third embodiments can be implemented in combination. In this case, the depth map, near-focus image, and edge information selected as alignment information in the second embodiment can also be changed to a resolution that matches the resolution of the overlapping area on the wide-angle side. As just one example, the flowchart shown in Figure 9 can be implemented in step S201 shown in Figure 6. As another example, the flowchart shown in Figure 9 can be implemented in steps S204, S205, S208, or S209 shown in Figure 6.

[0106] <4-2. Entity responsible for alignment> In the first to third embodiments, examples were given in which alignment by the alignment unit 5C is performed in the second device 5, but alignment can also be performed in the first imaging device 1. In this case, the position of the frame of the telephoto YC image on the wide-angle YC image can be transmitted from the first imaging device 1 to the second imaging device 5. For example, the processing of the determination unit 21 shown in Figure 6 can transmit the wide-angle depth map, the wide-angle near-focus image, and the wide-angle edge information as alignment information by reinterpreting the wide-angle side as the telephoto side and the telephoto side as the wide-angle side. Using such alignment information, the first imaging device 1 performs alignment and transmits the position of the frame of the telephoto YC image on the wide-angle YC image to the second imaging device 5. At this time, when the first imaging device 1 performs alignment, edge extraction and extraction of in-focus areas can be performed from the image acquired by the first imaging device 1, enabling high-speed alignment.

[0107] <4-3. Other variations> Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0108] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0109] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.

[0110] <<5. Hardware Configuration>> The information processing devices 10, 20, or 30 according to each embodiment described above are implemented by a computer 1000 having a configuration such as that shown in Figure 10. The following explanation will use the information processing devices 10, 20, or 30 according to the above embodiments as examples. Figure 10 is a hardware configuration diagram showing an example of a computer 1000. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD 1400, communication interface 1500, and input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0111] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0112] ROM1300 stores boot programs such as the BIOS (Basic Input Output System) executed by CPU1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0113] HDD1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU1100 and data used by such programs. Specifically, HDD1400 is a recording medium that records an information processing program related to this disclosure, which is an example of program data 1450.

[0114] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it generates to other devices via the communication interface 1500.

[0115] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0116] For example, when the computer 1000 functions as an information processing device 10, 20, or 30 according to the above embodiment, the CPU 1100 of the computer 1000 realizes each functional unit included in the control unit 15 by executing an information processing program loaded on the RAM 1200. The HDD 1400 stores the information processing program according to this disclosure and the data in the content storage unit 121. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0117] Furthermore, this technology can also be configured as follows. (1) A determination unit determines alignment information used to align a first image captured by the first imaging device with a second image captured by a second imaging device having a wider field of view than the field of view of the first imaging device, based on imaging-related information related to imaging by the first imaging device. An output unit that outputs the aforementioned alignment information, An information processing device having (2) The determination unit determines the alignment information used for alignment with the second image, The output unit outputs the alignment information to the second imaging device. The information processing device described in (1) above. (3) The aforementioned determination unit, A determination unit that determines whether the aforementioned imaging-related information satisfies predetermined conditions, Based on the determination result of the determination unit, the system includes a selection unit that selects at least one of the following: a depth map corresponding to the first image, a near-focus image corresponding to the vicinity of the subject in focus achieved by adjusting the focal length of the first image, edge information detected from the first image, or the first image itself. The information processing device described in (2) above. (4) The determination unit includes a determination unit that determines whether or not there are irregularities in the subject based on the depth distribution information of the subject in the first image, The selection unit selects the depth map as the alignment information if the subject has irregularities. The information processing device described in (3) above. (5) The determination unit determines whether or not the subject has irregularities based on whether or not the proportion of pixels within a predetermined depth range to the total number of pixels in the subject is within a predetermined threshold. The information processing device described in (4) above. (6) The determination unit determines whether the number of pixels in the region other than the area near focus where the focus has been adjusted by the focal length adjustment control is within a predetermined threshold, The selection unit selects the near-focus image if the number of pixels in a region other than the near-focus area where the amount of blur exceeds the threshold is not within the threshold. The information processing apparatus described in (3) to (5) above. (7) The determination unit determines whether the transmission bandwidth between the first imaging device and the second imaging device is above a predetermined threshold, The selection unit selects the first image if the transmission bandwidth is equal to or greater than the threshold, and selects the edge information if the transmission bandwidth is not equal to or greater than the threshold. The information processing apparatus described in (3) to (6) above. (8) The determination unit determines whether the processing capacity of the second imaging device is above a predetermined threshold, The selection unit selects the first image if the processing capacity is equal to or greater than the threshold, and selects the edge information if the processing capacity is not equal to or greater than the threshold. The information processing apparatus described in (3) to (7) above. (9) If the selection unit selects something other than the depth map, it further selects the depth to the subject as the alignment information. The information processing apparatus described in (3) to (8) above. (10) The aforementioned determination unit, A calculation unit calculates the number of pixels of the image sensor of the second imaging device included in the overlapping region where the field of view of the first imaging device and the field of view of the second imaging device overlap as the resolution of the overlapping region, The system includes a generation unit that generates a first image as alignment information, in which the first image has been modified to a resolution that matches the resolution of the overlapping region. The information processing apparatus described in (2) to (9) above. (11) The calculation unit performs the calculation of the resolution of the overlapping region when the zoom function is operating on either the first imaging device or the second imaging device. The information processing device described in (10) above. (12) The determination unit determines the alignment information used for alignment with the first image, The output unit outputs the alignment information to the first imaging device. The information processing device described in (1) above. (13) Based on imaging-related information associated with imaging by the first imaging device, alignment information is determined to be used for aligning the first image captured by the first imaging device with the second image captured by the second imaging device, which has a wider field of view than the field of view of the first imaging device. Outputting the aforementioned alignment information, An information processing method in which a computer performs the processing. (14) Based on imaging-related information associated with imaging by the first imaging device, alignment information is determined to be used for aligning the first image captured by the first imaging device with the second image captured by the second imaging device, which has a wider field of view than the field of view of the first imaging device. Outputting the aforementioned alignment information, An information processing program that instructs a computer to perform a task. [Explanation of symbols]

[0118] 1. First imaging device 1A Image sensor 1B Digital Signal Processing Unit 1C Display section 5. Second imaging device 5A Image Sensor 5B Digital Signal Processing Unit 5C Alignment section 5D display 10 Information Processing Devices 11 Acquisition Department 12. Decision Section 13 Output section

Claims

1. A determination unit determines alignment information used to align a first image captured by the first imaging unit with a second image captured by a second imaging unit having a different field of view than the field of view of the first imaging unit, based on imaging-related information related to imaging by the first imaging unit. An output unit that outputs the aforementioned alignment information, It has, The determination unit determines predetermined conditions based on the imaging-related information, and, according to the determination result, selects one of a plurality of alignment information sets, which include the first image and processing information generated from the first image and having a data structure predetermined to reduce the processing load of the alignment compared to using the first image. The determination unit determines the alignment information used for alignment with the second image, The output unit outputs the alignment information to the second imaging unit. Information processing device.

2. The second imaging unit has a wider field of view than the first imaging unit. The information processing apparatus according to claim 1.

3. The aforementioned determination unit, A determination unit that determines whether the aforementioned imaging-related information satisfies predetermined conditions, Based on the determination result of the determination unit, the system includes a selection unit that selects at least one of the following as alignment information: a depth map corresponding to the first image, a near-focus image corresponding to the vicinity of the subject in focus achieved by adjusting the focal length of the first image, edge information detected from the first image, or the first image itself. The information processing apparatus according to claim 1.

4. The determination unit determines whether or not there are irregularities in the subject based on the depth distribution information of the subject in the first image. The selection unit selects the depth map as the alignment information if the subject has irregularities. The information processing apparatus according to claim 3.

5. The determination unit determines whether or not the subject has irregularities based on whether or not the proportion of pixels within a predetermined depth range to the total number of pixels in the subject is within a predetermined threshold. The information processing apparatus according to claim 4.

6. The determination unit determines whether the number of pixels in the region other than the area near focus where the focus has been adjusted by the focal length adjustment control is within a predetermined threshold, The selection unit selects the near-focus image as the alignment information if the number of pixels in a region other than the near-focus area where the amount of blur exceeds the threshold is not within the threshold. The information processing apparatus according to claim 3.

7. The determination unit determines whether the transmission bandwidth between the first imaging unit and the second imaging unit is above a predetermined threshold, The selection unit selects the first image as the alignment information if the transmission bandwidth is equal to or greater than the threshold, and selects the edge information as the alignment information if the transmission bandwidth is not equal to or greater than the threshold. The information processing apparatus according to claim 3.

8. The imaging-related information further includes the processing capabilities of the second imaging unit, The determination unit determines whether the processing capacity of the second imaging unit is above a predetermined threshold, The selection unit selects the first image as the alignment information if the processing capacity is equal to or greater than the threshold, and selects the edge information as the alignment information if the processing capacity is not equal to or greater than the threshold. The output unit outputs the alignment information selected by the selection unit to the alignment unit of the second imaging unit, which performs alignment of the second image and the first image. The information processing apparatus according to claim 3.

9. If the selection unit selects something other than the depth map, it further selects the depth to the subject as the alignment information. The information processing apparatus according to claim 3.

10. The aforementioned determination unit, A calculation unit calculates the number of pixels of the image sensor of the second imaging unit included in the overlapping region where the field of view of the first imaging unit and the field of view of the second imaging unit overlap as the resolution of the overlapping region, The system includes a generation unit that generates a first image as alignment information, in which the first image has been modified to a resolution that matches the resolution of the overlapping region. The information processing apparatus according to claim 3.

11. The calculation unit performs the calculation of the resolution of the overlapping region when the zoom function is operating in either the first imaging unit or the second imaging unit. The information processing apparatus according to claim 10.

12. Based on imaging-related information related to imaging by the first imaging unit, alignment information is determined to be used for aligning the first image captured by the first imaging unit with the second image captured by the second imaging unit, which has a different field of view than the field of view of the first imaging unit. Outputting the aforementioned alignment information, The computer performs the process, The process of making the determination includes determining predetermined conditions based on the imaging-related information, and, according to the determination result, selecting one of a plurality of alignment information, which includes the first image and processing information generated from the first image and having a data structure predetermined to reduce the processing load of the alignment compared to when the first image is used. The process described above determines the alignment information used for alignment with the second image, The output process involves outputting the alignment information to the second imaging unit. Information processing methods.

13. Based on imaging-related information related to imaging by the first imaging unit, alignment information is determined to be used for aligning the first image captured by the first imaging unit with the second image captured by the second imaging unit, which has a different field of view than the field of view of the first imaging unit. Outputting the aforementioned alignment information, Let the computer perform the process, The process of making the determination includes determining predetermined conditions based on the imaging-related information, and, according to the determination result, selecting one of a plurality of alignment information, which includes the first image and processing information generated from the first image and having a data structure predetermined to reduce the processing load of the alignment compared to when the first image is used. The process described above determines the alignment information used for alignment with the second image, The output process involves outputting the alignment information to the second imaging unit. Information processing program.

Images