Information processing device, information processing method, and information processing program

Abstract

The information processing apparatus (10) includes: a decision unit that determines alignment information for aligning a first image imaged by the first imaging device and a second image imaged by the second imaging device having an imaging angle wider than the imaging angle of the first imaging device based on imaging-related information related to imaging captured by the first imaging device; and an output unit that outputs the alignment information.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. Background Art

[0002] As an example, when capturing an image with a camera, various viewfinders such as an optical viewfinder and an electronic viewfinder are used to check the composition.

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2018-11302 Summary of the Invention

[0006] Problems to be solved by the present invention

[0007] However, with the above-described viewfinder, trial and error of alternately moving the line of sight between the inside and outside of the viewfinder occurs until the composition intended by the photographer is accommodated within the frame of the camera, and thus the framing work becomes complicated.

[0008] Therefore, an object of the present disclosure is to provide an information processing device, an information processing method, and an information processing program that can simplify framing work.

[0009] Solution to the problem

[0010] In order to solve the above-mentioned problem, an information processing device according to an embodiment of the present disclosure includes: a decision unit, configured to decide alignment information to be used for aligning between a first image captured by a first imaging device and a second image captured by a second imaging device having a wider angle of view than that of the first imaging device based on imaging-related information related to imaging performed by the first imaging device; and an output unit, configured to output the alignment information. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram illustrating a configuration example of a system according to the first embodiment.

[0012] Figure 2 is a block diagram illustrating a functional configuration example of the information processing apparatus according to the first embodiment.

[0013] Figure 3 is a flowchart illustrating the procedure of information output processing according to the first embodiment.

[0014] Figure 4 is a block diagram illustrating a functional configuration example of an information processing apparatus according to the second embodiment.

[0015] Figure 5is a graph illustrating an example of depth distribution information.

[0016] Figure 6 is a flowchart illustrating the procedure of information output processing according to the second embodiment.

[0017] Figure 7 is a block diagram illustrating a functional configuration example of an information processing apparatus according to the third embodiment.

[0018] Figure 8 is a diagram illustrating an example of the correspondence relationship between focal length and angle of view.

[0019] Figure 9 is a flowchart illustrating the procedure of information output processing according to the third embodiment.

[0020] Figure 10 is a hardware configuration diagram illustrating an example of a computer. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described in detail based on the accompanying drawings. Note that in each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0022] Furthermore, the present disclosure will be described according to the following order of items.

[0023] 1. First embodiment

[0024] 1-1. System Configuration Example

[0025] 1-2. Example of use case

[0026] 1-3. Functional Configuration Example of First Imaging Device

[0027] 1-3-1. Imaging Sensor

[0028] 1-3-2. Digital signal processing unit

[0029] 1-3-3. Display unit

[0030] 1-4. Functional Configuration Example of Second Imaging Device

[0031] 1-4-1. Imaging sensor

[0032] 1-4-2. Digital signal processing unit

[0033] 1-4-3. Alignment unit

[0034] 1-4-4. Display unit

[0035] 1-5. One aspect of the problem

[0036] 1-6. One aspect of the problem-solving approach

[0037] 1-7. Functional Configuration Example of Information Processing Device

[0038] 1-7-1. Acquisition unit

[0039] 1-7-2. Decision Unit

[0040] 1-7-3. Output unit

[0041] 1-8. Processing Procedure of Information Processing Device

[0042] 1-9. One aspect of the effect

[0043] 2. Second embodiment

[0044] 2-1. Functional Configuration Example of Information Processing Device

[0045] 2-2. Determination Unit

[0046] 2-2-1. Determine the unit

[0047] 2-2-2. Select unit

[0048] 2-3. Processing Procedure of Information Processing Device

[0049] 2-4. One aspect of the effect

[0050] 3. Third embodiment

[0051] 3-1. Functional Configuration Example of Information Processing Device

[0052] 3-2. Determination unit

[0053] 3-2-1. Computational Unit

[0054] 3-2-2. Generation Unit

[0055] 3-3. Processing Procedure of Information Processing Device

[0056] 3-4. One aspect of the effect

[0057] 4. Modifications

[0058] 4-1. Combinations between Examples

[0059] 4-2. Targeting Execution Entity

[0060] 4-3. Other Modifications

[0061] 5. Hardware Configuration

[0062] <<1. First embodiment>>

[0063] <1-1. Example of system configuration>

[0064] Figure 1 1 is a diagram illustrating an example of a configuration of a system according to the first embodiment. Figure 1 The system shown in provides an imaging support function of displaying a mark M indicating the position of a frame of the image captured by the first imaging device 1 on the image 50 captured by the second imaging device 5 .

[0065] like Figure 1 As shown in , the system may include a first imaging device 1 and a second imaging device 5. The first imaging device 1 and the second imaging device 5 are communicably connected. As an example only, the first imaging device 1 and the second imaging device 5 may be connected via a Universal Serial Bus (USB) cable or the like. In addition, bidirectional communication does not necessarily need to be performed between the first imaging device 1 and the second imaging device 5, and the first imaging device 1 and the second imaging device 5 may be connected via a High-Definition Multimedia Interface (HDMI) (registered trademark) cable or the like.

[0066] As an aspect, the first imaging device 1 and the second imaging device 5 do not necessarily need to have the same angle of view. For example, although a telephoto lens having a focal length longer than that of the lens used for imaging by the second imaging device 5 may be mounted on the first imaging device 1, a wide-angle lens having a wider angle of view than that of the lens used for imaging by the first imaging device 1 may be mounted on the second imaging device 5.

[0067] As another aspect, the optical axes of the first imaging device 1 and the second imaging device 5 may coincide with each other, or the optical axes of the first imaging device 1 and the second imaging device 5 do not necessarily need to coincide with each other. Figure 1 , the first imaging device 1 and the second imaging device 5 are shown as different device groups as an example only, but the first imaging device 1 and the second imaging device 5 may be integrated into two devices.

[0068] Note that, Figure 1 An example is illustrated in which the first imaging device 1 and the second imaging device 5 are connected by wire, but the configuration is not limited to this example, and the first imaging device 1 and the second imaging device 5 may be connected by near-field wireless communication or wireless communication.

[0069] <1-2. Example of use case>

[0070] The first imaging device 1 and the second imaging device 5 do not necessarily need to be electronic devices dedicated to imaging. For example, it is not excluded that the first imaging device 1 and the second imaging device 5 are provided with functions other than the imaging function.

[0071] As an example only, there is a use case where a mark M indicating the position of a frame captured by a digital camera is superimposed and displayed on an image 50 captured by a smartphone or tablet terminal. In this case, the first imaging device 1 can be implemented as a digital camera, and the second device 5 can be implemented as a smartphone, tablet terminal, etc.

[0072] As another example, there is a use case in which a mark M indicating the position of a frame captured by a digital camera (such as an interchangeable lens camera with a stabilizer) is superimposed and displayed on an image 50 captured by a pair of augmented reality (AR) glasses. In this case, the first imaging device 1 can be implemented as a digital camera, and the second imaging device 5 can be implemented as AR glasses.

[0073] Note that, in Figure 1 , the mark M of the frame itself on which the first imaging device 1 captures an image is illustrated as an example only, but the shape and size of the mark M may be arbitrary. For example, the mark M may be a pointer, a graphic, a mark, etc. indicating the position of the frame on which the first imaging device 1 captures an image.

[0074] <1-3. Functional Configuration Example of First Imaging Device>

[0075] Figure 2 1 is a block diagram illustrating a functional configuration example of the information processing apparatus 10 according to the first embodiment. Figure 2 As shown in , the first imaging apparatus 1 includes an imaging sensor 1A, a digital signal processing unit 1B, a display unit 1C, and an information processing device 10 .

[0076] Here, although Figure 2 The example in which the information processing device 10 is incorporated into the first imaging device 1 is illustrated, but this is merely an example. For example, the information processing device 10 may be incorporated into the second imaging device 5, or may be implemented as a third device different from the first imaging device 1 and the second imaging device 5. Note that the information processing device 10 will be described after describing the functional configuration examples of the first imaging device 1 and the second imaging device 5.

[0077] <1-3-1. Imaging Sensor 1A>

[0078] As the imaging sensor 1A, an image sensor such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor can be employed. As an example only, light focused by the optical system (e.g., a telephoto lens) of the first imaging device 1 is photoelectrically converted by the imaging sensor 1A. The RAW image obtained through this photoelectric conversion is input to the digital signal processing unit 1B.

[0079] Here, an image plane phase difference sensor (not shown) may be incorporated into the imaging sensor 1A. For example, the distance to the subject (i.e., depth) can be measured based on the phase difference signal output from the image plane phase difference sensor. In this way, the depth measured for each image plane phase difference sensor can be output as a depth map to the digital signal processing unit 1B.

[0080] Note that although an image plane phase difference sensor is illustrated here, a phase difference sensor may be installed in place of the image plane phase difference sensor on the first imaging device 1, or both the image plane phase difference sensor and the phase difference sensor may be installed. Furthermore, although an image plane phase difference sensor and a phase difference sensor are illustrated here, this does not exclude the possibility that a depth sensor other than the image plane phase difference sensor and the phase difference sensor may be installed in the first imaging device 1.

[0081] <1-3-2. Digital Signal Processing Unit 1B>

[0082] The digital signal processing unit 1B is a functional unit that performs digital signal processing. In one embodiment, the digital signal processing unit 1B can be implemented using hardware such as a digital signal processor (DSP). Examples of the digital signal processing described above include processes such as converting a RAW image into an image in a predetermined format (e.g., a YC image), so-called RAW development, white balance adjustment, and chromatic aberration correction.

[0083] Note that an example in which the above-described digital signal processing unit 1B is realized by hardware has been described here, but the digital signal processing unit 1B may be realized by software such as a RAW development engine executed by a processor.

[0084] <1-3-3. Display Unit 1C>

[0085] The display unit 1C is a functional unit that displays various information. As an embodiment, the display unit 1C can be implemented by disposing a liquid crystal display, an organic electroluminescent (EL) display, or the like on the back of the housing of the first imaging device 1. For example, each time a YC image is output through the digital signal processing unit 1B, the display unit 1C displays the YC image in real time. Hereinafter, from the perspective of distinguishing the label of the YC image output by the digital signal processing unit 1B from the label of the YC image output by the digital signal processing unit 5B described later, the former may be referred to as a "telephoto YC image" and the latter may be referred to as a "wide-angle YC image". Therefore, a live view function of a telephoto YC image can be implemented.

[0086] Note that the display unit 1C can also be implemented as a touch panel by being integrated with an input unit (not shown). In addition, as described above, since there is an aspect in which a mark indicating the position of the frame in which the first imaging device 1 captures an image is displayed on the display unit 5D of the second imaging device 5, the display unit 1C does not necessarily need to be provided in the first imaging device 1.

[0087] <1-4. Functional Configuration Example of Second Imaging Device 5>

[0088] like Figure 2 As shown in FIG, the second imaging device 5 includes an imaging sensor 5A, a digital signal processing unit 5B, an alignment unit 5C, and a display unit 5D.

[0089] <1-4-1. Imaging Sensor 5A>

[0090] As imaging sensor 5A, an image sensor such as a CCD or CMOS can be employed. As an example only, light focused by the optical system (e.g., a telephoto lens) of second imaging device 5 is photoelectrically converted by imaging sensor 5A. The RAW image obtained through this photoelectric conversion is input to digital signal processing unit 5B.

[0091] In this imaging sensor 5A, an image plane phase difference sensor (not shown) can also be incorporated, similar to the imaging sensor 1A described above. In this way, the depth measured for each image plane phase difference sensor can be output as a depth map to the digital signal processing unit 5B. Note that although an image plane phase difference sensor has been illustrated here, a phase difference sensor can be installed on the second imaging device 5 instead of the image plane phase difference sensor, or both the image plane phase difference sensor and the phase difference sensor can be installed. In addition, although an image plane phase difference sensor and a phase difference sensor are illustrated here, it is not excluded that a depth sensor other than the image plane phase difference sensor and the phase difference sensor may be provided in the second imaging device 5.

[0092] <1-4-2. Digital Signal Processing Unit 5B>

[0093] The digital signal processing unit 5B is a functional unit that performs digital signal processing. As an embodiment, the digital signal processing unit 5B is virtually implemented by a processor such as a central processing unit (CPU) or a microprocessing unit (MPU). For example, the processor reads a program such as software of a RAW development engine, etc. from a storage device (not shown) in addition to reading an operating system (OS). The processor then executes the above-mentioned RAW development engine to carry out processing corresponding to the digital signal processing unit 5B on a memory such as a random access memory (RAM). Therefore, the digital signal processing unit 5B is virtually implemented as processing. Here, the CPU and MPU have been illustrated as examples of processors, but the digital signal processing unit 5B can be implemented by any processor, regardless of whether it is general-purpose or special-purpose. In addition, it is not excluded that the digital signal processing unit 5B is implemented by hard-wired logic such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0094] The digital signal processing unit 5B can perform functions similar to those of the digital signal processing unit 1B described above. For example, as an example of the above-mentioned digital signal processing, the digital signal processing unit 5B can perform processing such as converting a RAW image into an image in a predetermined format (e.g., a YC image), so-called RAW development, white balance adjustment, and chromatic aberration correction.

[0095] Note that an example has been described here in which the digital signal processing unit 5B is realized by software such as a RAW development engine executed by a processor, but the above-described digital signal processing unit 1B may be realized by hardware.

[0096] <1-4-3. Alignment Unit 5C>

[0097] Alignment unit 5C is a processing unit that aligns the image captured by first imaging device 1 with the image captured by second imaging device 5. As an example, similar to digital signal processing unit 5B, alignment unit 5C can be virtually implemented by a processor such as a CPU or MPU. For example, the processor reads an image processing program for aligning multiple images. The processor then executes the image processing program to execute the processing corresponding to alignment unit 5C on a memory such as RAM. Thus, alignment unit 5C is virtually implemented as a process.

[0098] Here, the above-mentioned alignment refers to general registration between two images, in which a transformation matrix such as translation, rotation, or deformation is applied to one image to align one image with the other. In one aspect, in registration, a transformation matrix that maximizes the similarity between the two images is searched. For example, as a similarity measure, the sum of squared differences (SSD), the sum of absolute differences (SAD), the correlation coefficient, etc. can be used. In addition, when the transformation matrix is ​​applied, feature points can be matched between the two images.

[0099] This registration achieves alignment between the image captured by the first imaging device 1 (e.g., the telephoto YC image output by the digital signal processing unit 1B) and the image captured by the second imaging device 5 (e.g., the wide-angle YC image output by the digital signal processing unit 5B). Furthermore, here, merely as an example, an example of a transformation matrix obtained through the above-described registration for aligning the telephoto YC image with the wide-angle YC image will be described.

[0100] As an example only, the alignment unit 5C may superimpose the telephoto YC image on the wide-angle YC image. At this time, if the optical axes of the first imaging device 1 and the second imaging device 5 are the same, then Figure 1 As in the example shown in , the frame of the telephoto YC image obtained by alignment maintains a rectangular shape. However, when the optical axes are different, there is a possibility that the frame of the telephoto YC image obtained by alignment does not have a rectangular shape but has a deformed shape.

[0101] From this aspect, the alignment unit 5C can superimpose a mark M indicating the position (e.g., the center position) of the frame of the telephoto YC image on the wide-angle YC image. Hereinafter, the wide-angle YC image on which the mark M indicating the position of the frame of the telephoto YC image is superimposed may be referred to as a "wide-angle image with a telephoto frame mark." The wide-angle image with a telephoto frame mark obtained in this manner is output to the display unit 5D.

[0102] <1-4-4. Display Unit 5D>

[0103] The display unit 5D is a functional unit that displays various information. As an example, the display unit 5D can be implemented by a liquid crystal display, an organic EL display, or the like, and can also be implemented as a touch panel by integrating with an input unit (not shown). For example, each time a wide-angle image with a telephoto frame mark is output from the alignment unit 5C, the display unit 5D displays the wide-angle image with the telephoto frame mark in real time.

[0104] <1-5. One aspect of the problem>

[0105] In Embodiment 1 of the present disclosure, as part of the aforementioned imaging support function, a wide-angle image with a telephoto frame mark is displayed on the display unit 5D. This display allows the mark M indicating the position of the telephoto frame on the wide-angle YC image to be viewed from above. This eliminates the need for trial and error in shifting the line of sight between the inner and outer sides of the viewfinder, simplifying framing work.

[0106] While this is true, in order to display a wide-angle image with a telephoto frame marker in real time, image processing such as alignment of the telephoto YC image with the wide-angle YC image may become a bottleneck, and delays may occur in the display of the position of the frame of the telephoto YC image to be superimposed on the wide-angle YC image. If this delay occurs, there will be a discrepancy between the position of the frame of the telephoto YC image on the display and the actual position of the frame of the telephoto YC image, making it impossible to perform imaging using the intended composition.

[0107] Various techniques for image alignment have been proposed, but they all rely on techniques that approach the alignment or registration algorithm itself. The ingenuity of such an algorithm lies solely in how to align two images. Therefore, only two images are used for alignment, and there is no viewpoint from which information that could aid alignment could be obtained when capturing one of the images. Furthermore, the ingenuity of such an algorithm is not utilized for alignment.

[0108] <1-6. One aspect of the solution to the problem>

[0109] Therefore, the information processing apparatus 10 according to an embodiment of the present disclosure provides an information output function that decides and outputs alignment information for image alignment between the first imaging device 1 and the second imaging device 5 based on imaging-related information related to imaging performed by the first imaging device 1 .

[0110] That is, the information processing apparatus 10 according to the embodiment of the present disclosure outputs alignment information that can contribute to reducing the amount of processing when performing alignment with an image captured by the second imaging device 5 from a viewpoint specific to imaging in the first imaging device 1 .

[0111] Therefore, in the information processing device 10 according to the embodiment of the present disclosure, since alignment of the telephoto YC image and the wide-angle YC image can be accelerated, the real-time characteristic of displaying the wide-angle image with the telephoto frame mark can be improved.

[0112] <1-7. Functional Configuration Example of Information Processing Device 10>

[0113] like Figure 2 As shown in , the information processing device 10 includes an acquisition unit 11 , a decision unit 12 , and an output unit 13 .

[0114] <1-7-1. Acquisition Unit 11>

[0115] The acquisition unit 11 is a processing unit that acquires imaging-related information related to imaging performed by the first imaging device 1 and the second imaging device 5. Here, as explained below, "imaging-related information" is not limited to the images themselves captured by the first imaging device 1 and the second imaging device 5, but can include general information related to imaging. Note that an example has been described herein where imaging-related information is acquired for both the first imaging device 1 and the second imaging device 5, but imaging-related information can be acquired for either the first imaging device 1 or the second imaging device 5.

[0116] a) Image size of imaging sensor 1A and imaging sensor 5A and imaging sensor size

[0117] b) Focal lengths of the first imaging device 1 and the second imaging device 5

[0118] c) Parameters for imaging by the first imaging device 1 and the second imaging device 5

[0119] d) Information that can be obtained from the telephoto YC image and the wide-angle YC image

[0120] e) Information on the bandwidth and stability of data transmission between the first imaging device 1 and the second imaging device 5

[0121] f) Information on the physical positional relationship between the first imaging device 1 and the second imaging device 5 and the optical axis offset

[0122] g) Processing capabilities of the arithmetic units installed on the first imaging device 1 and the second imaging device 5

[0123] The imaging-related information described in a) through g) above can be acquired using the frame of the image captured by the first imaging device 1 and the second imaging device 5 as a unit. For example, supplementing c) above, as an example only, the imaging parameters described above may include exposure, white balance, shutter speed, International Organization for Standardization (ISO) value, F-number, etc. Supplementing d) above, as an example only, the acquired information may include information regarding object recognition results, depth maps, appropriate exposure, light source information, flicker information, etc.

[0124] Such imaging-related information can be acquired from both the first imaging device 1 and the second imaging device 5. For example, the imaging-related information about the first imaging device 1 can be acquired from the digital signal processing unit 1B or a driver integrated circuit (IC) that drives the optical system of the first imaging device 1. In addition, the imaging-related information about the second imaging device 5 can be acquired from the digital signal processing unit 5B or a driver IC that drives the optical system of the second imaging device 5.

[0125] <1-7-2. Decision Unit 12>

[0126] The decision unit 12 is a processing unit that determines alignment information based on the imaging-related information acquired by the acquisition unit 11. In one aspect, the decision unit 12 determines alignment information used by the alignment unit 5C of the second imaging device 5 for alignment with the wide-angle YC image. This alignment information can be determined from the perspectives of reducing the amount of processing required for alignment with the wide-angle YC image and reducing the amount of transmission from the first imaging device 1 to the second imaging device 5.

[0127] As just one example, the decision unit 12 determines alignment information (at least one of a depth map, an image near focus, edge information, or a YC image, or a combination thereof). For example, a "depth map" refers to a depth measurement map using a phase difference sensor, an image plane phase difference sensor, or both. Furthermore, a "near focus image" refers to a portion of the telephoto YC image corresponding to the vicinity of the subject focused on by focus adjustment control. For example, focus adjustment control can be automatically performed using an autofocus (AF) function implemented by a phase difference sensor, an image plane phase difference sensor, or both. Alternatively, focus adjustment control can be performed manually using manual focus (MF). Furthermore, "edge information" refers to information indicating edges detected from the telephoto YC image. Furthermore, a "telephoto YC image" refers to the telephoto YC image itself output from the digital signal processing unit 1B. The "depth map," "near focus image," "edge information," and "telephoto YC image" can be changed to the same resolution as the wide-angle YC image during operation of the zoom function in the first imaging device 1 or the second imaging device 5.

[0128] <1-7-3. Output Unit 13>

[0129] The output unit 13 is a processing unit that outputs the alignment information decided by the decision unit 12. As one aspect, the output unit 13 transmits the alignment information decided by the decision unit 12 to the alignment unit 5C of the second imaging device 5.

[0130] <1-8. Processing Procedure of Information Processing Device>

[0131] Figure 3 1 is a flowchart illustrating the procedure of the information output process according to the first embodiment. As an example only, this process may be repeatedly performed for each frame of the image captured by the first imaging device 1 and the second imaging device 5.

[0132] like Figure 3 As shown in FIG, the acquisition unit 11 acquires imaging-related information related to imaging performed by the first imaging device 1 and the second imaging device 5, such as the above-mentioned a) to f) and the like (step S101).

[0133] Subsequently, the decision unit 12 decides alignment information to be used by the alignment unit 5C of the second imaging device 5 for alignment with the wide-angle YC image based on the imaging-related information acquired in step S101 (step S102 ).

[0134] Then, the output unit 13 transmits the alignment information decided in step S102 to the alignment unit 5C of the second imaging device 5 (step S103 ) and ends the process.

[0135] <1-9. One Aspect of the Effect>

[0136] As described above, the information processing apparatus 10 according to the first embodiment decides and outputs alignment information to be used for image alignment between the first imaging device 1 and the second imaging device 5 based on imaging-related information related to imaging performed by the first imaging device 1 .

[0137] That is, the information processing apparatus 10 according to the first embodiment outputs alignment information that can help reduce the amount of processing and improve accuracy when performing alignment with the wide-angle YC image captured by the second imaging device 5, from a perspective unique to imaging in the first imaging device 1.

[0138] Therefore, in the information processing apparatus 10 according to the first embodiment, since alignment of the telephoto YC image and the wide-angle YC image can be accelerated, the real-time characteristic of displaying the wide-angle image with the telephoto frame mark can be improved.

[0139] <<2. Second embodiment>>

[0140] In an embodiment of the present disclosure, an example of an algorithm that selects one or more of the following as the above-mentioned alignment information will be described: "Depth map", "Image near focus", "Edge information", and "Telephoto YC image".

[0141] <2-1. Functional Configuration Example of Information Processing Device 20>

[0142] Figure 4 2 is a block diagram illustrating a functional configuration example of the information processing apparatus 20 according to the second embodiment. Figure 4 As shown in FIG, the information processing device 20 is different in that the information processing device 20 includes a Figure 2 The processing contents of the decision unit 12 of the information processing apparatus 10 shown in FIG. 1 are partially different from the processing contents of the decision unit 21 .

[0143] <2-2. Decision Unit 21>

[0144] Figure 4 The decision unit 21 shown in FIG Figure 2The decision unit 12 shown in FIG. 1 is different in that the decision unit 21 includes a determination unit 21A and a selection unit 21B.

[0145] <2-2-1. Determination Unit 21A>

[0146] The determination unit 21A is a processing unit that determines whether the imaging-related information acquired by the acquisition unit 11 satisfies a predetermined condition.

[0147] As an example, the determination unit 21A obtains depth distribution information regarding a subject detected in the telephoto YC image acquired from the digital signal processing unit 1B. The "subject" referred to here can be a subject focused by the AF function or the MF function, a subject tracked by the AF function, or a subject detected as a result of object detection. The determination unit 21A then determines whether the subject has concavity or convexity based on the depth distribution information regarding the subject in the telephoto YC image. By way of example only, the determination unit 21A determines whether the subject has concavity or convexity based on whether the ratio of pixels within a predetermined depth range Th2 to the total number of pixels in the subject is within a predetermined threshold Th1, as condition 1. This "depth range Th2" can be set based on a peak in the subject's depth distribution. For example, a certain range before and after the peak in the depth distribution can be set as depth range Th2. In this case, the range before and after the peak included in depth range Th2 can vary and need not necessarily be the same.

[0148] Figure 5 is a graph illustrating an example of depth distribution information. Figure 5 The vertical axis of the graph shown in indicates the number of pixels, and the horizontal axis indicates the depth. The vertical axis and the horizontal axis may be a histogram representing the frequency and level of the depth. Figure 5 In , the portion corresponding to the total number of pixels of the object is indicated by whitening, and the portion corresponding to the depth range Th2 is indicated by hatching. Figure 5 In the example shown in , the presence or absence of unevenness of the subject can be determined based on whether the number of pixels included in the depth range Th2 (i.e., the ratio of the number of pixels in the shadow portion to the total number of pixels of the subject, expressed as a probability, percentage, etc.) is within a predetermined threshold Th1 (e.g., 50%). For example, if the ratio is within the threshold Th1, the subject is determined to have unevenness, and if the ratio exceeds the threshold Th1, the subject is determined to have no unevenness.

[0149] Then, when the above-described condition 1 is satisfied (i.e., when the ratio of the number of pixels in the depth range Th2 to the total number of pixels of the subject is within the threshold value Th1), the determination unit 21A determines the following condition 2. For example, the determination unit 21A determines as condition 2 whether the number of pixels whose blur amount exceeds a predetermined threshold value Th3 in an area other than the near-focus area focused by the AF function or the MF function in the telephoto YC image is within a predetermined threshold value Th4. In this case, if the number of pixels whose blur amount exceeds the threshold value Th3 in the area other than the near-focus area is within the threshold value Th4, it is determined that the above-described condition 2 is satisfied.

[0150] Furthermore, if the above-described condition 1 or condition 2 is not satisfied, the determination unit 21A determines condition 3 and condition 4, which will be described later. For example, the determination unit 21A determines whether the transmission band between the first imaging device 1 and the second imaging device 5 is greater than or equal to a predetermined threshold value Th5 as condition 3. Furthermore, if the above-described condition 3 is satisfied (i.e., if the transmission band is greater than or equal to the predetermined threshold value Th5), the determination unit 21A determines whether the processing capability of the second imaging device 5 (e.g., a performance value of the processor such as a clock frequency or the number of cores) is greater than or equal to a predetermined threshold value Th6 as condition 4.

[0151] <2-2-2. Select Unit 21B>

[0152] The selection unit 21B is a processing unit that selects one or more of the “depth map”, “near-focus image”, “edge information”, and “telephoto YC image” as alignment information based on the determination result of the determination unit 21A.

[0153] On the one hand, when the above conditions 1 and 2 are satisfied, the selection unit 21B selects the depth map as the alignment information. This conditional branching corresponds to the case where the ratio of the number of pixels in the depth range Th2 to the total number of pixels of the object is within the threshold Th1, and the number of pixels whose blur amount exceeds the threshold Th3 in the area other than the focus vicinity is within the threshold Th4.

[0154] There are several reasons why this alignment information is selected. Specifically, when condition 1 is met, there's a high probability that sufficient alignment accuracy can be achieved by performing alignment between the telephoto depth map and the wide-angle depth map, rather than between the telephoto and wide-angle YC images. Furthermore, while the blur amount in the second imaging device 5, which has a shorter focal length than the first imaging device 1, tends to be smaller than that in the first imaging device 1, when condition 2 is met, there's a high probability that the difference between the blur amounts in the first imaging device 1 and the second imaging device 5 will be large enough to support alignment between the depth maps. For these reasons, the depth map is selected as alignment information. By selecting the depth map as alignment information in this manner, the amount of processing can be reduced compared to having the alignment unit 5C perform alignment between the images. Furthermore, since the depth map contains less information than the telephoto YC image and edge information, transmission delay from the first imaging device 1 to the second imaging device 5 can be minimized.

[0155] On the other hand, when the above-mentioned condition 1 is satisfied but the above-mentioned condition 2 is not satisfied, the selection unit 21B selects the image near the focus as the alignment information. This conditional branching corresponds to a case where the ratio of the number of pixels in the depth range Th2 to the total number of pixels of the subject is within the threshold value Th1, and the number of pixels whose blur amount exceeds the threshold value Th3 in the area other than the focus is not within the threshold value Th4.

[0156] This selection of alignment information is performed because, even when condition 1 is satisfied, if condition 2 is not satisfied, there is a high probability that the following situation will occur. Specifically, there is a high probability that the difference between the blur amount of the first imaging device 1 and the blur amount of the second imaging device 5 exceeds the limit that can be tolerated for alignment between the depth maps. For this reason, the image near the focus point is selected as the alignment information. By selecting the image near the focus point as alignment information in this way, even blurred areas outside the focus point are used for alignment by the alignment unit 5C, thereby minimizing the reduction in alignment accuracy. Furthermore, since the image near the focus point is part of the telephoto YC image, the amount of alignment processing can be reduced compared to using the entire telephoto YC image for alignment. Furthermore, since the image near the focus point has less information than the telephoto YC image, transmission delay from the first imaging device 1 to the second imaging device 5 can be minimized.

[0157] On the other hand, if the above conditions 3 and 4 are satisfied, the selection unit 21B selects the telephoto YC image and the depth to the subject as the alignment information. This conditional branching corresponds to a case where the transmission band between the first imaging device 1 and the second imaging device 5 is greater than or equal to the threshold value Th5 and the processing capability of the second imaging device 5 is greater than or equal to the threshold value Th6.

[0158] The reasons for selecting this alignment information are as follows. Specifically, if both conditions 3 and 4 are met, there's a high probability that transmission delays will be less likely to occur even when the telephoto YC image is transmitted from the first imaging device 1 to the second imaging device 5. Furthermore, there's a high probability that processing delays will be less likely to occur even when the alignment unit 5C aligns the telephoto YC image with the wide-angle YC image. Furthermore, by including the depth of the subject in the alignment information, it's possible for the alignment unit 5C to narrow the area for alignment with the wide-angle YC image to a region near that depth, or to weight the area near that depth more heavily than the area at depths beyond that depth, and then have the alignment unit 5C perform alignment with the wide-angle YC image.

[0159] On the other hand, if the above-mentioned condition 3 or condition 4 is not satisfied, the selection unit 21B selects edge information and depth to the subject as alignment information. This conditional branching corresponds to a case where the transmission band between the first imaging device 1 and the second imaging device 5 is not greater than or equal to the threshold value Th5 or the processing capability of the second imaging device 5 is not greater than or equal to the threshold value Th6.

[0160] There are several reasons why this alignment information selection is performed. Specifically, if conditions 3 or 4 above are not met, there is a high probability of transmission delays when the telephoto YC image is transmitted from the first imaging device 1 to the second imaging device 5, or a high probability of processing delays when the alignment unit 5C aligns the telephoto YC image with the wide-angle YC image. In this case, selecting edge information as alignment information reduces the amount of processing required compared to having the alignment unit 5C perform image alignment. Furthermore, since edge information has a smaller amount of information than the telephoto YC image, transmission delays from the first imaging device 1 to the second imaging device 5 can be minimized. Furthermore, by including the depth of the subject in the alignment information, it is possible for the alignment unit 5C to narrow the area for alignment with the wide-angle edge information to an area near that depth, or to weight the area near that depth more heavily than the area at depths other than that depth, and then have the alignment unit 5C perform alignment with the wide-angle edge information.

[0161] Note that when selecting an image near the focus as alignment information, the following branch can be set in the selection of alignment information based on whether the above conditions 3 and 4 are met. For example, when the above conditions 3 and 4 are met, the YC image near the focus and the depth to the subject are selected, while when the above conditions 3 or 4 are not met, the edge information near the focus and the depth to the subject can be selected. In addition, when something other than the depth map is selected as alignment information (for example, when the image near the focus, edge information, or YC image is selected), this is a case where the subject has small unevenness. In this case, depth information with a smaller amount of information than the depth map, such as the depth to the subject, can also be selected as alignment information.

[0162] <2-3. Processing Procedure of Information Processing Device 20>

[0163] Figure 6 1 is a flowchart illustrating the procedure of the information output process according to the second embodiment. As an example only, this process may be repeatedly performed for each frame of the image captured by the first imaging device 1 and the second imaging device 5.

[0164] like Figure 6 As shown in FIG, the acquisition unit 11 acquires imaging-related information related to imaging performed by the first imaging device 1 and the second imaging device 5, such as the above-mentioned a) to f) and the like (step S101).

[0165] Subsequently, the determination unit 21A determines whether the subject has concavities and convexities based on the depth distribution information about the subject detected from the telephoto YC image included in the imaging-related information acquired in step S101, as condition 1. For example, the determination unit 21A determines whether the ratio of the number of pixels within a predetermined depth range Th2 to the total number of pixels of the subject is within a predetermined threshold value Th1, as condition 1 (step S201).

[0166] At this time, when the above-described condition 1 is satisfied (i.e., when the ratio of the number of pixels in the depth range Th2 to the total number of pixels of the object is within the threshold value Th1) ("YES" in step S202), the determination unit 21A determines the following condition 2. That is, the determination unit 21A determines, as condition 2, whether the number of pixels whose blur amount exceeds a predetermined threshold value Th3 in an area other than the near-focus area focused by the AF function or the MF function in the telephoto YC image is within a predetermined threshold value Th4 (step S203).

[0167] Here, if the above-mentioned condition 2 is satisfied (i.e., the number of pixels whose blur amount exceeds the threshold value Th3 in the area other than the near-focus area is within the threshold value Th4) (No in step S203), the selection unit 21B selects the depth map as the alignment information (step S204). Meanwhile, if the above-mentioned condition 2 is not satisfied (i.e., the number of pixels whose blur amount exceeds the threshold value Th3 in the area other than the near-focus area is not within the threshold value Th4) (Yes in step S203), the selection unit 21B selects the near-focus image as the alignment information (step S205).

[0168] In addition, when either the above-mentioned condition 1 or the above-mentioned condition 2 is not satisfied ("No" in step S202 or "No" in step S203), the determination unit 21A determines whether the transmission frequency band between the first imaging device 1 and the second imaging device 5 is greater than or equal to the predetermined threshold Th5 as condition 3 (step S206).

[0169] Moreover, when condition 3 of the above-mentioned device is satisfied (i.e., when the transmission band is greater than or equal to the predetermined threshold Th5) ("Yes" in step S206), the determination unit 21A determines the processing capability of the second imaging device 5 (for example, whether the performance value of the processor such as the clock frequency or the number of cores is greater than or equal to the predetermined threshold Th6) as condition 4 (step S207).

[0170] At this time, if the above-mentioned condition 4 is further satisfied (i.e., if the processing capability of the second imaging device 5 is greater than or equal to the threshold value Th6) ("Yes" in step S207), the selection unit 21B selects the telephoto YC image and the depth to the subject as the alignment information (step S208). Meanwhile, if either the above-mentioned condition 3 or the above-mentioned condition 4 is not satisfied ("No" in step S206 or "No" in step S207), the selection unit 21B selects the edge information and the depth to the subject as the alignment information (step S209).

[0171] Here, when the image near the focus is selected as the alignment information in step S205, the following branch can be set in the selection of alignment information based on whether the above conditions 3 and 4 are met. For example, when the above conditions 3 and 4 are met, the YC image near the focus and the depth to the subject are selected, while when the above conditions 3 or 4 are not met, the edge information near the focus and the depth to the subject are selected.

[0172] Thereafter, the output unit 13 transmits the alignment information selected in step S204 , step S208 , or step S209 to the alignment unit 5C of the second imaging device 5 (step S103 ) and ends the process.

[0173] <2-4. One aspect of the effect>

[0174] As described above, the information processing device 20 according to the second embodiment selects one or more of the following as alignment information: the depth map, the image near focus, edge information, and the telephoto YC image, based on the presence or absence of unevenness in the subject, the amount of blur, the transmission bandwidth, and the processing power of the second imaging device 5. This makes it possible to select alignment information that is appropriate for the imaging situation of the first imaging device 1. Consequently, the information processing device 20 according to the second embodiment makes it possible to achieve alignment that strikes a balance between reducing processing effort and maintaining accuracy.

[0175] <<3. Third embodiment>>

[0176] In the embodiment of the present disclosure, an example of generating a telephoto YC image having the same resolution as that of a wide-angle YC image as alignment information during operation of the zoom function in the first imaging device 1 or the second imaging device 5 will be described.

[0177] Hereinafter, merely as an example, an example in which an image output from the first imaging device 1 to the second imaging device 5 is a telephoto YC image will be described.

[0178] <3-1. Functional Configuration Example of Information Processing Device 30>

[0179] Figure 7 3 is a block diagram illustrating a functional configuration example of the information processing apparatus 30 according to the third embodiment. Figure 7 As shown in FIG, the information processing device 30 is different in that the information processing device 30 includes a Figure 2 The processing contents of the decision unit 12 of the information processing apparatus 10 shown in FIG. 1 are partially different from the processing contents of the decision unit 31 .

[0180] <3-2. Decision Unit 31>

[0181] Figure 7 The decision unit 31 shown in FIG Figure 2 The difference between the decision unit 12 shown in FIG. 1 and FIG. 2 is that the decision unit 31 includes a calculation unit 31A and a generation unit 31B.

[0182] <3-2-1. Calculation Unit 31A>

[0183] The calculation unit 31A is a processing unit that calculates the number of pixels of the imaging sensor 5A of the second imaging device 5 included in the overlapping area where the angle of view of capturing an image by the second imaging device 5 and the angle of view of capturing an image by the first imaging device 1 overlap with each other.

[0184] Figure 8is a diagram illustrating an example of the correspondence relationship between focal length and angle of view. Figure 8 An example in which the optical axis of the first imaging device 1 and the optical axis of the second imaging device 2 coincide with each other is illustrated only as an example. Figure 8 As shown in , the viewing angle on the wide-angle side is greater than the viewing angle on the telephoto side. In this case, the number of pixels of the imaging sensor 5A of the second imaging device 5 is included in the overlapping area where the viewing angle on the wide-angle side and the viewing angle on the telephoto side overlap each other, and the shaded portion in the figure can be calculated based on the aspects of the imaging sensor 1A and the imaging sensor 5A. For example, the horizontal viewing angle on the wide-angle side can be calculated according to the following equation (1). In addition, the horizontal viewing angle on the telephoto side can be calculated according to the following equation (2). It can be seen from equations (1) and (2) that when the focal length on the telephoto side or the focal length on the wide-angle side changes, the number of pixels of the imaging sensor 5A included in the above-mentioned overlapping area (that is, the resolution of the overlapping area on the wide-angle side) changes.

[0185] Horizontal viewing angle on the wide-angle side = 2×arctan[horizontal width of imaging sensor 5A / 2 / focal length on the wide-angle side]···(1)

[0186] Horizontal angle of view on the telephoto side = 2×arctan[horizontal width of the imaging sensor 1A / 2 / focal length on the telephoto side]···(2)

[0187] As described above, when the zoom function is in operation, the calculation unit 31A recalculates the resolution of the overlapped area on the wide-angle side. For example, the calculation unit 31A calculates the resolution of the overlapped area on the wide-angle side based on the focal length on the telephoto side after zooming and the focal length on the wide-angle side after zooming. By embedding this resolution calculation during the operation of the zoom function, it is possible to omit processing when the zoom is not changed.

[0188] <3-2-2. Generation Unit 31B>

[0189] The generation unit 31B generates a telephoto YC image with a resolution that matches the resolution of the wide-angle overlap area calculated by the calculation unit 31A as alignment information. For example, when the zoom function is in operation, the generation unit 31B changes the resolution of the telephoto YC image output from the digital signal processing unit 1B to match the resolution of the wide-angle overlap area calculated by the calculation unit 31A. As a result, the telephoto YC image is magnified or reduced according to the zoom amount.

[0190] <3-3. Processing Procedure of Information Processing Device 30>

[0191] Figure 91 is a flowchart illustrating the process of information output processing according to the third embodiment. As an example only, this processing can be repeatedly performed for each frame of the image captured by the first imaging device 1 and the second imaging device 5. Note that, Figure 9 The figure shows an excerpt of the processing corresponding to step S102, that is, Figure 3 The processing performed by the decision unit 31 in the series of processing shown in .

[0192] like Figure 9 As shown in FIG, in the case where the zoom function is in operation ("YES" in step S301), the calculation unit 31A calculates the resolution of the overlapping area on the wide-angle side based on the focal length on the telephoto side after zooming and the focal length on the wide-angle side after zooming (step S302). Thereafter, the generation unit 31B generates a telephoto YC image obtained by changing the resolution of the telephoto YC image output from the digital signal processing unit 1B to a resolution that matches the resolution of the overlapping area on the wide-angle side calculated in step S302 as alignment information (step S304).

[0193] Meanwhile, if the zoom function is not operating ("No" in step S301), the number of pixels of the imaging sensor 5A included in the overlapped area (i.e., the resolution of the overlapped area on the wide-angle side) remains unchanged. Therefore, the calculation unit 31A fixes the recently changed resolution as is (step S303). Then, the generation unit 31B generates a telephoto YC image obtained by changing the resolution of the telephoto YC image output from the digital signal processing unit 1B to match the resolution of the overlapped area on the wide-angle side fixed in step S304 as alignment information (step S304).

[0194] The alignment information generated in this manner in step S304 is output to the alignment unit 5C of the second imaging device 5 .

[0195] <3-4. One aspect of the effect>

[0196] As described above, the information processing device 30 according to the third embodiment generates, as alignment information, a telephoto YC image with a resolution that matches the resolution of the wide-angle overlap area calculated based on the telephoto focal length after zooming and the wide-angle focal length after zooming. Therefore, even when the zoom function is in operation, the alignment unit 5C can perform alignment while maintaining a consistent ratio between the telephoto YC image and the wide-angle YC image. Consequently, the information processing device 30 according to the third embodiment narrows the search range for the transformation matrix and improves alignment accuracy.

[0197] <<4. Modifications>>

[0198] Hereinafter, modifications of the first to third embodiments will be described.

[0199] <4-1. Combination between Examples>

[0200] In the second and third embodiments, examples of implementing both the second and third embodiments separately were described, but the second and third embodiments can also be implemented in combination. In this case, the depth map, the image near the focus, and the edge information selected as alignment information in the second embodiment can also be changed to a resolution that matches the resolution of the overlapping area on the wide-angle side. Just as an example, Figure 9 The flowchart shown in Figure 6 As another example, Figure 9 The flowchart shown in Figure 6 The method is implemented in step S204, step S205, step S208 or step S209 shown in FIG.

[0201] <4-2. Targeting Execution Body>

[0202] In the first to third embodiments, an example has been described in which the alignment of the alignment unit 5C is performed by the second device 5, but the alignment may also be performed by the first imaging device 1. In this case, it is only required to transmit the position of the frame of the telephoto YC image on the wide-angle YC image from the first imaging device 1 to the second imaging device 5. For example, Figure 6 In the processing of the decision unit 21 shown in FIG, the wide-angle side is replaced with the telephoto side when reading, and the telephoto side is replaced with the wide-angle side when reading. This allows the wide-angle depth map, the wide-angle image near the focus point, and the wide-angle edge information to be transmitted as alignment information. Using this alignment information, the first imaging device 1 performs alignment and transmits the position of the telephoto YC image frame on the wide-angle YC image to the second imaging device 5. In this case, by performing edge extraction or focus point extraction from the image acquired by the first imaging device 1 during alignment, alignment can be achieved at high speed.

[0203] <4-3. Other Modifications>

[0204] Furthermore, in the respective processes described in the above embodiments, all or part of the processes described as being automatically performed may be performed manually, or all or part of the processes described as being manually performed may be automatically performed by known methods. Furthermore, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be optionally changed. For example, the various types of information illustrated in each figure are not limited to the information illustrated.

[0205] In addition, each component of each device and apparatus shown in the drawings is conceptual in function and does not necessarily need to be physically configured as shown in the drawings. That is, the specific form of distribution and integration of each device and apparatus is not limited to the form shown in the drawings, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0206] Furthermore, the effects of each embodiment described in this specification are merely exemplary and not restrictive, and other effects may exist.

[0207] <<5. Hardware Configuration>>

[0208] For example, by having Figure 10 The computer 1000 of the configuration shown in embodies the information processing apparatus 10, 20, or 30 according to each embodiment described above. Hereinafter, the information processing apparatus 10, 20, or 30 according to the above-described embodiment will be exemplified and described. Figure 10 10 is a hardware configuration diagram illustrating an example of a computer 1000. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input and output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0209] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 in the RAM 1200 and executes processing corresponding to the various programs.

[0210] The ROM 1300 stores programs such as a boot program of a basic input and output system (BIOS) executed by the CPU 1100 when the computer 1000 starts up, programs depending on the hardware of the computer 1000 , and the like.

[0211] The HDD 1400 is a computer-readable recording medium that non-transitorily records a program to be executed by the CPU 1100 , data to be used by the program, etc. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of program data 1450 .

[0212] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0213] The input and output interface 1600 is an interface for connecting the input and output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input and output interface 1600. In addition, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input and output interface 1600. In addition, the input and output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase-change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0214] For example, when the computer 1000 is used as the information processing apparatus 10, 20, or 30 according to the above-described embodiment, the CPU 1100 of the computer 1000 implements each functional unit included in the control unit 15 by executing the information processing program loaded on the RAM 1200. In addition, the HDD 1400 stores the information processing program according to the present disclosure and the data in the content storage unit 121. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data 1450, but as another example, these programs may be acquired from another device via the external network 1550.

[0215] Note that the present technology can also have the following configurations.

[0216] (1) An information processing device comprising:

[0217] a decision unit configured to decide, based on imaging-related information related to imaging performed by the first imaging device, alignment information to be used for alignment between a first image captured by the first imaging device and a second image captured by a second imaging device having a wider angle of view than that of the first imaging device; and

[0218] The output unit is configured to output the alignment information.

[0219] (2) The information processing device according to (1), wherein

[0220] The decision unit is configured to decide alignment information to be used for alignment with the second image, and

[0221] The output unit is configured to output the alignment information to the second imaging device.

[0222] (3) The information processing device according to (2), wherein

[0223] The decision-making units include:

[0224] A determining unit configured to determine whether the imaging-related information satisfies a predetermined condition; and

[0225] A selection unit is configured to select at least one of a depth map corresponding to the first image, an image near the focus corresponding to the vicinity of the subject focused by adjustment control of a predetermined focal length in the first image, edge information detected from the first image, or the first image based on the determination result of the determination unit.

[0226] (4) The information processing device according to (3), wherein

[0227] The determining unit is configured to determine whether the subject has concavity and convexity based on the depth distribution information of the subject in the first image, and

[0228] The selection unit is configured to select the depth map as the alignment information if the subject has concavity and convexity.

[0229] (5) The information processing device according to (4), wherein

[0230] The determination unit is configured to determine whether the object has concavity and convexity based on whether a ratio of pixels within a predetermined depth range to a total number of pixels of the object is within a predetermined threshold.

[0231] (6) The information processing device according to (3) to (5), wherein

[0232] The determination unit is configured to determine whether the number of pixels whose blur amount exceeds a predetermined threshold in an area other than the near focus area focused by the adjustment control of the focal length is within a predetermined threshold, and

[0233] The selection unit is configured to select the image near the focus when the number of pixels having a blur amount exceeding the threshold in the area other than the focus near the focus is not within the threshold.

[0234] (7) The information processing device according to (3) to (6), wherein

[0235] The determining unit is configured to determine whether a transmission frequency band between the first imaging device and the second imaging device is greater than or equal to a predetermined threshold, and

[0236] The selection unit is configured to select the first image if the transmission band is greater than or equal to a threshold value, and select the edge information if the transmission band is not greater than or equal to the threshold value.

[0237] (8) The information processing device according to (3) to (7), wherein

[0238] The determining unit is configured to determine whether the processing capability of the second imaging device is greater than or equal to a predetermined threshold, and

[0239] The selection unit is configured to select the first image if the processing capability is greater than or equal to a threshold value, and select the edge information if the processing capability is not greater than or equal to the threshold value.

[0240] (9) The information processing device according to (3) to (8), wherein

[0241] The selection unit is configured to further select the depth to the subject as the alignment information in addition to the depth map.

[0242] (10) The information processing device according to (2) to (9), wherein

[0243] The decision-making units include:

[0244] a calculation unit configured to calculate, as a resolution of the overlapping area, the number of pixels of the imaging sensor of the second imaging device included in an overlapping area where the angle of view of the first imaging device and the angle of view of the second imaging device overlap with each other; and

[0245] A generating unit is configured to generate, as alignment information, a first image obtained by changing a resolution of the first image to a resolution matching a resolution of the overlapping area.

[0246] (11) The information processing device according to (10), wherein

[0247] The calculation unit is configured to calculate a resolution of the overlapping area when a zoom function is operated in the first imaging device or the second imaging device.

[0248] (12) The information processing device according to (1), wherein

[0249] The decision unit is configured to decide alignment information to be used for alignment with the first image, and

[0250] The output unit is configured to output the alignment information to the first imaging device.

[0251] (13) An information processing method that causes a computer to execute the following processing:

[0252] deciding, based on imaging-related information related to imaging performed by the first imaging device, alignment information to be used for alignment between a first image captured by the first imaging device and a second image captured by a second imaging device having a wider angle of view than the first imaging device; and

[0253] Output alignment information.

[0254] (14) An information processing program that causes a computer to execute the following processing:

[0255] deciding, based on imaging-related information related to imaging performed by the first imaging device, alignment information to be used for alignment between a first image captured by the first imaging device and a second image captured by a second imaging device having a wider angle of view than the first imaging device; and

[0256] Output alignment information.

[0257] Reference Signs List

[0258] 1First imaging device

[0259] 1A imaging sensor

[0260] 1B digital signal processing unit

[0261] 1C display unit

[0262] 5. Second imaging device

[0263] 5A imaging sensor

[0264] 5B digital signal processing unit

[0265] 5C alignment unit

[0266] 5D display unit

[0267] 10Information processing device

[0268] 11 Acquisition Unit

[0269] 12 Decision Unit

[0270] 13 output units

Claims

1. An information processing device, comprising: a decision unit configured to decide, based on imaging-related information related to imaging performed by the first imaging device, alignment information to be used for alignment between a first image captured by the first imaging device and a second image captured by a second imaging device having a wider angle of view than that of the first imaging device, wherein the alignment information is to be used for alignment with the second image; as well as The output unit is configured to output the alignment information to the second imaging device.

2. The information processing apparatus according to claim 1, wherein The decision-making units include: a determining unit configured to determine whether the imaging-related information satisfies a predetermined condition; as well as A selection unit is configured to select at least one of a depth map corresponding to the first image, an image near the focus corresponding to the vicinity of the subject focused by adjustment control of a predetermined focal length in the first image, edge information detected from the first image, or the first image based on the determination result of the determination unit.

3. The information processing apparatus according to claim 2, wherein The determining unit is configured to determine whether the subject has concavity and convexity based on the depth distribution information of the subject in the first image, and The selection unit is configured to select the depth map as the alignment information if the subject has concavity and convexity.

4. The information processing apparatus according to claim 3, wherein The determination unit is configured to determine whether the object has concavity and convexity based on whether a ratio of pixels within a predetermined depth range to a total number of pixels of the object is within a predetermined threshold. The information processing apparatus according to claim 2 , wherein The determination unit is configured to determine whether the number of pixels having a blur amount exceeding a predetermined threshold in an area other than the near-focus area focused by the adjustment control of the focal length is within a predetermined threshold, and The selection unit is configured to select the image near the focus when the number of pixels having a blur amount exceeding the threshold in the area other than the focus near the focus is not within the threshold. The information processing apparatus according to claim 2 , wherein The determining unit is configured to determine whether a transmission frequency band between the first imaging device and the second imaging device is greater than or equal to a predetermined threshold, and The selection unit is configured to select the first image if the transmission band is greater than or equal to a threshold value, and select the edge information if the transmission band is not greater than or equal to the threshold value.

7. The information processing apparatus according to claim 2, wherein The determining unit is configured to determine whether the processing capability of the second imaging device is greater than or equal to a predetermined threshold, and The selection unit is configured to select the first image if the processing capability is greater than or equal to a threshold value, and select the edge information if the processing capability is not greater than or equal to the threshold value. The information processing apparatus according to claim 2 , wherein The selection unit is configured to further select the depth to the subject as the alignment information in addition to the depth map.

9. The information processing apparatus according to claim 2, wherein The decision-making units include: a calculation unit configured to calculate, as a resolution of the overlapping area, the number of pixels of the imaging sensor of the second imaging device included in an overlapping area where the angle of view of the first imaging device and the angle of view of the second imaging device overlap with each other; as well as A generating unit is configured to generate, as alignment information, a first image obtained by changing a resolution of the first image to a resolution matching a resolution of the overlapping area.

10. The information processing apparatus according to claim 9, wherein The calculation unit is configured to calculate a resolution of the overlapping area when a zoom function is operated in the first imaging device or the second imaging device. The information processing apparatus according to claim 1 , wherein The decision unit is configured to decide alignment information to be used for alignment with the first image, and The output unit is configured to output the alignment information to the first imaging device.

12. An information processing method for causing a computer to perform the following processing: deciding, based on imaging-related information related to imaging performed by the first imaging device, alignment information to be used for alignment between a first image captured by the first imaging device and a second image captured by a second imaging device having a wider angle of view than that of the first imaging device, wherein the alignment information is to be used for alignment with the second image; and The alignment information is output to a second imaging device.

13. A recording medium storing an information processing program, wherein when the information processing program is executed by a computer, the computer is caused to execute the following processing: deciding, based on imaging-related information related to imaging performed by the first imaging device, alignment information to be used for alignment between a first image captured by the first imaging device and a second image captured by a second imaging device having a wider angle of view than that of the first imaging device, wherein the alignment information is to be used for alignment with the second image; and The alignment information is output to a second imaging device.

Images