Information generation apparatus, imaging apparatus, information generation method, and program

Abstract

To provide an information generation apparatus and an information generation method which suppress deterioration of focus accuracy.SOLUTION: An imaging apparatus as the information generation apparatus comprises: a first imaging information acquisition unit which acquires imaging information indicating an output from an imaging element, the imaging information being first imaging information indicating an output from the imaging element having imaged an object including a shielded area; and a second imaging information generation unit which generates second imaging information obtained by removing information on the shielded area from the first imaging information.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information generating device, an imaging device, an information generating method, and a program. [Background technology]

[0002] There is known an autofocus technique for automatically focusing on a subject in an imaging device such as a camera. However, when an object such as a splash of water is present between the subject and the imaging device, the autofocus may focus on the object, resulting in a decrease in focus adjustment accuracy. As a technique for preventing the focus adjustment accuracy from decreasing, for example, a technique is known in which the area of ​​the splash of water within the angle of view is determined, and the determined area is not used in the autofocus, or the determined area is interpolated (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-125743 A Summary of the Invention

[0004] One aspect of the present invention is an information generating device that includes a first imaging information acquisition unit that acquires first imaging information indicating output from an imaging element, the first imaging information indicating output from the imaging element that has imaged an object including an obstructed area, and a second imaging information generation unit that generates second imaging information in which information about the obstructed area has been removed from the first imaging information.

[0005] One aspect of the present invention is an imaging device including the above-described information generating device and an imaging optical system.

[0006] One aspect of the present invention is an information generation method including a first imaging information acquisition step of acquiring first imaging information indicating an output from an imaging element, the first imaging information indicating an output from the imaging element that has imaged an object including an obstructed area, and a second imaging generation step of generating second imaging information in which information about the obstructed area has been removed from the first imaging information.

[0007] One aspect of the present invention is a program for causing a computer to execute a first imaging information acquisition step of acquiring first imaging information indicating output from an imaging element, the first imaging information indicating output from the imaging element that has imaged an object including an obstructed area, and a second imaging generation step of generating second imaging information in which information about the obstructed area has been removed from the first imaging information. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram showing an example of an overview of an information generating method according to the first embodiment. [Diagram 2] 1 is a diagram illustrating an example of a hardware configuration of an imaging device 1 according to a first embodiment. [Diagram 3] 1 is a diagram illustrating an example of a functional configuration of an imaging device 1 according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an overview of autofocus processing according to the first embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging device 1a according to a first example of the first embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a learning process and an inference process according to a first example of the first embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of a functional configuration of an imaging device 1b according to a second example of the first embodiment. [Figure 8] 5A to 5C are diagrams illustrating an example of a learning process and an inference process according to the first embodiment. [Figure 9] FIG. 11 is a diagram illustrating an example of a functional configuration of an imaging device 1c according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (First embodiment) Hereinafter, the first embodiment will be described in detail with reference to the drawings. Fig. 1 is a diagram showing an example of an overview of an information generating method according to the present embodiment. In the present embodiment, the generated information is, for example, an image captured by an imaging device 1.

[0010] In the captured image P1, a shielded area X2 is captured together with the subject X1. As an example, the subject X1 is a person. As an example, the shielded area X2 is a splash of water. Therefore, the shielded area X2 exists between the subject X1 and the imaging device 1. In this embodiment, the shielded area is an area in the captured image in which an object that is closer to the imaging device 1 than the subject X1 and hides the subject X1 that is located behind is captured. In the captured image P1, the shielded area X2 that is closer to the imaging device 1 than the subject X1 hides the subject X1 that is located behind. Since the captured image P1 includes the shielded area X2, when the imaging device 1 performs AF (autofocus), the focus may be on the shielded area X2 instead of the subject X1.

[0011] Therefore, the imaging device 1 generates a captured image P2 from the captured image P1 based on the information generating method according to this embodiment. The captured image P2 is an image in which the occluded area X2 has been removed from the captured image P1. That is, in the captured image P2, the person is captured with the water splashes removed. Since the captured image P2 does not include the occluded area X2, the imaging device 1 can focus on the subject X1 by AF. The captured image P1 is an example of a first captured image. The captured image P2 is an example of a second captured image.

[0012] 2 is a diagram showing an example of a hardware configuration of the imaging device 1 according to this embodiment. The imaging device 1 captures still images or videos. The imaging device 1 is, as an example, a digital camera. Note that the imaging device 1 may also be a digital video camera, a camera mounted on a smartphone, or the like.

[0013] The imaging device 1 includes a processor 2, a storage device 3, an imaging unit 4, an LSI 5, an operation unit 6, a sensor 7, a display device 8, a communication I / F 9, and a bus 10. The processor 2, the storage device 3, the imaging unit 4, the LSI 5, the operation unit 6, the sensor 7, the display device 8, and the communication I / F 9 are connected to each other by the bus 10.

[0014] The processor 2 performs various controls on the imaging device 1. The processor 2 is, for example, a CPU (Central Processing Unit). The storage device 3 is a non-transient or temporary recording medium that stores various programs and data. The storage device 3 is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc. Note that a plurality of storage devices 3 may be implemented in the imaging device 1. At least one of the plurality of implemented storage devices 3 may be detachable from the imaging device 1.

[0015] The imaging unit 4 performs imaging under the control of the processor 2. The imaging unit 4 includes a drive unit 40, an AFE (Analog Front End) 41, a photographing optical system 42, and an imaging element 43.

[0016] The drive unit 40 drives and controls the photographing optical system 42. The drive unit 40 has a drive circuit and a drive source. The drive circuit controls the drive source according to instructions from the processor 2. The drive source is, for example, a motor. The drive source is controlled by the drive circuit to move the zoom lens 421 and the focusing lens 423 in the photographing optical system 42 in the optical axis direction. In addition, the drive source is controlled by the drive circuit to open and close the aperture 422 in the photographing optical system 42.

[0017] The photographing optical system 42 includes a plurality of lenses arranged in the optical axis direction and an aperture 422. As an example, the plurality of lenses are configured with a lens 420, a zooming lens 421, and a focusing lens 423. Note that the configuration of the plurality of lenses is not limited to this. The photographing optical system 42 collects subject light and outputs it to the image sensor 43.

[0018] The imaging element 43 receives subject light from the photographing optical system 42 and converts it into an electrical signal. The imaging element 43 is, for example, an XY address type solid-state imaging element (for example, a CMOS (Complementary Metal-Oxide Semiconductor)). The imaging element 43 may also be a sequential scan type solid-state imaging element (for example, a CCD (Charge Coupled Device)).

[0019] A plurality of light receiving elements (pixels) are arranged in a matrix on the light receiving surface of the image sensor 43. A plurality of types of color filters, each of which transmits light of a different color component, are arranged in a predetermined color array (e.g., a Bayer array) on the pixels of the image sensor 43. Each pixel of the image sensor 43 outputs an analog electrical signal corresponding to each color component through color separation by the color filter.

[0020] The AFE 41 is an analog front-end circuit that performs signal processing on an analog electric signal from the image sensor 43. The AFE 41 sequentially performs gain adjustment of the electric signal, analog signal processing (correlated double sampling, black level correction, etc.), A / D conversion processing, and digital signal processing (defective pixel correction, etc.) to generate RAW image data and output it to the LSI 5.

[0021] The LSI 5 is an integrated circuit that executes specific processing on the RAW image data output from the AFE 41. The specific processing includes image processing such as color interpolation, white balance adjustment, edge enhancement, gamma correction, and gradation conversion, as well as encoding processing, decoding processing, and compression / expansion processing. The LSI 5 is realized by, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a PLD (Programmable Logic Device).

[0022] The operation unit 6 receives various operations from a user and includes, for example, various buttons including a release button, switches, dials, and a touch panel.

[0023] The sensor 7 is a device that detects information, and includes, for example, an automatic focus (AF) sensor, an automatic exposure (AE) sensor, a gyro sensor, an acceleration sensor, and a temperature sensor.

[0024] The display device 8 displays image data and a setting screen. The display device 8 includes a rear monitor provided on the rear surface of the imaging device 1 and an electronic viewfinder. The communication I / F 9 connects to a network and transmits and receives data.

[0025] 3 is a diagram showing an example of a functional configuration of the imaging device 1 according to this embodiment. The imaging device 1 includes an imaging section 4, a control section 11, and a storage section 12. The imaging section 4 corresponds to the imaging section 4 shown in FIG. 2. The control section 11 is realized by the processor 2 shown in FIG. 2. The storage section 12 is realized by the storage device 3 shown in FIG. 2.

[0026] The imaging unit 4 captures a first captured image A1. The first captured image A1 is a captured image showing an output from the imaging element 43. The first captured image A1 is a captured image showing an output from the imaging element 43 that has captured an image of a target including a blocked area. Note that the first captured image A1 may be a captured image showing an output from the imaging element 43 that has captured an image of a target not including a blocked area.

[0027] The control unit 11 includes an image generating unit 110 and a focusing unit 113 . The image generating section 110 includes a first captured image acquiring section 111 and a second captured image generating section 112 . The first captured image acquisition section 111 acquires a first captured image A1 from the imaging element 43 . The second captured image generating section 112 generates a second captured image B1 by removing the image of the masked area from the first captured image A1. If the image of the masked area is not included in the first captured image A1, the second captured image generating section 112 reproduces the original first captured image A1 as the second captured image B1.

[0028] The second captured image generating unit 112 generates the second captured image B1 from the first captured image A1 using the learning model C1. That is, the second captured image B1 is generated from the first captured image A1 based on machine learning. The generation of the second captured image B1 from the first captured image A1 based on machine learning is also referred to as inferring the second captured image B1. The learning model C1 is stored in advance in the storage unit 12. Details of the learning model C1 will be described later. The learning model will also be referred to as a trained model.

[0029] The focusing section 113 has an AF function. The focusing section 113 performs focusing based on the second captured image B1 generated by the second captured image generation section 112. The focusing section 113 performs focusing by causing the drive section 40 to drive the focusing lens 423.

[0030] The storage unit 12 stores various types of information. For example, the storage unit 12 stores a learning model C1. The storage unit 12 also stores optical information for the focusing unit 113 to perform AF.

[0031] Next, an overview of the AF process performed by the imaging device 1 will be described with reference to FIG. Fig. 4 is a diagram showing an example of an outline of AF processing according to this embodiment. The AF processing is started, for example, when a setting to perform AF is made and the shutter button included in the operation unit 6 is half-pressed. The AF processing shown in Fig. 4 is executed by the control unit 11. Note that the AF processing may also be started when the shutter button is fully pressed.

[0032] Step S10: The first captured image acquisition section 111 acquires the first captured image A1. The first captured image acquisition section 111 outputs the acquired first captured image A1 to the second captured image generation section 112.

[0033] Step S20: The second captured image generation unit 112 generates the second captured image B1 by using the learning model C1. Here, the second captured image generation unit 112 reads out the learning model C1 from the storage unit 12. The second captured image generation unit 112 outputs the generated second captured image B1 to the focusing unit 113.

[0034] Step S30: The focusing unit 113 performs AF based on the second captured image B1 generated by the second captured image generating unit 112 and optical information for performing AF. With this, the control unit 11 ends the AF process.

[0035] (First embodiment) In the following, an example in which the learning model C1 is a learning model based on Principal Component Analysis (PCA) will be described. The imaging device 1 according to this embodiment is referred to as an imaging device 1a. The imaging device 1a is an example of the imaging device 1.

[0036] 5 is a diagram showing an example of a functional configuration of an image capturing device 1a according to this embodiment. The image capturing device 1a includes an image capturing unit 4, a control unit 11a, and a storage unit 12. The control unit 11a includes an image generating unit 110a and a focusing unit 113. The image generating section 110a includes a first captured image acquiring section 111 and a second captured image generating section 112a.

[0037] Comparing the imaging device 1a (FIG. 5) according to this embodiment with the imaging device 1 (FIG. 3) according to the first embodiment, the imaging device 1a is different in that it includes a second captured image generating unit 112a as the second captured image generating unit 112, and that a learning model C1a is used as the learning model C1. Here, the functions of the other components (the first captured image acquiring unit 111 and the focusing unit 113) are the same as those of the first embodiment. A description of the same functions as those of the first embodiment will be omitted, and in this embodiment, the description will be centered on the parts that are different from the first embodiment.

[0038] The learning model C1a is trained to generate first conversion information C2 based on principal component analysis from a first feature vector of a first dimension extracted from the image D1 without an occluded area by using at least a predetermined number of images D1 without an occluded area. The first feature vector is a vector indicating a predetermined number of feature amounts extracted from the image D1 without an occluded area. The first conversion information C2 indicates a first conversion for obtaining a second feature vector of a second dimension smaller than the number of dimensions of the first dimension. A part of the dimensions of the first feature vector is reduced to obtain the second feature vector. The second feature vector is a vector indicating a principal component in the space of feature amounts.

[0039] In this embodiment, the first transformation is a linear transformation that maps a first feature vector of a first dimension to a second feature vector of a second dimension that is smaller than the number of dimensions of the first dimension. The first transformation information C2 is expressed, for example, by a matrix indicating the first transformation.

[0040] The image D1 without the occluded area shows an output from the image sensor 43 capturing an image of a target that does not include an occluded area. In this embodiment, the image D1 without the occluded area is a captured image in which no water splashes are captured. The image D1 without the occluded area is an example of the third captured information.

[0041] The second captured image generating unit 112a includes a conversion information acquiring unit 114a. The conversion information acquiring unit 114a acquires second conversion information. The second conversion information indicates a second conversion generated to reconstruct a first feature vector from a second feature vector obtained by the first conversion. The second conversion information is included in the learning model C1a. The second conversion is an inverse conversion that reproduces a first feature vector of a first dimension from a second feature vector of a second dimension. The second conversion information is expressed, for example, by a matrix indicating the second conversion. The second conversion is approximately obtained as an inverse conversion of the first conversion.

[0042] The second captured image generating unit 112a generates a second feature vector from the first feature vector extracted from the first captured image A1 based on the first transformation. The second captured image generating unit 112a reconstructs the first feature vector from the generated second feature vector based on the second transformation indicated by the second transformation information. The second captured image generating unit 112a generates a second captured image B1 from the reconstructed first feature vector.

[0043] Here, the learning process and the inference process will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the learning process and the inference process according to this embodiment. In the learning process, a learning model C1a is generated based on principal component analysis. In the inference process, the second captured image B1 is inferred from the first captured image A1 based on the generated learning model C1a. In the description of Fig. 6, a case will be described in which the occluded region is an image of a water splash, as an example.

[0044] First, the learning process will be described. The learning process is executed before the imaging device 1a is used by a user, such as before being shipped from a factory. The learning process is executed by a learning device 20a (not shown). The learning device 20a is, for example, a computer such as a personal computer (PC) or a server.

[0045] Step S100: The learning device 20a acquires an image group D10 without an occluded region. The image group D10 without an occluded region is a collection of a predetermined number or more of images D1 without an occluded region. As described above, the image D1 without an occluded region is a captured image in which no splashing water is captured. It is preferable that the feature quantities of various types of images other than the feature quantities of the image of the splashing water are extracted by the principal component analysis. For this reason, in the image group D10 without an occluded region, it is preferable that various types of subjects other than splashing water are captured at various angles and various zooms in each of the predetermined number or more of images D1 without an occluded region. The various types of subjects other than splashing water are, for example, people, animals, plants, objects, buildings, and landscapes that can be captured by the imaging device 1.

[0046] The feature amounts extracted from the captured image include, for example, the number of pixels of each color, histograms of each RGB channel, and responses of various linear filters.

[0047] Step S110: The learning device 20a executes principal component analysis using the group of images without occlusion regions D10. Here, the learning device 20a extracts a first feature vector of a first dimension from each of the images without occlusion regions D1 included in the group of images without occlusion regions D10. The number of first dimensions is the number of types of features to be extracted.

[0048] The learning device 20a executes a principal component analysis on the extracted first feature vectors. The learning device 20a selects a predetermined number or more of dimensions of principal components from the dimensions obtained as a result of the principal component analysis based on the cumulative contribution rate. For example, the learning device 20a selects a predetermined number of dimensions of principal components in descending order of the cumulative contribution rate. Alternatively, the learning device 20a may select dimensions of principal components whose cumulative contribution rate is higher than a predetermined value. In either case where a predetermined number of dimensions of principal components are selected in descending order of the cumulative contribution rate or where dimensions of principal components whose cumulative contribution rate is higher than a predetermined value are selected, it is preferable to select a sufficiently large number of dimensions as the dimensions of the principal components.

[0049] The learning device 20a obtains first conversion information C2 as a result of the principal component analysis. The learning device 20a supplies the first conversion information C2 to the imaging device 1a as a learning model C1a. The imaging device 1a stores the supplied first conversion information C2 in the storage unit 12 as the learning model C1a. Therefore, the learning model C1a is stored in the storage unit 12 in advance.

[0050] Next, the inference process will be described. The inference process is executed by the imaging device 1a when the shutter button included in the operation unit 6 is half-pressed during imaging.

[0051] Step S200: The first captured image acquisition unit 111 acquires a first captured image A1 from the image sensor 43. Note that the first captured image A1 may or may not include an image of the water splash, which is a blocked area.

[0052] Step S210: The second captured image generating unit 112a generates a second feature vector from the first feature vector extracted from the first captured image A1 based on the first conversion indicated by the first conversion information C2. Here, the second captured image generating unit 112a first extracts a predetermined number of feature amounts from the first captured image A1 based on image recognition. The second captured image generating unit 112a generates a first feature vector V1 from the extracted feature amounts.

[0053] The second captured image generating unit 112a converts the generated first feature vector V1 into a second feature vector V2 based on a first conversion. The second feature vector V2 indicating the principal components is generated by the first conversion. In the second feature vector V2, the image features of the splash of water, which is an occluded region, are reduced.

[0054] Step S220: The second captured image generating unit 112a reconstructs the first feature vector V1 from the generated second feature vector V2 based on the second transformation indicated by the second transformation information. The second captured image generating unit 112a generates a second captured image B1 from the reconstructed first feature vector V1. In the second captured image B1, the water splash area is deleted. The second captured image B1 is used, for example, by the focusing unit 113 to perform focusing.

[0055] In addition, the reconstructed first feature vector V1 may have fewer features other than the occluded region, the water splash, compared to the first feature vector V1 extracted from the first captured image A1. However, if a sufficiently large number of dimensions is selected as the dimension of the principal component, the second captured image B1 generated from the extracted first feature vector V1 will be sufficiently similar to an image in which the occluded region, the water splash, has been deleted from the original first captured image A1. With this, the imaging device 1a ends the inference process.

[0056] In the imaging device 1a according to this embodiment, the second captured image B1 is generated using a learning model C1a based on principal component analysis, so the amount of data required for learning is smaller than that required for deep learning or the like.

[0057] (Second Example) In the following, an example in which the learning model C1 is a learning model based on deep learning will be described. The imaging device 1 according to this embodiment is referred to as an imaging device 1b. The imaging device 1b is an example of the imaging device 1.

[0058] 7 is a diagram showing an example of a functional configuration of an image capturing device 1b according to this embodiment. The image capturing device 1b includes an image capturing unit 4, a control unit 11b, and a storage unit 12. The control unit 11b includes an image generating unit 110b and a focusing unit 113. The image generating section 110b includes a first captured image acquiring section 111 and a second captured image generating section 112b.

[0059] Comparing the imaging device 1b according to this embodiment (FIG. 7) with the imaging device 1 according to the first embodiment (FIG. 3), the imaging device 1b is different in that it includes a second captured image generating unit 112b as the second captured image generating unit 112, and that a learning model C1b is used as the learning model C1. Here, the functions of the other components (the first captured image acquiring unit 111 and the focusing unit 113) are the same as those of the first embodiment. A description of the same functions as those of the first embodiment will be omitted, and in this embodiment, the description will be centered on the parts that are different from the first embodiment.

[0060] The learning model C1b is a learning model based on deep learning. As an example, the learning model C1b is an autoencoder. The learning model C1b is a model trained using a first captured image for learning and a second captured image for learning, and is trained to output a second captured image for learning when the first captured image for learning is input. Here, the first captured image for learning is a captured image showing an output from the image sensor 43 that captures an image of a learning object including a learning occlusion region. The second captured image for learning is a captured image in which an image of the learning occlusion region is removed from the first captured image for learning. In this embodiment, the learning occlusion region is a region of water splashes in the image.

[0061] In addition, the learning model C1b may be a variational autoencoder (VAE), a generative adversarial network, a diffusion model, etc., other than the autoencoder. The generative adversarial network is, for example, CycleGAN.

[0062] The second captured image generating unit 112b generates the second captured image B1 from the first captured image A1 based on the learning model C1b.

[0063] Here, the learning process and the inference process will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of the learning process and the inference process according to this embodiment. In the learning process, a learning model C1b is generated based on deep learning. In the inference process, the second captured image B1 is inferred from the first captured image A1 based on the generated learning model C1b. In the description of FIG. 8, a case will be described where the occluded region is an image of a water splash, as an example.

[0064] First, the learning process will be described. The learning process is executed before the imaging device 1b is used by a user, such as before being shipped from a factory. The learning process is executed by a learning device 20b (not shown). The learning device 20b is, for example, a computer such as a PC or a server.

[0065] Step S300: The learning device 20b acquires a learning image pair D20. The learning image pair D20 is an image group including a predetermined number or more pairs in which an image with a masked area D21 and an image without a masked area D22 are associated with each other. The image with a masked area D21 is an image captured of a certain subject in a state where there is water splash in front of the subject. The image without a masked area D22 is an image captured of the same subject as the subject captured in the corresponding image with a masked area D21 in a state where there is no water splash. It is preferable that various types of subjects are captured between the pairs included in the learning image pair D20.

[0066] The learning image pair D20 is prepared in advance. The image without occlusion region D22 is created by, for example, deleting the water splash region in the image with occlusion region D21 using an image processing technique such as inpainting. The image with a masked region D21 is an example of a first captured image for learning, and the image without a masked region D22 is an example of a second captured image for learning.

[0067] Step S310: The learning device 20b executes deep learning based on the learning model C2b. When the image with occluded area D21 is input, the learning model C2b outputs an estimated image without occluded area D23. The learning device 20b executes learning of the learning model C2b so that the image without occluded area D22, which is paired with the input image with occluded area D21, and the estimated image without occluded area D23 are similar. Here, an image is similar to another image when, for example, the difference between pixels between the image and the other image is smaller than a predetermined value.

[0068] Therefore, the learning device 20b executes learning so that when an image with a masked area D21 is input to the learning model C2b, the learning model C2b outputs an image without a masked area D22 that forms a pair with the image with a masked area D21.

[0069] As a result of the learning, the learning device 20b obtains a learning model C1b. The learning model C1b is a learning model C2b for which learning has been executed. The learning device 20b supplies the learning model C1b to the imaging device 1b. The imaging device 1b stores the supplied learning model C1b in the storage unit 12. Therefore, the learning model C1b is stored in advance in the storage unit 12.

[0070] Next, the inference process will be described. The inference process is executed by the imaging device 1b when the shutter button included in the operation unit 6 is half-pressed during imaging.

[0071] Step S400: The first captured image acquisition unit 111 acquires a first captured image A1 from the image sensor 43. Note that the first captured image A1 may or may not include an image of the water splash, which is a blocked area.

[0072] Step S410: The second captured image generating unit 112b generates the second captured image B1 from the first captured image A1 based on the learning model C1b. Here, the second captured image generating unit 112b inputs the first captured image A1 to the learning model C1b, thereby causing the learning model C1b to output the second captured image B1. With this, the imaging device 1b ends the inference process.

[0073] In the imaging device 1b according to the present embodiment, the second captured image B1 is generated using a learning model C1b based on deep learning, so there is no need to extract features from the first captured image A1. Note that in deep learning, the features of an image are considered to be extracted in a multi-layer neural network of deep learning.

[0074] In the above-described embodiment and example, the first captured image A1 is described as an example of the imaging information indicating the output from the imaging element. That is, an example of the imaging information being an image is described, but the present invention is not limited to this. The imaging information may be information that is output from the imaging element 43 and is not configured as an image. The imaging information may also be information detected by the sensor 7. The information detected by the sensor 7 is, for example, information on the distance for each ranging point used in a phase difference detection method.

[0075] When the imaging information is sensor information, the learning method is the same as when the imaging information is an image. When using principal component analysis as in the first embodiment, the feature amount extracted as the first feature vector is appropriately changed from the feature amount in the case of an image.

[0076] In the present embodiment and examples, an example in which the occluded region is a water splash has been described, but the present invention is not limited to this. The occluded region may be rain, snow, sand, mud, dust, or the like. In addition, the occluded region may include a plurality of types of objects. For example, the occluded region may be at least one of water splash, rain, snow, sand, mud, and dust. Additionally, an occluding region may be any object that hides the subject.

[0077] As described above, the information generating device of this embodiment (in this embodiment, the imaging device 1) includes a first imaging information acquisition section (in this embodiment, the first captured image acquisition section 111) and a second imaging information generating section (in this embodiment, the second captured image generating section 112). The first imaging information acquisition unit (in this embodiment, the first captured image acquisition unit 111) acquires first imaging information (in this embodiment, the first captured image A1) which indicates the output from the imaging element 43 and which captures an image of an object including a blocked area (in this embodiment, an image of a water splash). The second imaging information generation section (in this embodiment, the second captured image generation section 112) generates second imaging information (in this embodiment, the second captured image B1) in which information of the obstructed area (in this embodiment, the image of the water splash) is removed from the first imaging information (in this embodiment, the first captured image A1).

[0078] With this configuration, the information generating device according to this embodiment can perform AF based on the second imaging information from which the occluded area has been removed, even if an object such as a splash of water exists between the subject and the imaging device and the first imaging information includes information on the occluded area. Therefore, the information generating device according to this embodiment can prevent the focus adjustment accuracy from being reduced by focusing on the occluded area by AF.

[0079] The information generating device according to the present embodiment is preferably mounted on an imaging device such as a digital camera, a digital video camera, a smartphone camera, etc. The imaging device mounted with the information generating device according to the present embodiment is suitable for capturing images in rainy weather, in situations where water is splashing around when shooting near water, in situations where snow is falling when skiing or snowboarding, or in situations where sand is falling when long jumping, for example.

[0080] Second Embodiment The second embodiment of the present invention will now be described in detail with reference to the drawings. In the above-described first embodiment, the case where the learning model is stored in advance in a storage unit provided in the imaging device has been described. In the present embodiment, the case where the learning model is acquired from outside at the time of imaging will be described. The imaging device according to this embodiment is referred to as imaging device 1c.

[0081] 9 is a diagram showing an example of the functional configuration of an imaging device 1c according to this embodiment. The imaging device 1c includes an imaging section 4, a control section 11c, and a storage section 12. The control unit 11c includes an image generating unit 110c and a focusing unit 113. The image generating section 110c includes a first captured image acquiring section 111 and a second captured image generating section 112c.

[0082] Comparing the imaging device 1c according to this embodiment (FIG. 9) with the imaging device 1 according to the first embodiment (FIG. 3), the imaging device 1c is different in that it includes a learning model acquisition unit 115c and that the learning model is not pre-stored in the storage unit 12. Here, the functions of the other components (first captured image acquisition unit 111 and focusing unit 113) are the same as those of the first embodiment. Descriptions of the same functions as those of the first embodiment will be omitted, and the second embodiment will be described mainly with respect to the parts that are different from the first embodiment.

[0083] The image generating unit 110c includes a first captured image acquiring unit 111, a second captured image generating unit 112, and a learning model acquiring unit 115c. The learning model acquisition unit 115c acquires the learning model C1 from an external source. The external source is, for example, the server 13. Here, the learning model acquisition unit 115c communicates with the server 13 via the communication I / F 9. The learning model C1 is stored in advance in the server 13. The server 13 is, for example, a cloud server.

[0084] For example, when the imaging device 1 starts imaging, the learning model acquisition unit 115c communicates with the server 13 via the communication I / F 9. The learning model acquisition unit 115c acquires the learning model C1 from the server 13. The second captured image generation unit 112 generates the second captured image B1 using the learning model C1 acquired by the learning model acquisition unit 115c.

[0085] The timing when the learning model acquisition unit 115c starts communication with the server 13 is, for example, the timing when the imaging device 1 starts imaging as described above. Note that the timing when the learning model acquisition unit 115c starts communication with the server 13 may be immediately after the shutter button included in the operation unit 6 is half-pressed, or may be immediately after the power supply (not shown) of the imaging device 1 is turned on.

[0086] According to the imaging device 1c of this embodiment, the imaging device 1c does not need to store the learning model C1 in advance, and therefore the capacity required for the memory unit 12 can be reduced compared to when the learning model C1 is stored in the memory unit 12.

[0087] In addition, a part of the imaging devices 1, 1a, 1b, and 1c in the above-mentioned embodiment, for example, the first captured image acquisition unit 111, the second captured image generation unit 112, 112a, and 112b, the focusing unit 113, the conversion information acquisition unit 114a, and the learning model acquisition unit 115c may be realized by a computer. In that case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed. In addition, the "computer system" here refers to a computer system built into the imaging devices 1, 1a, 1b, and 1c, and includes hardware such as an OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built into a computer system. Furthermore, the term "computer-readable recording medium" may include a medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a medium that stores a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such a case. The above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. In addition, a part or all of the imaging devices 1, 1a, 1b, and 1c in the above-mentioned embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the imaging devices 1, 1a, 1b, and 1c may be individually processed, or a part or all of them may be integrated and processed. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0088] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0089] 1, 1a, 1b, 1c...imaging device, 111...first captured image acquisition section, 112, 112a, 112b...second captured image generation section, A1...first captured image, B1...second captured image, 43...imaging element

Claims

1. a first imaging information acquisition unit that acquires first imaging information indicating an output from an imaging element, the first imaging information indicating an output from the imaging element that has captured an image of an object including a blocked region; a second imaging information generating unit configured to generate second imaging information obtained by removing information on the blocked area from the first imaging information; An information generating device comprising:

2. The second imaging information generating unit generates the second imaging information by using a learning model. The information generating device according to claim 1 .

3. The learning model is trained to generate first transformation information indicating a first transformation for obtaining a second feature vector of a second dimension smaller than the number of dimensions of the first dimension based on a principal component analysis from a first feature vector of a first dimension extracted from the third imaging information, by using a predetermined number or more of third imaging information indicating an output from the imaging element capturing an image of a target not including the occluded region. The information generating device according to claim 2 .

4. a conversion information acquisition unit that acquires second conversion information indicating a second conversion generated so as to reconstruct the first feature vector from the second feature vector obtained by the first conversion, The second imaging information generating unit generates the second feature vector from the first feature vector extracted from the first imaging information based on the first transformation, reconstructs the first feature vector from the generated second feature vector based on the second transformation, and generates the second imaging information from the reconstructed first feature vector. The information generating device according to claim 3 .

5. The learning model is a model trained using first learning imaging information, which is the imaging information showing an output from the imaging element capturing an image of a learning object including a learning occluded region, and second learning imaging information, which is the imaging information obtained by removing information of the learning occluded region from the first learning imaging information, and is trained to output the second learning imaging information when the first learning imaging information is input, The second imaging information generating unit generates the second imaging information from the first imaging information based on the learning model. The information generating device according to claim 2 .

6. A learning model acquisition unit that acquires the learning model from an external source The information generating device according to claim 2 .

7. The learning model is an autoencoder, a variational autoencoder, a generative adversarial network, or a diffusion model. The information generating device according to claim 2 .

8. The occluded region is a region in which at least one of splashes of water, rain, snow, sand, mud, or dust is captured. The information generating device according to claim 1 .

9. The imaging device further includes a focusing unit that performs focusing based on the second imaging information. The information generating device according to claim 1 .

10. 10. An imaging device comprising: an information generating device according to claim 1; and an imaging optical system.

11. a first imaging information acquisition step of acquiring first imaging information indicating an output from an imaging element, the first imaging information indicating an output from the imaging element that has imaged an object including a blocked region; a second imaging generating step of generating second imaging information in which information about the blocked area is removed from the first imaging information; The information generating method includes:

12. On the computer, a first imaging information acquisition step of acquiring first imaging information indicating an output from an imaging element, the first imaging information indicating an output from the imaging element that has imaged an object including a blocked region; a second imaging generating step of generating second imaging information in which information about the blocked area is removed from the first imaging information; A program for executing the above.

Images