Image processing device, image processing method, and program

Abstract

To improve the accuracy of processing on input data related to an image.SOLUTION: Input data including an image and / or information on the image is acquired. A feature quantity of the input data is obtained through processing on the input data using a neural network. An integrated feature quantity is generated by integrating at least a portion of the input data with the feature quantity. A processing result corresponding to the input data is generated through processing on the integrated feature quantity.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program, and more particularly to focus control of an imaging device. [Background technology]

[0002] Image processing is used for various purposes. For example, Non-Patent Document 1 discloses a method for detecting a subject region from an image using a neural network.

[0003] On the other hand, imaging devices that perform focus adjustment to focus on a subject are known. For example, Patent Document 1 discloses a technique for performing focus adjustment based on defocus amounts in a plurality of autofocus (AF) areas. Patent Document 1 discloses a technique for focusing on a main subject while eliminating the influence of an obstruction that crosses in front of the main subject. According to Patent Document 1, the area of ​​the obstruction that crosses in front of the main subject is determined by using a statistical value of the distance value corresponding to the subject distance detected for each AF area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-137760 A [Patent Document 2] JP 2019-134431 A [Patent Document 3] JP 2017-98900 A [Non-patent literature]

[0005] [Non-Patent Document 1] W. Liu et al. "SSD: Single Shot MultiBox Detector", In ECCV 2016. [Non-Patent Document 2] W. Yang et al. "Deep Learning for Single Image Super-Resolution: A Brief Review", arXiv:1808.03344, 2018. [Non-Patent Document 3] L. Chen et al. "Simple Baselines for Image Restoration", arXiv:2204.04676, 2022. Summary of the Invention [Problem to be solved by the invention]

[0006] When focusing on a person's face, the arm or hand of the same person may cover the face. For example, when the arm covers the face, the face area included in the field of view of the imaging device includes an area where the face does not exist (i.e., the area of ​​the arm covering the face). In this case, the focus detection result in the face area changes continuously from the face to the arm. In such a situation, even if a technique using a statistical value of distance values ​​such as that in Patent Document 1 is used, it is difficult to suppress the influence of the arm and focus on the face. In addition, for example, when shooting is performed in a low-illumination environment, when the contrast of the subject is low, or when the F-number of the shooting optical system is large, the variation in the focus detection result is likely to be large. In this case, the distance value detected for each area may be accompanied by a relatively large error that follows a predetermined distribution such as a normal distribution. In such a case, it is difficult to suppress the influence of the error in a method using a statistical value of distance values ​​such as that in Patent Document 1.

[0007] The present inventors have considered using a neural network to perform focus adjustment with high accuracy. Generally, in processing using a neural network, feature extraction is performed in a previous layer and the result is output in a subsequent layer. In particular, the present inventors have invented a configuration that uses a neural network to estimate the defocus amount of a subject (e.g., a person's face) from a defocus map that indicates the defocus amount at each position of an image. However, it is considered that further improvement in the estimation accuracy is required. Also, in other processing using a neural network, there are cases where further improvement in the estimation accuracy is required.

[0008] An object of the present invention is to improve the accuracy of processing input data relating to an image. [Means for solving the problem]

[0009] An image processing apparatus according to an embodiment of the present invention includes the following configuration. - acquiring means for acquiring input data comprising an image and / or information relating to the image; a feature acquisition means for acquiring a feature of the input data by processing the input data using a neural network; an integration unit that generates an integrated feature by integrating at least a portion of the input data and the feature; a post-processing means for generating a processing result corresponding to the input data by processing the integrated feature; Equipped with. Effect of the Invention

[0010] The accuracy of processing input data relating to an image can be improved. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to an embodiment. [Diagram 3] 4 is a flowchart of a process performed in an image processing method according to an embodiment. [Figure 4] FIG. 11 is a diagram for explaining an example of a method for generating a feature amount. [Diagram 5] 11 is a flowchart illustrating an example of a process for generating an integrated feature. [Figure 6] 11 is a flowchart illustrating an example of a process for generating an integrated feature. [Figure 7] 11A and 11B are diagrams illustrating an example of processing for integrated features. [Figure 8] 5A and 5B are diagrams for explaining an input image and a defocus map. [Figure 9] 1 is a flowchart of a process in an image processing method according to an embodiment. [Figure 10] 11 is a flowchart illustrating an example of a process for generating an integrated feature. [Figure 11] 11A to 11C are diagrams for explaining a process of generating an integrated feature. [Figure 12] 11A to 11C are diagrams for explaining a process of generating an integrated feature. [Figure 13] FIG. 2 is a diagram showing an example of the functional configuration of a learning device according to an embodiment. [Figure 14] 1 is a flowchart of a process performed in a learning method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] An image processing device according to an embodiment of the present invention can be realized by a computer or information processing device including a processor and a memory. FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device that can be used as an image processing device according to each embodiment described later. The image processing device according to each embodiment may be realized by a single information processing device. On the other hand, each function of the information processing device described later may be distributed to multiple information processing devices as necessary. When the image processing device is composed of multiple information processing devices, the respective information processing devices are connected to each other so as to be able to communicate with each other. For the connection, for example, a local area network (LAN) or the like can be used.

[0014] In the example of FIG. 1, the information processing device 100 is connected to an input device 109, an output device 110, the Internet 111, and a camera 112. The connection method is not particularly limited. For example, at least some of these devices may be connected by wire. Furthermore, at least some of these devices may be connected via wireless communication. Furthermore, the input device 109 or the output device 110 may be an independent device separate from the information processing device 100. On the other hand, the input device 109 or the output device 110 may be integrally incorporated in the information processing device 100.

[0015] The information processing device 100 can perform processing according to each embodiment. The input device 109 is a device that accepts user input to the information processing device 100. The input device may be, for example, a pointing device or a keyboard. The output device 110 is a device that can output images and characters. The output device 110 is, for example, a monitor. The output device 110 can display data held by the information processing device 100, data input by a user, or the execution result of a program.

[0016] The camera 112 is an imaging device capable of acquiring captured images. The camera 112 can acquire successive captured images, for example, by capturing images at a predetermined interval Δt. The camera 112 can input the captured images thus acquired to a data acquisition unit 201, which will be described later. The number of cameras 112 is not particularly limited. For example, one camera 112 or multiple cameras 112 may be connected to the information processing device 100.

[0017] The CPU 101 is a central processing unit that controls the entire information processing device 100. The CPU 101 can execute processes according to each embodiment and control the operation of the information processing device 100 by executing various software (computer programs) stored in, for example, the external storage device 104.

[0018] The ROM 102 is a read only memory. The ROM 102 can store programs and parameters that do not require modification. The RAM 103 is a random access memory. The RAM 103 can temporarily store programs or data supplied from an external device or the like. The external storage device 104 is an external storage device that can be read by the information processing device 100. The external storage device 104 can store programs and data for a long period of time. The external storage device 104 may be, for example, a hard disk or a memory card fixedly installed in the information processing device 100. The external storage device 104 may also be an optical disk such as a flexible disk (FD) or a compact disk (CD) that is detachable from the information processing device 100, a magnetic card, an optical card, an IC card, or a memory card.

[0019] The input interface 105 is an interface with an input device 109. The output interface 106 is an interface with an output device 110. The communication interface 107 is an interface used for connecting to other devices. The information processing device 100 can be connected to the Internet 111 or a camera 112 via the communication interface 107. The camera 112 may be connected to the information processing device 100 via the Internet 111. The system 108 connects the above-mentioned units to each other so that they can communicate with each other.

[0020] In this way, a processor such as the CPU 101 executes a program stored in a memory such as the ROM 102, the RAM 103, or the external storage device 104, thereby realizing the functions of each unit shown in FIG. 2, which will be described later. For example, a program that realizes the functions of each unit and is stored in the external storage device 104 is read into the RAM 103. Then, the CPU 101 operates according to the program on the RAM 103, thereby realizing the functions of each unit. At this time, the device that stores the various programs and various setting data sets is not limited to the external storage device 104. For example, the information processing device 100 may obtain the various programs and various setting data sets from another device such as a server via a network such as the Internet 111.

[0021] 2 is a block diagram showing an example of a functional configuration of the information processing device 100. The information processing device 100 has a data acquisition unit 201, a parameter acquisition unit 202, and an inference unit 21. Each functional unit is connected to a storage unit 206. However, the information processing device 100 may have the storage unit 206. These functional units will be described below.

[0022] In the following, a case where the information processing device 100 performs a task of inferring the defocus range of a subject in an image will be mainly described. The defocus range of a subject is represented by the maximum and minimum values ​​of the defocus amount in an area corresponding to a specific subject in an image. The category of the subject is not particularly limited. The subject can be various objects such as a person, an animal (e.g., a dog or a cat), or a vehicle (e.g., a car or a train). On the other hand, the subject may be a part of an object such as a face or an eye. In the following, a case where a person is detected will be described. However, as described later, the process performed by the information processing device 100 according to this embodiment is not limited to the process of inferring the defocus range. In this specification, the defocus amount indicates how far the focus is deviated from the focal plane.

[0023] The data acquisition unit 201 acquires input data. In this embodiment, the input data is input data including an image and / or information about the image. In the following example, the data acquisition unit 201 acquires a captured image of a subject. The captured image may be an image obtained by capturing an image of the subject using the camera 112.

[0024] In this embodiment, the data acquisition unit 201 acquires a defocus map in addition to the captured image. The defocus map is a map that spreads in the spatial direction, and indicates the defocus amount for each position (for example, each region) of the captured image. The defocus map can be generated based on the defocus amount calculated for each region of the image. The data acquisition unit 201 can obtain such a defocus map from the camera 112. The method of generating the defocus map is not particularly limited. For example, the camera 112 having an image plane phase difference AF function can generate such a defocus map based on the defocus amount at each ranging point. The defocus map can be generated according to the method described in Patent Document 2, for example.

[0025] In this embodiment, the data acquisition unit 201 further acquires a subject area map indicating an area in which a subject exists in a captured image. The subject area map can indicate the position and size of the subject. The data acquisition unit 201 can generate a subject area map based on a position input. For example, a user can specify a subject area by touching a subject on an image displayed on the output device 110. In this case, the data acquisition unit 201 can acquire information specifying a subject area from the input device 109. The data acquisition unit 201 may also determine the area of ​​the subject using a method for automatically detecting a subject in an image. For example, the data acquisition unit 201 can detect a main subject or the like in an image using a method described in Patent Document 3. The data acquisition unit 201 may also determine the area of ​​the subject based on both the area designation and the object detection process. For example, the data acquisition unit 201 can use the area of ​​an object detected from a specified area as the area of ​​the subject. The data acquisition unit 201 can detect an object from an image using a method described in Non-Patent Document 1, for example.

[0026] Then, the data acquisition unit 201 can generate a subject region map based on information indicating the subject region in the image thus obtained. In the following example, the subject region in the image is represented by a bounding box (hereinafter, abbreviated as BB). Note that it is not essential to use the subject region map in this embodiment. That is, the information processing device 100 may estimate a defocus range for a subject present at any position in the captured image.

[0027] FIG. 8(A) shows an example of an image 801 acquired by the data acquisition unit 201. The image 801 is an image captured by the camera 112. A person 802, who is a subject, is captured in the image 801. In FIG. 8(B), a defocus map 803 and a BB 804 of the subject are superimposed on the image 801. FIG. 8(D) shows a subject region map 805. The subject region map 805 is generated based on the BB 804. The subject region map 805 is a two-dimensional map. In the subject region map 805, a value greater than 0 (for example, 1) is recorded in the pixels in the region inside the BB 804, and 0 is recorded in the other pixels.

[0028] On the other hand, in addition to the subject, the background is also captured in the BB804 of the image 801. Also, an obstacle that hides a part of the subject may be captured in the BB804 of the image 801. According to this embodiment, the defocus range for a subject of a specific category captured in the BB804 is estimated. For example, it is possible to estimate the defocus range for a part of the BB804 that is not a part of the background and is not a part of an obstacle, but is a part of a person's face.

[0029] The parameter acquisition unit 202 acquires parameters related to the processing performed by the feature generation unit 203 and the post-processing unit 205. The parameter acquisition unit 202 can acquire, for example, parameters related to a neural network used by the feature generation unit 203 and the post-processing unit 205. The parameter acquisition unit 202 can acquire parameters from the storage unit 206. These parameters are determined by learning as described below.

[0030] The inference unit 21 infers the defocus range of the subject based on the input data. The inference unit 21 includes a feature amount generation unit 203, a feature amount integration unit 204, and a post-processing unit 205.

[0031] The feature generating unit 203 generates features of input data based on input data related to an image acquired by the data acquiring unit 201. In this embodiment, the feature generating unit 203 generates features using an image 801, a defocus map 803, and a subject region map 805. The feature generating unit 203 can generate features using a neural network such as a Convolutional Neural Network (hereinafter abbreviated as CNN), for example.

[0032] The feature amount integration unit 204 generates an integrated feature amount by integrating at least a part of the input data acquired by the data acquisition unit 201 and the feature amount generated by the feature amount generation unit 203. In this embodiment, the feature amount integration unit 204 integrates the feature amount generated by the feature amount generation unit 203 and the defocus map acquired by the data acquisition unit 201.

[0033] The post-processing unit 205 generates a processing result corresponding to the input data acquired by the data acquisition unit 201 by processing the integrated feature generated by the feature integration unit 204. In this embodiment, the post-processing unit 205 outputs information indicating the defocus range of the subject based on the feature generated by the feature integration unit 204. FIG. 8(C) shows a histogram of the defocus amount in the BB804. FIG. 8(C) shows the defocus range estimated by the post-processing unit 205. In the embodiment described below, the post-processing unit 205 outputs the maximum and minimum values ​​of the defocus range. However, the format of the output of the post-processing unit 205 is not limited to this. For example, the post-processing unit 205 may output the defocus amount for each position where the subject is captured in the BB804.

[0034] 3 is a flowchart showing the flow of processing of an image processing method performed by the information processing device 100 in this embodiment. In the following, each process (step) is represented by a reference character preceded by an S, and the description of "process (step)" is omitted. However, the information processing device 100 does not need to perform all of the processes shown in this flowchart. Each process can be executed by the CPU 101 as described above.

[0035] In S301, the data acquisition unit 201 acquires a captured image of a subject as described above. In S302, the data acquisition unit 201 acquires a defocus map as described above. In S303, the data acquisition unit 201 acquires a subject region map as described above. The data acquisition unit 201 may acquire these data from the camera 112 connected to the information processing device 100. In addition, the data acquisition unit 201 may acquire data stored in the external storage device 104.

[0036] In S304, the parameter acquisition unit 202 acquires parameters related to the processing as described above. For example, the parameter acquisition unit 202 can acquire parameters used in the calculations in the Convolution layer and the Fully-Connected layer.

[0037] In S305, the feature generating unit 203 generates features using the input data acquired in S301 to S303. The feature generating unit 203 can generate features using, for example, CNN. The feature generating unit 203 may generate features using Multilayer Perceptron. The feature generating unit 203 may also generate features using Multi-head Self Attention. In this way, the process used by the feature generating unit 203 to generate features is not limited to a specific method.

[0038] On the other hand, in one embodiment, the process performed by the feature generator 203 on the input data to generate features includes a nonlinear operation such as an activation process. For example, the process performed by the feature generator 203 on the input data may be a process in a neural network including an activation layer.

[0039] In this embodiment, the feature generating unit 203 inputs the captured image, the defocus map, and the subject region map to the CNN. Then, the feature generating unit 203 performs calculations in the CNN to obtain the output from the CNN as features. FIG. 4 shows an example of the configuration of a CNN. The CNN shown in FIG. 4 has a combination of convolution layers (Convolution 401, 404) that perform convolution calculations, activation layers (ReLu 402, 405) that apply activation functions, and a pooling layer (MaxPooling 403) that performs pooling processing. However, the configuration of the CNN, i.e., the number of layers and the connections between layers, is not particularly limited.

[0040] The feature generating unit 203 can perform a process of aligning the resolutions of the captured image, the defocus map, and the subject region map before inputting them to the CNN. For example, the feature generating unit 203 can perform downsampling or upsampling. As an example, the feature generating unit 203 can upsample the resolution of the defocus map so that it matches the resolution of the captured image. At this time, the feature generating unit 203 can input the captured image, the defocus map, and the subject region map having the same number of elements in the vertical and horizontal directions to the CNN as multi-channel data.

[0041] In S306, the feature amount integration unit 204 generates an integrated feature amount as described above. In this embodiment, the feature amount integration unit 204 integrates the defocus map acquired by the data acquisition unit 201 and the feature amount generated by the feature amount generation unit 203.

[0042] 5 is a flowchart showing an example of the processing in S306. In S501 to S502, the feature amount integrating unit 204 performs processing to match the number of elements of the data to be integrated. For example, the feature amount integrating unit 204 can perform at least one of processing to match the spatial resolution and processing to match the number of channels between at least a portion of the input data and the features.

[0043] First, in S501, the feature integration unit 204 performs processing to align the spatial resolution (for example, the vertical and horizontal resolution) of the data to be integrated. By such processing, it is possible to align the number of elements in the spatial direction of the data to be integrated.

[0044] In this embodiment, the feature amount integrating unit 204 performs downsampling on the defocus map so as to match the resolution of the feature amount. The downsampling method is not particularly limited. For example, the feature amount integrating unit 204 can perform downsampling using a Nearest Neighbor method. Note that the feature amount integrating unit 204 may upsample the feature amount instead of downsampling the defocus map. Also, the feature amount integrating unit 204 may use a combination of downsampling and upsampling.

[0045] As an example, the feature amount integration unit 204 may perform downsampling using a combination of element rearrangement and convolution processing. An example of a method for rearranging elements will be described with reference to Figs. 12(A) and (B). Fig. 12(A) shows a defocus map 1203 before downsampling. The feature amount integration unit 204 can obtain maps 1204 to 1207 shown in Fig. 12(B) by rearranging the elements of the defocus map 1203. The maps 1204 to 1207 are arranged so as to spread in the channel direction. By rearranging, the spatial resolution of the maps 1204 to 1207 is lower than that of the defocus map 1203. When the elements are rearranged so that the resolution in the spatial direction (x and y directions) is 1 / 2, the elements can spread four times in the channel direction. The feature amount integration unit 204 can further perform convolution processing on the maps 1204 to 1207 thus obtained. For example, the feature amount integrating unit 204 can generate a map for one channel from the maps 1204 to 1207 by a convolution process using a 1×1 filter.

[0046] Next, in S503, the feature amount integrating unit 204 performs a process of matching the number of elements in the channel direction of the data to be integrated. An example of the process of S503 will be described with reference to FIG. 11. The feature amount integrating unit 204 can broadcast the defocus map 1102 after the process in S501 in the channel direction. Specifically, the feature amount integrating unit 204 can generate a defocus map 1103 having data of a plurality of channels by copying the defocus map 1102 in the channel direction. Data of each channel of the defocus map 1103 corresponds to the defocus map 1102. In this way, the feature amount integrating unit 204 can match the dimension in the channel direction of the feature amount 1101 and the dimension in the channel direction of the defocus map 1103.

[0047] Finally, in S504, the feature amount integration unit 204 integrates the defocus map and the feature amount after the processing in S503. In the example of Fig. 11, the feature amount integration unit 204 can integrate the feature amount 1101 and the defocus map 1103. In this embodiment, the feature amount integration unit 204 performs the integration by calculating an element product of the feature amount 1101 and the defocus map 1103. That is, the value of each element of the integrated feature amount obtained in S504 is the product of the corresponding element of the feature amount 1101 and the corresponding element of the defocus map 1103.

[0048] As another example of the process in S306, the feature amount integration unit 204 may perform a process of combining at least a part of the input data and the feature amount in the channel direction. In addition, the feature amount integration unit 204 may perform a product-sum operation on the data obtained by the combination. FIG. 6 is a flowchart showing another example of the process in S306. In S601, the feature amount integration unit 204 performs a process of aligning the resolution in the vertical and horizontal directions of the data to be integrated. In this embodiment, the feature amount integration unit 204 performs upsampling on the feature amount obtained in S305 so as to match the resolution of the defocus map. Next, in S602, the feature amount integration unit 204 combines the defocus map obtained in S302 and the feature amount after the process in S601 in the channel direction. Finally, in S603, the feature amount integration unit 204 generates an integrated feature amount by performing a convolution operation on the data obtained by the combination in S602. For example, the feature amount integrating unit 204 can perform convolution processing using a 1×1 filter (that is, weighted addition in the channel direction).

[0049] The feature amount integration unit 204 may perform noise reduction processing on the defocus map. Then, the feature amount integration unit 204 may integrate the defocus map that has been subjected to the noise reduction processing with the feature amount generated by the feature amount generation unit 203. The method of the noise reduction processing is not particularly limited. For example, the noise reduction processing can be performed using a filter such as a median filter or a Gaussian filter. Also, the noise reduction processing can be performed using a neural network. In this way, by performing the noise reduction processing on the defocus map, the accuracy of the estimation result of the defocus range obtained by the post-processing unit 205 can be improved.

[0050] Furthermore, the feature amount integration unit 204 may perform a high-resolution process on the defocus map. Then, the feature amount integration unit 204 may integrate the defocus map that has been subjected to the high-resolution process with the feature amount generated by the feature amount generation unit 203. An example of the high-resolution process is the super-resolution process described in Non-Patent Document 2. In this way, by performing the high-resolution process on the defocus map, it is possible to improve the accuracy of the estimation result of the defocus range in the fine details.

[0051] Next, in S307, the post-processing unit 205 generates a processing result by processing the integrated feature obtained in S306. In this embodiment, the post-processing unit 205 outputs the defocus range of the subject by processing the integrated feature. In this embodiment, the processing performed by the post-processing unit 205 on the integrated feature is a linear operation. Also, in this embodiment, the processing performed by the post-processing unit 205 on the integrated feature does not include non-linear processing. In the following example, the processing performed by the post-processing unit 205 to generate a processing result includes a full combination processing. Also, the processing performed by the post-processing unit 205 to generate a processing result includes a pooling processing.

[0052] FIG. 7 shows an example of the process performed by the post-processing unit 205 in S307. The post-processing unit 205 first performs a pooling process (Global Average Pooling 701) on the integrated feature amount. Furthermore, the post-processing unit 205 performs a fully-connected process (Fully-Connected 702) on the result of the pooling process. Through these processes, the defocus range of the subject (i.e., the maximum defocus amount Def max and the minimum value Def min ) is output. Note that the processing performed by the post-processing unit 205 is not limited to a specific method. For example, the post-processing unit 205 may perform a full combination process multiple times. In one embodiment, the amount of calculation of the processing performed by the post-processing unit 205 on the integrated features is less than the amount of calculation of the processing performed by the feature generating unit 203 on the input data. This amount of calculation may be a calculation time or the number of multiplication operations.

[0053] The result output by the post-processing unit 205 in this manner can be used for focus control of an imaging device (for example, the camera 112). For example, according to the method shown in FIG. max and the minimum value Def min Based on the result thus obtained, the CPU 101 can control the optical system of the camera 112 via the communication interface 107. For example, the output device 110 can detect whether the focus position of the camera 112 is Def max and Def min The CPU 101 can control the focus operation of the camera 112 so that the image is included in the range of Def max and Def min Alternatively, information indicating the focus position determined by the user between the input interface 105 and the camera 112 may be acquired via the input interface 105. Then, the CPU 101 can control the focus operation of the camera 112 according to the designated focus position. In this case, the CPU 101 determines whether the focus position designated by the user is within the range of Def max and Def min The user input can be restricted to be between .

[0054] According to this embodiment, the defocus amount of the main subject can be accurately inferred. That is, even if a background or an obstruction other than the subject exists in the subject area shown by the subject area map, the defocus amount for the subject can be estimated. Thus, according to this embodiment, when performing focus control to focus on the main subject, the influence of the obstruction or focus detection error can be reduced. Therefore, it becomes easier to maintain the main subject in focus.

[0055] Next, a method for learning parameters related to the processing performed by the information processing device 100 to estimate the defocus range of a subject (for example, the processing performed by the feature generating unit 203 and the post-processing unit 205) will be described. FIG. 13 shows an example of the functional configuration of a learning device 13 that performs the learning method according to this embodiment. The functions of the learning device 13 can be realized by a processor executing a program stored in a memory. The learning device 13 may have the same hardware configuration as the information processing device 100 shown in FIG. 1.

[0056] The data acquisition unit 1301 acquires input data for learning. The input data for learning includes a captured image for learning, a defocus map, and a subject region map. The data acquisition unit 1301 further acquires correct answer data for a defocus range corresponding to a set of the captured image for learning, the defocus map, and the subject region map.

[0057] The learning captured images, the defocus map, and the subject region map can be acquired as described above. The correct answer data for the defocus range may be generated based on a user input. The correct answer data may also be generated based on a defocus amount detected by an imaging device such as the camera 112.

[0058] For example, the camera 112 can calculate the defocus amount for each focus detection area based on the focus detection signal acquired at the same time as the image of the subject is captured. The calculation of the defocus amount may be performed by an external calculation device such as a personal computer based on the focus detection signal and the image signal recorded by the image capture device.

[0059] Then, in consideration of the subject area in the captured image and the defocus amount of the background of the subject or the obstacle in the foreground of the subject, the range of the defocus amount appropriate for the defocus amount of the subject can be used as the correct answer data. For example, the correct answer data can be determined by excluding the range of the defocus amount for the background and the range of the defocus amount for the obstacle from the range of the defocus amount for the subject area. The correct answer data for such a defocus range can be determined while the user visually checks the captured image and the defocus amount. As another method, the captured image can be divided by a segmentation process. Also, the defocus amount calculated for the focus detection area overlapping with the part area of ​​the subject that does not include the obstacle in the background or foreground can be determined as the correct defocus amount for the partial area of ​​the subject. Then, the correct answer data can be determined based on these correct defocus amounts determined for each partial area of ​​the subject.

[0060] The inference unit 21 infers the defocus range of a subject based on input data for learning, similar to the information processing device 100. The inference unit 21 includes a feature amount generation unit 203, a feature amount integration unit 204, and a post-processing unit 205, similar to the information processing device 100.

[0061] The loss calculation unit 1302 determines an error in the estimation result of the defocus range generated by the inference unit 21. For example, the data acquisition unit 1301 calculates the loss by comparing the defocus range generated by the inference unit 21 with the correct answer data for the defocus range acquired by the data acquisition unit 1301.

[0062] The parameter update unit 1303 updates parameters used by the inference unit 21 for processing (for example, parameters used by the feature generation unit 203 and the post-processing unit 205 for processing) based on the loss calculated by the loss calculation unit 1302. In this manner, parameter learning is performed. The parameter saving unit 1304 saves the parameters obtained by learning and used by the inference unit 21 for processing in the storage unit 206. These parameters are used for processing by the information processing device 100 (for example, the feature generation unit 203 and the post-processing unit 205).

[0063] Fig. 14 shows an example of a flowchart of the learning process according to this embodiment. S301 to S303 are the same as in Fig. 3. In this example, in S301 to S303, the data acquisition unit 1301 acquires input data for learning. In S1401, the data acquisition unit 1301 acquires correct answer data for the defocus range as described above. S305 to S307 are the same as in Fig. 3. In this example, an inference result of the defocus range for the input data for learning is generated.

[0064] In S1402, the loss calculation unit 1302 calculates the loss between the correct data of the defocus range acquired in S1401 and the inference result of the defocus range obtained in S307. The loss can be expressed, for example, by the L1 norm according to the following formula.

[0065]

number

[0066] In S1403, the parameter update unit 1303 updates the parameters based on the loss calculated in S1402. The parameters updated here are, for example, the weights of elements in a neural network. For example, the parameters updated may be weight coefficients of a convolution layer. Also, the parameters updated may be weight coefficients of a fully-connected layer. The parameter update method is not particularly limited. For example, the parameter update unit 1303 may update the parameters using back propagation based on Momentum SGD.

[0067] In S1404, the parameter storage unit 1304 stores the parameters updated in S1403. In the learning process, the method shown in Fig. 14 can be repeated using various learning input data.

[0068] In the above embodiment, the defocus range is estimated for one type of subject in the image. However, the subject is not limited to one type. For example, the defocus range may be estimated for each of two or more types of subjects. As a specific example, the information processing device 100 may estimate the defocus range for the entire person, the defocus range for the pupil region of the person, and the defocus range for the head region of the person in the captured image.

[0069] In another embodiment, the information processing device 100 may infer a depth range of the subject instead of the defocus range. The depth range can represent a range of distance from the imaging device to each position of the subject. The data acquisition unit 201 may acquire a depth map instead of the defocus map. The depth map can represent a depth value for each position of the captured image (for example, a distance from the imaging device to the subject corresponding to each position of the captured image). In this way, the input data acquired by the data acquisition unit 201 may include one of the defocus map and the depth map. In this case, one of the defocus map and the depth map is used to generate an integrated feature. The post-processing unit 205 can output an estimation result of at least one of the defocus range and the depth range of the subject.

[0070] In the above embodiment, the feature integration unit 204 generates an integrated feature by integrating the feature generated by the feature generation unit 203 with the defocus map. At this time, the feature integration unit 204 can perform integration without converting the magnitude or unit of the value indicated by the defocus map. Then, the post-processing unit 205 infers the defocus range based on the integrated feature. With this configuration, even if the processing performed by the feature generation unit 203 includes processing that is likely to change the value, such as nonlinear calculation, the value of the defocus map is likely to be maintained in the processing by the post-processing unit 205. For example, the integrated feature used by the post-processing unit 205 can better reflect the value of the defocus map. Therefore, the inference accuracy of the defocus range of the subject is improved. According to the study by the inventor of the present application, the inference accuracy of the defocus range is greatly improved by integrating the defocus map with the feature generated by the feature generation unit 203. In particular, the inference accuracy of the defocus range for a captured image in which the subject is out of focus is significantly improved.

[0071] In this manner, in the present embodiment, the feature amount of the input data obtained by processing the input data and at least a part of the input data (e.g., a defocus map) are integrated. The feature amount obtained by such processing of the input data often does not directly indicate the value of each of the input data (e.g., a defocus map). On the other hand, the integrated feature amount thus obtained reflects the value (particularly the absolute value) of at least a part of the input data (e.g., a defocus map) better than the feature amount obtained by processing the input data. Therefore, according to the present embodiment, the inference accuracy based on the processing of the integrated feature amount can be improved. Such a configuration is particularly effective when an inference is performed so that the value indicated by the input data to be integrated and the value indicated by the inference result are similar. In one embodiment, at least a part of the statistical value (e.g., a maximum value, a minimum value, an average value, or a weighted average value) of the value indicated by the input data to be integrated indicates a desirable inference result. For example, in one embodiment, the maximum value and the minimum value of the value indicated by the defocus map in the area corresponding to the subject can indicate a desirable defocus range as an inference result. This desirable defocus range as an inference result can be represented by the correct answer data of the defocus range used for learning (e.g., the maximum or minimum value of the defocus value).

[0072] In one embodiment, the integrated feature generated by the feature integration unit 204 is obtained based on an element product of at least a part of the input data (e.g., a defocus map) and the feature. In this case, the parameters used by the feature generation unit 203 are learned so that the feature generated by the feature generation unit 203 represents a part corresponding to the subject, regardless of how the value of the defocus to be inferred changes. That is, learning can be performed so that the feature generated by the feature generation unit 203 shows a high value in a part of the defocus map that is likely to show the defocus amount of the subject, and shows a value close to 0 in a part that is not likely to show the defocus amount of the subject. With this configuration, it becomes easier to learn the features required to obtain the defocus range of the subject. According to the study by the inventors of the present application, by integrating the defocus map based on the element product, the inference accuracy of the defocus range is greatly improved compared to the case of integrating the defocus map by combining the channel direction combination and the product-sum operation.

[0073] Moreover, in one embodiment, the post-processing unit 205 generates a processing result corresponding to the input data without using a nonlinear operation. With such a configuration, the absolute value indicated by the integrated feature amount can be easily maintained in the processing in the post-processing unit 205. Therefore, the post-processing unit 205 can easily output a value reflecting the absolute value indicated by the integrated feature amount (i.e., the value indicated by the integrated input data). Therefore, according to such an embodiment, the inference accuracy of the processing result (e.g., the defocus range) is stabilized.

[0074] The information processing device 100 according to the above embodiment can be used for various processes other than estimating the defocus range. In the following, a case where the information processing device 100 performs a task of reducing noise in an image will be described with reference to the flowchart in FIG.

[0075] In S901, the data acquisition unit 201 acquires a captured image of a subject as input data. In S902, the parameter acquisition unit 202 acquires parameters related to processing in the same manner as in S304. The parameter acquisition unit 202 can acquire parameters used in calculations in the Convolution layer and the Fully-Connected layer. In S903, the feature generation unit 203 generates features using the captured image acquired in S901. The feature generation unit 203 can generate features using a CNN as shown in FIG. 4.

[0076] In S904, the feature amount integrating unit 204 generates an integrated feature amount by integrating the captured image acquired in S901 and the feature amount generated in S903. The feature amount integrating unit 204 generates an integrated feature amount using the same method as in S306. For example, the feature amount integrating unit 204 can perform a process of matching the number of elements of the data to be integrated and a process of integrating each element.

[0077] FIG. 10 is a flowchart showing an example of the process performed in S904. In S1001, the feature amount integration unit 204 performs a process of aligning the resolution of the captured image with the resolution of the feature amount. For example, the feature amount integration unit 204 can perform an upsampling process on the feature amount. Next, in S1002, the feature amount integration unit 204 performs a process of aligning the number of channels of the captured image with the number of channels of the feature amount. For example, the feature amount integration unit 204 can perform a convolution process on the captured image. Note that the order of the processes in S1001 and S1002 may be interchanged. Next, in S1003, the feature amount integration unit 204 generates an integrated feature amount by calculating an element product of the input image and the feature amount after the processes in S1001 and S1002.

[0078] In S905, the post-processing unit 205 performs processing on the feature amount generated in S904. For example, the post-processing unit 205 can perform convolution processing. This convolution processing can be processing for converting the integrated feature amount so as to obtain an output having the number of channels that is ultimately desired to be output (for example, RGB three channels in the case of an image). In this way, the post-processing unit 205 can obtain a noise reduction result corresponding to the captured image.

[0079] In such a process, the learning of the parameters used by the information processing device 100 can be performed by the learning device 13 as already described. The ground truth data indicating the noise reduction result for the captured image for learning can be obtained by any method. For example, the ground truth data can be obtained by applying the noise reduction process as described in Non-Patent Document 3 to the captured image for learning.

[0080] In this embodiment, too, the integrated feature reflects the captured image data better than the feature obtained by processing the captured image. Therefore, this embodiment improves the accuracy of the noise reduction process. In particular, in one embodiment, the captured image and the feature are integrated using an element product. In this case, the parameters used by the feature generation unit 203 are learned so that the feature generated by the feature generation unit 203 represents the ratio between the captured image and the image to be output. Therefore, learning can be performed that is not dependent on the magnitude of the value indicated by the captured image. Such a configuration makes learning easier and improves noise reduction performance.

[0081] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0082] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) - acquiring means for acquiring input data comprising an image and / or information relating to the image; a feature acquisition means for acquiring a feature of the input data by processing the input data using a neural network; an integration unit that generates an integrated feature by integrating at least a portion of the input data and the feature; a post-processing means for generating a processing result corresponding to the input data by processing the integrated feature; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the integration means integrates at least a portion of the input data and the feature amount by calculating an element product. (Item 3) 3. The image processing device according to any one of items 1 to 2, characterized in that the integration means performs a process of matching spatial resolution between at least a portion of the input data and the feature amount, and integrates at least a portion of the input data after the process with the feature amount. (Item 4) 4. The image processing device according to any one of items 1 to 3, characterized in that the integration means performs a process of matching the number of channels between at least a portion of the input data and the feature amount, and integrates at least a portion of the input data after the process with the feature amount. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the integration means performs a process of combining at least a portion of the input data and the feature in a channel direction, and generates the integrated feature by performing a product-sum operation on data obtained by the combining process. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the integrating means performs a noise reduction process on at least a portion of the input data, and generates the integrated feature by integrating at least a portion of the input data after the noise reduction process with the feature. (Item 7) 7. The image processing device according to any one of items 1 to 6, wherein the integrating means performs a high-resolution processing on at least a part of the input data, and generates the integrated feature by integrating at least a part of the input data after the high-resolution processing with the feature. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the processing for the integrated feature amount is a linear operation. (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the processing for the integrated feature includes a full combination processing. (Item 10) 10. The image processing device according to any one of items 1 to 9, wherein the processing for the integrated feature does not include a nonlinear operation. (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the processing of the input data includes a nonlinear operation. (Item 12) 12. The image processing device according to any one of items 1 to 11, wherein the processing on the input data is processing in a neural network including an activation layer. (Item 13) 13. The image processing device according to any one of items 1 to 12, wherein an amount of calculation for processing the integrated feature is smaller than an amount of calculation for processing the input data. (Item 14) 14. The image processing device according to any one of items 1 to 13, wherein the post-processing means outputs an estimation result of a defocus range or a depth range for a subject in a captured image. (Item 15) the input data includes the captured image and one of a defocus map indicating a defocus amount for each position of the captured image and a depth map indicating a depth value for each position of the captured image, Item 15. The image processing device according to item 14, wherein at least a portion of the input data is one of the defocus map and the depth map. (Item 16) 16. The image processing device according to any one of items 14 to 15, wherein the input data further includes data for designating an area in the captured image where the subject exists. (Item 17) 14. The image processing device according to any one of items 1 to 13, wherein the post-processing means outputs a result of noise reduction for the captured image. (Item 18) the input data includes the captured image, Item 18. The image processing device according to item 17, wherein at least a portion of the input data is the captured image. (Item 19) An image processing method performed by an image processing device, comprising: obtaining input data relating to an image; acquiring features of the input data by processing the input data using a neural network; generating an integrated feature by integrating at least a portion of the input data and the feature; generating a processing result corresponding to the input data by processing the integrated feature; An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing device according to any one of items 1 to 18.

[0083] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0084] 100: information processing device, 201: data acquisition unit, 202: parameter acquisition unit, 203: feature generation unit, 204: feature integration unit, 205: post-processing unit, 206: storage unit

Claims

1. An acquisition means for acquiring input data including an image and a defocus map showing the amount of defocus for each position in the image, A feature acquisition means that acquires feature quantities of the input data by processing the input data using a neural network, An integration means for generating an integrated feature by integrating the defocus map and the feature quantities, Post-processing means that generates an estimation result of the defocus range for the subject in the image by processing the integrated feature quantities, An image processing device equipped with the following features.

2. The image processing apparatus according to claim 1, wherein the integration means integrates the defocus map and the feature quantity by calculating the element product.

3. The image processing apparatus according to claim 1, wherein the integrating means performs a process to match the spatial resolution between the defocus map and the feature quantity, and integrates the defocus map and the feature quantity after the process.

4. The image processing apparatus according to claim 1, wherein the integrating means performs a process to match the number of channels between the defocus map and the feature quantity, and integrates the defocus map after the process with the feature quantity.

5. The image processing apparatus according to claim 1, wherein the integration means performs a process of combining the defocus map and the feature quantity in the channel direction, and generates the integrated feature quantity by performing a sum-of-products operation on the data obtained by the combining process.

6. The image processing apparatus according to claim 1, characterized in that the integrating means performs noise reduction processing on the defocus map and generates the integrated feature by integrating the defocus map after the noise reduction processing with the feature.

7. The image processing apparatus according to claim 1, characterized in that the integration means performs a high-resolution processing on the defocus map and generates the integrated feature by integrating the defocus map after the high-resolution processing with the feature.

8. The image processing apparatus according to claim 1, characterized in that the processing of the integrated feature quantity is a linear operation.

9. The image processing apparatus according to claim 1, characterized in that the processing of the integrated feature quantity includes a fully connected processing.

10. The image processing apparatus according to claim 1, characterized in that the processing of the integrated features does not include nonlinear operations.

11. The image processing apparatus according to claim 1, characterized in that the processing of the input data includes nonlinear calculations.

12. The image processing apparatus according to claim 1, characterized in that the processing of the input data is processing in a neural network including an activation layer.

13. The image processing apparatus according to claim 1, characterized in that the computational load for processing the integrated feature quantity is less than the computational load for processing the input data.

14. The image processing apparatus according to claim 1, characterized in that the input data includes data specifying the region in the image in which the subject exists.

15. Acquisition means for acquiring input data including an image and a depth map showing depth values ​​for each position in the image, A feature acquisition means that acquires feature quantities of the input data by processing the input data using a neural network, An integration means for generating integrated features by integrating the depth map and the features, Post-processing means that generates an estimation result of the depth range for the subject in the image by processing the integrated feature quantities, An image processing device equipped with the following features.

16. The image processing apparatus according to claim 15, wherein the integration means integrates the depth map and the feature quantity by calculating the element product.

17. The image processing apparatus according to claim 15, wherein the integrating means performs a process to match the spatial resolution between the depth map and the feature quantity, and integrates the processed depth map and the feature quantity.

18. The image processing apparatus according to claim 15, wherein the integrating means performs a process to match the number of channels between the depth map and the feature quantity, and integrates the processed depth map and the feature quantity.

19. An image processing method performed by an image processing device, A step of acquiring input data including an image and a defocus map showing the amount of defocus for each position in the image, A step of obtaining feature quantities of the input data by processing the input data using a neural network, A step of generating an integrated feature by integrating the defocus map and the feature, A step of generating an estimated defocus range for the subject in the image by processing the integrated feature quantities, Image processing methods including [specific details omitted].

20. An image processing method performed by an image processing device, A step of acquiring input data including an image and a depth map showing depth values ​​for each position in the image, A step of obtaining feature quantities of the input data by processing the input data using a neural network, A step of generating an integrated feature by integrating the depth map and the feature, A step of generating an estimation result of the depth range for the subject in the image by processing the integrated feature quantities, Image processing methods including [specific details omitted].

21. A program for causing a computer to function as an image processing device according to any one of claims 1 to 18.

Images