Information processor, imaging apparatus, information processing method, information processing method in imaging apparatus

Abstract

To provide technology that enables the learning of models that robustly estimate imaging parameters.SOLUTION: A plurality of extended images with expanded information from the captured images are generated to train models for estimating imaging parameters based on these extended images.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to learning techniques. [Background technology]

[0002] In recent years, advances have been made in the development of cameras that allow users to customize their own settings. Technologies have been developed that automatically estimate shooting parameters such as exposure parameters and white balance according to the user's preferences. For example, there are systems that keep a log of changes to shooting parameters adjusted by the user and automatically calculate shooting parameter correction values ​​by referring to the past change log when shooting. There are also systems that generate shooting parameter recipes from various users in advance and then search and refer to the recipes when shooting to obtain the correct shooting parameters. However, these technologies determine shooting parameters based on rules and are therefore not flexible. Therefore, machine learning has recently been used to estimate shooting parameters. Patent Document 1 discloses a method for collecting training data for a learning model that uses captured images as input and can correctly infer the amount of shooting parameter correction. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-013061 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 uses parameters at the time of shooting as training data, and stores and accumulates live view images during shooting adjustment as learning data, making it possible to generate learning data naturally without the user's intention. However, because live view images during shooting adjustment tend to be concentrated around the appropriate shooting parameters, it has not been possible to robustly estimate shooting parameters for various input images. The present invention provides a technology that enables learning of a model that robustly estimates shooting parameters. [Means for solving the problem]

[0005] One aspect of the present invention is characterized by comprising a generation means for generating a plurality of extended images by extending information of a captured image, and a learning means for learning a model for estimating shooting parameters based on the extended images. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide a technique that enables learning of a model that robustly estimates imaging parameters. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration. [Figure 2] FIG. 1 is a block diagram showing an example of a hardware configuration applicable to a digital camera 102, a client terminal 105, a storage server 103, and a learning server 104. [Figure 3] 1 is a block diagram showing an example of the functional configuration of a digital camera 102, a client terminal 105, a storage server 103, and a learning server 104. [Figure 4] 10 is a flowchart of a process performed by the system to generate training data. [Figure 5] FIG. 4 is a diagram showing an example of the configuration of a shooting data list. [Figure 6] FIG. 4 is a diagram showing an example of the configuration of a shooting data list. [Figure 7] FIG. 3 is a block diagram showing an example of the functional configuration of a learning unit 312. [Figure 8] 4 is a flowchart of a photographing operation by the digital camera 102. [Figure 9] 10 is a flowchart of a process performed by the system to generate training data. [Figure 10] 4A and 4B are diagrams showing examples of histograms of images a and b. [Figure 11] 4 is a flowchart of a photographing operation by the digital camera 102. [Figure 12] 1 is a block diagram showing an example of the functional configuration of a digital camera 102, a client terminal 105, a storage server 103, and a learning server 104. [Figure 13] 10 is a flowchart of a process performed by the system to generate training data. [Figure 14] FIG. 4 is a diagram showing an example of the configuration of a shooting data list. [Figure 15] FIG. 3 is a block diagram showing an example of the functional configuration of a class-specific learning unit 312. [Figure 16] 4 is a flowchart of a photographing operation by the digital camera 102. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0009] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. A digital camera 102 and a client terminal 105 are connected to a local network 101 such as a LAN, and the local network 101 is connected to the Internet 100. A storage server 103 and a learning server 104 are connected to the Internet 100.

[0010] The digital camera 102 is an example of an imaging device capable of capturing moving images and still images. The digital camera 102 is an imaging device such as a mirrorless camera or a compact digital camera. The digital camera 102 generates imaging data including a captured image obtained by capturing an image and imaging parameters related to the capturing, and transmits (uploads) the generated imaging data to the accumulation server 103. The digital camera 102 also estimates "imaging parameters (correction amount)" using a trained estimation model, and corrects the current imaging parameters based on the estimated "imaging parameters (correction amount)."

[0011] The storage server 103 generates multiple extended images by extending the information of the captured image contained in the captured data transmitted from the digital camera 102, generates learning data including the captured data and the multiple extended images, and stores the learning data.

[0012] The learning server 104 learns the estimation model used in the digital camera 102 based on the augmented images stored in the storage server 103. The learning server 104 also deploys the trained estimation model to the digital camera 102 via the Internet 100 and the local network 101. If the camera to which the estimation model is deployed is different from the camera that acquires the captured data, the estimation model can be downloaded to the client terminal 105 and deployed to another camera via portable media such as a memory card. The client terminal 103 is a computer device such as a PC, smartphone, or tablet terminal.

[0013] Next, an example of a hardware configuration applicable to the digital camera 102, client terminal 105, storage server 103, and learning server 104 will be described using the block diagram in Fig. 2. When the hardware configuration shown in Fig. 2 is applied to the digital camera 102, this hardware configuration represents the hardware configuration of a calculation unit that handles images obtained by an optical system, an image sensor, an image processing circuit, etc.

[0014] The CPU 200 executes various processes using computer programs and data stored in the RAM 220. As a result, the CPU 200 controls the overall operation of an apparatus (application apparatus) to which the hardware configuration of Fig. 2 is applied, and executes or controls various processes described as processes performed by the application apparatus.

[0015] The ROM 210 stores setting data for the applicable device, computer programs and data relating to the startup of the applicable device, computer programs and data relating to the basic operation of the applicable device, and the like.

[0016] The RAM 220 has an area for storing computer programs and data loaded from the ROM 210 or the HDD 230, and an area for storing computer programs and data received from external devices via the communication unit 260. The RAM 220 also has a work area used by the CPU 200 when executing various processes. In this way, the RAM 220 can provide various areas as needed.

[0017] The HDD 230 stores an OS, computer programs and data for causing the CPU 200 to execute or control various processes described as processes performed by the application device.

[0018] An external storage device may also be used to fulfill a similar role. Here, the external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device connected via a network.

[0019] The input unit 240 is a user interface such as a keyboard, mouse, touch panel, button, lever, etc., and can be operated by the user to input various instructions and information to the applicable device.

[0020] The display unit 250 has an LCD screen or a touch panel screen, and can display the results of processing by the CPU 200 as images, text, etc. The display unit 250 may also be a projection device such as a projector that projects images and text. The learning server 104 may omit the input unit 240 and the display unit 250.

[0021] The communication unit 260 performs various processes for performing data communication with external devices. The CPU 200, the ROM 210, the RAM 220, the HDD 230, the input unit 240, the display unit 250, and the communication unit 260 are all connected to a system bus 270.

[0022] In this embodiment, the digital camera 102 (calculation unit), the client terminal 105, the storage server 103, and the learning server 104 are each described as having the hardware configuration shown in FIG. 2, but the present invention is not limited to this.

[0023] Next, an example of the functional configuration of each of the digital camera 102, client terminal 105, accumulation server 103, and learning server 104 will be described using the block diagram of FIG. 3. In this embodiment, the functional units shown in FIG. 3 (excluding storage units 302, 307, 310, and 316) are implemented by computer programs. In the following description, the functional units shown in FIG. 3 (excluding storage units 302, 307, 310, and 316) will be described as the main processing units, but in reality, the functions of these functional units are realized by the CPU 200 executing computer programs corresponding to these functional units. Note that one or more of the functional units shown in FIG. 3 may be implemented by hardware.

[0024] First, the digital camera 102 will be described. In response to a user pressing a shutter button (not shown), the photographing unit 301 performs a photographing operation based on parameters determined by the determination unit 308, thereby acquiring a photographed image and photographing parameters including various parameters related to the photographing operation (such as the shutter speed, aperture (F-number), ISO sensitivity, photographing position, and photographing date and time in the photographing operation). The photographing unit 301 then generates photographing data including the acquired photographed image and the acquired photographing parameters, and stores the photographing data in the storage unit 302.

[0025] The storage unit 302 stores the firmware execution program of the digital camera 102, shooting settings, user custom settings, trained estimation models, shooting data, etc. The storage unit 302 can be implemented using the ROM 210, RAM 220, HDD 230, etc.

[0026] The transmitting unit 303 transmits the photographic data stored in the storage unit 302 to the accumulation server 103. In this embodiment, the transmitting unit 303 reduces (resizes) the size of the photographic image included in the photographic data to a size of 32 pixels vertically by 32 pixels horizontally, and transmits the photographic data including the reduced photographic image and photographic parameters to the accumulation server 103. However, the reduced size is not limited to a specific size. Furthermore, the reduction may be performed using linear interpolation or nearest neighbor interpolation, and is not limited to a specific method. Furthermore, reducing the photographic image is not essential, and the photographic data including the photographic image and photographic parameters may be transmitted to the accumulation server 103 without being reduced.

[0027] The reading unit 304 reads out the estimation model stored in the storage unit 302 and loads the read estimation model into the RAM 220. The estimation unit 305 uses the estimation model loaded into the RAM 220 by the reading unit 304 to estimate shooting parameters (correction amounts) such as exposure, ISO sensitivity, shutter speed, and sharpness correction from a live view image (captured image) captured by the digital camera 102. The shooting parameters also include qualitative parameters such as whether or not HDR shooting is performed. In this embodiment, the estimation unit 305 reduces (resizes) the live view image captured by the digital camera 102 to a size of 32 pixels vertically by 32 pixels horizontally and uses the reduced image. Note that the reduced size of the live view image is not limited to a specific size as long as it is the same size as the captured image included in the capture data. Furthermore, the reduction may be performed using linear interpolation or nearest neighbor interpolation, and is not limited to a specific method. Furthermore, if the captured image included in the capture data is not reduced, the reduction of the live view image may be omitted. The determination unit 306 determines the parameters to be used for actual imaging based on the imaging parameters (correction amounts) estimated by the estimation unit 305.

[0028] Next, the accumulation server 103 will be described. The memory unit 307 stores the captured image data transmitted from the digital camera 102. The generation unit 308 generates multiple extended images by performing image processing on the captured image included in the captured image data stored in the memory unit 307. In this embodiment, the multiple extended images are generated by changing the brightness value of the captured image. The generation unit 308 then generates learning data including the captured image data and multiple extended images generated based on the captured image data, and stores the generated learning data in the memory unit 307. In response to an instruction from the learning server 104, the provision unit 309 transmits the learning data stored in the memory unit 307 to the learning server 104.

[0029] Next, we will explain the learning server 104. The memory unit 310 stores, for each project, definition files of estimation models, progress management of learning projects, learning data transmitted from the accumulation server 103, hyperparameters set during learning, estimation models, etc.

[0030] The generation unit 311 selects training data transmitted from the storage server 103 and performs preprocessing for inputting the training data into an estimation model. The generation unit 313 performs preprocessing for inputting the training data transmitted from the storage server 103 into an estimation model. For example, the generation unit 313 generates test data from the training data transmitted from the storage server 103. The method for generating test data from training data is not limited to a specific method, and for example, test data may be generated by processing part or all of the training data. Alternatively, test data may be generated without using training data.

[0031] The learning unit 312 learns an estimation model using learning data. For example, the learning unit 312 acquires "estimated shooting parameters," which are shooting parameters (correction amounts) estimated by the estimation model, by performing arithmetic processing of the estimation model to which an augmented image included in the learning data is input. The learning unit 312 then calculates the error (loss) between the shooting parameters included in the learning data and the acquired estimated shooting parameters, and updates the parameters (weights, etc.) of the estimation model so as to reduce the error, thereby learning the estimation model.

[0032] The test unit 314 performs estimation using the estimation model learned by the learning unit 312 using the test data generated by the generation unit 313, calculates the recall rate and precision rate based on specified test conditions, and outputs the performance of the estimation model obtained in this project.

[0033] For example, the test unit 314 performs computational processing on the estimation model to which the augmented images included in the test data are input to acquire "estimated imaging parameters," which are imaging parameters estimated (test estimated) by the estimation model. The test unit 314 then uses the acquired estimated imaging parameters to calculate a recall rate and a precision rate based on predetermined test conditions, and outputs "performance of the estimation model obtained in this project" based on the calculated results. Note that the method for acquiring "performance of the estimation model" from the estimated imaging parameters estimated by the estimation model is not limited to a specific method. For example, the "performance of the estimation model" may be acquired by processing only by the test unit 314, or may be acquired through user confirmation. Furthermore, the output destination and output format of the "performance of the estimation model" are not limited to a specific destination or output format. For example, the test unit 314 may display characters or graphs representing the "performance of the estimation model" as text or images on the display unit 250. Alternatively, for example, a message containing characters or graphs representing the "performance of the estimation model" may be sent to the client terminal 105 via the communication unit 260.

[0034] The deployment unit 315 transmits the trained estimation model, which has been authenticated by the user via test estimation as being OK to deploy to the digital camera 102, to the digital camera 102 in the form of binary information usable by the digital camera 102. For example, if the user checks the test estimation and determines that the trained estimation model can be deployed to the digital camera 102, the user operates the input unit 240 to input a deployment instruction. Upon receiving this instruction, the deployment unit 315 transmits the trained estimation model to the digital camera 102 in the form of binary information usable by the digital camera 102. The information is stored (downloaded) in the storage unit 302 of the digital camera 102 via the Internet 100 and the local network 101.

[0035] When the camera that generates the training data is different from the camera that deploys the estimation model, the transmission unit 316 transmits the trained estimation model to the client terminal 105. The transmitted estimation model is stored (downloaded) in the storage unit 317 of the client terminal 105 via the Internet 100 and the local network 101.

[0036] In this embodiment, a case will be described in which an "estimation model that estimates the exposure correction amount (exposure correction value) corresponding to a captured image" is trained using multiple extended images generated by varying the brightness value of the captured image. In this case, the digital camera 102 estimates the exposure correction value corresponding to the captured image using the estimation model, and corrects the exposure according to the estimated exposure correction value before capturing the image.

[0037] Next, the processing performed by the system according to this embodiment to generate learning data will be described with reference to the flowchart in Fig. 4. Loop L401 is a loop during shooting by the shooting unit 301. This loop starts when the power of the digital camera 102 is turned on. Within loop L401, shooting by the shooting unit 301 is repeatedly performed, and shooting data including the shot images obtained by the shooting and the shooting parameters related to the shooting are stored in the storage unit 302.

[0038] In step B401, the photographing unit 301 determines whether a photographing end condition has been met. The photographing end condition is not limited to a specific condition. For example, the photographing end condition may be that "a specified number / specified amount of photographed data has been stored in the storage unit 302" or "the elapsed time since photographing began exceeds a specified time." Further, for example, the photographing end condition may be that "the user has input an instruction to end photographing by operating the input unit 240 of the digital camera 102" or "the user has turned off the power by pressing the power button of the digital camera 102."

[0039] If the result of such determination is that the shooting end condition is met, the process exits loop L401 and proceeds to step S401, and if the shooting end condition is not met, loop L401 continues.

[0040] In step S401, the transmission unit 303 selects data necessary for learning from the imaging data stored in the storage unit 302, and transmits the selected data to the accumulation server 103. For ease of explanation, this embodiment will describe a case in which the transmission unit 303 transmits the imaging data stored in the storage unit 302 to the accumulation server 103. The imaging data transmitted to the accumulation server 103 is stored in the storage unit 307 of the accumulation server 103.

[0041] In this embodiment, the transmission unit 303 transmits a shooting data list, which is a list of shooting data (shot images, shooting parameters (ISO, shutter speed, aperture, shooting date and time)), to the accumulation server 103, as shown in FIG.

[0042] In step S402, the generating unit 308 generates a plurality of extended images from a captured image included in the captured data stored in the storage unit 307 by changing the brightness value of the captured image.

[0043] The following describes an example of processing by the generation unit 308. For each piece of shooting data in the shooting data list, the generation unit 308 generates an extended image from the captured image included in the shooting data that satisfies the following formula (1).

[0044]

number

[0045] …(1) where a ijindicates the luminance value at pixel position (i, j) in captured image a (the upper left corner of the captured image is the origin (0,0), with i being positive in the right direction and j being positive in the downward direction). β is a positive constant that represents the increase in luminance value from the captured image to the extended image, and x is the sum of the luminance values ​​of all pixels in the extended image. In this way, β is added in a large number of patterns to achieve a luminance value that is sufficiently far from the correct exposure, and multiple extended images are generated for one captured image. The luminance value of an image is calculated as the average of all pixel values ​​in the image. The luminance value is calculated by first averaging each of the three primary colors of the image: red (R), green (G), and blue (B), and then averaging the three RGB values.

[0046] As a result of processing the shooting data list of Figure 5 by the generation unit 308, the shooting data list is updated to include, for each shooting data, extended images 1, 2, ... generated from the shooting data and the brightness values ​​of the extended images, as shown in Figure 6.

[0047] In step S403, the generation unit 308 classifies the captured images in the capture data list into a plurality of groups in accordance with a specified condition. Various conditions can be applied to the "specified condition."

[0048] In this embodiment, the generation unit 308 identifies the earliest shooting date and time (first shooting date and time) and the latest shooting date and time (second shooting date and time) from the group of shooting dates and times included in the shooting data list. The generation unit 308 then equally divides the period between the first shooting date and time and the second shooting date and time into multiple divided periods, and includes the captured images with shooting dates and times that belong to each divided period in the group corresponding to the divided period. This allows each captured image in the shooting data list to be included in a group corresponding to one of the multiple divided periods, and as a result, the group of captured images in the shooting data list can be classified into multiple groups. Note that if the number of captured images belonging to a group is equal to or less than a threshold, the group may be deleted.

[0049] The group of photographed images in the photographing data list may be classified into multiple groups based on other information such as the photographing location, weather, etc. Also, the group of photographed images in the photographing data list may be classified into multiple groups based on the image recognition results for the photographed images.

[0050] In step S404, the generation unit 308 selects, for each group, captured images that meet a specified standard, and generates training data including multiple extended images generated from the selected captured images and the captured data of the selected captured images. For example, the generation unit 308 calculates the average luminance value of the captured images belonging to each group, and selects captured images belonging to the group whose luminance value difference from the average value is less than a threshold. That is, the generation unit 308 selects, for each group, captured images whose average luminance is within a specified range from the average luminance value of the captured images belonging to the group. The generation unit 308 then generates training data including multiple extended images generated from the selected captured images and the captured data of the selected captured images. The generation unit 308 then stores the generated training data in the storage unit 307.

[0051] The reason for performing the processing in step S403 is to group images with similar brightness values ​​and image configurations in order to select data within a range from the average brightness value in step S404. Without the grouping processing by shooting date and time in step S403, for example, there is a possibility that the average of images taken in the morning with high brightness values ​​and images taken in the evening with low brightness values ​​will be calculated, which would result in an inappropriate calculation using the data selection method in step S404.

[0052] An example of the functional configuration of the learning unit 312 will now be described with reference to the block diagram of FIG.

[0053] The acquisition unit 502 acquires the training data 501 stored in the storage unit 310 .

[0054] The estimator 503 performs calculation processing on an estimation model to which an extended image included in the learning data 501 acquired by the acquisition unit 502 is input, thereby acquiring an exposure compensation value estimated by the estimation model.

[0055] The loss calculation unit 504 calculates the error (loss) between the exposure correction value included in the training data 501 and the exposure correction value acquired by the estimator 503. The loss function used to calculate the error (loss) is the L1 loss, which is commonly used in regression tasks.

[0056] The weight update unit 505 learns the estimation model by updating the current "weights, which are parameters of the estimation model," stored in the storage unit 506, so that the error (loss) calculated by the loss calculation unit 504 becomes smaller.

[0057] In this embodiment, the estimation model having the weights stored in the storage unit 506 as parameters at the end of learning is output by the deploy unit 315 to a recording medium (such as an SD card) of the digital camera 102. However, the output destination of the estimation model is not limited to the digital camera 102, and may be, for example, the memory of a general-purpose computer or a control circuit inside the camera.

[0058] Note that various models can be applied to the above estimation model, such as a neural network like a CNN (Covolutional Neural Network), a ViT (Vision Transformer), SIFT, or an SVM (Support Vector Machine) combined with a feature extractor. In this embodiment, the explanatory variables are the extended images and the brightness values ​​corresponding to each extended image (collectively referred to as the extended images in the above description), and the objective variables are exposure parameters. Specifically, the exposure parameters are aperture, ISO sensitivity, and shutter speed.

[0059] Next, the photographing operation of the digital camera 102 will be described with reference to the flowchart in Fig. 8. In step S601, the photographing unit 301 starts live view and constantly exposes the image sensor. In step S602, the photographing unit 301 adjusts the aperture, ISO sensitivity, and shutter speed based on the set aperture value to achieve proper exposure.

[0060] In step S603, the estimation unit 305 inputs live view images sequentially output from the imaging unit 301 by live view using the imaging unit 301 into the estimation model that the reading unit 304 has loaded into the RAM 220.

[0061] In step S604, the estimation unit 305 estimates an exposure compensation value by performing calculation processing on the estimation model. The determination unit 306 determines parameters to be used for actual shooting based on the exposure compensation value estimated by the estimation unit 305. The shooting unit 301 then performs exposure compensation in accordance with the determined parameters. In step S605, the shooting unit 301 performs a shooting operation in response to the user pressing a shutter button (not shown) to obtain a shot image.

[0062] Thus, according to this embodiment, by performing image processing to change the brightness value of the captured image and expanding the data, it is possible to generate learning data that includes not only images around the correct exposure but also expanded images that are far from the correct exposure.

[0063] Furthermore, by using the shooting parameters of the captured image as the correct answer and using the extended image obtained by data extension from the captured image as the learning data, it is possible to collect learning data while shooting without the user being aware of it.

[0064] In this embodiment, an extended image is generated by changing the brightness value through image processing, but the image processing is not limited to processing that changes the brightness value, and may be, for example, image processing that changes the white balance, sharpness, or amount of blur.

[0065] <Modification> In the first embodiment, the luminance value of the extended image is calculated as a useful explanatory variable for estimating the exposure compensation value. However, it is assumed that the shooting parameters to be estimated are parameters other than exposure, such as shutter speed or white balance. Furthermore, the method for selecting the captured images in step S404 may not be able to select a good captured image because RGB changes significantly when the shooting scene changes rapidly.

[0066] Therefore, in this modified example, multiple extended images are generated using image processing other than image processing that changes brightness values. Furthermore, in this modified example, histogram intersection is utilized to select more useful information as learning data.

[0067] The process performed by the system according to this embodiment to generate training data will be described with reference to the flowchart in Fig. 9. In Fig. 9, the same process steps as those shown in Fig. 4 are assigned the same step numbers as those steps, and the description of those process steps will be omitted.

[0068] In step S802, the generation unit 308 applies motion blur of each of the multiple motion blur amounts (shake amounts) to the captured image included in the shooting data stored in the memory unit 307, thereby generating an extended image corresponding to the shake amount.

[0069] In step S804, the generation unit 308 selects, for each group, captured images that meet a specified criterion, and generates learning data including a plurality of augmented images generated from the selected captured images and the captured data of the selected captured images. For example, for each group, the generation unit 308 calculates the similarity between a target captured image belonging to the group and each captured image in the group other than the target captured image. If the number of captured images whose similarity with the target captured image is equal to or greater than a threshold is less than a specified number, the generation unit 308 designates the target captured image as a non-target captured image. The generation unit 308 performs this process for all captured images in the group.

[0070] For example, the generating unit 308 identifies the non-target captured images by utilizing a histogram intersection of the brightness values ​​of the captured images, which is a method for evaluating the degree to which two images are similar in color.

[0071] For example, if one of two images belonging to a group of interest is image a and the other is image b, the generation unit 308 first generates a histogram of the luminance values ​​of image a and a histogram of the luminance values ​​of image b, as shown in Fig. 10. The histograms in Fig. 10 represent the frequency (number of pixels) corresponding to each luminance value from 0 to 255. Here, the frequency corresponding to the i-th bin (0≦i≦255) in image a (the number of pixels having luminance value i in image a) is denoted as ai, and the frequency corresponding to the i-th bin in image b (the number of pixels having luminance value i in image b) is denoted as bi. In this case, the similarity D between image a and image b based on the histogram intersection can be calculated according to the following equation (2):

[0072]

number

[0073] …(2) Since equation (2) is an equation for grayscale images, when the captured image is a color image, for example, the color image may be converted into a grayscale image and the converted grayscale image may be used to calculate the similarity D. However, when the captured image is a color image, the method for calculating the similarity D is not limited to a specific calculation method, and for example, the value of D may be found for each color component of RGB according to equation (2), and the average value of the values ​​of D found for each of RGB may be used as the similarity.

[0074] The generating unit 308 then selects, for each group, a captured image that is not a non-target captured image in the group, and generates learning data including a plurality of extended images generated from the selected captured images and the captured data of the selected captured images. The generating unit 308 then stores the generated learning data in the storage unit 307.

[0075] Next, the photographing operation of the digital camera 102 will be described with reference to the flowchart of Fig. 11. In step S901, the photographing unit 301 sets the photographing mode to aperture priority mode.

[0076] In step S902, the image capturing unit 301 starts live view and constantly exposes the image sensor. In step S903, the image capturing unit 301 sets a fixed aperture value. In step S904, the image capturing unit 301 sets exposure parameters other than the aperture, that is, the ISO and shutter speed.

[0077] In step S905, the estimation unit 305 inputs live view images sequentially output from the imaging unit 301 by live view using the imaging unit 301 into the estimation model that the reading unit 304 has loaded into the RAM 220.

[0078] In step S906, the estimation unit 305 estimates an exposure compensation value by performing calculation processing on the estimation model. The determination unit 306 determines parameters to be used for actual shooting based on the exposure compensation value estimated by the estimation unit 305. The shooting unit 301 then performs exposure compensation (correction of ISO and shutter speed) in accordance with the determined parameters. In step S907, the shooting unit 301 performs a shooting operation in response to the user pressing a shutter button (not shown) to acquire a shot image.

[0079] The effect of this modification is that by performing image processing of motion blur on the captured image and expanding the data, it is possible to generate training data for estimating the appropriate shutter speed. Furthermore, by utilizing histogram intersection, it is possible to select more useful data as training data.

[0080] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified, it is assumed that the present embodiment is the same as the first embodiment. In this embodiment, learning data is generated for learning an estimation model for estimating shooting parameters (correction amounts) for detecting a subject from a captured image.

[0081] An example of the functional configuration of each of the digital camera 102, client terminal 105, storage server 103, and learning server 104 will be described using the block diagram in Fig. 12. In Fig. 12, functional units that are the same as those shown in Fig. 3 are given the same reference numerals, and descriptions of those functional units will be omitted.

[0082] The detection unit 318 performs a detection process to detect a subject from a captured image using a trained model (detector) for subject detection, thereby obtaining a detection set including "classification results of the subject detected from the captured image," "reliability of the classification result," and "BB (Bounding Box) information that defines the BB of the subject in the captured image." If the "reliability of the classification result" in the detection set is equal to or greater than a threshold, the detection unit 318 outputs the detection set as a detection result, and if the "reliability of the classification result" in the detection set is less than the threshold, the detection unit 318 outputs an undetected flag instead of the detection set as a detection result.

[0083] The BB information may be, for example, information indicating the coordinate position of the vertex of the upper left corner of the BB in the photographed image and the vertical and horizontal sizes of the BB, information indicating the coordinate positions of opposing vertices of the BB, or information indicating the coordinate positions of the vertices of the four corners of the BB. In addition, in this embodiment, the classification result of the subject class will be described as being one of person, dog, cat, bird, and car, but is not limited to this.

[0084] In this embodiment, for each class, an image containing a subject of that class, the class of the subject, and BB information of the subject in the image are prepared in advance as training data, and the detector is trained in advance using such training data.

[0085] The detection unit 318 generates a detection result by inputting a captured image to a detector and performing arithmetic processing on the detector, and assigns the detection result to the captured data including the captured image. The storage unit 302 stores, for each class, an estimation model for estimating shooting parameters (correction amounts) for detecting a subject of that class from the captured image. The reading unit 304 reads the estimation model corresponding to each class into the RAM 220.

[0086] The process performed by the system according to this embodiment to generate training data will be described with reference to the flowchart in Fig. 13. In Fig. 13, the same process steps as those shown in Fig. 4 are assigned the same step numbers as those steps, and the description of those process steps will be omitted.

[0087] In step S1001, the transmission unit 303 performs the same operation as in step S401 above, and the photographic data transmitted to the accumulation server 103 is stored in the memory unit 307 of the accumulation server 103. In this embodiment, the transmission unit 303 transmits to the accumulation server 103 a photographic data list, which is a list of sets of photographic data (photographed image, photographing parameters (ISO, shutter speed, aperture, photographing date and time)) and detection results from the photographed image, as shown in Fig. 14 .

[0088] In step S1002, the generating unit 308 generates a plurality of extended images from a captured image included in the captured data stored in the storage unit 307 by changing the brightness value of the captured image.

[0089] More specifically, if the detection result assigned to the photographic data is a detection set (i.e., not an undetected flag), the generation unit 308 crops the image in the photographic image in the photographic data within the "BB defined by the BB information in the detection result assigned to the photographic data" as a cropped image, and replaces the photographic image in the photographic data with the cropped image.The generation unit 308 then sets the cropped image as the photographic image in the photographic data, and generates from the photographic image a plurality of extended images in which the luminance values ​​of the photographic image are changed, in the same manner as in the first embodiment.

[0090] On the other hand, if the detection result assigned to the shooting data is an undetected flag, the generation unit 308 generates multiple extended images from the shooting image included in the shooting data by changing the brightness value of the shooting image, as in the first embodiment.

[0091] In this embodiment, the generating unit 308 generates training data in the same manner as in the modified example of the first embodiment, but groups the generated training data according to the detection results included in the training data.

[0092] For example, if the detection result included in the training data is an undetected flag, the generation unit 308 places the training data in an "undetected" group. On the other hand, suppose that the detection result included in the training data is not an undetected flag. In this case, if the classification result included in the detection result included in the training data is "person," the generation unit 308 places the training data in the "person" group, and if the classification result is "dog," the generation unit 308 places the training data in the "dog" group. Furthermore, if the classification result is "cat," the generation unit 308 places the training data in the "cat" group; if the classification result is "bird," the generation unit 308 places the training data in the "bird" group; and if the classification result is "car," the generation unit 308 places the training data in the "car" group.

[0093] In this embodiment, as shown in FIG. 15 , a learning unit 312 having the same configuration as in the first embodiment is provided for each class. FIG. 15 shows, as an example, a learning unit 312 corresponding to one class and a learning unit 312 corresponding to the other class. Learning data corresponding to the class group corresponding to the learning unit 312 is input to each learning unit 312, and the learning process in the learning unit 312 is performed based on the learning data. As a result, the learning unit 312 can learn an estimation model for estimating shooting parameters (correction amounts) for detecting a subject of the corresponding class. As in the first embodiment, the estimation model corresponding to each class is output to a recording medium (such as an SD card) of the digital camera 102 by the deploy unit 315, for example.

[0094] Next, the photographing operation of the digital camera 102 will be described with reference to the flowchart in Fig. 16. In Fig. 16, the same processing steps as those shown in Fig. 8 are assigned the same step numbers as those steps, and the description of those processing steps will be omitted.

[0095] In step S1103, the detection unit 318 inputs live view images sequentially output from the imaging unit 301 by live view using the imaging unit 301 to a detector, and performs arithmetic processing on the detector to obtain a detection result.

[0096] In step B1101, the detection unit 318 determines whether or not a subject (subject class) has been detected from the live-view image. For example, if the detection result is an undetected flag, the detection unit 318 determines that a subject has not been detected from the live-view image, and the process proceeds to step S1104. On the other hand, if the detection result is not an undetected flag, the detection unit 318 determines that a subject has been detected from the live-view image, and the process proceeds to step S1107.

[0097] In step S1104, the estimation unit 305 inputs the live view image to an estimation model trained using the learning data of the "undetected" group. In step S1105, the estimation unit 305 estimates an exposure compensation value by performing arithmetic processing on the estimation model trained using the learning data of the "undetected" group. The determination unit 306 determines parameters to be used for actual shooting based on the exposure compensation value estimated by the estimation unit 305. The shooting unit 301 then performs exposure compensation according to the determined parameters.

[0098] In step S1107, the estimation unit 305 crops the image in the area defined by the "BB information included in the detection result" in the live view image as a cropped image.

[0099] In step S1108, the estimation unit 305 selects, from the trained estimation models, an estimation model trained using training data of a group corresponding to the class of the classification result included in the detection result, as a selected estimation model, and then inputs the cropped image to the selected estimation model.

[0100] In step S1109, the estimation unit 305 estimates an exposure compensation value by performing calculation processing on the selected estimation model. The determination unit 306 determines parameters to be used for actual shooting based on the exposure compensation value estimated by the estimation unit 305. The shooting unit 301 then performs exposure compensation in accordance with the determined parameters.

[0101] An effect of this embodiment is that by using training data including extended images obtained by data-extending cropped images, it is possible to generate an estimation model that estimates the correction amount of shooting parameters specific to the detected subject.

[0102] In the above embodiments and variants, the client terminal 105, the storage server 103, and the learning server 104 are each described as separate devices, but this is not limited to this, and two or more of these devices may be integrated into a single device.

[0103] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in the above embodiments and variant examples are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0104] Furthermore, some or all of the embodiments and modifications described above may be used in appropriate combination, and some or all of the embodiments and modifications described above may be used selectively.

[0105] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0106] The invention of this specification includes the following information processing device, image capturing device, information processing method, information processing method in image capturing device, and computer program. (Item 1) a generating means for generating a plurality of extended images by extending information of the captured image; a learning means for learning a model for estimating photographing parameters based on the augmented image; An information processing device comprising: (Item 2) 2. The information processing device according to item 1, wherein the generating means generates a plurality of extended images by extending information of the captured images captured until a shooting end condition is satisfied. (Item 3) 3. The information processing device according to item 1 or 2, wherein the generating means generates a plurality of extended images by performing image processing on the captured image. (Item 4) 4. The information processing device according to item 3, wherein the generating means generates a plurality of extended images by changing the brightness value of the captured image. (Item 5) 4. The information processing device according to item 3, wherein the generating means generates a plurality of extended images by applying motion blur with different amounts of blur to the captured image. (Item 6) The generating means 4. The information processing device according to item 3, characterized in that, when a subject is detected from the captured image, a plurality of extended images are generated by changing the brightness value of the image of the subject. (Item 7) The generating means 4. The information processing device according to item 3, characterized in that, when a subject is not detected from the captured image, a plurality of extended images are generated by changing the brightness value of the captured image. (Item 8) The learning means An information processing device according to any one of items 1 to 7, characterized in that a plurality of captured images are classified into a plurality of groups based on the date and time of capture, and the learning is performed based on a plurality of extended images obtained by extending information on captured images in the group that meet specified criteria. (Item 9) The learning means Item 9. The information processing device according to item 8, characterized in that the learning is performed based on a plurality of extended images obtained by extending information on captured images having brightness within a specified range from the average brightness value of the captured images belonging to the group. (Item 10) The learning means The information processing device according to item 8 is characterized in that the similarity between a target photographic image belonging to the group and each photographic image in the group other than the target photographic image is calculated, and if the number of photographic images having a similarity to the target photographic image equal to or greater than a threshold is less than a specified number, the target photographic image is designated as a non-target photographic image, and the learning is performed based on a plurality of extended images obtained by extending information of photographic images in the plurality of groups that are not non-target photographic images. (Item 11) The learning means 9. The information processing device according to item 8, wherein the information processing device performs learning of an estimation model for estimating shooting parameters for detecting a subject of a class corresponding to a captured image that satisfies a specified standard in the group. (Item 12) 12. The information processing device according to any one of items 1 to 11, wherein the shooting parameters include an exposure compensation value. (Item 13) moreover, 13. The information processing device according to any one of items 1 to 12, further comprising a deployment unit that deploys the model trained by the learning unit. (Item 14) Photography means, An estimation means for estimating the shooting parameters for the current captured image using a learning model in which learning for estimating the shooting parameters is performed based on a plurality of augmented images obtained by augmenting information on images captured in the past. An imaging device comprising: (Item 15) An information processing method performed by an information processing device, a generation step in which a generation means of the information processing device generates a plurality of extended images by extending information of the captured image; a learning step in which a learning means of the information processing device learns a model for estimating shooting parameters based on the extended image; An information processing method comprising: (Item 16) The shooting process and an estimation step of estimating the shooting parameters for the current captured image using a learning model in which learning for estimating the shooting parameters is performed based on a plurality of augmented images obtained by augmenting information on images captured in the past; 1. An information processing method for an imaging device, comprising: (Item 17) A computer program for causing a computer to function as each means of the information processing device according to any one of items 1 to 13.

[0107] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0108] 102: Digital camera 103: Storage server 104: Learning server 105: Client terminal 301: Photography unit 302: Memory unit 303: Transmission unit 304: Reading unit 305: Estimation unit 306: Determination unit 307: Memory unit 308: Generation unit 309: Provision unit 310: Memory unit 311: Generation unit 312: Learning unit 313: Generation unit 314: Test unit 315: Deployment unit 316: Transmission unit 317: Memory unit

Claims

1. a generating means for generating a plurality of extended images by extending information of the captured image; a learning means for learning a model for estimating photographing parameters based on the augmented image; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the generating means generates a plurality of extended images by extending information of the images captured until a shooting termination condition is satisfied.

3. 2. The information processing apparatus according to claim 1, wherein the generating means generates a plurality of extended images by performing image processing on the photographed image.

4. 4. The information processing apparatus according to claim 3, wherein the generating means generates a plurality of extended images by changing a luminance value of the photographed image.

5. 4. The information processing apparatus according to claim 3, wherein the generating means generates a plurality of extended images by applying motion blur to the photographed image with different amounts of blur.

6. The generating means 4. The information processing apparatus according to claim 3, wherein, when a subject is detected from the photographed image, a plurality of extended images are generated by changing a luminance value of the image of the subject.

7. The generating means 4. The information processing apparatus according to claim 3, wherein, when a subject is not detected from the photographed image, a plurality of extended images are generated by changing a luminance value of the photographed image.

8. The learning means The information processing device according to claim 1, characterized in that a plurality of captured images are classified into a plurality of groups based on the date and time of capture, and the learning is performed based on a plurality of extended images obtained by extending information of captured images in the group that meet specified criteria.

9. The learning means The information processing apparatus according to claim 8, wherein the learning is performed based on a plurality of extended images obtained by extending information on photographed images having brightness values ​​within a specified range from an average brightness value of the photographed images belonging to the group.

10. The learning means The information processing device described in claim 8, characterized in that the similarity between a target photographic image belonging to the group and each photographic image in the group other than the target photographic image is calculated, and if the number of photographic images whose similarity with the target photographic image is equal to or greater than a threshold is less than a specified number, the target photographic image is designated as a non-target photographic image, and the learning is performed based on a plurality of extended images obtained by extending information of photographic images in the plurality of groups that are not non-target photographic images.

11. The learning means 9. The information processing apparatus according to claim 8, wherein the information processing apparatus performs learning of an estimation model for estimating shooting parameters for detecting a subject of a class corresponding to a photographed image that satisfies a specified standard in the group.

12. 2. The information processing apparatus according to claim 1, wherein the photographing parameters include an exposure compensation value.

13. moreover, The information processing apparatus according to claim 1 , further comprising: a deploying unit that deploys the model trained by the learning unit.

14. Photography means, An estimation means for estimating the shooting parameters for the current captured image using a learning model in which learning for estimating the shooting parameters is performed based on a plurality of augmented images obtained by augmenting information on images captured in the past. An imaging device comprising:

15. An information processing method performed by an information processing device, a generation step in which a generation means of the information processing device generates a plurality of extended images by extending information of the captured image; a learning step in which a learning means of the information processing device learns a model for estimating shooting parameters based on the extended image; An information processing method comprising:

16. The shooting process and an estimation step of estimating the shooting parameters for the current captured image using a learning model in which learning for estimating the shooting parameters is performed based on a plurality of augmented images obtained by augmenting information on images captured in the past; 1. An information processing method for an imaging device, comprising:

17. A computer program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 13.

Images