Information processing device and information processing method

Abstract

To provide a technique for acquiring photographed images that suit a user's image processing preferences.SOLUTION: An information processing device acquires recommended shooting parameters on the basis of the content of image processing performed on shooting information obtained by shooting, and associates the shooting information with the recommended shooting parameters and registers them as learning data.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to information processing technology. [Background technology]

[0002] In recent years, not only digital cameras but also smartphones have become capable of recording raw sensor values, known as RAW, as shooting data. RAW images have a deeper bit depth than the common image format JPEG, and offer greater freedom in image editing. Therefore, RAW image editing is essential for correcting images taken with incorrect camera settings or for pursuing higher quality images.

[0003] Patent Document 1 proposes a method of creating a database of image feature amounts and correction processes, extracting feature amounts from a new image, and determining suitable correction process parameters from the database. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-187215 Summary of the Invention [Problem to be solved by the invention]

[0005] However, because image correction is a process that corrects an already captured image through digital numerical calculations, there is a problem that the larger the correction value, the more the image quality deteriorates. For example, this tendency is particularly pronounced when the signal volume of the RAW image itself is low, such as when shooting at night. This invention provides a technology for acquiring captured images that meet the user's image processing preferences. [Means for solving the problem]

[0006] One aspect of the present invention is characterized by comprising an acquisition means for acquiring recommended shooting parameters based on the content of image processing performed on shooting information obtained by shooting, and a registration means for associating the shooting information with the recommended shooting parameters and registering them as learning data. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique for acquiring a captured image that meets the user's image processing preferences. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration. [Figure 2] FIG. 1 is a block diagram showing an example of a hardware configuration applicable to a digital camera 102, a client terminal 103, a storage server 104, and a learning server 105. [Figure 3] 1 is a block diagram showing an example of the functional configuration of a digital camera 102, a client terminal 103, a storage server 104, and a learning server 105. [Figure 4] 10 is a flowchart of a process performed by the system to generate training data. [Figure 5] FIG. 3 is a block diagram showing an example of the functional configuration of a learning unit 316. [Figure 6] 4 is a flowchart of a photographing operation by the digital camera 102. [Figure 7] 10 is a flowchart of a series of processes for performing image editing on an image captured in aperture priority mode and generating learning data. [Figure 8] FIG. 10 is a diagram showing an example of a GUI display. [Figure 9] 10 is a flowchart of a process performed by the system to generate training data. [Figure 10] 4 is a flowchart of a photographing operation by the digital camera 102. [Figure 11] FIG. 10 is a diagram showing a display example of a recipe screen. [Figure 12]10 is a flowchart of a process performed by the client terminal 103 to display a recipe screen on the display unit 250. [Figure 13] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 14] 10 is a flowchart of a process performed by the system to generate training data. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. A digital camera 102 and a client terminal 103 are connected to a local network 101 such as a LAN, and the local network 101 is connected to the Internet 100. A storage server 104 and a learning server 105 are connected to the Internet 100. The digital camera 102 is an example of an imaging device capable of capturing moving images and still images.

[0011] The client terminal 103 is a computer device such as a PC, smartphone, or tablet terminal, and performs image processing (image processing / image editing) on ​​a RAW image captured by the digital camera 102 in response to a user operation. The client terminal 103 then generates a data set obtained by the image processing, and transmits (uploads) the data set to the storage server 104. The storage server 104 generates learning data to be used in supervised learning from the data set transmitted from the client terminal 103, and stores the generated learning data.

[0012] The learning server 105 performs training of the inference model used in the digital camera 102 in response to instructions from the user, tests the inference model obtained through the training, and creates a report. The learning server 105 also deploys the trained inference model to the digital camera 102 via the Internet 100 and the local network 101. If the digital camera 102 cannot connect to the Internet 100, the trained inference model can be downloaded to the client terminal 103 and deployed to the digital camera 102 via portable media such as a memory card.

[0013] Next, an example of a hardware configuration applicable to the digital camera 102, client terminal 103, storage server 104, and learning server 105 will be described using the block diagram in Fig. 2. When the hardware configuration shown in Fig. 2 is applied to the digital camera 102, this hardware configuration represents the hardware configuration of a calculation unit that handles images obtained by an optical system, an image sensor, an image processing circuit, etc.

[0014] The CPU 200 executes various processes using computer programs and data stored in the RAM 220. As a result, the CPU 200 controls the overall operation of an apparatus (application apparatus) to which the hardware configuration of Fig. 2 is applied, and executes or controls various processes described as processes performed by the application apparatus.

[0015] The ROM 210 stores setting data for the applicable device, computer programs and data relating to the startup of the applicable device, computer programs and data relating to the basic operation of the applicable device, and the like.

[0016] The RAM 220 has an area for storing computer programs and data loaded from the ROM 210 or the HDD 230, and an area for storing computer programs and data received from external devices via the communication unit 260. The RAM 220 also has a work area used by the CPU 200 when executing various processes. In this way, the RAM 220 can provide various areas as needed.

[0017] The HDD 230 stores an OS, computer programs and data for causing the CPU 200 to execute or control various processes described as processes performed by the application device.

[0018] An external storage device may also be used to fulfill a similar role. Here, the external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device connected via a network.

[0019] The input unit 240 is a user interface such as a keyboard, mouse, touch panel, button, lever, etc., and can be operated by the user to input various instructions and information to the applicable device.

[0020] The display unit 250 has a liquid crystal screen or a touch panel screen, and can display, as images or text, the results of processing by the CPU 200. Note that the display unit 250 may also be a projection device such as a projector that projects images or text.

[0021] The communication unit 260 performs various processes for performing data communication with external devices. The CPU 200, the ROM 210, the RAM 220, the HDD 230, the input unit 240, the display unit 250, and the communication unit 260 are all connected to a system bus 270.

[0022] In this embodiment, the digital camera 102 (calculation unit), the client terminal 103, the storage server 104, and the learning server 105 are each described as having the hardware configuration shown in FIG. 2, but the present invention is not limited to this.

[0023] Next, an example of the functional configuration of each of the digital camera 102, client terminal 103, accumulation server 104, and learning server 105 will be described using the block diagram of FIG. 3. In this embodiment, the functional units shown in FIG. 3 (excluding storage units 301, 311, and 314) are implemented by computer programs. In the following description, the functional units shown in FIG. 3 (excluding storage units 301, 311, and 314) will be described as the main processing units, but in reality, the functions of these functional units are realized by the CPU 200 executing computer programs corresponding to these functional units. Note that one or more of the functional units shown in FIG. 3 may be implemented by hardware.

[0024] First, we will explain the digital camera 102. The storage unit 301 stores the firmware execution program, shooting settings, user custom settings, trained inference models, etc. of the digital camera 102. The storage unit 301 can be implemented using the ROM 210, RAM 220, HDD 230, etc.

[0025] The reading unit 302 reads out the inference model stored in the storage unit 301 and expands the read-out inference model into the RAM 220. The inference unit 303 uses the inference model expanded into the RAM 220 by the reading unit 302 to infer shooting parameters, which are correction values ​​for parameters such as exposure, ISO sensitivity, shutter speed, etc., from a live view image (reduced developed image) captured by the digital camera 102. The shooting parameters also include qualitative parameters such as whether or not to perform HDR shooting.

[0026] The determination unit 304 determines parameters to be used for actual shooting based on the shooting parameters inferred by the inference unit 303. The shooting unit 305 acquires a RAW image by performing a shooting operation based on the parameters determined by the determination unit 304 in response to a user pressing a shutter button (not shown). The shooting unit 305 then assigns well-known Exif information to the RAW image, including various information related to the shooting operation (such as the shutter speed, aperture (F-number), and ISO sensitivity in the shooting operation), and stores the RAW image with the Exif information assigned in the storage unit 301.

[0027] Next, a description will be given of the client terminal 103. The client terminal 103 acquires a RAW image taken by the digital camera 102, and performs image processing on the RAW image in accordance with a user operation.

[0028] A GUI (Graphical User Interface) control unit 306 controls the display of software GUI that performs image processing on a RAW image according to user operations, by displaying the GUI on the display unit 250. The GUI control unit 306 can, for example, display a list of image editing parameters, display editing results that are output in accordance with parameter changes, and display a comparison of images before and after editing.

[0029] In response to a user operation, the image reading unit 307 reads out a RAW image to be subjected to image processing from the HDD 230 to the RAM 220 from the RAW image group received from the digital camera 102 and stored in the HDD 230 .

[0030] The processing unit 308 performs image processing (adjustment of brightness, white balance, etc.) on the RAW image read into the RAM 220 by the image reading unit 307 in accordance with a user operation on the GUI, and outputs a captured image obtained by developing the RAW image that has been subjected to image processing to the GUI control unit 306. This enables the GUI control unit 306 to display, on the GUI, a captured image obtained by developing the RAW image that has been subjected to image processing by the processing unit 308.

[0031] The output unit 309 generates a captured image by performing various image processes, including development processing, on the RAW image that has been subjected to image processing by the processing unit 308. The transmission unit 310 generates a data set including "a thumbnail image (a reduced image developed at the time of shooting) of the RAW image read into the RAM 220 by the image reading unit 307," "the captured image that has been subjected to development processing by the processing unit 308," and "image processing details." The transmission unit 310 then transmits (uploads) the generated data set to the accumulation server 104.

[0032] Next, we will explain the accumulation server 104. The accumulation server 104 functions as an information processing device that acquires recommended shooting parameters based on the content of image processing performed on shooting information obtained by shooting, and associates the shooting information with the recommended shooting parameters and registers them as learning data.

[0033] The storage unit 311 stores a data set uploaded from the client terminal 103. The storage unit 311 can be implemented using the ROM 210, the RAM 220, the HDD 230, and the like.

[0034] The conversion unit 312 acquires recommended shooting parameters (recommended shooting parameters) from the Exif information included in the dataset stored in the storage unit 311 and the image processing details included in the dataset. The recommended shooting parameters include, for example, one or more of an ISO sensitivity correction value, an aperture correction value, a shutter speed correction value, etc. The conversion unit 312 then generates learning data including the dataset and the recommended shooting parameters, and stores the generated learning data in the storage unit 311. In response to an instruction from the learning server 105, the provision unit 313 transmits some or all of the learning data stored in the storage unit 311 to the learning server 105.

[0035] Next, we will explain the learning server 105. The learning server 105 can learn an inference model that the inference unit 303 uses to infer shooting parameters, test the inference model, and deploy the inference model to the digital camera 102.

[0036] The memory unit 314 stores, for each project, definition files of inference models, progress management of learning projects, data used for learning and testing, hyperparameters set for learning, trained inference models, and so on.

[0037] The generation unit 315 adjusts the frequency of occurrence of each piece of training data transmitted from the provision unit 313 and performs preprocessing for inputting the training data into an inference model. The generation unit 317 does not adjust the frequency of occurrence of each piece of training data transmitted from the provision unit 313, but performs preprocessing for inputting the training data into an inference model. For example, the generation unit 317 generates test data from the training data transmitted from the provision unit 313. The method for generating test data from training data is not limited to a specific method, and for example, test data may be generated by processing part or all of the training data. Alternatively, test data may be generated without using training data.

[0038] The learning unit 316 uses the learning data to learn the inference model. For example, the learning unit 316 acquires "inferred shooting parameters," which are shooting parameters inferred by the inference model, by performing calculation processing on the inference model to which a "thumbnail image of a RAW image" included in the learning data is input. The learning unit 316 then calculates the error between the recommended shooting parameters included in the learning data and the acquired inferred shooting parameters, and learns the inference model by updating the parameters (weights, etc.) of the inference model so as to reduce the error.

[0039] The test unit 318 performs inference using the inference model learned by the learning unit 316 using the test data generated by the generation unit 317, calculates the recall rate and precision rate based on specified test conditions, and outputs the performance of the inference model obtained in this project.

[0040] For example, the test unit 318 acquires "inference shooting parameters," which are shooting parameters inferred (test inference) by the inference model by performing calculations on the inference model to which a "thumbnail image of a RAW image" included in the test data is input. The learning unit 316 then uses the acquired inference shooting parameters to calculate a recall rate and a precision rate based on predetermined test conditions, and outputs "performance of the inference model obtained in this project" based on the calculated results. Note that the method for acquiring "performance of the inference model" from the inference shooting parameters inferred by the inference model is not limited to a specific method. The "performance of the inference model" may be acquired by processing only by the test unit 318, or may be acquired through user confirmation. Furthermore, the output destination and output format of the "performance of the inference model" are not limited to a specific destination or output format. For example, the test unit 318 may display characters or graphs representing the "performance of the inference model" as text or an image on the display unit 250. Alternatively, for example, a message containing characters or graphs representing the "performance of the inference model" may be sent to the client terminal 103 via the communication unit 260.

[0041] The deployment unit 319 transmits to the digital camera 102, via test inference, an inference model that has been authenticated by the user as being OK to deploy to the digital camera 102, in the form of binary format information usable by the digital camera 102. For example, if the user checks the test inference and determines that the trained inference model can be deployed to the digital camera 102, the user operates the input unit 240 to input a deployment instruction. Upon receiving this instruction, the deployment unit 319 transmits the trained inference model to the digital camera 102 in the form of binary format information usable by the digital camera 102. The information is stored (downloaded) in the storage unit 301 of the digital camera 102 via the Internet 100 and the local network 101.

[0042] In the following, it is assumed that the user performs an adjustment operation to adjust the brightness of a RAW image as image processing for the RAW image by operating the client terminal 103. In this case, the learning server 105 learns an inference model that infers an exposure compensation value corresponding to a thumbnail image of the input RAW image, and the digital camera 102 uses the inference model to infer an exposure compensation value corresponding to the captured RAW image.

[0043] Next, the process performed by the system according to this embodiment to generate training data will be described with reference to the flowchart in FIG. 4. When a user of the client terminal 103 operates the input unit 240 to input an instruction to launch image editing software, the GUI control unit 306 displays the GUI of the image editing software on the display unit 250. The output unit 309 performs various image processing, including development processing, on the RAW image read by the image reading unit 307 from the HDD 230 to the RAM 220 to generate a captured image. The GUI control unit 306 then displays the captured image developed by the output unit 309 on the GUI. When the user operates the input unit 240 to input an instruction to adjust brightness, in step S401, the processing unit 308 adjusts the brightness of the RAW image in accordance with the instruction, and the output unit 309 develops the RAW image to which the brightness adjustment value has been applied to generate a captured image and outputs the developed image to the GUI control unit 306. This allows the GUI control unit 306 to display the captured image on the GUI.

[0044] The user then checks the captured image displayed on the GUI and determines whether the brightness has been adjusted to the desired level. If the user determines that the captured image displayed on the GUI has been adjusted to the desired level and inputs an instruction to that effect by operating input unit 240, the process proceeds to step S403 via step S402. On the other hand, if such an instruction has not been input, the process proceeds to step S401 via step S402.

[0045] In step S403, the output unit 309 generates a data set including “thumbnail images of RAW images read into the RAM 220 by the image reading unit 307,” “captured images developed according to brightness adjustment values,” and “contents of brightness adjustment.” The thumbnail images of RAW images are reduced, unedited images developed with the settings at the time of shooting used for GUI display.

[0046] In step S404, the transmission unit 310 transmits (uploads) the data set generated in step S403 to the accumulation server 104. These images may include images that have not been edited as a result.

[0047] In step S405, the conversion unit 312 stores the data set uploaded from the client terminal 103 in the storage unit 311. Then, the conversion unit 312 acquires a recommended exposure compensation value as a recommended shooting parameter from the Exif information included in the data set stored in the storage unit 311 and the brightness adjustment content (brightness adjustment value) included in the data set.

[0048] Here, if the parameter for adjusting brightness is the gain amount of digital gain as it is, the brightness adjustment value is adopted as it is as the recommended exposure compensation value. The recommended exposure compensation value is GT (Ground Truth) data in this embodiment, and can be thought of as the exposure compensation value that should have been corrected in order to obtain an exposure that the user considers appropriate for an image captured by automatically or manually determining the exposure.

[0049] The conversion unit 312 then generates learning data including the data set and the recommended exposure compensation value obtained using the data set, and stores the generated learning data in the storage unit 311.

[0050] An example of the functional configuration of the learning unit 316 will now be described with reference to the block diagram in Fig. 5. The acquisition unit 502 acquires learning data 501. The inference unit 503 performs arithmetic processing of an inference model to which a "thumbnail image of a RAW image" included in the learning data 501 acquired by the acquisition unit 502 is input, thereby acquiring an exposure compensation value, which is a shooting parameter inferred by the inference model.

[0051] The loss calculation unit 504 calculates the error (loss) between the recommended exposure compensation value included in the training data 501 and the exposure compensation value acquired by the inference unit 503. The loss function used to calculate the error (loss) is the L1 loss, which is commonly used in regression tasks.

[0052] The weight update unit 505 learns the inference model by updating the current ``weights, which are parameters of the inference model'' stored in the storage unit 506 so that the error (loss) calculated by the loss calculation unit 504 becomes smaller.

[0053] In this embodiment, the inference model having the weights stored in the storage unit 506 as parameters at the end of learning is output by the deployment unit 319 to the recording media (such as an SD card) of the digital camera 102. However, the output destination of the inference model is not limited to the digital camera 102, and may be, for example, the memory of a general-purpose computer or a control circuit inside the camera.

[0054] Note that various models can be applied to the above inference model, such as neural networks like CNN (Covolutional Neural Network), ViT (Vision Transformer), and SVM (Support Vector Machine) combined with a feature extractor.

[0055] Next, the photographing operation of the digital camera 102 will be described with reference to the flowchart in Fig. 6. In step S601, the photographing unit 305 sets the photographing mode to aperture priority mode in response to a mode switching instruction input by the user operating the input unit 240. In step S602, the photographing unit 305 starts live view and constantly exposes the image sensor.

[0056] In step S603, the photographing unit 305 sets the aperture value input by the user operating the input unit 240. In step S604, the photographing unit 305 adjusts exposure parameters other than the aperture, specifically the ISO sensitivity and shutter speed, based on the aperture value set in step S603, to achieve proper exposure.

[0057] In step S605, the inference unit 303 inputs intermediate images developed from RAW images sequentially output from the imaging unit 305 by live view from the imaging unit 305 into the inference model expanded in RAM 220 by the reading unit 302.

[0058] In step S606, the inference unit 303 infers shooting parameters by performing calculation processing of the inference model. The determination unit 304 determines parameters to be used for actual shooting based on the shooting parameters inferred by the inference unit 303. The shooting unit 305 then performs exposure correction according to the determined parameters. The shooting unit 305 then performs a shooting operation in response to a user pressing a shutter button (not shown) to acquire a RAW image.

[0059] <Variation 1> In exposure correction in the digital camera 102, exposure parameters are generally determined definitively according to a diagram such as a program diagram. However, there are cases where a user wishes to perform exposure correction using a different combination of exposure parameters.

[0060] Therefore, in this modified example, a GUI is provided that allows the user to virtually move the exposure parameters, and an inference model is generated to infer the exposure parameters from the GT data created there.

[0061] A series of processes for performing image editing on images taken in aperture priority mode and generating learning data will be described with reference to the flowchart in Fig. 7. In Fig. 7, the same process steps as those shown in Fig. 4 are assigned the same step numbers as those steps, and descriptions of those process steps will be omitted.

[0062] The user operates the input unit 240 to change the ISO sensitivity and Tv value and select a combination of ISO sensitivity and Tv value that will achieve the brightness adjustment performed in step S401, and in step S702, the processing unit 308 acquires the selected combination.

[0063] An example of the GUI display according to this embodiment is shown in FIG. 8(a). In the GUI 800, a captured image being edited is displayed in a display area 801. A slider 802 is a slider for adjusting brightness. When the user operates the input unit 240 to move a knob 803 left or right, the brightness of the captured image displayed in the display area 801 is adjusted according to the brightness adjustment value corresponding to the position of the knob 803. In other words, a preview of the captured image that has been developed according to the brightness adjustment value currently set by the user is presented to the user. The brightness adjustment value corresponding to the position of the knob 803 is also displayed in a numeric window 804 with a spin button. Note that the brightness adjustment value can also be adjusted using the spin button.

[0064] Area 805 is an area where an operation unit is provided for changing pseudo shooting parameters (pseudo shooting parameters) in accordance with brightness adjustment. Slider 806 is a slider for adjusting ISO sensitivity. When the user operates input unit 240 to move knob 807 left or right, the ISO sensitivity is updated to the ISO sensitivity corresponding to the position of knob 807. In addition, the ISO sensitivity corresponding to the position of knob 807 is displayed in numeric window 808 with a spin button. Note that the ISO sensitivity can also be adjusted using the spin button.

[0065] The slider 809 is a slider for adjusting the aperture value. When the user operates the input unit 240 to move the knob 810 left or right, the aperture value is updated to the aperture value corresponding to the position of the knob 810. The aperture value corresponding to the position of the knob 810 is also displayed in a numeric window 811 with a spin button. The aperture value can also be adjusted using the spin button.

[0066] The slider 812 is a slider for adjusting the shutter speed. When the user operates the input unit 240 to move the knob 813 left or right, the shutter speed is updated to the shutter speed corresponding to the position of the knob 813. The shutter speed corresponding to the position of the knob 813 is also displayed in a numeric window 814 with a spin button. The shutter speed can also be adjusted using the spin button.

[0067] The ISO sensitivity and shutter speed values ​​change in conjunction with the brightness adjustment value. Note that, since the captured image was taken in aperture priority mode, the controls related to the aperture value (slider 809, knob 810, and numerical window 811 with spin button) are disabled in Figure 8(a).

[0068] Check box 815 is used to determine whether or not to change the brightness adjustment accordingly when the pseudo shooting parameters are changed. If the user operates the input unit 240 to check the check box 815, when the ISO sensitivity is changed, the shutter speed value changes accordingly to offset the change in ISO sensitivity. For example, in FIG. 8(a), when the ISO sensitivity is changed from 1600 to 3200, the shutter speed changes from 1 / 250 to 1 / 500, and the EV value is kept constant. On the other hand, if the user operates the input unit 240 to check the check box 815, the ISO sensitivity and shutter speed can be freely changed, and the brightness adjustment value corresponding to the changed exposure is reflected in the position of the knob 803, the numerical window 804 with spin button, and the display area 801. If the user operates the input unit 240 to select button 816, the shooting parameters are returned to the original shooting parameters (the brightness adjustment value is the origin (0.00)).

[0069] The ISO sensitivity and shutter speed that can be set in GUI 800 in Fig. 8(a) are merely pseudo, and it is of course not possible to change these settings after shooting to create a new image. However, the user can manually record the ISO sensitivity and shutter speed that will achieve the exposure compensation that they want to achieve.

[0070] In step S705, the transmitting unit 310 converts the pseudo shooting parameters into displacement amounts in units of APEX (Additive system of Photographic EXposure) from the original shooting parameters. A specific method for converting pseudo shooting parameters in this modification will be described with reference to FIG. 8(b).

[0071] Figure 8(b) shows the GUI800 screen when the brightness adjustment has been changed to +2.00. To achieve a brightness adjustment of +2.00, the pseudo shooting parameters ISO sensitivity and shutter speed are each adjusted in the direction of one stop of overexposure from Figure 8(a). In this case, the amount of change in ISO sensitivity in APEX units is +1.00 [EV], and the amount of change in shutter speed is also +1.00 [EV], and these values ​​are used as the pseudo shooting parameters (GT data).

[0072] In step S706, the transmission unit 310 stores the pseudo shooting parameters converted in step S705 in the data set generated in step S403, and transmits (uploads) the data set storing the pseudo shooting parameters to the accumulation server 104.

[0073] In this modified example, the configuration of the learning unit 316 is the same as in Figure 5, but the learning data 501 includes a data set and pseudo shooting parameters, and the inference unit 503 uses the pseudo shooting parameters as GT data to learn an inference model, thereby inferring the amount of change in exposure parameters for a RAW image.

[0074] The photographing operation by the digital camera 102 basically follows the process shown in the flowchart of FIG. 6, but since the result of the inference in step S606 is the amount of change in the ISO sensitivity and Tv value, the actual ISO sensitivity and Tv value are changed accordingly.

[0075] <Variation 2> In this modification, for image processing such as adjusting the dynamic range, an HDR shooting recommendation flag indicating whether or not to perform HDR shooting is stored in a data set as GT data, and the HDR shooting recommendation flag is used as GT data to train an inference model for inferring whether or not to perform HDR shooting. The process performed by the system according to this embodiment to generate training data will be described with reference to the flowchart in Fig. 9.

[0076] When a user of the client terminal 103 operates the input unit 240 to input an instruction to launch image editing software, the GUI control unit 306 displays the GUI of the image editing software on the display unit 250. The output unit 309 performs various image processes, including development processing, on the RAW image read from the HDD 230 to the RAM 220 by the image reading unit 307 to generate a captured image. The GUI control unit 306 then displays the captured image developed by the output unit 309 on the GUI. When the user operates the input unit 240 to input an instruction to adjust highlights and shadows, in step S901, the processing unit 308 performs highlight and shadow adjustment on the RAW image in accordance with the instruction, and the output unit 309 generates a captured image by developing the RAW image in accordance with the highlight and shadow adjustment values ​​and outputs the image to the GUI control unit 306. This allows the GUI control unit 306 to display the captured image on the GUI.

[0077] The user then checks the captured image displayed on the GUI for overexposure, blocked up shadows, etc., and determines whether it has been adjusted to the desired gradation expression. If the user determines that the captured image displayed on the GUI has been adjusted to the desired gradation expression and inputs an instruction to that effect by operating input unit 240, the process proceeds to step S903 via step S902. On the other hand, if such an instruction has not been input, the process proceeds to step S901 via step S902.

[0078] In step S903, the transmitting unit 310 generates a data set including "thumbnail images (reduced pre-edited images) of the RAW images read into RAM 220 by the image reading unit 307," "captured images that have been developed according to highlight and shadow adjustment values," and "contents of highlight and shadow adjustment."

[0079] In step S904, the transmission unit 310 transmits (uploads) the data set generated in step S903 to the accumulation server 104. These images may include images that have not been edited as a result.

[0080] In step S905, the conversion unit 312 stores the data set uploaded from the client terminal 103 in the storage unit 311. Then, the conversion unit 312 acquires an HDR photography recommendation flag as a recommended shooting parameter from the highlight / shadow adjustment contents (highlight / shadow adjustment values) included in the data set stored in the storage unit 311. The highlight / shadow adjustment values ​​include a highlight adjustment value and a shadow adjustment value.

[0081] Here, the value of the HDR shooting recommendation flag is set to 1 if the value obtained by subtracting the highlight adjustment amount from the shadow adjustment amount is equal to or greater than a threshold, and set to 0 if the value is less than the threshold. In this way, it is possible to attach a flag recommending HDR shooting to an image that has been edited to compress the dynamic range to a value greater than the threshold.

[0082] Therefore, the learning data according to this embodiment includes a data set and an HDR shooting recommendation flag. The learning unit 316 according to this modification has the functional configuration example shown in Fig. 5, similar to the first modification of the first embodiment, but the operation differs from that of the first modification of the first embodiment in the following points.

[0083] The acquisition unit 502 acquires learning data 501. The inference unit 503 performs calculations on an inference model that receives as input the "thumbnail image of the RAW image" included in the learning data 501 acquired by the acquisition unit 502, thereby acquiring a "recommended HDR shooting value indicating whether HDR shooting should be performed for the input image," which is a shooting parameter inferred by the inference model. The recommended HDR shooting value is a scalar value with a value range of [0, 1].

[0084] The loss calculation unit 504 calculates the error (loss) between the HDR photography recommendation flag included in the training data 501 and the HDR photography recommendation value acquired by the inference unit 503. Because this task involves binary classification, binary cross entropy is used as the loss function used to calculate the error (loss).

[0085] The weight update unit 505 learns the inference model by updating the current ``weights, which are parameters of the inference model'' stored in the storage unit 506 so that the error (loss) calculated by the loss calculation unit 504 becomes smaller.

[0086] Next, the photographing operation of the digital camera 102 will be described with reference to the flowchart in Fig. 10. In the flowchart in Fig. 10, the same processing steps as those shown in Fig. 6 are assigned the same step numbers as those of the corresponding processing steps, and the description of those processing steps will be omitted.

[0087] In step S1001, the inference unit 303 acquires a recommended value for HDR photography by performing calculation processing on the inference model. Then, in step S1002, the inference unit 303 determines whether the recommended value for HDR photography acquired in step S1001 is 0.8 or greater.

[0088] If the result of this determination is that the recommended value for HDR shooting is 0.8 or greater, the process proceeds to step S1004, and if the recommended value for HDR shooting is less than 0.8, the process proceeds to step S1003. In step S1003, the imaging unit 305 performs SDR shooting, i.e., normal shooting, and in step S1004, the imaging unit 305 performs HDR shooting.

[0089] <Variation 3> In this modification, a photography recipe is output using a trained inference model. The photography recipe is information related to photography for capturing a certain image, and is information for capturing a beautiful and attractive image.

[0090] An example of the display of a recipe screen showing a shooting recipe for shooting a sports scene of a person is shown in Fig. 11. The recipe screen 1100 shown in Fig. 11 is displayed on the display unit 250 of the client terminal 103. However, the recipe screen 1100 may also be displayed on another device.

[0091] A sample image of a sports scene is displayed in the display area 1101. This sample image may be a captured image of a sports scene captured by the imaging unit 305, or may be a captured image of a sports scene stored in advance in the HDD 230. By viewing the sample image displayed in the display area 1101, the user can specifically imagine (imagine) the image that can be obtained by capturing images according to the imaging recipe.

[0092] The shooting setting information 1102 includes information about the digital camera 102, such as camera information and lens information, the shooting mode, and the settings of the digital camera 102 used when shooting (focal length, exposure compensation, ISO sensitivity, metering mode, white balance, shutter speed, and aperture value).

[0093] Photography tips 1103 is information explaining tips for taking better images (equipment and props to be used when taking a photo, composition showing the position of the subject, focus and blur, and post-processing methods).

[0094] The process performed by the client terminal 103 to display the above recipe screen on the display unit 250 will be described with reference to the flowchart in Fig. 12. After the image editing software is started, in step S1201, the CPU 200 selects an image for which a shooting recipe is to be generated from the processed RAW images. The selection may be performed by the user in response to an operation input via the input unit 240, or may be performed by the CPU 200 in accordance with a predetermined standard.

[0095] In step S1202, the CPU 200 creates a new shooting recipe for the RAW image selected in step S1201. At this point, the shooting setting information of the shooting recipe (shooting setting information 1102 in the example of FIG. 11) contains the same information as the Exif information. Also, each item in the shooting tips 1103 remains blank.

[0096] In step S1203, the CPU 200 inputs the RAW image selected in step S1201 into the inference model held by the digital camera 102. In step S1204, the CPU 200 causes the digital camera 102 to perform calculations on the inference model to which the RAW image selected in step S1201 has been input, thereby acquiring a recommended exposure compensation value and a recommended value for HDR photography.

[0097] In step S1205, the CPU 200 changes the exposure parameters (that is, the ISO sensitivity and the Tv value) by the recommended exposure compensation value acquired in step S1204, and stores the changed exposure parameters in the RAM 220.

[0098] In step S1206, CPU 200 determines whether the recommended value for HDR capture acquired in step S1204 is equal to or greater than 0.8. If the result of this determination is that the recommended value for HDR capture is equal to or greater than 0.8, processing proceeds to step S1208, and if the recommended value for HDR capture is less than 0.8, processing proceeds to step S1207.

[0099] In step S1207, the CPU 200 sets the special photography in the photography tips of the photography recipe as "none" and saves the setting in the RAM 220. On the other hand, in step S1208, the CPU 200 sets the special photography in the RAM 220 as "HDR photography."

[0100] In step S1209, the CPU 200 updates the corresponding items of the shooting recipe with the information stored in the RAM 220 in step S1205, step S1207, step S1208, etc. In step S1210, the CPU 200 causes the display unit 250 to display the shooting recipe after updating in step S1209.

[0101] In step S1211, the CPU 200 accepts additional comments from the user regarding the shooting recipe, such as a description of the composition and focus / bokeh, which are not definitively determined from the camera settings and shooting parameter inference results.

[0102] In this way, by capturing an image by inferring the shooting parameters that should be set from the image editing parameters, it is possible to obtain an image that is close to an image that has been edited to the user's liking at the time of capture. This not only reduces the burden on post-processing, but also narrows the adjustment range related to image editing, thereby reducing acquired noise caused by image editing.

[0103] Furthermore, even for images captured with undesired shooting parameters, the shooting parameters that should have been set can be inferred and reflected in the shooting recipe, so that a shooting recipe that is in line with the user's intentions can be output.

[0104] Although the inference model of this embodiment is configured to infer exposure parameters and a flag for HDR shooting, other shooting parameters can also be inferred as long as they can be set in the same way during image editing and shooting. Specifically, noise reduction, sharpness intensity, saturation, etc. can also be inferred in the same way.

[0105] [Second embodiment] In this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed that the present embodiment is the same as the first embodiment. An example of the functional configuration of a system according to this embodiment will be described using the block diagram of FIG. 13. In FIG. 13, functional units similar to those shown in FIG. 3 are assigned the same reference numbers as those of the functional units, and descriptions of those functional units will be omitted. In the first embodiment, the conversion unit 312 was included in the storage server 104, but in this embodiment, the conversion unit 312 is included in the client terminal 103. The processing performed by the system according to this embodiment to generate training data will be described with reference to the flowchart of FIG. 14.

[0106] In step S1401, the GUI control unit 306 acquires a range (search range) for searching for data to be learned, which is input by the user operating the input unit 240. The search range is, for example, an image search condition.

[0107] In step S1402, the image reading unit 307 acquires RAW images within the search range acquired in step S1401, and the conversion unit 312 generates recommended exposure compensation values ​​in a batch from the acquired RAW images.

[0108] In step S1403, the transmitting unit 310 samples RAW images so that the recommended exposure compensation value is uniformly distributed. For example, the recommended exposure compensation values ​​are divided into seven groups: less than -2.5, -2.5 or greater but less than -1.5, -1.5 or greater but less than -0.5, -0.5 or greater but less than 0.5, 0.5 or greater but less than 1.5, 1.5 or greater but less than 2.5, and 2.5 or greater. The transmitting unit 310 samples RAW images so that the number of RAW images belonging to each group is equal. Furthermore, unedited RAW images within the search range are sampled with a recommended exposure compensation value of 0.0.

[0109] In step S1404, the transmitting unit 310 generates a data set including "thumbnail images (reduced unedited images) of the RAW images sampled by the image reading unit 307," "contents of brightness adjustment," and "recommended exposure compensation value." The transmitting unit 310 then transmits (uploads) the generated data set to the accumulation server 104.

[0110] In this way, a dataset can be generated from a large amount of edited or unedited RAW images present on the client terminal 103, making it easier to improve the accuracy of the inference model.

[0111] In this embodiment, RAW images are sampled so that the objective variable to be inferred has a uniform distribution. However, if there are not enough RAW images available, it is possible to simply sample from edited RAW images only, or to sample according to the time required for image processing. For example, sampling may be performed from data with a timestamp newer than the date and time of shooting. Alternatively, for example, the time spent on processing and editing an image may be accumulated, and a set of shooting information and recommended shooting parameters sampled from data with an image timestamp newer than the date and time of shooting may be registered.

[0112] In each of the above embodiments, the client terminal 103, the storage server 104, and the learning server 105 are each described as separate devices, but this is not limited to this, and two or more of these devices may be integrated into a single device.

[0113] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in the above embodiments and variant examples are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0114] Furthermore, some or all of the embodiments and modifications described above may be used in appropriate combination, and some or all of the embodiments and modifications described above may be used selectively.

[0115] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0116] The invention of this specification includes the following information processing device, information processing method, and computer program. (Item 1) an acquisition means for acquiring recommended photographing parameters based on the content of image processing performed on photographing information obtained by photographing; a registration means for registering the imaging information and the recommended imaging parameters as learning data in association with each other; An information processing device comprising: (Item 2) 2. The information processing device according to item 1, wherein the shooting information includes a thumbnail image of a RAW image and Exif information. (Item 3) 3. The information processing device according to item 2, wherein the thumbnail image of the RAW image is a reduced image developed at the time of shooting. (Item 4) 4. The information processing device according to item 2 or 3, wherein the acquisition means acquires a recommended exposure compensation value from a brightness adjustment value in brightness adjustment performed on the RAW image and the Exif information. (Item 5) 5. The information processing device according to any one of items 1 to 4, wherein the recommended shooting parameters include one or more of an ISO sensitivity correction value, an aperture value correction value, and a shutter speed correction value. (Item 6) The acquisition means further acquires pseudo-shooting parameters including ISO sensitivity, aperture value, and shutter speed, which are set as pseudo-shooting parameters according to brightness adjustment; The pseudo-photography parameters change in conjunction with the brightness adjustment value in the brightness adjustment. 6. The information processing device according to any one of items 1 to 5, (Item 7) The information processing device described in any one of items 1 to 6, characterized in that the acquisition means acquires a flag indicating whether or not to perform HDR shooting, which has a value of 1 if the value obtained by subtracting the highlight adjustment value from the shadow adjustment value is equal to or greater than a threshold, and a value of 0 if the value is less than the threshold. (Item 8) moreover, an inference means for inferring photographing parameters by performing calculation processing of an inference model based on the photographing information; a learning means for learning the inference model based on an error between the imaging parameters inferred by the inference means and the recommended imaging parameters; 8. The information processing device according to any one of items 1 to 7, comprising: (Item 9) moreover, Item 9. The information processing device according to item 8, further comprising a deployment means for deploying the inference model learned by the learning means to an imaging device. (Item 10) moreover, An information processing device described in any one of items 1 to 9, characterized in that it is equipped with a means for inferring shooting parameters by performing calculations on an inference model to which a sample image is input, generating a screen showing a shooting recipe based on the inferred shooting parameters, and outputting the screen. (Item 11) The information processing device described in any one of items 1 to 10, characterized in that the acquisition means acquires recommended shooting parameters based on the content of image processing performed on the shooting information in the search range, and associates the sampled shooting information with the recommended shooting parameters and registers them. (Item 12) Item 12. The information processing device according to item 11, wherein the sampled photographing information is photographing information sampled based on recommended photographing parameters. (Item 13) Item 12. The information processing device according to item 11, wherein the sampled photographing information is photographing information sampled based on the time of image processing. (Item 14) Item 12. The information processing device according to item 11, wherein the sampled photographing information is photographing information sampled based on a timestamp of the photographing information. (Item 15) An information processing method performed by an information processing device, an acquisition step in which an acquisition means of the information processing device acquires recommended shooting parameters based on the content of image processing performed on shooting information obtained by shooting; a registration step in which a registration means of the information processing device associates the shooting information with the recommended shooting parameters and registers them as learning data; An information processing method comprising: (Item 16) A computer program for causing a computer to function as each means of the information processing device according to any one of items 1 to 14.

[0117] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0118] 102: Digital camera 103: Client terminal 104: Storage server 105: Learning server 301: Memory unit 302: Reading unit 303: Inference unit 304: Determination unit 305: Photography unit 306: GUI control unit 307: Image reading unit 308: Processing unit 309: Output unit 310: Transmission unit 311: Memory unit 312: Conversion unit 313: Provision unit 314: Memory unit 315: Generation unit 316: Learning unit 317: Generation unit 318: Test unit 319: Deployment unit

Claims

1. an acquisition means for acquiring recommended photographing parameters based on the content of image processing performed on photographing information obtained by photographing; a registration means for registering the imaging information and the recommended imaging parameters as learning data in association with each other; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the photographing information includes a thumbnail image of a RAW image and Exif information.

3. 3. The information processing apparatus according to claim 2, wherein the thumbnail image of the RAW image is a reduced image developed at the time of shooting.

4. 3. The information processing apparatus according to claim 2, wherein the acquisition unit acquires the recommended exposure compensation value from a brightness adjustment value in brightness adjustment performed on the RAW image and the Exif information.

5. 2. The information processing apparatus according to claim 1, wherein the recommended shooting parameters include one or more of an ISO sensitivity correction value, an aperture value correction value, and a shutter speed correction value.

6. The acquisition means further acquires pseudo-photography parameters including ISO sensitivity, aperture value, and shutter speed, which are set as pseudo-photography parameters according to brightness adjustment; The pseudo-photography parameters change in conjunction with the brightness adjustment value in the brightness adjustment.

2. The information processing apparatus according to claim 1, wherein:

7. 2. The information processing device according to claim 1, wherein the acquisition means acquires a flag indicating whether or not to perform HDR shooting, the flag having a value of 1 if the value obtained by subtracting the highlight adjustment value from the shadow adjustment value is equal to or greater than a threshold value, and a value of 0 if the value is less than the threshold value.

8. moreover, an inference means for inferring photographing parameters by performing calculation processing of an inference model based on the photographing information; a learning means for learning the inference model based on an error between the imaging parameters inferred by the inference means and the recommended imaging parameters; 2. The information processing apparatus according to claim 1, further comprising:

9. moreover, The information processing device according to claim 8 , further comprising a deployment unit that deploys the inference model learned by the learning unit to an imaging device.

10. moreover, The information processing device described in claim 1, characterized in that it is equipped with a means for inferring shooting parameters by performing calculations on an inference model to which a sample image is input, generating a screen showing a shooting recipe based on the inferred shooting parameters, and outputting the screen.

11. The information processing device according to claim 1, characterized in that the acquisition means acquires recommended shooting parameters based on the content of image processing performed on the shooting information in the search range, and associates the sampled shooting information with the recommended shooting parameters and registers them.

12. 12. The information processing apparatus according to claim 11, wherein the sampled photographing information is photographing information sampled based on recommended photographing parameters.

13. 12. The information processing apparatus according to claim 11, wherein the sampled photographing information is photographing information sampled based on a time period for image processing.

14. 12. The information processing apparatus according to claim 11, wherein the sampled photographic information is photographic information sampled based on a time stamp of the photographic information.

15. An information processing method performed by an information processing device, an acquisition step in which an acquisition means of the information processing device acquires recommended shooting parameters based on the content of image processing performed on shooting information obtained by shooting; a registration step in which a registration means of the information processing device associates the shooting information with the recommended shooting parameters and registers them as learning data; An information processing method comprising:

16. A computer program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 14.

Images