Information processing apparatus, information processing method, computer program, and system

Abstract

Provided is an information processing apparatus that provides a learning model based on a user's past shooting and image editing tendencies. The information processing apparatus is provided with: a collection unit that collects metadata at the time of shooting, shooting setting values, and editing setting values used at the time of image editing of a captured image in association with the captured image from an image pickup apparatus, and accumulates such metadata, shooting setting values, and editing setting values in a storage apparatus; a generation unit that causes a model to learn a correlation between the metadata, shooting setting values, and editing setting values accumulated in the storage apparatus, and generates a learning model for estimating shooting setting values and editing setting values from metadata; and a transmission unit that transmits the generated learning model to the image pickup apparatus.

Description

Technical Field

[0001] The technology disclosed in this specification (hereinafter referred to as "this disclosure") relates to information processing apparatus and information processing method, computer program, image capture apparatus, and system for performing processing related to image capture by a camera device. Background Technology

[0002] So-called "digital data," which converts externally received light into digital data and stores it using an image sensor made of semiconductors, is widely used. When performing image capture using a digital camera device, image capture settings such as aperture and shutter speed are typically set via a user interface on the main body of the camera device. Furthermore, brightness adjustments, color temperature adjustments, color corrections, contrast adjustments, etc., are typically performed on the captured images using editing applications. Recently, many products have emerged that have an automatic image capture function that automatically controls the image capture settings of the camera device based on detection values ​​from sensors contained within the digital camera device. For example, several techniques related to focus control for automatic focusing on the subject have been proposed (see, for example, Patent Document 1).

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: WO2023 / 286301

[0006] Patent Document 2: JP 2020-39123 A, Paragraphs 0020 to 0030 Summary of the Invention

[0007] Technical issues

[0008] The purpose of this disclosure is to provide an information processing apparatus and method for performing processing related to the setting of a camera device, a computer program, an image capture device, a system, a camera device system, and a camera device equipment for facilitating setting operations.

[0009] Solution to the problem

[0010] This disclosure is made in view of the above-mentioned issues, and the first aspect of this disclosure is:

[0011] An information processing device, comprising:

[0012] The collection unit is configured to collect metadata and image capture settings associated with the captured image of the camera device, as well as editing information during image editing of the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device.

[0013] A generation unit is configured to enable the model to learn the correlations between metadata, image capture settings, and editing information accumulated in a storage device, and to generate a learned model for estimating image capture settings and editing information based on the metadata; and

[0014] The transmission unit is configured to transmit the generated learning model to the camera device.

[0015] Metadata includes at least one of the following: first information obtained by analyzing the features of the subject scene, second information obtained by converting the state at the time of image capture into data, or third information obtained by reducing the size of the image of the subject scene.

[0016] The generation unit, based on a specific user's image capture and editing operations, enables the model to learn the correlation between metadata, image capture settings, and editing information, and generates a learning model that reflects a specific user's past image capture and editing tendencies.

[0017] The second aspect of this disclosure is an information processing method, including:

[0018] The image capture metadata and image capture settings are collected in association with the captured images of the camera device, as well as the editing information during image editing of the captured images, and the metadata, image capture settings, and editing information are accumulated in the storage device;

[0019] The model learns the correlations between metadata accumulated in the storage device, image capture settings, and editing information, and generates a learned model for estimating image capture settings and editing information based on the metadata; and

[0020] The generated learning model is transmitted to the camera device.

[0021] A third aspect of this disclosure is a computer program written in a computer-readable format that enables a computer to function as:

[0022] The collection unit is configured to collect metadata and image capture settings associated with the captured image of the camera device, as well as editing information during image editing of the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device.

[0023] A generation unit is configured to enable the model to learn the correlations between metadata, image capture settings, and editing information accumulated in a storage device, and to generate a learned model for estimating image capture settings and editing information based on the metadata; and

[0024] The transmission unit is configured to transmit the generated learning model to the camera device.

[0025] The computer program according to the third aspect of this disclosure is defined as a computer program written in a computer-readable form to perform predetermined processing operations on a computer. The computer program can be provided to a computer capable of executing various program codes using a storage medium or communication medium provided in a computer-readable format, such as an optical disc, magnetic disk, or semiconductor memory, or a communication medium such as a network. By installing the computer program according to the third aspect of this disclosure on a computer via any medium, synergistic effects can be exhibited on the computer, and the same and / or similar effects and functions as those of the information processing apparatus according to the first aspect of this disclosure can be obtained.

[0026] The fourth aspect of this disclosure is:

[0027] An image capture device includes an image capture functional unit. The image capture device includes:

[0028] The generation unit is configured to analyze the subject scene and generate metadata; and

[0029] The estimation unit is configured to estimate the setting value based on the metadata using a learned model that has already learned the correlation between the metadata and the setting value.

[0030] The image capture function unit is controlled by using setting values ​​to perform image capture operations.

[0031] The estimation unit estimates image capture settings and editing information based on metadata using a learning model. The image capture apparatus according to the fourth aspect also includes a conversion unit configured to convert editing information into parameter values ​​that can be processed in the camera device, and to control the image capture function unit to perform image capture operations using the image capture settings and parameter values.

[0032] The image capture device according to the fourth aspect also includes a monitoring function unit configured to generate a monitoring image of the image capture function unit, wherein the monitoring function unit is controlled by using image capture setting values ​​and parameter values ​​so that the monitoring screen is displayed.

[0033] The fifth aspect of this disclosure is a system comprising:

[0034] A camera device, an editing device configured to edit images captured by the camera device, and a model providing device configured to provide a learning model to the camera device, wherein,

[0035] The camera device outputs metadata related to image capture and image capture settings associated with the captured image.

[0036] The editing device outputs editing information during image editing of the captured image, and

[0037] The model providing apparatus includes: a collection unit configured to collect metadata, image capture settings, and editing information and accumulate the metadata, image capture settings, and editing information into a storage device; a generation unit configured to enable the model to learn the correlation between the metadata, image capture settings, and editing information accumulated in the storage device and generate a learning model for estimating the image capture settings and editing information based on the metadata; and a transmission unit configured to transmit the generated learning model to a camera device.

[0038] However, the "system" mentioned here refers to a logically aggregated collection of multiple devices (or functional modules used to implement a specific function), and it is not important whether each device or functional module is in a single housing. In other words, a single device comprising multiple components or functional modules and an aggregate of multiple devices each correspond to a "system".

[0039] The sixth aspect of this disclosure is a camera device system, which includes:

[0040] The camera device unit includes a first learning model unit; and

[0041] A generative AI model unit is configured to generate a second image based on a first image captured by a camera device unit, wherein...

[0042] The first learning model is modified by using a learning model that has already been learned based on the first and second images.

[0043] The seventh aspect of this disclosure is a camera device including a machine learning model, the camera device comprising:

[0044] Control unit; and

[0045] The first learning model, in which...

[0046] The control unit modifies the first learning model based on the second learning model, which has been learned by comparing the first image captured by the camera device with the second image generated by the generative AI model unit based on the first image. Attached Figure Description

[0047] [ Figure 1 ] Figure 1This is a diagram illustrating the configuration of system 10 to which the present disclosure is applied.

[0048] [ Figure 2 ] Figure 2 It is a graph showing the state of metadata, image capture settings, and editing settings associated with and accumulated for each captured image for each user.

[0049] [ Figure 3 ] Figure 3 It is a graph showing the state of the learning model generated by using a large amount of data stored for each user.

[0050] [ Figure 4 ] Figure 4 This is a diagram illustrating the operation of system 10 up to the generation of the learning model by model providing device 13.

[0051] [ Figure 5 ] Figure 5 This is a table showing examples of image capture setting values ​​set in the camera device 11 when capturing an image.

[0052] [ Figure 6 ] Figure 6 This is a table showing examples of "information obtained by analyzing the features of the subject scene" included in the metadata.

[0053] [ Figure 7 ] Figure 7 This is a table that shows an example of "information obtained by converting the state at the time of image capture into data" included in the metadata.

[0054] [ Figure 8 ] Figure 8 This is a table showing examples of editing settings values ​​set in an image editing application.

[0055] [ Figure 9 ] Figure 9 This is a diagram showing the operation of system 10 after camera device 11 acquires the learning model.

[0056] [ Figure 10 ] Figure 10 This is a diagram illustrating the operation of a system 10 for displaying images for monitoring using a learning model, for a camera device 11.

[0057] [ Figure 11 ] Figure 11 This is a diagram illustrating a variation of system operation.

[0058] [ Figure 12 ] Figure 12 This is a diagram showing the configuration of a camera device 11 to which the present disclosure can be applied.

[0059] [ Figure 13 ] Figure 13 This is a diagram showing the internal configuration of the image processing unit 1203.

[0060] [ Figure 14 ] Figure 14 It shows that it has been applied. Figure 12 A diagram illustrating the specific operation of the system 10 of the camera device 11.

[0061] [ Figure 15 ] Figure 15 This is a flowchart illustrating the processing procedure of the camera device 11 performing an image capture operation.

[0062] [ Figure 16 ] Figure 16 This is a flowchart illustrating the process by which the editing device 12 performs editing operations on the captured image output from the camera device 11.

[0063] [ Figure 17 ] Figure 17 This is a flowchart illustrating the process by which the model providing device 13 generates the learning model.

[0064] [ Figure 18 ] Figure 18 This is a diagram illustrating an example of system operation when monitoring a scene of a subject.

[0065] [ Figure 19 ] Figure 19 This is a flowchart illustrating the processing procedure for monitoring the scene of the subject using the camera device 11.

[0066] [ Figure 20 ] Figure 20 This is a diagram illustrating a first application example of system 10.

[0067] [ Figure 21 ] Figure 21 This is a diagram illustrating a second application example of system 10.

[0068] [ Figure 22 ] Figure 22 This is a diagram illustrating a third application example of system 10.

[0069] [ Figure 23 ] Figure 23 This is a diagram illustrating a third application example of system 10.

[0070] [ Figure 24 ] Figure 24 This is a diagram illustrating an example of the hardware configuration of an information processing device. Detailed Implementation

[0071] In the following description, embodiments of the present disclosure will be described with reference to the accompanying drawings in the following order.

[0072] A. Overview

[0073] A-1. System Configuration

[0074] A-2. System Operation Example

[0075] A-3. Another example of system operation

[0076] A-4. Variation

[0077] A-5. Effects

[0078] B. Specific Examples

[0079] B-1. Configuration of camera equipment

[0080] B-2. System Operation Example

[0081] B-3. ​​Another example of system operation

[0082] C. Application Examples

[0083] C-1. First Application Example

[0084] C-2. Second Application Example

[0085] C-3. Third Application Example

[0086] D. Configuration of information processing devices

[0087] A. Overview

[0088] Photographers need to perform various settings on their camera equipment to capture images that reflect their intentions and preferences. Some photographers may find this setup process complex and difficult. Camera equipment offering automatic image capture functionality is configured to automatically set image capture settings, which are calculated based on the results of subject scene state analysis using various sensors, including an image sensor. However, the images generated using automatic image capture are determined based on logical conditions. That is, the image capture settings calculated by automatic image capture from the same subject scene are consistent, therefore the images generated using automatic image capture differ from each photographer's various intentions and preferences.

[0089] The types of subject scenes are essentially infinite, requiring complex algorithms involving numerous conditional branches to automatically calculate image capture settings for the camera device based on the analysis results of the subject scene's state. This introduces the risk of compromising operational quality due to errors in the program, among other things.

[0090] Recently, image processing techniques have been developed for editing images after they have been captured. Photographers can use editing applications to perform image processing on captured images to reflect their intentions and preferences. However, as the number of captured images increases with the evolution of camera equipment, editing work requires a significant amount of time.

[0091] If the image could be generated during image capture while considering image processing through editing, the aforementioned editing work would be unnecessary. However, the setup and operation of a camera device that considers image processing through editing is complex and requires specialized knowledge. Even if a device exists to perform image processing to reflect the photographer's intentions and preferences, capturing and editing images are different from each other, and it is impossible to immediately confirm what kind of image will ultimately be obtained during image capture. Furthermore, if an inexperienced user attempts to imitate an image captured and edited by a skilled user of the camera device, the inexperienced user cannot easily replicate the process.

[0092] By providing users with editing settings for specific pre-edited images based on past trends derived from the correlation between a large number of pre-edited images and the corresponding editing settings for each image, the editing process can be facilitated. However, the setup operation of camera devices, considering the image processing performed through editing, remains complex and difficult. Furthermore, a large number of images prior to editing, which are of significant size, are required to calculate the aforementioned correlations, leading to high storage costs for accumulating data and high computational costs for calculating correlations by processing image data with a large amount of information.

[0093] Therefore, this disclosure proposes a technique for generating images that reflect the diverse intentions and preferences of various photographers based on their past image capture and image editing trends. Specifically, in this disclosure, an artificial intelligence (AI) model (hereinafter, the "AI model" is also referred to as the "learning model") is learned to estimate appropriate image capture and editing settings based on metadata, the correlation between image capture settings of the camera device, and editing settings used for image editing in past image captures. Upon subsequent instructions to capture an image, image capture processing is performed by setting the estimated image capture and editing settings based on the metadata using the learning model. Thus, according to this disclosure, by using the learning model, setting operations of the camera device that take into account the behavior of image processing performed through editing are executed, and the camera device can generate a captured image identical to the captured image after editing. Note that in this disclosure, a "dictionary" deriving appropriate image capture and editing settings from metadata can be used instead of the "learning model" that has learned the correlation between metadata, image capture settings used for image editing, and editing settings. When the term "learning model" is simply referred to in this specification, it includes both the model being learned and the model that has already been learned.

[0094] Metadata includes, for example, features of the subject scene and the conditions at the time of image capture, and can be obtained from image analysis results, hardware configuration information of the camera device, and detection results from sensors mounted on the camera device. Metadata may include a scaled-down image obtained by reducing the size of the image captured by the camera device. In this disclosure, by using metadata in which information is compressed, the model is learned, thus reducing storage and computational costs compared to learning the model by using captured images of large data sizes as is.

[0095] A-1. System Configuration

[0096] Figure 1 The configuration of a system 10 applying the present disclosure is illustrated schematically. The system 10 in the figure includes a camera device 11 through which a user performs image capture, an editing device 12 through which a user performs editing work on the captured images of the camera device 11, and a model providing device 13 for providing a learning model in which past image capture and image editing tendencies of an individual user are learned. Furthermore, the user of the camera device 11 and the user of the editing device 12 are assumed to be substantially the same, but may also be different from each other.

[0097] Whenever a user performs an image capture, the camera device 11 outputs the captured image along with the image capture settings and metadata set by the user at the time of image capture. The captured image output from the camera device 11 is, for example, a RAW image before editing, but it can also be an image in other formats. The metadata includes, for example, features of the subject scene, the conditions at the time of image capture, etc., and the specific types of metadata will be described in detail below. Furthermore, the specific types of parameters for the image capture settings will also be described in detail below.

[0098] Upon receiving a captured image from the camera device 11, the editing device 12 performs image processing such as image processing on the captured image, for example, by using an image editing application and according to the user's editing operations. In addition to outputting the edited image, the editing device 12 also outputs editing settings values ​​set by the user during image editing. The edited image output from the editing device 12 is in, for example, YUV format, but may also be in other formats such as Joint Photographic Experts Group (JPEG). Furthermore, the specific types of parameters for the editing settings values ​​will be described in detail below. The editing device 12 may also output some metadata.

[0099] The model providing device 13 collects image capture settings and metadata for each captured image from the camera device 11, collects editing settings for the captured images from the editing device 12, and accumulates metadata, image capture settings, and editing settings associated with each user and each captured image, such as... Figure 2 As shown. Then, the model providing device 13, by using a large amount of data stored for each user, enables the model to learn the correlation between metadata, image capture settings, and editing settings, and generates a learned model for estimating image capture settings and editing settings based on the metadata, such as... Figure 3 As shown. The learning model generated by the model providing device 13 is a personalized learning model used to estimate appropriate image capture settings and editing settings based on the target user's past image capture and image editing tendencies. The learning model generated by the model providing device 13 is provided to the camera device 11 of the target user. For example, the model providing device 13 can relearn the model each time new accumulated data is collected, and provide the relearned model to the camera device 11.

[0100] After acquiring the learning model, the camera device 11 can input metadata into the learning model each time an image capture is performed, and can control the image capture operation by using the image capture settings and editing settings output from the learning model. That is, even if the user does not input the image capture settings and editing settings one after another, the camera device 11 can generate a captured image with high reproducibility that is equivalent to the captured image edited by the editing device 12.

[0101] Even if a user does not have specialized knowledge of the setup operation or image editing of the camera device 11, an image equivalent to one captured and edited by a skilled user can be mimicked by introducing a learning model generated using accumulated data from skilled users into the camera device 11.

[0102] The camera device 11 can be a common digital camera device. The editing device 12 is, for example, an information processing device such as a personal computer (PC), or an information terminal such as a smartphone or tablet computer with image editing applications installed. The camera device 11 and the editing device 12 can be... Figure 1 The device shown can be a physically independent device, or it can be a physically integrated device (e.g., an information terminal equipped with a camera).

[0103] The model providing device 13 is, for example, an information processing device such as a PC equipped with a relevant learning application. The relevant learning application mentioned here is an application that has the function of enabling the model to learn the correlation between information related to image capture conditions, image capture settings, and editing settings to generate a learning model.

[0104] The model provides device 13, for example, a server installed in the cloud. Figure 1 For simplicity, only one camera device 11, one editing device 12, and one model providing device 13 are shown. However, in practice, the model providing device 13 can collect data from multiple camera devices 11 and editing devices 12. The model providing device 13 can then perform correlation learning based on a large amount of data accumulated for each user to generate a learning model for each user, and can provide the learning model for each user to each of the multiple camera devices 11.

[0105] As a variation, the model providing device 13 can be an information processing device integrated with the editing device 12, or it can incorporate all of the camera device 11, the editing device 12, and the model providing device 13 into a single device. The model providing device 13, integrated with the camera device 11 and the editing device 12, generates a learning model for a specific user.

[0106] A-2. System Operation Example

[0107] Figure 4 The operation of system 10 up to the generation of the learning model by model providing device 13 is shown.

[0108] The camera device 11 includes an image capture function unit 401. The image capture function unit 401 includes an image capture execution unit having a lens and an image sensor, and a signal processing unit for generating a YUV image (or an RGB image or a JPEG image) by performing signal processing such as development processing on the image signal (RAW image) output from the image sensor. For simplicity, the figures are shown below. Figure 4 The image capture execution unit and signal processing unit are not shown.

[0109] Each time an image is captured, the user accesses the user interface of the camera device 11 (in...). Figure 4 (Not shown in the image) to set image capture settings. Image capture function unit 401 performs image capture based on the set image capture settings. Figure 5 An example of image capture settings set in the camera device 11 during image capture is shown. Aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type are included in the image capture settings, but it is not necessary to include all of these, and conversely, other image capture parameters can be included in the image capture settings. The image capture function unit 401 can perform image capture of still images or videos.

[0110] The image capture function unit 401 generates metadata when performing image capture based on the set image capture settings. The metadata includes, for example, features of the subject scene, the conditions during image capture, and can be obtained from image analysis results, hardware configuration information of the camera device, and detection results from sensors mounted on the camera device. The metadata may include a scaled-down image obtained by reducing the size of the captured image. The image capture function unit generates metadata each time it receives an instruction from the user to capture an image.

[0111] Figure 6 An example of "information obtained by analyzing the features of the subject scene" included in the metadata is shown. "Information obtained by analyzing the features of the subject scene" includes "frequency values ​​obtained by classifying the brightness values ​​of each pixel in the image into five levels," "the ratio of the values ​​of the brightest and darkest pixels," "the level of the brightness of the pixel at the focus location in the five-level classification," "frequency values ​​obtained by classifying the color of each pixel in the image into five colors," "the level of the color of the pixel at the focus location in the five-level classification," "whether there is a moving object in the subject scene," "distance information of the subject," etc., but it is not necessary to include all of the above, and conversely, other information may also be included. While in Figure 6 The description is omitted, but the metadata may include a scaled-down image obtained by capturing an image through a scaled-down camera device.

[0112] also, Figure 7 An example of "information obtained by converting the conditions at the time of image capture into data" included in the metadata is shown. "Information obtained by converting the conditions at the time of image capture into data" includes "name of the image capture device," "name of the lens used," "image capture date and time," "ambient light intensity value," "tilt of the camera device (or camera device shake)," etc., but does not need to include all of the above, and conversely, may include other information. The camera device 11 includes a metadata acquisition unit (not shown) for acquiring metadata and accumulating it in a file each time the image capture function unit performs an image capture operation.

[0113] When the camera device 11 converts the captured image into a file, the metadata and image capture settings are also stored in the file. The metadata and image capture settings can be stored in the same file as the captured image, or in a different file, as long as they are associated with the captured image. Figure 4 In the figure, reference numeral 411 indicates the file group of captured images output from the camera device 11, and reference numerals 412 and 413 respectively indicate the file groups of metadata and image capture settings associated with the corresponding captured images.

[0114] Refer again Figure 4 The editing device 12 receives the captured image 411 from the camera device 11 and performs editing processes, such as image processing, on the captured image according to the user's editing operations using the image editing application 402. However, the image editing application 402 can operate in the camera device 11 instead of in the editing device 12. The image editing application 402 processes the captured image input from the camera device 11 based on editing settings values ​​set by the user. Figure 8 An example of editing settings values ​​set to image editing application 402 is shown. Editing settings values ​​include brightness, color temperature adjustment, color correction, contrast, white level, black level, and chroma, but not necessarily all of the above, and conversely, editing settings values ​​may include other editing parameters.

[0115] When an image edited by image editing application 402 is converted to a file and output, the editing settings are also stored in the file in association with the original captured image. Figure 4 In the figure, reference numeral 414 indicates the file group of the edited image output from the image editing application 402, and reference numeral 415 indicates the file group of editing settings associated with the original captured image. Note that the editing device 12 may store a portion of the metadata used for image editing in the metadata file.

[0116] The model providing device 13 collects file groups 412, 413, and 415, which are stored in association with the original captured image, respectively, and accumulates the collected file groups in association with the captured image into the storage device 403 for each user (see [link to storage device 13]). Figure 2 The model providing device 13 can periodically perform the operation of collecting file groups 412, 413 and 415, or it can perform the operation in response to instructions from the user.

[0117] In the model providing device 13, the model learning unit 404, for a specific user, enables the model to learn the correlation between metadata associated with each captured image, image capture settings, and editing settings, generating a learning model 405 for estimating image capture settings and editing settings based on the metadata, and converting the learning model 405 into a file for storage. The learning model 405 generated by the model providing device 13 is a model for a specific user, used to estimate suitable image capture settings and editing settings for each subject scene based on the specific user's past image capture and image editing tendencies. The model providing device 13 transmits the learning model 405 to the camera device 11 for the specific user. The model providing device 13 may transmit the file of the learning model 405 in response to a download request from the camera device 11, or it may push the file of the learning model 405 to deliver it.

[0118] The model providing device 13 performs the generation process of the learning model by using metadata with compressed information, and thus, it can reduce storage and computing costs compared to learning the model by using captured images with large data sizes as is.

[0119] Figure 9 The operation of system 10 after the camera device 11 acquires the learning model is shown. However, in Figure 9 In the middle, assuming according to Figure 4 The operation of the system 10 shown involves a learning model generated for a specific user being transmitted to the camera device 11, and this learning model is available on the camera device 11 side.

[0120] When the imaging device 11 receives an instruction from the user to capture an image, the image capture function unit 401 analyzes the subject scene to generate metadata as indicated by reference numeral 417. The metadata 417 is input to the learning model 405 provided by the model providing device 13, and image capture settings and editing settings are estimated as indicated by reference numerals 418 and 419, respectively. Because the editing settings 419 are parameter values ​​for the image editing application 402, the editing settings 419 are converted into parameter values ​​in a format that can be processed in the imaging device 11, as indicated by reference numeral 420. Based on the image capture settings 418 estimated using the learning model 405, the image capture function unit 401 sets the aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type in the image capture execution unit (as described above) to perform an image capture operation, and sets the transformed parameter values ​​420 to the signal processing unit (as described above) to perform signal processing of the image signal (such as RAW image development processing).

[0121] As described above, by using the learning model 405 provided from the model providing device 13, the camera device 11 can generate an image 421 based on the subject scene solely through the user's image capture operation (in other words, without image editing), which has a visual effect that reflects the trend of the visual effects of the image group 414 generated by the same user in the past by performing image capture and image editing.

[0122] Note that after the camera device 11 acquires the learning model 405 from the model providing device 13, whenever an image capture is performed using image capture settings set by the user, the camera device 11 also stores metadata and image capture settings associated with the captured image in a file. Furthermore, whenever the user performs image editing on the captured image using the editing device 12, the editing settings are stored associated with the original captured image in a file. Additionally, after transmitting the learning model 405 to the camera device 11, the model providing device 13 also relearns the model using each filegroup of newly collected metadata, image capture settings, and editing settings, and retransmits the file of the relearned learning model to the camera device 11.

[0123] A-3. Another example of system operation

[0124] In section A-3, the system operation will be described when the subject scene is monitored through the viewfinder before image capture.

[0125] Figure 10 The operation of a system 10 for displaying images for monitoring using a learning model is shown for a camera device 11. However, in Figure 10 In the middle, assuming according to Figure 4 The operation of the system 10 shown involves a learning model generated for a specific user being transmitted to the camera device 11, and this learning model is available on the camera device 11 side.

[0126] In addition to the image capture function unit 401, the camera device 11 also includes a viewfinder 408 and a monitoring function unit 407 for generating a monitoring image to be displayed on the viewfinder 408.

[0127] The monitoring function unit 407 continuously analyzes the subject scene captured in real time by the image capture function unit 401 to generate metadata 417. The metadata 417 is input into the learning model 405 to estimate the image capture setting value 418 and the editing setting value 419. Since the editing setting value 419 is a parameter value used for the image editing application 402, the editing setting value 419 is converted into a parameter value 420 in a format that can be processed in the camera device 11.

[0128] When acquiring a live view image from the image capture function unit 401 based on the image capture setting value 418 estimated by using the learning model 405, the monitoring function unit 407 generates a monitoring image equivalent to the live view image after editing using the image editing application 402 based on the parameter value 420 converted from the editing setting value, and displays the monitoring image in the viewfinder 408.

[0129] As described above, by using the learning model 405 provided by the model providing device 13, the camera device 11 can display a monitoring image with visual effects on the viewfinder 408 according to the subject scene at the time of monitoring. The visual effects reflect the trend of the visual effects of the same user's past image sets generated by performing image capture and image editing.

[0130] A-4. Variation

[0131] In section A-2 above, the operation of system 10 has been described, in which a learned model that has learned the correlation between image capture settings, editing settings, and metadata is generated and used on camera device 11. In section A-4, reference will be made to... Figure 11 A variation thereof is described in which a learning model is generated that learns only the correlation between editing settings and metadata without using image capture settings for model learning, and the learning model is used in the camera device 11.

[0132] Each time an image is captured, the user accesses the user interface of the camera device 11 (in...). Figure 11(Not shown in the image) to set image capture settings. Image capture function unit 401 performs image capture based on the set image capture settings. Image capture function unit 401 can perform image capture of still images or videos.

[0133] When the camera device 11 converts captured images into files, the metadata in the image capture is also stored in the files. The metadata can be stored in the same file as the captured images, or it can be stored in a different file than the captured images, as long as it is associated with the captured images. Reference numeral 411 indicates a group of captured images output from the camera device 11, and reference numeral 412 indicates the metadata associated with each captured image. Figure 4 The system operation examples shown are different; the image capture settings for each image capture operation are not stored in the file.

[0134] The editing device 12 receives the captured image 411 from the camera device 11 and performs editing processes, such as image processing, on the captured image according to the user's editing operations using the image editing application 402. When the editing device 12 converts the edited image into a file for output, it also stores the editing settings values ​​associated with the original captured image in the file. Reference numeral 414 indicates the file group of the edited image output from the image editing application 402, and reference numeral 415 indicates the file group of editing settings values ​​associated with the original captured image. Note that the editing device 12 may store a portion of the metadata used for image editing in a metadata file.

[0135] The model providing device 13 collects file groups 412 and 415, which are stored in association with the original captured image, and stores these file groups in association with the captured image for each user in the storage device 403. The model providing device 13 may perform the operation of collecting file groups 412 and 415 periodically, or may perform the operation in response to instructions from the user.

[0136] In the model providing device 13, for a specific user, the model learning unit 1101 enables the model to learn the correlation between metadata and editing settings associated with each captured image, generates a learning model 1103 for estimating editing settings based on the metadata, and converts the learning model 1103 into a file for storage. The learning model generated by the model providing device 13 is a model for a specific user, used to estimate appropriate editing settings for each subject scene based on the specific user's past image editing tendencies. The model providing device 13 transmits the generated learning model file 1104 to the camera device 11 of the specific user. The model providing device 13 may transmit the learning model file 1104 in response to a download request from the camera device 11, or it may push the learning model file 1104 to deliver it.

[0137] When the camera device 11 receives an instruction from the user to capture an image after acquiring the learning model 1103, the image capture function unit 401 analyzes the subject scene to generate metadata 417. The metadata 417 is input to the learning model 1103 to estimate the editing setting value 1105. Because the editing setting value 1105 is a parameter value for the image editing application 402, it is converted into a parameter value 1106 in a format that can be processed in the camera device 11. The image capture function unit 401 sets image capture setting values ​​such as aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type based on the user's instruction to capture the image, to perform the image capture operation, and performs signal processing of the image signal (such as RAW image development processing) using the converted parameter value 1106.

[0138] exist Figure 11 In the system operation shown, although the user needs to set the image capture settings each time they give an instruction to capture an image, it is also possible to generate an image based on the subject scene without image editing. This image has a visual effect that reflects the trend of the visual effects of the same user's past image sets generated by performing image capture and image editing.

[0139] A-5. Effect

[0140] Section A-5 outlines the effects achieved by system 10 through the application of this disclosure. However, the effects described herein are merely examples, and the effects exhibited by this disclosure are not limited thereto. Furthermore, in addition to the effects described above, this disclosure may also exhibit additional effects. Other objects, features, and advantages of this disclosure will become apparent from the detailed description of the embodiments described in this specification and the accompanying drawings.

[0141] In system 10, after the camera device 11 acquires the learning model, it is not necessary for the user to perform complex setting operations for image capture settings each time an image capture is performed, nor is it necessary to perform image processing on the captured image using image editing application 402. The system can capture images that reflect the user's intentions and preferences based on the subject scene.

[0142] In system 10, when a user monitors the subject scene before image capture using the viewfinder of the camera device 11, a monitoring image reflecting the user's intentions and preferences can be displayed based on the subject scene without performing complex image capture setting operations. Therefore, the user can perform image capture while simultaneously checking an image equivalent to the final image after image processing using the image editing application 402 before image capture.

[0143] In system 10, by applying a learning model generated from image capture and image editing performed by a skilled user to a camera device 11 for an inexperienced user, the inexperienced user can imitate an image equivalent to one captured and edited by a skilled user.

[0144] In system 10, control of the camera device 11 is performed using a simple algorithm that does not involve a large number of conditional branch determinations by using a learning model generated from captured images and image editing by the user. Therefore, the risk of impairing the quality of operation due to errors in the program can be reduced.

[0145] B. Specific Examples

[0146] B-1. Configuration of camera equipment

[0147] Figure 12 The configuration of the camera device 11 to which this disclosure applies is schematically shown. The camera device 11 shown in the figure includes an image capture optical system 1201, an image capture unit 1202, an image processing unit 1203, a display unit 1204, a storage unit 1205, an operation unit 1206, a control unit 1207, a sensor unit 1208, and a bus 1209.

[0148] The image capture optical system 1201 is configured by combining multiple optical lenses, such as a focusing lens and a zoom lens (neither shown). The image capture optical system 1201 drives the focusing lens and zoom lens based on control signals from the image processing unit 1203 and the control unit 1207, and forms an optical image of the subject on the image capture plane of the image capture unit 1202. The image capture optical system 1201 also includes an aperture stop mechanism, a shutter mechanism, etc.

[0149] The image capture unit 1202 is configured using an image sensor such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD). The image capture unit 1202 performs photoelectric conversion to generate an analog image capture signal corresponding to the optical image of the subject. The image capture unit 1202 performs correlated double sampling (CDS) processing, automatic gain control (AGC) processing, and analog-to-digital (AD) conversion processing on the analog image capture signal, and outputs the captured image as a digital image signal.

[0150] The image processing unit 1203 performs development processing on the digital image signal, and also processes the developed digital image signal to generate a captured image. The image processing unit 1203 will be described in detail below.

[0151] For example, the display unit 1204 is a display device configured with a liquid crystal display (LCD), a plasma display panel (PDP), an organic electroluminescent (EL) panel, or the like. The display unit 1204 primarily functions as a viewfinder, displaying live-view images captured during image capture, as well as still images and videos stored in the storage unit 1205. The display unit 1204 also serves as a user interface by overlaying a touch panel on the screen, displaying menus on the screen and receiving touch operations.

[0152] Storage unit 1205 is configured using a recording medium such as a hard disk drive (HDD) or a solid-state drive (SSD). The recording medium can be fixedly disposed within the camera device 11 or detachably attached to it. Storage unit 1205 stores captured images processed by image processing unit 1203 in the recording medium. For example, storage unit 1205 stores the image signal of a still image in a file format based on a predetermined standard compression such as JPEG, and also stores information about the stored image associated with it (e.g., Exchangeable Image File Format (EXIF) data including additional information such as the image capture date and time). For example, metadata (see...). Figure 6 and Figure 7 At least some information of the captured image can be stored in storage unit 1205 as an Exif associated with the captured image. Specifically, a scaled-down image of the captured image can be stored if included in the Exif. Storage unit 1205 stores the video image signal in a file format compressed based on a predetermined standard such as Moving Picture Experts Group 2 (MPEG2). The compression and decompression processing of the image signal can be performed by storage unit 1205 or by image processing unit 1203. Storage unit 1205 stores the learning model transmitted from model providing device 13.

[0153] For example, the operation unit 1206 includes a power button for switching the power on and off of the main body of the camera device 11, a release button for indicating the start of storing captured images, an operation knob for zoom adjustment, and a touch panel integrated with the screen of the display unit 1204. The operation unit 1206 receives operations from the user (photographer) and generates operation signals corresponding to the user's operations, which are then output to the control unit 107. Image capture settings and some editing settings can be input via the operation unit 1206.

[0154] The control unit 1207 is configured via a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), etc. The ROM stores, in a non-volatile manner, information related to the configuration of the main body of the imaging device 11, such as the name of the imaging device 11 and the name of the lens used in the image capture optical system 1201, information related to a portion of metadata, and programs read and executed by the CPU. The RAM serves as working memory when the CPU executes programs. The CPU performs various processing operations according to the programs stored in the ROM and issues commands to comprehensively control the operation of each unit, causing the imaging device 111 to perform operations according to user operations received by the operation unit 1206.

[0155] In this embodiment, the ROM stores: a learning model (as described above) that estimates appropriate image capture settings and editing settings based on metadata such as the characteristics of the subject scene and the conditions at the time of image capture; a control program that performs image capture processing by estimating appropriate image capture settings and editing settings based on the conditions at the time of image capture and the characteristics of the subject scene when image capture is instructed using the learning model; a control program that converts the metadata and image capture settings data (learning data) required to generate the learning model into a file for storage; and so on. When image capture is instructed via the operation unit 1206, the CPU comprehensively controls the image capture operation of the camera device 11 based on the image capture settings and editing settings estimated using the learning model.

[0156] The program executed by the control unit 1207 can be pre-installed when the product is shipped, or can be appropriately updated after shipment by downloading or via a recording medium. The ROM can be an electrically erasable programmable ROM (EEPROM), which is electrically rewritable, allowing the stored program and other data (learning models, etc.) to be updated.

[0157] In addition to image capture operations, sensor unit 1208 also includes various sensors for obtaining information necessary to control the operation of the camera device 11. For example, sensor unit 1208 includes a clock for counting dates and times, a Global Positioning System (GPS) sensor for receiving GPS signals, a light intensity sensor, an inertial measurement unit (IMU) for detecting tilt and camera shake of the main body of camera device 11, etc. Any sensor element can be incorporated into sensor unit 1208.

[0158] Bus 1209 electrically interconnects the aforementioned units to allow these units to send and receive image signals, control signals, etc. Bus 1209 may include various types of buses.

[0159] Note that a stacked CMOS sensor with a multi-layer structure can be used in the image capture unit 1202 (see, for example, Patent Document 2). When using a stacked CMOS, pixel units including a pixel array are formed in the first layer, and logic units such as the image processing unit 1203 and memory that can be used as at least a portion of the storage area of ​​the storage unit 1205 are formed in the second and subsequent layers.

[0160] Figure 13 The internal configuration of the image processing unit 1203 is schematically shown. The image processing unit 1203 shown in the figure includes a developing unit 1301, a post-processing unit 1302, a detection unit 1303, and a focus control unit 1304.

[0161] The developing unit 1301 performs clamping processing, defect correction processing, and demosaic processing on the digital image signal supplied from the image capture unit 1202. The developing unit 1301 performs clamping processing to set the signal level of pixels in a black state (without light) to a predetermined level on the image signal generated by the image capture unit 1202. For example, the developing unit 1301 performs defect correction processing by using the pixel signals of surrounding pixels to correct the pixel signals of defective pixels. Furthermore, the developing unit 1301 performs demosaic processing to generate an image signal where one pixel indicates each component of YUV (Y is luminance, U is the difference between luminance and blue, and V is the difference between luminance and red) based on a pixel signal where one pixel indicates one color component generated by the image capture unit 1202.

[0162] The post-processing unit 1302 performs noise reduction, white balance adjustment, gamma adjustment, distortion adjustment, camera shake correction, image quality improvement, image magnification / reduction, and encoding processing on the image signal after development processing by the developing unit 1301, as needed. For example, the post-processing unit 1302 performs white balance processing by using a white balance coefficient calculated by the detection processing unit 1303. The post-processing unit 1302 can perform some post-processing on the image signal based on image capture settings and editing settings input via the operation unit 1206. The post-processing unit 1302 outputs the image signal for displaying the image (live view image) to the display unit 1204 and outputs the image signal for storing the image to the storage unit 1205.

[0163] The detection processing unit 1303 calculates the white balance coefficient based on the brightness value and integral value by using a detection area portion of the image signal after development processing by the development processing unit 1301, which is set automatically or by user setting, and outputs the white balance coefficient to the post-processing unit 1302. The detection processing unit 1303 can determine the exposure state based on the image signal of the detection area, generate an exposure control signal based on the determination result, and output the exposure control signal to the image capture optical system 1201 to drive the aperture, etc., so that the subject in the detection area has appropriate brightness.

[0164] The focus control unit 1304 generates a control signal for focusing on a predetermined subject based on the image signal after development processing by the development processing unit 1301, and outputs the control signal to the image capture optical system 1201. For example, the focus control unit 1304 generates a depth map of the image capture area or calculates the depth value of the predetermined subject based on the image signal after development processing by the development processing unit 1301, and performs focus control. For example, the depth map can be generated by using a deep learning-based DNN model, or it can be generated in other ways.

[0165] B-2. System Operation Example

[0166] Figure 14 It shows that the application is used Figure 12 The specific operation of the system 10 of the camera device 11 shown.

[0167] In the control unit 1207 of the camera device 11, in addition to the control software (SW) for image capture operation based on the image capture setting values ​​indicated by reference numeral 1401, the following algorithms are also run: a metadata generation algorithm (indicated by reference numeral 1402) for generating metadata that converts the features of the subject scene and the image capture status into data; a file conversion algorithm (indicated by reference numeral 1403) for converting each of the metadata and the image capture setting values ​​into a file and storing the file in association with the captured image; an estimation algorithm (indicated by reference numeral 1404) for estimating the image capture setting values ​​and editing setting values ​​based on the metadata using a learning model (or a dictionary lookup algorithm for deriving the image capture setting values ​​and editing setting values ​​from the metadata using a dictionary); a conversion algorithm (indicated by reference numeral 1405) for converting the editing setting values ​​into parameter values ​​that can be processed by the camera device 11; and a control algorithm (indicated by reference numeral 1406) for controlling the image capture operation via the control software 1401 based on the converted image capture setting values ​​and parameter values.

[0168] The CPU of the editing device 12 executes an editing application. The editing application processes the captured image (RAW image) generated by the camera device 11 to generate an edited image (e.g., a YUV image, RGB image, JPEG image, etc.). It processes the edited image after image processing by the camera device 11, or processes an already processed edited image again to output the edited image. Furthermore, the editing application includes a file conversion algorithm, indicated by reference numeral 1411, for converting the editing settings values ​​used to edit the captured image into a file and associating them with the original captured image.

[0169] On the CPU of the model providing device 13, a file collection algorithm, indicated by reference numeral 1421, and a model learning algorithm, indicated by reference numeral 1422, each operate independently. The file collection algorithm 1421 collects file groups containing metadata, image capture settings, and editing settings converted into files by file conversion algorithms 1403 and 1411, and accumulates these file groups in association with the original captured images for each user in the storage device 403. For a specific user, the model learning algorithm 1422 uses the file groups accumulated in the storage device 403 to learn a model by using the correlation between the metadata, image capture settings, and editing settings associated with each captured image, and generates a learned model for estimating the image capture settings and editing settings based on the metadata.

[0170] Figure 15 The flowchart shows the process in which... Figure 14 The system shown illustrates the image capture process performed by the camera device 11.

[0171] First, the user captures the setting value by inputting an image using the operation unit 1206, etc. (step S1501).

[0172] Based on the image capture settings input in step S1501, the control software 1401 controls the image capture optical system 1201 and the image capture unit 1202 to generate a captured image (RAW image) (step S1502).

[0173] Next, the metadata generation algorithm 1402 controls the image processing unit 1203 to generate metadata by converting information about the situation at the time of image capture into data while calculating information obtained by analyzing the features of the subject scene based on the captured image, such as a RAW image (step S1503).

[0174] The file conversion algorithm 1403 converts the image capture settings obtained in step S1501 and the metadata generated in step S1503 into files respectively, and stores these files in association with the captured image at that time (step S1504).

[0175] Next, it is checked whether the camera device 11 has acquired the learning model, that is, whether the learning model exists in the storage unit 1205 (step S1505).

[0176] When the camera device 11 has acquired the learning model (yes in step S1505), the estimation algorithm 1404 estimates the image capture settings and editing settings based on the metadata generated in step S1503 using the learning model (step S1506).

[0177] The conversion algorithm 1405 converts the estimated editing settings value in step S1506 into parameter values ​​that can be processed by the camera device 11 (step S1507).

[0178] The control algorithm 1406 controls the image capture optical system 1201 and the image capture unit 1202 via the control software 1401 to perform image capture based on the image capture setting values ​​estimated in step S1506 and generate captured images (RAW images, etc.) (step S1508).

[0179] Next, based on the parameter values ​​converted in step S1507, control algorithm 1406 controls image processing unit 1203 via control software 1401 to perform development processing on the captured image, such as a RAW image, generated in step S1508. Then, control algorithm 1406 generates a processed image (e.g., YUV image, RGB image, JPEG image, etc.) with a visual effect that reflects the tendency of the same user to generate a group of images in the past by performing image capture and image editing (step S1509), and terminates the image capture operation.

[0180] On the other hand, before the camera device 11 has acquired a learning model (no in step S1505), the control software 1401 controls the image processing unit 1203 to perform development processing on the captured image, such as a RAW image, generated in step S1502, based on the image capture settings input in step S1501. Then, the control software 1401 generates the processed image (e.g., a YUV image, an RGB image, a JPEG image, etc.) (step S1510) and terminates the image capture operation.

[0181] Figure 16 The flowchart shows the process in which... Figure 14 In the system shown, the editing device 12 performs the processing of editing operations on the captured image output from the camera device 11.

[0182] The image editing application executed by the CPU of the editing device 12 performs editing processing using a captured image (RAW image) or a processed image (e.g., YUV image, RGB image, JPEG image, etc.) from the camera device 11, and outputs the edited image (e.g., YUV image, RGB image, JPEG image, etc.) (step S1601). Here, the image editing application is based on editing settings such as brightness, color temperature adjustment, color correction, contrast, white level, black level, and chroma input by the user (see [link to image editing application]). Figure 8 ( ) to process RAW or YUV images captured by camera device 11.

[0183] The file conversion algorithm 11 converts the editing settings values ​​used by the image editing application to edit the captured image in step S1601 into a file, stores the file in association with the original captured image (step S1602), and terminates the editing operation.

[0184] Figure 17 The flowchart shows the process in which... Figure 14 The system shown illustrates the process by which the model-providing device 13 generates the model.

[0185] File collection algorithm 1421 collects file groups containing metadata and image capture settings converted into files by file conversion algorithm 1403 in camera device 11, and file groups containing editing settings converted into files by file conversion algorithm 1411 in editing device 12, and associates these file groups with captured images (see reference). Figure 2 The data is associated with the accumulation in storage device 403 (step S1701).

[0186] Before the number of files for a certain user stored in storage device 403 reaches a certain number (not in step S1702), model learning algorithm 1422 repeatedly collects the file group returned to step S1701 without performing model learning for that user.

[0187] On the other hand, when the number of files for a certain user accumulated in storage device 403 reaches a certain number ("Yes" in step S1702), model learning algorithm 1422 uses the file group accumulated in storage device 403 to learn a model for the user by using the correlation between metadata associated with each captured image, image capture setting value and editing setting value, generate a learning model for estimating image capture setting value and editing setting value that appropriately reflect the user's past image capture and image editing tendencies based on metadata, and convert the learning model into a file for storage (step S1703).

[0188] Then, the model providing device 13 transmits the learning model file to the corresponding user's camera device 11 (step S1704) and terminates the process. For example, the model providing device 13 may transmit the learning model file 416 in response to a download request from the camera device 11, or it may push the learning model file to deliver it.

[0189] B-3. ​​Another operational example of the system

[0190] In section B-3, the system operation of the camera device 11 when monitoring the subject scene using the display unit 1201 (viewfinder) will be described.

[0191] Figure 18 An example of system operation when monitoring a subject scene is shown. However, it is assumed that the camera device 11 has already obtained a learned model from the model providing device 13, and therefore, in Figure 18 The description of the editing device 12 and the model providing device 13 is omitted.

[0192] In the control unit 1207 of the camera device 11, in addition to the control software (SW) indicated by reference numeral 1801 for image capture operation based on image capture setting values, the following algorithms are also run: a metadata generation algorithm (indicated by reference numeral 1802) for generating metadata that converts the features of the subject scene and the image capture status into data; an estimation algorithm (indicated by reference numeral 1803) for estimating image capture setting values ​​and editing setting values ​​based on metadata using a learning model (or a dictionary lookup algorithm for deriving image capture setting values ​​and editing setting values ​​from metadata using a dictionary); a conversion algorithm (indicated by reference numeral 1804) for converting editing setting values ​​into parameter values ​​that can be processed by the camera device 11; and a control algorithm (indicated by reference numeral 1805) for controlling image capture operation and display operation via the control software 1801 based on the converted image capture setting values ​​and parameter values.

[0193] Figure 19 The process for monitoring the subject scene using camera device 11 is illustrated in the form of a flowchart.

[0194] First, the user inputs image capture settings using the operation unit 1206, etc. (step S1901). The control software 1801 controls the image capture optical system 1201 and the image capture unit 1202 based on the image capture settings input in step S1501 to generate a RAW image (step S1902).

[0195] Next, the metadata generation algorithm 1802 controls the image processing unit 1203 to calculate information obtained by analyzing the features of the subject scene from captured images such as RAW images, and also generates metadata by converting information about the condition at the time of image capture into data while generating information obtained by reducing the size of the captured image (step S1903).

[0196] When it is necessary to continue displaying the monitoring image on the display unit (viewfinder) 1207 (yes in step S1904), the camera device 11 is subsequently checked to see if the learning model has been acquired, that is, whether the learning model exists in the storage unit 1205 (step S1905).

[0197] When the camera device 11 has acquired the learning model (yes in step S1905), the estimation algorithm 1803 estimates the image capture settings and editing settings based on the metadata generated in step S1903 using the learning model (step S1906).

[0198] Next, the conversion algorithm 1804 converts the edit setting value estimated in step S1906 into parameter values ​​that can be processed by the camera device 11 (step S1907).

[0199] Then, the control algorithm 1805 controls the image capture optical system 1201 and the image capture unit 1202 via the control software 1801 to perform image capture based on the image capture setting values ​​estimated in step S1906, and generates a captured image such as a RAW image (step S1908).

[0200] Next, based on the parameter values ​​converted in step S1907, control algorithm 1805 controls image processing unit 1203 via control software 1801 to perform development processing on captured images such as RAW images generated in step S1908, and generates processed images (e.g., YUV images, RGB images, JPEG images, etc.) with visual effects that reflect the tendency of the same user to generate image groups in the past by performing image capture and image editing (step S1909).

[0201] Next, based on the image capture settings estimated in step S1906, control algorithm 1805 controls display unit 1207 via control software 1801, and displays the processed image (e.g., YUV image, RGB image, JPEG image, etc.) generated in step S1909 (step S1910). Then, control software 1801 repeats the same process described above and returns to step S1904.

[0202] On the other hand, when the camera device 11 has not yet acquired the learning model (no in step S1905), the control software 1801 controls the image processing unit 1203 based on the image capture setting value input in step S1901, and performs development processing on the captured image such as a RAW image generated in step S1902 to generate a processed image (e.g., a YUV image, an RGB image, a JPEG image, etc.) (step S1911).

[0203] Next, the control software 1801 controls the display unit 1207 based on the image capture settings input in step S1901, so that the processed image (e.g., YUV image, RGB image, JPEG image, etc.) generated in step S1910 is displayed (step S1911). Then, the control software 1801 repeats the same process described above and returns to step S1904.

[0204] When the display of the monitored image on the display unit (viewfinder) 1207 is terminated (no in step S1904), this process also terminates.

[0205] C. Application Examples

[0206] C-1. First Application Example

[0207] As Figure 4 and Figure 9 The example shown is an application example of the operation of system 10. Figure 20 The generation of a learning model is illustrated when the difference information between the original captured image and the edited image is used as learning data instead of the editing settings values ​​of the image editing application, and the operation of the system 10 after the camera device 11 acquires the learning model.

[0208] First, the description will continue until... Figure 20 The operation of generating the learning model in System 10 shown.

[0209] The user accesses the camera device 11 via the user interface (in) each time an image is captured. Figure 4 (Not shown in the image) to set image capture settings (aperture value, shutter speed, ISO value, exposure correction value, color temperature, focus area type, etc.). Image capture function unit 401 performs image capture based on the set image capture settings. Image capture function unit 401 generates metadata (see [reference]) when performing image capture based on the set image capture settings. Figure 6 and Figure 7 ).

[0210] When the camera device 11 converts the captured image into a file, the image capture metadata and image capture settings are also stored in the file. The metadata and image capture settings can be stored in the same file as the captured image, or they can be stored in a different file, as long as they are associated with the captured image. Figure 20 In the figure, reference numeral 411 indicates the file group of captured images output from the camera device 11, and reference numerals 412 and 413 respectively indicate the file groups of metadata and image capture settings associated with the captured images.

[0211] Subsequently, upon receiving the captured image 411 from the camera device 11, the editing device 12 performs image processing such as image processing on the captured image based on the user's editing operations using the image editing application 402. The image editing application 402 is based on editing settings set by the user (see...). Figure 8 ( ) to process the captured image input from the camera device 11.

[0212] When an image edited by image editing application 402 is output after being converted to a file, the difference information between the original captured image and the edited image is stored in the file in association with the original captured image. Note that the editing device 12 may store a portion of the metadata used for image editing in the metadata file. Figure 20In this context, reference numeral 414 indicates the file group of edited images output from image editing application 402, and reference numeral 2201 indicates the difference information between the original captured image and the edited image. The "difference information" between the images mentioned herein includes, for example, the following information.

[0213] The difference in brightness values ​​of each corresponding pixel between the original captured image and the edited image.

[0214] Color difference between each corresponding pixel between the original captured image and the edited image

[0215] The model providing device 13 collects file groups 412, 413, and 2201, which are respectively associated with the original captured image, image capture settings, and difference information between the captured image and the edited image, and stores these file groups in the storage device 403 in association with the captured image for each user.

[0216] In the model providing device 13, the model learning unit 2202, for a specific user, enables the model to learn the correlation between metadata associated with each captured image, image capture settings, and difference information between captured and edited images. This generates a learning model 2203 for estimating image capture settings and difference information between captured and edited images based on the metadata, and the learning model 2203 is converted into a file for storage. The learning model 2203 generated by the model providing device 13 is a user-specific model used to estimate suitable image capture settings and appropriate editing settings for each subject scene based on the user's past image capture and editing tendencies. The model providing device 13 performs the learning model generation process using compressed metadata and difference information between images, thus reducing storage and computational costs compared to learning a model using captured images with large data sizes as is. The model providing device 13 transmits the learning model 2203 to the user's camera device 11.

[0217] Next, it will be described in Figure 20 The operation of the camera device 11 in the system 10 shown after acquiring the learning model.

[0218] When the imaging device 11 receives an instruction from the user to capture an image, the image capture function unit 401 analyzes the subject scene to generate metadata as indicated by reference numeral 417. The metadata 417 is input to the learning model 2203 provided by the model providing device 13, and as indicated by reference numerals 418 and 2204, the image capture setting values ​​and the difference information between the captured and edited images are estimated. Furthermore, as indicated by reference numeral 2205, the difference information between the captured and edited images is converted into parameter values ​​in a format that can be processed in the imaging device 11. Then, based on the image capture setting values ​​418 estimated using the learning model 2203, the image capture function unit 401 sets the aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type in the image capture execution unit (as described above) to perform an image capture operation, and sets the converted parameter values ​​2205 in the signal processing unit (as described above) to perform signal processing of the image signal (such as RAW image development processing).

[0219] As described above, by using the learning model 2203 provided by the model providing device 13, the camera device 11 can generate an image 2206 based on the subject scene solely through the user's image capture operation (in other words, even without image editing), which has a visual effect that reflects the trend of the visual effects of the same user's past image sets 414 generated by performing image capture and image editing.

[0220] C-2. Second Application Example

[0221] As Figure 20 Another application example of the operation of system 10 shown is as follows: Figure 21 The diagram illustrates the generation of a learning model when the editing device 12 automatically edits a captured image using a generative AI model instead of an image editing application, and the operation of the system 10 after the camera device 11 acquires the generative AI model.

[0222] First, the description will continue until... Figure 21 The operation of generating the learning model in System 10 shown.

[0223] The operation of the camera device 11 before acquiring the learning model is the same as described above. Figure 20 The application examples shown operate the same way, and therefore their descriptions will be omitted here.

[0224] Upon receiving the captured image 411 from the camera device 11, the editing device 12 performs automatic editing processing on the captured image using a generative AI model 2101. The generative AI model 2101 can generate an edited image of the captured image. The generative AI model 2101 can be an image editing model that has been learned to edit captured images or generate edited images of captured images. Model parameters, etc., are input to the generative AI model 2101 via an external input device 2110.

[0225] When an image edited by the generative AI model 2101 is converted into a file for output, the difference information between the original captured image and the edited image is stored in the file in association with the original captured image. Note that the editing device 12 may store a portion of the metadata used for image editing in the metadata file. Figure 21 In the figure, reference numeral 2102 indicates the file group of images after editing the output from the generative AI model 2101, and reference numeral 2103 indicates the difference information between the original captured image and the edited image. The “difference information” between the images mentioned herein is the same as described above.

[0226] The model providing device 13 collects file groups 412, 413, and 2103, which are respectively associated with the original captured image, image capture settings, and difference information between the captured image and the edited image, and stores these file groups in the storage device 403 in association with the captured image for each user.

[0227] In the model providing device 13, the model learning unit 2104 enables the model to learn the correlation between metadata associated with each captured image of a specific user, image capture setting values, and difference information between captured and edited images. This generates a learning model 2105 for estimating image capture setting values ​​and difference information between captured and edited images based on the metadata, and the learning model 2105 is converted into a file for storage. The learning model 2105 generated by the model providing device 13 is a model for a specific user, used to estimate suitable image capture and editing settings for each subject scene based on the specific user's past image capture and editing tendencies. The model providing device 13 transmits the learning model 2105 to the camera device 11 of the specific user. When the learning model 2100 has already been saved, the camera device 11 updates or changes the learning model 2100 based on the newly provided learning model 2105 from the model providing device 13.

[0228] Next, it will be described in Figure 21 The operation of the camera device 11 in the system 10 shown after acquiring the learning model.

[0229] When the imaging device 11 receives an instruction from the user to capture an image, the image capture function unit 401 analyzes the subject scene to generate metadata as indicated by reference numeral 417. The metadata 417 is input to the learning model 2100, and as indicated by reference numerals 2106 and 2107, image capture settings and difference information between the captured and edited images are estimated. Furthermore, as indicated by reference numeral 2108, the difference information between the captured and edited images is converted into parameter values ​​in a format that can be processed in the imaging device 11. Then, based on the image capture settings 2106 estimated using the learning model 2100, the image capture function unit 401 sets the aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type in the image capture execution unit (as described above) to perform an image capture operation, and sets the converted parameter values ​​2108 to the signal processing unit (as described above) to perform signal processing of the image signal (such as RAW image development processing).

[0230] As described above, by using the learning model 2100 provided from the model providing device 13, the camera device 11 can generate an image 2109 based on the subject scene simply through the user's image capture operation (in other words, even without image editing), which has a tendency to reflect the visual effect of the image group 2102 generated by automatic editing using the generative AI model 2101.

[0231] C-3. Third Application Example

[0232] Figure 22 It shows Figure 1 The example of application of System 10 shown. Figure 1 The system 10 shown is configured such that only the model providing device 13 is located in the cloud, and ordinary users perform image capture and image editing operations by using the camera device 11 and the editing device 12. In contrast, Figure 22 The system 10 shown is configured such that the editing device 12 is also located in the cloud along with the model providing device 13, and ordinary users perform image capture by using the camera device 11.

[0233] The following will describe Figure 22 The operation of system 10 shown.

[0234] The user uploads captured images (e.g., RAW images) from camera device 11 to editing device 12 in the cloud, and in response, editing device 12 performs editing processing on the uploaded captured images. Camera device 11 can download the edited images (e.g., YUV images, RGB images, JPEG images, etc.) from editing device 12. Camera device 11 uploads metadata and image capture settings associated with the captured images to model providing device 13. The metadata, as already referenced... Figure 6 and Figure 7 As described, and may also include a scaled-down version of the captured image.

[0235] For example, the editing device 12 performs editing processing on a captured image uploaded from the camera device 11 based on the user's editing operations on the camera device 11 (or other information terminal owned by the user). The editing device 12 can perform image editing by using an image editing application, or it can generate the edited image by using a generative AI model. The editing device 12 associates the editing information during image editing with the captured image and transmits the editing information to the model providing device 13. The editing information includes at least one of the editing settings values ​​for image editing or difference information between the original captured image and the edited image.

[0236] The model providing device 13 collects metadata, image capture settings, and editing information associated with the captured image from the camera device 11 and the editing device 12. By using the large amount of collected metadata, image capture settings, and editing information, the model providing device 13 enables the model to learn the correlations between the metadata, image capture settings, and editing information, and generates a learned model for estimating the image capture settings and editing information based on the metadata. The model providing device 13 then transmits the learned model to the camera device 11.

[0237] By using the learning model provided by the model providing device 13, the camera device 11 can generate an image based on the subject scene solely through the user's image capture operation (in other words, even without image editing), which has a visual effect that tends to reflect the visual effect of the image that has been edited by the editing device 12.

[0238] like Figure 22 The system configuration indicated, in which the editing device 12 is also located in the cloud along with the model providing device 13, has the following advantages: Figure 23 As shown, services implemented by a combination of model providing device 13 and editing device 12 can be provided to multiple camera devices 11.

[0239] The model providing device 13 can use a large amount of metadata, image capture settings, and editing information associated with captured images from multiple camera devices 11 to provide each camera device 11 with a deep learning-based learning model. This learning model should be considered as being designed for a general user rather than a specific user.

[0240] Furthermore, the model providing device 13 can provide a learning model with professional specifications to multiple camera devices 11. This learning model has been learned by using metadata and image capture settings associated with captured images by experts such as professional photographers, as well as editing information from professional image editors.

[0241] The editing device 12 can perform image editing by using an image editing application, or it can generate edited images by using a generative AI model. Because it is shared by a large number of users, image editing services or edited image generation services can be provided by using expensive and deep learning-intensive generative AI models.

[0242] D. Configuration of information processing devices

[0243] Figure 24 An example of the hardware configuration of an information processing device 2000 that can operate as an editing device or a model providing device is shown. The information processing device 2000 includes: a central processing unit (CPU) 2001, a read-only memory (ROM) 2002, a random access memory (RAM) 2003, a host bus 2004, a bridge 2005, an expansion bus 2006, an interface unit 2007, an input unit 2008, an output unit 2009, a storage unit 2010, a driver 2011, and a communication unit 2013. For example, the information processing device 2000 may be configured as a personal computer or the like.

[0244] CPU 2001 controls the overall operation of information processing device 2000 according to various programs. Furthermore, when performing computationally demanding processes such as model learning on information processing device 2000, CPU 2001 is preferably a multi-core CPU (e.g., Apple M1 Max), and in addition to CPU 2001, information processing device 2000 preferably also includes multi-core processors such as graphics processing unit (GPU) or general-purpose computing on graphics processing unit (GPGPU) (e.g., "Quadro A6000", available from NVIDIA, etc.). However, for convenience, they will be collectively referred to as CPU 2001 below.

[0245] ROM 2002 stores programs (basic input / output system, etc.) and operating parameters used by CPU 2001 in a non-volatile manner. RAM 2003 is used to load programs to be executed by CPU 2001 and to temporarily store parameters such as working data that may change occasionally during program execution. For example, programs loaded into RAM 2003 and executed by CPU 2001 are various application programs, operating systems (OS), etc.

[0246] CPU 2001, ROM 2002, and RAM 2003 are interconnected via a host bus 2004 configured by a CPU bus, etc. Through the coordinated operation of ROM 2002 and RAM 2003, CPU 2001 executes various applications in a runtime environment provided by the OS, thereby implementing various functions and services. In the case where the information processing device 2000 is a personal computer, the OS may be, for example, Microsoft Windows, Unix, or a successor OS. For example, applications or modules within applications may be existing libraries stored, shared, or distributed by source code management services.

[0247] When the information processing device 2000 operates as an editing device, an editing application for performing image processing such as image processing on images captured by the camera device is executed on the CPU 2001. This editing application includes a file conversion algorithm for converting editing settings values ​​used to edit the captured images into files for association with the original captured images. Furthermore, when the information processing device 2000 operates as a model providing device, a file collection algorithm is run on the CPU 2001 for collecting metadata, image capture settings values, and editing settings values ​​for each file group that have been converted into files. A model learning algorithm is also run to estimate image capture settings and editing settings values ​​reflecting the user's past image capture and editing tendencies based on the correlation between the metadata, image capture settings values, and editing settings values ​​associated with each captured image.

[0248] The host bus 2004 is connected to the expansion bus 2006 via bridge 2005. The expansion bus 2006 is, for example, a Peripheral Component Interconnect (PCI) bus or PCI Express, and bridge 2005 conforms to the PCI standard. Note that the information processing device 2000 does not need to be configured such that circuit components are separated by the host bus 2004, bridge 2005, and expansion bus 2006, and can be implemented such that virtually all circuit components are interconnected via a single bus (not shown).

[0249] Interface unit 2007 connects peripheral devices such as input unit 2008, output unit 2009, storage unit 2010, driver 2011, and communication unit 2013 according to the expansion bus 2006 standard. Note that... Figure 24 All peripheral devices shown are not necessarily required, and the information processing apparatus 2000 may also include peripheral devices (not shown). Peripheral devices may be built into the main body of the information processing apparatus 2000, or some peripheral devices may be externally connected to the main body of the information processing apparatus 2000.

[0250] Input unit 2008 includes input control circuitry for generating an input signal based on input from the user and outputting the input signal to CPU 2001. When information processing device 2000 is a personal computer, input unit 2008 may include a keyboard, mouse, and touch panel. For example, output unit 2009 includes display devices such as liquid crystal display (LCD), organic electroluminescent (EL) display devices, and light-emitting diode (LED) display devices, as well as sound output devices such as speakers.

[0251] Storage unit 2010 stores files containing programs (applications, operating system, etc.) executed by CPU 2001, as well as various types of data. Storage unit 2010 is configured with a high-capacity storage device such as a solid-state drive (SSD) or hard disk drive (HDD), but may include external storage devices. When information processing device 2000 operates as a model providing device, for each user, a filegroup of metadata, image capture settings, and editing settings is stored in storage unit 2010 associated with the original captured image.

[0252] The removable storage medium 2012 is a recording medium configured as a cassette tape, such as a microSD card. The drive 2011 performs read and write operations on the loaded removable storage medium 2012. The drive 2011 outputs data read from the removable recording medium 2012 to RAM 2003 and storage unit 2010, and writes data from RAM 2003 or storage unit 2010 to the removable recording medium 2012.

[0253] The communication unit 2013 is a device for performing wireless communication such as WiFi and Bluetooth, as well as cellular communication networks such as 4G and 5G. The communication unit 2013 may include terminals such as Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI), and may also include the function of performing HDMI communication with USB devices such as scanners, printers, and displays. When operating as an editing device, the information processing device 2000 acquires captured images from the camera device or transmits files of editing settings values ​​to the model providing device via the communication unit 2013. Furthermore, when operating as a model providing device, the information processing device 2000 acquires metadata, image capture settings values, and each file group of editing settings values ​​from the camera device or editing device via the communication unit 2013, or transmits the generated learning model to the camera device.

[0254] [Industrial Applicability]

[0255] This disclosure has been described in detail with reference to specific embodiments. However, this disclosure should not be construed as limiting itself to the above embodiments, and it is apparent that those skilled in the art can modify and substitute the embodiments without departing from the spirit of this disclosure. The effects described in this specification are merely examples, and the effects demonstrated by this disclosure are not limited, and may provide additional effects beyond those described in this specification.

[0256] This specification primarily describes implementations of this disclosure applied to systems comprising physically independent camera devices, editing devices, and model-providing devices, but the scope of this disclosure is not limited thereto. For example, this disclosure can be similarly applied to systems comprising camera devices with editing capabilities and model-providing devices. Furthermore, the model-providing device is, for example, assumed to be a server located in the cloud, and can generate models reflecting past image capture and image editing trends of each user and provide them to a large number of camera devices. Alternatively, the camera device may be equipped with model generation capabilities and can internally generate learning models reflecting past image capture and image editing trends of users of the camera device.

[0257] In short, this disclosure has been described by way of example and should not be construed as restrictive. The scope of the claims should be considered in order to determine the gist of this disclosure.

[0258] The series of processing operations described in this specification can be performed by hardware, software, or a combination of hardware and software. When processing is performed by software, a program containing the processing sequence related to the implementation of this disclosure is installed in the memory of a computer incorporated in dedicated hardware and executed. Alternatively, programs capable of performing various processing operations can be installed in a general-purpose computer to enable the general-purpose computer to perform processing related to the implementation of this disclosure.

[0259] For example, the program can be pre-stored on an HDD or SSD as a recording medium, or it can be stored on a recording medium provided in a computer, such as a ROM. Alternatively, the program can be temporarily or permanently stored on a removable recording medium, such as a floppy disk, CD-ROM, magneto-optical (MO) disk, digital versatile disk (DVD), Blu-ray disc (BD), magnetic disk, or Universal Serial Bus (USB) storage. By using the aforementioned removable recording media, the program related to the implementation of this disclosure can be provided as so-called packaged software.

[0260] The program can be transmitted wirelessly or wiredly from the download site to the computer via a network such as a wide area network (WAN), local area network (LAN), or the Internet, as represented by a cellular network. The computer can receive the program transmitted in the above manner and install the program on a large-capacity storage device such as an HDD or SSD within the computer.

[0261] Note that this disclosure may also have the following configurations.

[0262] (1) An information processing device, comprising:

[0263] A collection unit is configured to collect metadata and image capture settings associated with the captured image by the camera device, as well as editing information during image editing of the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device.

[0264] A generation unit, configured to enable a model to learn the correlations between metadata, image capture settings, and editing information accumulated in the storage device, and to generate a learning model for estimating image capture settings and editing information based on the metadata; and

[0265] A transmission unit configured to transmit the generated learning model to the camera device.

[0266] (2) The information processing apparatus according to (1), wherein,

[0267] The metadata includes at least one of the following: first information obtained by analyzing the features of the subject scene, second information obtained by converting the state at the time of image capture into data, or third information obtained by reducing the size of the image of the subject scene.

[0268] (3) The information processing apparatus according to (2), wherein,

[0269] The first information includes at least one of the following: frequency values ​​obtained by classifying the brightness values ​​of each pixel in the image into five levels; the ratio of the values ​​of the brightest and darkest pixels; the brightness level of the pixel at the focus position in the five-level classification; frequency values ​​obtained by classifying the color of each pixel in the image into five colors; the color level of the pixel at the focus position in the five-level classification; whether there is a moving object in the subject scene; and distance information of the subject.

[0270] The second information includes at least one of the following: the name of the image capturing device, the name of the lens used, the date and time of image capture, the ambient light intensity value, and the tilt of the camera device (or camera device shake).

[0271] (4) The information processing apparatus according to any one of (1) to (3), wherein,

[0272] The image capture settings include at least one of the following: aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type.

[0273] (5) The information processing apparatus according to any one of (1) to (4), wherein,

[0274] The editing information includes editing settings, which include at least one of the following: brightness, color temperature adjustment, color correction, contrast, white level, black level, and chroma.

[0275] (6) The information processing apparatus according to (1), wherein,

[0276] The editing information includes information on the differences between the captured image and the edited image.

[0277] (6-1) The information processing apparatus according to (6), wherein,

[0278] The difference information between the captured image and the edited image includes at least one of the following: the difference in brightness value of each corresponding pixel between the original captured image and the edited image, or the difference in color of each corresponding pixel between the original captured image and the edited image.

[0279] (7) The information processing apparatus according to any one of (1) to (6), wherein,

[0280] The generation unit enables the model to learn the correlation between metadata, image capture settings, and editing information based on a specific user's image capture and editing operations, and to generate a learning model that reflects the specific user's past image capture and image editing tendencies.

[0281] (8) An information processing method, comprising:

[0282] The image capture metadata and image capture settings are collected in association with the captured images of the camera device, as well as the editing information during image editing of the captured images, and the metadata, image capture settings, and editing information are accumulated in the storage device;

[0283] The model learns the correlations between metadata, image capture settings, and editing information accumulated in the storage device, and generates a learned model for estimating image capture settings and editing information based on the metadata; and

[0284] The generated learning model is transmitted to the camera device.

[0285] (9) A computer program written in a computer-readable format, said computer program being used to cause a computer to function as:

[0286] A collection unit is configured to collect metadata and image capture settings associated with the captured image by the camera device, as well as editing information during image editing of the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device.

[0287] A generation unit, configured to enable a model to learn the correlations between metadata, image capture settings, and editing information accumulated in the storage device, and to generate a learning model for estimating image capture settings and editing information based on the metadata; and

[0288] A transmission unit configured to transmit the generated learning model to the camera device.

[0289] (10) An image capturing device including an image capturing function unit, the image capturing device comprising:

[0290] A generation unit, configured to analyze the subject scene and generate metadata; and

[0291] An estimation unit is configured to estimate a setting value based on the metadata using a learning model that has learned the correlation between the metadata and the setting value.

[0292] The image capture function unit is controlled by using the setting value to perform an image capture operation.

[0293] (11) The image capture apparatus according to (10), wherein,

[0294] The estimation unit uses the learning model to estimate image capture settings and editing information based on the metadata.

[0295] The image capture device further includes a conversion unit configured to convert the editing information into parameter values ​​that can be processed by the image capture device.

[0296] The image capture function unit is controlled to perform image capture operations by using the image capture settings and the parameter values.

[0297] (12) The image capture device according to (11) further includes:

[0298] A monitoring function unit, configured to generate a monitoring image from the image capture function unit, wherein...

[0299] The monitoring function unit is controlled by using the image capture settings and the parameter values ​​so that the monitoring screen is displayed.

[0300] (13) The image capturing apparatus according to any one of (10) to (12), wherein,

[0301] The metadata and each of the settings values ​​are stored during image capture.

[0302] (14) The image capturing apparatus according to any one of (10) to (13), wherein,

[0303] The estimation unit uses a learning model generated by using the image capture settings and metadata at the time of image capture, as well as editing information when editing the captured image.

[0304] (15) A system comprising:

[0305] A camera device, an editing device configured to edit images captured by the camera device, and a model providing device configured to provide a learning model to the camera device, wherein,

[0306] The camera device outputs metadata related to image capture and image capture settings associated with the captured image.

[0307] The editing device outputs editing information during image editing of the captured image, and

[0308] The model providing apparatus includes: a collection unit configured to collect metadata, image capture settings, and editing information and accumulate the metadata, image capture settings, and editing information into a storage device; a generation unit configured to enable a model to learn the correlation between the metadata, image capture settings, and editing information accumulated in the storage device, and to generate a learning model for estimating the image capture settings and editing information based on the metadata; and a transmission unit configured to transmit the generated learning model to the camera device.

[0309] (16) A camera device system, comprising:

[0310] The camera device unit includes a first learning model unit; and

[0311] A generative AI model unit is configured to generate a second image based on a first image captured by the camera device unit, wherein...

[0312] The first learning model is modified using a learning model that has already been learned based on the first image and the second image.

[0313] (17) A camera device including a machine learning model, the camera device comprising:

[0314] Control unit; and

[0315] The first learning model, in which...

[0316] The control unit modifies the first learning model based on a second learning model, which has been learned by comparing a first image captured by the camera device with a second image generated by the generative AI model unit based on the first image.

[0317] List of reference numerals

[0318] 10 System, 11 Camera Equipment, 12 Editing Device

[0319] 13 Model Providing Device

[0320] 401 Image capture function unit, 402 Image editing application

[0321] 403 Storage device, 404 Model learning unit

[0322] 407 Monitoring Function Unit, 408 Viewfinder

[0323] 1101 Model Learning Unit

[0324] 1201 Image capture optical system, 1202 Image capture unit, 1203 Image processing unit

[0325] 1204 Display unit, 1205 Storage unit, 1206 Operation unit

[0326] 1207 Control Unit, 1208 Sensor Unit, 1209 Bus

[0327] 1301 Developing Unit, 1302 Post-processing Unit

[0328] 1303 Detection and Processing Unit, 1304 Focusing Control Unit

[0329] 2000 Information Processing Device, 2001 CPU, 2002 ROM

[0330] 2003 RAM, 2004 Host Bus, 2005 Bridge

[0331] 2006 Expansion Bus, 2007 Interface Unit

[0332] 2008 Input Unit, 2009 Output Unit, 2010 Storage Unit

[0333] 2011 Drive, 2012 Removable Recording Media

[0334] 2013 Communication Unit Claims (as amended under Article 19 of the Treaty) 1. (Modified) An information processing apparatus, comprising: A collection unit is configured to collect metadata and image capture settings of the captured image, as well as editing information for the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device. A generation unit, configured to enable a model to learn the correlations between metadata accumulated in the storage device, image capture settings, and editing information, and to generate a learned model for estimating editing information based on the metadata; and A transmission unit configured to transmit the generated learning model to a camera device. 2. The information processing apparatus according to claim 1, wherein, The metadata includes at least one of the following: first information obtained by analyzing the features of the subject scene, and second information obtained by converting the state at the time of image capture into data. 3. (Modified) The information processing apparatus according to claim 2, wherein, The first information includes at least one of the following: frequency values ​​obtained by classifying the brightness values ​​of each pixel in the image into a predetermined number of levels; the ratio of the values ​​of the brightest and darkest pixels; the brightness level of the pixel at the focus position in the predetermined number of levels; frequency values ​​obtained by classifying the color of each pixel in the image into a predetermined number of colors; the color level of the pixel at the focus position in the predetermined number of levels; whether there is a moving object in the subject scene; and distance information of the subject. The second information includes at least one of the following: the name of the image capturing device, the name of the lens used, the date and time of image capture, the ambient light intensity value, and the tilt of the camera device (or camera device shake). 4. The information processing apparatus according to claim 1, wherein, The image capture settings include at least one of the following: aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type. 5. The information processing apparatus according to claim 1, wherein, The editing information includes editing settings, which include at least one of the following: brightness, color temperature adjustment, color correction, contrast, white level, black level, and chroma. 6. The information processing apparatus according to claim 1, wherein, The editing information includes information on the differences between the captured image and the edited image. 7. (Modified) The information processing apparatus according to claim 1, wherein, The generation unit enables the model to learn the correlation between metadata of image capture based on a specific user, image capture settings, and editing information based on the editing operations of the specific user, and generates a learning model that reflects the past image capture and image editing tendencies of the specific user. 8. (Revised) An information processing method, comprising: Collect metadata and image capture settings of the captured image, as well as editing information for the captured image, and accumulate the metadata, image capture settings, and editing information into a storage device; The model learns the correlations between metadata accumulated in the storage device, image capture settings, and editing information, and generates a learned model for estimating editing information based on the metadata; and The generated learning model is transmitted to the camera device. 9. (Amended) A computer program written in a computer-readable format, said computer program being used to cause a computer to function as: A collection unit is configured to collect metadata and image capture settings of the captured image, as well as editing information for the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device. A generation unit, configured to enable a model to learn the correlations between metadata accumulated in the storage device, image capture settings, and editing information, and to generate a learned model for estimating editing information based on the metadata; and A transmission unit configured to transmit the generated learning model to a camera device. 10. (Modified) An image capturing device, comprising: A generation unit, configured to generate metadata; An estimation unit, configured to estimate editing information based on metadata generated by the generation unit, using a learning model that has learned the correlation between metadata of the captured image, image capture settings, and editing information for the captured image; and A control unit configured to control the image capture operation based on the estimated editing information. 11. (Modified) The image capture apparatus according to claim 10, further comprising: A conversion unit is configured to convert the editing information into parameter values ​​that can be processed by the image capture device, and The control unit controls the image capture operation by using the parameter values. 12. (Modified) The image processing apparatus according to claim 11, further comprising: Display unit, wherein The control unit generates a monitoring image of the captured image using the parameter values, and displays the monitoring image on the display unit. 13. (Deleted) 14. (Deleted) 15. (Revised) A system comprising: The system includes a camera device, an editing device configured to edit captured images captured by the camera device, and a model providing device configured to provide a learning model to the camera device, wherein... The camera device outputs metadata of the captured image and the image capture settings of the camera device when capturing the captured image. The editing device outputs editing information for the captured image, and The model providing apparatus includes: a collection unit configured to collect metadata, image capture settings, and editing information and accumulate the metadata, image capture settings, and editing information in a storage device; a generation unit configured to enable a model to learn the correlation between the metadata, image capture settings, and editing information accumulated in the storage device and generate a learning model for estimating editing information based on the metadata; and a transmission unit configured to transmit the generated learning model to the camera device. 16. (Amended) A camera device system, comprising: The camera device unit includes a first learning model unit; and A generative AI model unit is configured to generate a second image based on a first image captured by the camera device unit, wherein... The first learning model is modified by using a learning model that has already learned the metadata of the first image, the image capture settings of the camera unit when capturing the first image, and the correlation between the difference information between the first image and the second image. 17. (Amended) A camera device including a machine learning model, the camera device comprising: Control unit; and The first learning model, in which... The control unit modifies the first learning model based on a second learning model, which has learned the metadata of the first image captured by the camera device, the image capture settings of the camera device when the first image was captured, and the correlation between the first image and the difference information between the first image and the second image generated by the generative AI model unit based on the first image. 18. (Additional) The information processing apparatus according to claim 1, wherein, The editing information accumulated in the storage device includes information generated by an editing device different from the camera device that captured the captured image. 19. (Additional) The information processing apparatus according to claim 1, wherein, The estimated editing information is information converted by the conversion unit in the camera device into parameter values ​​that can be processed in the camera device. 20. (Additional) The information processing apparatus according to claim 1, wherein, The estimated editing information is information used to generate monitoring images in the camera device. 21. (Additional) The information processing apparatus according to claim 1, wherein, The metadata includes a scaled-down image obtained by scaling down the captured image. 22. (Additional) The image capture apparatus according to claim 11, further comprising: Image capture unit and image processing unit, wherein, The control unit controls the image processing unit so that the image processing unit uses the parameter values ​​to process the image signal generated by the image capture unit.

Claims

1. An information processing apparatus, comprising: A collection unit is configured to collect metadata and image capture settings associated with the captured image by the camera device, as well as editing information during image editing of the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device. A generation unit is configured to enable a model to learn the correlation between metadata, image capture settings and editing information accumulated in the storage device, and to generate a learning model for estimating image capture settings and editing information based on the metadata. as well as A transmission unit configured to transmit the generated learning model to the camera device.

2. The information processing apparatus according to claim 1, wherein, The metadata includes at least one of the following: first information obtained by analyzing the characteristics of the subject scene, and second information obtained by converting the state at the time of image capture into data.

3. The information processing apparatus according to claim 2, wherein, The first information includes at least one of the following: frequency values ​​obtained by classifying the brightness values ​​of each pixel in the image into five levels; the ratio of the values ​​of the brightest and darkest pixels; the brightness level of the pixel at the focus position in the five-level classification; frequency values ​​obtained by classifying the color of each pixel in the image into five colors; the color level of the pixel at the focus position in the five-level classification; whether there is a moving object in the subject scene; and distance information of the subject. The second information includes at least one of the following: the name of the image capturing device, the name of the lens used, the date and time of image capture, the ambient light intensity value, and the tilt of the camera device (or camera device shake).

4. The information processing apparatus according to claim 1, wherein, The image capture settings include at least one of the following: aperture value, shutter speed, ISO value, exposure correction value, color temperature, and focus area type.

5. The information processing apparatus according to claim 1, wherein, The editing information includes editing settings, which include at least one of the following: brightness, color temperature adjustment, color correction, contrast, white level, black level, and chroma.

6. The information processing apparatus according to claim 1, wherein, The editing information includes information on the differences between the captured image and the edited image.

7. The information processing apparatus according to claim 1, wherein, The generation unit enables the model to learn the correlation between metadata, image capture settings, and editing information based on a specific user's image capture and editing operations, and to generate a learning model that reflects the specific user's past image capture and image editing tendencies.

8. An information processing method, comprising: The metadata and image capture settings at the time of image capture, as well as the editing information during image editing of the captured image, are collected in association with the captured image of the camera device, and the metadata, image capture settings, and editing information are accumulated in the storage device; The model learns the correlation between metadata, image capture settings, and editing information accumulated in the storage device, and generates a learning model for estimating image capture settings and editing information based on the metadata. as well as The generated learning model is transmitted to the camera device.

9. A computer program written in a computer-readable format, said computer program being used to cause a computer to function as: A collection unit is configured to collect metadata and image capture settings associated with the captured image by the camera device, as well as editing information during image editing of the captured image, and to accumulate the metadata, image capture settings, and editing information into a storage device. A generation unit is configured to enable a model to learn the correlation between metadata, image capture settings and editing information accumulated in the storage device, and to generate a learning model for estimating image capture settings and editing information based on the metadata. as well as A transmission unit configured to transmit the generated learning model to the camera device.

10. An image capturing device, comprising an image capturing functional unit, the image capturing device comprising: A generation unit, configured to analyze the subject scene and generate metadata; as well as An estimation unit is configured to estimate a setting value based on the metadata using a learning model that has learned the correlation between the metadata and the setting value. The image capture function unit is controlled by using the setting value to perform an image capture operation.

11. The image capture apparatus according to claim 10, wherein, The estimation unit uses the learning model to estimate image capture settings and editing information based on the metadata. The image capture device further includes a conversion unit configured to convert the editing information into parameter values ​​that can be processed by the image capture device. The image capture function unit is controlled to perform image capture operations by using the image capture settings and the parameter values.

12. The image processing apparatus according to claim 11, further comprising: A monitoring function unit, configured to generate a monitoring image from the image capture function unit, wherein... The monitoring function unit is controlled by using the image capture settings and the parameter values ​​so that the monitoring screen is displayed.

13. The image capture apparatus according to claim 10, wherein, The metadata and each of the settings values ​​are stored during image capture.

14. The image capture apparatus according to claim 10, wherein, The estimation unit uses the learning model generated by using metadata and image capture settings at the time of image capture, as well as editing information when editing the captured image.

15. A system comprising: A camera device, an editing device configured to edit images captured by the camera device, and a model providing device configured to provide a learning model to the camera device, wherein, The camera device outputs metadata related to image capture and image capture settings associated with the captured image. The editing device outputs editing information during image editing of the captured image, and The model providing apparatus includes: a collection unit configured to collect metadata, image capture settings, and editing information and accumulate the metadata, image capture settings, and editing information into a storage device; a generation unit configured to enable a model to learn the correlation between the metadata, image capture settings, and editing information accumulated in the storage device, and to generate a learning model for estimating the image capture settings and editing information based on the metadata; and a transmission unit configured to transmit the generated learning model to the camera device.

16. A camera device system, comprising: A camera device unit, the camera device unit including a first learning model unit; as well as A generative AI model unit is configured to generate a second image based on a first image captured by the camera device unit, wherein... The first learning model is modified using a learning model that has already been learned based on the first image and the second image.

17. A camera device including a machine learning model, the camera device comprising: Control unit; as well as The first learning model, in which... The control unit modifies the first learning model based on a second learning model, which has been learned by comparing a first image captured by the camera device with a second image generated by the generative AI model unit based on the first image.

Images