Image processing device, image processing method and program

Abstract

To provide an image processing apparatus that can obtain an image content on which the intention of acquisition of the content is more appropriately reflected.SOLUTION: An image processing apparatus has: content acquisition means that acquires a first image content; degree acquisition means that, with an element having fluctuation being a variation in a state from among elements constituting an image as a fluctuation element, acquires a degree of fluctuation of a fluctuation element of a content intention acquisition unit 105 first image content; intention acquisition means that acquires information indicating a user's intention of photographing; and creation means that uses a learned learning model to create a second image content different in the degree of fluctuation of the fluctuation element of the image content from the content intention acquisition unit 105 first image content. Here, the learning model creates a content intention acquisition unit 105 second image content in which the degree of fluctuation acquired in the content intention acquisition unit 105 first image content is a degree corresponding to information indicating a content intention acquisition unit 105 photographing intention.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] It is known that the act of photography has two aspects: "recording" the subject as image content and "expressing" what the photographer wants to communicate through the image content. When the act of photography emphasizes "expression" through image content, it is particularly important that the photographer's intention (hereinafter referred to as the photographer's intention to capture the content) is reflected in the content. However, in actual photography scenes, the facial expressions and movements of the subjects, and the relative positions of the subjects to one another, are often not in line with the photographer's intention, so the photographer has to wait until the subject's condition matches the photographer's intention to capture the content, and constantly concentrate so as not to miss the shot.

[0003] On the other hand, when "expression" through image content is emphasized, the necessity for the obtained image content to be "image content obtained by the photographer through the act of taking a photograph" is diminished. Patent Document 1 proposes a technology for generating aggregated content using captured images or video content to provide a rich retrospective experience, including atmosphere. Furthermore, a technology using a deep neural network model with a generative adversarial network (GAN) has been proposed as a technology for generating non-existent image content. Patent Document 2 proposes a technology for generating images with transformed gaze or facial direction using a trained GAN model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-51270 [Patent Document 2] Japanese Patent Application Publication No. 2019-148980 Summary of the Invention [Problem to be solved by the invention]

[0005] With the technology proposed in Patent Document 1, if the intention to acquire the content is not reflected in the original image or video content, the intention to acquire the content cannot be reflected in the aggregated content generated using the image, etc. Furthermore, the technology proposed in Patent Document 2 is a technology for generating an image in which the line of sight or face direction is converted, and does not take into consideration generating content that reflects the intention to acquire the content of the image content.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to realize a technology that makes it possible to obtain image content that more appropriately reflects the intention of the user to acquire the content. [Means for solving the problem]

[0007] In order to solve this problem, for example, an image processing device of the present invention has the following configuration: That is, it has a content acquisition means for acquiring first image content, a degree acquisition means for acquiring a fluctuation degree of a fluctuation element of the first image content, where an element having fluctuation, which is a variation in state, among elements constituting an image is defined as a fluctuation element, an intention acquisition means for acquiring information indicating a user's intention to take a photograph, and a generation means for generating, from the first image content, a second image content having a different fluctuation degree of the fluctuation element of the image content using a trained learning model, wherein the learning model generates the second image content such that the fluctuation degree acquired in the first image content corresponds to the information indicating the intention to take a photograph. [Effects of the Invention]

[0008] According to the present invention, it is possible to obtain image content that more appropriately reflects the intention of acquiring the content. [Brief explanation of the drawings]

[0009] [Figure 1A]FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus according to an embodiment; [Figure 1B] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment; [Figure 2] 1 is a diagram illustrating fluctuations of elements that constitute image content according to an embodiment; [Figure 3] 1 is a flowchart showing the operation of a learning process of a fluctuation model according to an embodiment; [Figure 4] 1 is a flowchart showing the operation of image content generation processing (reconstruction processing) according to an embodiment; [Figure 5] FIG. 10 is a diagram showing an example of generating fluctuation rules for image content according to an embodiment; [Figure 6] FIG. 10 is a diagram showing an example of generating image content according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following, an example of an image processing device will be described, in which a digital camera capable of generating image content is used. However, this embodiment is not limited to digital cameras and can also be applied to other devices capable of generating image content. These devices may include, for example, mobile phones including smartphones, game consoles, personal computers, tablet devices, other wearable information terminals, server devices, etc.

[0012] <Example of digital camera functional configuration> Fig. 1A is a diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing device that generates image content according to an embodiment. An example of the hardware configuration of the digital camera will be described later with reference to Fig. 1B. Note that part or all of the example functional configuration shown in Fig. 1A may be realized by, for example, a CPU 122 or a GPU 126 (described later) of the digital camera 100 executing a computer program.

[0013] The digital camera 100 includes, for example, an image content acquisition unit 101, a fluctuation element extraction unit 102, a fluctuation model generation unit 103, a fluctuation model database 104, and a content intention acquisition unit 105. The digital camera 100 also includes a fluctuation rule determination unit 106, an image content reconstruction unit 107, a display unit 108, and a user instruction acquisition unit 109.

[0014] First, the image content acquisition unit 101 performs an acquisition process of the image content. In this embodiment, the image content acquisition unit 101 may acquire not only the image content but also meta information for the image content. The meta information for the image content includes, for example, date and time information and acquisition location information when the image content was acquired.

[0015] The image content acquisition unit 101 controls the acquisition of image content by the imaging device 129 described below, and outputs the acquired image content to the fluctuation element extraction unit 102 and image content reconstruction unit 107 described below. The image content acquisition unit 101 may perform image processing such as arbitrary trimming and resizing on the image content to suit the output destination, and then output the image content after normalizing it.

[0016] Here, with reference to FIG. 2, "fluctuation" and "fluctuation elements" according to this embodiment will be described. FIG. 2 shows the "fluctuation" of elements constituting image content. In FIG. 2, the horizontal axis represents the time axis, and the vertical axis represents the magnitude of the degree of each element. 201, 202, and 203 in the figure indicate changes in the elements constituting the image content on the time axis. For example, 201 indicates changes in the "smile level" of the "expression" of the main subject on the time axis. 202 indicates changes in the "composition position" on the time axis, and 203 indicates changes in the "amount of cloudiness" of the "weather" on the time axis. In this embodiment, "fluctuation" refers to the variation in the state of the elements constituting an image. For example, the variation (change) in the state of a single element, such as the smile level, is described as "fluctuation." An element having "fluctuation" is called a "fluctuation element." The degree of variation in the state of the fluctuation element can be measured from the image content.

[0017] In the example shown in Figure 2, we will explain a case where the photographer's intention when taking an image (i.e., the intention to obtain image content) is to have a high ``smile level,'' the subject appear on the left side as the ``composition position,'' or there is little ``cloud cover.''

[0018] The timings when the fluctuation of the fluctuation elements is highest are when the "smile level" is 204, when the "composition position" is 205, and when the "amount of clouds" is 206. The image contents acquired at times 204, 205, and 206 are contents 207, 208, and 209, respectively.

[0019] The fluctuation element extraction unit 102 extracts fluctuation elements contained in the image content. For example, in an example where a person's facial expression is used as the fluctuation element, the fluctuation element extraction unit 102 detects the person's face in the image content and extracts the fluctuation element. When the fluctuation element extraction unit 102 detects the person's face, it further performs a process of acquiring the fluctuation degree for the person's facial expression. For example, by acquiring this degree, the fluctuation element extraction unit 102 quantifies the degree of smile, degree of joy, anger, sadness, happiness, degree of eye opening, degree of mouth opening, etc. Note that when acquiring the fluctuation degree, the fluctuation degree may be calculated from the image content, or the fluctuation degree corresponding to the image content may be acquired via a network.

[0020] Other fluctuation elements may include, for example, the posture of a person in the image content, the composition of the image content, the lighting in the image content, the weather in the image content, or the clothing of a subject in the image content. The degree of fluctuation of a person's posture may be calculated from at least one of, for example, the direction of the face, the direction of the body, or the amount of blur in the person's movements. The degree of fluctuation of the composition of the image content may be calculated from at least one of, for example, the positional relationship between subjects or the distance between subjects. The degree of fluctuation of lighting may be calculated from, for example, the position of a light source. The degree of fluctuation of weather may be calculated from at least one of, for example, the weather, cloud cover, etc. The degree of fluctuation of clothing may be calculated from at least one of, for example, the type and color of the clothing. The fluctuation element extraction unit 102 outputs the calculated degrees of fluctuation elements together with the image content to the fluctuation rule determination unit 106. Furthermore, the fluctuation element extraction unit 102 outputs the image content and the fluctuation degree of the fluctuation element to the fluctuation model generation unit 103 as learning data for a fluctuation model, which will be described later.

[0021] The fluctuation model generation unit 103 performs a process of learning a learning model for each fluctuation element (hereinafter referred to as a fluctuation model) using the image content obtained from the fluctuation element extraction unit 102 and the fluctuation degree of the extracted fluctuation element. A fluctuation model is generated for each fluctuation element and is trained to generate image content corresponding to a specified fluctuation degree. For example, a fluctuation model using a person's facial expression as a fluctuation element is trained to generate image content of the specified facial expression. Note that even for the same fluctuation element, multiple fluctuation models may be generated for each period such as one month, for each area where the user has stayed, or in response to an instruction from the user.

[0022] The fluctuation model may be configured with a known machine learning algorithm capable of generating images, such as a Generative Adversarial Network (GAN). A GAN is configured with two neural networks: a generator that generates image content, and a classifier that identifies whether the image content generated by the generator is a genuine image.

[0023] In the processing of the fluctuation model learning stage, the generator and the classifier share a loss function with each other, and the generator repeatedly updates their respective neural networks so that the loss function is minimized and the classifier maximizes it. As a result, the image content generated by the generator becomes natural-looking. Note that the configuration of the neural network in GAN and the learning algorithm are well-known technologies, and therefore will not be described in this embodiment. In this way, the data used in learning is associated with the learned fluctuation model and stored in the fluctuation model database 104. In other words, the image content included in the learning data and the degree of the fluctuation element of the image content are associated with information indicating the fluctuation element (corresponding to the model) and stored in the fluctuation model database 104.

[0024] The fluctuation model database 104 is stored in the HDD 125 described below, and stores the fluctuation model for each fluctuation element generated by the fluctuation model generating unit 103 and data used in learning.

[0025] In this embodiment, an example will be described in which the fluctuation model generation unit 103 and the fluctuation model database 104 are included within the digital camera 100. However, a configuration may be adopted in which a communication unit is provided within the digital camera 100, and the fluctuation model generation unit 103 and the fluctuation model database 104 are located on an external server or cloud. Alternatively, the fluctuation model generation unit 103 and the fluctuation model database 104 may be located both in the digital camera 100 and the external server, and these may be used depending on the application or purpose.

[0026] For example, the digital camera 100 may have a generation unit for a fluctuation model associated with a fluctuation element that is expected to be used frequently, such as the facial expression of the main subject, and a database. On the other hand, the external server may store a generation unit for a fluctuation model that is used less frequently, a fluctuation model in the middle of learning, and learning data. The external server or cloud service may also manage the update history of the fluctuation model.

[0027] The content intention acquisition unit 105 acquires the content acquisition intention that the photographer wants to express in the input image content, and outputs an identifier of the content acquisition intention indicating the content acquisition intention to the fluctuation rule determination unit 106.

[0028] In this embodiment, for example, a relationship between fluctuation elements included in image content and content acquisition intention identifiers is defined in advance, and the fluctuation elements included in acquired image content are converted into content acquisition intention identifiers. That is, the content intention acquisition unit 105 can acquire the content acquisition intention identifier based on image information of the image content. The content acquisition intention identifier includes keywords used for tagging general image content, such as "fun" and "souvenir photo." Furthermore, the content intention acquisition unit 105 may accept an instruction or selection regarding the content acquisition intention identifier from the user. Furthermore, the content intention acquisition unit 105 may estimate information about the content acquisition intention identifier from a user behavior history, such as a history of operations performed to acquire image content and the number of photo attempts.

[0029] The content intention acquisition unit 105 may further use sound information to output a content acquisition intention identifier. For example, the content intention acquisition unit 105 can convert sound information of the shooting space, including the voice of the photographer, into a content acquisition intention identifier by using surrounding sound information at the time of content acquisition.

[0030] The fluctuation rule determination unit 106 calculates a fluctuation degree change amount (hereinafter referred to as a fluctuation rule) for each fluctuation element of the image content to be reconstructed and its degree, using the identifier of the content acquisition intention described above. The fluctuation rule determination unit 106 also specifies a fluctuation model to be used in the image content reconstruction unit 107 described below. Details of the processing by the fluctuation rule determination unit 106 will be described later.

[0031] The image content reconstructing unit 107 reads out a fluctuation model from the fluctuation model database 104 in accordance with the rule (fluctuation degree change amount of the fluctuation element) determined by the fluctuation rule determining unit 106. Then, the image content reconstructing unit 107 reconstructs the image content by inputting the image content to be reconstructed and the parameters for reconstruction into the fluctuation model. Details of the reconstruction of the image content will be described later. The image content reconstructing unit 107 outputs the reconstructed image content to the display unit 108.

[0032] The display unit 108 displays various image contents on the display device 128. In this embodiment, the display unit 108 displays at least the image contents acquired by the image content acquisition unit 101 or the image contents reconstructed by the image content reconstruction unit 107 on the display device 128.

[0033] The user instruction acquisition unit 109 receives various instructions from the user regarding the reconstruction of image content via the input device 127, and prompts each processing unit of the digital camera 100 to perform predetermined processing. For example, the user instruction acquisition unit 109 receives instructions from the user to acquire image content or instructions to reconstruct the image content. In addition, the user instruction acquisition unit 109 may also receive specifications of parameters required for the reconstruction of image content, such as an identifier of the content acquisition intention or a fluctuation model.

[0034] <Example of hardware configuration for a digital camera> 1B, an example of the hardware configuration of digital camera 100 will be described. Digital camera 100 includes, for example, a system bus 121, a CPU 122, a ROM 123, a RAM 124, an HDD 125, a GPU 126, an input device 127, a display device 128, and an imaging device 129. Each unit of digital camera 100 is connected to system bus 121.

[0035] The CPU 122 is an arithmetic circuit such as a CPU (Central Processing Unit) that implements various functions of the digital camera 100 by loading computer programs stored in the ROM 123 or HDD 125 onto the RAM 124 and executing them. The ROM 123 includes a non-volatile storage medium such as a semiconductor memory and stores, for example, programs executed by the CPU 122 and necessary data. The RAM 124 includes a volatile storage medium such as a semiconductor memory and temporarily stores, for example, calculation results of the CPU 122. The HDD 125 includes a hard disk drive and stores, for example, computer programs executed by the CPU 122 and the processing results thereof. While this example illustrates a case in which the digital camera 100 includes a hard disk, the digital camera 100 may include a storage medium such as an SSD instead of a hard disk. The GPU (Graphics Processing Unit) 126 includes an arithmetic circuit and may execute, for example, part or all of the learning stage processing and inference stage processing of a learning model. Compared to a CPU, a GPU can process more data in parallel, making it effective to use a GPU for deep learning processing, which involves repeated calculations using the above-mentioned neural network.

[0036] The input device 127 includes operating members such as buttons and a touch panel that accept operational inputs to the digital camera 100. The display device 128 includes a display panel such as an OLED. The imaging device 129 includes an optical system unit such as a lens, aperture, and shutter, and an imaging element such as a CMOS sensor. The optical system unit may be configured with a compound lens or multiple lenses. The optical system unit may also be capable of changing optical characteristics such as zoom and aperture (for example, depending on the image content to be acquired).

[0037] <Learning process of fluctuation model> The fluctuation model learning process performed by the fluctuation model generation unit 103 and the like will be described with reference to Fig. 3. This process can be realized by the units shown in Fig. 1A, which are realized by the CPU 122 or GPU 126 of the digital camera 100 executing a computer program, for example. This process can basically be executed when a shooting instruction is received from the user, or at any time before or after that. However, even when a shooting instruction is not received from the user, for example, if the image content acquisition unit 101 is constantly running and can capture the photographer's surrounding environment, this process may be executed at regular intervals.

[0038] In S301, the image content acquisition unit 101 acquires image content for learning via the imaging device 129. For example, the acquired image content for learning is still image data. Alternatively, the image content acquisition unit 101 may extract still image data from video content. The image content acquisition unit 101 outputs the acquired still image data to the fluctuation element extraction unit 102. Note that the acquired image content is not limited to that output from the imaging device 129, and image content acquired in advance and stored in the HDD 125 may also be used. The image content for learning may be limited to image content acquired during a specific period or at a specific location. For example, the image content for learning may be image content acquired between a predetermined start instruction and a predetermined end instruction by the user as a shooting period or a learning data collection period. Alternatively, the image content for learning may be acquired according to the image content to be reconstructed. The image content for learning may be image content acquired during a specific period before or after the acquisition date and time of the image content to be reconstructed. Alternatively, the image content for learning may be image content acquired within a predetermined range around the acquisition position of the image content to be processed for reconstruction.

[0039] In S302, the fluctuation element extraction unit 102 extracts predetermined fluctuation elements from the input still image data and calculates (acquires) the fluctuation degree (score) for the extracted fluctuation element. The image content acquisition unit 101 normalizes the still image data in a region including the extracted fluctuation element, and outputs the normalized data together with fluctuation degree information to the fluctuation model generation unit 103 (as learning data for the fluctuation model).

[0040] In this explanation, it is assumed that this process is performed for each fluctuation element of one still image data. However, the extraction frequency of the fluctuation element may be determined for each fluctuation element. For example, an element with drastic fluctuation changes may be extracted more frequently, and an element with gradual change may be extracted less frequently.

[0041] In S303, the fluctuation model generation unit 103 reads out fluctuation model information of the learning target from the fluctuation model database 104, and performs machine learning processing of the fluctuation model using the input learning data. The machine learning processing of the fluctuation model is, for example, the processing in the learning stage of the GAN described above. Then, the fluctuation model generation unit 103 updates the fluctuation model information in the fluctuation model database 104 together with the data used for learning. Note that if the fluctuation model of the learning target does not exist in the fluctuation model database 104, a new fluctuation model is added.

[0042] Through the above process, the fluctuations of the fluctuation elements in the image content acquired by the user or obtained through user experience are used as learning data for each fluctuation element model. This makes it possible to construct a neural network for the GAN generator that can tune the fluctuation of the fluctuation elements (i.e., can generate images according to the specified degree of fluctuation).

[0043] <Reconstruction process operation> Next, with reference to FIG. 4, the reconstruction process of image content using the fluctuation element model will be described. Note that this process can be realized by the respective units shown in FIG. 1A, which are realized by, for example, the CPU 122 or GPU 126 of the digital camera 100 executing a computer program. Note that this process is started in response to receiving an instruction from a user. To start the process, it is sufficient that one image content to be reconstructed is selected, and the timing of the instruction may be arbitrary. In this embodiment, the description will be given assuming that the image content 208 in FIG. 2 has been selected. For example, the process may be started in response to a user instruction to acquire the image content. Alternatively, the reconstruction instruction may be received while a recorded image is being displayed after the image content has been acquired, or while the image content is being played back.

[0044] In S401, the image content acquisition unit 101 acquires image content to be reconstructed. Here, for example, a case where the image content 208 is the image content to be reconstructed will be described as an example.

[0045] In S402, the fluctuation element extraction unit 102 receives the image content to be reconstructed from the image content acquisition unit 101, extracts the fluctuation elements contained in the image content, and calculates (acquires) the degree of the fluctuation elements. The operation of the fluctuation element extraction unit 102 is the same as the processing in the learning processing.

[0046] In S403, the content intention acquisition unit 105 acquires an identifier of the intention to acquire the content from an arbitrary information group associated with the image content. For example, an identifier of the intention to acquire the content, such as "travel," "souvenir photo," or "fun," is acquired from people and their expressions reflected in the image content 208 and background objects, and associates the identifier with the image content.

[0047] The content intention acquisition unit 105 may acquire the content acquisition intention identifier based on additional information other than the image content. For example, if the digital camera 100 is equipped with voice recognition technology, the content intention acquisition unit 105 may use the results of the voice recognition to acquire the content acquisition intention identifier. For example, the content intention acquisition unit 105 may acquire the content acquisition intention identifier based on user utterance information recorded during a predetermined period before and after the image content was captured, or user utterance information input during a predetermined period after the image content was played. Specifically, if the content intention acquisition unit 105 recognizes user utterances such as "It's cloudy," "I can't see because of the clouds," or "I wish it was sunny" when acquiring the image content 208 or issuing a reconstruction instruction, the unit may use "weather" or "sunny," which is considered an ideal weather condition, as a keyword. In this case, the keyword is associated with the image content as a content acquisition intention identifier.

[0048] In addition to the above examples, the identifier of the content acquisition intention may be predicted and calculated from the user's operation history information, behavior history information, and text information entered by the user before and after the act of photographing the image content 208 selected in S401.

[0049] Thereafter, the content intention acquisition unit 105 associates the content acquisition intention identifier with the image content 208 and outputs it to the fluctuation rule determination unit 106 .

[0050] In S404, the fluctuation rule determination unit 106 determines a fluctuation rule that will serve as control information for the image content reconstruction unit 107 using the image content to be reconstructed, the fluctuation element information associated with the image content, and the content acquisition intention identifier.

[0051] A method for creating a fluctuation rule according to this embodiment will be described with reference to Fig. 5. Fig. 5 shows the relationship between the degree of fluctuation of the fluctuation elements of image content to be reconstructed and various types of information.

[0052] The fluctuation rule determination unit 106 selects and reads out fluctuation model information related to the fluctuation elements of the image content 208 to be reconstructed from the fluctuation model database 104. Note that the read out fluctuation model information is information on a fluctuation model trained using training data, and the training data includes at least the image content including the fluctuation elements to be reconstructed.

[0053] The fluctuation rule determination unit 106 uses the read fluctuation model information and the related training data group to calculate information on the fluctuation range that can be reconstructed in the fluctuation model. For example, an example distribution of training data for a fluctuation model related to smiles is shown in FIG. 5(a). In the training of the above-mentioned GAN, the system is trained to be able to generate images with the degree of fluctuation contained in the training data. Therefore, from the distribution of the degree of smiles in the training data shown in FIG. 5(a), it can be understood that the fluctuation range of image content that can be reconstructed by specifying the degree of the fluctuation element is a range from degree 1 to 6.

[0054] Next, the fluctuation rule determination unit 106 calculates a recommended value for the degree of fluctuation of the fluctuation element after reconfiguration from the content acquisition intention identifier. In this embodiment, for example, the digital camera 100 stores in advance information associating the above-mentioned content acquisition intention identifier with the ideal degree of fluctuation of the fluctuation element as conversion table information between intention and ideal degree of fluctuation. The fluctuation rule determination unit 106 calculates the degree of fluctuation of the fluctuation element after reconfiguration by referring to the conversion table information.

[0055] For example, the conversion table for the content acquisition intention identifier "fun" associates the fluctuation elements of "facial expression" and "composition" as shown in Fig. 5(b). In this example, the ideal fluctuation degree of the fluctuation element of "facial expression" is associated so that the degree of smiling in "facial expression" is 7, which is the maximum value.

[0056] The fluctuation rule determination unit 106 determines a fluctuation model to be used and calculates parameters to be set for the determined fluctuation model. The set parameters are calculated so that they fall within the above-mentioned fluctuation range in which reconstruction is possible and approach the ideal degree of fluctuation of the fluctuation element according to the content acquisition intention. For example, the fluctuation rule determination unit 106 first determines whether the ideal degree of fluctuation corresponding to the shooting intention corresponds to a degree that can be set for reconstruction among the fluctuation degrees (whether it is a degree from 1 to 6 in the above example). If the ideal degree of fluctuation corresponds to a degree that can be set for reconstruction among the fluctuation degrees, the fluctuation rule determination unit 106 sets the ideal degree of fluctuation as the degree to be set for reconstruction. If the ideal degree of fluctuation does not correspond to a degree that can be set for reconstruction among the fluctuation degrees, the fluctuation rule determination unit 106 sets the degree that is closest to the ideal degree among the degrees that can be set for reconstruction as the degree to be set for reconstruction. That is, the adjusted degree according to the ideal degree of fluctuation is set for reconstruction. For example, as shown in Fig. 5(c), the parameter set in the fluctuation model of the fluctuation element "facial expression" is that the ideal degree of fluctuation is 7, while the upper limit of the reconstructible range of the fluctuation model is 6. Therefore, the set value is 6.

[0057] Furthermore, the fluctuation rule determination unit 106 determines the order of reconstruction processing using multiple fluctuation models. The processing order of the fluctuation models here is arbitrary and may be determined based on various factors. In this embodiment, for example, the processing is performed in the order from the fluctuation model with the largest difference between the recommended value of the fluctuation degree and the degree of fluctuation in the image content to be reconstructed to the fluctuation model with the smallest difference. In this case, for example, as shown in FIG. 5(d), the reconstruction processing of the fluctuation models is performed in the order of first the fluctuation model for "facial expression," then the fluctuation model for "amount of clouds," and finally the fluctuation model for "composition."

[0058] In this way, the fluctuation rule determination unit 106 outputs the fluctuation model information, the parameter information to be passed to the fluctuation model, and the reconstruction processing order information of the fluctuation model to the image content reconstruction unit 107 as fluctuation rules.

[0059] In S405, the image content reconstructing unit 107 executes reconstruction processing using the image content to be reconstructed and the fluctuation rule determined by the fluctuation rule determining unit 106. For example, as a result of the reconstruction processing, an image as shown in Fig. 6 is generated. The reconstructed image shown in Fig. 6 is new image content that maintains the atmosphere of the image content 208 to be reconstructed, does not change significantly in "composition", has a large degree of smiling in "expression", and has a small degree of "cloudiness".

[0060] The generated image may be displayed on the display unit 108 to prompt the user to confirm it and to receive feedback on the reconstruction process. For example, if the user issues an instruction to record the reconstructed image content, positive feedback may be given to the fluctuation model along with the recording process, and if not, negative feedback may be given to perform a new reconstruction process.

[0061] As described above, in this embodiment, the fluctuation degree of the fluctuation element of the acquired image content and information indicating the user's intention to capture the image are acquired, and image content with different fluctuation degrees is generated from the acquired image content using a trained learning model. At this time, the learning model generates image content in which the fluctuation degree acquired in the acquired image content corresponds to the information indicating the intention to capture the image content. In this way, it is possible to obtain image content that more appropriately reflects the intention to capture the content.

[0062] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0063] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0064] 101...image content acquisition unit, 102...fluctuation element extraction unit, 103...fluctuation model generation unit, 105...content intention acquisition unit, 106...fluctuation rule determination unit, 107...image content reconstruction unit

Claims

1. a content acquisition means for acquiring a first image content; a degree acquiring means for acquiring a degree of fluctuation of the fluctuation element of the first image content, the fluctuation element being a fluctuation that is a variation in state among elements constituting the image; an intention acquisition means for acquiring information indicating a user's intention to take a photograph; generating means for generating, from the first image content, second image content having a different degree of fluctuation of the fluctuation element of the image content by using a trained learning model; An image processing device characterized in that the learning model generates the second image content in which the degree of fluctuation obtained in the first image content corresponds to the information indicating the shooting intention.

2. 2. The image processing device according to claim 1, wherein the intention acquisition means acquires information indicating the photographing intention based on image information of the first image content or based on information associated with the first image content, which is at least one of text information, operation history information, behavior history information, and sound information input by the user.

3. 3. The image processing device according to claim 2, wherein the intention acquisition means acquires information indicating the shooting intention based on user speech information during a predetermined period before and after the first image content is shot or during a predetermined period after the first image content is played back.

4. The method further includes a determination unit that determines whether the information indicating the photographing intention corresponds to a settable degree of fluctuation of the fluctuation element, An image processing device as described in any one of claims 1 to 3, characterized in that when the information indicating the shooting intention corresponds to a settable degree of fluctuation of the fluctuation element, the learning model generates the second image content in which the degree of fluctuation of the fluctuation element extracted in the first image content corresponds to the degree of fluctuation of the information indicating the shooting intention.

5. The image processing device described in claim 4, characterized in that when the information indicating the shooting intention does not correspond to a settable degree of fluctuation of the fluctuation element, the learning model generates the second image content in which the degree of fluctuation of the fluctuation element extracted in the first image content is adjusted in accordance with the information indicating the shooting intention.

6. 6. The image processing device according to claim 5, wherein the adjusted degree is a degree of fluctuation that is closest to a degree corresponding to the information indicating the photographing intention, among settable degrees of the fluctuation element.

7. 7. The image processing device according to claim 4, wherein the determination means determines whether the information indicating the photographing intention corresponds to the settable degree based on a correspondence between a distribution of fluctuation degrees of each of a plurality of image contents used as learning data for training the learning model and the information indicating the photographing intention.

8. The image processing device according to any one of claims 1 to 7, characterized in that the learning model is trained to generate image content from input image content, with the degree of fluctuation of the image content set to a specified degree of fluctuation, using learning data including photographed image content and the degree of fluctuation of the fluctuation elements of the photographed image content.

9. further comprising an imaging means for capturing an image of the image content; 9. The image processing device according to claim 1, wherein the learning data for the learning model is data configured to include image content captured by the imaging means and the degree of fluctuation of fluctuation elements of the captured image content.

10. The plurality of image contents used as training data for the training model include: Image content acquired between a predetermined start instruction and an end instruction by a user; Image content acquired during a predetermined period before and after the acquisition date and time of the image content to be processed; and image content acquired within a predetermined range around the acquisition position of the image content to be processed.

11. 11. The image processing device according to claim 1, wherein the fluctuation elements of the image content include at least one of the facial expression or posture of a person in the image content, the composition of the image content, the weather perceived in the image content, and the clothing of the subject perceived in the image content.

12. An image processing method executed in an image processing device, a content acquisition step of acquiring a first image content; a degree acquisition step of acquiring a degree of fluctuation of the fluctuation element of the first image content, the fluctuation element being an element having fluctuation, which is a variation in state, among elements constituting the image; an intention acquisition step of acquiring information indicating a user's intention to take a photograph; a generating step of generating, from the first image content, second image content having a different degree of fluctuation of the fluctuation element of the image content, using a trained learning model; An image processing method characterized in that the learning model generates the second image content in which the degree of fluctuation obtained in the first image content corresponds to the information indicating the shooting intention.

13. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images