Image processing apparatus and method, program, and storage medium

Abstract

The present invention comprises: an image acquisition means for acquiring a first image; a fluctuation degree acquisition means for acquiring a fluctuation degree of a fluctuation element having a fluctuation as a change in state among the elements constituting the first image; a generation means for generating, using the first image, a second image in which the degree of fluctuation of the fluctuation element is different from that of the first image, using the trained learning model; and a display control means for displaying an image on the display means, in which the generation means also generates a third image in which the degree of fluctuation of the fluctuation element is smaller than that of the second image using the first image in a case where a deviation of the first image from the second image is greater than or equal to a predetermined deviation, and displays the image on the display means. And a display control section performing control to display the first image and then display the third image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and method, a computer program, and a storage medium. Background Art

[0002] In recent years, as technologies related to image generation, new technologies related to image processing using AI technologies and advanced computational processing have been proposed. This includes a great deal of research on technologies related to generating non-existent images using generative adversarial networks (GANs), which are a type of unsupervised learning, as well as many related published papers and invention proposals. Against this background, it has become possible to use image processing technologies represented by GANs to process an image obtained by shooting (hereinafter referred to as a "captured image") and reconstruct the image to reflect the intention of the user who captured the image.

[0003] On the other hand, recent imaging devices including digital cameras and smartphones are generally provided with a display unit for displaying images. In a normal shooting operation, a live view image is displayed during shooting, and after the development of the captured image is completed, the user can switch to a preview display of the captured image that has been developed (hereinafter referred to as a "recorded image") and visually confirm whether the recorded image is as intended by the user.

[0004] Similarly, when reconstructing a recorded image to generate a new image, the user can visually confirm whether the reconstructed image is as intended by the user by previewing the image obtained by reconstruction (hereinafter referred to as a "reconstructed image") on the display unit. However, depending on the shooting scene, there may be a large deviation between the recorded image and the reconstructed image, and in the case where the reconstructed image is generated and previewed immediately after shooting, the user may feel a strong sense of discomfort due to the difference from the scene in front of the user's eyes, or it may take time to confirm.

[0005] Examples of the deviation between the live view image and the recorded image include cases where there is a difference between images due to a deviation in the shooting timing and the recording timing caused by a shutter release delay or the like. In response, Patent Document 1 discloses a technique according to which, before previewing the recorded image, a live view image or an image obtained by performing smoothing filtering processing on the recorded image is displayed to reduce the sense of discomfort felt during preview display caused by the timing deviation between shooting and recording.

[0006] In addition, Patent Document 2 discloses that when sequentially displaying a plurality of recorded images captured with a single shooting instruction, a plurality of processed images are generated and sequentially displayed, and the composite ratio of the next recorded image to be displayed gradually increases with respect to the currently displayed recorded image. Thereby, the sense of discomfort when switching to the preview display of the recorded image can be reduced.

[0007] Citation List

[0008] Patent Document

[0009] Patent Document 1: Japanese Patent Application Laid-Open No. 2005-204210

[0010] Patent Document 2: Japanese Patent Application Laid-Open No. 2014-127966 Summary of the Invention

[0011] Problems to be Solved by the Invention

[0012] However, there is a problem that when a reconstructed image is displayed, the image before reconstruction and the image after reconstruction may be very different. Therefore, simply displaying the processed image by the methods disclosed in Patent Document 1 and Patent Document 2 is not sufficient to reduce the sense of incongruity felt during preview display.

[0013] The present invention has been made in view of the above problems, and an object of the present invention is to reduce the sense of incongruity felt during preview display of a reconstructed image when the reconstructed image is generated.

[0014] Solutions to the Problems

[0015] To achieve the above object, an image processing apparatus according to the present invention includes: an image acquisition unit configured to acquire a first image; a fluctuation degree acquisition unit configured to acquire a fluctuation degree of a fluctuating element having fluctuations among elements constituting the first image, the fluctuation being a change in state; a generation unit configured to generate a second image having a different fluctuation degree of the fluctuating element compared to the first image by using a trained learning model and the first image; and a display control unit configured to display an image on a display unit, wherein when a deviation between the first image and the second image is greater than or equal to a predetermined deviation, the generation unit further generates a third image having a smaller fluctuation degree of the fluctuating element compared to the second image by using the first image, and the display control unit performs control to display the first image and then display the third image.

[0016] Effects of the Invention

[0017] According to the present invention, when a reconstructed image is generated, the sense of incongruity felt during preview display of the reconstructed image can be reduced.

[0018] Other features and advantages of the present invention will become apparent from the following description in conjunction with the accompanying drawings. Note that in the entire drawings, the same reference numerals denote the same or similar components. Brief Description of the Drawings

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0020] Figure 1A is a block diagram showing an example functional configuration of an image processing apparatus according to an embodiment of the present invention.

[0021] Figure 1B is a block diagram showing an example hardware configuration of an image processing apparatus according to an embodiment.

[0022] Figure 2 is a diagram illustrating the fluctuations of elements constituting an image according to an embodiment.

[0023] Figure 3 is a flowchart showing a fluctuation model training process according to an embodiment.

[0024] Figure 4 is a flowchart showing an image reconstruction process according to an embodiment.

[0025] Figure 5A is a diagram showing an example of image fluctuation rule generation according to an embodiment.

[0026] Figure 5B is a diagram showing an example of image fluctuation rule generation according to an embodiment.

[0027] Figure 5C is a diagram showing an example of image fluctuation rule generation according to an embodiment.

[0028] Figure 5D is a diagram showing an example of image fluctuation rule generation according to an embodiment.

[0029] Figure 6 is a diagram showing an example of image generation according to an embodiment.

[0030] Figure 7 is a flowchart showing the operations of an image generation and display process according to the first embodiment.

[0031] Figure 8 is a diagram showing the flow of an image generation process according to the first embodiment.

[0032] Figure 9 is a diagram showing an example image generation according to the first embodiment.

[0033] Figure 10A is a diagram showing the flow of a different image generation process according to the first embodiment.

[0034] Figure 10B is a diagram showing the flow of a different image generation process according to the first embodiment.

[0035] Figure 11It is a flowchart showing the operations of image generation and display processing according to the second embodiment.

[0036] Figure 12 It is a diagram showing an example image generation according to the second embodiment.

[0037] Figure 13 It is a flowchart showing the operations of image generation and display processing according to the third embodiment.

[0038] Figure 14 It is a diagram showing the process of image generation processing according to the third embodiment.

[0039] Figure 15 It is a diagram showing an example image generation according to the third embodiment. Detailed Description of the Embodiment

[0040] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but the invention is not limited to requiring all of these features, and multiple of these features can be appropriately combined. In addition, in the drawings, the same or similar configurations are given the same reference numerals, and their repeated descriptions are omitted.

[0041] <First Embodiment>

[0042] Hereinafter, a digital camera capable of generating an image will be described as an example of an image processing apparatus according to the first embodiment. Note that this embodiment is not limited to a digital camera, and can also be applied to other devices capable of generating an image. Examples of these devices include mobile phones such as smart phones, game machines, personal computers, tablet terminals, wearable information terminals, and server devices.

[0043] ● Example Configuration of Digital Camera

[0044] Figure 1A It is a diagram showing an example functional configuration of a digital camera 100 as an example of an image processing apparatus in the embodiment, and Figure 1B is a diagram showing Figure 1A an example hardware configuration of the digital camera shown. Figure 1A Part or all of the functional configuration of the digital camera 100 shown can be implemented by, for example, Figure 1B a computer program executed by the CPU 122 or GPU 126 shown.

[0045] As Figure 1AAs shown, the digital camera 100 includes an image acquisition unit 101, a fluctuation factor extraction unit 102, a fluctuation model generation unit 103, a fluctuation model database 104, and a shooting intention acquisition unit 105. In addition, the digital camera 100 includes a fluctuation rule determination unit 106, an image reconstruction unit 107, a display control unit 108, a user instruction acquisition unit 109, an image difference calculation unit 110, and a recording unit 111.

[0046] In addition, as Figure 1B shown, the hardware configuration of the digital camera 100 includes a system bus 121, a CPU 122, a ROM 123, a RAM 124, an HDD 125, a GPU 126, an input device 127, a display device 128, and an imaging device 129. These components are connected to the system bus 121.

[0047] The CPU 122 is a computing circuit such as a CPU (central processing unit), and realizes the functions of the digital camera 100 by extracting the computer programs stored in the ROM 123 or the HDD 125 into the RAM 124 and executing these computer programs. The ROM 123 includes, for example, a non-volatile storage medium such as a semiconductor memory, and stores the programs and required data executed by the CPU 122. The RAM 124 includes, for example, a volatile storage medium such as a semiconductor memory, and temporarily stores, for example, the calculation results of the CPU 122.

[0048] The HDD 125 includes a hard disk drive, and stores, for example, the computer programs executed by the CPU 122 and their processing results, etc. In addition, the HDD 125 (recording medium) stores the images recorded by the recording unit 111. Note that in this example, the digital camera 100 is described as having a hard disk, but the digital camera 100 may also have a storage medium such as an SSD instead of a hard disk.

[0049] The GPU (Graphics Processing Unit) 126 includes a computing circuit, and can execute, for example, part or all of the processing in the learning model training stage and the inference stage. Since the GPU can process more data in parallel than the CPU, it is effective to use the GPU for processing in deep learning processing that performs repetitive operations using neural networks.

[0050] The input device 127 includes operation members such as buttons and touch panels that receive operation inputs for the digital camera 100. The display device 128 includes, for example, a display panel such as an OLED. The imaging device 129 includes, for example, an optical system unit such as a lens, a diaphragm, and a shutter, and an image sensor such as a CMOS sensor. The optical system unit may further include a compound eye lens or a multi-eye lens. Additionally, the optical unit may be capable of changing optical characteristics such as zoom and aperture according to, for example, the acquired image.

[0051] In the digital camera 100 having the above configuration, first, the image acquisition unit 101 performs image acquisition processing. Note that, in the present embodiment, the image acquisition unit 101 can not only acquire an image but also acquire meta information for the image. The meta information for the image includes, for example, date and time information of acquiring the image and acquisition location information. The image acquisition unit 101 controls the imaging device 129 to acquire an image and outputs the acquired image to the fluctuation factor extraction unit 102, the shooting intention acquisition unit 105, the image reconstruction unit 107, the display control unit 108, and the image difference calculation unit 110. Note that the image acquisition unit 101 may output the image after standardizing the acquired image by performing optional image processing such as cropping or resizing according to the output destination.

[0052] Here, reference will be made to Figure 2 describe "fluctuation" and "fluctuation factor" according to the present embodiment.

[0053] Figure 2 denote the "fluctuation" of the elements constituting the image. In Figure 2 the horizontal axis represents time, and the vertical axis represents the magnitude of the degree of each element. The bar graphs 201, 202, and 203 show the changes of the elements constituting the image over time. For example, the bar graph 201 shows the change of the "smile degree" in the "facial expression" of the main subject over time. The bar graph 202 shows the change of the "position" in the "composition" over time, and the bar graph 203 shows the change of the "sunny degree" in the "weather" over time.

[0054] In the present embodiment, the change in the state of the elements constituting the image is called "fluctuation". For example, the change (variation) in the state of one element such as the smile degree is described as "fluctuation". Note that the element having "fluctuation" is called a "fluctuation factor". Additionally, "fluctuation", that is, the change in the state, can be detected by measuring the degree of the state in a plurality of acquired images.

[0055] Here, a case where the intention of the user to capture an image (the intention to obtain an image) is one of a high "smile degree" of the "facial expression", the subject appears on the left side of the "position" as the "composition", and a high "sunny degree" of the "climate" will be described as an example.

[0056] In Figure 2 the example shown, the timing when the fluctuation of each fluctuation factor is closest to the shooting intention is the timing 204 with the highest "smile degree", the timing 205 with the highest "position" (the subject is on the left), and the timing 206 with the highest "sunny degree". In addition, the images obtained at the timings 204, 205, and 206 are given as images 207, 208, and 209, respectively.

[0057] Returning to Figure 1A and Figure 1B the description, the fluctuation factor extraction unit 102 extracts the fluctuation factors included in the image. For example, in an example where the facial expression of a person is a fluctuation factor, the fluctuation factor extraction unit 102 performs detection of the face of the person in the image and extracts the fluctuation factors. In addition, when the face of a person is detected, the fluctuation factor extraction unit 102 performs processing for obtaining the degree of fluctuation of the expression of the person. For example, the fluctuation factor extraction unit 102 obtains the degree of fluctuation and quantifies the smile degree, the emotion degree, the eye opening degree, the mouth opening degree, etc. Note that when obtaining the degree of fluctuation, the degree of fluctuation can be calculated based on the image, or the degree of fluctuation corresponding to the image can be obtained via the network.

[0058] In addition, other fluctuation factors may include, for example, the posture of the person in the image, the composition of the image, the illumination in the image, the climate in the image, and the clothing of the subject in the image. The posture of a person includes, for example, at least one of the orientation of the face, the orientation of the body, and the amount of blur of the movement of the person. In addition, the composition of the image includes, for example, at least one of the positional relationship of the subjects and the distance between the subjects. The illumination in the image includes, for example, the position of the light source. The climate in the image includes, for example, at least one of the weather and the cloud amount. The clothing in the image includes, for example, at least one of the type and color of the clothing.

[0059] The fluctuation factor extraction unit 102 outputs the degree of fluctuation of the calculated fluctuation factors together with the image to the fluctuation rule determination unit 106. In addition, the fluctuation factor extraction unit 102 outputs the image and the degree of fluctuation of the fluctuation factors to the fluctuation model generation unit 103 as learning data of the fluctuation model described later.

[0060] The fluctuation model generation unit 103 uses the image obtained from the fluctuation factor extraction unit 102 and the degree of fluctuation of the extracted fluctuation factors to perform processing for training a learning model (hereinafter referred to as "fluctuation model") for each fluctuation factor. A fluctuation model is generated for each fluctuation factor, and the fluctuation model is trained to generate an image corresponding to a specified degree of fluctuation. For example, a fluctuation model in which the fluctuation factor is a human facial expression is trained to generate an image of a specified facial expression. Note that even for the same fluctuation factor, multiple fluctuation models can be generated for each area the user has visited and for each time period (such as one month) according to a user instruction or the like.

[0061] In addition, the fluctuation model can be generated, for example, using a known machine learning algorithm (such as GAN, etc.) capable of generating images. GAN consists of two neural networks, namely a generator that generates images and a discriminator that discriminates whether the images generated by the generator are real images. In the processing in the training stage of the fluctuation model, the above-mentioned generator and discriminator share a loss function with each other, and repeatedly update their respective neural networks so that the generator minimizes the loss function and the discriminator maximizes the loss function. As a result, the images generated by the generator look more natural. Note that since known techniques are applied in relation to the learning algorithm and the configuration of the neural network in GAN, the description thereof is omitted in this embodiment.

[0062] In this way, the data used in training is saved in the fluctuation model database 104 in association with the trained fluctuation model. In other words, the image included in the learning data and the degree of the fluctuation factor of the image are held in the fluctuation model database 104 in association with the information representing the fluctuation factor (corresponding to the model).

[0063] The fluctuation model database 104 is stored in the HDD 125 and stores the fluctuation models for each fluctuation factor generated by the fluctuation model generation unit 103 and the data used in training.

[0064] Note that in this embodiment, the fluctuation model generation unit 103 and the fluctuation model database 104 are described as being included in the digital camera 100. However, a configuration can be adopted in which a communication unit is provided in the digital camera 100 and the fluctuation model generation unit 103 and / or the fluctuation model database 104 are arranged on an external server or in the cloud. Alternatively, the fluctuation model generation unit 103 and the fluctuation model database 104 can be arranged on both the digital camera 100 and an external server and selectively used according to the application or purpose.

[0065] For example, in the digital camera 100, a database and a fluctuation model generation unit are arranged. The fluctuation model generation unit is used to generate a fluctuation model associated with fluctuation factors (such as the facial expression of the main subject, etc.) with high expected usage frequency. On the other hand, the fluctuation model generation unit for generating a fluctuation model with low usage frequency, the fluctuation model during training, and / or the learning data can be stored in an external server. The update history of the fluctuation model can also be managed on the external server or cloud service side.

[0066] The shooting intention acquisition unit 105 acquires the shooting intention that the user wants to express when shooting the input image from the input image, and outputs a shooting intention identifier indicating the shooting intention to the fluctuation rule determination unit 106.

[0067] In this embodiment, for example, the relationship between the fluctuation factors included in the image and the shooting intention identifier is predefined, and the fluctuation factors included in the acquired image are converted into a shooting intention identifier. That is, the shooting intention acquisition unit 105 can acquire the shooting intention identifier based on the image information of the image. The shooting intention identifier includes keywords such as keywords for marking general images, such as "fun" and "commemorative photo", etc. In addition, the shooting intention acquisition unit 105 can receive an indication or selection of the shooting intention identifier from the user. Additionally, the shooting intention acquisition unit 105 can infer the information of the shooting intention identifier based on the operation history of the operation performed for the purpose of image acquisition and the user behavior history such as the number of shooting attempts.

[0068] The shooting intention acquisition unit 105 can also use the sound information to output the shooting intention identifier. For example, the shooting intention acquisition unit 105 can also convert the sound information of the shooting space including the user's voice into a shooting intention identifier by using the ambient sound information during shooting.

[0069] The fluctuation rule determination unit 106 calculates the change amount (hereinafter referred to as "fluctuation rule") of the fluctuation degree for each fluctuation factor by using the shooting intention identifier for the fluctuation factors and their degrees of the image that the user wants to reconstruct. In addition, the fluctuation rule determination unit 106 designates the fluctuation model to be used by the image reconstruction unit 107 described later. Note that the processing of the fluctuation rule determination unit 106 will be described in detail later.

[0070] The image reconstruction unit 107 reads a fluctuation model from the fluctuation model database 104 according to the fluctuation rule determined by the fluctuation rule determination unit 106. Then, the image reconstruction unit 107 performs image reconstruction by inputting the image that the user wants to reconstruct and the parameters for reconstruction into the fluctuation model. Note that the image reconstruction unit 107 is not limited to generating one image, and can generate and output multiple images with different degrees of change in the fluctuation elements. Note that image reconstruction will be described in detail later. The image reconstruction unit 107 outputs the reconstructed image to the display control unit 108.

[0071] The display control unit 108 causes the display device 128 to display various images. In the present embodiment, the display control unit 108 causes the display device 128 to display at least the image acquired by the image acquisition unit 101 or the image reconstructed by the image reconstruction unit 107.

[0072] The user instruction acquisition unit 109 receives various instructions related to the reconstruction of an image from the user via the input device 127, and prompts the processing unit of the digital camera 100 to perform a predetermined process. For example, the user instruction acquisition unit 109 receives an image acquisition instruction and a reconstruction instruction from the user. The user instruction acquisition unit 109 can additionally receive the designation of a shooting intention identifier and parameters (such as a fluctuation model, etc.) required for image reconstruction.

[0073] The image difference calculation unit 110 calculates the difference between two input images and determines the degree of deviation between the two images. Note that the image calculation method will be described in detail later.

[0074] The recording unit 111 records the image to the HDD 125. Note that in the present embodiment, the case where the digital camera 100 includes a recording device for recording an image is described as an example, but a communication unit can be provided in the digital camera 100, and the image can be recorded on an external server or in the cloud.

[0075] ● Training process of the fluctuation model

[0076] Next, reference will be made to Figure 3 to describe the training process of the fluctuation model performed by the fluctuation model generation unit 103 and the like. Note that this process can be implemented, for example, by the CPU 122 or GPU 126 of the digital camera 100 executing a computer program, and can be implemented by Figure 1A the various units shown in. Additionally, this process can basically be executed at the timing of receiving a shooting instruction from the user and during an arbitrary period including this timing. However, the present invention is not limited to this, and even if no shooting instruction is received from the user, for example, when the image acquisition unit 101 is operating and the user can take images of his or her surrounding environment, shooting can be performed at regular intervals.

[0077] At the start of the training process, first, in step S301, the image acquisition unit 101 controls the imaging device 129 to acquire an image for training. The acquired training image is, for example, a still image. Additionally, the image acquisition unit 101 may capture a moving image and obtain a still image from the moving image. Note that the acquired image is not limited to the image output by the imaging device 129, and an image acquired in advance and stored in the HDD 125 may be used. Further, the training image may be limited to an image acquired during a specific period or at a specific location. For example, the training image may be an image acquired during a period (as a shooting period or a learning data collection period) between a user's predetermined start instruction and end instruction. Alternatively, the training image may be acquired according to the image to be reconstructed. In addition, the training image may be an image acquired within a predetermined period including the acquisition date and time of the image to be the reconstruction processing object. Alternatively, the training image may be an image acquired within a predetermined range around the acquisition location of the image to be the reconstruction processing object.

[0078] The image acquisition unit 101 outputs the image data of the acquired still image to the fluctuation factor extraction unit 102.

[0079] Next, in step S302, the fluctuation factor extraction unit 102 extracts predetermined fluctuation factors from the image data of the input still image, and calculates (obtains) the degree of fluctuation (score) for the extracted fluctuation factors. Additionally, the fluctuation factor extraction unit 102 normalizes the calculated degree of fluctuation in the region including the fluctuation factors extracted from the image data of the input still image, and outputs the normalized degree of fluctuation as learning data for the fluctuation model to the fluctuation model generation unit 103 together with the degree of fluctuation information.

[0080] Note that in this specification, it is assumed that this process is performed on the image data of each still image for each fluctuation factor. However, the extraction frequency of the fluctuation factors can be determined for each fluctuation factor. For example, factors with rapidly changing fluctuations can be extracted at a higher frequency, and factors with gradually changing fluctuations can be extracted at a lower frequency.

[0081] In step S303, the fluctuation model generation unit 103 reads out the information of the fluctuation model to be trained from the fluctuation model database 104, and performs machine learning processing of the fluctuation model using the input learning data. The machine learning processing of the fluctuation model is, for example, the processing in the training phase of the above-described GAN. After that, the fluctuation model generation unit 103 updates the fluctuation model information in the fluctuation model database 104 together with the data used in the learning. Note that when there is no fluctuation model to be trained in the fluctuation model database 104, a new fluctuation model is added.

[0082] The above processing uses the fluctuations of the fluctuation elements in the image acquired by the user as the learning data for each fluctuation element model. Thus, a neural network of the generator of the GAN that can tune the fluctuations of the fluctuation elements (i.e., can generate an image corresponding to a specified degree of fluctuation) can be constructed.

[0083] ● Reconstruction processing

[0084] Next, the Figure 4 image reconstruction processing using the fluctuation model will be described with reference to. Note that this processing can be implemented, for example, by the CPU 122 or GPU 126 of the digital camera 100 executing a computer program, and can be implemented by Figure 1A the various units shown in. Note that this processing starts in response to receiving an instruction from the user. At the start of the processing, an image to be reconstructed can be selected, and an instruction can be given at any appropriate timing. Note that here, the processing starts in response to receiving an image acquisition instruction as an instruction from the user. Additionally, the reconstruction instruction can be received during the display of the recorded image after image acquisition, or can be received during image playback.

[0085] When starting the image reconstruction processing, in step S401, the image acquisition unit 101 acquires an image to be reconstructed. Note that as a specific example of the following description, the Figure 2 image 208 shown is the case of the image to be reconstructed.

[0086] Next, in step S402, the fluctuation element extraction unit 102 receives the image to be reconstructed from the image acquisition unit 101, extracts the fluctuation elements included in the image, and calculates (acquires) the degree of fluctuation of the fluctuation elements. The operation of the fluctuation element extraction unit 102 performed here is similar to the processing in Figure 3 step S302 of the training processing.

[0087] In step S403, the shooting intention acquisition unit 105 acquires a shooting intention identifier from an arbitrary information group attached to the image. For example, shooting intention identifiers such as "travel", "commemorative photo", or "fun" are acquired from the people, their facial expressions, or background objects appearing in the image 208, and the shooting intention identifier is associated with the image.

[0088] Note that the shooting intention acquisition unit 105 can acquire a shooting intention identifier based on information other than images. For example, in the case where the digital camera 100 is equipped with voice recognition technology, the shooting intention acquisition unit 105 utilizes the result of voice recognition when acquiring the shooting intention identifier. For example, the shooting intention acquisition unit 105 can acquire the shooting intention identifier based on the user's speech information recorded during a predetermined period including the date and time of the captured image or the user's speech information input during a predetermined period after image replay. Specifically, when the user is recognized as saying something like "It's cloudy", "There are too many clouds to see", or "Hope it clears up" when acquiring the image 208 or indicating reconstruction, "climate" or "sunny day" which is considered an ideal state can be used as a keyword. In this case, the keyword is associated with the image as the shooting intention identifier.

[0089] In addition to the above examples, the following configuration can also be adopted: Calculate the shooting intention identifier by prediction based on the user's operation history information or behavior history information during a period including the time of the shooting action of the image 208 selected in step S401, the text information input by the user, etc.

[0090] After that, the shooting intention acquisition unit 105 outputs the shooting intention identifier together with the image 208 associated with the shooting intention identifier to the fluctuation rule determination unit 106.

[0091] In step S404, the fluctuation rule determination unit 106 uses the image to be reconstructed, the fluctuation element information associated with the image, and the shooting intention identifier to determine the fluctuation rule that will be used as control information for the image reconstruction unit 107.

[0092] Here, reference will be made to Figures 5A to 5D to describe the method for creating the fluctuation rule according to the present embodiment. Figures 5A to 5D Shows the relationship between the degree of fluctuation of the fluctuation elements of the image to be reconstructed and various information.

[0093] The fluctuation rule determination unit 106 selects and reads out the fluctuation model information related to the fluctuation elements of the image 208 to be reconstructed from the fluctuation model database 104. Note that the read fluctuation model information is the information of the fluctuation model trained using learning data, and the learning data at least includes images containing the fluctuation elements to be reconstructed.

[0094] The fluctuation rule determination unit 106 uses the read fluctuation model information and the related learning data set to calculate the information on the reconstructable fluctuation range in the fluctuation model. For example, in Figure 5AAn example distribution of learning data showing a fluctuation model related to a smile is shown. In the training of the above GAN, the GAN is trained to be able to generate an image with a degree of fluctuation included in the learning data. Therefore, from Figure 5A From the distribution of the degree of smile in the shown learning data, it can be understood that the fluctuation range of the image that can be reconstructed by specifying the degree of the fluctuation factor is within the range of fluctuation degrees 1 to 6.

[0095] Next, the fluctuation rule determination unit 106 calculates a recommended value of the degree of fluctuation of the reconstructed fluctuation factor according to the shooting intention identifier. In the present embodiment, for example, the digital camera 100 pre-holds information associating the above shooting intention identifier with the ideal degree of fluctuation of the fluctuation factor as conversion table information of the shooting intention and the ideal degree of fluctuation. The fluctuation rule determination unit 106 calculates the degree of fluctuation of the reconstructed fluctuation factor by referring to the conversion table information.

[0096] For example, in the conversion table for the shooting intention identifier "fun", as Figure 5B shown, the fluctuation factors "facial expression" and "composition" are associated. In this example, the ideal degree of fluctuation of the fluctuation factor "facial expression" is associated in such a way that the degree of smile of the "facial expression" is the degree of fluctuation of 7 as the maximum value.

[0097] The fluctuation rule determination unit 106 determines the fluctuation model to be used and calculates the parameters to be set in the determined fluctuation model. The parameters to be set are calculated so as to fall within the above-mentioned reconstructable fluctuation range and be close to the ideal degree of fluctuation of the fluctuation factor according to the shooting intention.

[0098] For example, first, the fluctuation rule determination unit 106 determines whether the ideal degree of fluctuation corresponding to the shooting intention corresponds to the degree of fluctuation that can be set for reconstruction among the degrees of fluctuation (in the above example, between the degrees of fluctuation 1 and 6). When the ideal degree of fluctuation corresponds to the degree of fluctuation that can be set for reconstruction among the degrees of fluctuation, the fluctuation rule determination unit 106 sets the ideal degree of fluctuation as the degree of fluctuation set for reconstruction. When the ideal degree of fluctuation does not correspond to the degree of fluctuation that can be set for reconstruction among the degrees of fluctuation, the fluctuation rule determination unit 106 sets the degree of fluctuation closest to the ideal degree of fluctuation among the degrees of fluctuation that can be set for reconstruction as the degree of fluctuation set for reconstruction. That is, for reconstruction, an adjusted degree of fluctuation adjusted according to the ideal degree of fluctuation is set. For example, as Figure 5B shown, the parameter set as the ideal degree of fluctuation in the fluctuation model of the fluctuation factor "facial expression" is the degree of fluctuation 7, and as Figure 5A shown, the upper limit of the reconstructable range of the fluctuation model is the degree of fluctuation 6. Therefore, as Figure 5C shown, the value set is the degree of fluctuation 6.

[0099] In addition, the fluctuation rule determination unit 106 determines the order of the reconstruction processes using multiple fluctuation models. The order of the processes of the fluctuation models mentioned here is not particularly limited and can be determined by various factors. In the present embodiment, for example, the fluctuation models are processed in the order from the fluctuation model having the largest difference between the recommended value of the above-mentioned fluctuation degree and the fluctuation degree in the image to be reconstructed to the fluctuation model having the smallest difference. In this case, for example, as Figure 5D shown, the fluctuation model reconstruction process is carried out in the order of first "facial expression", second "sunny degree", and finally "composition".

[0100] In this way, the fluctuation rule determination unit 106 outputs the fluctuation model information, the parameter information to be passed to the fluctuation model, and the reconstruction process order information of the fluctuation model as the fluctuation rule to the image reconstruction unit 107.

[0101] Return Figure 4 , in step S405, the image reconstruction unit 107 performs the reconstruction process using the image to be reconstructed and the fluctuation rule determined by the fluctuation rule determination unit 106. For example, as a result of the reconstruction process, an image such as Figure 6 shown is generated. Figure 6 The reconstructed image shown is a new image in which the "composition" does not change much while maintaining the atmosphere of the image 208 to be reconstructed, the smiling degree of the "facial expression" is large, and the degree of the "sunny degree" is large (few clouds).

[0102] Note that the following configuration can be adopted: the generated image is presented to the user for confirmation via the display control unit 108, and feedback on the reconstruction process is received. For example, by applying positive feedback to the fluctuation model when the user issues a recording instruction to record the reconstructed image and by applying negative feedback when it is not the case, the reconstruction process can be newly implemented together with the recording process.

[0103] Due to the above processing, the fluctuation degree of the fluctuation elements of the acquired image and the information indicating the shooting intention of the user are obtained, and an image with a different fluctuation degree is generated from the acquired image using the trained learning model. At this time, the learning model generates an image in which the fluctuation degree acquired in the acquired image is set to a degree corresponding to the information indicating the shooting intention. By adopting such a configuration, it becomes possible to obtain an image that more appropriately reflects the shooting intention.

[0104] ● Image generation and display processing

[0105] Next, reference will be made to Figure 7Describe the post - capture image capture and display process in this embodiment when image reconstruction is implemented. The digital camera 100 in this embodiment captures an image using the imaging device 129 when receiving an image acquisition instruction from the input device 127. In the case where the acquired image is not reconstructed, after the image acquired by the imaging device 129 is displayed on the display device 128 for a certain period, the post - capture display process ends. On the other hand, in the case where the acquired image is reconstructed, the process shown in Figure 7 is started when receiving an image acquisition instruction from the user.

[0106] In step S701, when receiving an image acquisition instruction from the user, the image acquisition unit 101 controls the imaging device 129 to capture an image.

[0107] In step S702, the display control unit 108 causes the display device 128 to display the image acquired by the image acquisition unit 101.

[0108] In step S703, the display control unit 108 performs the reconstruction process on the image acquired by the image acquisition unit 101 as described previously with reference to Figure 4 the above.

[0109] In step S704, the image difference calculation unit 110 compares the pre - reconstruction image acquired by the image acquisition unit 101 with the post - reconstruction image generated by the image reconstruction unit 107, and calculates the difference between the pre - reconstruction image and the post - reconstruction image.

[0110] As a method for the image difference calculation unit 110 to calculate the difference between images, for example, the difference in the degree of fluctuation can be calculated for each fluctuation element included in the image, and the difference in the degree of fluctuation can be output as a difference result associated with the difference between the fluctuation element and the degree of fluctuation. For example, in Figure 5C the degree of fluctuation of the fluctuation element "smile" in the image to be reconstructed is 2, and the degree of fluctuation in the post - reconstruction image is 6. Therefore, 4 is output as the difference result related to "smile". In a similar calculation method, for example, 2 is output as the difference result related to "eye open", and 5 is output as the difference result for "mouth open".

[0111] Alternatively, the output of the difference result can be achieved by normalizing the degree of fluctuation of all the fluctuation factors included in the image and calculating the sum value or average value of the differences in the degree of fluctuation. In addition, the differences in the degree of fluctuation can be weighted according to the degree of influence on the image during reconstruction due to the change in the degree of fluctuation. For example, in the reconstruction of an image related to "facial expression", even when there is a large change in the degree of fluctuation, only the face of the main subject and its surroundings change, so the difference between the pre-reconstruction image and the post-reconstruction image is small. On the other hand, in the reconstruction of an image related to "composition", even when the change in the degree of fluctuation is small, the positions of the subjects in the image change, so the difference between the pre-reconstruction image and the post-reconstruction image tends to be large. Therefore, even when the difference in the degree of fluctuation related to "composition" is small, the weight is set in such a way that the difference result is high.

[0112] Note that the method for calculating the difference between images is not limited to the above method, and for example, the image difference calculation unit 110 can calculate the difference information by using the inter-frame difference method to judge the composition of the pre-reconstruction image and the post-reconstruction image and the amount of movement of the subject.

[0113] In step S705, the image difference calculation unit 110 also judges whether the deviation between the image acquired by the image acquisition unit 101 and the image generated by the image reconstruction unit 107 is greater than or equal to a predetermined deviation based on the difference information calculated in step S704. If it is judged that the deviation between the pre-reconstruction image and the post-reconstruction image is greater than or equal to the predetermined deviation, the image reconstruction unit 107 and the display control unit 108 are notified, and the process proceeds to step S706. If it is judged that the deviation between the pre-reconstruction image and the post-reconstruction image is less than the predetermined deviation, the process proceeds to step S707.

[0114] Here, for example, when the difference in the degree of fluctuation is calculated for each fluctuation factor in step S704, a threshold is set for each fluctuation factor, and when any one of the calculated differences in the degree of fluctuation exceeds the threshold, it is judged that the deviation is greater than or equal to the predetermined deviation. Considering the difference amount of the degree of fluctuation between images, the threshold for the degree of fluctuation of each fluctuation factor is determined in advance for each fluctuation factor. For example, the threshold for "facial expression" is set high, and the threshold for "composition" is set low. Alternatively, the threshold can be determined at the timing of the shooting action by associating with the user behavior history before and after the shooting action.

[0115] In addition, when calculating the difference between images using the inter-frame difference method, for example, when the difference between the images is greater than or equal to a specific percentage of the frame area, it can be judged that the deviation is greater than or equal to the predetermined deviation.

[0116] In step S706, when receiving a notification from the image difference calculation unit 110, the image reconstruction unit 107 performs image reconstruction. Here, through a process similar to the image reconstruction process in step S703, the image reconstruction unit 107 generates a new image with a smaller degree of fluctuation compared to the image generated in step S703.

[0117] Here, reference will be made to Figure 8 and Figure 9 to describe the deviation between the pre-reconstruction image and the post-reconstruction image generated in step S703 and the process of the image reconstruction unit 107 generating an image with suppressed fluctuation degree in step S706. Figure 8 shows a process flow for reconstructing an image to be reconstructed based on the order of reconstruction processing determined by the fluctuation rule determination unit 106, and Figure 9 shows an example of the image to be reconstructed and the image generated through the reconstruction processing.

[0118] Image 801 is the image to be reconstructed. Here, for example, Figure 9 the image 9a shown is the image to be reconstructed 801.

[0119] In the reconstruction process 802, by passing the fluctuation parameter 804 to the fluctuation model 803, reconstruction using the fluctuation model 803 is performed on the image 801. Then, as a result of the reconstruction process, image 805 is generated. For example, when the fluctuation model 803 is "facial expression", image 9b is generated by reconstructing the "facial expression" in image 9a.

[0120] The image reconstruction unit 107 reconstructs the image by performing all the reconstruction processes based on the order of reconstruction processing determined by the fluctuation rule determination unit 106. As a result, the reconstructed image 806 is generated. For example, by performing the reconstruction process 807 and the reconstruction process 808 on the image 9b generated by the reconstruction process 802, a reconstruction result such as the image 9c can be obtained.

[0121] Both image 9b and image 9c are output results obtained by reconstructing image 9a using the fluctuation model. However, image 9b is an image that only reconstructs the "facial expression", while image 9c is an image that reconstructs other fluctuation elements such as the "composition" and "atmosphere" of a photo that were not implemented in the reconstruction process 802. In step S703, the image 9c that reconstructs all the fluctuation elements is output.

[0122] On the other hand, the difference between Image 9c and Image 9a is particularly large due to the reconstruction of "composition". However, since the reconstruction of "composition" is not carried out, the difference between Image 9b and Image 9a is small. In this way, the image reconstruction unit 107 can generate an image with suppressed fluctuation degree by reducing the number of fluctuation models used and the reconstruction processing amount. In step S706, the reconstructed Image 9b for some fluctuation elements is output.

[0123] Note that as a method for generating an image with suppressed fluctuation degree, the parameter information passed to the fluctuation model can be changed. The following will describe the concept of the image generation method when changing the parameter information with reference to Figure 10A and Figure 10B

[0124] In Figure 10A and Figure 10B the same fluctuation model 1002 is used to perform the reconstruction process on the object image 1001. In Figure 10A the reconstruction process is performed by passing the fluctuation parameter A1003 to the fluctuation model 1002. As a result, an image with the fluctuation degree of the object image changed from "3" to "7" can be obtained. In step S703, the reconstructed image using the fluctuation parameter A1003 with such a high fluctuation degree is output.

[0125] On the other hand, in Figure 10B the reconstruction process is performed by passing the fluctuation parameter B1004 to the fluctuation model 1002. As a result, an image with the fluctuation degree of the object image changed from "3" to "5" can be obtained. In this way, by changing the parameter information, an image with suppressed fluctuation degree can be generated. In step S706, the reconstructed image using the fluctuation parameter B1004 with such suppressed fluctuation degree is output.

[0126] In step S706, an image with suppressed fluctuation degree is generated through the above processing and output to the display control unit 108.

[0127] In step S707, the display control unit 108 switches the image displayed on the display device 128 from the image acquired by the image acquisition unit 101 to the image generated by the image reconstruction unit 107. Note that if a notification is received from the image difference calculation unit 110, the image as the output result of step S706 is displayed, and if no notification is received, the image as the output result of step S703 is displayed.

[0128] For example, when switching the display from Figure 9 ​When the pre-reconstruction image 9a shown is switched to the image 9c as the reconstruction result, due to the "composition" within the image being reconstructed, the difference between the images is large. Therefore, when the user switches the display of the images, a sense of incongruity may be felt. On the other hand, when the display is switched from the image 9a to the image 9b with the degree of fluctuation suppressed, the difference between the images is small. Therefore, it is less likely for the user to feel a sense of incongruity. In addition, the reconstruction effect of specific fluctuation elements such as the "facial expression" of the subject becomes visually recognizable.

[0129] In step S708, regardless of the judgment result of the image difference calculation unit 110, the recording unit 111 records the image generated by the image reconstruction unit 107 in step S703 into the HDD 125.

[0130] According to the first embodiment as described above, when the degree of deviation between the pre-reconstruction image and the post-reconstruction image is greater than or equal to a predetermined deviation, a new image with the degree of fluctuation suppressed is reconstructed and displayed. By adopting such a configuration, when reconstructing an image, it becomes possible to reduce the sense of incongruity felt during preview display.

[0131] <Second Embodiment>

[0132] Next, the second embodiment of the present invention will be described. Note that since the image processing device in this embodiment has a configuration similar to that of the image processing device described with reference to Figure 1A and Figure 1B in the first embodiment, its description is omitted here.

[0133] Figure 11 is a flowchart showing post-shot image display and image recording control when image reconstruction is performed in this embodiment. Note that in Figure 11 , the same reference numerals are given to the processes similar to those shown in Figure 7 in the first embodiment, and their descriptions are omitted.

[0134] In step S706, when the reconstruction of the image with the degree of fluctuation suppressed is completed, in step S1101, the image difference calculation unit 110 calculates the difference between the image acquired by the image acquisition unit 101 and the image generated by the image reconstruction unit 107 in step S706.

[0135] In step S1102, the image difference calculation unit 110 also determines whether the deviation between the image acquired by the image acquisition unit 101 and the image generated by the image reconstruction unit 107 in step S706 is greater than or equal to a predetermined deviation based on the difference information between the pre-reconstruction image and the post-reconstruction image calculated in step S1101. If it is determined that the deviation between the pre-reconstruction image and the post-reconstruction image is greater than or equal to the predetermined deviation, the display control unit 108 is notified, and the process proceeds to step S1103. If it is determined that the deviation between the pre-reconstruction image and the post-reconstruction image is less than the predetermined deviation, the process proceeds to step S707. Note that the predetermined deviation mentioned here can be the same threshold as in step S705, or it can be a different threshold.

[0136] In step S1103, the display control unit 108 switches the image displayed on the display device 128 from the image acquired by the image acquisition unit 101 to an arbitrary image.

[0137] Here, the arbitrary image is, for example, an image having a predetermined color such as black, or an image 12d indicating that the process is in progress as shown. Note that an example is shown where image 12a is the image to be reconstructed acquired in step S701, image 12c is the reconstructed image in S703, and image 12b is the reconstructed image in step S706. In step S707, image 12d is used to reduce the sense of incongruity felt when switching the image displayed on the display device 128 from image 12a to image 12b. Image 12d can be any image that does not deviate from this usage application. Figure 12 According to the second embodiment described above, in the case where the degree of deviation between the pre-reconstruction image and the post-reconstruction image is greater than or equal to a predetermined deviation, for the newly generated image with the degree of fluctuation suppressed, before displaying the image reconstructed for all fluctuation elements, an arbitrary image is displayed. By adopting such a configuration, when reconstructing an image, it becomes possible to reduce the sense of incongruity felt during preview display.

[0138] <Third Embodiment>

[0139] Next, a third embodiment of the present invention will be described. Note that since the image processing device in this embodiment also has a configuration similar to that of the image processing device described with reference to the first embodiment

[0140] and Figure 1A and Figure 1B the description thereof is omitted here.

[0141] Figure 13 is a flowchart showing the post-shot image display and image recording control in the case of performing image reconstruction in this embodiment. Note that in Figure 13 for those related to the first embodimentFigure 7 The controls shown for similar processes are given the same reference numerals and their descriptions are omitted.

[0142] In step S705, if it is determined that the deviation between the image acquired by the image acquisition unit 101 in step S701 and the image generated by the image reconstruction unit 107 in step S703 is greater than or equal to a predetermined deviation, the image reconstruction unit 107 and the display control unit 108 are notified, and the process proceeds to step S1301. If it is determined that the deviation between the pre-reconstruction image and the post-reconstruction image is less than the predetermined deviation, the process proceeds to step S707.

[0143] In step S1301, when receiving a notification from the image difference calculation unit 110, the image reconstruction unit 107 performs image reconstruction. Here, using a process similar to the image reconstruction process in step S703, the image reconstruction unit 107 generates a new image in which the degree of fluctuation is less than that in the image generated in step S703.

[0144] Reference will be made here Figure 14 and Figure 15 to describe the deviation between the pre-reconstruction image and the post-reconstruction image generated in step S703 and the process performed in step S1301 in which the image reconstruction unit 107 generates an image with suppressed fluctuation degree. Figure 14 shows a flow of a process for reconstructing an image to be reconstructed based on the order of reconstruction processing determined by the fluctuation rule determination unit 106, and Figure 15 shows examples of the image to be reconstructed and the image generated by the reconstruction processing.

[0145] Image 1401 is the image to be reconstructed. Here, for example, Figure 15 the image 15a shown is the image to be reconstructed 1401.

[0146] In the reconstruction process 1402, by passing the fluctuation parameter 1404 to the fluctuation model 1403, reconstruction using the fluctuation model 1403 is performed on the image 1401. Then, as a result of the reconstruction process, image 1405(1) is generated. For example, when the fluctuation model 1403 is "facial expression", image 15b is generated by performing the reconstruction of the "facial expression" of image 15a.

[0147] The image reconstruction unit 107 reconstructs an image by performing all reconstruction processes based on the fluctuation rules determined by the fluctuation rule determination unit 106 in the order of the reconstruction processes. As a result, a reconstructed image 1406 is generated. For example, by performing the reconstruction process 1407 on the image 15b generated by the reconstruction process 1402, an image 1405(2) is generated, and an image 15c as the reconstruction result can be obtained. Similarly, by performing the reconstruction process 1408 on the image 15c generated by the reconstruction process 1407, an image 1406 is generated, and an image 15d as the reconstruction result can be obtained.

[0148] Note that the fluctuation rule determination unit 106 can determine the order of the reconstruction processes for each fluctuation element as shown in Figure 15 or can determine the order for each arbitrary region. Alternatively, the order can be determined such that the fluctuation elements or arbitrary regions are in the order from the near distance to the far distance or from the far distance to the near distance, or alternatively in ascending order or descending order of the difference between images.

[0149] In step S1302, the image difference calculation unit 110 compares the image generated by the image reconstruction unit 107 in step S703 with the image generated by the image reconstruction unit 107 in step S1301, and calculates the difference between the pre-reconstruction image and the post-reconstruction image.

[0150] Furthermore, in step S1303, based on the difference information between the pre-reconstruction image and the post-reconstruction image calculated in step S1302, the image difference calculation unit 110 determines whether the deviation between the image generated by the image reconstruction unit 107 in step S703 and the image generated by the image reconstruction unit 107 in step S1302 is greater than or equal to a predetermined deviation. If it is determined that the deviation between the pre-reconstruction image and the post-reconstruction image is greater than or equal to the predetermined deviation, the process proceeds to step S1305. If it is determined that the deviation between the pre-reconstruction image and the post-reconstruction image is less than the predetermined deviation, the display control unit 108 is notified, and the process proceeds to step S707.

[0151] In step S1304, the display control unit 108 switches the image displayed on the display device 128 to the image generated by the image reconstruction unit 107 in step S1301, and returns to step S1301. Thus, an image with a gradually increasing degree of fluctuation is generated and displayed until the deviation between the image generated by the image reconstruction unit 107 in step S703 and the image generated by the image reconstruction unit 107 in step S1302 becomes less than the predetermined deviation.

[0152] According to the third embodiment described above, images with different fluctuations in the pre-reconstruction image and the post-reconstruction image are newly generated and displayed according to the degree of deviation between the pre-reconstruction image and the post-reconstruction image. By adopting such a configuration, when generating a reconstructed image, it becomes possible to reduce the sense of incongruity felt during preview display.

[0153] Note that in the first to third embodiments described above, a digital camera capable of generating images is described as an example of an image processing device. However, the present invention is not limited to a device capable of generating images, and can be applied to a device capable of inputting images from an external device. For example, reconstruction processing can be performed on an image acquired by connecting a device to a camera or an image saved on a server or cloud acquired via a network. In this case, a configuration of the above-described processing that starts Figure 7 , Figure 11 , Figure 13 as shown can be adopted.

[0154] <Other Embodiments>

[0155] Note that the present invention can be applied to a system composed of multiple devices or to a device composed of a single device.

[0156] In addition, the present invention can be implemented by supplying a program for implementing one or more functions of the above-described embodiments to a system or device via a network or a storage medium and causing one or more processors in a computer of the system or device to read and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0157] The present invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit or scope of the present invention. Therefore, the following claims are appended to disclose the scope of the present invention.

[0158] This application claims the benefit of the priority of Japanese Patent Application No. 2022-182795 filed on November 15, 2022, which is incorporated herein by reference in its entirety.

Claims

1. An image processing apparatus, characterized in that Comprising: An image acquisition component for acquiring a first image; A fluctuation degree acquisition component for acquiring the fluctuation degree of a fluctuating element having fluctuations among the elements constituting the first image, where the fluctuation is a change in state; A generation component for using a trained learning model to generate a second image with a different fluctuation degree of the fluctuating element compared to the first image using the first image; And A display control component for displaying an image on a display component, wherein, when the deviation between the first image and the second image is greater than or equal to a predetermined deviation, the generation component further uses the first image to generate a third image with a smaller fluctuation degree of the fluctuating element compared to the second image, and the display control component controls to display the first image and then display the third image.

2. The image processing apparatus according to claim 1, wherein the display control component controls to display the first image and then display the third image.

3. The image processing apparatus according to claim 1, wherein when the deviation between the first image and the third image is greater than or equal to a predetermined deviation, the display control component controls to display the first image, then display an arbitrary image different from the first image, the second image, and the third image, and display the third image after displaying the arbitrary image.

4. The image processing apparatus according to claim 3, wherein the arbitrary image includes at least one of an image having a predetermined color and an image indicating that the processing of the generation component is in progress.

5. The image processing apparatus according to claim 1, wherein when the deviation between the second image and the third image is greater than or equal to a predetermined deviation, the generation component further uses the first image to increase the fluctuation degree of the fluctuating element and regenerate a third image with a smaller fluctuation degree of the fluctuating element compared to the second image, and the display control component controls to display the first image and then display the third image in the order of generation.

6. The image processing apparatus according to claim 5, wherein the generation component: generates the second image by performing reconstruction processing using a plurality of fluctuation models as the learning model, and generates the third image by performing reconstruction processing for each of the fluctuating elements or for each predetermined region.

7. The image processing apparatus according to claim 6, wherein the generation component generates the third image by performing the reconstruction processing for the fluctuating elements or the predetermined regions in an order from close range to far range or from far range to close range.

8. The image processing apparatus according to claim 6, wherein the generation component generates the third image by performing the reconstruction processing for the fluctuating elements or the predetermined regions in ascending order or descending order of the deviation between the first image and the second image.

9. The image processing apparatus according to any one of claims 1 to 8, characterized in that Further comprising: A recording component for recording an image onto a recording medium, wherein the recording component records the second image without recording the third image.

10. The image processing apparatus according to any one of claims 1 to 9, characterized in that, Further comprising: A determination component for determining the deviation state between images based on the difference in the degree of fluctuation of the same fluctuation factor between the images.

11. The image processing apparatus according to any one of claims 1 to 9, characterized in that, Further comprising: A determination component for determining the deviation state between images by means of an inter-frame difference method.

12. The image processing apparatus according to any one of claims 1 to 11, characterized in that the generation component: generates the second image by performing a reconstruction process using a plurality of fluctuation models as the learning model, and generates the third image by reducing the number of fluctuation models used among the plurality of fluctuation models.

13. The image processing apparatus according to any one of claims 1 to 12, characterized in that the generation component: generates the second image by performing a reconstruction process using a plurality of fluctuation models as the learning model, and generates the third image by changing the parameter representing the degree of fluctuation assigned to the fluctuation model.

14. The image processing apparatus according to any one of claims 1 to 13, characterized in that the display control component performs display in response to the acquisition of the first image, and the generation component generates the second image in response to the acquisition of the first image.

15. An image processing method, comprising: an image acquisition step for acquiring a first image; a degree-of-fluctuation acquisition step for acquiring the degree of fluctuation of a fluctuation factor having fluctuations among the factors constituting the first image, where the fluctuation is a change in state; a first generation step for generating, using the first image and a trained learning model, a second image having a different degree of fluctuation of the fluctuation factor compared to the first image; and a second generation step for generating, when the deviation between the first image and the second image is greater than or equal to a predetermined deviation, a third image having a smaller degree of fluctuation of the fluctuation factor compared to the second image using the first image; and a display control step for, when the deviation between the first image and the second image is greater than or equal to the predetermined deviation, performing control to display the first image on a display component and then display the third image.

16. A computer program for causing a computer to function as a component of the image processing apparatus according to any one of claims 1 to 14.

17. A computer-readable storage medium storing the computer program according to claim 16.

Images