IMAGE PROCESSING DEVICE AND METHOD, COMPUTER PROGRAM AND STORAGE MEDIUM

Abstract

An image processing device comprises: image acquisition means for acquiring a first image; fluctuation degree detection means configured to detect a fluctuation degree of a fluctuation element having a fluctuation that is a state variation among elements constituting the first image; generation means configured to use the first image to generate a second image in which the fluctuation degree of the fluctuation element differs from the first image using a trained learning model;and a display control device configured to control the display of an image on a display device, wherein, in a case where the first image and the second image diverge by more than or equal to a predetermined divergence, the generating device further uses the first image to generate a third image in which the degree of fluctuation of the fluctuation element is smaller than in the second image, and the display control device performs control to display the first image and thereafter display the third image.;

Description

TECHNICAL FIELD

[0001] The present invention relates to an image processing apparatus and method, a computer program and a storage medium. BACKGROUND

[0002] In recent years, new technologies related to image processing, using AI and advanced computational processing, have been proposed as technologies for generating images. This includes numerous research papers on technologies for generating nonexistent images using generative adversarial networks (GANs), which employ a type of unsupervised learning, as well as many related paper presentations and invention proposals. Against this backdrop, it has become possible to manipulate an image obtained by taking a picture (also known as a "shot") using image processing technologies represented by GANs and reconstruct it to reflect the intent of the user who took the picture.

[0003] On the other hand, newer image capture devices, including digital cameras and smartphones, are generally equipped with a display unit for displaying images. During a normal shooting process, a live image is displayed at the time of shooting. After the development of the captured image is complete, the user can switch to the preview display of the developed image ("captured image") and visually confirm whether the captured image meets the user's expectations.

[0004] When a captured image is reconstructed to generate a new image, the user can visually confirm whether the image obtained by reconstruction ("reconstructed image") conforms to the user's expectations by previewing the reconstructed image on a display unit. However, depending on the shooting situation, there may be a large divergence (also deviation, difference) between the captured image and the reconstructed image. If the reconstructed image is generated and previewed immediately after shooting, the user may feel a strong sense of incongruity due to the difference from the scene in front of their eyes, or confirmation may take some time.

[0005] Examples of divergence between the live view image and the recorded image include a case where a difference occurs between the images due to a discrepancy in the shutter release timing (shooting timing) and the recording timing due to a shutter delay or the like. In response to this, PTL 1 discloses a technology according to which a live view image or an image obtained by performing smoothing filter processing on the recorded image is displayed before the preview of the recorded image, in order to reduce the sense of incongruity that occurs at the time of preview display due to the time discrepancy between shooting and recording.

[0006] Furthermore, PTL 2 discloses the generation and sequential display of a plurality of manipulated images, in which the composition ratio of the next recorded image to be displayed relative to the currently displayed recorded image is gradually increased when a plurality of recorded images captured with a single instruction are sequentially displayed. This can reduce the sense of incongruity when switching to the preview display of the recorded images. CITATION LISTPATENT LITERATURE PTL 1: JP 2005-204210A PTL 2: JP 2014-127966A SUMMARY OF THE INVENTION TECHNICAL PROBLEM

[0007] However, one problem is that when displaying a reconstructed image, the images before and after reconstruction may differ greatly, so that displaying manipulated images using the methods described in PTL 1 and PTL 2 alone is not enough to reduce the sense of incongruity in the preview display.

[0008] One aim of the invention is to reduce the feeling of incongruity that occurs when displaying the reconstructed images in the preview when generating reconstructed images. SOLUTION TO THE PROBLEM

[0009] To achieve the above object, an image processing apparatus of the present invention includes image acquiring means for acquiring a first image; fluctuation degree detecting means for detecting a fluctuation degree of a fluctuation element having fluctuation, which is a variation of a state, among elements constituting the first image; generating means for using the first image to generate a second image in which the degree of fluctuation of the fluctuation element is different from that in the first image, using a trained learning model; and display controlling means for displaying an image on display means;wherein, in a case where the first image and the second image diverge by more than or equal to a predetermined divergence, the generating means further uses the first image to generate a third image in which the degree of fluctuation of the fluctuation element is smaller than in the second image, and the display controlling means performs control to display the first image and subsequently the third image.; ADVANTAGEOUS EFFECTS OF THE INVENTION

[0010] According to the present invention, when generating reconstructed images, the feeling of incongruity when displaying the reconstructed images in the preview can be reduced.

[0011] Further features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings. Note that the same reference numerals indicate the same or similar components throughout the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are part of the specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention. Fig. 1A is a block diagram showing an exemplary functional configuration of an image processing apparatus according to an embodiment of the present invention. Fig. 1B is a block diagram showing an example hardware configuration of the image processing apparatus according to the embodiment. Fig. 2 is a diagram illustrating the fluctuation of picture elements according to the embodiment. Fig. 3 is a flowchart showing the processing of fluctuation model training according to the embodiment. Fig. 4 is a flowchart showing the image reconstruction processing according to the embodiment. Fig. 5A is a diagram showing an embodiment of generating fluctuation rules of images. Fig. 5B is a diagram showing an embodiment of generating fluctuation rules of images. Fig. Figure 5C is a diagram showing an embodiment of generating fluctuation rules of images. Fig. Figure 5D is a diagram showing an example of generating fluctuation rules of images. Fig. 6 is a diagram showing an example of image generation according to the embodiment. Fig. 7 is a flowchart showing the processes of generating and displaying images according to a first embodiment. Fig. 8 is a diagram showing the flow of image generation processing according to the first embodiment. Fig. 9 is a diagram showing an exemplary generation of images according to the first embodiment. Fig. 10A is a diagram showing the flow of further processing for generating images according to the first embodiment. Fig. 10B is a diagram showing the flow of further processing for generating images according to the first embodiment. Fig. 11 is a flowchart showing the processes of generating and displaying images according to a second embodiment. Fig. 12 is a diagram for exemplary image generation according to the second embodiment. Fig. 13 is a flowchart showing the processes of generating and displaying images according to a third embodiment. Fig. 14 is a diagram showing the flow of image generation processing according to the third embodiment. Fig. 15 is a diagram for exemplary image generation according to the third embodiment. DESCRIPTION OF THE EMBODIMENTS

[0013] Embodiments will be described in detail below with reference to the accompanying drawings. It should be noted that the following embodiments are not intended to limit the scope of the claimed invention. Several features are described in the embodiments, but no limitation is placed on an invention requiring all of these features; rather, some of these features may be combined if necessary. Furthermore, in the accompanying drawings, identical or similar embodiments are denoted by the same reference numerals, and redundant descriptions are omitted. <Erstes Ausführungsbeispiel>

[0014] In the following, a digital camera capable of generating images serves as an example of an image processing device according to a first embodiment. The present embodiment is not limited to a digital camera and can also be applied to other devices capable of generating images. Examples of such devices include mobile phones such as smartphones, game consoles, personal computers, tablet terminals, portable information terminals, server devices, and the like. • Example configuration of the digital camera

[0015] Fig. 1A is a diagram showing an exemplary functional configuration of a digital camera 100 as an example of an image processing apparatus in the embodiment, and Fig. 1B is a diagram showing an example hardware configuration of the Fig. 1A. Some or all of the features shown in Fig. 1A can be realized, for example, by a CPU 122 or a GPU 126, which in Fig. 1B and execute a computer program.

[0016] As in Fig. As shown in Figure 1A, the digital camera 100 includes an image acquisition unit 101, a fluctuation element extraction unit 102, a fluctuation model generation unit 103, a fluctuation model database 104, and a shooting intention detection unit (also called a triggering intention detection unit) 105. In addition, the digital camera 100 includes a fluctuation rule determination unit 106, an image reconstruction unit 107, a display control unit 108, a user instruction detection unit 109, an image difference calculation unit 110, and a recording unit 111.

[0017] As in Fig. 1B, the hardware configuration of the digital camera 100 includes a system bus 121, the CPU 122, a ROM 123, a RAM 124, an HDD 125, the GPU 126, an input device 127, a display device 128, and an image pickup device 129. These components are connected to the system bus 121.

[0018] The CPU 122 is a computing circuit, such as a CPU (Central Processing Unit), and implements the functions of the digital camera 100 by extracting a computer program stored in the ROM 123 or the HDD 125 into the RAM 124 and executing the computer program. The ROM 123 comprises, for example, a non-volatile storage medium, such as a semiconductor memory, and stores programs executed by the CPU 122 and necessary data. The RAM 124 comprises a volatile storage medium, such as a semiconductor memory, and temporarily stores, for example, calculation results of the CPU 122.

[0019] The HDD 125 comprises a hard disk drive and stores, for example, computer programs executed by the CPU 122, their processing results, and the like. Furthermore, the HDD 125 (recording medium) stores images recorded by the recording unit 111. Note that in this example, the digital camera 100 is described as being equipped with a hard disk, but the digital camera 100 may have a storage medium such as an SSD instead of a hard disk.

[0020] The GPU (Graphics Processing Unit) 126 includes a computing circuit and can, for example, perform all or part of the processing of the learning model training stage and the processing of the inference stage. Because a GPU is capable of processing more data in parallel than a CPU, processing with a GPU is effective in deep learning, which involves performing repetitive operations with neural networks.

[0021] The input device 127 includes control elements such as buttons and a touch panel that receive control inputs for the digital camera 100. The display device 128 includes a display panel, such as an OLED. The image capture device 129 includes optical system units such as a lens, an aperture, and a shutter, as well as an image sensor such as a CMOS sensor. The optical system units may also include a fly's eye lens or a multi-lens lens. The optical units may also be capable of changing optical properties, such as zoom and aperture, depending on the captured image.

[0022] In the digital camera 100 configured as described above, the image acquisition unit 101 first performs image acquisition processing. Note that in the present embodiment, the image acquisition unit 101 can acquire not only an image but also meta information for the image. The meta information for an image includes, for example, information about the date and time the image was captured and information about the shooting position. The image acquisition unit 101 controls the acquisition of images by the image pickup device 129 and outputs the acquired images to the fluctuation element extraction unit 102, the shooting intention detection unit 105, the image reconstruction unit 107, the display control unit 108, and the image difference calculation unit 110.Note that the image acquisition unit 101 may output the acquired images after normalizing the images by performing optional image processing, such as cropping or scaling the images, to conform to the output target.

[0023] Here, “fluctuation” and “fluctuation element” according to the present embodiment are defined with reference to Fig. 2 described.

[0024] Fig. 2 represents the “fluctuation” of the elements that make up the images. In Fig. In Figure 2, the horizontal axis represents time, and the vertical axis represents the degree of each element. Bar charts 201, 202, and 203 show the temporal changes of the elements that comprise the images. For example, bar chart 201 shows the temporal change of the "smile level" from the "facial expressions" of a main subject. Bar chart 202 shows the temporal change of the "position" from "compositions," and bar chart 203 shows the temporal change of the "sunnyness" of the "weather condition."

[0025] In the present embodiment, the change in the state of the elements constituting an image is referred to as "fluctuation." For example, the variation (change) in the state of an element such as smile level is referred to as "fluctuation." Note that elements with "fluctuation" are referred to as "fluctuation elements." A "fluctuation," that is, a change in state, can also be detected by measuring the degree of state in a plurality of captured images.

[0026] Here, the case is described where the user's intention to take a picture is that the "smile level" of the "facial expression" is high, the subject appears on the left side as the "position" of the "composition", and the "sunnyness" of the "weather condition" is high.

[0027] In the Fig. In the example shown in Figure 2, the time at which the fluctuation of each fluctuation element is closest to the shooting intention is a time 204 at which the "smile level" is highest, a time 205 at which the "position" is highest (subject on the left), and a time 206 at which the "sunshine" is highest. The images captured at times 204, 205, and 206 are indicated as images 207, 208, and 209, respectively.

[0028] Returning to the description of Fig. 1, the fluctuation element extraction unit extracts a fluctuation element contained in an image. For example, in an example where a person's facial expression is a fluctuation element, the fluctuation element extraction unit 102 performs detection of a person's face in an image and extracts the fluctuation element. When a person's face is detected, the fluctuation element extraction unit 102 further performs processing to detect the degree of fluctuation in the person's facial expression. For example, the fluctuation element extraction unit 102 detects this degree of fluctuation and quantifies the degree of smile, the degree of emotion, the degree of eye opening, the degree of mouth opening, and the like.It should be noted that when detecting the degree of fluctuation, the degree of fluctuation can be calculated from the image, or the degree of fluctuation corresponding to the image can be detected through a network.

[0029] Other fluctuation elements can be, for example, the posture of a person in the image, the composition of the image, the weather conditions in the image and the clothing of the person in the image. The posture of the person includes, for example, the orientation of the face, the orientation of the body and the degree of blurriness of the person's movements. The composition of the image also includes at least one of the following elements: the positional relationship of the people and the distance between the people. The lighting of the image includes, for example, the position of the light source. The weather conditions in the image include at least one of, for example, weather and cloud cover. The clothing in the image includes at least one of the following: type and color of clothing, for example.

[0030] The fluctuation element extraction unit 102 outputs the calculated fluctuation degree of the fluctuation element along with the image to the fluctuation rule determination unit 106. Furthermore, the fluctuation element extraction unit 102 outputs the image and the fluctuation degree of the fluctuation element to the fluctuation model generation unit 103 as learning data for a fluctuation model described later.

[0031] The fluctuation model generation unit 103 performs processing to train a learning model for each fluctuation item (hereinafter, "fluctuation model") using the image and the fluctuation degree of the extracted fluctuation item obtained by the fluctuation item extraction unit 102. The fluctuation model is generated and trained for each fluctuation item to generate an image corresponding to a specific fluctuation degree. For example, a fluctuation model whose fluctuation element is a person's facial expression is trained to generate an image of a specific facial expression. Note that even with the same fluctuation item, a plurality of fluctuation models can be generated for each period of time, for example, one month, for each area visited by the user, according to the user's instructions, and the like.

[0032] The fluctuation models can also be generated using well-known machine learning algorithms that can generate images, such as GANs. GANs consist of two neural networks: a generator that generates images and a discriminator that distinguishes whether the images generated by the generator are real images. During the training stage of the fluctuation model, the generator and the discriminator described above share a loss function and repeatedly update the respective neural networks so that the generator minimizes the loss function and the discriminator maximizes it. The images generated by the generator therefore appear more natural. Since known technologies are applied in GANs with regard to the learning algorithm and the configuration of the neural networks, their description is omitted in this embodiment.

[0033] In this way, the data used in training are stored in the fluctuation model database 104 in association with the trained fluctuation model. In other words, images included in the training data and the degrees of fluctuation elements of the images are stored in the fluctuation model database 104 in association with information indicating the fluctuation elements (corresponding to the models).

[0034] The fluctuation model database 104 is stored on the hard disk 125 and stores a fluctuation model for each fluctuation item generated by the fluctuation model generation unit 103 and the data used in training.

[0035] Note that in the present embodiment, the fluctuation model generation unit 103 and the fluctuation model database 104 are described as being included in the digital camera 100. However, a configuration may be adopted in which a communication unit is provided in the digital camera 100 and the fluctuation model generation unit 103 and / or the fluctuation model database 104 are located on an external server or a cloud. Alternatively, the fluctuation model generation unit 103 and the fluctuation model database 104 may be located both in the digital camera 100 and on an external server and used selectively depending on the application or purpose.

[0036] For example, a database and a fluctuation model generation unit associated with fluctuation elements expected to be used frequently, such as the facial expression of the main subject, are arranged in the digital camera 100. On the other hand, a fluctuation model generation unit with a low usage frequency may store fluctuation models in training and / or learning data on an external server. The update history of the fluctuation models may also be managed on the external server or on a cloud service.

[0037] The shooting intention detection unit 105 detects, from an input image, the shooting intention that the user who took the image wishes to express, and outputs a shooting intention identifier indicating the shooting intention to the fluctuation rule determination unit 106.

[0038] In the present embodiment, for example, the relationship between fluctuation elements contained in images and shooting intent identifiers is defined in advance, and fluctuation elements contained in acquired images are converted into shooting intent identifiers. That is, the shooting intent acquisition unit 105 is capable of acquiring shooting intent identifiers based on the image information of the images. The identifiers for the shooting intent include keywords used in labeling general images, such as "fun" and "souvenir photo." Furthermore, the shooting intent acquisition unit 105 can receive an instruction or selection regarding the shooting intent identifier from the user. Furthermore, the shooting intent acquisition unit 105 can acquire information about the shooting intent identifier from the history of operations performed for the purpose of image acquisition and the history of user behavior, such asthe number of recording attempts.

[0039] The shooting intent detection unit 105 can further output a shooting intent identifier using sound information. For example, the shooting intent detection unit 105 can also convert sound information from a recording room, including the user's voice, into a shooting intent identifier using ambient sound information at the time of recording.

[0040] The fluctuation rule determination unit 106 calculates the amount of change in the fluctuation degree for each fluctuation element (hereinafter, "fluctuation rule") based on the fluctuation element of the image the user wishes to reconstruct and its degree, using the shooting intent identifier. Furthermore, the fluctuation rule determination unit 106 determines the fluctuation model to be used by the image reconstruction unit 107 described later. The processing by the fluctuation rule determination unit 106 will be described in detail later.

[0041] The image reconstruction unit 107 reads a fluctuation model from the fluctuation model database 104 in accordance with the fluctuation rule determined by the fluctuation rule determination unit 106. The image reconstruction unit 107 then performs image reconstruction by inputting the image the user wishes to reconstruct and the parameters to be used for reconstruction into the fluctuation model. Note that the image reconstruction unit 107 is not limited to generating one image, but can generate and output a plurality of images with different degrees of change of the fluctuation element. The image reconstruction will be described in detail later. The image reconstruction unit 107 outputs the reconstructed image to the display control unit 108.

[0042] The display control unit 108 causes the display device 128 to display various images. In the present embodiment, the display control unit 108 causes the display device 128 to display at least the image acquired by the image acquisition unit 101 or the image reconstructed by the image reconstruction unit 107.

[0043] The user instruction acquisition unit 109 receives various image reconstruction instructions from the user via the input device 127 and causes the processing units of the digital camera 100 to perform predetermined processing. For example, the user instruction acquisition unit 109 receives image acquisition instructions and reconstruction instructions from the user. The user instruction acquisition unit 109 can additionally receive a shooting intent identifier and parameters required for image reconstruction, such as a fluctuation model.

[0044] The image difference calculation unit 110 calculates the difference between the two input images and determines the degree of divergence between the two images. The image calculation method is described in detail below.

[0045] The recording unit 111 records images on the hard disk 125. Note that, in the present embodiment, the case where a recording unit that records images is included in the digital camera 100 is described, but a communication unit may also be provided in the digital camera 100, and images may be recorded on an external server or a cloud. • Training process of the fluctuation model

[0046] Next, the processing of the fluctuation model by the fluctuation model generation unit 103 and the like will be described with reference to Fig. 3. Note that this processing may be performed, for example, by the CPU 122 or GPU 126 of the digital camera 100 executing a computer program, and by the various Fig. 1A. This processing can also be performed in principle at the time a recording instruction is received from the user and during any period including that time. However, the present invention is not limited to this, and even if no instruction is received from the user, recordings can be performed at regular intervals, for example, when the image capture unit 101 is operating and the user is able to record an image of their surroundings.

[0047] When training processing is started, the image acquisition unit 101 first controls the image pickup device 129 for training in step S301. The captured image for training is, for example, a still image. The image acquisition unit 101 may also capture a moving image and create a still image from the moving image. Note that the captured image is not limited to the image output by the image pickup device 129; an image captured in advance and stored on the hard disk 125 may also be used. Also, the image for training may be limited to an image captured within a specific period of time or at a specific position. For example, the image to be trained may be an image captured between the user's specified start and end instructions, such as during the capture period or during the learning data acquisition period.Alternatively, the training image can be acquired corresponding to the image intended for reconstruction. Furthermore, the training image can be an image acquired during a predetermined period of time, including the acquisition date and the processing time of the image intended for reconstruction. Alternatively, the training image can be an image acquired within a predetermined area around the acquisition position of the image intended for reconstruction.

[0048] The image acquisition unit 101 outputs the image data of the captured still image to the fluctuation element extraction unit 102.

[0049] Next, in step S302, the fluctuation element extraction unit 102 extracts a predetermined fluctuation element from the image data of the input still image and calculates (acquires) the fluctuation degree (score) for the extracted fluctuation element. Furthermore, the fluctuation element extraction unit 102 normalizes the calculated fluctuation degree in the area containing the extracted fluctuation element from the image data of the input still image and outputs the normalized fluctuation degree to the fluctuation model generation unit 103 as fluctuation model learning data along with fluctuation degree information.

[0050] This description assumes that this processing is performed for each fluctuation element on the image data of each still image. However, the extraction frequency of the fluctuation elements can be determined individually for each fluctuation element. For example, elements whose fluctuation changes drastically can be extracted at a higher frequency, while elements whose fluctuation changes gradually can be extracted at a lower frequency.

[0051] In step S303, the fluctuation model generation unit 103 reads information about the fluctuation model to be trained from the fluctuation model database 104 and performs machine learning processing on the fluctuation model using the input learning data. The machine learning processing of the fluctuation model is, for example, processing of the training stage of GANs described above. After that, the fluctuation model generation unit 103 updates the information about the fluctuation model in the fluctuation model database 104 along with the data used in the learning. If a fluctuation model to be trained does not exist in the fluctuation model database 104, a new fluctuation model is added.

[0052] The above processing uses the fluctuation of the fluctuation elements in the images captured by the user as training data for each fluctuation element model. This allows the neural network of the GAN generator to be constructed, which is capable of adapting the fluctuation of fluctuation elements (i.e., capable of generating images corresponding to a certain degree of fluctuation). • Reconstruction

[0053] Next, with reference to Fig. 4 describes the processing of image reconstruction using a fluctuation model. It should be noted that this processing is performed, for example, by the CPU 122 or GPU 126 of the digital camera 100 executing a computer program, and by the various Fig. 1A. It should also be noted that this processing is initiated in response to the receipt of an instruction from the user. At the beginning of processing, an image to be reconstructed can be selected. The instruction can be given at any suitable time. Processing is initiated in response to the receipt of an image acquisition instruction as an instruction from the user, and in addition, a reconstruction instruction can be received during the display of a recorded image after image acquisition or at the time of image playback.

[0054] When the image reconstruction processing is started, the image acquisition unit 101 acquires an image intended for reconstruction in step S401. Note that, as a specific example for the following description, the case where the image in Fig. Figure 208 shown in Figure 2 is the target of the reconstruction.

[0055] Next, in step S402, the fluctuation element extraction unit 102 receives the image to be reconstructed from the image acquisition unit 101, extracts a fluctuation element contained in the image, and calculates (detects) the degree of fluctuation of the fluctuation element. The operations of the fluctuation element extraction unit 102 performed here are similar to the processing in the training processing performed in step S302 of Fig. 3 is carried out.

[0056] In step S403, the shooting intent detection unit 105 detects a shooting intent identifier from any information group associated with the image. For example, a shooting intent identifier such as "travel," "memory photo," or "fun" is detected from the person appearing in the image 208, their facial expression, or a background object and associated with the image.

[0057] Note that the shooting intent detection unit 105 may detect a shooting intent identifier based on information other than the image. For example, if the digital camera 100 is equipped with voice recognition technology, the shooting intent detection unit 105 uses the result of the voice recognition to detect a shooting intent identifier. For example, the shooting intent detection unit 105 may detect a shooting intent identifier based on user utterance information recorded during a predetermined period, including the date and time the image was captured, or based on user utterance information input during a predetermined period after the image was played back.If the user says something like "it's cloudy," "it's too cloudy to see," or "I wish it were sunny" when capturing image 208 or during the reconstruction instruction, the "weather condition" or "sunny" considered ideal can be used as a keyword. In this case, the keyword is linked to the image as a shooting intent identifier.

[0058] In addition to the examples described above, a configuration may be adopted in which the shooting intention identifier is calculated by prediction from the information on the operation history or behavior history of the user, text information input by the user, and the like during a period including the shooting time of the image 208 selected in step S401.

[0059] Thereafter, the shooting intention detection unit 105 outputs the shooting intention identifier to the fluctuation rule determination unit 106 with the associated image 208.

[0060] In step S404, the fluctuation rule determining unit 106 determines a fluctuation rule serving as control information for the image reconstruction unit 107 using the image to be reconstructed, the fluctuation element information associated with the image, and the shooting intention identifier.

[0061] In the following, a method for creating fluctuation rules according to the present embodiment will be described with reference to the Fig. 5A to 5D. Fig. 5A to 5D show the relationship between the degree of fluctuation of a fluctuation element of the image to be reconstructed and various information.

[0062] The fluctuation rule determination unit 106 selects and reads fluctuation model information related to the fluctuation element of the image 208 to be reconstructed from the fluctuation model database 104. Note that the read fluctuation model information is information of a fluctuation model trained using training data, and the training data includes at least the image with the fluctuation element to be reconstructed.

[0063] The fluctuation rule determination unit 106 calculates information about a fluctuation range that can be reconstructed in the fluctuation model by using the read fluctuation model information and the associated learning data group. An example of the distribution of learning data of a fluctuation model with respect to the degree of smile is shown in Fig. 5A. In the training of GANs described above, the GANs are trained to be able to generate an image of the degree of fluctuation contained in the training data. From the distribution of the degree of smile in the Fig. From the learning data shown in Figure 5A, it can be seen that the fluctuation range of the images that can be reconstructed by determining the fluctuation degree of the fluctuation element is in the range of fluctuation degrees 1 to 6.

[0064] Next, the fluctuation rule determination unit 106 calculates a recommended value for the fluctuation degree of the fluctuation element after reconstruction from the shooting intention identifier. In the present embodiment, for example, the digital camera 100 has information linking the aforementioned shooting intention identifiers with ideal fluctuation degrees of the fluctuation elements in advance as conversion table information of the shooting intention and the ideal fluctuation degree. The fluctuation rule determination unit 106 calculates the fluctuation degree of the fluctuation element after reconstruction by referring to the conversion table information.

[0065] In the conversion table for the shooting intention “fun”, for example, the fluctuation elements “facial expression” and “composition” are assigned, as in Fig. 5B. In this example, the ideal smile level of the fluctuation element "Facial Expression" is assigned such that the smile level of "Facial Expression" is a smile level of 7, which is the maximum value.

[0066] The fluctuation rule determination unit 106 determines the fluctuation model to be used and calculates the parameter to be adjusted in the determined fluctuation model. The parameter to be adjusted is calculated so that it falls within the aforementioned reconstructable fluctuation range and approximates the ideal fluctuation degree of the fluctuation element according to the shooting intention.

[0067] For example, the fluctuation rule determination unit 106 first determines whether the ideal fluctuation corresponding to the shooting intention corresponds to a fluctuation that can be set for reconstruction among the fluctuation degrees (in the above example, between fluctuation degrees 1 and 6). If the ideal fluctuation degree corresponds to a fluctuation degree that can be set for reconstruction among the fluctuation degrees, the fluctuation rule determination unit 106 sets the ideal fluctuation degree as the fluctuation degree set for reconstruction. When the ideal fluctuation degree does not agree with a fluctuation degree settable for reconstruction among the fluctuation degrees, the fluctuation rule determining unit 106 sets the fluctuation degree closest to the ideal fluctuation degree among the fluctuation degrees settable for reconstruction as the fluctuation degree set for reconstruction.That is, for reconstruction, the adjusted fluctuation level, which is adjusted according to the ideal fluctuation level, is set. For example, the parameter set as the ideal fluctuation level in the fluctuation model of the "facial expression" fluctuation element is fluctuation level 7, as shown in . Fig. 5B, while the upper limit of the reconstructable range of the fluctuation model is the fluctuation degree 6, as shown in Fig. 5A. The value that is set is therefore the fluctuation level 6, as shown in Fig. 5C.

[0068] In addition, the fluctuation rule determining unit 106 determines the order of processing the reconstruction using a plurality of fluctuation models. The order of processing the fluctuation models referred to here is not necessarily limited and can be determined by various factors. In the present embodiment, for example, the fluctuation models are processed in the order from the fluctuation model with the largest difference between the above-mentioned recommended value of the fluctuation degree and the fluctuation degree in the image to be reconstructed to the fluctuation model with the smallest difference. In this case, for example, as in Fig. 5D, the processing of the fluctuation model reconstruction is performed in the order of “facial expression” first, “sunshine” next, and “composition” last.

[0069] In this way, the fluctuation rule determining unit 106 outputs fluctuation model information, parameter information to be transferred to the fluctuation models, and information about the order of processing the fluctuation models as a fluctuation rule to the image reconstruction unit 107.

[0070] Back to Fig. 4, in step S405, the image reconstruction unit 107 performs reconstruction processing using the image determined for reconstruction and the fluctuation rule determined by the fluctuation rule determination unit 106. As a result of the reconstruction processing, for example, an image as shown in Fig. 6 shown. The Fig. The reconstructed image shown in Figure 6 is a new image in which the “composition” has not changed much, the degree of smile of the “facial expression” is high, and the degree of “sunny” is high (few clouds), while the atmosphere of the image 208 intended for reconstruction is preserved.

[0071] Note that a configuration can be adopted in which the generated image prompts the user for confirmation via the display control unit 108 and receives feedback about the reconstruction. For example, reconstruction can be reimplemented together with recording processing by providing the fluctuation model with positive feedback when the user issues a recording instruction to record the reconstructed image and negative feedback when the user does not.

[0072] Through the above processing, the degree of fluctuation of the fluctuation element of the captured image and information indicating the user's shooting intention are captured, and images with different degrees of fluctuation are generated from the captured image using the trained learning model. At this time, the learning model generates an image with the degree of fluctuation in the captured image adjusted to the degree corresponding to the information indicating the shooting intention. Through this configuration, it becomes possible to obtain an image that better reflects the shooting intention. • Image generation and display processing

[0073] Hereinafter, the shooting of images after the capture and the display processing in the present embodiment in the case of implementing the image reconstruction will be described with reference to Fig. 7. The digital camera 100 of the present embodiment captures an image with the image capturing device 129 after receiving an image capturing instruction from the user from the input device 127. If the reconstruction of the captured image is not implemented, the post-capture display processing is terminated as soon as the image captured by the image capturing device 129 has been displayed on the display device 128 for a certain time. On the other hand, if the reconstruction of the captured image is implemented, after receiving an image capturing instruction from the user, the display processing shown in Fig. The processing shown in Figure 7 is started.

[0074] In step S701, the image acquisition unit 101 controls the image pickup device 129 after receiving an instruction to capture images from the user.

[0075] In step S702, the display control unit 108 causes the display device 128 to display the image captured by the image capture unit 101.

[0076] In step S703, the display control unit 108 performs reconstruction processing of the image acquired by the image acquisition unit 101, as previously described with reference to Fig. 4 described.

[0077] In step S704, the image difference calculation unit 110 compares the image acquired by the image acquisition unit 101 before reconstruction with the image generated by the image reconstruction unit 107 after reconstruction and calculates the difference between the images before and after reconstruction.

[0078] As a method for calculating the difference between images by the image difference calculation unit 110, the difference in the degree of fluctuation may be calculated, for example, for each fluctuation element in an image and output as a difference result associated with the difference between the fluctuation element and the degree of fluctuation. Fig. 5C, for example, the fluctuation degree of the fluctuation element "smile degree" in the image targeted for reconstruction is 2, and the fluctuation degree in the image after reconstruction is 6, so the difference result with respect to "smile degree" is output as 4. For example, with a similar calculation method, 2 is output as the difference result with respect to "eye opening" and 5 as the difference result with respect to "mouth opening."

[0079] The difference result can also be output by normalizing the fluctuation degree of all fluctuation elements contained in the image and calculating the total value or average of the differences in the fluctuation degrees. Furthermore, the difference in the fluctuation degrees can be weighted according to the degree of influence of the change in the fluctuation degree on the image at the time of reconstruction. For example, when reconstructing an image with the theme of "facial expression," even if the fluctuation changes significantly, only the face of the main subject and its surroundings change, so the difference between the images before and after reconstruction is small.On the other hand, when reconstructing an image related to "composition," the position of the subject within the image changes even with a small change in the degree of fluctuation, and therefore the difference between the images before and after reconstruction tends to be large. Accordingly, the weighting is adjusted so that the difference between the images before and after reconstruction is large, even if the difference (difference result) in the degree of fluctuation related to "composition" is small.

[0080] Note that the method for calculating the image difference is not limited to the method described above. For example, the image difference calculation unit can calculate the difference information by determining the composition of the image before and after reconstruction and the amount of subject movement using an inter-frame difference method.

[0081] In step S705, the image difference calculation unit 110 further determines whether the image acquired by the image acquisition unit 101 and the image generated by the image reconstruction unit 107 diverge by more than or equal to a predetermined divergence based on the difference information calculated in step S704. If it is determined that the images before and after reconstruction diverge by more than or equal to the predetermined divergence, the image reconstruction unit 107 and the display control unit 108 are notified, and processing proceeds to step S706. If it is determined that the images before and after reconstruction diverge by less than the predetermined divergence, processing proceeds to step S707.

[0082] Here, for example, when calculating the difference in the degree of fluctuation for each fluctuation element in step S704, a threshold value is set for the fluctuation element. If any of the calculated differences in the degree of fluctuation exceeds the threshold value, it is determined that the divergence is greater than or equal to the predetermined divergence. The threshold value of the degree of fluctuation for each fluctuation element is determined in advance for each fluctuation element, taking into account the magnitude of the difference in the degree of fluctuation between the images. For example, the threshold value for "facial expression" is set high, and the threshold value for "composition" is set low. Alternatively, the threshold value may be determined at the time of shooting by linking it to the user's behavior before and after shooting.

[0083] When calculating the difference between images using the interframe difference method, it can also be determined that the divergence is greater than or equal to a given divergence, for example, if the difference between the images is greater than or equal to a certain percentage of the image area.

[0084] In step S706, the image reconstruction unit 107 performs image reconstruction after receiving notification from the image difference calculation unit 110. Here, the image reconstruction unit 107 generates a new image in which the degree of fluctuation is smaller (suppressed) than the image generated in step S703, using processing similar to the image reconstruction processing in step S703.

[0085] The divergence between the images generated in step S703 before and after reconstruction and the procedure performed in step S706 in which the image reconstruction unit 107 generates an image in which the degree of fluctuation is suppressed will be explained with reference to the Fig. 8 and Fig. 9 described. Fig. Fig. 8 shows the flow of processing for reconstructing an image intended for reconstruction based on the order of reconstruction processing determined by the fluctuation rule determining unit 106, and Fig. 9 shows an example of an image intended for reconstruction and images generated by the reconstruction processing.

[0086] An image 801 is an image that is to be reconstructed. In this case, for example, the image in Fig. Figure 9a shown in Figure 9 is the image 801 to be reconstructed.

[0087] In reconstruction processing 802, reconstruction is performed using a fluctuation model 803 on image 801 by passing a fluctuation parameter 804 to fluctuation model 803. As a result of reconstruction processing, an image 805 is then generated. For example, if fluctuation model 803 is "facial expression," an image 9b is generated by reconstructing the "facial expression" in image 9a.

[0088] The image reconstruction unit 107 reconstructs the image by performing all reconstruction processing based on the order of reconstruction processing determined by the fluctuation rule determination unit 106. As a result, a reconstructed image 806 is generated. For example, by performing reconstruction processing 807 and reconstruction processing 808 on the image 9b generated by the reconstruction processing 802, a reconstruction result as shown in an image 9c can be obtained.

[0089] Images 9b and 9c are both output results obtained by reconstructing image 9a using a fluctuation model. However, image 9b is an image in which only the "facial expression" is reconstructed, while image 9c is an image in which other fluctuation elements that were not implemented in reconstruction processing 802, such as the "composition" and "weather condition" of the photo, are reconstructed. In step S703, image 9c in which all fluctuation elements are reconstructed is output.

[0090] On the other hand, the difference between image 9b and image 9a is large, particularly due to the "composition" reconstruction, while the difference between image 9c and image 9a is small because the "composition" reconstruction is not implemented. In this way, the image reconstruction unit 107 is able to generate an image in which the degree of fluctuation is suppressed by reducing the number of fluctuation models used and the amount of reconstruction processing. In step S706, the image 9b reconstructed for some fluctuation elements is output.

[0091] It should be noted that in an image generation method that suppresses the degree of fluctuation, the parameter information passed to the fluctuation model may be changed. The following describes the concept of an image generation method in the case of a change in the parameter information with reference to Fig. 10A and Fig. 10B described.

[0092] In Fig. 10A and Fig. 10B, the processing of the reconstruction of a target image 1001 is performed using the same fluctuation model 1002. In Fig. 10A, the reconstruction processing is implemented by passing a fluctuation parameter A1003 to the fluctuation model 1002. As a result, an image can be acquired in which the degree of fluctuation of the target image is changed from "3" to "7." In step S703, an image reconstructed using the fluctuation parameter A1003 in which the degree of fluctuation is thus high is output.

[0093] On the other hand, in Fig. 10B, the reconstruction processing is implemented by passing a fluctuation parameter B1004 to the fluctuation model 1002. As a result, an image can be acquired in which the degree of fluctuation of the target image is changed from "3" to "5." In this way, it is possible to generate an image in which the fluctuation is suppressed by changing the parameter information. In step S706, an image reconstructed using the fluctuation parameter B1004 in which the degree of fluctuation is suppressed in this way is output.

[0094] In step S706, an image in which the degree of fluctuation is suppressed is generated by the above processing and output to the display control unit 108.

[0095] In step S707, the display control unit 108 switches the image displayed on the display device 128 from the image acquired by the image acquisition unit 101 to the image generated by the image reconstruction unit 107. Note that when a notification is received from the image difference calculation unit 110, an image that is the output result of step S706 is displayed, and when no notification is received, an image that is the output result of step S703 is displayed.

[0096] For example, if the Fig. When switching from image 9a before reconstruction shown in Figure 9 to image 9c, which is the result of reconstruction, the difference between the images is large due to the "composition" within the image to be reconstructed, so the user is likely to feel a sense of incongruity when switching the display of the images. On the other hand, when switching from image 9a to image 9b, in which the degree of fluctuation is suppressed, the difference between the images is small, so the user is less likely to feel a sense of incongruity. In addition, the reconstruction effects of certain fluctuation elements, such as the "facial expression" of the subject, become visually apparent.

[0097] In step S708, the recording unit 111 records the image generated by the image reconstruction unit 107 in step S703 on the hard disk 125, regardless of the determination result of the image difference calculation unit 110.

[0098] According to the first embodiment, as described above, when the degree of divergence between the images before and after reconstruction is greater than or equal to a predetermined divergence, an image in which the degree of fluctuation is suppressed is reconstructed and displayed. With such a configuration, it becomes possible to reduce the sense of incongruity that arises when displaying the preview image when reconstructing an image. <Zweites Ausführungsbeispiel>

[0099] A second embodiment of the present invention will be described below. Since the image processing apparatus of the present embodiment is similar in structure to that shown in Fig. 1 described image processing apparatus of the first embodiment, its description is omitted here.

[0100] Fig. Fig. 11 is a flowchart showing the display and recording control of an image after capture in the case of implementing image reconstruction in the present embodiment. It should be noted that in Fig. 11 similar processing operations as those in Fig. 7 of the first embodiment are provided with the same reference numerals and the description thereof is omitted.

[0101] In step S706, when the reconstruction of an image in which the degree of fluctuation is suppressed is completed, the image difference calculation unit 110 calculates the difference between the image acquired by the image acquisition unit 101 and the image generated by the image reconstruction unit 107 in step S706 in step S1101.

[0102] In step S1102, the image difference calculation unit 110 further determines whether the image acquired by the image acquisition unit 101 and the image generated by the image reconstruction unit 107 in step S706 diverge by more than or equal to a predetermined divergence, based on difference information between the images before and after reconstruction calculated in step S1101. If it is determined that the images before and after reconstruction diverge by more than or equal to the predetermined divergence, the display control unit 108 is notified, and processing proceeds to step S1103. If it is determined that the images before and after reconstruction diverge by less than the predetermined divergence, processing proceeds to step S707. Note that the predetermined divergence mentioned here may be the same threshold as in step S705 or a different threshold.

[0103] In step S1103, the display control unit 108 switches the image displayed on the display device 128 from the image acquired by the image acquisition unit 101 to an arbitrary image.

[0104] Here, the arbitrary image is, for example, an image with a given color such as black or an image 12d indicating that processing is in progress, as in Fig. 12. An example is shown in which image 12a is the image to be reconstructed, acquired in step S701, image 12c is the image reconstructed in step S703, and image 12b is the image reconstructed in step S706. In step S707, image 12d is used to reduce the sense of incongruity that occurs when switching the image displayed on the display device 128 from image 12a to image 12b. Image 12d can be any image that does not deviate from this usage application (purpose).

[0105] According to the second embodiment, as described above, when the degree of divergence between the images before and after generation is greater than or equal to a predetermined divergence, with respect to a newly generated image in which the degree of fluctuation is suppressed, an arbitrary image is displayed before displaying an image reconstructed for all fluctuation elements. With such a configuration, it becomes possible to reduce the sense of incongruity in the preview display when reconstructing an image. <Drittes Ausführungsbeispiel>

[0106] Next, a third embodiment of the present invention will be described. Since the image processing apparatus of the present embodiment is similar in structure to that shown in Fig. 1 described image processing apparatus of the first embodiment, its description is omitted here.

[0107] Fig. Fig. 13 is a flowchart showing the display and recording control of an image after capture in the case of implementing image reconstruction in the present embodiment. It should be noted that in Fig. 13 similar processing operations as in Fig. 7 of the first embodiment are provided with the same reference numerals and the description thereof is omitted.

[0108] If it is determined in step S705 that the image acquired by the image acquisition unit 101 in step S701 and the image generated by the image reconstruction unit 107 in step S703 diverge by more than or equal to a predetermined divergence, the image reconstruction unit 107 and the display control unit 108 are notified, and processing proceeds to step S1301. If it is determined that the images before and after reconstruction diverge by less than the predetermined divergence, processing proceeds to step S707.

[0109] In step S1301, the image reconstruction unit 107 performs image reconstruction after receiving notification from the image difference calculation unit 110. Here, the image reconstruction unit 107 generates a new image with a smaller degree of fluctuation than the image generated in step S703, using processing similar to the image reconstruction processing in step S703.

[0110] Here, the divergence between the images generated in step S703 before and after reconstruction and the process performed in step S1301 in which the image reconstruction unit 107 generates an image in which the degree of fluctuation is suppressed will be described with reference to Fig. 14 and Fig. 15 described. Fig. 14 shows the flow of processing for reconstructing an image intended for reconstruction based on the order of reconstruction processing determined by the fluctuation rule determining unit 106, and Fig. 15 shows an example of an image intended for reconstruction and an image generated by the reconstruction processing.

[0111] An image 1401 is an image that is to be reconstructed. The image in Fig. For example, image 15a shown in Figure 15 is image 1401 to be reconstructed.

[0112] In the reconstruction processing 1402, reconstruction is performed using a fluctuation model 1403 on the image 1401 by passing a fluctuation parameter 1404 to the fluctuation model 1403. As a result of the reconstruction processing, an image 1405-1 is then generated. For example, if the fluctuation model 1403 is "facial expression," an image 15b is generated by reconstructing the "facial expression" of the image 15a.

[0113] The image reconstruction unit 107 reconstructs the image by performing all reconstruction processing based on the order of reconstruction processing determined by the fluctuation rule determination unit 106. As a result, a reconstructed image 1406 is generated. For example, by performing reconstruction processing 1407 on the image 15b generated by the reconstruction processing 1402, an image 1405-2 is generated, and an image 15c that is the result of the reconstruction can be acquired. Similarly, by applying processing 1408 to the image 15c generated by the reconstruction processing 1407, an image 1406 can be generated, and an image 15d that is the result of the reconstruction can be acquired.

[0114] Note that the fluctuation rule determining unit 106 may determine the order of reconstruction processing for each fluctuation element, as shown in Fig. 15, or can determine the order for any arbitrary region. Alternatively, the order can be determined such that the fluctuation elements or the arbitrary regions are in the order from near to far, or in the order from far to near, or alternatively in ascending order of the difference between the images, or in descending order of the difference between the images.

[0115] In step S1302, the image difference calculation unit 110 compares the image generated by the image reconstruction unit 107 in step S703 with the image generated by the image reconstruction unit 107 in step S1301 and calculates the difference between the images before and after reconstruction.

[0116] In step S1303, the image difference calculation unit 110 also determines whether the image generated by the image reconstruction unit 107 in step S703 and the image generated by the image reconstruction unit 107 in step S1302 diverge by more than or equal to a predetermined divergence, based on the difference information between the images before and after reconstruction calculated in step S1302. If it is determined that the images before and after reconstruction diverge by more than or equal to the predetermined divergence, processing proceeds to step S1305. If it is determined that the images before and after reconstruction diverge by less than the predetermined divergence, the display control unit 108 is notified, and processing proceeds to step S707.

[0117] In step S1304, the display control unit 108 switches the image displayed on the display device 128 to the image generated by the image reconstruction unit 107 in step S1301 and returns to step S1301. This generates and displays images in which the degree of fluctuation gradually increases until the divergence between the image generated by the image reconstruction unit 107 in step S1303 and the image generated by the image reconstruction unit 107 in step S1302 becomes smaller than the prescribed divergence.

[0118] According to the third embodiment described above, images where the fluctuations between the images before and after reconstruction diverge are regenerated and displayed according to the degree of divergence between the images before and after reconstruction. With such a configuration, it is possible to reduce the sense of incongruity in the preview display when generating reconstructed images.

[0119] It should be noted that in the embodiments described above, a digital camera capable of generating images was described as an example of an image processing device. However, the present invention is not limited to devices capable of generating images and can be applied to devices capable of accepting images from an external device. For example, reconstruction processing can be performed on images captured by connecting the device to a camera, or on images stored on a server or in a cloud and captured via a network. In such cases, a configuration can be adopted in which the above-described processing described in the Fig. 7, Fig. 11 and Fig.13, is started in response to a recording instruction in an external device or an image input instruction from an external device. <Andere Ausführungsbeispiele>

[0120] It should be noted that the present invention can be applied to a system having multiple devices or to an apparatus having a single device.

[0121] The present invention may also be implemented by processing in which a program for implementing one or more functions of the above-described embodiments is delivered to a system or device via a network or storage medium, and one or more processors in the system or device's computer are caused to read and execute the program. The present invention may also be implemented by a circuit (e.g., an ASIC) for implementing one or more functions.

[0122] The present invention is not limited to the above embodiments, and various changes and modifications may be made without departing from the spirit or scope of the invention. Accordingly, the following claims are appended to provide the invention to the public.

[0123] This application claims priority to Japanese Patent Application No. 2022-182795, filed on November 15, 2022, which is hereby incorporated by reference in its entirety. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2005-204210A

[0006] JP 2014-127966A

[0006] JP 2022-182795

[0123]

Claims

[1] An image processing device, characterized by that it has: Image capturing means configured to capture a first image, fluctuation degree detecting means configured to detect a fluctuation degree of a fluctuation element having fluctuation, which is a variation of a state, among elements constituting the first image, Generation device configured to use the first image to generate a second image in which the degree of fluctuation of the fluctuation element is different from that in the first image, using a trained learning model; and Display control device arranged to control the display of an image on display devices; wherein, in a case where the first image and the second image diverge by more than or equal to a predetermined divergence, the generating means further uses the first image to generate a third image in which the degree of fluctuation is lower than in the second image, and the display control device carries out control to display the first image and subsequently the third image. [2] An image processing apparatus according to claim 1, characterized in that the display control unit performs control to display the first image and subsequently the third image. [3] The image processing apparatus according to claim 1, characterized byin a case where the first image and the third image diverge by more than or equal to a predetermined divergence, the display control means performs control to display the first image, subsequently an arbitrary image different from the first, second and third images, and to display the third image after displaying the arbitrary image. [4] The image processing apparatus according to claim 3, characterized by that the arbitrary image includes at least one of an image having a predetermined color and an image indicating that processing by the generating means is in progress. [5] The image processing apparatus according to claim 1, characterized byin a case where the second image and the third image diverge by more than or equal to a predetermined divergence, the generating means further uses the first image to increase the degree of fluctuation of the fluctuation element and again generate a third image in which the degree of fluctuation of the fluctuation element is less than in the second image, and the display controlling means performs control to display the first image and subsequently the third images in the order of their generation. [6] The image processing apparatus according to claim 4, characterized by that the generating device generates the second image by performing reconstruction processing that uses a variety of fluctuation models as a learning model, and generates the third image by performing reconstruction processing for each of the fluctuation elements or for each of the predetermined regions. [7] The image processing apparatus according to claim 5, characterized by that the generating means generates the third image by performing the reconstruction processing of the fluctuation elements or the predetermined areas in the order from the near area to the far area or from the far area to the near area. [8] The image processing apparatus according to claim 6, characterized by that the generating means generates the third image by performing reconstruction processing of the fluctuation elements or the predetermined regions in ascending order of divergence or in descending order of divergence between the first image and the second image. [9] The image processing apparatus according to any one of claims 1 to 8, characterized by that it also has: a recording device for recording an image on a recording medium, wherein the recording device records the second image and does not record the third image. [10] The image processing apparatus according to any one of claims 1 to 9, characterized by that it also has: a determining means for determining a divergence state between images based on a difference in the degree of fluctuation of the same fluctuation element between the images. [11] The image processing apparatus according to any one of claims 1 to 9, characterized by that it also has: Determining device for determining a divergence state between images by an interframe difference method. [12] The image processing apparatus according to any one of claims 1 to 11, characterized by that the generating device generates the second image by performing reconstruction processing using a plurality of fluctuation models as a learning model, and the third image is generated by reducing the number of fluctuation models used from the multitude of fluctuation models. [13] The image processing apparatus according to any one of claims 1 to 12, characterized by that the generating device generates the second image by performing reconstruction processing using a variety of fluctuation models as a learning model, and the third image is generated by changing a parameter that indicates the degree of fluctuation of the fluctuation model. [14] The image processing apparatus according to any one of claims 1 to 13, characterized bythat the display control device performs the display in response to detection of the first image and the generation device generates the second image in response to detection of the first image. [15] An image processing method comprising: an image capturing step for capturing a first image; a fluctuation degree detecting step of detecting a fluctuation degree of a fluctuation element having fluctuation, which is variation of a state, among elements constituting the first image; a first generation step in which the first image is used to generate a second image in which the degree of fluctuation of the fluctuation element differs from the first image, using a trained learning model; and a second generation step in which, in a case where the first image and the second image diverge by more than or equal to a predetermined divergence, the first image is used to generate a third image in which the degree of fluctuation of the fluctuation element is lower than that of the second image; and a display control step of performing, in a case where the first image and the second image diverge by more than or equal to the predetermined divergence, control for displaying the first image and subsequently displaying the third image on a device. [16] A computer program for causing a computer to function as a device of the image processing apparatus according to any one of claims 1 to 14. [17] A computer-readable storage medium storing the computer program of claim 16.

Images