Image processing apparatus, control method of image processing apparatus, storage medium, and program product

Abstract

The invention relates to an image processing apparatus, a control method of the image processing apparatus, a storage medium, and a program product. The image processing apparatus includes: a conversion unit configured to convert an input image to acquire an output image; a calculation unit configured to calculate a similarity between the first input image and the second input image; and a determination unit configured to determine whether the second input image is an image captured in the same scene as the first input image based on the similarity, in a case where it is determined that the second input image is an image captured in the same scene, to determine whether the second input image is an image captured in the same scene. The conversion unit converts the second input image to acquire a second output image by using data based on a first output image converted by the conversion unit from the first input image.

Description

Technical Field

[0001] This disclosure relates to image processing equipment, control methods for image processing equipment, storage media, and program products. Background Technology

[0002] In recent years, generative artificial intelligence (generative AI) technology has rapidly gained popularity. In particular, in the field of image generation, it is easy to generate specified images and moving images, thus becoming more widely used in various fields such as corporate promotion and film production, in addition to personal content creation.

[0003] In image generation using generative AI, text-to-image generation based on character input and image-to-image generation based on image input are known as representative patterns. Image-to-image generation can alter a portion of the subject in the input image and generate a new perspective not present in the input image. In this case, images closely resembling the user's intent can be generated by inputting instruction data indicating the desired changes along with the image.

[0004] However, since the generated images are ultimately produced by AI, the output cannot be completely predicted. Even with identical input images and instruction data, the generated result can vary between the first and second attempts. When inputting images from a series of consecutively captured images of the same scene into a generative AI for purposes such as widening the field of view or altering the background's color perception, the position, size, appearance, color perception, brightness, etc., of the newly generated subject or background may differ from image to image. Therefore, there is a concern that consistency may be lost when the output image is viewed as a whole scene. For example, when inputting a series of images containing several pictures of birds flying in the sky into a generative AI for the purpose of changing the sky from cloudy to sunny, the position and size of the sun added to the sky may vary from image to image.

[0005] As a method for suppressing variations in the output of image processing, Japanese Patent Application Laid-Open No. 10-290469 discloses a method for determining the white balance of current image data by weighting the similarity between previously acquired image data and the latest image data. Japanese Patent Application Laid-Open No. 2013-192057 discloses a method for suppressing brightness and color variations between images in image processing involving multiple consecutive image captures, such as automatic exposure bracketing (AE bracketing) and HDR shooting.

[0006] However, the techniques described in Japanese Patent Application Publication No. 10-290469 and Japanese Patent Application Publication No. 2013-192057 only reduce variations in the output of image processing related to brightness and color perception, such as color correction processing and white balance processing. For this reason, they cannot reduce variations in image processing used for generating images with randomness, such as generative AI. Summary of the Invention

[0007] This disclosure is made in view of the above-mentioned problems and provides a technique for suppressing variations in the output of image generation.

[0008] According to one aspect of this disclosure, an image processing apparatus is provided, comprising: a conversion unit configured to convert an input image to obtain an output image; a calculation unit configured to calculate a similarity between a first input image and a second input image; and a determination unit configured to determine, based on the similarity, whether the second input image is an image taken in the same scene as the first input image, wherein, if it is determined that the second input image is an image taken in the same scene, the conversion unit converts the second input image to obtain a second output image by using data based on a first output image obtained by converting the first input image by the conversion unit.

[0009] According to one aspect of this disclosure, a control method for an image processing apparatus is provided, the image processing apparatus being used to convert an input image to obtain an output image, the control method comprising: calculating a similarity between a first input image and a second input image; determining, based on the similarity, whether the second input image is an image taken in the same scene as the first input image; and, if it is determined that the second input image is an image taken in the same scene, converting the second input image to obtain a second output image by using data based on a first output image converted from the first input image.

[0010] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is given by way of example. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the embodiments.

[0012] Figure 1 This is a view illustrating an example of the hardware configuration of an image processing device according to one embodiment.

[0013] Figure 2This is a view illustrating an example of the functional configuration of an image processing device according to a first embodiment.

[0014] Figure 3 This is an operational illustration of the image processing apparatus according to the first embodiment.

[0015] Figure 4 This is a flowchart illustrating the processing procedure performed by the image processing apparatus according to the first embodiment.

[0016] Figure 5 This is a view illustrating an example of the functional configuration of an image processing device according to a second embodiment.

[0017] Figure 6 This is an operational illustration of the image processing apparatus according to the second embodiment.

[0018] Figure 7 This is a flowchart illustrating the processing procedure performed by the image processing apparatus according to the second embodiment.

[0019] Figure 8 This is a view illustrating an example of the functional configuration of an image processing device according to a third embodiment.

[0020] Figure 9 This is an operational illustration of the image processing apparatus according to the third embodiment.

[0021] Figure 10 This is a flowchart illustrating the processing procedure performed by the image processing apparatus according to the third embodiment. Detailed Implementation

[0022] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are required, and several such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions are omitted.

[0023] First Embodiment

[0024] Hardware configuration

[0025] First, refer to Figure 1 An example of the hardware configuration of the image processing apparatus according to this embodiment is described. The image processing apparatus 10 includes a central processing unit (CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13, an auxiliary storage device 14, a display unit 15, an operation unit 16, a communication I / F 17, and a bus 18.

[0026] CPU 11 controls the entire image processing device 10 by using computer programs and data stored in ROM 12 and RAM 13. Figure 1 The image processing device 10 shown illustrates various functions. Note that the image processing device 10 may include one or more dedicated hardware components different from the CPU 11, and at least a portion of the processing performed by the CPU 11 may be executed by the dedicated hardware. Examples of dedicated hardware include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and digital signal processors (DSPs). ROM 12 stores programs that do not need to be changed. RAM 13 temporarily stores programs and data supplied from auxiliary storage device 14 and data supplied externally via communication I / F 17. Auxiliary storage device 14 includes, for example, a hard disk drive and stores various types of data such as image data and audio data.

[0027] Display unit 15 includes, for example, a liquid crystal display or an LED, and displays a graphical user interface (GUI) for user operation of image processing device 10. Operation unit 16 includes, for example, a keyboard, mouse, joystick, and touchpad, and inputs various instructions to CPU 11 in response to user operations. Communication I / F 17 is used for communication with devices external to image processing device 10. For example, if image processing device 10 is wired to an external device, a communication cable is connected to communication I / F 17. If image processing device 10 has the function of wireless communication with external devices, communication I / F 17 includes an antenna. Bus 18 connects the various units of image processing device 10 to transmit information.

[0028] This embodiment assumes that the display unit 15 and the operation unit 16 are located inside the image processing device 10; however, at least one of the display unit 15 and the operation unit 16 can exist as another device outside the image processing device 10. In this case, the CPU 11 can operate as a display control unit that controls the display unit 15 and an operation control unit that controls the operation unit 16.

[0029] Functional Configuration

[0030] Next, we will refer to Figure 2 An example of the functional configuration of the image processing apparatus according to this embodiment is described. The image processing apparatus 10 includes an image input unit 201, a similarity calculation unit 202, a similar scene determination unit 203, an instruction data acquisition unit 204, an image conversion unit 205, and an image output unit 206.

[0031] Image input unit 201 acquires an input image. Here, the input image is an image acquired by a digital camera, smartphone, tablet terminal, or any other device capable of taking pictures, and is, for example, one of a group of images taken in the same scene.

[0032] The similarity calculation unit 202 calculates the similarity between the input image input to the image input unit 201 in the previous processing using generative AI and the input image input to the image input unit 201 in the current processing using generative AI. The similarity can be calculated based on information related to the capture time of the previous input image and the capture time of the current input image, and / or any statistical measure obtained from the previous input image and any statistical measure obtained from the current input image. For example, the similarity can be calculated based on the difference in capture time between the images. Statistical measures of the image brightness values ​​can be compared as statistical measures, and the similarity can be calculated based on their differences. Furthermore, existing methods disclosed in Japanese Patent Application Publication No. 10-290469 and Japanese Patent Application Publication No. 2013-192057 can be used to calculate the similarity.

[0033] The same scene determination unit 203 determines whether the previous input image and the current input image were taken in the same scene based on the similarity calculated by the similarity calculation unit 202. For example, if the similarity calculated by the similarity calculation unit 202 is equal to or greater than a threshold, it can be determined that the previous input image and the current input image were taken in the same scene.

[0034] The instruction data acquisition unit 204 acquires instruction data indicating the user's desired transformation content. Instruction data is data indicating a user request related to a change in the input image and is referred to as a prompt. For example, instruction data can be text information such as "clear blue sky" or "sun and clouds." By inputting such instruction data as a prompt when a bird appears in the input image, it is possible to generate an image where the background is changed to a clear blue sky while the bird remains unchanged, even when a bird flying in a cloudy sky appears in the input image.

[0035] Based on the determination result of the same scene determination unit 203, the image conversion unit 205 performs image conversion on the input image using the instruction data acquired by the instruction data acquisition unit 204 or using the output image itself generated in the previous process. Details of the processing by the image conversion unit 205 will be described later. Note that the image conversion processing according to this embodiment assumes processing for converting at least a portion of the input image using generative AI and performing image generation with randomness. Examples of processing for converting at least a portion include various examples of changing the size, shape, color, brightness, etc., of subjects appearing in the input image, deleting or adding arbitrary subjects, and changing the background of the image. Other examples include increasing the viewpoint of the input image to generate a viewpoint portion that does not yet exist and creating an output image.

[0036] The image output unit 206 outputs the output image converted and generated by the image conversion unit 205 to the display unit 15 and other units.

[0037] Description of processing

[0038] Here, we will refer to Figure 3 Detailed description Figure 2 The processing flow of the image processing device 10 described herein. Figure 3 The upper part represents the previous processing, and the lower part represents the current processing. In the previous processing, the previous input image 310 was input to the image input unit 201 and acquired by the similarity calculation unit 202. Here, the similarity between the previous input image 310 and the previous input image is calculated. Then, the same scene determination unit 203 determines that the previous input image 310 and the previous input image are not images taken in the same scene. In this case, the image conversion unit 205 performs image conversion based on the previous instruction data 350 acquired by the instruction data acquisition unit 204, and the image output unit 206 outputs the previous output image 340.

[0039] Then, in the current processing, the current input image 300 is input to the image input unit 201 and acquired by the similarity calculation unit 202. At the same time, the previous input image 310 is also input to the image input unit 201 and acquired by the similarity calculation unit 202. The similarity calculation unit 202 determines the similarity between the current input image 300 and the previous input image 310. Then, if the same scene determination unit 203 determines that the current input image 300 and the previous input image 310 are not images taken in the same scene, the instruction data acquisition unit 204 acquires instruction data 320 indicating the conversion content expected by the user. Then, the image conversion unit 205 performs image conversion based on the instruction data 320 acquired by the instruction data acquisition unit 204, and the image output unit 206 outputs the output image 330.

[0040] On the other hand, if the same scene determination unit 203 determines that the current input image 300 and the previous input image 310 are images taken in the same scene, the input to the image conversion unit 205 is switched. Specifically, the previous output image 340 is input as a replacement for the instruction data 320. The image conversion unit 205 performs image conversion by using the previous output image 340 as a replacement for the prompt, and the image output unit 206 outputs the output image 330. For example, if the previous output image 340 is a bird image with a background that has been converted to a clear blue sky, using the previous output image 340 as a replacement for the prompt will also produce similar conversion content in the current output image 330 (similar position of the sun in the image, degree of blue sky, etc.).

[0041] Even when the same instruction data (cue) is used each time, for example, even if the instruction data is the same "clear blue sky" in both the previous and current instances, there are still cases where the position of the sun in the output image differs between the previous and current images, or the color perception of the blue sky differs between the two. On the other hand, using the previous output image 340 as a cue can suppress such variations in the generated result. Therefore, for example, when performing image conversion on a group of images of the same scene obtained by continuously shooting birds in flight, it is possible to reduce the loss of consistency caused by changes in the position of the sun, the position of the clouds, the clarity of the sky, the color of the sky, etc., for each image.

[0042] Processing flow

[0043] Figure 4 This is a flowchart illustrating the processing procedure performed by the image processing apparatus according to this embodiment. According to this embodiment, the processing involves the CPU 11 reading and executing a computer program stored in ROM 12 or RAM 13 to perform... Figure 2 The functions of the image processing device 10 described herein are implemented according to the functional block diagram.

[0044] In S401, the image input unit 201 acquires the current input image 300 and the previous input image 310. In S402, the similarity calculation unit 202 calculates the similarity between the current input image 300 and the previous input image 310.

[0045] In S403, the same scene determination unit 203 determines whether the current input image 300 and the previous input image 310 were taken in the same scene. If the same scene determination unit 203 determines that they are taken in the same scene, the process proceeds to S404. On the other hand, if the same scene determination unit 203 determines that they are not taken in the same scene, the process proceeds to S405.

[0046] In S404, the image input unit 201 acquires the previous output image 340. In S405, the instruction data acquisition unit 204 acquires instruction data 320 indicating the conversion content desired by the user.

[0047] In S406, the image conversion unit 205 performs image conversion on the current input image 300. If it is determined in S403 that the images were not taken in the same scene, the image conversion unit 205 performs image conversion based on the instruction data 320 acquired by the instruction data acquisition unit 204. Conversely, if it is determined in S403 that the images were taken in the same scene, the image conversion unit 205 uses the previous output image 340 as a substitute for the instruction data 320 during image conversion.

[0048] In S407, the image output unit 206 outputs the output image 330 after image conversion in S406. The above is... Figure 4 A series of processes within.

[0049] As described above, according to this embodiment, when image processing for generating random images is applied to a group of images taken in the same scene, variations in the output can be reduced.

[0050] Second Embodiment

[0051] In this embodiment, an example will be described as follows: image conversion is performed by generating instruction data from the previous output image instead of using it as a prompt, and then using the generated instruction data as a prompt. Since the hardware configuration of the image processing device according to this embodiment is similar to that of the first embodiment, its description will be omitted.

[0052] Functional Configuration

[0053] Reference Figure 5 An example of the functional configuration of the image processing apparatus according to this embodiment is described. In addition to the image input unit 201, similarity calculation unit 202, same scene determination unit 203, instruction data acquisition unit 204, image conversion unit 205 and image output unit 206 described in the first embodiment, the image processing apparatus 50 also includes an instruction data generation unit 501.

[0054] The instruction data generation unit 501 automatically generates generation instruction data from the previous output image. The generation instruction data includes image modification instructions to ensure consistency between the output image after image conversion from the current input image and the previous output image. For example, instruction data can be generated that specifies in detail the position and size of the sun, the color of the blue sky, the shape of the clouds, etc.

[0055] Description of processing

[0056] Here, we will refer to Figure 6 Detailed description Figure 5 The processing flow of the image processing device 50 described herein. Figure 6The upper part represents the previous processing, and the lower part represents the current processing. In the previous processing, the previous input image 310 was input to the image input unit 201 and acquired by the similarity calculation unit 202. Here, the similarity between the previous input image 310 and the previous input image is calculated. Then, the same scene determination unit 203 determines that the previous input image 310 and the previous input image are not images taken in the same scene. In this case, the image conversion unit 205 performs image conversion based on the previous instruction data 350 acquired by the instruction data acquisition unit 204, and the image output unit 206 outputs the previous output image 340. The processing so far is similar to Figure 3 In the previous embodiment, the instruction data generation unit 501 then acquires the previous output image 340 and generates generation instruction data 600 based on the previous output image 340.

[0057] Then, in the current processing, the current input image 300 is input to the image input unit 201 and acquired by the similarity calculation unit 202. At this time, the previous input image 310 is also input to the image input unit 201 and acquired by the similarity calculation unit 202. The similarity calculation unit 202 determines the similarity between the current input image 300 and the previous input image 310. Then, if the same scene determination unit 203 determines that the current input image 300 and the previous input image 310 are not images taken in the same scene, the instruction data acquisition unit 204 acquires instruction data 320 indicating the conversion content expected by the user. Then, the image conversion unit 205 performs image conversion based on the instruction data 320 acquired by the instruction data acquisition unit 204, and the image output unit 206 outputs the output image 330. The processing up to this point is similar to... Figure 3 The processing in the middle.

[0058] On the other hand, if the same scene determination unit 203 determines that the current input image 300 and the previous input image 310 were taken in the same scene, the input to the image conversion unit 205 is switched. Specifically, the input is the generation instruction data 600 acquired by the instruction data acquisition unit 204.

[0059] Image conversion unit 205 performs image conversion using generation instruction data 600 as a prompt, and image output unit 206 outputs output image 330. For example, if the previous output image 340 was a bird image with a background that has been converted to a clear blue sky, using generation instruction data 600 as a prompt will also produce similar conversion content in the current output image 330 (e.g., similar position of the sun in the image, similar degree of blue sky, etc.). Using generation instruction data 600 generated from the previous output image 340 as a prompt can suppress variations in the generated result.

[0060] Processing flow

[0061] Figure 7 This is a flowchart illustrating the processing procedure performed by the image processing apparatus according to this embodiment. The same step numbers are assigned to the reference. Figure 4 The process described herein is the same as the process described above, and its detailed description will be omitted. According to this embodiment, the process involves the CPU 11 reading and executing a computer program stored in ROM 12 or RAM 13 to perform... Figure 5 The functions of the image processing device 50 described herein are implemented according to the functional block diagram.

[0062] In this embodiment, if the same scene determination unit 203 determines in S403 that the images are taken in the same scene, the process proceeds to S701. In S701, the instruction data generation unit 501 acquires the previous output image 340 and generates generation instruction data 600 based on the previous output image 340. In S702, the instruction data acquisition unit 204 acquires the generation instruction data 600 generated by the instruction data generation unit 501. Afterwards, the process proceeds to S703.

[0063] In S703, the image conversion unit 205 performs image conversion on the current input image 300. If in S403 it is determined that the images were not taken in the same scene, the image conversion unit 205 performs image conversion based on the instruction data 320 acquired by the instruction data acquisition unit 204. Conversely, if in S403 it is determined that the images were taken in the same scene, the image conversion unit 205 uses the generation instruction data 600 generated from the previous output image 340 as a prompt to perform image conversion instead of the instruction data 320.

[0064] As described above, in this embodiment, under the condition of the same scene, instruction data (hints) is generated from the output image that is the result of the previous generation, and the generated instruction data is used to perform image transformation. This can reduce the variation of the output results when image processing for random image generation is applied to a group of images taken in the same scene.

[0065] Third Embodiment

[0066] In this embodiment, an example will be described as follows: image conversion is performed by generating depth information from the previous output image instead of using it as a cue, and using the generated depth data as a cue. Since the hardware configuration of the image processing device according to this embodiment is similar to that of the first embodiment, its description will be omitted.

[0067] Functional Configuration

[0068] Reference Figure 8 An example of the functional configuration of the image processing apparatus according to this embodiment is described. In addition to the image input unit 201, similarity calculation unit 202, same scene determination unit 203, instruction data acquisition unit 204, image conversion unit 205 and image output unit 206 described in the first embodiment, the image processing apparatus 80 also includes a depth information acquisition unit 801.

[0069] The depth information acquisition unit 801 acquires depth data 900 from the previous output image 340. Using depth data (depth map) during image conversion helps maintain the shape and positional relationships of the subject. The depth data 900 includes information for matching the positional relationships of the subject in the converted region of the output image 330, which is the result of the image conversion of the current input image 300, with the positional relationships in the previous output image 340. The depth data 900 can be acquired from the previous output image 340 itself, or it can be acquired based on metadata such as that of the previous output image 340.

[0070] Description of processing

[0071] Here, we will refer to Figure 9 Detailed description Figure 8 The processing flow of the image processing device 80 described herein. Figure 9 The upper part represents the previous processing, and the lower part represents the current processing. In the previous processing, the previous input image 310 was input to the image input unit 201 and acquired by the similarity calculation unit 202. Here, the similarity between the previous input image 310 and the previous input image is calculated. Then, the same scene determination unit 203 determines that the previous input image 310 and the previous input image are not images taken in the same scene. In this case, the image conversion unit 205 performs image conversion based on the previous instruction data 350 acquired by the instruction data acquisition unit 204, and the image output unit 206 outputs the previous output image 340. The processing up to this point is similar to... Figure 3 In the previous embodiment, the depth information acquisition unit 801 acquired the previous output image 340 and generated depth data 900 based on the previous output image 340.

[0072] Then, in the current processing, the current input image 300 is input to the image input unit 201 and acquired by the similarity calculation unit 202. At this time, the previous input image 310 is also input to the image input unit 201 and acquired by the similarity calculation unit 202. The similarity calculation unit 202 determines the similarity between the current input image 300 and the previous input image 310. Then, if the same scene determination unit 203 determines that the current input image 300 and the previous input image 310 are not images taken in the same scene, the instruction data acquisition unit 204 acquires instruction data 320 indicating the conversion content expected by the user. Then, the image conversion unit 205 performs image conversion based on the instruction data 320 acquired by the instruction data acquisition unit 204, and the image output unit 206 outputs the output image 330. The processing up to this point is similar to... Figure 3 The processing in the middle.

[0073] On the other hand, if the same scene determination unit 203 determines that the current input image 300 and the previous input image 310 were taken in the same scene, the input to the image conversion unit 205 is switched. Specifically, the depth data 900 acquired by the depth information acquisition unit 801 is input.

[0074] Image conversion unit 205 performs image conversion by using depth data 900 as a cue, and image output unit 206 outputs output image 330. For example, if the previous output image 340 is a bird image with a background that has been converted to a clear blue sky, using depth data 900 as a cue will also produce similar conversion content in the current output image 330 (e.g., the position of the sun in the image is similar). Using depth data 900 generated from the previous output image 340 as a cue can produce similar positional relationships of the subject, and thus can suppress variations in the generated result.

[0075] Processing flow

[0076] Figure 10 This is a flowchart illustrating the processing procedure performed by the image processing apparatus according to this embodiment. The same step numbers are assigned to the reference. Figure 4 The process described herein is the same as the process described above, and its detailed description will be omitted. According to this embodiment, the process involves the CPU 11 reading and executing a computer program stored in ROM 12 or RAM 13 to perform... Figure 8 The functions of the image processing device 80 described herein are implemented according to the functional block diagram.

[0077] In this embodiment, if the same scene determination unit 203 determines in S403 that the images are taken in the same scene, the process proceeds to S1001. In S1001, the depth information acquisition unit 801 acquires the previous output image 340 and generates depth data 900 based on the previous output image 340. In S1002, the instruction data acquisition unit 204 acquires the depth data 900 generated by the depth information acquisition unit 801. Thereafter, the process proceeds to S1003.

[0078] In S1003, the image conversion unit 205 performs image conversion on the current input image 300. If in S403 it is determined that the images were not taken in the same scene, the image conversion unit 205 performs image conversion based on the instruction data 320 acquired by the instruction data acquisition unit 204. Conversely, if in S403 it is determined that the images were taken in the same scene, the image conversion unit 205 uses the depth data 900 acquired from the previous output image 340 instead of the instruction data 320 for image conversion.

[0079] As described above, in this embodiment, in the case of the same scene, depth data (hint) is generated from the output image that is the result of the previous generation, and the generated depth data is used to perform image transformation. This can reduce the variation of the output when image processing for random image generation is applied to a group of images taken in the same scene.

[0080] [Variation Example]

[0081] The same scene determination process described in the first to third embodiments above can be in the following form: enabling switching between whether to perform the process after adding the processing execution conditions. That is, the same scene determination process can be performed when predetermined conditions are met. For example, it can be applied in the form of performing the process only when the user desires it, or in the form of performing the process limited to images captured in any shooting mode, such as image data during continuous shooting. For the user-desired situation, for example, when the user receives input instructing the same scene determination process to be performed, this process can be performed.

[0082] In the first to third embodiments, the image processing device has been described as an example, but this disclosure can be implemented in any electronic device. It may also include personal computers, tablet terminals, mobile phones, smartphones, digital cameras, digital video cameras, etc. Furthermore, it includes transmissive goggles used in game consoles, augmented reality (AR), mixed reality (MR), etc., but this disclosure is not limited to them. In particular, when this disclosure is applied to a device capable of taking pictures through the main body of a device such as a digital camera or mobile phone, this disclosure can be applied not only to post-editing processing of images, but also to the form of simultaneously recording the captured image and the output of the image conversion unit during shooting.

[0083] In the above embodiments, an example of calculating the similarity between the current input image and the previous input image has been described; however, this disclosure is not limited to this example. The similarity between the current input image and input images taken before the current input image can be calculated. That is, the input image is not limited to the previous input image, and input images preceding the previous input image can be used.

[0084] According to the above embodiments, when it is determined that the images were taken in the same scene, the current input image is transformed using data from the previous output image after the previous input image has been transformed to obtain the current output image. This can reduce changes in the position, size, appearance, color, brightness, etc. of newly generated subjects or backgrounds in the output image when image transformation is applied to a group of images taken in the same scene.

[0085] According to this disclosure, it is possible to suppress variations in the output of image generation.

[0086] (Other embodiments)

[0087] The embodiments of the present invention can also be implemented by the following method: providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0088] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: A conversion unit configured to convert an input image to obtain an output image; A computing unit configured to calculate the similarity between a first input image and a second input image; as well as A determining unit is configured to determine, based on the similarity, whether the second input image was taken in the same scene as the first input image. Wherein, if it is determined that the second input image is an image taken in the same scene, the conversion unit converts the second input image to obtain the second output image by using data based on the first output image after the first input image has been converted by the conversion unit.

2. The image processing apparatus according to claim 1, wherein, If it is determined that the second input image is not an image taken in the same scene, the conversion unit uses instruction data indicating a user request related to a change in the input image to convert the second input image to obtain a second output image.

3. The image processing apparatus according to claim 1, wherein, The data based on the first output image is the first output image itself, instruction data generated from the first output image related to changes in the input image, or depth data obtained from the first output image.

4. The image processing apparatus according to claim 3, wherein, Instruction data generated from the first output image, relating to changes in the input image, indicates image change instructions to ensure consistency between the second output image and the first output image.

5. The image processing apparatus according to claim 1, wherein, The determining unit makes a determination when predetermined conditions are met.

6. The image processing apparatus according to claim 5, wherein, The conditions for meeting the predetermined conditions include receiving input from the user to instruct a determination.

7. The image processing apparatus according to claim 5, wherein, The conditions that satisfy the predetermined conditions include the second input image and the first input image being images acquired through continuous shooting.

8. The image processing apparatus according to claim 1, wherein, The first input image is an image taken before the second input image.

9. The image processing apparatus according to claim 2, wherein, The instruction data is a prompt.

10. The image processing apparatus according to claim 1, wherein, The conversion unit performs image conversions with randomness.

11. The image processing apparatus according to claim 1, wherein, The conversion unit uses generative artificial intelligence (AI) to perform image conversion.

12. A control method for an image processing device, the image processing device being used to convert an input image to obtain an output image, the control method comprising: Calculate the similarity between the first input image and the second input image; Based on the similarity, it is determined whether the second input image was taken in the same scene as the first input image; as well as If it is determined that the second input image is an image taken in the same scene, the second input image is converted to obtain a second output image by using data based on the first output image after the first input image has been converted.

13. A computer-readable storage medium storing a program that causes a computer to perform the control method of the image processing apparatus according to claim 12.

14. A program product storing a program for causing a computer to execute a control method for an image processing apparatus according to claim 12.

Images