Image processing device, imaging device, image processing method, and program

Abstract

The present invention provides an image processing apparatus, an imaging apparatus, an image processing method, and a program that can obtain an image in which the influence of the first AI processing is less noticeable than that of the first image obtained by performing the first AI processing on the image to be processed. [Solution] The image processing device includes a processor. The processor obtains a first image obtained by performing a first AI processing on the image to be processed, and a second image obtained without performing the first AI processing on the image to be processed, and adjusts for any excess or deficiency of the first AI processing by combining the first image and the second image.

Description

Technical Field

[0001] The technology of the present disclosure relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program.

Background Art

[0002] Patent Document 1 discloses an image processing system including a processing unit that performs processing on an input image input to an input layer using a neural network having an input layer, an output layer, and an intermediate layer provided between the input layer and the output layer, and an adjustment unit that adjusts at least one internal parameter of one or more nodes included in the intermediate layer, which is an internal parameter calculated by learning, based on data related to the input image when performing processing after learning.

[0003] Further, in the image processing system described in Patent Document 1, the input image is an image including noise, and the input image is subjected to processing performed by the processing unit to remove or reduce noise from the input image.

[0004] Further, in the image processing system described in Patent Document 1, the neural network includes a first neural network, a second neural network, and a division unit that divides an input image into a high-frequency component image and a low-frequency component image, inputs the high-frequency component image to the first neural network, and inputs the low-frequency component image to the second neural network, and a synthesis unit that synthesizes a first output image output from the first neural network and a second output image output from the second neural network. The adjustment unit adjusts the internal parameters of the first neural network based on data related to the input image, while not adjusting the internal parameters of the second neural network.

[0005] Furthermore, Patent Document 1 discloses an image processing system comprising a processing unit that generates a noise-reduced output image from an input image using a neural network, and an adjustment unit that adjusts the internal parameters of the neural network according to the imaging conditions of the input image.

[0006] Patent Document 2 discloses a medical image processing device comprising: an acquisition unit that acquires a first image, which is a medical image of a predetermined part of a subject; an image enhancement unit that generates a second image, which is of higher quality than the first image, from the first image using an image enhancement engine including a machine learning engine; and a display control unit that displays a composite image obtained by combining the first image and the second image according to a ratio obtained using information about at least a part of the region of the first image on a display unit.

[0007] Patent Document 3 discloses an electronic device that includes a memory for storing at least one instruction word, and a processor electrically connected to the memory, which executes the instruction word to obtain a noise map indicating the quality of the input image from the input image, applies the input image and the noise map to a learning network model including multiple layers, and obtains an output image with improved quality of the input image, wherein the processor provides the noise map to at least one intermediate layer among the multiple layers, and the learning network model is a trained artificial intelligence model obtained by learning the relationship between multiple sample images, a noise map for each sample image, and the original image for each sample image through an artificial intelligence algorithm. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2018-206382 [Patent Document 2] Japanese Patent Publication No. 2020-166814 [Patent Document 3] Japanese Patent Publication No. 2020-184300 [Overview of the project]

[0009] One embodiment of the technology of this disclosure provides an image processing apparatus, an imaging apparatus, an image processing method, and a program that can obtain an image in which the influence of the first AI processing is less noticeable than a first image obtained by performing the first AI processing on an image to be processed. [Means for solving the problem]

[0010] A first aspect of the technology of this disclosure is an image processing device comprising a processor, the processor acquiring a first image obtained by performing a first AI processing on an image to be processed, and a second image obtained without performing the first AI processing on the image to be processed, and adjusting the excess or deficiency of the first AI processing by combining the first image and the second image.

[0011] A second aspect of the technology of this disclosure is an image processing apparatus according to the first aspect, wherein the second image is an image obtained by performing a non-AI method of processing on the image to be processed, which does not use a neural network.

[0012] A third aspect of the technology of this disclosure is an image processing device comprising a processor, the processor acquiring a first image obtained by adjusting the non-noise elements of the image to be processed by performing a first AI processing on the image to be processed, and a second image obtained without performing the first AI processing on the image to be processed, and adjusting the non-noise elements by combining the first image and the second image.

[0013] A fourth aspect of the technology of this disclosure is an image processing apparatus according to the third aspect, wherein the second image is an image in which non-noise elements have been adjusted by performing a non-AI method of processing on the image to be processed, without using a neural network.

[0014] A fifth aspect of the technology of this disclosure is an image processing apparatus according to the third aspect, wherein the second image is an image in which no non-noise elements have been adjusted.

[0015] A sixth aspect of the technology of this disclosure is an image processing apparatus according to any one of the first to fifth aspects, wherein the processor combines a first image and a second image at a ratio that adjusts for any excess or deficiency of the first AI processing.

[0016] A seventh aspect of the technology of this disclosure is an image processing apparatus according to a sixth aspect, wherein the image to be processed is an image obtained by imaging by an imaging device, the first AI processing includes a first correction processing that corrects phenomena appearing in the image to be processed due to the characteristics of the imaging device using an AI method, the first image includes a first corrected image obtained by performing the first correction processing, and the processor adjusts elements derived from the first correction processing by combining the first corrected image and the second image in proportion.

[0017] An eighth aspect of the technology of this disclosure is an image processing apparatus according to a seventh aspect, wherein the processor performs a second correction process to correct a phenomenon in a non-AI manner, the second image includes a second corrected image obtained by the second correction process, and the processor adjusts elements derived from the first correction process by combining the first corrected image and the second corrected image in proportion.

[0018] A ninth aspect of the technology of this disclosure is an image processing apparatus according to the sixth or eighth aspect, wherein the characteristics include the optical characteristics of an imaging device.

[0019] A tenth aspect of the technology of this disclosure is an image processing apparatus according to any one of the sixth to ninth aspects, wherein the first AI processing includes a first modification processing that modifies factors governing the visual impression given by an image to be processed in an AI manner, the first image includes a first modified image obtained by performing the first modification processing, and the processor adjusts the elements derived from the first modification processing by combining the first modified image and the second image in proportion.

[0020] An eleventh aspect of the technology of this disclosure is an image processing apparatus according to a tenth aspect, wherein the processor performs a second modification process that modifies factors in a non-AI manner, the second image includes a second modified image obtained by the second modification process, and the processor adjusts elements derived from the first modification process by combining the first modified image and the second modified image in proportion.

[0021] A twelfth aspect of the technology of this disclosure is an image processing apparatus according to the tenth or eleventh aspect, wherein the factors include clarity, color, gradation, resolution, blur, edge enhancement, style, and / or skin-related image quality.

[0022] A thirteenth aspect of the technology of this disclosure is an image processing device according to any one of the sixth to twelfth aspects, wherein the image to be processed is an image obtained by imaging the subject light that is imaged on a light-receiving surface by the lens of the imaging device, the first image includes a first aberration-corrected image obtained by performing an aberration region correction process as part of the first AI process, which corrects the region of the image where lens aberrations are reflected in the image using an AI method, the second image includes a second aberration-corrected image obtained by performing a process that corrects the region of the image where lens aberrations are reflected in the image using a non-AI method, and the processor adjusts the elements derived from the aberration region correction process by combining the first aberration-corrected image and the second aberration-corrected image by a ratio.

[0023] A fourteenth aspect of the technology of this disclosure is an image processing apparatus according to any one of the sixth to thirteenth aspects, wherein the first image includes a first colored image obtained by performing a coloring process as part of a first AI process in which a first region and a second region which is a region different from the first region are colored in an AI manner on an image to be processed, and the second image includes a second colored image obtained by performing a process to change the color of the image to be processed in a non-AI manner, and the processor adjusts elements derived from the coloring process by combining the first colored image and the second colored image in proportion.

[0024] A 15th aspect according to the technology of the present disclosure is an image processing apparatus according to the 14th aspect, in which a second color image is an image obtained by performing a process of colorizing a first region and a second region of a processing target image in a non-AI manner so as to be distinguishable.

[0025] A 16th aspect according to the technology of the present disclosure is an image processing apparatus according to the 14th or 15th aspect, in which a processing target image is an image obtained by capturing a first subject, and a first region is a region in the processing target image where a specific subject included in the first subject is depicted.

[0026] A 17th aspect according to the technology of the present disclosure includes a first contrast adjustment image obtained by performing a first contrast adjustment process for adjusting the contrast of a processing target image in an AI manner as a process included in a first AI process, and a second contrast adjustment image obtained by performing a second contrast adjustment process for adjusting the contrast of the processing target image in a non-AI manner. A processor synthesizes the first contrast adjustment image and the second contrast adjustment image at a ratio, and thus, an image processing apparatus according to any one of the 6th to 16th aspects for adjusting elements derived from the first contrast adjustment process. is provided.

[0027] The 18th aspect of the technology of this disclosure is an image processing apparatus according to the 17th aspect, wherein the image to be processed is an image obtained by capturing a second subject, the first contrast adjustment process includes a third contrast adjustment process that adjusts the contrast of the image to be processed according to the second subject using an AI method, the second contrast adjustment process includes a fourth contrast adjustment process that adjusts the contrast of the image to be processed according to the second subject using a non-AI method, the first image includes a third contrast image obtained by performing the third contrast adjustment process, the second image includes a fourth contrast image obtained by performing the fourth contrast adjustment process, and the processor adjusts elements derived from the third contrast adjustment process by combining the third contrast image and the fourth contrast image in proportion.

[0028] A 19th aspect of the technology of this disclosure is an image processing apparatus according to the 17th or 18th aspect, wherein the first contrast adjustment process includes a fifth contrast adjustment process that adjusts the contrast between a central pixel and a plurality of adjacent pixels adjacent to the central pixel in an AI-based manner in an image to be processed, the second contrast adjustment process includes a sixth contrast adjustment process that adjusts the contrast between a central pixel and a plurality of adjacent pixels in a non-AI-based manner, the first image includes a fifth contrast image obtained by performing the fifth contrast adjustment process, the second image includes a sixth contrast image obtained by performing the sixth contrast adjustment process, and the processor adjusts elements derived from the fifth contrast adjustment process by combining the fifth contrast image and the sixth contrast image in proportion.

[0029] A 20th aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 19th aspects, wherein the first image includes a first resolution adjustment image obtained by performing a first resolution adjustment process, which adjusts the resolution of an image to be processed using an AI method, as part of the first AI process, and the second image includes a second resolution adjustment image obtained by performing a second resolution adjustment process, which adjusts the resolution using a non-AI method, and the processor adjusts elements derived from the first resolution adjustment process by combining the first resolution adjustment image and the second resolution adjustment image in proportion.

[0030] A 21st aspect of the technology of this disclosure is an image processing apparatus according to the 20th aspect, wherein the first resolution adjustment process is a process that enhances the resolution of the image to be processed using an AI method, and the second resolution adjustment process is a process that enhances the resolution of the image to be processed using a non-AI method.

[0031] A 22nd aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 21st aspects, wherein the first image includes a first high dynamic range image obtained by performing an expansion process to widen the dynamic range of an image to be processed using an AI method as part of the first AI process, and the second image includes a second high dynamic range image obtained by performing a process to widen the dynamic range of an image to be processed using a non-AI method, and the processor adjusts elements derived from the expansion process by combining the first high dynamic range image and the second high dynamic range image by a ratio.

[0032] A 23rd aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 22nd aspects, wherein the first image includes a first edge-enhanced image obtained by performing an enhancement process as part of the first AI process in which the edge regions in the image to be processed are enhanced more than non-edge regions which are regions different from the edge regions using an AI method, and the second image includes a second edge-enhanced image obtained by performing a process in which the edge regions are enhanced more than non-edge regions using a non-AI method, and the processor adjusts the elements derived from the enhancement process by combining the first edge-enhanced image and the second edge-enhanced image by ratio.

[0033] A 24th aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 23rd aspects, wherein the first image includes a first point image adjustment image obtained by performing a point image adjustment process to adjust the amount of point image blur on an image to be processed using an AI method as part of the first AI process, and the second image includes a second point image adjustment image obtained by performing a process to adjust the amount of blur using a non-AI method, and the processor adjusts elements derived from the point image adjustment process by combining the first point image adjustment image and the second point image adjustment image by a ratio.

[0034] A 25th aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 24th aspects, wherein the image to be processed is an image obtained by capturing a third subject, the first image includes a first blurred image obtained by performing a blurring process in which an AI method applies blur to the image to be processed according to the third subject as part of the first AI process, the second image includes a second blurred image obtained by performing a process that applies blur to the image to be processed in a non-AI method, and the processor adjusts elements derived from the blurring process by combining the first blurred image and the second blurred image by a ratio.

[0035] A 26th aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 25th aspects, wherein the first image includes a first bokeh image obtained by performing a bokeh processing that applies a first bokeh to an image to be processed using an AI method as part of the first AI processing, and the second image includes a second bokeh image obtained by adjusting the first bokeh from the image to be processed using a non-AI method, or by performing a processing that applies a second bokeh to the image to be processed using a non-AI method, and the processor adjusts elements derived from the bokeh processing by combining the first bokeh image and the second bokeh image in proportion.

[0036] A 27th aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 26th aspects, wherein the first image includes a first tone adjustment image obtained by performing a first tone adjustment process, which adjusts the tone of an image to be processed using an AI method, as part of the first AI process, and the second image includes a second tone adjustment image obtained by performing a second tone adjustment process, which adjusts the tone of an image to be processed using a non-AI method, and the processor adjusts elements derived from the first tone adjustment process by combining the first tone adjustment image and the second tone adjustment image by ratio.

[0037] A 28th aspect of the technology of this disclosure is an image processing apparatus according to the 27th aspect, wherein the image to be processed is an image obtained by capturing a fourth subject, the first tone adjustment process is an AI method for adjusting the tone of the image to be processed according to the fourth subject, and the second tone adjustment process is a non-AI method for adjusting the tone of the image to be processed according to the fourth subject.

[0038] A 29th aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 28th aspects, wherein the first image includes a style-changed image obtained by performing a style-change processing that changes the style of an image to be processed using an AI method as part of the first AI processing, and the processor adjusts elements derived from the style-change processing by combining the style-changed image and the second image in proportion.

[0039] A 30th aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 29th aspects, wherein the image to be processed is an image obtained by capturing skin, the first image includes a skin image quality adjustment image obtained by performing a skin image quality adjustment process as part of the first AI process, which adjusts the image quality of the skin in the image to be processed using an AI method, and the processor adjusts elements derived from the skin image quality adjustment process by combining the skin image quality adjustment image and the second image in proportion.

[0040] A 31st aspect of the technology of this disclosure is an image processing apparatus according to any one of the 6th to 30th aspects, wherein the first AI processing includes a plurality of purpose-specific processing performed in an AI manner, the first image includes a plurality of processed images obtained by performing the plurality of purpose-specific processing on the image to be processed, and the processor synthesizes the plurality of processed images and the second image in proportion.

[0041] A 32nd aspect of the technology of this disclosure is an image processing apparatus according to a 31st aspect, in which multiple purpose-specific processes are performed in an order based on the degree of influence they have on the image to be processed.

[0042] A 33rd aspect of the technology of this disclosure is an image processing apparatus according to a 32nd aspect, in which multiple purpose-specific processing is performed in stages, from purpose-specific processing with a small degree of impact to purpose-specific processing with a large degree of impact.

[0043] A 34th aspect of the technology of this disclosure is an image processing apparatus relating to any one of the 5th to 32nd aspects, wherein the ratio is determined based on the difference between the image to be processed and the first image, and / or the difference between the first image and the second image.

[0044] A 35th aspect of the technology of this disclosure is an image processing apparatus relating to any one of the 6th to 34th aspects, wherein the processor adjusts a ratio according to relevant information related to the image to be processed.

[0045] A 36th aspect of the technology of this disclosure is an imaging device comprising an image processing device according to any one of the first to 34 aspects and an image sensor, wherein the image to be processed is an image obtained by imaging performed by the image sensor.

[0046] A 37th aspect of the technology of this disclosure is an image processing method that includes obtaining a first image obtained by performing a first AI processing on an image to be processed, and a second image obtained without performing the first AI processing on the image to be processed, and adjusting for any excess or deficiency of the first AI processing by combining the first image and the second image.

[0047] A 38th aspect of the technology of this disclosure is an image processing method that includes obtaining a first image obtained by adjusting the non-noise elements of an image to be processed by performing a first AI processing on the image to be processed, and a second image obtained without performing the first AI processing on the image to be processed, and adjusting the non-noise elements by combining the first image and the second image.

[0048] A 39th aspect of the technology of this disclosure is a program for causing a computer to perform a process that includes acquiring a first image obtained by performing a first AI process on an image to be processed, and a second image obtained without performing the first AI process on the image to be processed, and adjusting for any excess or deficiency of the first AI process by combining the first image and the second image.

[0049] A forty-th aspect of the technology of this disclosure is a program for causing a computer to perform a process that includes acquiring a first image obtained by adjusting the non-noise elements of an image to be processed by performing a first AI process on the image to be processed, and a second image obtained without performing the first AI process on the image to be processed, and adjusting the non-noise elements by combining the first image and the second image. [Brief explanation of the drawing]

[0050] [Figure 1] This is a schematic diagram showing an example of the overall configuration of an imaging device. [Figure 2] This is a schematic diagram showing an example of the hardware configuration of the optical and electrical systems of an imaging device. [Figure 3] This block diagram shows an example of the functions of an image processing engine. [Figure 4] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit. [Figure 5] This is a conceptual diagram showing an example of the processing content of the image adjustment and synthesis units. [Figure 6] This is a flowchart illustrating an example of the image synthesis process. [Figure 7] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the first modified example. [Figure 8] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the first modified example. [Figure 9] This flowchart shows an example of the image synthesis process flow related to the first modified example. [Figure 10] This is a conceptual diagram illustrating an example of how a non-AI processing unit performs coloring in a way that distinguishes between human and background areas using a non-AI method. [Figure 11] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the second modified example. [Figure 12] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the second modified example. [Figure 13] This flowchart shows an example of the image synthesis process flow related to the second modified example. [Figure 14] This is a conceptual diagram showing an example of the content of the first and second clarity processing steps. [Figure 15] This is a conceptual diagram illustrating an example of how the processor adjusts contrast according to the subject. [Figure 16] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the third modified example. [Figure 17] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the third modified example. [Figure 18] This flowchart shows an example of the image synthesis process flow related to the third modified example. [Figure 19] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the fourth modified example. [Figure 20] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the fourth modified example. [Figure 21] This flowchart shows an example of the image synthesis process flow related to the fourth modified example. [Figure 22] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the fifth modified example. [Figure 23] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the fifth modified example. [Figure 24] This flowchart shows an example of the image synthesis process flow related to the fifth modified example. [Figure 25] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the sixth modified example. [Figure 26] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the sixth modified example. [Figure 27] This flowchart shows an example of the image synthesis process flow related to the sixth modified example. [Figure 28] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the seventh modified example. [Figure 29] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the seventh modified example. [Figure 30] This flowchart shows an example of the image synthesis process flow related to the seventh modified example. [Figure 31] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the eighth modified example. [Figure 32]This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the eighth modified example. [Figure 33] This flowchart shows an example of the image synthesis process flow related to the eighth modified example. [Figure 34] This is a conceptual diagram illustrating the first example of a process in which a non-AI processing unit generates a second bokeh effect by filtering the first bokeh effect generated by the AI ​​method. [Figure 35] This is a conceptual diagram illustrating a second example of the processing method in which a non-AI processing unit generates a second bokeh effect by filtering the first bokeh effect generated by the AI ​​method. [Figure 36] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the 9th modified example. [Figure 37] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the ninth modified example. [Figure 38] This flowchart shows an example of the image synthesis process flow related to the ninth modified example. [Figure 39] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the 10th modified example. [Figure 40] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the 10th modified example. [Figure 41] This flowchart shows an example of the image synthesis process flow related to the 10th modified example. [Figure 42] This is a conceptual diagram showing an example of the processing content of the AI-based processing unit and the non-AI-based processing unit related to the 11th modified example. [Figure 43] This is a conceptual diagram showing an example of the processing content of the image adjustment unit and the image synthesis unit in the 11th modified example. [Figure 44] This flowchart shows an example of the image synthesis process flow related to the 11th modified example. [Figure 45] This is a conceptual diagram illustrating an example of how an AI processing unit performs multiple purpose-specific processing using an AI method. [Figure 46] This is a conceptual diagram illustrating an example of a process in which the processor derives a percentage based on the difference between the image to be processed and the first image. [Figure 47] This is a conceptual diagram illustrating an example of a process in which a processor derives a percentage based on the difference between the first and second images. [Figure 48] This is a conceptual diagram illustrating an example of how a processor adjusts ratios based on relevant information. [Figure 49] This is a conceptual diagram showing an example of the configuration of an imaging system. [Modes for carrying out the invention]

[0051] Hereinafter, an example of an embodiment of the image processing apparatus, imaging apparatus, image processing method, and program relating to the technology of this disclosure will be described with reference to the attached drawings.

[0052] First, let's explain the terminology used in the following explanation.

[0053] CPU stands for "Central Processing Unit." GPU stands for "Graphics Processing Unit." TPU stands for "Tensor Processing Unit." NVM stands for "Non-volatile memory." RAM stands for "Random Access Memory." It refers to an abbreviation for "emory". IC refers to an abbreviation for "Integrated Circuit". ASIC is PLD stands for "Application Specific Integrated Circuit." FPGA stands for "Field-Programmable Gate Array." SoC stands for "System-on-a-chip". SSD stands for "Solid State Drive". USB stands for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". fps is an abbreviation for "frames per second". MF is , refers to an abbreviation for “Manual Focus”. AF refers to an abbreviation for “Auto Focus”. CMOS refers to an abbreviation for “Complementary Metal Oxide Semiconductor”. CCD refers to an abbreviation for “Charge Coupled Device”. LAN refers to an abbreviation for “Local Area Network”. WAN refers to an abbreviation for “Wide Area Network”. AI refers to an abbreviation for “Artificial Intelligence”. A / D refers to an abbreviation for “Analog / Digital”. FIR refers to an abbreviation for “Finite Impulse Response”. IIR refers to an abbreviation for “Infinite Impulse Response”. VAE refers to an abbreviation for “Variational Auto-Encoder”. GAN refers to an abbreviation for “Generative Adversarial Network”. FIR refers to an abbreviation for “Finite Impulse Response”.

[0054] In this embodiment, noise refers to noise generated by imaging by the imaging device (for example, electrical noise appearing in the image obtained by imaging (i.e., an electronic image)). In other words, noise refers to unavoidably occurring electrical noise (for example, noise unavoidably caused by electrical factors). Specific examples of noise include noise generated with increasing analog gain, dark current noise, pixel defects, and / or heat noise. Furthermore, in the following, elements other than noise that appear in the image obtained by imaging (i.e., elements that represent the image other than noise) will be referred to as "non-noise elements".

[0055] As an example, as shown in Figure 1, the imaging device 10 is a device for imaging a subject and comprises an image processing engine 12, an imaging device body 16, and an interchangeable lens 18. The imaging device 10 is an example of an "imaging device" according to the technology of this disclosure. The interchangeable lens 18 is an example of a "lens" according to the technology of this disclosure. The image processing engine 12 is an example of an "image processing device" and "computer" according to the technology of this disclosure.

[0056] The image processing engine 12 is built into the imaging device body 16 and controls the entire imaging device 10. The interchangeable lens 18 is interchangeably mounted on the imaging device body 16. The interchangeable lens 18 is provided with a focus ring 18A. The focus ring 18A is operated by the user of the imaging device 10 (hereinafter simply referred to as "user") when the user manually adjusts the focus on the subject using the imaging device 10.

[0057] In the example shown in Figure 1, a lens-interchangeable digital camera is shown as an example of the imaging device 10. However, this is merely an example, and it may be a lens-fixed digital camera, or a digital camera built into various electronic devices such as smart devices, wearable devices, cell observation devices, ophthalmic observation devices, or surgical microscopes.

[0058] The imaging device body 16 is equipped with an image sensor 20. The image sensor 20 is an example of an "image sensor" related to the technology of this disclosure. The image sensor 20 is a CMOS image sensor. The image sensor 20 generates and outputs image data representing an image by capturing an image of a subject. When the interchangeable lens 18 is attached to the imaging device body 16, the subject light representing the subject passes through the interchangeable lens 18 and is imaged onto the image sensor 20, and image data is generated by the image sensor 20.

[0059] In this embodiment, a CMOS image sensor is given as an example of the image sensor 20, but the technology of this disclosure is not limited to this, and the technology of this disclosure can be established even if the image sensor 20 is another type of image sensor such as a CCD image sensor.

[0060] A release button 22 and a dial 24 are provided on the top surface of the imaging device body 16. The dial 24 is operated when setting the operating modes of the imaging system and the playback system, and by operating the dial 24, the imaging device 10 can be selectively set to an imaging mode, a playback mode, and a setting mode. The imaging mode is the operating mode that causes the imaging device 10 to take an image. The playback mode is the operating mode that plays back images (for example, still images and / or moving images) obtained by taking an image for recording in the imaging mode. The setting mode is the operating mode that is set for the imaging device 10 when setting various setting values ​​used in control related to imaging.

[0061] The release button 22 functions as both an imaging preparation instruction unit and an imaging instruction unit, and can detect two-stage pressing operations: an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state where the button is pressed from, for example, the standby position to an intermediate position (half-press position), and the imaging instruction state refers to a state where the button is pressed beyond the intermediate position to the final pressed position (full-press position). Hereinafter, the state where the button is pressed from the standby position to the half-press position will be referred to as the "half-press state," and the state where the button is pressed from the standby position to the full-press position will be referred to as the "full-press state." Depending on the configuration of the imaging device 10, the imaging preparation instruction state may be a state where the user's finger is in contact with the release button 22, and the imaging instruction state may be a state where the user's finger has moved away from contact with the release button 22.

[0062] The back of the imaging device body 16 is provided with instruction keys 26 and a touch panel display 32.

[0063] The touch panel display 32 comprises a display 28 and a touch panel 30 (see also Figure 2). An example of the display 28 is an EL display (e.g., an organic EL display or an inorganic EL display). The display 28 may be other types of displays, such as a liquid crystal display, instead of an EL display.

[0064] The display 28 displays images and / or text information. When the imaging device 10 is in imaging mode, the display 28 is used to display the live view image obtained by imaging for the live view image, that is, by continuous imaging. Here, "live view image" refers to a moving image for display based on image data obtained by imaging by the image sensor 20. The imaging performed to obtain the live view image (hereinafter also referred to as "imaging for the live view image") is performed according to a frame rate of, for example, 60fps. 60fps is merely an example; the frame rate may be less than 60fps or more than 60fps.

[0065] The display 28 is also used to display still images obtained when the imaging device 10 is instructed to take still images via the release button 22. The display 28 is also used to display playback images when the imaging device 10 is in playback mode. Furthermore, when the imaging device 10 is in setting mode, the display 28 is used to display a menu screen where various menus can be selected, and a setting screen for setting various setting values ​​used in control related to imaging.

[0066] The touch panel 30 is a transparent touch panel and is superimposed on the surface of the display area of ​​the display 28. The touch panel 30 is touched by a finger or an object such as a stylus pen. By detecting this, the system accepts instructions from the user. For the sake of explanation, in the following, the "fully pressed state" mentioned above also includes the state in which the user has turned on the soft key for starting imaging via the touch panel 30.

[0067] In this embodiment, an out-cell type touch panel display in which the touch panel 30 is superimposed on the surface of the display area of ​​the display 28 is given as an example of the touch panel display 32, but this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be used as the touch panel display 32.

[0068] The instruction key 26 accepts various instructions. Here, "various instructions" refers to, for example, instructions to display the menu screen, instructions to select one or more menus, instructions to confirm the selection, instructions to delete the selection, zoom in, zoom out, and various other instructions such as frame-by-frame playback. These instructions may also be given via the touch panel 30.

[0069] As an example, as shown in Figure 2, the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light-receiving surface 72A, and subject light is imaged onto the light-receiving surface 72A via an interchangeable lens 18. The light-receiving surface 72A is an example of a "light-receiving surface" according to the technology of this disclosure. The photoelectric conversion element 72 is arranged in the imaging device body 16 such that the center of the light-receiving surface 72A coincides with the optical axis OA (see also Figure 1). The photoelectric conversion element 72 has a plurality of photosensitive pixels arranged in a matrix, and the light-receiving surface 72A is formed by the plurality of photosensitive pixels. Each photosensitive pixel has a microlens (not shown). Each photosensitive pixel is a physical pixel having a photodiode (not shown), which photoelectrically converts the received light and outputs an electrical signal corresponding to the amount of light received.

[0070] Furthermore, multiple photosensitive pixels have red (R), green (G), or blue (B) color filters (not shown) arranged in a matrix in a predetermined pattern (e.g., Bayer array, G-striped R / G checkerboard, X-Trans® array, or honeycomb array). For the sake of explanation, in the following, photosensitive pixels having a microlens and an R color filter will be referred to as R pixels, photosensitive pixels having a microlens and a G color filter will be referred to as G pixels, and photosensitive pixels having a microlens and a B color filter will be referred to as B pixels.

[0071] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 has an objective lens 40A, a focusing lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, focusing lens 40B, zoom lens 40C, and aperture 40D are arranged in the order of objective lens 40A, focusing lens 40B, zoom lens 40C, and aperture 40D along the optical axis OA from the subject side (i.e., the object side) to the imaging device body 16 side (i.e., the image side).

[0072] The interchangeable lens 18 also includes a control device 36, a first actuator 37, a second actuator 38, and a third actuator 39. The control device 36 controls the entire interchangeable lens 18 according to instructions from the imaging device body 16. The control device 36 is a device having a computer including, for example, a CPU, NVM, and RAM. The NVM of the control device 36 is, for example, an EEPROM. The RAM of the control device 36 temporarily stores various information and is used as work memory. In the control device 36, the CPU reads the necessary programs from the NVM and controls the entire imaging lens 40 by executing the read programs on the RAM.

[0073] Here, a device with a computer is given as an example of a control device 36, but this is merely one example, and devices including ASICs, FPGAs, and / or PLDs are also included. A vice may be applied. Furthermore, the control device 36 may be, for example, a device implemented by a combination of hardware and software configurations.

[0074] The first actuator 37 includes a focusing slide mechanism (not shown) and a focusing motor (not shown). A focusing lens 40B is mounted on the focusing slide mechanism so as to be slidable along the optical axis OA. A focusing motor is also connected to the focusing slide mechanism, and the focusing slide mechanism operates by receiving power from the focusing motor, thereby moving the focusing lens 40B along the optical axis OA.

[0075] The second actuator 38 includes a zoom slide mechanism (not shown) and a zoom motor (not shown). A zoom lens 40C is mounted on the zoom slide mechanism so as to be slidable along the optical axis OA. A zoom motor is also connected to the zoom slide mechanism, and the zoom slide mechanism operates by receiving power from the zoom motor, thereby moving the zoom lens 40C along the optical axis OA.

[0076] The third actuator 39 includes a power transmission mechanism (not shown) and an aperture motor (not shown). The aperture 40D has an opening 40D1, and the size of the opening 40D1 is variable. The opening 40D1 is formed, for example, by a plurality of aperture blades 40D2. The plurality of aperture blades 40D2 are connected to the power transmission mechanism. An aperture motor is also connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the aperture motor to the plurality of aperture blades 40D2. The plurality of aperture blades 40D2 change the size of the opening 40D1 by operating in response to the power transmitted from the power transmission mechanism. The aperture 40D adjusts the exposure by changing the size of the opening 40D1.

[0077] The focus motor, zoom motor, and aperture motor are connected to the control device 36, and the control device 36 controls the drive of each of these motors. In this embodiment, stepping motors are used as an example of the focus motor, zoom motor, and aperture motor. Therefore, the focus motor, zoom motor, and aperture motor operate in synchronization with pulse signals in response to commands from the control device 36. Here, an example is shown in which the focus motor, zoom motor, and aperture motor are provided on the interchangeable lens 18, but this is merely an example, and at least one of the focus motor, zoom motor, and aperture motor may be provided on the imaging device body 16. The components and / or operating method of the interchangeable lens 18 can be changed as needed.

[0078] In the imaging device 10, when in imaging mode, MF mode and AF mode are selectively set according to instructions given to the imaging device body 16. MF mode is an operation mode in which the focus is adjusted manually. In MF mode, for example, the user operates the focus ring 18A, etc., and the focus lens 40B moves along the optical axis OA by an amount of movement corresponding to the amount of operation of the focus ring 18A, etc., thereby adjusting the focus.

[0079] In AF mode, the imaging device body 16 calculates the focus position according to the subject distance and adjusts the focus by moving the focus lens 40B toward the calculated focus position. Here, the focus position refers to the position of the focus lens 40B on the optical axis OA when the image is in focus.

[0080] The imaging device body 16 includes an image processing engine 12, an image sensor 20, a system controller 44, an image memory 46, a UI device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. The sensor 20 includes a photoelectric conversion element 72 and an A / D converter 74.

[0081] The input / output interface 70 is connected to an image processing engine 12, an image memory 46, a UI device 48, an external I / F 50, a photoelectric conversion element driver 54, a mechanical shutter driver 56, and an A / D converter 74. The input / output interface 70 is also connected to a control device 36 for the interchangeable lens 18.

[0082] The system controller 44 includes a CPU (not shown), an NVM (not shown), and RAM (not shown). In the system controller 44, the NVM is a non-temporary storage medium that stores various parameters and programs. The NVM of the system controller 44 is, for example, an EEPROM. However, this is merely an example, and an HDD and / or SSD may be used as the NVM of the system controller 44 instead of, or in conjunction with, the EEPROM. The RAM of the system controller 44 temporarily stores various information and is used as work memory. In the system controller 44, the CPU reads the necessary programs from the NVM and controls the entire imaging device 10 by executing the read programs on the RAM. That is, in the example shown in Figure 2, the image processing engine 12, image memory 46, UI device 48, external I / F 50, communication I / F 52, photoelectric conversion element driver 54, and control device 36 are controlled by the system controller 44.

[0083] The image processing engine 12 operates under the control of the system controller 44. The image processing engine 12 comprises a processor 62, an NVM 64, and RAM 66. Here, the processor 62 is an example of a "processor" related to the technology of this disclosure.

[0084] The processor 62, NVM 64, and RAM 66 are connected via a bus 68, which is connected to an input / output interface 70. In the example shown in Figure 2, for illustrative purposes, only one bus is shown as bus 68, but there may be multiple buses. Bus 68 may be a serial bus, or a parallel bus including a data bus, address bus, and control bus, etc.

[0085] The processor 62 has a CPU and a GPU, with the GPU operating under the control of the CPU and primarily responsible for image processing. The processor 62 may consist of one or more CPUs with integrated GPU functionality, or one or more CPUs without integrated GPU functionality. Furthermore, the processor 62 may include a multi-core CPU or a TPU.

[0086] NVM64 is a non-temporary storage medium that stores various parameters and programs different from those stored in the NVM of the system controller 44. NVM64 is, for example, an EEPROM. However, this is merely one example, and an HDD and / or SSD may be used as NVM64 instead of, or in conjunction with, an EEPROM. RAM66 temporarily stores various information and is used as work memory.

[0087] The processor 62 reads the necessary program from the NVM64 and executes the read program in RAM66. The processor 62 performs various image processing operations according to the program executed on RAM66.

[0088] A photoelectric conversion element driver 54 is connected to the photoelectric conversion element 72. The photoelectric conversion element driver 54 supplies an imaging timing signal to the photoelectric conversion element 72, which defines the timing of imaging performed by the photoelectric conversion element 72, according to instructions from the processor 62. The conversion element 72 performs reset, exposure, and output of electrical signals according to the imaging timing signals supplied from the photoelectric conversion element driver 54. Examples of imaging timing signals include a vertical synchronization signal and a horizontal synchronization signal.

[0089] When the interchangeable lens 18 is attached to the imaging device body 16, the subject light incident on the imaging lens 40 is imaged onto the light-receiving surface 72A by the imaging lens 40. Under the control of the photoelectric conversion element driver 54, the photoelectric conversion element 72 converts the subject light received by the light-receiving surface 72A into electrical signals and outputs an electrical signal corresponding to the amount of subject light as analog image data representing the subject light to the A / D converter 74. Specifically, the A / D converter 74 reads out the analog image data from the photoelectric conversion element 72 in units of one frame and for each horizontal line using an exposure sequential readout method.

[0090] The A / D converter 74 generates a processing target image 75A by digitizing analog image data. The processing target image 75A is an image captured by the imaging device 10, and is an example of the "processing target image" and "image captured" related to the technology of this disclosure. The processing target image 75A is an image in which R pixels, G pixels, and B pixels are arranged in a mosaic pattern.

[0091] In this embodiment, as an example, the processor 62 of the image processing engine 12 acquires the image to be processed 75A from the A / D converter 74 and performs various image processing on the acquired image to be processed 75A.

[0092] The processed image 75B is stored in the image memory 46. The processed image 75B is an image obtained by performing various image processing operations on the image to be processed 75A by the processor 62.

[0093] The UI device 48 includes a display 28, and the processor 62 displays various information on the display 28. The UI device 48 also includes a reception device 76. The reception device 76 includes a touch panel 30 and a hard key section 78. The hard key section 78 consists of multiple hard keys, including an instruction key 26 (see Figure 1). The processor 62 operates according to the various instructions received by the touch panel 30. Although the hard key section 78 is included in the UI device 48 here, the technology of this disclosure is not limited to this, and for example, the hard key section 78 may be connected to an external I / F 50.

[0094] The external I / F 50 is responsible for the exchange of various types of information between the imaging device 10 and devices located outside of it (hereinafter also referred to as "external devices"). An example of the external I / F 50 is a USB interface. External devices such as smart devices, personal computers, servers, USB memory, memory cards, and / or printers (not shown) can be directly or indirectly connected to the USB interface.

[0095] The communication interface 52 is connected to a network (not shown). The communication interface 52 is responsible for the exchange of information between the system controller 44 and communication devices (not shown), such as servers, on the network. For example, the communication interface 52 transmits information to the communication device via the network in response to a request from the system controller 44. The communication interface 52 also receives information transmitted from the communication device and outputs the received information to the system controller 44 via the input / output interface 70.

[0096] As an example, as shown in Figure 3, the NVM64 of the imaging device 10 stores an image synthesis processing program 80. The image synthesis processing program 80 is a "program" related to the technology of this disclosure. This is an example of "ram".

[0097] The NVM64 of the imaging device 10 stores the generative model 82A. An example of the generative model 82A is a trained generative network. An example of a generative network is a GAN or VAE. The processor 62 performs AI-based processing on the image to be processed 75A (see Figure 2). An example of AI-based processing is processing using the generative model 82A. For the sake of explanation, in the following, processing using the generative model 82A will be described as processing that the generative model 82A actively performs. That is, for the sake of explanation, the generative model 82A will be described as a function that processes the input information and outputs the processing result.

[0098] The NVM64 of the imaging device 10 stores a digital filter 84A. An example of a digital filter 84A is an FIR filter. However, the FIR filter is merely an example, and other digital filters such as IIR filters may also be used. For the sake of explanation, in the following, processing using the digital filter 84A will be described as processing that is actively performed by the digital filter 84A. In other words, for the sake of explanation, the digital filter 84A will be described as a function that processes the input information and outputs the processing result.

[0099] The processor 62 reads the image synthesis processing program 80 from the NVM 64 and executes the read image synthesis processing program 80 on the RAM 66. The processor 62 performs image synthesis processing (see Figure 6) according to the image synthesis processing program 80 executed on the RAM 66. The image synthesis processing is realized by the processor 62 operating as an AI method processing unit 62A1, a non-AI method processing unit 62B1, an image adjustment unit 62C1, and a synthesis unit 62D1 according to the image synthesis processing program 80. The generation model 82A is used by the AI ​​method processing unit 62A1, and the digital filter 84A is used by the non-AI method processing unit 62B1.

[0100] As an example, as shown in Figure 4, the AI-type processing unit 62A1 and the non-AI-type processing unit 62B1 receive the image to be processed 75A1 as input. The image to be processed 75A1 is an example of the image to be processed 75A shown in Figure 2. In the example shown in Figure 4, the image region 75A1a is shown as the image region of the image to be processed 75A1 that is affected by the aberration (hereinafter simply referred to as "aberration") of the imaging lens 40 (see Figure 2) (i.e., the image region in which the aberration is reflected).

[0101] The image to be processed 75A1 is an image that has non-noise elements. An example of a non-noise element is the image region 75A1a. ​​The image region 75A1a is an example of "non-noise elements of the image to be processed," "phenomena that appear in the image to be processed due to the characteristics of the imaging device," "blur," and "regions in the captured image that reflect lens aberrations" related to the technology of this disclosure.

[0102] In the example shown in Figure 4, an image region 75A1a is shown as an example of an image region where field curvature is reflected. In addition, in the example shown in Figure 4, the image region 75A1a is shown as a manner in which the image is gradually darkened from the center of the processed image 75A1 outward in the radial direction due to field curvature (i.e., a manner in which background blur is reflected).

[0103] Here, field curvature is given as an example of an aberration reflected in the processed image 75A1, but this is merely one example, and other types of aberrations may be reflected in the processed image 75A1, such as spherical aberration, coma aberration, astigmatism, distortion aberration, axial chromatic aberration, or lateral chromatic aberration. Aberrations are examples of the "characteristics of the imaging device" and "optical characteristics of the imaging device" related to the technology of this disclosure.

[0104] The AI ​​processing unit 62A1 performs AI-based processing on the image 75A1 to be processed. One example of AI-based processing on the image 75A1 is processing using the generative model 82A1. The generative model 82A1 is an example of the generative model 82A shown in Figure 3. The generative model 82A1 is a generative network that has already been trained to reduce the effects of aberrations (here, as an example, field curvature). The AI ​​processing unit 62A1 generates the first aberration-corrected image 86A1 by processing the image 75A1 using the generative model 82A1. In other words, the AI ​​processing unit 62A1 generates the first aberration-corrected image 86A1 by adjusting non-noise elements (here, as an example, the image region 75A1a) within the image 75A1 to be processed using the AI ​​method. To put it another way, the AI ​​processing unit 62A1 generates a first aberration-corrected image 86A1 by correcting the image region 75A1a (i.e., the region where aberrations are reflected) within the image to be processed 75A1 using the AI ​​method. Here, the processing using the generation model 82A1 is an example of the "first AI processing," "first correction processing," and "first aberration region correction processing" related to the technology of this disclosure. Also, here, "generating the first aberration-corrected image 86A1" is an example of "acquiring the first image" related to the technology of this disclosure.

[0105] The generation model 82A1 receives the image to be processed 75A1 as input. Based on the input image to be processed 75A1, the generation model 82A1 generates and outputs a first aberration-corrected image 86A1. The first aberration-corrected image 86A1 is an image obtained by adjusting the non-noise elements by the generation model 82A1 (i.e., an image obtained by adjusting the non-noise elements by processing the image to be processed 75A1 using the generation model 82A1). In other words, the first aberration-corrected image 86A1 is an image in which the non-noise elements in the image to be processed 75A1 have been corrected by the generation model 82A1 (i.e., an image in which the non-noise elements have been corrected by processing the image to be processed 75A1 using the generation model 82A1). To put it another way, the first aberration-corrected image 86A1 is an image in which the image region 75A1a has been corrected by the generation model 82A1 (i.e., an image in which the image region 75A1a has been corrected so that the effect of aberrations is reduced by processing the image to be processed 75A1 using the generation model 82A1). The first aberration-corrected image 86A1 is an example of the "first image," "first corrected image," and "first aberration-corrected image" related to the technology of this disclosure.

[0106] The non-AI processing unit 62B1 performs non-AI processing on the image 75A1 to be processed. Non-AI processing refers to processing that does not use a neural network. For example, processing that does not use a neural network is processing that does not use the generative model 82A1.

[0107] One example of non-AI processing on the image to be processed 75A1 is processing using a digital filter 84A1. The digital filter 84A1 is a digital filter configured to reduce the effects of aberrations (in this case, field curvature). The non-AI processing unit 62B1 generates a second aberration-corrected image 88A1 by processing (i.e., filtering) the image to be processed 75A1 using the digital filter 84A1. In other words, the non-AI processing unit 62B1 generates a second aberration-corrected image 88A1 by adjusting non-noise elements (in this case, image region 75A1a) within the image to be processed 75A1 using a non-AI method. To put it another way, the non-AI processing unit 62B1 generates a second aberration-corrected image 88A1 by correcting the image region 75A1a (i.e., the region where aberrations are reflected) within the image to be processed 75A1 using a non-AI method. Here, the processing using the digital filter 84A1 is an example of the "non-AI method processing without using a neural network," "second correction processing," and "processing that corrects using a non-AI method" related to the technology of this disclosure. Also, here, "generating a second aberration-corrected image 88A1" is an example of "acquiring a second image" related to the technology of this disclosure.

[0108] The image to be processed, 75A1, is input to the digital filter 84A1. The 84A1 generates a second aberration-corrected image 88A1 based on the input image 75A1 to be processed. The second aberration-corrected image 88A1 is an image obtained by adjusting the non-noise elements with the digital filter 84A1 (i.e., an image obtained by adjusting the non-noise elements by processing the image 75A1 to be processed using the digital filter 84A1). In other words, the second aberration-corrected image 88A1 is an image in which the non-noise elements within the image 75A1 to be processed have been corrected by the digital filter 84A1 (i.e., an image in which the non-noise elements have been corrected by processing the image 75A1 to be processed using the digital filter 84A1). To put it another way, the second aberration-corrected image 88A1 is an image in which the image region 75A1a has been corrected by the digital filter 84A1 (i.e., an image in which the image region 75A1a has been corrected so that the effect of aberrations is reduced by processing the image 75A1 to be processed using the digital filter 84A1). The second aberration-corrected image 88A1 is an example of the "second image," "second corrected image," and "second aberration-corrected image" related to the technology of this disclosure.

[0109] Incidentally, some users prefer to retain a moderate amount of aberration in the image rather than completely eliminating its effects. In the example shown in Figure 4, the first aberration-corrected image 86A1 has reduced aberration effects compared to the second aberration-corrected image 88A1. In other words, the second aberration-corrected image 88A1 retains more aberration effects than the first aberration-corrected image 86A1. However, users may feel that the aberration effects in the first aberration-corrected image 86A1 are too minimal, while the aberration effects in the second aberration-corrected image 88A1 are too pronounced. Therefore, if only one of the first or second aberration-corrected image 86A1 is ultimately output, the user will be provided with an image that does not suit their preferences. By increasing the amount of training data for the generative model 82A1 or increasing the number of intermediate layers in the generative model 82A1, the performance of the generative model 82A1 can be improved, increasing the likelihood of obtaining an image closer to the user's preferences. However, the cost required to create the generation model 82A1 is high, which may ultimately lead to an increased price for the imaging device 10.

[0110] Therefore, in light of these circumstances, as shown in Figure 5 as an example, the imaging device 10 performs processing by the image adjustment unit 62C1 and the synthesis unit 62D1 on the first aberration-corrected image 86A1 and the second aberration-corrected image 88A1, thereby synthesizing the first aberration-corrected image 86A1 and the second aberration-corrected image 88A1.

[0111] As an example, as shown in Figure 5, the NVM64 stores a ratio of 90A. Ratio 90A is the ratio for combining the first aberration correction image 86A1 and the second aberration correction image 88A1, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A1 (i.e., processing using the generative model 82A1).

[0112] The ratio 90A is broadly divided into a first ratio 90A1 and a second ratio 90A2. The first ratio 90A1 is a value between 0 and 1, and the second ratio 90A2 is the value obtained by subtracting the value of the first ratio 90A1 from "1". In other words, the first ratio 90A1 and the second ratio 90A2 are set such that the sum of the first ratio 90A1 and the second ratio 90A2 is "1". The first ratio 90A1 and the second ratio 90A2 are variable values ​​that can be changed by user instructions. User instructions are received by the receiving device 76 (see Figure 2).

[0113] The image adjustment unit 62C1 adjusts the first aberration-corrected image 86A1 generated by the AI ​​method processing unit 62A1 using a first ratio 90A1. For example, the image adjustment unit 62C1 adjusts the pixel value of each pixel in the first aberration-corrected image 86A1 by multiplying the pixel value of each pixel in the first aberration-corrected image 86A1 by the first ratio 90A1.

[0114] The image adjustment unit 62C1 processes the second aberration-corrected image generated by the non-AI processing unit 62B1. The image 88A1 is adjusted using the second ratio 90A2. For example, the image adjustment unit 62C1 adjusts the pixel value of each pixel in the second aberration-corrected image 88A1 by multiplying the pixel value of each pixel in the second aberration-corrected image 88A1 by the second ratio 90A2.

[0115] The combining unit 62D1 generates a composite image 92A by combining the first aberration-corrected image 86A1, which has been adjusted by the image adjustment unit 62C1 with a first ratio 90A1, and the second aberration-corrected image 88A1, which has been adjusted by the image adjustment unit 62C1 with a second ratio 90A2. In other words, the combining unit 62D1 adjusts for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A1 by combining the first aberration-corrected image 86A1, which has been adjusted with a first ratio 90A1, and the second aberration-corrected image 88A1, which has been adjusted with a second ratio 90A2. To put it another way, the combining unit 62D1 adjusts for non-noise elements (here, as an example, the image region 75A1a) by combining the first aberration-corrected image 86A1, which has been adjusted with a first ratio 90A1, and the second aberration-corrected image 88A1, which has been adjusted with a second ratio 90A2. To put it another way, the synthesis unit 62D1 adjusts elements derived from processing using the generation model 82A1 (for example, the pixel values ​​of pixels whose aberration effect has been reduced by the generation model 82A1) by synthesizing the first aberration-corrected image 86A1 adjusted by the first ratio 90A1 and the second aberration-corrected image 88A1 adjusted by the second ratio 90A2.

[0116] The synthesis performed by the synthesis unit 62D1 is the addition of pixel values ​​at corresponding pixel positions between the first aberration-corrected image 86A1 and the second aberration-corrected image 88A1. Here, addition refers to, for example, simple addition. In the example shown in Figure 5, as an example of a synthesized image 92A, an image is shown when the first image 86A and the second aberration-corrected image 88A1 are synthesized when the values ​​of the first ratio 90A1 and the second ratio 90A2 are both "0.5". In this case, the influence of the first aberration-corrected image 86A1 (i.e., the influence of processing using the generation model 82A1) and the influence of the second aberration-corrected image 88A1 (i.e., the influence of processing using the digital filter 84A1) are reflected equally in the synthesized image 92A.

[0117] If the first ratio 90A1 is made larger than the second ratio 90A2, the influence of the first aberration correction image 86A1 will be greater than the influence of the second aberration correction image 88A1, and this will be reflected in the composite image 92A. Conversely, if the second ratio 90A2 is made larger than the first ratio 90A1, the influence of the second aberration correction image 88A1 will be greater than the influence of the first aberration correction image 86A1, and this will be reflected in the composite image 92A.

[0118] The compositing unit 62D1 performs various image processing operations on the composite image 92A (for example, known image processing operations such as offset correction, white balance correction, demosaicing, color correction, gamma correction, color space conversion, luminance processing, color difference processing, and resizing). The compositing unit 62D1 outputs the image obtained by performing various image processing operations on the composite image 92A as the processed image 75B (see Figure 2) to a predetermined output destination (for example, the image memory 46 shown in Figure 2).

[0119] Next, the operation of the imaging device 10 will be explained with reference to Figure 6. Figure 6 shows an example of the flow of image synthesis processing performed by the processor 62. The flow of image synthesis processing shown in Figure 6 is an example of an "image processing method" related to the technology of this disclosure.

[0120] In the image synthesis process shown in Figure 6, first, in step ST10, the AI ​​method processing unit 62A1 determines whether or not the image to be processed 75A1 has been generated by the image sensor 20 (see Figure 2). If the image to be processed 75A1 has not been generated by the image sensor 20 in step ST10, the determination is denied, and the image synthesis process proceeds to step ST32. If the image to be processed 75A1 has been generated by the image sensor 20 in step ST10, the determination is affirmed, and the image synthesis process proceeds to step ST12.

[0121] In step ST12, the AI-type processing unit 62A1 and the non-AI-type processing unit 62B1 acquire the image to be processed 75A1 from the image sensor 20. The processing in step ST12 is executed. After that, the image synthesis process proceeds to step ST14.

[0122] In step ST14, the AI ​​processing unit 62A1 inputs the image to be processed 75A1 acquired in step ST12 into the generation model 82A1. After the processing in step ST14 is executed, the image synthesis process proceeds to step ST16.

[0123] In step ST16, the AI ​​processing unit 62A1 acquires the first aberration-corrected image 86A1 output from the generation model 82A1, which was input to the generation model 82A1 in step ST14. After the processing in step ST16 is executed, the image synthesis process moves on to step ST18.

[0124] In step ST18, the non-AI processing unit 62B1 corrects the effects of aberrations (i.e., the image region 75A1a) by applying a digital filter 84A1 to the image to be processed 75A1 acquired in step ST12. After the processing in step ST18 is executed, the image synthesis process moves on to step ST20.

[0125] In step ST20, the non-AI processing unit 62B1 acquires a second aberration-corrected image 88A1 obtained in step ST18 by processing the target image 75A1 using the digital filter 84A1. After the processing in step ST20 is completed, the image synthesis process moves on to step ST22.

[0126] In step ST22, the image adjustment unit 62C1 obtains the first ratio 90A1 and the second ratio 90A2 from the NVM 64. After the processing in step ST22 is completed, the image synthesis process proceeds to step ST24.

[0127] In step ST24, the image adjustment unit 62C1 adjusts the first aberration-corrected image 86A1 using the first ratio 90A1 acquired in step ST22. After the processing in step ST24 is completed, the image synthesis process proceeds to step ST26.

[0128] In step ST26, the image adjustment unit 62C1 adjusts the second aberration-corrected image 88A1 using the second ratio 90A2 acquired in step ST22. After the processing in step ST26 is completed, the image synthesis process proceeds to step ST28.

[0129] In step ST28, the synthesis unit 62D1 adjusts the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A1 by combining the first aberration-corrected image 86A1 adjusted in step ST24 and the second aberration-corrected image 88A1 adjusted in step ST26. The synthesis of the first aberration-corrected image 86A1 adjusted in step ST24 and the second aberration-corrected image 88A1 adjusted in step ST26 generates a composite image 92A. After the processing in step ST28 is executed, the image synthesis process moves on to step ST30.

[0130] In step ST30, the image merging unit 62D1 performs various image processing on the composite image 92A. The image merging unit 62D1 then outputs the image obtained by performing various image processing on the composite image 92A as the processed image 75B to a predetermined output destination. After the processing in step ST30 is completed, the image merging process moves on to step ST32.

[0131] In step ST32, the image synthesis unit 62D1 determines whether the conditions for terminating the image synthesis process (hereinafter referred to as "termination conditions") have been met. Examples of termination conditions include the condition that an instruction to terminate the image synthesis process has been received by the receiving device 76. If the termination conditions are not met in step ST32, the determination is negated, and the image synthesis process proceeds to step ST10. If the termination conditions are met in step ST32, If the result is positive, the image synthesis process is terminated.

[0132] As described above, in the imaging device 10, the image to be processed 75A1 is acquired by the AI ​​processing unit 62A1 and the non-AI processing unit 62B1 as an image having an image region 75A1a that reflects the effects of aberrations. The image to be processed 75A1 is processed using the AI ​​method by the AI ​​processing unit 62A1 (i.e., processing using the generation model 82A1). This generates the first aberration-corrected image 86A1. The image to be processed 75A1 is also processed using the non-AI method by the non-AI processing unit 62B1 (i.e., processing using the digital filter 84A1). This generates the second aberration-corrected image 88A1.

[0133] However, if the first aberration-corrected image 86A1 were used as is as the final image provided to the user, the effects of processing using the generation model 82A1 would be noticeable, potentially resulting in an image that does not suit the user's preferences. Therefore, in the imaging device 10, the first aberration-corrected image 86A1 and the second aberration-corrected image 88A1 are adjusted by ratio 90A. Specifically, the first aberration-corrected image 86A1 is adjusted using the first ratio 90A1, and the second aberration-corrected image 88A1 is adjusted using the second ratio 90A2. Then, the first aberration-corrected image 86A1 adjusted using the first ratio 90A1 and the second aberration-corrected image 88A1 adjusted using the second ratio 90A2 are combined. As a result, an image (i.e., the combined image 92A) can be obtained in which the effects of processing using the generation model 82A1 (i.e., the effects of adjusting non-noise elements by processing using the generation model 82A1) are less noticeable than in the first aberration-corrected image 86A1.

[0134] In this embodiment, the second aberration-corrected image 88A1 is an image obtained by processing the image to be processed 75A1 using a digital filter 84A1, and the first aberration-corrected image 86A1 and the second aberration-corrected image 88A1 are combined to generate a composite image 92A. This allows the composite image 92A to include the effect of processing using the digital filter 84A1.

[0135] In this embodiment, the second aberration-corrected image 88A1 is an image in which the non-noise elements of the image to be processed 75A1 have been adjusted by a non-AI method of processing, and the first aberration-corrected image 86A1 and the second aberration-corrected image 88A1 are combined to generate a composite image 92A. This makes it possible to include the result of adjusting the non-noise elements of the image to be processed 75A1 by a non-AI method of processing in the composite image 92A1.

[0136] In this embodiment, the ratio 90A is set to adjust for any excess or deficiency of processing using the generation model 82A1. The first aberration-corrected image 86A1 and the second aberration-corrected image 88A1 are then combined using ratio 90A. This prevents the image (i.e., the combined image 92A) from becoming unsuitable for the user due to excessive influence from processing using the generation model 82A1. Furthermore, since the ratio 90A is changed according to user instructions, the degree to which the effects of processing using the generation model 82A1 remain in the combined image 92A, and the degree to which the effects of aberrations remain in the combined image 92A, can be adjusted to the user's preference.

[0137] In this embodiment, the first aberration-corrected image 86A1 is an image obtained by correcting, using an AI method, a phenomenon that appears in the image to be processed 75A1 due to the characteristics of the imaging device 10 (here, as an example, the optical characteristics of the imaging lens 40) (here, as an example, the effect of aberration). The first aberration-corrected image 86A1 is an image obtained by correcting, using a non-AI method, a phenomenon that appears in the image to be processed 75A1 due to the characteristics of the imaging device 10. The first aberration-corrected image 86A1 and the second aberration-corrected image 88A1 are then combined to form a composite image. Image 92A is generated. Therefore, it is possible to suppress the excessive or insufficient amount of correction applied to the composite image 92A by the AI ​​method for phenomena appearing in the processing target image 75A1 due to the characteristics of the imaging device 10 (here, as an example, the effect of aberrations). Also, since the effect of aberrations is not completely eliminated, it is possible to mitigate the unnatural appearance of the composite image 92A (i.e., the unnaturalness caused by the reduction of the effect of aberrations by the generation model 82A1). Furthermore, the effect of processing using the generation model 82A1 is not excessively reflected in the composite image 92A, and the effect of aberrations can be appropriately retained.

[0138] In the above embodiment, an example of a form in which the first aberration-corrected image 86A1 and the second aberration-corrected image 88A1 are combined was described, but the technology of this disclosure is not limited thereto. For example, instead of the second aberration-corrected image 88A1, the image to be processed 75A1 (i.e., an image in which non-noise elements have not been adjusted) may be combined with the first aberration-corrected image 86A1 so that elements originating from AI-based processing are adjusted. In other words, by combining the first aberration-corrected image 86A1 and the image to be processed 75A1, the image region in which the effect of aberration has been reduced by the AI ​​method (for example, here, as an example, the pixel values ​​of pixels in which the effect of aberration has been reduced by the generation model 82A1 may be adjusted. In this case, the influence that elements originating from the AI ​​method have on the combined image 92A is mitigated by elements originating from the image to be processed 75A1 (for example, the image region 75A1a). Therefore, it is possible to suppress the excess or deficiency of the correction amount applied to the combined image 92A by the AI ​​method for phenomena appearing in the image to be processed 75A1 due to the characteristics of the imaging device 10 (here, as an example, the effect of aberration). The image to be processed 75A1 that is combined with the first aberration-corrected image 86A1 is an example of the "second image" related to the technology of this disclosure.

[0139] In the above embodiment, the effect of aberration (image region 75A1a in the example shown in Figure 4) was given as an example of a phenomenon that appears in the image 75A1 to be processed due to the characteristics of the imaging device 10, but the technology of this disclosure is not limited thereto. For example, the phenomenon that appears in the image 75A1 to be processed due to the characteristics of the imaging device 10 may be flare and / or ghosting caused by the imaging lens 40, and in this case as well, the flare and / or ghosting can be reduced by AI-type processing and non-AI-type processing. In addition, the brightness determined according to the aperture of the imaging lens 40 can be adjusted by AI-type processing and non-AI-type processing.

[0140] In the above embodiment, the second aberration-corrected image 88A1 was exemplified as an image obtained by performing a non-AI method of processing on the image to be processed 75A1, but the technology of this disclosure is not limited thereto. For example, an image obtained without performing processing using the generative model 82A1 on an image different from the image to be processed 75A1 (for example, an image other than the image to be processed 75A1 among a plurality of images including the image to be processed 75A1 obtained by continuous shooting) may be applied instead of the second aberration-corrected image 88A1. The same applies to the following first and subsequent modifications.

[0141] [First variation] As an example, as shown in Figure 7, the processor 62 according to this first modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A2 instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B2 instead of a non-AI method processing unit 62B1. In this first modified example, explanations of matters that are the same as those explained before this first modified example will be omitted, and matters that are different from those explained before this first modified example will be explained.

[0142] The AI ​​processing unit 62A2 and the non-AI processing unit 62B2 receive the image to be processed 75A2 as input. The image to be processed 75A2 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A2 is a chromatic image and has a person region 94 and a background region 96. The person region 94 is the image region in which a person is depicted. The background region 96 is the image region in which the background is depicted. This is the image region.

[0143] Here, the person and background in the image to be processed 75A2 are examples of the "first subject" related to the technology of this disclosure. The person region 94 is an example of the "first region" and the "region in which a specific subject is depicted" related to the technology of this disclosure. The background region 96 is an example of the "second region, which is a region different from the first region" related to the technology of this disclosure. The color of the person region 94 and the color of the background region 96 are examples of the "non-noise element of the image to be processed," the "factor governing the visual impression given by the image to be processed," and the "color" related to the technology of this disclosure.

[0144] The AI ​​processing unit 62A2 performs AI-based processing on the image 75A2 to be processed. One example of AI-based processing on the image 75A2 is processing using the generative model 82A2. The generative model 82A2 is an example of the generative model 82A shown in Figure 3. The generative model 82A2 is a generative network that has already been trained to change the color of the person region 94 and the background region 96 so that the person region 94 and the background region 96 can be discriminated against.

[0145] The AI ​​processing unit 62A2 modifies the factors that govern the visual impression derived from the image 75A2 using an AI method. Specifically, the AI ​​processing unit 62A2 modifies the factors that govern the visual impression derived from the image 75A2 as non-noise elements by performing processing on the image 75A2 using the generative model 82A2. The factors that govern the visual impression derived from the image 75A2 are the color of the person region 94 and the color of the background region 96. In the example shown in Figure 7, the AI ​​processing unit 62A2 generates a first colored image 86A2 by performing processing on the image 75A2 using the generative model 82A2. The first colored image 86A2 is an image in which the person region 94 and the background region 96 are colored in a way that makes them distinguishable. For example, the person region 94 is colored with a chromatic color, and the background region 96 is colored with a grayscale color.

[0146] Here, the processing using the generation model 82A2 is an example of the "first AI processing," "first modification processing," and "coloring processing" related to the technology of this disclosure. The first colored image 86A2 is an example of the "first modified image" and "first colored image" related to the technology of this disclosure. "Generating the first colored image 86A2" is an example of "acquiring the first image" related to the technology of this disclosure.

[0147] The generative model 82A2 receives the image to be processed 75A2 as input. Based on the input image to be processed 75A2, the generative model 82A2 generates and outputs a first colored image 86A2.

[0148] The non-AI processing unit 62B2 performs non-AI processing on the image 75A2 to be processed. Non-AI processing refers to processing that does not use a neural network. In this first modified example, an example of processing that does not use a neural network is processing that does not use the generative model 82A2.

[0149] One example of non-AI processing of the image to be processed 75A2 is processing using a digital filter 84A2. The digital filter 84A2 is a digital filter configured to change chromatic colors in the image to be processed 75A2 to achromatic colors. The non-AI processing unit 62B2 generates a second colored image 88A2 by processing (i.e., filtering) the image to be processed 75A2 using the digital filter 84A2. In other words, the non-AI processing unit 62B2 generates a second colored image 88A2 by adjusting non-noise elements (here, color as an example) in the image to be processed 75A2 using a non-AI method. To put it another way, the non-AI processing unit 62B2 generates a second colored image 88A2 by changing chromatic colors in the image to be processed 75A2 to achromatic colors using a non-AI method.

[0150] Here, the processing using the digital filter 84A2 is an example of the "non-AI method processing that does not use a neural network" and the "second modification processing that modifies factors in a non-AI method" related to the technology of this disclosure. "Generating a second colored image 88A2" is an example of "acquiring a second image" related to the technology of this disclosure.

[0151] The digital filter 84A2 receives the image to be processed 75A2 as input. The digital filter 84A2 generates a second colored image 88A2 based on the input image to be processed 75A2. The second colored image 88A2 is an image obtained by changing the non-noise elements of the digital filter 84A2 (i.e., an image obtained by changing the non-noise elements of the image to be processed 75A2 through processing using the digital filter 84A2). In other words, the second colored image 88A2 is an image in which the colors in the image to be processed 75A2 have been changed by the digital filter 84A2 (i.e., an image in which chromatic colors have been changed to achromatic colors through processing using the digital filter 84A2 on the image to be processed 75A2). The second colored image 88A2 is an example of the "second image," "second modified image," and "second colored image" related to the technology of this disclosure.

[0152] Incidentally, the first colored image 86A2 obtained by applying the AI ​​method to the image 75A2 may contain colors that differ from the user's preferences due to the characteristics of the generative model 82A2 (e.g., the number of intermediate layers and / or the amount of training). If the influence of the AI ​​method is excessively reflected in the image 75A2, it is possible that colors that differ from the user's preferences will become prominent.

[0153] Therefore, in light of these circumstances, the imaging device 10 combines the first color image 86A2 and the second color image 88A2 by performing processing by the image adjustment unit 62C2 and processing by the synthesis unit 62D2 on the first color image 86A2 and the second color image 88A2, as shown in Figure 8 as an example.

[0154] As an example, as shown in Figure 8, the NVM64 stores a ratio of 90B. The ratio of 90B is the ratio used to combine the first colored image 86A2 and the second colored image 88A2, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A2 (i.e., processing using the generative model 82A2).

[0155] The ratio 90B is broadly divided into the first ratio 90B1 and the second ratio 90B2. The first ratio 90B1 is a value between 0 and 1, and the second ratio 90B2 is the value obtained by subtracting the value of the first ratio 90B1 from "1". In other words, the first ratio 90B1 and the second ratio 90B2 are set so that the sum of the first ratio 90B1 and the second ratio 90B2 is "1". The first ratio 90B1 and the second ratio 90B2 are variable values ​​that can be changed by user instructions.

[0156] The image adjustment unit 62C2 adjusts the first colored image 86A2 generated by the AI ​​method processing unit 62A2 using the first ratio 90B1. For example, the image adjustment unit 62C2 adjusts the pixel value of each pixel in the first colored image 86A2 by multiplying the pixel value of each pixel in the first colored image 86A2 by the first ratio 90B1.

[0157] The image adjustment unit 62C2 adjusts the second colored image 88A2 generated by the non-AI method processing unit 62B2 using the second ratio 90B2. For example, the image adjustment unit 62C2 adjusts the pixel value of each pixel in the second colored image 88A2 by multiplying the pixel value of each pixel in the second colored image 88A2 by the second ratio 90B2.

[0158] The composite unit 62D2 adjusts the first color by the image adjustment unit 62C2 at a first ratio 90B1. A composite image 92B is generated by combining the color image 86A2 and the second colored image 88A2, which has been adjusted by the second ratio 90B2 by the image adjustment unit 62C2. In other words, the compositing unit 62D2 adjusts the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A2 by combining the first colored image 86A2, which has been adjusted by the first ratio 90B1, and the second colored image 88A2, which has been adjusted by the second ratio 90B2. To put it another way, the compositing unit 62D2 adjusts non-noise elements (here, color as an example) by combining the first colored image 86A2, which has been adjusted by the first ratio 90B1, and the second colored image 88A2, which has been adjusted by the second ratio 90B2. To put it another way, the synthesis unit 62D2 adjusts elements derived from processing using the generation model 82A2 (for example, the pixel values ​​of pixels whose color has been changed by the generation model 82A2) by synthesizing the first colored image 86A2 adjusted by the first ratio 90B1 and the second colored image 88A2 adjusted by the second ratio 90B2.

[0159] The synthesis performed by the synthesis unit 62D2 is the addition of pixel values ​​at corresponding pixel positions between the first colored image 86A2 and the second colored image 88A2. The synthesis by the synthesis unit 62D2 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92B by the synthesis unit 62D2 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92B, after various image processing has been performed, is then output to a predetermined output destination by the synthesis unit 62D2.

[0160] Figure 9 shows an example of the image synthesis process flow according to this first modified example. The flowchart in Figure 9 differs from the flowchart in Figure 6 in that steps ST50 to ST68 are applied instead of steps ST12 to ST30.

[0161] In the image synthesis process shown in Figure 9, in step ST50, the AI-type processing unit 62A2 and the non-AI-type processing unit 62B2 acquire the image to be processed 75A2 from the image sensor 20. After the processing in step ST50 is completed, the image synthesis process proceeds to step ST52.

[0162] In step ST52, the AI ​​processing unit 62A2 inputs the image to be processed 75A2 acquired in step ST50 into the generation model 82A2. After the processing in step ST52 is executed, the image synthesis process proceeds to step ST54.

[0163] In step ST54, the AI ​​processing unit 62A2 obtains the first colored image 86A2 output from the generation model 82A2 after the image to be processed 75A2 is input to the generation model 82A2 in step ST52. After the processing in step ST54 is executed, the image synthesis process moves on to step ST56.

[0164] In step ST56, the non-AI processing unit 62B2 adjusts the colors within the image 75A2 acquired in step ST50 by applying a digital filter 84A2 to the image 75A2. After the processing in step ST56 is completed, the image synthesis process moves on to step ST58.

[0165] In step ST58, the non-AI processing unit 62B2 acquires a second colored image 88A2 obtained in step ST56 by processing the target image 75A2 using the digital filter 84A2. After the processing in step ST58 is completed, the image synthesis process moves on to step ST60.

[0166] In step ST60, the image adjustment unit 62C2 obtains the first ratio 90B1 and the second ratio 90B2 from the NVM 64. After the processing in step ST60 is completed, the image synthesis process proceeds to step ST62.

[0167] In step ST62, the image adjustment unit 62C2 adjusts the first colored image 86A2 using the first ratio 90B1 acquired in step ST60. After the processing in step ST62 is completed, the image synthesis process proceeds to step ST64.

[0168] In step ST64, the image adjustment unit 62C2 adjusts the second colored image 88A2 using the second ratio 90B2 acquired in step ST60. After the processing in step ST64 is completed, the image synthesis process proceeds to step ST66.

[0169] In step ST66, the synthesis unit 62D2 adjusts for any excess or deficiency in the AI ​​processing performed by the AI ​​processing unit 62A2 by combining the first colored image 86A2 adjusted in step ST62 and the second colored image 88A2 adjusted in step ST64. The synthesis of the first colored image 86A2 adjusted in step ST62 and the second colored image 88A2 adjusted in step ST64 generates the synthesized image 92B. After the processing in step ST66 is executed, the image synthesis process moves on to step ST68.

[0170] In step ST68, the image merging unit 62D2 performs various image processing on the composite image 92B. The image merging unit 62D2 then outputs the image obtained by performing various image processing on the composite image 92B as the processed image 75B to a predetermined output destination. After the processing in step ST68 is completed, the image merging process proceeds to step ST32.

[0171] As explained above, in the imaging device 10 according to this first modified example, a first colored image 86A2 is generated by changing the factors governing the visual impression given by the image to be processed 75A2 (here, as an example, color) using AI-based processing. A second colored image 88A2 is generated by changing the factors governing the visual impression given by the image to be processed 75A2 using non-AI-based processing. Furthermore, the first colored image 86A2 is adjusted according to a first ratio 90B1, and the second colored image 88A2 is adjusted according to a second ratio 90B2. Then, a composite image 92B is generated by combining the first colored image 86A2 adjusted according to the first ratio 90B1 and the second colored image 88A2 adjusted according to the second ratio 90B2. As a result, elements derived from the AI-based processing (for example, the color in the first colored image 86A2) are adjusted. In other words, the influence of elements derived from AI processing on the composite image 92B is mitigated by elements derived from non-AI processing (for example, the colors in the second colored image 88A2). Therefore, it is possible to suppress the excessive or insufficient amount of change made by the AI ​​method to the factors governing the visual impression given by the processed image 75A2 to the composite image 92B. As a result, the composite image 92B is an image in which the influence of AI processing is less noticeable compared to the first colored image 86A2, and an image suitable for users who do not want the influence of AI processing to be overly noticeable can be provided.

[0172] In this first modified example, the first colored image 86A2 is generated by coloring the person region 94 and the background region 96 within the image to be processed 75A2 in a discriminable manner using an AI method. Then, the first colored image 86A2 and the second colored image 88A2 are combined. This makes it possible to suppress excessive or insufficient coloring in the AI ​​processing for the combined image 92B. As a result, the combined image 92B is an image in which the coloring from the AI ​​processing method is less noticeable compared to the first colored image 86A2, and an image suitable for users who do not like the coloring from the AI ​​processing method to be overly noticeable can be provided.

[0173] In this first modified example, the person region 94 and the background region 96 within the image to be processed 75A2 are colored in a discriminable manner using the AI ​​method, and then the first colored image 86A2 and the second colored image 88A2 are combined. This suppresses excessive or insufficient coloring of the person region 94 by the AI ​​method. As a result, the combined image 92B is an image in which the coloring of the person region 94 by the AI ​​method is less noticeable compared to the first colored image 86A2, and the A This method can provide images suitable for users who do not like the colorization in the I-type processing to be overly prominent.

[0174] In the examples shown in Figures 7 to 9, the non-AI processing unit 62B2 was described as changing the color in the image 75A2 from chromatic to achromatic, regardless of the subject depicted in the image 75A2. However, the technology of this disclosure is not limited to this. For example, the non-AI processing unit 62B2 may use a non-AI method to color the person area 94 and the background area 96 in a way that makes them distinguishable.

[0175] In this case, for example, as shown in Figure 10, the non-AI processing unit 62B2 performs processing on the image to be processed 75A2 using a digital filter 84A2a. The digital filter 84A2a is configured to colorize the person area 94 and the background area 96 in the image to be processed 75A2 in a discriminable manner. The digital filter 84A2a may be configured to make one of the person area 94 and the background area 96 chromatic and the other achromatic. Alternatively, the digital filter 84A2a may be configured to make both the person area 94 and the background area 96 chromatic or achromatic, and to change the gradation between the person area 94 and the background area 96.

[0176] The non-AI processing unit 62B2 processes the image to be processed 75A2 using a digital filter 84A2a to generate a second colored image 88A2 in which the person region 94 and the background region 96 are colored in a way that makes them distinguishable. By combining the second colored image 88A2 and the first colored image 86A2, the user can easily visually recognize the difference between the person region 94 and the background region 86 in the combined image 92B.

[0177] In this first modified example, the person region 94 is given as an example of the "first region" and the "region in which a specific subject is depicted" relating to the technology of this disclosure. However, this is merely an example, and the technology of this disclosure is also applicable to regions other than the person region 94 (for example, a region in which a specific vehicle is depicted, a region in which a specific animal is depicted, a region in which a specific plant is depicted, a region in which a specific building is depicted, and / or a region in which a specific aircraft is depicted, etc.).

[0178] In this first modification, an example of a form in which the first colored image 86A2 and the second colored image 88A2 are combined was given, but this is merely one example. For example, instead of the second colored image 88A2, the image to be processed 75A2 (i.e., an image in which non-noise elements have not been adjusted) may be combined with the first colored image 86A2 so that elements derived from the AI ​​method processing (for example, the colors in the first colored image 86A2) are adjusted. In this case, the influence that elements derived from the AI ​​method processing have on the combined image 92B is mitigated by elements derived from the image to be processed 75A2 (for example, the colors in the image to be processed 75A2). Therefore, it is possible to suppress the excess or deficiency of the amount of change made by the AI ​​method to the factors governing the visual impression given by the image to be processed 75A2 to the combined image 92B. The image to be processed 75A2 combined with the first colored image 86A2 is an example of the "second image" related to the technology of this disclosure.

[0179] [Second variation] As an example, as shown in Figure 11, the processor 62 in this second modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A3 instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B3 instead of a non-AI method processing unit 62B1. In this second modified example, explanations of matters that are the same as those explained earlier will be omitted, and matters that differ from those explained earlier will be explained.

[0180] The AI ​​processing unit 62A3 and the non-AI processing unit 62B3 receive the image 75A3 to be processed. The image to be processed, 75A3, is an example of the image to be processed, 75A3, shown in Figure 2. The image to be processed, 75A3, is a chromatic image and has a person region 98 and a background region 100. The person region 98 is the image region in which the person is depicted. The background region 100 is the image region in which the background is depicted. Although a chromatic image is used as an example for the image to be processed, the image to be processed, 75A3, may also be an achromatic image.

[0181] The AI-based processing unit 62A3 and the non-AI-based processing unit 62B3 perform a process to adjust the contrast of the input image 75A3 to be processed. In this second modified example, the process of adjusting the contrast refers to a process of strengthening or weakening the contrast. The contrast of the image 75A3 to be processed is an example of the "non-noise elements of the image to be processed," the "factors governing the visual impression given by the image to be processed," and the "contrast of the image to be processed" related to the technology of this disclosure.

[0182] The AI ​​processing unit 62A3 performs AI-based processing on the image 75A3. One example of AI-based processing on the image 75A3 is processing using the generative model 82A3. The generative model 82A3 is an example of the generative model 82A shown in Figure 3. The generative model 82A3 is a generative network that has already been trained to adjust the contrast of the image 75A3.

[0183] The AI ​​processing unit 62A3 modifies the factors that govern the visual impression derived from the image 75A3 using the AI ​​method. Specifically, the AI ​​processing unit 62A3 modifies the factors that govern the visual impression derived from the image 75A3 as non-noise elements of the image 75A3 by performing processing on the image 75A3 using the generative model 82A3. The factors that govern the visual impression derived from the image 75A3 are the contrast of the image 75A3. In the example shown in Figure 11, the AI ​​processing unit 62A3 generates a first contrast-adjusted image 86A3 by performing processing on the image 75A3 using the generative model 82A3. The first contrast-adjusted image 86A3 is an image in which the contrast of the image 75A3 has been adjusted using the AI ​​method.

[0184] Here, the processing using the generation model 82A3 is an example of the "first AI processing," "first modification processing," and "first contrast adjustment processing" related to the technology of this disclosure. The first contrast adjustment image 86A3 is an example of the "first modified image" and "first contrast adjustment image" related to the technology of this disclosure. "Generating the first contrast adjustment image 86A3" is an example of "acquiring the first image" related to the technology of this disclosure.

[0185] The generation model 82A3 receives the image to be processed 75A3 as input. Based on the input image to be processed 75A3, the generation model 82A3 generates and outputs a first contrast-adjusted image 86A3. In the example shown in Figure 11, an example of the first contrast-adjusted image 86A3 is shown, which has a higher contrast than the image to be processed 75A3.

[0186] The non-AI processing unit 62B3 performs non-AI processing on the image 75A3 to be processed. Non-AI processing refers to processing that does not use a neural network. In this second modification, an example of processing that does not use a neural network is processing that does not use the generative model 82A3.

[0187] An example of non-AI processing of the image to be processed 75A3 is processing using a digital filter 84A3. The digital filter 84A3 is a digital filter configured to adjust the contrast of the image to be processed 75A3. The non-AI processing unit 62B3 generates a second contrast-adjusted image 88A3 by performing processing (i.e., filtering) on ​​the image to be processed 75A3 using the digital filter 84A3. In other words, the non-AI processing unit 62B3 generates a second contrast-adjusted image 88A3 by adjusting the non-noise elements (in this case, contrast) of the image 75A3 to be processed using a non-AI method. To put it another way, the non-AI processing unit 62B3 generates a second contrast-adjusted image 88A3 by changing the contrast of the image 75A3 to be processed using a non-AI method.

[0188] Here, the processing using the digital filter 84A3 is an example of the "non-AI method processing that does not use a neural network" and the "second modification processing that changes factors in a non-AI method" related to the technology of this disclosure. "Generating the second contrast-adjusted image 88A3" is an example of "acquiring the second image" related to the technology of this disclosure.

[0189] The digital filter 84A3 receives the image to be processed 75A3 as input. The digital filter 84A3 generates a second contrast-adjusted image 88A3 based on the input image to be processed 75A3. The second contrast-adjusted image 88A3 is an image obtained by changing the non-noise elements of the digital filter 84A3 (i.e., an image obtained by changing the non-noise elements of the image to be processed 75A3 through processing using the digital filter 84A3). In other words, the second contrast-adjusted image 88A3 is an image in which the contrast of the image to be processed 75A3 has been changed by the digital filter 84A3 (i.e., an image in which the contrast has been changed through processing using the digital filter 84A3 on the image to be processed 75A3). In the example shown in Figure 11, an example of a second contrast-adjusted image 88A3 is shown, which has higher contrast than the image to be processed 75A3 and lower contrast than the first contrast-adjusted image 86A3. The second contrast-adjusted image 88A3 is an example of the "second image," "second modified image," and "second contrast-adjusted image" related to the technology of this disclosure.

[0190] Incidentally, the first contrast-adjusted image 86A3 obtained by applying AI-based processing to the target image 75A3 may contain contrasts that differ from the user's preference due to the characteristics of the generative model 82A3 (e.g., the number of intermediate layers and / or the amount of training). If the influence of AI-based processing is excessively reflected in the target image 75A3, it is possible that contrasts that differ from the user's preference may become prominent.

[0191] Therefore, in light of these circumstances, the imaging device 10, as an example as shown in Figure 12, performs processing by the image adjustment unit 62C3 and the synthesis unit 62D3 on the first contrast adjustment image 86A3 and the second contrast adjustment image 88A3, thereby synthesizing the first contrast adjustment image 86A3 and the second contrast adjustment image 88A3.

[0192] As an example, as shown in Figure 12, the NVM64 stores a ratio of 90C. Ratio 90C is the ratio used to combine the first contrast adjustment image 86A3 and the second contrast adjustment image 88A3, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A3 (i.e., processing using the generative model 82A3).

[0193] The ratio 90C is broadly divided into the first ratio 90C1 and the second ratio 90C2. The first ratio 90C1 is a value between 0 and 1, and the second ratio 90C2 is the value obtained by subtracting the value of the first ratio 90C1 from "1". In other words, the first ratio 90C1 and the second ratio 90C2 are set such that the sum of the first ratio 90C1 and the second ratio 90C2 is "1". The first ratio 90C1 and the second ratio 90C2 are variable values ​​that can be changed by user instructions.

[0194] The image adjustment unit 62C3 adjusts the first contrast adjustment image 86A3 generated by the AI ​​method processing unit 62A3 using the first ratio 90C1. For example, the image adjustment unit 62C3 multiplies the pixel value of each pixel in the first contrast adjustment image 86A3 by the first ratio 90C1. This adjusts the pixel value of each pixel in the first contrast adjustment image 86A3.

[0195] The image adjustment unit 62C3 adjusts the second contrast adjustment image 88A3 generated by the non-AI method processing unit 62B3 using the second ratio 90C2. For example, the image adjustment unit 62C3 adjusts the pixel value of each pixel in the second contrast adjustment image 88A3 by multiplying the pixel value of each pixel in the second contrast adjustment image 88A3 by the second ratio 90C2.

[0196] The synthesis unit 62D3 generates a composite image 92C by combining the first contrast adjustment image 86A3, which has been adjusted by the image adjustment unit 62C3 with a first ratio 90C1, and the second contrast adjustment image 88A3, which has been adjusted by the image adjustment unit 62C3 with a second ratio 90C2. In other words, the synthesis unit 62D3 adjusts for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A3 by combining the first contrast adjustment image 86A3, which has been adjusted with a first ratio 90C1, and the second contrast adjustment image 88A3, which has been adjusted with a second ratio 90C2. To put it another way, the synthesis unit 62D3 adjusts for non-noise elements (here, contrast as an example) by combining the first contrast adjustment image 86A3, which has been adjusted with a first ratio 90C1, and the second contrast adjustment image 88A3, which has been adjusted with a second ratio 90C2. To put it another way, the synthesis unit 62D3 adjusts elements derived from processing using the generation model 82A3 (for example, the pixel values ​​of pixels whose contrast has been changed by the generation model 82A3) by synthesizing the first contrast adjustment image 86A3 adjusted by the first ratio 90C1 and the second contrast adjustment image 88A3 adjusted by the second ratio 90C2.

[0197] The synthesis performed by the synthesis unit 62D3 is the addition of pixel values ​​at corresponding pixel positions between the first contrast adjustment image 86A3 and the second contrast adjustment image 88A3. The synthesis by the synthesis unit 62D3 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92C by the synthesis unit 62D3 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92C, after various image processing has been performed, is then output to a predetermined output destination by the synthesis unit 62D3.

[0198] Figure 13 shows an example of the image synthesis process flow according to this second modified example. The flowchart shown in Figure 13 differs from the flowchart shown in Figure 6 in that steps ST100 to ST118 are applied instead of steps ST12 to ST30.

[0199] In the image synthesis process shown in Figure 13, in step ST100, the AI ​​method processing unit 62A3 and the non-AI method processing unit 62B3 acquire the image to be processed 75A3 from the image sensor 20. After the processing in step ST100 is completed, the image synthesis process proceeds to step ST102.

[0200] In step ST102, the AI ​​processing unit 62A3 inputs the image to be processed 75A3 acquired in step ST100 into the generation model 82A3. After the processing in step ST102 is completed, the image synthesis process proceeds to step ST104.

[0201] In step ST104, the AI ​​processing unit 62A3 acquires the first contrast-adjusted image 86A3 output from the generation model 82A3 after the image to be processed 75A3 is input to the generation model 82A3 in step ST102. After the processing in step ST104 is executed, the image synthesis process moves on to step ST106.

[0202] In step ST106, the non-AI processing unit 62B3 adjusts the contrast of the image 75A3 acquired in step ST100 by applying a digital filter 84A3 to it. After the processing in step ST106 is completed, the image synthesis process moves on to step ST108.

[0203] In step ST108, the non-AI processing unit 62B3 acquires a second contrast-adjusted image 88A3 obtained in step ST106 by processing the target image 75A3 using the digital filter 84A3. After the processing in step ST108 is executed, the image synthesis process moves on to step ST110.

[0204] In step ST110, the image adjustment unit 62C3 obtains the first ratio 90C1 and the second ratio 90C2 from the NVM 64. After the processing in step ST110 is completed, the image synthesis process proceeds to step ST112.

[0205] In step ST112, the image adjustment unit 62C3 adjusts the first contrast adjustment image 86A3 using the first ratio 90C1 acquired in step ST110. After the processing in step ST112 is completed, the image synthesis process proceeds to step ST114.

[0206] In step ST114, the image adjustment unit 62C3 adjusts the second contrast adjustment image 88A3 using the second ratio 90C2 acquired in step ST110. After the processing in step ST114 is completed, the image synthesis process proceeds to step ST116.

[0207] In step ST116, the synthesis unit 62D3 adjusts the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A3 by combining the first contrast adjustment image 86A3 adjusted in step ST112 and the second contrast adjustment image 88A3 adjusted in step ST114. The synthesis of the first contrast adjustment image 86A3 adjusted in step ST112 and the second contrast adjustment image 88A3 adjusted in step ST114 generates the synthesized image 92C. After the processing in step ST116 is executed, the image synthesis process moves on to step ST118.

[0208] In step ST118, the image merging unit 62D3 performs various image processing on the composite image 92C. The image merging unit 62D3 then outputs the image obtained by performing various image processing on the composite image 92C as the processed image 75B to a predetermined output destination. After the processing in step ST118 is completed, the image merging process proceeds to step ST32.

[0209] As explained above, in the imaging device 10 according to this second modified example, a first contrast-adjusted image 86A3 is generated by adjusting the contrast of the image to be processed 75A3 using the AI ​​method. A second contrast-adjusted image 88A3 is generated by adjusting the contrast of the image to be processed 75A3 using a non-AI method. The first contrast-adjusted image 86A3 and the second contrast-adjusted image 88A3 are then combined. This makes it possible to suppress excessive or insufficient contrast in the combined image 92C due to the AI ​​processing. As a result, the combined image 92C is an image in which the contrast from the AI ​​processing is less noticeable compared to the first contrast-adjusted image 86A3, and a suitable image can be provided for users who do not like the contrast from the AI ​​processing to be excessively noticeable.

[0210] In the examples shown in Figures 11 to 13, the processor 62 adjusts the contrast of the entire image 75A3 to be processed. However, the technology of this disclosure is not limited to this, and the processor 62 may also perform a process to adjust the clarity of the image 75A3 to be processed. Clarity refers to the contrast between the central pixel and the pixels adjacent to the central pixel within a pixel block consisting of multiple pixels. There are two types of clarity adjustment processes: AI-based clarity adjustment and non-AI-based clarity adjustment.

[0211] The process of adjusting clarity using an AI method is, for example, a process using the generative model 82A3a. In this case, the generative model 82A3a is a generative network that has already been trained to adjust the contrast and perform the first clarity processing in the manner described above. The first clarity processing refers to the process of adjusting clarity using the AI ​​method, that is, the process of locally adjusting the contrast using the AI ​​method. Local contrast adjustment using the AI ​​method is achieved, for example, by increasing or decreasing the difference between the pixel value of the central pixel 104A among the multiple pixels 104 that constitute the edge region of the person region 98 and the pixel values ​​of the multiple adjacent pixels 104B adjacent to the central pixel 104A, as shown in Figure 14.

[0212] An example of a non-AI method for adjusting clarity is a process using the generative model 82A3a. In this case, the digital filter 84A3a is a digital filter configured to adjust the contrast in the manner described above and to perform a second clarity processing. The second clarity processing refers to a process for adjusting clarity in a non-AI method, that is, a process for locally adjusting the contrast in a non-AI method. Local contrast adjustment in a non-AI method is achieved, for example, by increasing or decreasing the difference between the pixel value of the central pixel 106A among the multiple pixels 106 that constitute the edge region of the person region 98 and the pixel values ​​of the multiple adjacent pixels 106B adjacent to the central pixel 106A, as shown in Figure 14.

[0213] If the first clarity processing is performed here, the clarity of the first contrast-adjusted image 86A3 may be strengthened too much, causing an unnatural border to appear in the person region 98. Conversely, if the clarity of the first contrast-adjusted image 86A3 is weakened too much, fine details of the person region 98 may become unclear. Therefore, the first contrast-adjusted image 86A3, which has undergone the first clarity processing, and the second contrast-adjusted image 88A3, which has undergone the second clarity processing, are combined at a ratio of 90C. This adjusts the elements derived from the first clarity processing (for example, the pixel values ​​of pixels whose contrast has been changed by the generative model 82A3a). As a result, a composite image 92C can be obtained in which the influence of the first clarity processing is mitigated.

[0214] Here, an example is given in which a first contrast-adjusted image 86A3, which has undergone first clarity processing, and a second contrast-adjusted image 88A3, which has undergone second clarity processing, are combined. However, the first contrast-adjusted image 86A3, which has undergone first clarity processing, may be combined with the second contrast-adjusted image 88A3 or the image to be processed 75A3, which has not undergone second clarity processing. In this case as well, a similar effect can be expected.

[0215] The first clarity processing is an example of the "fifth contrast adjustment processing" related to the technology of this disclosure. The second clarity processing is an example of the "sixth contrast adjustment processing" related to the technology of this disclosure. The first contrast-adjusted image 86A3 obtained by performing the first clarity processing is an example of the "fifth contrast image" related to the technology of this disclosure. The second contrast-adjusted image 88A3 obtained by performing the second clarity processing is an example of the "sixth image" related to the technology of this disclosure.

[0216] The examples shown in Figures 11 to 13 describe the case where a person is included in the image to be processed 74A3, but the technology of this disclosure is not limited to this. For example, as shown in Figure 15, the image to be processed 74A3 may include a person and a vehicle (here, an automobile as an example). In this case, the image to be processed 74A3 has a person region 98 and a vehicle region 108. The vehicle region 108 is the image region in which the vehicle is included.

[0217] The AI ​​processing unit 62A adjusts the contrast of the image 75A3 to be processed according to the subject using an AI method. To achieve this, in the example shown in Figure 15, the AI ​​processing unit 62A performs processing using the generative model 82A3b. The generative model 82A3b is a generative network that has already been trained to adjust the contrast according to the subject. The AI ​​processing unit 62A uses the generation model 82A3b to adjust the contrast according to the person region 98 and the vehicle region 108 within the image 75A3 to be processed. Specifically, the person region 98 is given a contrast corresponding to the person represented by the person region 98, and the vehicle region 108 is given a contrast corresponding to the vehicle represented by the vehicle region 108. In the example shown in Figure 15, the contrast of the vehicle region 108 is higher than the contrast of the person region 98.

[0218] The non-AI processing unit 62B adjusts the contrast of the image to be processed 75A3 according to the subject, using a non-AI method. To achieve this, in the example shown in Figure 15, the non-AI processing unit 62B performs processing using a digital filter 84A3b. The digital filter 84A3b is a digital filter configured to adjust the contrast according to the subject. The non-AI processing unit 62B uses the digital filter 84A3b to adjust the contrast according to the person area 98 and the vehicle area 108 in the image to be processed 75A3. In the example shown in Figure 15, the contrast of the vehicle area 108 is higher than the contrast of the person area 98. Also, the contrast of the vehicle area 108 in the second contrast-adjusted image 88A3 is lower than the contrast of the vehicle area 108 in the first contrast-adjusted image 86A3. Also, the contrast of the person area 98 in the second contrast-adjusted image 88A3 is lower than the contrast of the person area 98 in the first contrast-adjusted image 86A3.

[0219] Here, if the first contrast-adjusted image 86A3 is excessively affected by the processing using the generation model 82A3b, the contrast of the person area 98 and the vehicle area 108 in the first contrast-adjusted image 86A3 may not suit the user's preference. For example, the user may feel that the contrast of the person area 98 and the vehicle area 108 in the first contrast-adjusted image 86A3 is too high. Therefore, the first contrast-adjusted image 86A3, obtained by processing the target image 75A3 using the generation model 82A3b, and the second contrast-adjusted image 88A3, obtained by processing the target image 75A3 using the digital filter 84A3b, are combined at a ratio of 90C. This adjusts the elements originating from the processing using the generation model 82A3b (for example, the pixel values ​​of pixels whose contrast has been changed by the generation model 82A3b). As a result, a composite image 92C can be obtained in which the influence of the processing using the generation model 82A3b has been mitigated.

[0220] Here, an example is given in which a first contrast-adjusted image 86A3 obtained by processing the target image 75A3 using the generation model 82A3b and a second contrast-adjusted image 88A3 obtained by processing the target image 75A3 using the digital filter 84A3b are combined. However, the technology of this disclosure is not limited to this. For example, the first contrast-adjusted image 86A3 obtained by processing the target image 75A3 using the generation model 82A3b may be combined with the second contrast-adjusted image 88A3 or the target image 75A3 which has not been processed using the digital filter 84A3b. In this case as well, similar effects can be expected.

[0221] The processing using the generation model 82A3b is an example of the "third contrast adjustment processing" related to the technology of this disclosure. The processing using the digital filter 84A3b is an example of the "fourth contrast adjustment processing" related to the technology of this disclosure. The first contrast adjustment image 86A3 obtained by performing processing using the generation model 82A3b on the image to be processed 75A3 is an example of the "third contrast adjustment image" related to the technology of this disclosure. The second contrast adjustment image 88A3 obtained by performing processing using the digital filter 84A3b on the image to be processed 75A3 is an example of the "fourth contrast adjustment image" related to the technology of this disclosure.

[0222] [Third variation] As an example, as shown in Figure 16, the processor 62 in this third modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A4 instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B4 instead of a non-AI method processing unit 62B1. In this third modified example, explanations of matters that are the same as those explained earlier will be omitted, and matters that are different from those explained earlier will be explained.

[0223] The AI ​​processing unit 62A4 and the non-AI processing unit 62B4 receive the image to be processed 75A4 as input. The image to be processed 75A4 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A4 is a chromatic image and has a person region 110. The person region 110 is the image region in which a person is depicted. Although a chromatic image is used as an example for the image to be processed 75A4 here, the image to be processed 75A4 may also be an achromatic image.

[0224] The AI-based processing unit 62A4 and the non-AI-based processing unit 62B4 perform a process to adjust the resolution of the input image 75A4 to be processed. In this third modified example, the process of adjusting the resolution refers to a process of increasing or decreasing the resolution. The resolution of the image 75A4 to be processed is an example of the "non-noise elements of the image to be processed," the "factors governing the visual impression given by the image to be processed," and the "resolution of the image to be processed" related to the technology of this disclosure.

[0225] The AI ​​processing unit 62A4 performs AI-based processing on the image 75A4 to be processed. An example of AI-based processing on the image 75A4 to be processed is processing using the generative model 82A4. The generative model 82A4 is an example of the generative model 82A shown in Figure 3. The generative model 82A4 is a generative network that has already undergone training to adjust the resolution of the image 75A4 to be processed. In this third modified example, the training to adjust the resolution of the image 75A4 to be processed refers to training to super-enhance the resolution of the image 75A4 to be processed.

[0226] The AI ​​processing unit 62A4 modifies the factors that govern the visual impression derived from the image 75A4 using the AI ​​method. Specifically, the AI ​​processing unit 62A4 modifies the factors that govern the visual impression derived from the image 75A4 as non-noise elements by performing processing on the image 75A4 using the generative model 82A4. The factor that governs the visual impression derived from the image 75A4 is the resolution of the image 75A4. In the example shown in Figure 16, the AI ​​processing unit 62A4 generates a first resolution-adjusted image 86A4 by performing processing on the image 75A4 using the generative model 82A4. The first resolution-adjusted image 86A4 is an image in which the resolution of the image 75A4 has been adjusted using the AI ​​method. Here, an image in which the resolution of the image 75A4 has been adjusted using the AI ​​method refers to an image in which the image 75A4 has been super-resolved using the AI ​​method.

[0227] Here, the processing using the generation model 82A4 is an example of the "first AI processing," "first modification processing," and "first resolution adjustment processing" related to the technology of this disclosure. The first resolution adjustment image 86A4 is an example of the "first modified image" and "first resolution adjustment image" related to the technology of this disclosure. "Generating the first resolution adjustment image 86A4" is an example of "acquiring the first image" related to the technology of this disclosure.

[0228] The generation model 82A4 receives the image to be processed 75A4 as input. Based on the input image to be processed 75A4, the generation model 82A4 generates and outputs a first resolution-adjusted image 86A4. In the example shown in Figure 16, an example of the first resolution-adjusted image 86A4 is shown, which is an image of the image to be processed 75A4 that has been super-resolved.

[0229] The non-AI processing unit 62B4 performs non-AI processing on the image 75A3 to be processed. Non-AI processing refers to processing that does not use a neural network. In this third modification, an example of processing that does not use a neural network is processing that does not use the generative model 82A4.

[0230] One example of non-AI processing of the image 75A4 is processing using a digital filter 84A4. The digital filter 84A4 is a digital filter configured to adjust the resolution of the image 75A4. In the following explanation, we will use a digital filter 84A4 configured to super-enhance the resolution of the image 75A4 as an example.

[0231] The non-AI processing unit 62B4 generates a second resolution-adjusted image 88A4 by performing processing (i.e., filtering) on ​​the image to be processed 75A4 using a digital filter 84A4. In other words, the non-AI processing unit 62B4 generates a second resolution-adjusted image 88A4 by adjusting the non-noise elements (here, resolution as an example) of the image to be processed 75A4 using a non-AI method. To put it another way, the non-AI processing unit 62B4 generates a second resolution-adjusted image 88A4 by adjusting the resolution of the image to be processed 75A4 using a non-AI method. An image in which the resolution of the image to be processed 75A4 has been adjusted using a non-AI method refers to an image in which the image to be processed 75A4 has been super-adjusted using a non-AI method.

[0232] Here, the processing using the digital filter 84A4 is an example of the "non-AI method processing that does not use a neural network" and the "second modification processing that changes factors in a non-AI method" related to the technology of this disclosure. "Generating a second resolution-adjusted image 88A4" is an example of "acquiring a second image" related to the technology of this disclosure.

[0233] The digital filter 84A4 receives the image to be processed 75A4 as input. The digital filter 84A4 generates a second resolution-adjusted image 88A4 based on the input image to be processed 75A4. The second resolution-adjusted image 88A4 is an image obtained by changing the non-noise elements of the digital filter 84A4 (i.e., an image obtained by changing the non-noise elements of the image to be processed 75A4 through processing using the digital filter 84A4). In other words, the second resolution-adjusted image 88A4 is an image in which the resolution of the image to be processed 75A4 has been adjusted by the digital filter 84A4 (i.e., an image in which the resolution has been adjusted through processing using the digital filter 84A4). In the example shown in Figure 16, an example of a second resolution-adjusted image 88A4 is shown in which the image to be processed 75A4 has been super-resolved and has a lower resolution than the first resolution-adjusted image 86A4. The second resolution-adjusted image 88A4 is an example of the "second image," "second modified image," and "second resolution-adjusted image" related to the technology of this disclosure.

[0234] Incidentally, the resolution of the first resolution-adjusted image 86A4 obtained by applying AI-based processing to the target image 75A4 may differ from the user's preference due to the characteristics of the generative model 82A4 (e.g., the number of intermediate layers and / or the amount of training). If the influence of AI-based processing is excessively reflected on the target image 75A4, the resolution may end up being too high or too low compared to the user's preference.

[0235] Therefore, in light of these circumstances, the imaging device 10, as an example as shown in Figure 17, performs processing by the image adjustment unit 62C4 and the synthesis unit 62D4 on the first resolution adjustment image 86A4 and the second resolution adjustment image 88A4, thereby synthesizing the first resolution adjustment image 86A4 and the second resolution adjustment image 88A4.

[0236] As an example, as shown in Figure 17, the NVM64 stores a percentage of 90D. 90D is the ratio at which the first resolution adjustment image 86A4 and the second resolution adjustment image 88A4 are combined, and is set to adjust for any excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A4 (i.e., processing using the generative model 82A4).

[0237] The ratio 90D is broadly divided into the first ratio 90D1 and the second ratio 90D2. The first ratio 90D1 is a value between 0 and 1, and the second ratio 90D2 is the value obtained by subtracting the value of the first ratio 90D1 from "1". In other words, the first ratio 90D1 and the second ratio 90D2 are set so that the sum of the first ratio 90D1 and the second ratio 90D2 is "1". The first ratio 90D1 and the second ratio 90D2 are variable values ​​that can be changed by user instructions.

[0238] The image adjustment unit 62C4 adjusts the first resolution adjustment image 86A4 generated by the AI ​​method processing unit 62A4 using the first ratio 90D1. For example, the image adjustment unit 62C4 adjusts the pixel value of each pixel in the first resolution adjustment image 86A4 by multiplying the pixel value of each pixel in the first resolution adjustment image 86A4 by the first ratio 90D1.

[0239] The image adjustment unit 62C4 adjusts the second resolution adjustment image 88A4 generated by the non-AI method processing unit 62B4 using the second ratio 90D2. For example, the image adjustment unit 62C4 adjusts the pixel value of each pixel in the second resolution adjustment image 88A4 by multiplying the pixel value of each pixel in the second resolution adjustment image 88A4 by the second ratio 90D2.

[0240] The compositing unit 62D4 generates a composite image 92D by compositing the first resolution adjustment image 86A4, which has been adjusted by the image adjustment unit 62C4 with a first ratio 90D1, and the second resolution adjustment image 88A4, which has been adjusted by the image adjustment unit 62C4 with a second ratio 90D2. In other words, the compositing unit 62D4 adjusts for any excess or deficiency in the AI ​​method processing performed by the AI ​​method processing unit 62A4 by compositing the first resolution adjustment image 86A4, which has been adjusted with a first ratio 90D1, and the second resolution adjustment image 88A4, which has been adjusted with a second ratio 90D2. To put it another way, the compositing unit 62D4 adjusts for non-noise elements (here, resolution as an example) by compositing the first resolution adjustment image 86A4, which has been adjusted with a first ratio 90D1, and the second resolution adjustment image 88A4, which has been adjusted with a second ratio 90D2. To put it another way, the synthesis unit 62D4 synthesizes the first resolution-adjusted image 86A4 adjusted by the first ratio 90D1 and the second resolution-adjusted image 88A4 adjusted by the second ratio 90D2, thereby adjusting the elements derived from processing using the generation model 82A4 (for example, the pixel values ​​of pixels whose resolution has been adjusted by the generation model 82A4).

[0241] The synthesis performed by the synthesis unit 62D4 is the addition of pixel values ​​at corresponding pixel positions between the first resolution adjustment image 86A4 and the second resolution adjustment image 88A4. The synthesis by the synthesis unit 62D4 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92D by the synthesis unit 62D4 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92D, after various image processing has been performed, is then output to a predetermined output destination by the synthesis unit 62D4.

[0242] Figure 18 shows an example of the image synthesis process flow according to this third modified example. The flowchart shown in Figure 18 differs from the flowchart shown in Figure 6 in that steps ST150 to ST168 are applied instead of steps ST12 to ST30.

[0243] In the image synthesis process shown in Figure 18, in step ST150, the AI ​​method processing unit 62A4 and the non-AI method processing unit 62B4 acquire the image to be processed 75A4 from the image sensor 20. After the processing in step ST150 is completed, the image synthesis process proceeds to step ST152.

[0244] In step ST152, the AI ​​method processing unit 62A4 processes the data acquired in step ST150. The image to be processed, 75A4, is input to the generation model, 82A4. This causes the image to be processed, 75A4, to be super-resolution using the AI ​​method. After the processing in step ST152 is executed, the image synthesis process moves on to step ST154.

[0245] In step ST154, the AI ​​processing unit 62A4 acquires the first resolution-adjusted image 86A4 output from the generation model 82A4 after the image to be processed 75A4 is input to the generation model 82A4 in step ST152. After the processing in step ST154 is executed, the image synthesis process moves on to step ST156.

[0246] In step ST156, the non-AI processing unit 62B4 adjusts the resolution of the image 75A4 acquired in step ST150 by applying a digital filter 84A4 to it. This makes the image 75A4 super-resolution using the non-AI method. After the processing in step ST156 is completed, the image synthesis process moves on to step ST158.

[0247] In step ST158, the non-AI processing unit 62B4 acquires a second resolution-adjusted image 88A4 obtained in step ST156 by processing the target image 75A4 using the digital filter 84A4. After the processing in step ST158 is executed, the image synthesis process moves on to step ST160.

[0248] In step ST160, the image adjustment unit 62C4 acquires the first ratio 90D1 and the second ratio 90D2 from the NVM 64. After the processing of step ST160 is executed, the image synthesis process proceeds to step ST162.

[0249] In step ST162, the image adjustment unit 62C4 adjusts the first resolution adjustment image 86A4 using the first ratio 90D1 acquired in step ST160. After the processing of step ST162 is executed, the image synthesis process proceeds to step ST164.

[0250] In step ST164, the image adjustment unit 62C4 adjusts the second resolution adjustment image 88A4 using the second ratio 90D2 acquired in step ST160. After the processing of step ST164 is executed, the image synthesis process proceeds to step ST166.

[0251] In step ST166, the synthesis unit 62D4 synthesizes the first resolution adjustment image 86A4 adjusted in step ST162 and the second resolution adjustment image 88A4 adjusted in step ST164, thereby adjusting the excess or deficiency of the AI method processing by the AI method processing unit 62A4. The synthesized image 92D is generated by synthesizing the first resolution adjustment image 86A4 adjusted in step ST162 and the second resolution adjustment image 88A4 adjusted in step ST164. After the processing of step ST166 is executed, the image synthesis process proceeds to step ST168.

[0252] In step ST168, the synthesis unit 62D4 performs various image processes on the synthesized image 92D. Then, the synthesis unit 62D4 outputs the image obtained by performing various image processes on the synthesized image 92D as the processed image 75B to a predetermined output destination. After the processing of step ST168 is executed, the image synthesis process proceeds to step ST32.

[0253] As explained above, in the imaging device 10 according to this third modified example, the resolution of the image to be processed 75A4 is adjusted using the AI ​​method to generate a first resolution-adjusted image 86A4. Then, the first resolution-adjusted image 86A4 and the second resolution-adjusted image 88A4 are combined. This makes it possible to suppress the excess or deficiency of resolution in the AI-based processing for the combined image 92D. As a result, the combined image 92D is, compared to the first resolution-adjusted image 86A4, The AI-processed image has less noticeable resolution, making it suitable for users who do not like the AI-processed resolution to be overly prominent.

[0254] In this third modification, the first resolution adjustment image 86A4 is an image in which the image to be processed 75A4 has been super-resolution processed using the AI ​​method, and the second resolution adjustment image 88A4 is an image in which the image to be processed 75A4 has been super-resolution processed using a non-AI method. Then, the composite image 92D is generated by combining the image in which the image to be processed 75A4 has been super-resolution processed using the AI ​​method and the image in which the image to be processed 75A4 has been super-resolution processed using a non-AI method. Therefore, it is possible to suppress the issue of the resolution obtained by the AI ​​super-resolution processing being too high or too low in relation to the composite image 92D.

[0255] Here, an example is given in which a first resolution-adjusted image 86A4 obtained by processing the target image 75A4 using the generation model 82A4 and a second resolution-adjusted image 88A4 obtained by processing the target image 75A4 using the digital filter 84A4 are combined. However, the technology of this disclosure is not limited to this. For example, the first resolution-adjusted image 86A4 obtained by processing the target image 75A4 using the generation model 82A4 and the target image 75A4 (i.e., an image in which non-noise elements have not been adjusted) may be combined. In this case as well, similar effects can be expected.

[0256] [Fourth variation] As an example, as shown in Figure 19, the processor 62 according to this fourth modification differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A5 instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B5 instead of a non-AI method processing unit 62B1. In this fourth modification, explanations of matters that are the same as those explained before this fourth modification will be omitted, and matters that are different from those explained before this fourth modification will be explained.

[0257] The AI ​​processing unit 62A5 and the non-AI processing unit 62B5 receive the image to be processed 75A5 as input. The image to be processed 75A5 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A5 is a chromatic image. Although a chromatic image is used as an example here for the image to be processed 75A5, the image to be processed 75A5 may also be an achromatic image.

[0258] The AI-based processing unit 62A5 and the non-AI-based processing unit 62B5 perform processing to expand the dynamic range of the input image to be processed 75A5. The dynamic range of the image to be processed 75A5 is an example of the "non-noise elements of the image to be processed," the "factors governing the visual impression given by the image to be processed," and the "dynamic range of the image to be processed" related to the technology of this disclosure.

[0259] The AI ​​processing unit 62A5 performs AI-based processing on the image 75A5. One example of AI-based processing on the image 75A5 is processing using the generative model 82A5. The generative model 82A5 is an example of the generative model 82A shown in Figure 3. The generative model 82A5 is a generative network that has already been trained to expand the dynamic range of the image 75A5. In this fourth modification, the training to adjust the dynamic range of the image 75A5 refers to training to make the image 75A5 high dynamic range. Hereafter, "high dynamic range" will be referred to as "HDR".

[0260] The AI ​​processing unit 62A5 modifies the factors that govern the visual impression derived from the image 75A5 to be processed using the AI ​​method. In other words, the AI ​​processing unit 62A5 modifies the factors that govern the visual impression derived from the image 75A5 to be processed. By processing A5 using the generative model 82A5, the non-noise elements of the image 75A5 being processed are modified, specifically the factors that govern the visual impression derived from the image 75A5. The factors that govern the visual impression derived from the image 75A5 being processed are the dynamic range of the image 75A5. In the example shown in Figure 19, the AI ​​processing unit 62A5 generates the first HDR image 86A5 by processing the image 75A5 using the generative model 82A5. The first HDR image 86A5 is an image in which the dynamic range of the image 75A5 has been expanded using the AI ​​method.

[0261] Here, the processing using the generation model 82A5 is an example of the "first AI processing," "first modification processing," and "extension processing" related to the technology of this disclosure. The first HDR image 86A5 is an example of the "first modified image" and "first HDR image" related to the technology of this disclosure. "Generating the first HDR image 86A5" is an example of "acquiring the first image" related to the technology of this disclosure.

[0262] The generation model 82A5 receives the image to be processed 75A5 as input. Based on the input image to be processed 75A5, the generation model 82A5 generates and outputs a first HDR image 86A5.

[0263] The non-AI processing unit 62B5 performs non-AI processing on the image 75A5 to be processed. Non-AI processing refers to processing that does not use a neural network. In this fourth modification, an example of processing that does not use a neural network is processing that does not use the generative model 82A5.

[0264] One example of non-AI processing for the image 75A5 is processing using a digital filter 84A5. The digital filter 84A5 is configured to expand the dynamic range of the image 75A5. In the following explanation, we will use a digital filter 84A5 configured to convert the image 75A5 to HDR as an example.

[0265] The non-AI processing unit 62B5 generates a second HDR image 88A5 by performing processing (i.e., filtering) on ​​the image to be processed 75A5 using a digital filter 84A5. In other words, the non-AI processing unit 62B5 generates a second HDR image 88A5 by changing the non-noise elements of the image to be processed 75A5 using a non-AI method. To put it another way, the non-AI processing unit 62B5 generates a second HDR image 88A5 by expanding the dynamic range of the image to be processed 75A5 using a non-AI method.

[0266] Here, the processing using the digital filter 84A5 is an example of the "non-AI method processing that does not use a neural network" and the "second modification processing that modifies factors in a non-AI method" related to the technology of this disclosure. "Generating a second HDR image 88A5" is an example of "acquiring a second image" related to the technology of this disclosure.

[0267] The digital filter 84A5 receives the image to be processed 75A5 as input. The digital filter 84A5 generates a second HDR image 88A5 based on the input image to be processed 75A5. The second HDR image 88A5 is an image obtained by changing the non-noise elements of the digital filter 84A5 (i.e., an image obtained by changing the non-noise elements of the image to be processed 75A5 through processing using the digital filter 84A5). In other words, the second HDR image 88A5 is an image in which the dynamic range of the image to be processed 75A5 has been changed by the digital filter 84A5 (i.e., an image in which the dynamic range has been expanded through processing using the digital filter 84A5 on the image to be processed 75A5). The second HDR image 88A5 is an example of the "second image," "second modified image," and "second HDR image" related to the technology of this disclosure.

[0268] Incidentally, the dynamic range of the first HDR image 86A5 obtained by applying AI-based processing to the target image 75A5 may differ from the user's preference due to the characteristics of the generative model 82A5 (e.g., the number of intermediate layers and / or the amount of training). If the influence of AI-based processing is excessively reflected on the target image 75A5, the dynamic range may become too wide or too narrow compared to the user's preference.

[0269] Therefore, in light of these circumstances, the imaging device 10 combines the first HDR image 86A5 and the second HDR image 88A5 by performing processing in the image adjustment unit 62C5 and processing in the synthesis unit 62D5 on the first HDR image 86A5 and the second HDR image 88A5, as shown in Figure 20 as an example.

[0270] As an example, as shown in FIG. 20, a ratio 90E is stored in the NVM64. The ratio 90E is a ratio for synthesizing the first HDR image 86A5 and the second HDR image 88A5, and is determined to adjust the excess or deficiency of the AI method processing (that is, the processing using the generation model 82A5) by the AI method processing unit 62A5.

[0271] The ratio 90E is roughly divided into a first ratio 90E1 and a second ratio 90E2. The first ratio 90E1 is a value of 0 or more and 1 or less, and the second ratio 90E2 is a value obtained by subtracting the value of the first ratio 90E1 from "1". That is, the first ratio 90E1 and the second ratio 90E2 are determined such that the sum of the first ratio 90E1 and the second ratio 90E2 is "1". The first ratio 90E1 and the second ratio 90E2 are variable values that are changed according to an instruction from the user.

[0272] The image adjustment unit 62C5 adjusts the first HDR image 86A5 generated by the AI method processing unit 62A5 using the first ratio 90E1. For example, the image adjustment unit 62C5 adjusts the pixel value of each pixel of the first HDR image 86A5 by multiplying the pixel value of each pixel of the first HDR image 86A5 by the first ratio 90E1.

[0273] The image adjustment unit 62C5 adjusts the second HDR image 88A5 generated by the non-AI method processing unit 62B5 using the second ratio 90E2. For example, the image adjustment unit 62C5 adjusts the pixel value of each pixel of the second HDR image 88A5 by multiplying the pixel value of each pixel of the second HDR image 88A5 by the second ratio 90E2.

[0274] The synthesis unit 62D5 generates a composite image 92E by combining the first HDR image 86A5, which has been adjusted by the image adjustment unit 62C5 with a first ratio 90E1, and the second HDR image 88A5, which has been adjusted by the image adjustment unit 62C5 with a second ratio 90E2. In other words, the synthesis unit 62D5 adjusts for any excess or deficiency in the AI ​​processing performed by the AI ​​processing unit 62A5 by combining the first HDR image 86A5, which has been adjusted with a first ratio 90E1, and the second HDR image 88A5, which has been adjusted with a second ratio 90E2. To put it another way, the synthesis unit 62D5 adjusts for non-noise elements (in this case, dynamic range as an example) by combining the first HDR image 86A5, which has been adjusted with a first ratio 90E1, and the second HDR image 88A5, which has been adjusted with a second ratio 90E2. To put it another way, the synthesis unit 62D5 synthesizes the first HDR image 86A5 adjusted with a first ratio 90E1 and the second HDR image 88A5 adjusted with a second ratio 90E2, thereby adjusting elements derived from processing using the generative model 82A5 (for example, the pixel values ​​of pixels whose dynamic range has been expanded by the generative model 82A5).

[0275] The synthesis performed by the synthesis unit 62D5 is the addition of pixel values ​​at corresponding pixel positions between the first HDR image 86A5 and the second HDR image 88A5. The synthesis by the synthesis unit 62D5 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Also, the synthesized image 92E Similarly, the compositing unit 62D5 performs various image processing on the composite image 92E, in the same manner as the composite image 92A shown in Figure 5. The composite image 92E, after various image processing steps have been performed, is then output to a predetermined output destination by the compositing unit 62D5.

[0276] Figure 21 shows an example of the image synthesis process flow according to this fourth modified example. The flowchart shown in Figure 21 differs from the flowchart shown in Figure 6 in that steps ST200 to ST218 are applied instead of steps ST12 to ST30.

[0277] In the image synthesis process shown in Figure 21, in step ST200, the AI ​​method processing unit 62A5 and the non-AI method processing unit 62B5 acquire the image to be processed 75A5 from the image sensor 20. After the processing in step ST200 is completed, the image synthesis process proceeds to step ST202.

[0278] In step ST202, the AI ​​processing unit 62A5 inputs the image to be processed 75A5 acquired in step ST200 to the generation model 82A5. As a result, the image to be processed 75A5 is converted to HDR using the AI ​​method. After the processing in step ST202 is completed, the image synthesis process moves on to step ST204.

[0279] In step ST204, the AI ​​processing unit 62A5 acquires the first HDR image 86A5 output from the generation model 82A5, which was created when the image to be processed 75A5 was input to the generation model 82A5 in step ST202. After the processing in step ST204 is completed, the image synthesis process moves on to step ST206.

[0280] In step ST206, the non-AI processing unit 62B5 expands the dynamic range of the image 75A5 by applying a digital filter 84A5 to the image 75A5 acquired in step ST200. As a result, the image 75A5 is converted to HDR using the non-AI method. After the processing in step ST206 is executed, the image synthesis process moves on to step ST208.

[0281] In step ST208, the non-AI processing unit 62B5 acquires a second HDR image 88A5 obtained in step ST206 by processing the target image 75A5 using the digital filter 84A5. After the processing in step ST208 is completed, the image synthesis process moves on to step ST210.

[0282] In step ST210, the image adjustment unit 62C5 obtains the first ratio 90E1 and the second ratio 90E2 from the NVM 64. After the processing in step ST210 is completed, the image synthesis process proceeds to step ST212.

[0283] In step ST212, the image adjustment unit 62C5 adjusts the first HDR image 86A5 using the first ratio 90E1 acquired in step ST210. After the processing in step ST212 is completed, the image synthesis process proceeds to step ST214.

[0284] In step ST214, the image adjustment unit 62C5 adjusts the second HDR image 88A5 using the second ratio 90E2 acquired in step ST210. After the processing in step ST214 is completed, the image synthesis process proceeds to step ST216.

[0285] In step ST216, the synthesis unit 62D5 synthesizes the first HDR image 86A5 adjusted in step ST212 and the second HDR image 88A5 adjusted in step ST214, thereby adjusting the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A5. The composite image 92E is generated by combining it with the DR image 88A5. After the processing in step ST216 is executed, the image synthesis process moves on to step ST218.

[0286] In step ST218, the image merging unit 62D5 performs various image processing on the composite image 92E. The image merging unit 62D5 then outputs the image obtained by performing various image processing on the composite image 92E as the processed image 75B to a predetermined output destination. After the processing in step ST218 is completed, the image merging process proceeds to step ST32.

[0287] As explained above, in the imaging device 10 according to this fourth modified example, the dynamic range of the image to be processed 75A5 is converted to HDR using the AI ​​method to generate a first HDR image 86A5. Then, the first HDR image 86A5 and the second HDR image 88A5 are combined. This makes it possible to suppress any excess or deficiency of the dynamic range in the AI ​​method processing in the combined image 92E. As a result, the combined image 92E is an image in which the dynamic range in the AI ​​method processing is less noticeable compared to the first HDR image 86A5, and an image suitable for users who do not like to have the dynamic range in the AI ​​method processing be overly noticeable can be provided.

[0288] Here, we have given an example of a configuration in which a first HDR image 86A5 obtained by processing the target image 75A5 using the generation model 82A5 and a second HDR image 88A5 obtained by processing the target image 75A5 using the digital filter 84A5 are combined. However, the technology of this disclosure is not limited to this. For example, the first HDR image 86A5 obtained by processing the target image 75A5 using the generation model 82A5 and the target image 75A5 (i.e., an image in which non-noise elements have not been adjusted) may be combined. In this case as well, similar effects can be expected.

[0289] [Fifth variation] As an example, as shown in Figure 22, the processor 62 in this fifth modified example differs from the processor 62 shown in Figure 4 in that it has an AI-type processing unit 62A6 instead of an AI-type processing unit 62A1, and a non-AI-type processing unit 62B6 instead of a non-AI-type processing unit 62B1. In this fifth modified example, explanations of matters that are the same as those explained before this fifth modified example will be omitted, and matters that are different from those explained before this fifth modified example will be explained.

[0290] The AI ​​processing unit 62A6 and the non-AI processing unit 62B6 receive the image to be processed 75A6 as input. The image to be processed 75A6 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A6 is a chromatic image and has edge regions 112. The edge regions 112 are image regions where the edges of the subject are captured (for example, high-frequency components above a certain value). Here, a chromatic image is used as an example of the image to be processed 75A6, but the image to be processed 75A6 may also be an achromatic image.

[0291] The AI-based processing unit 62A6 and the non-AI-based processing unit 62B6 perform processing to emphasize the edge region 112 in the input image 75A6 to be processed more than the non-edge region (hereinafter simply referred to as the "non-edge region") which is a different region from the edge region 112. The edge region 112 is an example of the "edge region in the image to be processed" related to the technology of this disclosure. The degree of emphasis of the edge region 112 is an example of the "non-noise element of the image to be processed," the "factor governing the visual impression given by the image to be processed," and the "degree of emphasis of the edge region" related to the technology of this disclosure.

[0292] The AI ​​processing unit 62A6 performs AI-based processing on the image 75A6 to be processed. An example of AI-based processing on the image 75A6 to be processed is processing using the generation model 82A6. The generation model 82A6 is an example of the generation model 82A shown in Figure 3. The generation model 82A6 divides the edge regions 112 in the image 75A6 to be processed from the non-edge regions. This is a generative network that has already undergone learning to emphasize certain aspects.

[0293] The AI ​​processing unit 62A6 modifies the factors that govern the visual impression derived from the image 75A6 using the AI ​​method. Specifically, the AI ​​processing unit 62A6 modifies the factors that govern the visual impression derived from the image 75A6 as non-noise elements by performing processing on the image 75A6 using the generative model 82A6. The factors that govern the visual impression derived from the image 75A6 are the edge regions 112 within the image 75A6. In the example shown in Figure 22, the AI ​​processing unit 62A6 generates a first edge-enhanced image 86A6 by performing processing on the image 75A6 using the generative model 82A6. The first edge-enhanced image 86A6 is an image in which the edge regions 112 within the image 75A6 are enhanced more than the non-edge regions using the AI ​​method.

[0294] Here, the processing using the generative model 82A4 is an example of the "first AI processing," "first modification processing," and "enhancement processing" related to the technology of this disclosure. The first edge-enhanced image 86A6 is an example of the "first modified image" and "first edge-enhanced image" related to the technology of this disclosure. "Generating the first edge-enhanced image 86A6" is an example of "acquiring the first image" related to the technology of this disclosure.

[0295] The generative model 82A6 receives the image to be processed 75A6 as input. Based on the input image to be processed 75A6, the generative model 82A6 generates and outputs a first edge-enhanced image 86A6.

[0296] The non-AI processing unit 62B6 performs non-AI processing on the image 75A6 to be processed. Non-AI processing refers to processing that does not use a neural network. In this fifth modification, an example of processing that does not use a neural network is processing that does not use the generative model 82A6.

[0297] An example of a non-AI method of processing the image 75A6 to be processed is processing using a digital filter 84A6. The digital filter 84A6 is configured to emphasize the edge regions 112 in the image 75A6 to be processed more than the non-edge regions.

[0298] The non-AI processing unit 62B6 generates a second edge-enhanced image 88A6 by performing processing (i.e., filtering) on ​​the image 75A6 to be processed using a digital filter 84A6. In other words, the non-AI processing unit 62B6 generates a second edge-enhanced image 88A6 by enhancing non-noise elements (here, for example, edge regions 112) of the image 75A6 to be processed using a non-AI method compared to non-edge regions. To put it another way, the non-AI processing unit 62B6 generates a second edge-enhanced image 88A6 by enhancing the edge regions 112 in the image 75A6 to be processed using a non-AI method compared to non-edge regions.

[0299] Here, the processing using the digital filter 84A6 is an example of the "non-AI processing method that does not use a neural network" and the "second modification processing that modifies factors in a non-AI manner" related to the technology of this disclosure. "Generating a second edge-enhanced image 88A6" is an example of "acquiring a second image" related to the technology of this disclosure.

[0300] The digital filter 84A6 receives the image to be processed 75A6 as input. The digital filter 84A6 generates a second edge-enhanced image 88A6 based on the input image to be processed 75A6. The second edge-enhanced image 88A6 is an image obtained by modifying the non-noise elements of the digital filter 84A6 (i.e., an image obtained by modifying the non-noise elements of the image to be processed 75A6 through processing using the digital filter 84A6). The second edge-enhanced image 88A6 is an image in which the edge regions 112 in the image to be processed 75A6 have been adjusted by the digital filter 84A6 (i.e., an image in which the edge regions 112 are enhanced more than the non-edge regions by processing the image to be processed 75A6 using the digital filter 84A6). The intensity of the edge regions 112 in the second edge-enhanced image 88A6 is lower than the intensity of the edge regions 112 in the first edge-enhanced image 86A6. The difference in intensity of the edge regions 112 in the second edge-enhanced image 88A6 compared to the edge regions 112 in the first edge-enhanced image 86A6 is such that, for example, the difference between the edge regions 112 in the second edge-enhanced image 88A6 and the edge regions 112 in the first edge-enhanced image 86A6 is at least visually recognizable. The second edge-enhanced image 88A6 is an example of the "second image," "second modified image," and "second edge-enhanced image" related to the technology of this disclosure.

[0301] Incidentally, the intensity (e.g., brightness) of the first edge-enhanced image 86A6 obtained by applying AI-based processing to the target image 75A6 may differ from the user's preference due to the characteristics of the generative model 82A6 (e.g., the number of hidden layers and / or the amount of training). If the influence of AI-based processing is excessively reflected on the target image 75A6, the intensity may be too high or too low compared to the user's preference.

[0302] Therefore, in light of these circumstances, the imaging device 10 performs processing by the image adjustment unit 62C6 and the synthesis unit 62D6 on the first edge-enhanced image 86A6 and the second edge-enhanced image 88A6, as shown in Figure 23 as an example, thereby synthesizing the first edge-enhanced image 86A6 and the second edge-enhanced image 88A6.

[0303] As an example, as shown in Figure 23, the NVM64 stores a ratio of 90F. The ratio of 90F is the ratio used to combine the first edge-enhanced image 86A6 and the second edge-enhanced image 88A6, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A6 (i.e., processing using the generative model 82A6).

[0304] The ratio 90F is broadly divided into the first ratio 90F1 and the second ratio 90F2. The first ratio 90F1 is a value between 0 and 1, and the second ratio 90F2 is the value obtained by subtracting the value of the first ratio 90F1 from "1". In other words, the first ratio 90F1 and the second ratio 90F2 are set so that the sum of the first ratio 90F1 and the second ratio 90F2 is "1". The first ratio 90F1 and the second ratio 90F2 are variable values ​​that can be changed by user instructions.

[0305] The image adjustment unit 62C6 adjusts the first edge-enhanced image 86A6 generated by the AI ​​method processing unit 62A6 using a first ratio 90F1. For example, the image adjustment unit 62C6 adjusts the pixel value of each pixel in the first edge-enhanced image 86A6 by multiplying the pixel value of each pixel in the first edge-enhanced image 86A6 by the first ratio 90F1.

[0306] The image adjustment unit 62C6 adjusts the second edge-enhanced image 88A6 generated by the non-AI method processing unit 62B6 using a second ratio 90F2. For example, the image adjustment unit 62C6 adjusts the pixel value of each pixel in the second edge-enhanced image 88A6 by multiplying the pixel value of each pixel in the second edge-enhanced image 88A6 by the second ratio 90F2.

[0307] The synthesis unit 62D6 generates a composite image 92F by combining the first edge-enhanced image 86A6, which has been adjusted by the image adjustment unit 62C6 at a first ratio 90F1, and the second edge-enhanced image 88A6, which has been adjusted by the image adjustment unit 62C6 at a second ratio 90F2. In other words, the synthesis unit 62D6 adjusts the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A6 by combining the first edge-enhanced image 86A6, which has been adjusted at a first ratio 90F1, and the second edge-enhanced image 88A6, which has been adjusted at a second ratio 90F2. In other words, the synthesis unit 62D6, The non-noise elements (here, as an example, the edge region 112) are adjusted by combining the first edge-enhanced image 86A6 adjusted with a first ratio of 90F1 and the second edge-enhanced image 88A6 adjusted with a second ratio of 90F2. In other words, the compositing unit 62D6 adjusts elements derived from processing using the generative model 82A6 (for example, the pixel values ​​of pixels in which the edge region 112 is enhanced more than the non-edge region by the generative model 82A6) by combining the first edge-enhanced image 86A6 adjusted with a first ratio of 90F1 and the second edge-enhanced image 88A6 adjusted with a second ratio of 90F2.

[0308] The synthesis performed by the synthesis unit 62D6 is the addition of pixel values ​​at corresponding pixel positions between the first edge-enhanced image 86A6 and the second edge-enhanced image 88A6. The synthesis by the synthesis unit 62D6 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92F by the synthesis unit 62D6 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92F, after various image processing has been performed, is then output to a predetermined output destination by the synthesis unit 62D6.

[0309] Figure 24 shows an example of the image synthesis process flow according to this fifth modified example. The flowchart shown in Figure 24 differs from the flowchart shown in Figure 6 in that steps ST250 to ST268 are applied instead of steps ST12 to ST30.

[0310] In the image synthesis process shown in Figure 24, in step ST250, the AI-type processing unit 62A6 and the non-AI-type processing unit 62B6 acquire the image to be processed 75A6 from the image sensor 20. After the processing in step ST250 is completed, the image synthesis process proceeds to step ST252.

[0311] In step ST252, the AI ​​processing unit 62A6 inputs the image to be processed 75A6 acquired in step ST250 to the generation model 82A6. After the processing in step ST252 is executed, the image synthesis process proceeds to step ST254.

[0312] In step ST254, the AI ​​processing unit 62A6 obtains the first edge-enhanced image 86A6 output from the generation model 82A6, which was input to the generation model 82A6 in step ST252. After the processing in step ST254 is completed, the image synthesis process moves on to step ST256.

[0313] In step ST256, the non-AI processing unit 62B6 applies a digital filter 84A6 to the image 75A6 acquired in step ST250, thereby emphasizing the edge regions 112 within the image 75A6 compared to the non-edge regions. After the processing in step ST256 is completed, the image synthesis process proceeds to step ST258.

[0314] In step ST258, the non-AI processing unit 62B6 acquires a second edge-enhanced image 88A6 obtained in step ST256 by processing the target image 75A6 using the digital filter 84A6. After the processing in step ST258 is completed, the image synthesis process moves on to step ST260.

[0315] In step ST260, the image adjustment unit 62C6 obtains the first ratio 90F1 and the second ratio 90F2 from the NVM 64. After the processing in step ST260 is completed, the image synthesis process proceeds to step ST262.

[0316] In step ST262, the image adjustment unit 62C6 adjusts the first edge-enhanced image 86A6 using the first ratio 90F1 acquired in step ST260. After the processing in step ST262 is completed, the image synthesis process proceeds to step ST264.

[0317] In step ST264, the image adjustment unit 62C6 adjusts the second edge-enhanced image 88A6 using the second ratio 90F2 acquired in step ST260. After the processing in step ST264 is completed, the image synthesis process proceeds to step ST266.

[0318] In step ST266, the synthesis unit 62D6 adjusts the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A6 by combining the first edge-enhanced image 86A6 adjusted in step ST262 and the second edge-enhanced image 88A6 adjusted in step ST264. The synthesis of the first edge-enhanced image 86A6 adjusted in step ST262 and the second edge-enhanced image 88A6 adjusted in step ST264 generates the synthesized image 92F. After the processing in step ST266 is executed, the image synthesis process moves on to step ST268.

[0319] In step ST268, the image merging unit 62D6 performs various image processing on the composite image 92F. The image merging unit 62D6 then outputs the image obtained by performing various image processing on the composite image 92F as the processed image 75B to a predetermined output destination. After the processing in step ST268 is completed, the image merging process proceeds to step ST32.

[0320] As described above, in the imaging device 10 according to this fifth modified example, a first edge-enhanced image 86A6 is generated by enhancing the edge regions 112 in the image to be processed 75A6 more than the non-edge regions using the AI ​​method. A second edge-enhanced image 88A6 is generated by enhancing the edge regions 112 in the image to be processed 75A6 more than the non-edge regions using the non-AI method. The first edge-enhanced image 86A6 and the second edge-enhanced image 88A6 are then combined. This makes it possible to suppress the excessive or insufficient intensity of the edge regions 112 in the combined image 92F due to the AI ​​processing. As a result, the combined image 92F is an image in which the intensity of the edge regions 112 in the AI ​​processing is less noticeable compared to the first edge-enhanced image 86A6, providing a suitable image for users who do not want the intensity of the edge regions 112 in the AI ​​processing to be excessively noticeable.

[0321] Here, an example is given in which a first edge-enhanced image 86A6 obtained by processing the target image 75A6 using the generative model 82A6 and a second edge-enhanced image 88A6 obtained by processing the target image 75A6 using the digital filter 84A6 are combined. However, the technology of this disclosure is not limited to this. For example, the first edge-enhanced image 86A6 obtained by processing the target image 75A6 using the generative model 82A6 and the target image 75A6 (i.e., an image in which non-noise elements have not been adjusted) may be combined. In this case as well, similar effects can be expected.

[0322] [Sixth variation] As an example, as shown in Figure 25, the processor 62 in this sixth modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A7 instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B7 instead of a non-AI method processing unit 62B1. In this sixth modified example, explanations of matters that are the same as those explained before this sixth modified example will be omitted, and matters that are different from those explained before this sixth modified example will be explained.

[0323] As an example, as shown in Figure 25, the AI-type processing unit 62A7 and the non-AI-type processing unit 62B7 receive the image to be processed 75A7 as input. The image to be processed 75A7 is an example of the image to be processed 75A shown in Figure 2. In the example shown in Figure 25, the image to be processed 75A7 includes point images 114. The point images 114 are subject images obtained when subject light representing a point subject is imaged on the light-receiving surface 72A, and are due to the point spreading phenomenon originating from the optical characteristics of the imaging lens 40. As a result, the point image 114 appears blurred within the processed image 75A7 compared to the original subject image. The amount of blur of the point image 114 is expressed by a generally known point spread function.

[0324] The image to be processed, 75A7, is an image having point images 114 as non-noise elements. The point images 114 are examples of "non-noise elements in the image to be processed," "phenomena appearing in the image to be processed due to the characteristics of the imaging device," and "blur" related to the technology of this disclosure. The amount of blur of the point images 114 is an example of "amount of blur of point images" related to the technology of this disclosure. The point spreading phenomenon is an example of "characteristics of the imaging device" and "optical characteristics of the imaging device" related to the technology of this disclosure.

[0325] The AI ​​processing unit 62A7 performs AI-based processing on the image 75A7 to be processed. An example of AI-based processing on the image 75A7 is processing using the generative model 82A7. The generative model 82A7 is an example of the generative model 82A shown in Figure 3. The generative model 82A7 is a generative network that has already been trained to adjust the amount of blur of the point image 114. Below, as an example of adjusting the amount of blur of the point image 114, we will explain by reducing the amount of blur of the point image 114 (i.e., reducing the point spread).

[0326] The AI ​​processing unit 62A7 generates a first point image adjustment image 86A7 by performing processing on the image to be processed 75A7 using the generation model 82A7. In other words, the AI ​​processing unit 62A7 generates a first point image adjustment image 86A7 by adjusting non-noise elements (in this case, point images 114 as an example) in the image to be processed 75A7 using the AI ​​method. To put it another way, the AI ​​processing unit 62A7 generates a first point image adjustment image 86A7 by reducing the amount of blur of point images 114 in the image to be processed 75A7 using the AI ​​method. Here, the processing using the generation model 82A7 is an example of the "first AI processing," "first correction processing," and "point image adjustment processing" related to the technology of this disclosure. Also, here, "generating a first point image adjustment image 86A7" is an example of "acquiring a first image" related to the technology of this disclosure.

[0327] The generative model 82A7 receives the image to be processed 75A7 as input. Based on the input image to be processed 75A7, the generative model 82A7 generates and outputs a first point image adjustment image 86A7. The first point image adjustment image 86A7 is an image obtained by adjusting the non-noise elements by the generative model 82A7 (i.e., an image obtained by adjusting the non-noise elements by processing the image to be processed 75A7 using the generative model 82A7). In other words, the first point image adjustment image 86A7 is an image in which the non-noise elements in the image to be processed 75A7 have been corrected by the generative model 82A7 (i.e., an image in which the non-noise elements have been corrected by processing the image to be processed 75A7 using the generative model 82A7). To put it another way, the first point image adjustment image 86A7 is an image in which the point spread of the point image 114 has been corrected by the generation model 82A7 (that is, an image in which the point spread of the point image 114 has been corrected to be reduced by processing the image to be processed 75A7 using the generation model 82A7). Note that the first point image adjustment image 86A7 is an example of the "first image," "first corrected image," and "first point image adjustment image" related to the technology of this disclosure.

[0328] The non-AI processing unit 62B7 performs non-AI processing on the image 75A7 to be processed. Non-AI processing refers to processing that does not use a neural network. For example, processing that does not use a neural network is processing that does not use the generative model 82A7.

[0329] An example of non-AI processing for the image 75A7 to be processed is processing using a digital filter 84A7. The digital filter 84A7 is a digital filter configured to reduce the point spread of the point image 114. An example of a digital filter configured to reduce the point spread of the point image 114 is a resolution correction filter that cancels out the point spread indicated by the point spread function representing the point image 114. A filter is applied to a visible light image that is blurred compared to the original visible light image due to the point spreading phenomenon. An example of a resolution correction filter is an FIR filter. Since resolution correction filters are well-known filters, further detailed explanation will be omitted.

[0330] The non-AI processing unit 62B7 generates a second point image adjustment image 88A7 by performing processing (i.e., filtering) on ​​the image to be processed 75A7 using a digital filter 84A7. In other words, the non-AI processing unit 62B7 generates a second point image adjustment image 88A7 by adjusting the non-noise elements in the image to be processed 75A7 using a non-AI method. To put it another way, the non-AI processing unit 62B7 generates a second point image adjustment image 88A7 by correcting the image to be processed 75A7 so that the point spread within the image to be processed 75A7 is reduced using a non-AI method. Here, processing using the digital filter 84A7 is an example of the "non-AI processing without using a neural network," "second correction processing," and "processing to adjust the amount of blur using a non-AI method" related to the technology of this disclosure. Also, "generating a second point image adjustment image 88A7" here is an example of "acquiring a second image" related to the technology of this disclosure.

[0331] The digital filter 84A7 receives the image to be processed 75A7 as input. The digital filter 84A7 generates a second point image adjustment image 88A7 based on the input image to be processed 75A7. The second point image adjustment image 88A7 is an image obtained by adjusting the non-noise elements with the digital filter 84A7 (i.e., an image obtained by adjusting the non-noise elements with processing using the digital filter 84A7 on the image to be processed 75A7). In other words, the second point image adjustment image 88A7 is an image in which the non-noise elements in the image to be processed 75A7 have been corrected by the digital filter 84A7 (i.e., an image in which the non-noise elements have been corrected with processing using the digital filter 84A7 on the image to be processed 75A7). To put it another way, the second point image adjustment image 88A7 is an image obtained by correcting the processing target image 75A7 with the digital filter 84A1 (that is, an image corrected so that the point spread is reduced by processing the processing target image 75A7 with the digital filter 84A7). The second point image adjustment image 88A7 is an example of the "second image," "second corrected image," and "second point image adjustment image" related to the technology of this disclosure.

[0332] Incidentally, some users prefer to leave a moderate amount of point spread in the image rather than completely eliminating it. In the example shown in Figure 25, the point spread of point image 114 is reduced in the first point image adjustment image 86A7 than in the second point image adjustment image 88A7. In other words, the blur of point image 114 remains in the second point image adjustment image 88A7 more than in the first point image adjustment image 86A7. However, users may feel that the amount of blur of point image 114 in the first point image adjustment image 86A7 is insufficient, and the amount of blur of point image 114 in the second point image adjustment image 88A7 is too much. Therefore, if only one of the first point image adjustment image 86A7 or the second point image adjustment image 88A7 is ultimately output, the user will be provided with an image that does not suit their preferences. Improving the performance of the generative model 82A7 by increasing the amount of training data it receives or increasing the number of intermediate layers in the generative model 82A7 increases the likelihood of obtaining images that closely match the user's preferences. However, this increases the cost required to create the generative model 82A7, which may ultimately lead to a higher price for the imaging device 10.

[0333] Therefore, in light of these circumstances, as shown in Figure 26 as an example, the imaging device 10 performs processing by the image adjustment unit 62C7 and the synthesis unit 62D7 on the first point image adjustment image 86A7 and the second point image adjustment image 88A7, thereby synthesizing the first point image adjustment image 86A7 and the second point image adjustment image 88A7.

[0334] As an example, as shown in Figure 26, the NVM64 stores a percentage of 90G. 90G is the ratio for combining the first point image adjustment image 86A7 and the second point image adjustment image 88A7, and is set to adjust for any excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A7 (i.e., processing using the generative model 82A7).

[0335] The ratio 90G is broadly divided into the first ratio 90G1 and the second ratio 90G2. The first ratio 90G1 is a value between 0 and 1, and the second ratio 90G2 is the value obtained by subtracting the value of the first ratio 90G1 from "1". In other words, the first ratio 90G1 and the second ratio 90G2 are set so that the sum of the first ratio 90G1 and the second ratio 90G2 is "1". The first ratio 90G1 and the second ratio 90G2 are variable values ​​that can be changed by user instructions.

[0336] The image adjustment unit 62C7 adjusts the first point image adjustment image 86A7 generated by the AI ​​method processing unit 62A7 using the first ratio 90G1. For example, the image adjustment unit 62C7 adjusts the pixel value of each pixel in the first point image adjustment image 86A7 by multiplying the pixel value of each pixel in the first point image adjustment image 86A7 by the first ratio 90G1.

[0337] The image adjustment unit 62C7 adjusts the second point image adjustment image 88A7 generated by the non-AI method processing unit 62B7 using the second ratio 90G2. For example, the image adjustment unit 62C7 adjusts the pixel value of each pixel in the second point image adjustment image 88A7 by multiplying the pixel value of each pixel in the second point image adjustment image 88A7 by the second ratio 90G2.

[0338] The combining unit 62D7 generates a composite image 92G by combining the first point image adjustment image 86A7, which has been adjusted by the image adjustment unit 62C7 with a first ratio 90G1, and the second point image adjustment image 88A7, which has been adjusted by the image adjustment unit 62C7 with a second ratio 90G2. In other words, the combining unit 62D7 adjusts for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A7 by combining the first point image adjustment image 86A7, which has been adjusted with a first ratio 90G1, and the second point image adjustment image 88A7, which has been adjusted with a second ratio 90G2. To put it another way, the synthesis unit 62D7 synthesizes the first point image adjustment image 86A7 adjusted with a first ratio 90G1 and the second point image adjustment image 88A7 adjusted with a second ratio 90G2, thereby adjusting elements derived from processing using the generation model 82A7 (for example, the pixel values ​​of pixels whose point spread has been reduced by the generation model 82A7).

[0339] The synthesis performed by the synthesis unit 62D7 is the addition of pixel values ​​at corresponding pixel positions between the first point image adjustment image 86A7 and the second point image adjustment image 88A7. The synthesis by the synthesis unit 62D7 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92G by the synthesis unit 62D7 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92G, after various image processing has been performed, is then output to a predetermined output destination by the synthesis unit 62D7.

[0340] Figure 27 shows an example of the image synthesis process flow according to this sixth modified example. The flowchart shown in Figure 27 differs from the flowchart shown in Figure 6 in that steps ST300 to ST318 are applied instead of steps ST12 to ST30.

[0341] In the image synthesis process shown in Figure 27, in step ST300, the AI ​​method processing unit 62A7 and the non-AI method processing unit 62B7 acquire the image to be processed 75A7 from the image sensor 20. After the processing in step ST300 is completed, the image synthesis process proceeds to step ST302.

[0342] In step ST302, the AI ​​method processing unit 62A7 processes the data acquired in step ST300. The target image 75A7 is input to the generation model 82A7. After the processing in step ST302 is executed, the image synthesis process proceeds to step ST304.

[0343] In step ST304, the AI ​​processing unit 62A7 acquires the first point image adjustment image 86A7 output from the generation model 82A7 after the image to be processed 75A7 is input to the generation model 82A7 in step ST302. After the processing in step ST304 is executed, the image synthesis process moves on to step ST306.

[0344] In step ST306, the non-AI processing unit 62B7 corrects the point spreading phenomenon in the image 75A7 acquired in step ST300 by applying a digital filter 84A7 to the image 75A7. After the processing in step ST306 is completed, the image synthesis process moves on to step ST308.

[0345] In step ST308, the non-AI processing unit 62B7 acquires the second point image adjustment image 88A7 obtained in step ST306 by processing the target image 75A7 using the digital filter 84A7. After the processing in step ST308 is executed, the image synthesis process moves on to step ST310.

[0346] In step ST310, the image adjustment unit 62C7 obtains the first ratio 90G1 and the second ratio 90G2 from the NVM 64. After the processing in step ST310 is completed, the image synthesis process proceeds to step ST312.

[0347] In step ST312, the image adjustment unit 62C7 adjusts the first point image adjustment image 86A7 using the first ratio 90G1 acquired in step ST310. After the processing in step ST312 is completed, the image synthesis process proceeds to step ST314.

[0348] In step ST314, the image adjustment unit 62C7 adjusts the second point image adjustment image 88A7 using the second ratio 90G2 acquired in step ST310. After the processing in step ST314 is completed, the image synthesis process proceeds to step ST316.

[0349] In step ST316, the synthesis unit 62D7 adjusts the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A7 by combining the first point image adjustment image 86A7 adjusted in step ST312 and the second point image adjustment image 88A7 adjusted in step ST314. The synthesis of the first point image adjustment image 86A7 adjusted in step ST312 and the second point image adjustment image 88A7 adjusted in step ST314 generates a composite image 92G. After the processing in step ST316 is executed, the image synthesis process moves on to step ST318.

[0350] In step ST318, the image merging unit 62D7 performs various image processing on the composite image 92G. The image merging unit 62D7 then outputs the image obtained by performing various image processing on the composite image 92G as the processed image 75B to a predetermined output destination. After the processing in step ST318 is completed, the image merging process proceeds to step ST32.

[0351] As explained above, in the imaging device 10 according to this sixth modified example, the point spread of the point image 114 in the image to be processed 75A7 is reduced by the AI ​​method to generate the first point image adjustment image 86A7. In addition, the point spread of the point image 114 in the image to be processed 75A7 is reduced by the non-AI method to generate the second point image adjustment image 88A7. Then, the first point image adjustment image 86A7 and the second point image adjustment image 88A7 are combined. This makes it possible to suppress the excess or deficiency of the correction amount for the point spread phenomenon in the AI ​​method processing (i.e., the correction amount for the blur amount of the point image 114) in the combined image 92G. As a result, the combined image 92G is the first point Compared to image adjustment image 86A7, this image shows less noticeable correction of point spreading in the AI-based processing, making it suitable for users who do not want the correction of point spreading in the AI-based processing to be overly noticeable.

[0352] Here, an example is given in which a first point image adjustment image 86A7 obtained by processing the target image 75A7 using the generation model 82A7 and a second point image adjustment image 88A7 obtained by processing the target image 75A7 using the digital filter 84A7 are combined. However, the technology of this disclosure is not limited to this. For example, the first point image adjustment image 86A7 obtained by processing the target image 75A7 using the generation model 82A7 and the target image 75A7 (i.e., an image in which non-noise elements have not been adjusted) may be combined. In this case as well, similar effects can be expected.

[0353] [7th variation] As an example, as shown in Figure 28, the processor 62 in this seventh modification differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A8 instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B8 instead of a non-AI method processing unit 62B1. In this seventh modification, explanations of matters that are the same as those explained before this seventh modification will be omitted, and matters that are different from those explained before this seventh modification will be explained.

[0354] The AI ​​processing unit 62A8 and the non-AI processing unit 62B8 receive the image to be processed 75A8 as input. The image to be processed 75A8 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A8 is a chromatic image and has a person region 116. The person region 116 is the image region in which a person is depicted. Although a chromatic image is used as an example for the image to be processed 75A8 here, the image to be processed 75A8 may also be an achromatic image.

[0355] The AI-based processing unit 62A8 and the non-AI-based processing unit 62B8 perform processing to apply a blur to the input image 75A8 that corresponds to the subject contained within it. In this seventh modified example, the subject contained within the image 75A8 refers to a person. A person contained within the image 75A8 is an example of the "third subject" related to the technology of this disclosure. The blur corresponding to the subject is an example of the "non-noise element of the image to be processed," the "factor governing the visual impression given by the image to be processed," and the "blur corresponding to the third subject" related to the technology of this disclosure.

[0356] The AI ​​processing unit 62A8 performs AI-based processing on the image 75A8. One example of AI-based processing on the image 75A8 is processing using the generative model 82A8. The generative model 82A8 is an example of the generative model 82A shown in Figure 3. The generative model 82A8 is a generative network that has already been trained to add blur to the human region 116.

[0357] The AI ​​processing unit 62A8 modifies the factors that govern the visual impression derived from the image 75A8 using an AI method. Specifically, the AI ​​processing unit 62A8 modifies the factors that govern the visual impression derived from the image 75A8 as non-noise elements of the image 75A8 by performing processing on the image 75A8 using the generative model 82A8. The factors that govern the visual impression derived from the image 75A8 are the blur corresponding to the person region 116 within the image 75A8. In the example shown in Figure 28, the AI ​​processing unit 62A8 generates the first blurred image 86A8 by performing processing on the image 75A8 using the generative model 82A8. The first blurred image 86A8 is an image in which the person region 116 within the image 75A8 has been blurred using an AI method.

[0358] Here, the processing using the generative model 82A8 is referred to as the "first AI processing" related to the technology of this disclosure. This is an example of the "first modification process" and the "blurring process". The first blurred image 86A8 is an example of the "first modified image" and the "first blurred image" related to the technology of this disclosure. "Generating the first blurred image 86A8" is an example of "acquiring the first image" related to the technology of this disclosure.

[0359] The generation model 82A8 receives the image to be processed 75A8 as input. Based on the input image to be processed 75A8, the generation model 82A8 generates and outputs the first blurred image 86A8.

[0360] The non-AI processing unit 62B8 performs non-AI processing on the image 75A8 to be processed. Non-AI processing refers to processing that does not use a neural network. In this seventh modification, an example of processing that does not use a neural network is processing that does not use the generative model 82A8.

[0361] An example of non-AI processing of the image 75A8 to be processed is processing using a digital filter 84A8. The digital filter 84A8 is a digital filter configured to blur the person region 116 within the image 75A8 to be processed.

[0362] The non-AI processing unit 62B8 generates a second blurred image 88A8 by performing processing (i.e., filtering) on ​​the image to be processed 75A8 using a digital filter 84A8. In other words, the non-AI processing unit 62B8 generates a second blurred image 88A8 by changing the non-noise elements of the image to be processed 75A8 using a non-AI method. To put it another way, the non-AI processing unit 62B8 generates a second blurred image 88A8 by applying blur to the person region 116 within the image to be processed 75A8 using a non-AI method.

[0363] Here, the processing using the digital filter 84A8 is an example of the "non-AI method processing that does not use a neural network" and the "second modification processing that changes the factors in a non-AI method" related to the technology of this disclosure. "Generating the second blurred image 88A8" is an example of "acquiring the second image" related to the technology of this disclosure.

[0364] The digital filter 84A8 receives the image to be processed 75A8 as input. The digital filter 84A8 generates a second blurred image 88A8 based on the input image to be processed 75A8. The second blurred image 88A8 is an image obtained by changing the non-noise elements with the digital filter 84A8 (i.e., an image obtained by changing the non-noise elements by processing the image to be processed 75A8 using the digital filter 84A8). In other words, the second blurred image 88A8 is an image in which the person region 116 in the image to be processed 75A8 has been adjusted by the digital filter 84A8 (i.e., an image in which the person region 116 has been blurred by processing the image to be processed 75A8 using the digital filter 84A8). The degree of blur applied to the person region 116 in the second blurred image 88A8 is less than the degree of blur applied to the person region 116 in the first blurred image 86A8. The second blurred image 88A8 is an example of the "second image," "second modified image," and "second blurred image" related to the technology of this disclosure.

[0365] Incidentally, the amount of blur in the first blurred image 86A8 obtained by applying the AI ​​method to the target image 75A8 may differ from the user's preference due to the characteristics of the generative model 82A8 (e.g., the number of intermediate layers and / or the amount of training). If the influence of the AI ​​method is excessively reflected in the target image 75A8, the amount of blur may be too much or too little compared to the user's preference.

[0366] Therefore, in light of these circumstances, the imaging device 10, as an example as shown in Figure 29, processes the first blurred image 86A8 and the second blurred image 88A8 with the processing of the image adjustment unit 62C8 and The first blurred image 86A8 and the second blurred image 88A8 are combined by the processing performed by the compositing unit 62D8.

[0367] As an example, as shown in Figure 29, the NVM64 stores a ratio of 90H. Ratio 90H is the ratio used to combine the first blurred image 86A8 and the second blurred image 88A8, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A8 (i.e., processing using the generative model 82A8).

[0368] The ratio 90H is broadly divided into the first ratio 90H1 and the second ratio 90H2. The first ratio 90H1 is a value between 0 and 1, and the second ratio 90H2 is the value obtained by subtracting the value of the first ratio 90H1 from "1". In other words, the first ratio 90H1 and the second ratio 90H2 are set so that the sum of the first ratio 90H1 and the second ratio 90H2 is "1". The first ratio 90H1 and the second ratio 90H2 are variable values ​​that can be changed by user instructions.

[0369] The image adjustment unit 62C8 adjusts the first blurred image 86A8 generated by the AI ​​method processing unit 62A8 using a first ratio 90H1. For example, the image adjustment unit 62C8 adjusts the pixel value of each pixel in the first blurred image 86A8 by multiplying the pixel value of each pixel in the first blurred image 86A8 by the first ratio 90H1.

[0370] The image adjustment unit 62C8 adjusts the second blurred image 88A8 generated by the non-AI method processing unit 62B8 using the second ratio 90H2. For example, the image adjustment unit 62C8 adjusts the pixel value of each pixel in the second blurred image 88A8 by multiplying the pixel value of each pixel in the second blurred image 88A8 by the second ratio 90H2.

[0371] The compositing unit 62D8 generates a composite image 92H by compositing the first blurred image 86A8, which has been adjusted by the image adjustment unit 62C8 with a first ratio 90H1, and the second blurred image 88A8, which has been adjusted by the image adjustment unit 62C8 with a second ratio 90H2. In other words, the compositing unit 62D8 adjusts for any excess or deficiency in the AI ​​processing performed by the AI ​​processing unit 62A8 by compositing the first blurred image 86A8, which has been adjusted with a first ratio 90H1, and the second blurred image 88A8, which has been adjusted with a second ratio 90H2. To put it another way, the synthesis unit 62D8 adjusts elements derived from processing using the generative model 82A8 (for example, the pixel values ​​of pixels to which blurring has been applied by the generative model 82A8) by synthesizing the first blurred image 86A8 adjusted with a first ratio 90H1 and the second blurred image 88A8 adjusted with a second ratio 90H2.

[0372] The image merging performed by the merging unit 62D8 involves adding the pixel values ​​of corresponding pixel positions between the first blurred image 86A8 and the second blurred image 88A8. The merging by the merging unit 62D8 is performed in the same manner as the merging by the merging unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the merging image 92H by the merging unit 62D8 in the same manner as the merging image 92A shown in Figure 5. The merging image 92H, after various image processing has been performed, is then output to a predetermined output destination by the merging unit 62D8.

[0373] Figure 30 shows an example of the image synthesis process flow according to this seventh modified example. The flowchart shown in Figure 30 differs from the flowchart shown in Figure 6 in that steps ST350 to ST368 are applied instead of steps ST12 to ST30.

[0374] In the image synthesis process shown in Figure 30, in step ST350, the AI ​​method processing unit 62A8 and the non-AI method processing unit 62B8 acquire the image to be processed 75A8 from the image sensor 20. After the processing in step ST350 is completed, the image synthesis process proceeds to step ST352. do.

[0375] In step ST352, the AI ​​processing unit 62A8 inputs the image to be processed 75A8 acquired in step ST350 to the generation model 82A8. After the processing in step ST352 is executed, the image synthesis process proceeds to step ST354.

[0376] In step ST354, the AI ​​processing unit 62A8 acquires the first blurred image 86A8 output from the generation model 82A8 after the image to be processed 75A8 is input to the generation model 82A8 in step ST352. After the processing in step ST354 is executed, the image synthesis process moves on to step ST356.

[0377] In step ST356, the non-AI processing unit 62B8 applies a digital filter 84A8 to the image 75A8 acquired in step ST350, thereby blurring the person region 116 within the image 75A8. After the processing in step ST356 is completed, the image synthesis process moves on to step ST358.

[0378] In steps ST352 and ST356, examples are given where blur is applied to the person region 116. However, this is merely an example, and blur may be applied to image regions other than the person region 116, according to the method defined for the person region 116. Alternatively, blur may not be applied to the person region 116, but blur may be applied to image regions other than the person region 116, according to the method defined for the person region 116. Furthermore, while the person region 116 is used as an example here, this is merely an example, and the image region may contain subjects other than people (for example, a specific vehicle, a specific plant, a specific animal, a specific building, or a specific aircraft, etc.). In this case as well, blur may be applied to the image according to the subject in a similar manner.

[0379] In step ST358, the non-AI processing unit 62B8 acquires a second blurred image 88A8 obtained in step ST356 by processing the target image 75A8 using the digital filter 84A8. After the processing in step ST358 is completed, the image synthesis process moves on to step ST360.

[0380] In step ST360, the image adjustment unit 62C8 obtains the first ratio 90H1 and the second ratio 90H2 from the NVM 64. After the processing in step ST360 is completed, the image synthesis process proceeds to step ST362.

[0381] In step ST362, the image adjustment unit 62C8 adjusts the first blurred image 86A8 using the first ratio 90H1 acquired in step ST360. After the processing in step ST362 is completed, the image synthesis process proceeds to step ST364.

[0382] In step ST364, the image adjustment unit 62C8 adjusts the second blurred image 88A8 using the second ratio 90H2 acquired in step ST360. After the processing in step ST364 is completed, the image synthesis process proceeds to step ST366.

[0383] In step ST366, the compositing unit 62D8 adjusts for any excess or deficiency in the AI ​​processing performed by the AI ​​processing unit 62A8 by compositing the first blurred image 86A8 adjusted in step ST362 and the second blurred image 88A8 adjusted in step ST364. The composite image 92H is generated by compositing the first blurred image 86A8 adjusted in step ST362 and the second blurred image 88A8 adjusted in step ST364. After the processing in step ST366 is executed, the image compositing process moves on to step ST368.

[0384] In step ST368, the image merging unit 62D8 performs various image processing on the composite image 92H. The image merging unit 62D8 then outputs the image obtained by performing various image processing on the composite image 92H as the processed image 75B to a predetermined output destination. After the processing in step ST368 is completed, the image merging process proceeds to step ST32.

[0385] As explained above, in the imaging device 10 according to this seventh modified example, a first blurred image 86A8 is generated by applying a blur corresponding to the person region 116 in the image to be processed 75A8 using an AI method. A second blurred image 88A8 is generated by applying a blur corresponding to the person region 116 in the image to be processed 75A8 using a non-AI method. The first blurred image 86A8 and the second blurred image 88A8 are then combined. This makes it possible to suppress any excess or deficiency of the blur corresponding to the person region 116 in the combined image 92H due to the AI ​​processing. As a result, the combined image 92H is an image in which the blur corresponding to the person region 116 in the AI ​​processing is less noticeable compared to the first blurred image 86A8, and a suitable image can be provided for users who do not like the blur in the first blurred image 86A8 due to the AI ​​processing to be excessively noticeable.

[0386] Here, an example is given in which a first blurred image 86A8 obtained by processing the target image 75A8 using the generative model 82A8 and a second blurred image 88A8 obtained by processing the target image 75A8 using the digital filter 84A8 are combined. However, the technology of this disclosure is not limited to this. For example, the first blurred image 86A8 obtained by processing the target image 75A8 using the generative model 82A8 and the target image 75A8 (i.e., an image in which non-noise elements have not been adjusted) may be combined. In this case as well, a similar effect can be expected.

[0387] [8th variation] As an example, as shown in Figure 31, the processor 62 according to this eighth modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A9 ​​instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B9 instead of a non-AI method processing unit 62B1. In this eighth modified example, explanations of matters that are the same as those explained before this eighth modified example will be omitted, and matters that are different from those explained before this eighth modified example will be explained.

[0388] The AI ​​processing unit 62A9 ​​and the non-AI processing unit 62B9 receive the image to be processed 75A9 as input. The image to be processed 75A9 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A9 is a chromatic image. Although a chromatic image is used as an example here for the image to be processed 75A9, the image to be processed 75A9 may also be an achromatic image.

[0389] The AI-based processing unit 62A9 ​​and the non-AI-based processing unit 62B9 perform a process to apply bokeh to the input image 75A9. The bokeh applied to the image 75A9 is an example of the "non-noise elements of the image to be processed," the "factors governing the visual impression given by the image to be processed," the "first bokeh," and the "second bokeh" related to the technology of this disclosure.

[0390] The AI ​​processing unit 62A9 ​​performs AI-based processing on the image 75A9. One example of AI-based processing on the image 75A9 is processing using the generative model 82A9. The generative model 82A9 is an example of the generative model 82A shown in Figure 3. The generative model 82A9 is a generative network that has already been trained to add bokeh to the image 75A9.

[0391] The AI ​​processing unit 62A9 ​​uses the AI ​​method to process the visual information provided by the image 75A9. The factors governing the impression are changed. Specifically, the AI ​​processing unit 62A9 ​​performs processing on the image to be processed 75A9 using the generative model 82A9, thereby changing the factors governing the visual impression given by the image to be processed 75A9 as non-noise elements of the image to be processed 75A9. The factors governing the visual impression given by the image to be processed 75A9 are the bokeh applied to the image to be processed 75A9. In the example shown in Figure 31, the AI ​​processing unit 62A9 ​​generates the first bokeh image 86A9 by performing processing on the image to be processed 75A9 using the generative model 82A9. The first bokeh image 86A9 is an image to which the first bokeh 118 has been applied to the image to be processed 75A9 using the AI ​​method.

[0392] Here, the processing using the generative model 82A8 is an example of the "first AI processing," "first modification processing," and "bokeh processing" related to the technology of this disclosure. The first bokeh 118 is an example of the "first bokeh" related to the technology of this disclosure. The first bokeh image 86A9 is an example of the "first modified image" and "first bokeh image" related to the technology of this disclosure. "Generating the first bokeh image 86A9" is an example of "acquiring the first image" related to the technology of this disclosure.

[0393] The generative model 82A9 receives the image to be processed 75A9 as input. Based on the input image to be processed 75A9, the generative model 82A9 generates and outputs the first bokeh image 86A9.

[0394] The non-AI processing unit 62B9 performs non-AI processing on the image 75A9 to be processed. Non-AI processing refers to processing that does not use a neural network. In this eighth modification, an example of processing that does not use a neural network is processing that does not use the generative model 82A9.

[0395] An example of non-AI processing of the image 75A9 to be processed is processing using a digital filter 84A9. The digital filter 84A9 is a digital filter configured to add bokeh to the image 75A9 to be processed.

[0396] The non-AI processing unit 62B9 generates a second bokeh image 88A9 by performing processing (i.e., filtering) on ​​the image to be processed 75A9 using a digital filter 84A9. In other words, the non-AI processing unit 62B9 generates a second bokeh image 88A9 by changing the non-noise elements of the image to be processed 75A9 using a non-AI method. To put it another way, the non-AI processing unit 62B9 generates a second bokeh image 88A9 by adding a second bokeh 120 to the image to be processed 75A9 using a non-AI method.

[0397] Here, the processing using the digital filter 84A9 is an example of the "non-AI method processing that does not use a neural network" and the "second modification processing that changes the factors in a non-AI method" related to the technology of this disclosure. "Generating the second blurred image 88A9" is an example of "acquiring the second image" related to the technology of this disclosure.

[0398] The digital filter 84A9 receives the image to be processed 75A9 as input. The digital filter 84A9 generates a second bokeh image 88A9 based on the input image to be processed 75A9. The second bokeh image 88A9 is an image obtained by changing the non-noise elements of the digital filter 84A9 (i.e., an image obtained by changing the non-noise elements of the image to be processed 75A9 by processing with the digital filter 84A9). In other words, the second bokeh image 88A9 is an image to be processed 75A9 to which a second bokeh 120 has been applied (i.e., an image to be processed 75A9 to which a second bokeh 120 has been applied by processing with the digital filter 84A9). The characteristics of the second bokeh 120 (e.g., color, sharpness, and / or size) are different from the characteristics of the first bokeh 118. Note that the second bokeh image 88A9 is the "second image" related to the technology of this disclosure. These are examples of the "Second Modified Image" and the "Second Bokeh Image".

[0399] Incidentally, the characteristics of the first bokeh image 86A9 obtained by applying AI-based processing to the target image 75A9 may differ from the user's preference due to the characteristics of the generative model 82A9 (e.g., the number of intermediate layers and / or the amount of training). If the influence of AI-based processing is excessively reflected in the target image 75A9, it is possible that the bokeh preferred by the user may not be expressed.

[0400] Therefore, in light of these circumstances, the imaging device 10 performs processing by the image adjustment unit 62C9 and the synthesis unit 62D9 on the first blurred image 86A9 and the second blurred image 88A9, as shown in Figure 32 as an example, thereby synthesizing the first blurred image 86A9 and the second blurred image 88A9.

[0401] As an example, as shown in Figure 32, the NVM64 stores a ratio of 90I. Ratio 90I is the ratio used to combine the first bokeh image 86A9 and the second bokeh image 88A9, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A9 ​​(i.e., processing using the generative model 82A9).

[0402] The ratio 90I is broadly divided into the first ratio 90I1 and the second ratio 90I2. The first ratio 90I1 is a value between 0 and 1, and the second ratio 90I2 is the value obtained by subtracting the value of the first ratio 90I1 from "1". In other words, the first ratio 90I1 and the second ratio 90I2 are set so that the sum of the first ratio 90I1 and the second ratio 90I2 is "1". The first ratio 90I1 and the second ratio 90I2 are variable values ​​that can be changed by user instructions.

[0403] The image adjustment unit 62C9 adjusts the first bokeh image 86A9 generated by the AI ​​method processing unit 62A9 ​​using a first ratio 90I1. For example, the image adjustment unit 62C9 adjusts the pixel value of each pixel in the first bokeh image 86A9 by multiplying the pixel value of each pixel in the first bokeh image 86A9 by the first ratio 90I1.

[0404] The image adjustment unit 62C9 adjusts the second bokeh image 88A9 generated by the non-AI method processing unit 62B9 using the second ratio 90I2. For example, the image adjustment unit 62C9 adjusts the pixel value of each pixel in the second bokeh image 88A9 by multiplying the pixel value of each pixel in the second bokeh image 88A9 by the second ratio 90I2.

[0405] The synthesis unit 62D9 generates a composite image 92I by combining the first bokeh image 86A9, which has been adjusted by the image adjustment unit 62C9 with a first ratio 90I1, and the second bokeh image 88A9, which has been adjusted by the image adjustment unit 62C9 with a second ratio 90I2. In other words, the synthesis unit 62D9 adjusts for any excess or deficiency in the AI ​​processing performed by the AI ​​processing unit 62A9 ​​by combining the first bokeh image 86A9, which has been adjusted with a first ratio 90I1, and the second bokeh image 88A9, which has been adjusted with a second ratio 90I2. To put it another way, the synthesis unit 62D9 adjusts elements derived from processing using the generation model 82A9 (for example, the pixel values ​​of pixels to which the first bokeh 118 has been applied by the generation model 82A9) by synthesizing the first bokeh image 86A9 adjusted by the first ratio 90I1 and the second bokeh image 88A9 adjusted by the second ratio 90I2.

[0406] The synthesis performed by the synthesis unit 62D9 is the addition of pixel values ​​at corresponding pixel positions between the first bokeh image 86A9 and the second bokeh image 88A9. The synthesis by the synthesis unit 62D9 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Also, the synthesized image 92I Similarly, the compositing unit 62D9 performs various image processing on the composite image 92I, in the same manner as the composite image 92A shown in Figure 5. The composite image 92I, after various image processing steps have been performed, is then output to a predetermined output destination by the compositing unit 62D9.

[0407] Figure 33 shows an example of the image synthesis process flow according to this eighth modified example. The flowchart shown in Figure 33 differs from the flowchart shown in Figure 6 in that steps ST400 to ST418 are applied instead of steps ST12 to ST30.

[0408] In the image synthesis process shown in Figure 33, in step ST400, the AI ​​method processing unit 62A9 ​​and the non-AI method processing unit 62B9 acquire the image to be processed 75A9 from the image sensor 20. After the processing in step ST400 is completed, the image synthesis process proceeds to step ST402.

[0409] In step ST402, the AI ​​processing unit 62A9 ​​inputs the image to be processed 75A9 acquired in step ST400 into the generation model 82A9. After the processing in step ST402 is completed, the image synthesis process proceeds to step ST404.

[0410] In step ST404, the AI ​​processing unit 62A9 ​​acquires the first bokeh image 86A9 output from the generation model 82A9 after the image to be processed 75A9 is input to the generation model 82A9 in step ST402. After the processing in step ST404 is executed, the image synthesis process moves on to step ST406.

[0411] In step ST406, the non-AI processing unit 62B9 applies a second bokeh effect 120 to the image 75A9 acquired in step ST400 by using a digital filter 84A9. After the processing in step ST406 is completed, the image synthesis process moves on to step ST408.

[0412] In step ST402 and step ST406, examples are given of how bokeh is generated regardless of the subject in the image 75A9 to be processed. However, this is merely an example, and a predetermined bokeh corresponding to the subject in the image 75A9 to be processed (for example, a specific person, a specific vehicle, a specific plant, a specific animal, a specific building, and / or a specific aircraft, etc.) may be generated and applied to the image 75A9 to be processed.

[0413] In step ST408, the non-AI processing unit 62B9 acquires a second bokeh image 88A9 obtained in step ST406 by processing the target image 75A9 using the digital filter 84A9. After the processing in step ST408 is completed, the image synthesis process moves on to step ST410.

[0414] In step ST410, the image adjustment unit 62C9 obtains the first ratio 90I1 and the second ratio 90I2 from the NVM 64. After the processing in step ST410 is completed, the image synthesis process proceeds to step ST412.

[0415] In step ST412, the image adjustment unit 62C9 adjusts the first bokeh image 86A9 using the first ratio 90I1 acquired in step ST410. After the processing in step ST412 is completed, the image synthesis process moves on to step ST414.

[0416] In step ST414, the image adjustment unit 62C9 adjusts the second bokeh image 88A9 using the second ratio 90I2 acquired in step ST410. After the processing in step ST414 is completed, the image synthesis process proceeds to step ST416.

[0417] In step ST416, the synthesis unit 62D9 adjusts for any excess or deficiency in the AI ​​processing performed by the AI ​​processing unit 62A9 ​​by combining the first bokeh image 86A9 adjusted in step ST412 and the second bokeh image 88A9 adjusted in step ST414. The synthesis of the first bokeh image 86A9 adjusted in step ST412 and the second bokeh image 88A9 adjusted in step ST414 generates a composite image 92I. After the processing in step ST416 is executed, the image synthesis process moves on to step ST418.

[0418] In step ST418, the image merging unit 62D9 performs various image processing on the composite image 92I. The image merging unit 62D9 then outputs the image obtained by performing various image processing on the composite image 92I as the processed image 75B to a predetermined output destination. After the processing in step ST418 is completed, the image merging process proceeds to step ST32.

[0419] As explained above, in the imaging device 10 according to this eighth modified example, a first bokeh image 86A9 is generated by applying a first bokeh 118 to the image to be processed 75A9 using the AI ​​method. A second bokeh image 88A9 is generated by applying a second bokeh 120 to the image to be processed 75A9 using the non-AI method. The first bokeh image 86A9 and the second bokeh image 88A9 are then combined. This makes it possible to suppress any excess or deficiency of the first bokeh 118 elements in the combined image 92I due to the AI ​​processing. As a result, the combined image 92I is less conspicuous with the first bokeh 118 from the AI ​​processing compared to the first bokeh image 86A9, providing a suitable image for users who do not want the characteristics of the first bokeh image 86A9 from the AI ​​processing to be excessively prominent.

[0420] Here, an example is given in which a first bokeh image 86A9 obtained by processing the target image 75A9 using the generative model 82A9 and a second bokeh image 88A9 obtained by processing the target image 75A9 using the digital filter 84A9 are combined. However, the technology of this disclosure is not limited to this. For example, the first bokeh image 86A9 obtained by processing the target image 75A9 using the generative model 82A9 and the target image 75A9 (i.e., an image in which non-noise elements have not been adjusted) may be combined. In this case as well, a similar effect can be expected.

[0421] In the examples shown in Figures 31 to 33, an example was described in which the non-AI processing unit 62B9 applies a non-AI processing method to the image 75A9 to be processed, thereby adding a second bokeh effect 120 to the image 75A9 to be processed. However, the technology of this disclosure is not limited to this. For example, as shown in Figure 34, the non-AI processing unit 62B9 may apply a non-AI processing method to the first bokeh image 86A9 generated by the AI ​​processing unit 62A9 ​​to generate a second bokeh image 86B9 that includes the second bokeh effect 120.

[0422] Furthermore, as shown in Figure 35, for example, the non-AI processing unit 62B9 may perform non-AI processing on the first bokeh image 86A9 generated by the AI ​​processing unit 62A9 ​​to generate a second bokeh image 88A9 that includes a second bokeh 120 with higher clarity than the first bokeh 120.

[0423] Furthermore, in the examples shown in Figures 31 to 35, we have described an example in which a first bokeh 118 is applied to the image 75A9 to be processed using the AI ​​method. However, if the image 75A9 to be processed contains bokeh, the AI ​​processing unit 62A9 ​​may remove the bokeh from the image 75A9 to be processed using the AI ​​method.

[0424] [9th variation] As an example, as shown in Figure 36, the processor 62 according to this ninth modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A10 instead of an AI method processing unit 62A1, and a non-AI method processing unit 62B10 instead of a non-AI method processing unit 62B1. In this ninth modified example, explanations of matters that are the same as those explained before this ninth modified example will be omitted, and matters that are different from those explained before this ninth modified example will be explained.

[0425] The AI ​​processing unit 62A10 and the non-AI processing unit 62B10 receive the image to be processed 75A10 as input. The image to be processed 75A10 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A10 is a chromatic image and has a person region 124 and a background region 126. The person region 124 is the image region in which a person is depicted. The background region 126 is the image region in which the background is depicted. Here, a chromatic image is used as an example for the image to be processed 75A10, but this is merely an example, and the image to be processed 75A10 may also be an achromatic image.

[0426] Here, the person depicted in the image to be processed 75A10 is an example of the "fourth subject" related to the technology of this disclosure. The gradation of the image to be processed 75A10 is an example of the "non-noise element of the image to be processed," the "factor governing the visual impression given by the image to be processed," and the "gradation of the image to be processed" related to the technology of this disclosure.

[0427] The AI ​​processing unit 62A10 performs AI-based processing on the image 75A10 to be processed. An example of AI-based processing on the image 75A10 to be processed is processing using the generative model 82A10. The generative model 82A10 is an example of the generative model 82A shown in Figure 3. The generative model 82A10 is a generative network that has already been trained to adjust the gradation of the image 75A10 to be processed according to the person region 124. Examples of training to adjust the gradation of the image 75A10 to be processed according to the person region 124 include training to change the gradation of the image 75A10 to be processed depending on whether or not a person is depicted in the image 75A10, or training to change the gradation of the image 75A10 to be processed according to the features of the person depicted in the image 75A10.

[0428] The AI ​​processing unit 62A10 modifies the factors that govern the visual impression derived from the image 75A10 using an AI method. Specifically, the AI ​​processing unit 62A10 modifies the factors that govern the visual impression derived from the image 75A10 as non-noise elements of the image 75A10 by performing processing on the image 75A10 using the generative model 82A10. The factors that govern the visual impression derived from the image 75A10 are the gradation of the image 75A10. In the example shown in Figure 36, the AI ​​processing unit 62A10 generates the first gradation-adjusted image 86A10 by performing processing on the image 75A10 using the generative model 82A10. The first gradation-adjusted image 86A10 is an image in which the gradation of the image 75A10 has been modified according to the person region 124. For example, the gradation of the image 75A10 to be processed is changed by raising or lowering the pixel values ​​of the R pixel, G pixel, and B pixel by a similar amount. Alternatively, the gradation of the image 75A10 to be processed may be changed by raising or lowering the pixel value of at least one specified pixel among the R, G, and B pixels. The extent to which the pixel values ​​of each color are changed is determined according to the person region 124 (for example, the presence or absence of the person region 124, or the characteristics of the person indicated by the person region 124).

[0429] Here, the processing using the generation model 82A10 is an example of the "first AI processing," "first modification processing," and "first grayscale adjustment processing" related to the technology of this disclosure. The first grayscale adjustment image 86A10 is an example of the "first modified image" and "first grayscale adjustment image" related to the technology of this disclosure. "Generating the first grayscale adjustment image 86A10" is an example of "acquiring the first image" related to the technology of this disclosure.

[0430] The generation model 82A10 receives the image to be processed 75A10 as input. Based on the input image to be processed 75A10, the generation model 82A10 generates and outputs the first grayscale adjustment image 86A10.

[0431] The non-AI processing unit 62B10 performs non-AI processing on the image 75A10 to be processed. Non-AI processing refers to processing that does not use a neural network. In this ninth modification, an example of processing that does not use a neural network is processing that does not use the generative model 82A10.

[0432] An example of non-AI processing of the image to be processed 75A10 is processing using a digital filter 84A10. The digital filter 84A10 is a digital filter configured to adjust the gradation of the image to be processed 75A10. For example, the digital filter 84A10 is used when the image to be processed 75A10 contains a person region 124. In this case, for example, the non-AI processing unit 62B10 determines whether or not the image to be processed 75A10 contains a person region 124 by performing a known person detection process on the image to be processed 75A10. Then, if the non-AI processing unit 62B10 determines that the image to be processed 75A10 contains a person region 124, it performs processing on the image to be processed 75A10 using the digital filter 84A10.

[0433] Furthermore, a digital filter 84A10 may be prepared in advance for each feature of a person indicated by the person region 124. In this case, for example, the non-AI processing unit 62B10 can perform known image recognition processing on the image to be processed 75A10 to acquire the features of a person indicated by the person region 124, and then perform processing on the image to be processed 75A10 using the digital filter 84A10 corresponding to the acquired features.

[0434] The non-AI processing unit 62B10 generates a second tone-adjusted image 88A10 by performing processing (i.e., filtering) on ​​the image to be processed 75A10 using the digital filter 84A10. In other words, the non-AI processing unit 62B10 generates a second tone-adjusted image 88A10 by adjusting the non-noise elements of the image to be processed 75A10 (here, as an example, the tone of the image to be processed 75A10) using a non-AI method.

[0435] Here, the processing using the digital filter 84A10 is an example of the "non-AI method processing that does not use a neural network" and the "second modification processing that changes factors in a non-AI method" related to the technology of this disclosure. "Generating the second grayscale adjusted image 88A10" is an example of "acquiring the second image" related to the technology of this disclosure.

[0436] The digital filter 84A10 receives the image to be processed 75A10 as input. The digital filter 84A10 generates a second tone-adjusted image 88A10 based on the input image to be processed 75A10. The second tone-adjusted image 88A10 is an image obtained by changing the non-noise elements of the digital filter 84A10 (i.e., an image obtained by changing the non-noise elements of the image to be processed 75A10 through processing using the digital filter 84A10). In other words, the second tone-adjusted image 88A10 is an image in which the tone of the image to be processed 75A10 has been changed by the digital filter 84A10 (i.e., an image in which the tone has been changed through processing using the digital filter 84A10 on the image to be processed 75A10). The second tone-adjusted image 88A10 is an example of the "second image," "second modified image," and "second tone-adjusted image" related to the technology of this disclosure.

[0437] By the way, the gradation of the first gradation-adjusted image 86A10 obtained by performing AI-based processing on the image to be processed 75A10 is determined by the characteristics of the generation model 82A10 (for example, medium The resulting gradation may differ from the user's preference due to factors such as the number of layers and / or the amount of training data. If the AI ​​processing method is excessively applied to the image 75A10 being processed, it is possible that gradations that differ from the user's preference may become noticeable.

[0438] Therefore, in light of these circumstances, the imaging device 10, as an example shown in Figure 37, performs processing by the image adjustment unit 62C10 and processing by the synthesis unit 62D10 on the first grayscale adjustment image 86A10 and the second grayscale adjustment image 88A10, thereby synthesizing the first grayscale adjustment image 86A10 and the second grayscale adjustment image 88A10.

[0439] As an example, as shown in Figure 37, the NVM64 stores a ratio of 90J. The ratio of 90J is the ratio used to combine the first grayscale adjustment image 86A10 and the second grayscale adjustment image 88A10, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A10 (i.e., processing using the generation model 82A10).

[0440] The ratio 90J is broadly divided into the first ratio 90J1 and the second ratio 90J2. The first ratio 90J1 is a value between 0 and 1, and the second ratio 90J2 is the value obtained by subtracting the value of the first ratio 90J1 from "1". In other words, the first ratio 90J1 and the second ratio 90J2 are set such that the sum of the first ratio 90J1 and the second ratio 90J2 is "1". The first ratio 90J1 and the second ratio 90J2 are variable values ​​that can be changed by user instructions.

[0441] The image adjustment unit 62C10 adjusts the first tone adjustment image 86A10 generated by the AI ​​method processing unit 62A10 using the first ratio 90J1. For example, the image adjustment unit 62C10 adjusts the pixel value of each pixel in the first tone adjustment image 86A10 by multiplying the pixel value of each pixel in the first tone adjustment image 86A10 by the first ratio 90J1.

[0442] The image adjustment unit 62C10 adjusts the second grayscale adjustment image 88A10 generated by the non-AI method processing unit 62B10 using the second ratio 90J2. For example, the image adjustment unit 62C10 adjusts the pixel value of each pixel in the second grayscale adjustment image 88A10 by multiplying the pixel value of each pixel in the second grayscale adjustment image 88A10 by the second ratio 90J2.

[0443] The compositing unit 62D10 generates a composite image 92J by compositing the first grayscale adjustment image 86A10, which has been adjusted by the image adjustment unit 62C10 with a first ratio 90J1, and the second grayscale adjustment image 88A10, which has been adjusted by the image adjustment unit 62C10 with a second ratio 90J2. In other words, the compositing unit 62D10 adjusts for any excess or deficiency in the AI ​​method processing performed by the AI ​​method processing unit 62A10 by compositing the first grayscale adjustment image 86A10, which has been adjusted with a first ratio 90J1, and the second grayscale adjustment image 88A10, which has been adjusted with a second ratio 90J2. To put it another way, the compositing unit 62D10 adjusts for non-noise elements (here, as an example, the grayscale of the image to be processed 75A10) by compositing the first grayscale adjustment image 86A10, which has been adjusted with a first ratio 90J1, and the second grayscale adjustment image 88A10, which has been adjusted with a second ratio 90J2. To put it another way, the combining unit 62D10 adjusts elements derived from processing using the generation model 82A10 (for example, the pixel values ​​of pixels whose gradation has been changed by the generation model 82A10) by combining the first gradation adjustment image 86A10 adjusted by the first ratio 90J1 and the second gradation adjustment image 88A10 adjusted by the second ratio 90J2.

[0444] The synthesis performed by the synthesis unit 62D10 is the addition of pixel values ​​at corresponding pixel positions between the first grayscale adjustment image 86A10 and the second grayscale adjustment image 88A10. The synthesis by the synthesis unit 62D10 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92J by the synthesis unit 62D10 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92J, after various image processing has been performed, is then output to a predetermined output destination by the synthesis unit 62D10.

[0445] Figure 38 shows an example of the image synthesis process flow according to this ninth modified example. The flowchart shown in Figure 38 differs from the flowchart shown in Figure 6 in that steps ST450 to ST468 are applied instead of steps ST12 to ST30.

[0446] In the image synthesis process shown in Figure 38, in step ST450, the AI ​​method processing unit 62A10 and the non-AI method processing unit 62B10 acquire the image to be processed 75A10 from the image sensor 20. After the processing in step ST450 is completed, the image synthesis process proceeds to step ST452.

[0447] In step ST452, the AI ​​processing unit 62A10 inputs the image to be processed 75A10 acquired in step ST450 into the generation model 82A10. After the processing in step ST452 is executed, the image synthesis process proceeds to step ST454.

[0448] In step ST454, the AI ​​processing unit 62A10 acquires the first grayscale adjustment image 86A10, which was output from the generation model 82A10 after the image to be processed 75A10 was input to the generation model 82A10 in step ST452. After the processing in step ST454 is completed, the image synthesis process moves on to step ST456.

[0449] In step ST456, the non-AI processing unit 62B10 adjusts the gradation of the image 75A10 acquired in step ST450 by applying a digital filter 84A10 to the image 75A10. After the processing in step ST456 is completed, the image synthesis process moves to step ST458.

[0450] In step ST458, the non-AI processing unit 62B10 acquires the second grayscale adjustment image 88A10 obtained in step ST456 by processing the target image 75A10 using the digital filter 84A10. After the processing in step ST458 is executed, the image synthesis process moves on to step ST460.

[0451] In step ST460, the image adjustment unit 62C10 obtains the first ratio 90J1 and the second ratio 90J2 from the NVM 64. After the processing in step ST460 is completed, the image synthesis process proceeds to step ST462.

[0452] In step ST462, the image adjustment unit 62C10 adjusts the first grayscale adjustment image 86A10 using the first ratio 90J1 acquired in step ST460. After the processing in step ST462 is completed, the image synthesis process moves on to step ST464.

[0453] In step ST464, the image adjustment unit 62C10 adjusts the second grayscale image 88A10 using the second ratio 90J2 acquired in step ST460. After the processing in step ST464 is completed, the image synthesis process moves on to step ST466.

[0454] In step ST466, the synthesis unit 62D10 adjusts the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A10 by combining the first grayscale adjustment image 86A10 adjusted in step ST462 and the second grayscale adjustment image 88A10 adjusted in step ST464. The synthesis of the first grayscale adjustment image 86A10 adjusted in step ST462 and the second grayscale adjustment image 88A10 adjusted in step ST464 generates the synthesized image 92J. After the processing in step ST466 is executed, the image synthesis process moves on to step ST468.

[0455] In step ST468, the synthesis unit 62D10 performs various image processing on the synthesized image 92J. The image synthesis unit 62D10 then performs various image processing on the synthesized image 92J and outputs the resulting image as the processed image 75B to a predetermined output destination. After the processing in step ST468 is executed, the image synthesis process moves on to step ST32.

[0456] As explained above, in the imaging device 10 according to this ninth modified example, a first tone-adjusted image 86A10 is generated by adjusting the tone of the image to be processed 75A10 using the AI ​​method. A second tone-adjusted image 88A10 is generated by adjusting the tone of the image to be processed 75A10 using a non-AI method. The first tone-adjusted image 86A10 and the second tone-adjusted image 88A10 are then combined. This makes it possible to suppress the amount of tone adjustment in the AI ​​method processing on the combined image 92J from being excessive or insufficient. As a result, the combined image 92J is an image in which the amount of tone adjustment in the AI ​​method processing is less noticeable compared to the first tone-adjusted image 86A10, and a suitable image can be provided for users who do not like the amount of tone adjustment in the AI ​​method processing to be excessively noticeable.

[0457] In this ninth modification, the first tonal adjustment image 86A10 is generated by adjusting the tonal range of the AI-processed image 75A10 according to the person region 124. Furthermore, the second tonal adjustment image 88A10 is generated by adjusting the tonal range of the non-AI-processed image 75A10 according to the person region 124. The first tonal adjustment image 86A10 and the second tonal adjustment image 88A10 are then combined. This prevents the adjustment amount applied by the AI-processed tonal range according to the person region 124 from being excessive or insufficient in relation to the combined image 92J.

[0458] Here, an example of how the gradation is adjusted according to the person area 124 has been given and explained, but this is merely one example, and the gradation may be adjusted according to the background area 126. Alternatively, the gradation may be adjusted according to the combination of the person area 124 and the background area 126. Furthermore, the gradation may be adjusted according to areas other than the person area 124 and the background area 126 (for example, an area containing a specific vehicle, an area containing a specific animal, an area containing a specific plant, an area containing a specific building, and / or an area containing a specific aircraft, etc.).

[0459] Furthermore, while an example of a configuration in which a first tone-adjusted image 86A10 obtained by processing the target image 75A10 using the generation model 82A10 and a second tone-adjusted image 88A10 obtained by processing the target image 75A10 using the digital filter 84A10 are combined is given here, the technology of this disclosure is not limited to this. For example, the first tone-adjusted image 86A10 obtained by processing the target image 75A10 using the generation model 82A10 and the target image 75A10 (i.e., an image in which non-noise elements have not been adjusted) may be combined. In this case as well, similar effects can be expected.

[0460] [10th variation] As an example, as shown in Figure 39, the processor 62 in this tenth modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A11 instead of an AI method processing unit 62A1. In this tenth modified example, explanations of matters that are the same as those explained earlier will be omitted, and matters that are different from those explained earlier will be explained.

[0461] The AI ​​processing unit 62A11 receives the image to be processed 75A11 as input. The image to be processed 75A11 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A11 is a chromatic image. Here, a chromatic image is used as an example of the image to be processed 75A11, but this is merely an example, and the image to be processed 75A11 can also be an achromatic image. That's fine.

[0462] The AI ​​processing unit 62A11 performs AI-based processing on the image 75A11 to be processed. One example of AI-based processing on the image 75A11 is processing using the generative model 82A11. The generative model 82A11 is an example of the generative model 82A shown in Figure 3. The generative model 82A11 is a generative network that has already been trained to change the style of the image 75A11 to be processed.

[0463] Here, the style of the image to be processed 75A10 is an example of the "non-noise elements of the image to be processed," the "factors governing the visual impression given by the image to be processed," and the "style of the image to be processed" related to the technology of this disclosure.

[0464] The AI ​​processing unit 62A11 modifies the factors that govern the visual impression derived from the image 75A11 using an AI method. Specifically, the AI ​​processing unit 62A11 modifies the factors that govern the visual impression derived from the image 75A11 as non-noise elements of the image 75A11 by performing processing on the image 75A11 using the generative model 82A11. The factors that govern the visual impression derived from the image 75A11 are the style of the image 75A11. In the example shown in Figure 39, the AI ​​processing unit 62A11 generates a style-modified image 86A11 by performing processing on the image 75A11 using the generative model 82A11. The style-modified image 86A11 is an image in which the style of the image 75A11 has been changed. In the example shown in Figure 39, the style of the style-modified image 86A11 differs from the style of the image to be processed 75A11 in that multiple spiral patterns have been added.

[0465] Here, the processing using the generation model 82A11 is an example of the "first AI processing," "first modification processing," and "style modification processing" related to the technology of this disclosure. The style-modified image 86A11 is an example of the "first modified image" and "style-modified image" related to the technology of this disclosure. The image to be processed 75A11 is an example of the "second image" related to the technology of this disclosure. "Generating the style-modified image 86A11" is an example of "acquiring the first image" related to the technology of this disclosure.

[0466] The generation model 82A11 receives the image to be processed 75A11 as input. Based on the input image to be processed 75A11, the generation model 82A11 generates and outputs a style-modified image 86A11.

[0467] Incidentally, the style of the style-modified image 86A11 obtained by applying AI-based processing to the target image 75A11 may differ from the user's preference due to the characteristics of the generative model 82A11 (e.g., the number of intermediate layers and / or the amount of training). If the influence of AI-based processing is excessively reflected on the target image 75A11, it is possible that a style different from the user's preference may become prominent.

[0468] Therefore, in light of these circumstances, the imaging device 10, as an example as shown in Figure 40, performs processing by the image adjustment unit 62C11 and processing by the synthesis unit 62D11 on the style-changed image 86A11 and the image to be processed 75A11, thereby synthesizing the style-changed image 86A11 and the image to be processed 75A11.

[0469] As an example, as shown in Figure 40, the NVM64 stores a ratio of 90K. The ratio of 90K is the ratio used to combine the style-changed image 86A11 and the image to be processed 75A11, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A11 (i.e., processing using the generation model 82A11).

[0470] The ratio 90K is broadly divided into the first ratio 90K1 and the second ratio 90K2. The first ratio 90K1 is a value between 0 and 1, and the second ratio 90K2 is the value obtained by subtracting the value of the first ratio 90K1 from "1". In other words, the first ratio 90K1 and the second ratio 90K2 are set so that the sum of the first ratio 90K1 and the second ratio 90K2 is "1". The first ratio 90K1 and the second ratio 90K2 are variable values ​​that can be changed by user instructions.

[0471] The image adjustment unit 62C11 adjusts the style-changed image 86A11 generated by the AI ​​method processing unit 62A11 using a first ratio 90K1. For example, the image adjustment unit 62C11 adjusts the pixel value of each pixel in the style-changed image 86A11 by multiplying the pixel value of each pixel in the style-changed image 86A11 by the first ratio 90K1.

[0472] The image adjustment unit 62C11 adjusts the image to be processed 75A11 using a second ratio 90K2. For example, the image adjustment unit 62C11 adjusts the pixel value of each pixel in the image to be processed 75A11 by multiplying the pixel value of each pixel in the image to be processed 75A11 by the second ratio 90K2.

[0473] The compositing unit 62D11 generates a composite image 92K by compositing the style-changed image 86A11, which has been adjusted by the image adjustment unit 62C11 at a first ratio of 90K1, with the processing target image 75A11, which has been adjusted by the image adjustment unit 62C11 at a second ratio of 90K2. In other words, the compositing unit 62D11 adjusts for any excess or deficiency in the AI ​​method processing performed by the AI ​​method processing unit 62A11 by compositing the style-changed image 86A11, which has been adjusted at a first ratio of 90K1, with the processing target image 75A11, which has been adjusted at a second ratio of 90K2. To put it another way, the compositing unit 62D11 adjusts for non-noise elements (here, as an example, the style of the processing target image 75A11) by compositing the style-changed image 86A11, which has been adjusted at a first ratio of 90K1, with the processing target image 75A11, which has been adjusted at a second ratio of 90K2. To put it another way, the synthesis unit 62D11 synthesizes the style-modified image 86A11 adjusted by the first ratio 90K1 and the processing target image 75A11 adjusted by the second ratio 90K2, thereby adjusting elements derived from processing using the generation model 82A11 (for example, the pixel values ​​of pixels whose style has been modified by the generation model 82A11).

[0474] The synthesis performed by the synthesis unit 62D11 is the addition of pixel values ​​at corresponding pixel positions between the style-changed image 86A11 and the image to be processed 75A11. The synthesis by the synthesis unit 62D11 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92K by the synthesis unit 62D11 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92K, after various image processing, is then output to a predetermined output destination by the synthesis unit 62D11.

[0475] Figure 41 shows an example of the image synthesis process flow according to this 10th modified example. The flowchart shown in Figure 41 differs from the flowchart shown in Figure 6 in that steps ST500 to ST514 are applied instead of steps ST12 to ST30.

[0476] In the image synthesis process shown in Figure 41, in step ST500, the AI ​​method processing unit 62A11 acquires the image to be processed 75A11 from the image sensor 20. After the processing in step ST500 is completed, the image synthesis process proceeds to step ST502.

[0477] In step ST502, the AI ​​processing unit 62A11 inputs the image to be processed 75A11 acquired in step ST500 into the generation model 82A11. After the processing in step ST512 is executed, the image synthesis process proceeds to step ST504.

[0478] In step ST504, the AI ​​method processing unit 62A11 processes the target in step ST512. Image 75A11 is input to the generation model 82A11, and the style-modified image 86A11 output from the generation model 82A11 is obtained. After the processing in step ST514 is executed, the image synthesis process moves to step ST506.

[0479] In step ST506, the image adjustment unit 62C11 obtains the first ratio 90K1 and the second ratio 90K2 from the NVM 64. After the processing in step ST506 is completed, the image synthesis process proceeds to step ST508.

[0480] In step ST508, the image adjustment unit 62C11 adjusts the style-changed image 86A11 using the first ratio 90K1 acquired in step ST506. After the processing in step ST508 is completed, the image synthesis process moves on to step ST510.

[0481] In step ST510, the image adjustment unit 62C11 adjusts the image to be processed 75A11 using the second ratio 90K2 acquired in step ST506. After the processing in step ST510 is completed, the image synthesis process proceeds to step ST512.

[0482] In step ST512, the synthesis unit 62D11 synthesizes the style-changed image 86A11 adjusted in step ST508 and the processing target image 75A11 adjusted in step ST510, thereby adjusting the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A11. The synthesis of the style-changed image 86A11 adjusted in step ST508 and the processing target image 75A11 adjusted in step ST510 generates a synthesized image 92K. After the processing in step ST512 is executed, the image synthesis process moves on to step ST514.

[0483] In step ST514, the image merging unit 62D11 performs various image processing on the composite image 92K. The image merging unit 62D11 then outputs the image obtained by performing various image processing on the composite image 92K as the processed image 75B to a predetermined output destination. After the processing in step ST514 is completed, the image merging process moves on to step ST32.

[0484] As explained above, in the imaging device 10 according to this 10th modified example, the style of the image to be processed 75A11 is adjusted using the AI ​​method to generate a style-modified image 86A11. Then, the style-modified image 86A11 and the image to be processed 75A11 are combined. This makes it possible to suppress any excess or deficiency of the style modified by the AI ​​method in the combined image 92K. As a result, the combined image 92K is an image in which the style modified by the AI ​​method is less noticeable compared to the style-modified image 86A11, and a suitable image can be provided for users who do not like the style modified by the AI ​​method to be overly noticeable.

[0485] [11th variation] As an example, as shown in Figure 42, the processor 62 in this 11th modified example differs from the processor 62 shown in Figure 4 in that it has an AI method processing unit 62A12 instead of an AI method processing unit 62A1. In this 11th modified example, explanations of matters that are the same as those explained earlier will be omitted, and matters that are different from those explained earlier will be explained.

[0486] The AI ​​processing unit 62A12 receives the image to be processed 75A12 as input. The image to be processed 75A12 is an example of the image to be processed 75A shown in Figure 2. The image to be processed 75A12 is a chromatic image. Here, a chromatic image is used as an example of the image to be processed 75A12, but this is merely an example, and the image to be processed 75A12 may also be an achromatic image.

[0487] The image to be processed, 75A12, has a person region, 128. The person region, 128 is an image region representing a person. The person region, 128, has a skin region, 128A, representing skin. The skin region, 128A, also contains a blemish region, 128A1. The blemish region, 128A1, is an image region representing a blemish on the skin. Although a blemish is used as an example here, it is not limited to blemishes; it could be a mole and / or a scar, or any element that detracts from the aesthetics of the skin.

[0488] The AI ​​processing unit 62A12 performs AI-based processing on the image 75A12 to be processed. One example of AI-based processing on the image 75A12 to be processed is processing using the generative model 82A12. The generative model 82A12 is an example of the generative model 82A shown in Figure 3. The generative model 82A12 is a generative network that has already been trained to adjust the image quality related to skin in the image 75A12 to be processed (i.e., the image quality of skin region 128A). Adjusting the image quality related to skin refers to, for example, making the blemish region 128A1 in the image 75A12 less noticeable (for example, erasing the blemish region 128A1).

[0489] Here, the image quality of the skin region 128A is an example of the "non-noise elements of the image to be processed," the "factors governing the visual impression given by the image to be processed," and the "image quality related to skin" related to the technology of this disclosure.

[0490] The AI ​​processing unit 62A12 modifies the factors that govern the visual impression derived from the image 75A12 using an AI method. Specifically, the AI ​​processing unit 62A12 modifies the factors that govern the visual impression derived from the image 75A12 as non-noise elements of the image 75A12 by performing processing on the image 75A12 using the generative model 82A12. The factors that govern the visual impression derived from the image 75A12 are the image quality of the skin region 128A. In the example shown in Figure 42, the AI ​​processing unit 62A12 generates a skin quality adjustment image 86A12 by performing processing on the image 75A12 using the generative model 82A12. The skin quality adjustment image 86A12 is an image in which the image quality of the skin region 128A contained in the image 75A12 has been adjusted. In the example shown in Figure 42, the skin quality adjustment image 86A12 differs from the processed image 75A12 in that the blemish area 128A1 has been removed.

[0491] Here, the processing using the generation model 82A12 is an example of the "first AI processing," "first modification processing," and "skin quality adjustment processing" related to the technology of this disclosure. The skin quality adjustment image 86A12 is an example of the "first modified image" and "skin quality adjustment image" related to the technology of this disclosure. The image to be processed 75A12 is an example of the "second image" related to the technology of this disclosure. "Generating the skin quality adjustment image 86A12" is an example of "acquiring the first image" related to the technology of this disclosure.

[0492] The generation model 82A12 receives the image to be processed 75A12 as input. Based on the input image to be processed 75A12, the generation model 82A12 generates and outputs a skin quality adjustment image 86A12.

[0493] By the way, the image quality of the skin region 128A in the skin quality adjustment image 86A12 obtained by performing AI-based processing on the target image 75A12 may differ from the user's preference due to the characteristics of the generative model 82A12 (e.g., the number of intermediate layers and / or the amount of training). If the influence of the AI-based processing is excessively reflected on the target image 75A12, it is possible that an image quality different from the user's preference will become noticeable. For example, there is a risk that the image will become unnatural because the blemish region 128A1 is completely erased.

[0494] Therefore, in light of these circumstances, the imaging device 10, as an example as shown in Figure 43, applies the image adjustment unit 62C12 to the skin quality adjustment image 86A12 and the image to be processed 75A12. The processing and synthesis unit 62D12 performs the necessary operations to synthesize the skin quality adjustment image 86A12 and the image to be processed 75A12.

[0495] As an example, as shown in Figure 43, the NVM64 stores a ratio of 90L. Ratio 90L is the ratio used to combine the skin quality adjustment image 86A12 and the image to be processed 75A12, and is set to adjust for any excess or deficiency in the AI ​​method processing by the AI ​​method processing unit 62A12 (i.e., processing using the generative model 82A12).

[0496] The ratio 90L is broadly divided into the first ratio 90L1 and the second ratio 90L2. The first ratio 90L1 is a value between 0 and 1, and the second ratio 90L2 is the value obtained by subtracting the value of the first ratio 90L1 from "1". In other words, the first ratio 90L1 and the second ratio 90L2 are set so that the sum of the first ratio 90L1 and the second ratio 90L2 is "1". The first ratio 90L1 and the second ratio 90L2 are variable values ​​that can be changed by user instructions.

[0497] The image adjustment unit 62C12 adjusts the skin quality adjustment image 86A12 generated by the AI ​​method processing unit 62A12 using a first ratio 90L1. For example, the image adjustment unit 62C12 adjusts the pixel value of each pixel in the skin quality adjustment image 86A12 by multiplying the pixel value of each pixel in the skin quality adjustment image 86A12 by the first ratio 90L1.

[0498] The image adjustment unit 62C12 adjusts the image to be processed 75A12 using a second ratio 90L2. For example, the image adjustment unit 62C12 adjusts the pixel value of each pixel in the image to be processed 75A12 by multiplying the pixel value of each pixel in the image to be processed 75A12 by the second ratio 90L2.

[0499] The synthesis unit 62D12 generates a composite image 92L by combining the skin quality adjustment image 86A12, which has been adjusted by the image adjustment unit 62C12 at a first ratio 90L1, with the processing target image 75A12, which has been adjusted by the image adjustment unit 62C12 at a second ratio 90L2. In other words, the synthesis unit 62D12 adjusts for any excess or deficiency in the AI ​​processing performed by the AI ​​processing unit 62A12 by combining the skin quality adjustment image 86A12, which has been adjusted at a first ratio 90L1, with the processing target image 75A12, which has been adjusted at a second ratio 90L2. To put it another way, the synthesis unit 62D12 adjusts for non-noise elements (here, as an example, the image quality of the skin region 128A) by combining the skin quality adjustment image 86A12, which has been adjusted at a first ratio 90L1, with the processing target image 75A12, which has been adjusted at a second ratio 90L2. To put it another way, the synthesis unit 62D12 synthesizes the skin quality adjustment image 86A12 adjusted with a first ratio 90L1 and the processing target image 75A12 adjusted with a second ratio 90L2, thereby adjusting elements derived from processing using the generation model 82A12 (for example, the pixel values ​​of pixels whose image quality has been changed by the generation model 82A12).

[0500] The synthesis performed by the synthesis unit 62D12 is the addition of pixel values ​​at corresponding pixel positions between the skin quality adjustment image 86A12 and the image to be processed 75A12. The synthesis by the synthesis unit 62D12 is performed in the same manner as the synthesis by the synthesis unit 62D1 shown in Figure 5. Furthermore, various image processing is performed on the synthesized image 92L by the synthesis unit 62D12 in the same manner as the synthesized image 92A shown in Figure 5. The synthesized image 92L, after various image processing has been performed, is then output to a predetermined output destination by the synthesis unit 62D12.

[0501] Figure 44 shows an example of the image synthesis process flow according to this 11th modified example. The flowchart shown in Figure 44 differs from the flowchart shown in Figure 6 in that steps ST550 to ST564 are applied instead of steps ST12 to ST30.

[0502] In the image synthesis process shown in Figure 44, in step ST550, the AI ​​method processing unit 62A12 Next, the image to be processed, 75A12, is acquired from the image sensor 20. After the processing in step ST550 is completed, the image synthesis process proceeds to step ST552.

[0503] In step ST552, the AI ​​processing unit 62A12 inputs the image to be processed 75A12 acquired in step ST550 into the generation model 82A12. After the processing in step ST552 is executed, the image synthesis process proceeds to step ST554.

[0504] In step ST554, the AI ​​processing unit 62A12 acquires the skin quality adjustment image 86A12 output from the generation model 82A12, which was input to the generation model 82A12 in step ST552. After the processing in step ST554 is completed, the image synthesis process moves on to step ST556.

[0505] In step ST556, the image adjustment unit 62C12 obtains the first ratio 90L1 and the second ratio 90L2 from the NVM 64. After the processing in step ST556 is completed, the image synthesis process proceeds to step ST558.

[0506] In step ST558, the image adjustment unit 62C12 adjusts the skin quality adjustment image 86A12 using the first ratio 90L1 acquired in step ST556. After the processing in step ST558 is completed, the image synthesis process moves on to step ST560.

[0507] In step ST560, the image adjustment unit 62C12 adjusts the image to be processed 75A12 using the second ratio 90L2 acquired in step ST556. After the processing in step ST560 is completed, the image synthesis process proceeds to step ST562.

[0508] In step ST562, the synthesis unit 62D12 synthesizes the skin quality adjustment image 86A12 adjusted in step ST558 and the processing target image 75A12 adjusted in step ST560, thereby adjusting the excess or deficiency of the AI ​​method processing by the AI ​​method processing unit 62A11. The synthesis of the skin quality adjustment image 86A12 adjusted in step ST558 and the processing target image 75A12 adjusted in step ST560 generates a synthesized image 92L. After the processing in step ST562 is executed, the image synthesis process moves on to step ST564.

[0509] In step ST564, the image merging unit 62D12 performs various image processing on the composite image 92L. The image merging unit 62D12 then outputs the image obtained by performing various image processing on the composite image 92L as the processed image 75B to a predetermined output destination. After the processing in step ST564 is completed, the image merging process proceeds to step ST32.

[0510] As described above, in the imaging device 10 according to this 11th modified example, the image quality of the skin region 128A of the image to be processed 75A12 is adjusted using the AI ​​method to generate a skin quality adjustment image 86A12. Then, the skin quality adjustment image 86A12 and the image to be processed 75A12 are combined. This makes it possible to suppress the amount of image quality adjustment adjusted by the AI ​​method from being excessive or insufficient in the combined image 92L. As a result, the combined image 92L is an image in which the amount of image quality adjustment changed by the AI ​​method is less noticeable compared to the skin quality adjustment image 86A12, and a suitable image can be provided for users who do not want the amount of image quality adjustment changed by the AI ​​method to be excessively noticeable (for example, users who do not want the blemish region 128A1 to be completely erased).

[0511] Here, we have described an example of how the blemish area 128A1 is removed, but the technology of this disclosure is not limited to this. For example, by changing the brightness of the skin area 128A or changing the color of the skin area 128A using an AI method, the blemish area in the image 75A12 being processed is removed. The skin may be made to appear more beautiful. In this case as well, the processing described in steps ST556 to ST564 above should be performed so that the skin of the person in the image does not look unnatural due to excessive skin beautification.

[0512] In the following, for the sake of explanation, if it is not necessary to distinguish between the processed images 75A1 to 75A12, they will be referred to as "processed image 75A". Also, in the following, for the sake of explanation, if it is not necessary to distinguish between the ratios 90A to 90L, they will be referred to as "ratio 90". Also, in the following, for the sake of explanation, if it is not necessary to distinguish between the first aberration correction image 86A1, the first color image 86A2, the first contrast adjustment image 86A3, the first resolution adjustment image 86A4, the first HDR image 86A5, the first edge enhancement image 86A6, the first point image adjustment image 86A7, the first blur image 86A8, the first bokeh image 86A9, the first gradation adjustment image 86A10, the style change image 86A11, and the skin quality adjustment image 86A12, they will be referred to as "first image 86A". When it is not necessary to distinguish between the second aberration correction image 88A1, the second color image 88A2, the second contrast adjustment image 88A3, the second resolution adjustment image 88A4, the second HDR image 88A5, the second edge enhancement image 88A6, the second point image adjustment image 88A7, the second blur image 88A8, the second bokeh image 88A9, the second grayscale adjustment image 88A10, the image to be processed 75A11, and the image to be processed 75A12, they will be referred to as "second image 88A". Also, for the sake of explanation, when it is not necessary to distinguish between the generation models 82A1 to 82A12, they will be referred to as "generation model 82A". Also, for the sake of explanation, when it is not necessary to distinguish between the AI ​​method processing units 62A1 to 62A12, they will be referred to as "AI method processing unit 62A". Furthermore, for the sake of clarity, in the following explanation, if it is not necessary to distinguish between composite images 92A to 92L, they will be referred to as "composite image 92".

[0513] [12th variation] In the examples shown in Figures 1 to 44, the processor 62 generates the first image 86A by performing a single, purpose-specific process using an AI method. However, the technology of this disclosure is not limited to this. For example, the processor 62 may perform multiple processes using an AI method.

[0514] In this case, as shown in Figure 45 as an example, the AI ​​processing unit 62A13 performs multiple purpose-specific processing 130 on the image to be processed 75A using the AI ​​method. That is, the AI ​​processing unit 62A13 performs processing on the image to be processed 75A using multiple generative models 82A. Multiple purpose-specific processing 130 is performed on the image to be processed 75A using the AI ​​method, generating multiple processed images 132. The multiple processed images 132 and the second image 88A are combined at a ratio of 90.

[0515] The multiple purpose-specific processing 130 includes aberration correction processing 130A, point image adjustment processing 130B, gradation adjustment processing 130C, contrast adjustment processing 130D, dynamic range adjustment processing 130E, resolution adjustment processing 130F, edge enhancement processing 130G, clarity adjustment processing 130H, bokeh generation processing 130I, blurring processing 130J, skin tone quality adjustment processing 130K, color adjustment processing 130L, and style change processing 130M.

[0516] An example of aberration correction processing 130A is the processing performed by the AI-type processing unit 62A1 shown in Figure 4. An example of point image adjustment processing 130B is the processing performed by the AI-type processing unit 62A7 shown in Figure 25. An example of gradation adjustment processing 130C is the processing performed by the AI-type processing unit 62A10 shown in Figure 36. An example of contrast adjustment processing 130D is the processing performed by the AI-type processing unit 62A3 shown in Figure 19. An example of dynamic range adjustment processing 130E is the processing performed by the AI-type processing unit 62A5 shown in Figure 19. An example of resolution adjustment processing 130F is the processing performed by the AI-type processing unit 62A4 shown in Figure 16. An example of edge enhancement processing 130G is the processing performed by the AI-type processing unit 62A6 shown in Figure 22. The processes performed are as follows: An example of clarity adjustment processing 130H is the process performed by the AI ​​processing unit 62A3 shown in Figure 14. An example of bokeh generation processing 130I is the process performed by the AI ​​processing unit 62A9 ​​shown in Figure 31. An example of blur application processing 130J is the process performed by the AI ​​processing unit 62A8 shown in Figure 28. An example of skin quality adjustment processing 130K is the process performed by the AI ​​processing unit 62A12 shown in Figure 42. An example of color adjustment processing 130L is the process performed by the AI ​​processing unit 62A2 shown in Figure 7. An example of style change processing 130M is the process performed by the AI ​​processing unit 62A11 shown in Figure 39.

[0517] Multiple purpose-specific processes 130 are performed in an order based on the degree of influence they have on the image 75A to be processed. For example, the multiple purpose-specific processes 130 are performed in stages, from purpose-specific processes 130 that have a small influence on the image 75A to purpose-specific processes 130 that have a large influence on the image 75A to be processed. In the example shown in Figure 45, the processes performed on the image 75A to be processed are performed in the following order: aberration correction process 130A, point image adjustment process 130B, gradation adjustment process 130C, contrast adjustment process 130D, dynamic range adjustment process 130E, resolution adjustment process 130F, edge enhancement process 130G, clarity adjustment process 130H, bokeh generation process 130I, blur application process 130J, skin tone quality adjustment process 130K, color adjustment process 130L, and style change process 130M.

[0518] Thus, in this 12th modified example, multiple purpose-specific processing 130 is performed on the image 75A using the AI ​​method, and the resulting multiple processed images 132 and the second image 88A are combined at a ratio of 90, so that the same effect as the examples shown in Figures 1 to 44 can be obtained.

[0519] Furthermore, in this twelfth modification, since the multiple purpose-specific processes 130 are performed in an order based on the degree of influence they have on the image 75A to be processed, it is possible to suppress the unnatural appearance of the multiple processed images 132 compared to the case where the multiple purpose-specific processes 130 are performed on the image 75A in an order determined without considering the degree of influence they have on the image 75A to be processed.

[0520] Furthermore, in this twelfth modification, the multiple purpose-specific processes 130 are performed in stages, from purpose-specific processes 130 that have a small impact on the image 75A to purpose-specific processes 130 that have a large impact on the image 75A. Therefore, compared to the case where the multiple purpose-specific processes 130 are performed in stages, from purpose-specific processes 130 that have a large impact on the image 75A to purpose-specific processes 130 that have a small impact on the image 75A, it is possible to suppress the unnatural appearance of the multiple processed images 132.

[0521] [13th variation] In the examples shown in Figures 1 to 45, an example is given in which the ratio 90 is determined according to instructions from the user. However, the technology of this disclosure is not limited to this, and the ratio 90 may be determined by other methods. For example, if the difference between the image to be processed 75A and the first image 86A, or the difference between the first image 86A and the second image 88A is excessively large, or if the difference between the image to be processed 75A and the first image 86A, or the difference between the first image 86A and the second image 88A is excessively small, it can be determined that the AI ​​processing has a large impact on the first image 86A. Therefore, the ratio 90 may be determined according to the difference between the image to be processed 75A and the first image 86A, or the difference between the first image 86A and the second image 88A, or the difference between the image to be processed 75A and the first image 86A, or the difference between the first image 86A and the second image 88A.

[0522] In this case, for example, as shown in Figure 46, the processor 62 derives a ratio 90 based on the difference 134 between the image to be processed 75A and the first image 86A. The ratio 90 may be calculated from an arithmetic formula in which the difference 134 is the independent variable and the ratio 90 is the dependent variable, or the difference The ratio 90 may be derived from a table that associates the difference 134 with the ratio 90. Alternatively, a division value may be used instead of the difference 134. An example of a division value used instead of the difference 134 is the ratio of one of the following: the statistical value of the pixel values ​​(e.g., average pixel value) of multiple pixels in the image to be processed 75A (e.g., all pixels or multiple pixels constituting the area in which the main subject is depicted) and the statistical value of the pixel values ​​(e.g., average pixel value) of multiple pixels in the first image 86A (e.g., all pixels or multiple pixels constituting the area in which the main subject is depicted).

[0523] Furthermore, as shown in Figure 47, for example, the processor 62 may derive the ratio 90 based on the difference 136 between the first image 86A and the second image 88A. In this case as well, the ratio 90 may be calculated from an arithmetic formula in which the difference 136 is the independent variable and the ratio 90 is the dependent variable, or the ratio 90 may be derived from a table that associates the difference 136 with the ratio 90. Alternatively, a division value may be used instead of the difference 136. An example of a division value used instead of the difference 136 is the ratio of one of the following: the statistical value of the pixel values ​​(e.g., average pixel value) of multiple pixels in the first image 86A (e.g., all pixels or multiple pixels constituting the area in which the main subject is depicted) and the statistical value of the pixel values ​​(e.g., average pixel value) of multiple pixels in the second image 88A (e.g., all pixels or multiple pixels constituting the area in which the main subject is depicted).

[0524] Alternatively, the ratio 90 may be derived based on the differences 134 and 136. In this case, the ratio 90 may be calculated from an arithmetic formula in which the differences 134 and 136 are independent variables and the ratio 90 is the dependent variable, or the ratio 90 may be derived from a table that associates the differences 134 and 136 with the ratio 90.

[0525] Thus, according to this 13th modified example, the ratio 90 is determined based on the difference between the image to be processed 75A and the first image 86A, and / or the difference between the first image 86A and the second image 88A. Therefore, compared to the case where the ratio 90 is a fixed value determined without considering the first image 86A, it is possible to suppress the unnatural appearance of the image obtained by combining the first image 86A and the second image 88A due to the effects of AI processing.

[0526] [14th variation] As an example, as shown in Figure 48, the processor 62 may adjust the ratio 90 according to related information 138 related to the image to be processed 75A. Here, a first example of related information 138 is information related to the sensitivity of the image sensor 20 (e.g., ISO sensitivity). A second example of related information 138 is information related to the brightness of the image to be processed 75A (e.g., the average, median, or mode of the pixel values ​​of the image to be processed 75A). A third example of related information 138 is information indicating the spatial frequency of the image to be processed 75A. A fourth example of related information 138 is the subject image within the image to be processed 75A (e.g., a person image or a part image).

[0527] Thus, according to this 14th modified example, the ratio 90 is adjusted according to the related information 138 associated with the image to be processed 75A. Therefore, compared to the case where the ratio 90 is changed without considering the related information 138 at all, it is possible to suppress the deterioration of the image quality of the image obtained by combining the first image 86A and the second image 88A due to the related information 138.

[0528] [Other variations] In the above, an example of a configuration in which the AI ​​processing unit 62A performs processing using the generation model 82A was given, but multiple types of generation models 82A may be used by the AI ​​processing unit 62A depending on the conditions. For example, the generation model 82A used by the AI ​​processing unit 62A may be switched depending on the imaging scene captured by the imaging device 10. Alternatively, the ratio 90 may be changed according to the generation model 82A used by the AI ​​processing unit 62A.

[0529] In the above description, examples were given of a configuration in which a chromatic or achromatic image obtained by imaging with the imaging device 10 is used as the image to be processed 75A. However, the technology of this disclosure is not limited thereto, and the image to be processed 75A may also be a depth image.

[0530] In the above description, an example was given in which a second image 88A is obtained by performing a non-AI method of processing on the target image 75A. However, the technology of this disclosure is not limited thereto, and the second image 88A may be an image obtained by performing a non-AI method of processing on the target image 75A and processing using a trained model different from the generative model 82A.

[0531] The above description illustrates an example in which image synthesis processing is performed by the processor 62 of the image processing engine 12 included in the imaging device 10. However, the technology of this disclosure is not limited to this, and the device that performs image synthesis processing may be provided outside the imaging device 10. In this case, as an example, an imaging system 140 may be used, as shown in Figure 49. The imaging system 140 comprises the imaging device 10 and an external device 142. The external device 142 is, for example, a server. The server is implemented, for example, by cloud computing. Cloud computing is given as an example here, but this is merely an example. For example, the server may be implemented by a mainframe, or by network computing such as fog computing, edge computing, or grid computing. A server is given here as an example of an external device 142, but this is merely an example. Instead of a server, at least one personal computer or the like may be used as the external device 142.

[0532] The external device 142 includes a processor 144, an NVM 146, RAM 148, and a communication interface 150. The processor 144, NVM 146, RAM 148, and communication interface 150 are connected by a bus 152. The communication interface 150 is connected to the imaging device 10 via a network 154. The network 154 is, for example, the internet. Note that the network 154 is not limited to the internet, but may also be a WAN and / or a LAN such as an intranet.

[0533] The NVM146 stores the image synthesis processing program 80, the generation model 82A, and the digital filter 84A. The processor 144 executes the image synthesis processing program 80 on the RAM 146. The processor 144 performs the image synthesis processing described above according to the image synthesis processing program 80 executed on the RAM 146. When performing image synthesis processing, the processor 144 processes the image to be processed 75A using the generation model 82A and the digital filter 84A as described in each of the above examples. The image to be processed 75A is transmitted, for example, from the imaging device 10 to the external device 142 via the network 154. The communication I / F 150 of the external device 142 receives the image to be processed 75A. The processor 144 performs image synthesis processing on the image to be processed 75A received by the communication I / F 150. The processor 144 generates a composite image 92 by performing image synthesis processing and transmits the generated composite image 92 to the imaging device 10. The imaging device 10 receives the composite image 92 transmitted from the external device 142 via the communication I / F 52 (see Figure 2).

[0534] In the example shown in Figure 49, the external device 142 is an example of an "image processing device" and a "computer" related to the technology of this disclosure, and the processor 144 is an example of a "processor" related to the technology of this disclosure.

[0535] Furthermore, the image synthesis process may be performed in a distributed manner by multiple devices, including the imaging device 10 and the external device 142.

[0536] Although processor 62 was used as an example above, at least one other CPU, at least one GPU, and / or at least one TPU may be used instead of, or in conjunction with, processor 62.

[0537] The above description uses an example where the image synthesis processing program 80 is stored in the NVM 62, but the technology of this disclosure is not limited to this. For example, the image synthesis processing program 80 may be stored in a portable non-temporary storage medium such as an SSD or USB memory. The image synthesis processing program 80 stored in the non-temporary storage medium is installed in the image processing engine 12 of the imaging device 10. The processor 62 performs image synthesis processing according to the image synthesis processing program 80.

[0538] Alternatively, the image synthesis processing program 80 may be stored in a storage device such as another computer or server connected to the imaging device 10 via a network, and the image synthesis processing program 80 may be downloaded and installed in the image processing engine 12 upon request from the imaging device 10.

[0539] It is not necessary to store the entire image synthesis processing program 80 in a storage device such as another computer or server connected to the imaging device 10, or in the NVM 62; it is acceptable to store only a portion of the image synthesis processing program 80.

[0540] Furthermore, although the imaging device 10 shown in Figures 1 and 2 has a built-in image processing engine 12, the technology of this disclosure is not limited to this, and for example, the image processing engine 12 may be provided outside the imaging device 10.

[0541] Alth...

Claims

1. Equipped with a processor, The aforementioned processor, A first image obtained by performing a first AI processing on the image to be processed, and a second image obtained without performing the first AI processing on the image to be processed are acquired. The image to be processed is an image obtained by capturing the subject light, which is imaged on a light-receiving surface by the lens of the imaging device, with the imaging device. The first image includes a first aberration-corrected image obtained by performing an aberration region correction process, which corrects the region of the captured image in which the aberration of the lens is reflected, using an AI method, as a process included in the first AI process. The second image includes a second aberration-corrected image obtained by performing a process to correct the region of the captured image in which the aberration of the lens is reflected using a non-AI method. The processor adjusts the elements originating from the aberration region correction process by combining the first aberration-corrected image and the second aberration-corrected image at a ratio that adjusts for any excess or deficiency of the first AI processing. Image processing device.

2. Equipped with a processor, The aforementioned processor, A first image obtained by performing a first AI processing on the image to be processed, and a second image obtained without performing the first AI processing on the image to be processed are acquired. The first image includes a first colored image obtained by performing a coloring process as part of the first AI processing, in which a first region and a second region which is a region different from the first region are colored in an AI manner on the image to be processed. The second image includes a second colored image obtained by performing a process to change the color of the image to be processed using a non-AI method. The processor adjusts the elements derived from the coloring process by combining the first colored image and the second colored image at a ratio that adjusts for any excess or deficiency of the first AI processing. Image processing device.

3. The second image is an image obtained by performing a non-AI method of processing on the image to be processed, which does not use a neural network. The image processing apparatus according to claim 1.

4. The second colored image is an image obtained by performing a process on the image to be processed in which the first region and the second region are colored in a way that allows them to be distinguished using a non-AI method. The image processing apparatus according to claim 2.

5. The image to be processed is an image obtained by capturing the first subject, The first region is a region within the image to be processed in which a specific subject included in the first subject is depicted. The image processing apparatus according to claim 2.

6. An image processing apparatus according to any one of claims 1 to 5, Equipped with an image sensor, The image to be processed is an image obtained by capturing images using the image sensor. Imaging device.

7. An image processing method, The image processing method includes obtaining a first image obtained by performing a first AI processing on the image to be processed, and a second image obtained without performing the first AI processing on the image to be processed. The image to be processed is an image obtained by capturing the subject light, which is imaged on a light-receiving surface by the lens of the imaging device, with the imaging device. The first image includes a first aberration-corrected image obtained by performing an aberration region correction process, which corrects the region of the captured image in which the aberration of the lens is reflected, using an AI method, as a process included in the first AI process. The second image includes a second aberration-corrected image obtained by performing a process to correct the region of the captured image in which the aberration of the lens is reflected using a non-AI method. The image processing method includes adjusting elements derived from the aberration region correction process by combining the first aberration-corrected image and the second aberration-corrected image at a ratio that adjusts for any excess or deficiency of the first AI processing. Image processing methods.

8. An image processing method, The image processing method includes obtaining a first image obtained by performing a first AI processing on the image to be processed, and a second image obtained without performing the first AI processing on the image to be processed. The first image includes a first colored image obtained by performing a coloring process as part of the first AI processing, in which a first region and a second region which is a region different from the first region are colored in an AI manner on the image to be processed. The second image includes a second colored image obtained by performing a process to change the color of the image to be processed using a non-AI method. The image processing method includes adjusting elements derived from the coloring process by combining the first colored image and the second colored image at a ratio that adjusts for any excess or deficiency of the first AI processing. Image processing methods.

9. A program that causes a computer to perform a process, The process includes obtaining a first image obtained by performing a first AI process on the image to be processed, and a second image obtained without performing the first AI process on the image to be processed. The image to be processed is an image obtained by capturing the subject light, which is imaged on a light-receiving surface by the lens of the imaging device, with the imaging device. The first image includes a first aberration-corrected image obtained by performing an aberration region correction process, which corrects the region of the captured image in which the aberration of the lens is reflected, using an AI method, as a process included in the first AI process. The second image includes a second aberration-corrected image obtained by performing a process to correct the region of the captured image in which the aberration of the lens is reflected using a non-AI method. The process includes adjusting elements derived from the aberration region correction process by combining the first aberration-corrected image and the second aberration-corrected image at a ratio that adjusts for any excess or deficiency of the first AI processing. program.

10. A program that causes a computer to perform a process, The process includes obtaining a first image obtained by performing a first AI process on the image to be processed, and a second image obtained without performing the first AI process on the image to be processed. The first image includes a first colored image obtained by performing a coloring process as part of the first AI processing, in which a first region and a second region which is a region different from the first region are colored in an AI manner on the image to be processed. The second image includes a second colored image obtained by performing a process to change the color of the image to be processed using a non-AI method. The process includes adjusting elements derived from the coloring process by combining the first colored image and the second colored image at a ratio that adjusts for any excess or deficiency of the first AI processing. program.

Images