Image processing device

Abstract

In multi-frame synthesis processing, the present invention provides an image processing device that can suppress the degradation of the synthesized image that occurs particularly in two processing steps: image resizing and image data movement, thereby generating a higher quality synthesized image. [Solution] The image processing apparatus 11 comprises a super-resolution processing unit 11a that expands the resolution of each reference image data RI to a higher resolution super-resolution image data; an alignment processing unit 11b that aligns the super-resolution image data processed by the super-resolution processing unit 11a; and a synthesis unit 11d that generates a final single composite image data using the super-resolution image data after alignment by the alignment processing unit 11b. Furthermore, it comprises an oversampling unit 11c that performs resizing and compression processing of the super-resolution image data, and the synthesis unit 11d generates a final single composite image data from the compressed image data after compression processing by the oversampling unit 11c. With this configuration, the image processing apparatus 11 can generate a higher quality image after multi-frame synthesis processing.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program for performing multi-frame synthesis processing.

Background Art

[0002] In recent years, as a technology for creating high-quality image data, multi-frame synthesis is widely used in camera shooting, super-resolution of games, etc. When using multi-frame synthesis, even under shooting conditions that are usually difficult (such as low illuminance at night, high contrast, camera shake, etc.), multiple consecutive image data are captured, and they are synthesized to perform processing such as noise processing and camera shake correction. As a result, new low-noise image data can be generated.

[0003] Multi-frame synthesis is basically composed of the steps shown in FIG. 13. (1) First, as shown in 1301 of FIG. 13, a plurality of consecutive image frames are acquired. These consecutive image frames have slightly different exposure, focus, shutter speed, etc. (2) Next, as shown in 1302 of FIG. 13, alignment is performed on these multiple image frames. Usually, the acquired consecutive image frames are slightly misaligned due to camera shake, etc., and these images are accurately overlapped using image recognition technology, etc. (3) Finally, as shown in 1303 and 1304 of FIG. 13, the target information is extracted / synthesized from the aligned images, and a final single high-resolution image 904 is generated. In this multi-frame synthesis technology, while more computational processing is required, it is possible to generate a clearer, less noisy, and wider dynamic range image.

[0004] And, for example, an image processing apparatus capable of suppressing a decrease in resolution perception caused by camera shake correction using continuously captured images is disclosed (see, for example, Patent Document 1). Also, multi-frame interpolation of Bayer images using optical flow is disclosed (see, for example, Non-Patent Document 1).

Prior Art Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2009-76984 [Non-patent literature]

[0006] [Non-Patent Document 1] "Multi-frame interpolation of Bayer images using optical flow" [Accessed December 1, 2025], Internet<URL:https: / / fukushima.web.nitech.ac.jp / paper / 2020_iwait_sato.pdf> [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, conventional multi-frame synthesis, while involving several detailed technical elements, basically involves taking multiple consecutive image frames as input, adjusting their size, and precisely aligning them to combine them into a single high-quality image file.

[0008] Therefore, in at least two processing steps—resizing (enlarging / reducing) image data and moving / adjusting image data in decimal units—it is unavoidable that high-frequency components will be lost from the original image frame. As a result, even if conventional multi-frame composite images are high-resolution, they are still degraded by the amount of high-frequency components lost during these processing steps.

[0009] The present invention has been made in view of the above problems, and also aims to provide an image processing apparatus that can suppress the degradation of the composite image that occurs in two processing processes in multi-frame synthesis, particularly image resizing and image data movement, and generate a higher quality composite image. [Means for solving the problem]

[0010] To achieve the above objective, the present invention provides an image processing apparatus that performs multi-frame synthesis processing using a plurality of consecutive reference image data, comprising: a super-resolution processing unit that expands the resolution of each reference image data to a higher resolution super-resolution image data; an alignment processing unit that aligns the super-resolution image data processed by the super-resolution processing unit; and a synthesis unit that generates a final single composite image data using the super-resolution image data aligned by the alignment processing unit.

[0011] In this image processing apparatus, it is preferable to further include an oversampling unit that performs resizing and compression processing of the super-resolution image data, and the synthesis unit generates a final single synthesized image data from the compressed image data after the compression processing in the oversampling unit.

[0012] In this image processing apparatus, if the reference image data and the composite image data are at the same magnification before and after the multi-frame synthesis process, it is preferable that the super-resolution processing unit enlarges each of the reference image data to a higher resolution super-resolution image data, and the oversampling unit performs compression processing to return the super-resolution image data, which has been aligned by the alignment processing unit, to its original magnification.

[0013] In this image processing device, if the composite image data before and after multi-frame synthesis processing involves an increase in the target magnification of the reference image data, the super-resolution processing unit shall enlarge each of the reference image data to a higher resolution super-resolution image data with a magnification equal to or greater than the target magnification. ,before The oversampling unit preferably performs a compression process to return the super-resolution image data, which has been aligned by the alignment processing unit, back to the target magnification.

[0014] Furthermore, in order to achieve the above objective, the present invention provides an image processing apparatus that performs multi-frame synthesis processing using a plurality of consecutive reference image data, comprising: an enlargement processing unit that performs an enlargement processing by an integer multiple without interpolation for each pixel of the reference image data; an alignment processing unit that aligns the plurality of enlarged image data processed by the enlargement processing unit; and a synthesis unit that generates a final single composite image data from the plurality of enlarged image data after alignment by the alignment processing unit.

[0015] Furthermore, in order to achieve the above objective, the present invention provides an image processing method for performing multi-frame synthesis processing using a plurality of consecutive reference image data, characterized by comprising: a super-resolution processing step of increasing the resolution of each reference image data to a higher resolution super-resolution image data; an alignment processing step of aligning the super-resolution image data processed in the super-resolution processing step; and a synthesis step of generating a final single composite image data using the super-resolution image data aligned in the alignment processing step.

[0016] Furthermore, in order to achieve the above objective, the present invention provides an image processing method for performing multi-frame synthesis processing using a plurality of consecutive reference image data, characterized in that it includes: an enlargement processing step of enlarging each pixel of the reference image data by an integer multiple without interpolation; an alignment processing step of aligning the plurality of enlarged image data obtained in the enlargement processing step; and a synthesis step of generating a final single composite image data from the plurality of enlarged image data obtained after alignment in the alignment processing step.

[0017] Furthermore, in order to achieve the above objective, the present invention provides a program that is executed in an image processing device that performs multi-frame synthesis processing using a plurality of consecutive reference image data, and is characterized by including: a super-resolution processing step of increasing the resolution of each reference image data to a higher resolution super-resolution image data; a positioning processing step of aligning the super-resolution image data processed in the super-resolution processing step; and a synthesis step of generating a final single composite image data using the super-resolution image data after positioning in the positioning processing step.

[0018] Furthermore, in order to achieve the above objective, the present invention is a program executed by an image processing device that performs multi-frame synthesis processing using a plurality of consecutive reference image data, characterized in that it includes: an enlargement processing step that performs an enlargement processing by an integer multiple without interpolation for each pixel of the reference image data; an alignment processing step that aligns the plurality of enlarged image data obtained in the enlargement processing step; and a synthesis step that generates a final single composite image data from the plurality of enlarged image data obtained after alignment in the alignment processing step. [Effects of the Invention]

[0019] The image processing apparatus according to the present invention includes a super-resolution processing unit that enlarges the resolution of reference image data to higher-resolution super-resolution image data for each piece of the reference image data, an alignment processing unit that aligns the super-resolution image data super-resolved by the super-resolution processing unit, and a composition unit that generates final one-piece composite image data using the super-resolution image data after alignment in the alignment processing unit. Further, an oversampling unit that performs resizing and compression processing of the super-resolution image data is provided, and the composition unit generates final one-piece composite image data from the compressed image data after the compression processing in the oversampling unit. With this configuration, in multi-frame composition processing, particularly in two processing steps of image resizing and image data movement, it is possible to provide an image processing apparatus that can further suppress degradation of the composite image and generate a higher-quality composite image.

Brief Description of the Drawings

[0020] [Figure 1] It is a block diagram showing the configuration of a terminal device including the image processing apparatus according to an embodiment of the present invention. [Figure 2] It is a functional block diagram of the image processing apparatus described above. [Figure 3] It is an explanatory diagram showing an example of multi-frame composition performed in the image processing apparatus described above. [Figure 4] It is an explanatory diagram showing another example of multi-frame composition performed in the image processing apparatus described above. [Figure 5] It is an explanatory diagram showing another example of multi-frame composition performed in the image processing apparatus described above. [Figure 6] It is a functional block diagram of the image processing apparatus according to Modification 1 of the embodiment of the present invention. [Figure 7] It is an explanatory diagram showing an example of multi-frame composition performed in the image processing apparatus according to Modification 1 described above. [Figure 8] It is an explanatory diagram showing another example of multi-frame composition performed in the image processing apparatus according to Modification 1 described above. [Figure 9]This is an explanatory diagram showing an example of multi-frame synthesis in the prior art relating to the modified example 2 described above. [Figure 10] This is an explanatory diagram showing an example of multi-frame synthesis performed in the image processing device according to the modified example 2 described above. [Figure 11] This is an explanatory diagram showing an example of multi-frame synthesis in the prior art relating to the above modified example 3. [Figure 12] This is an explanatory diagram showing an example of multi-frame synthesis performed in the image processing device according to the modified example 3 described above. [Figure 13] This diagram illustrates the conventional procedure for multi-frame synthesis. [Modes for carrying out the invention]

[0021] (Embodiment) An image processing apparatus according to an embodiment of the present invention will be described with reference to Figures 1 to 5.

[0022] First, the main processing unit of the terminal device 1 equipped with the image processing device according to this embodiment will be described with reference to Figure 1. This terminal device 1 includes mobile terminals such as smartphones and mobile phones with imaging functions, imaging devices such as digital cameras, and information processing devices such as computers that perform information processing such as games.

[0023] As shown in Figure 1, terminal device 1 comprises a control unit 10, an image processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, a reading unit 16, an imaging unit 17, and an imaging condition processing unit 18. Although this figure shows terminal device 1 with a camera function, the imaging unit 17 and the imaging condition processing unit 18 are not necessary if the terminal device is, for example, a game console.

[0024] The control unit 10 uses a processor such as a CPU and memory to control the components of the terminal device 1 and realize various functions.

[0025] The image processing device 11 generates higher-quality digital image data by performing multi-frame synthesis processing on the continuous digital image data captured by the imaging unit 17. Here, the image processing device 11 uses a processor such as a GPU or dedicated circuit and memory to execute an image processing program for multi-frame synthesis in response to control instructions from the control unit 10. Note that the control unit 10 and the image processing device 11 may be configured as a single hardware (SoC: System on a Chip) integrating a processor such as a CPU or GPU, memory, and also a storage unit 12 and a communication unit 13.

[0026] Multi-frame synthesis processing is used for super-resolution, denoising, and frame generation of image data. For example, when used for image stabilization in a smartphone camera, it involves taking multiple consecutive shots at a high shutter speed and combining them to create new image data that is less blurry but noisier.

[0027] The memory unit 12 uses a hard disk or flash memory. The memory unit 12 stores an image processing program 1P and a machine learning library 1L that enables the machine learning model (e.g., a CNN (Convolutional Neural Network)) to function. The memory unit 12 also stores definition data that defines the machine learning model, parameters including settings for the trained machine learning model, and so on.

[0028] The communication unit 13 is a communication module that enables communication connection to a communication network such as the Internet. The communication unit 13 uses a network card, a wireless communication device, or a carrier communication module.

[0029] The display unit 14 uses a liquid crystal panel or an organic EL (Electro-Luminescence) display, etc. The display unit 14 can display an image through processing by the image processing unit 11 as instructed by the control unit 10.

[0030] The operation unit 15 includes a user interface such as a keyboard or mouse that receives user operations such as shutter clicks when imaging landscapes or objects. Physical buttons provided on the housing may be used, or if the terminal device 1 is a smartphone, software buttons displayed on the display unit 14 may be used. The operation unit 15 notifies the control unit 10 of the user's operation information.

[0031] The reading unit 16 can read the image processing program 2P and the machine learning library 3L stored on the recording medium 2, such as an optical disc, using, for example, a disk drive. The image processing program 1P and the machine learning library 1L stored in the storage unit 12 may be copies of the image processing program 2P and the machine learning library 3L read by the reading unit 16 from the recording medium 2, copied to the storage unit 12 by the control unit 10, or acquired via the communication unit 13.

[0032] The imaging unit 17 is a component that converts light into electrical signals and has an image sensor made of, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor). It receives incident light, converts it into an electrical signal, and constructs digital image data.

[0033] The frame buffer 17a is a working memory that stores reference image data captured by the imaging unit 17 and used for multi-frame synthesis processing. For example, if the terminal device 1 is a smartphone, raw frames are continuously recorded in the frame buffer 17a while the user is operating the camera application. When the shutter, which is the operation unit 15, is pressed, the most recently recorded consecutive frames are sent to the multi-frame synthesis pipeline side (i.e., the image processing device 11 side) as reference image data.

[0034] The shooting condition control unit 18 determines and controls the imaging conditions in the imaging unit 17, such as the F-number (aperture), shutter speed, ISO sensitivity, frame rate, and focus, based on user input, for example.

[0035] Next, the functions of the image processing device 11 will be described with reference to Figure 2. The image processing device 11 according to this embodiment includes a super-resolution processing unit 11a, a positioning processing unit 11b, an oversampling unit 11c, and a synthesis unit 11d.

[0036] First, when the shutter (operation unit 15) is pressed, the most recent, related, consecutive reference image data RI stored in the frame buffer 17a is input to the super-resolution processing unit 11a. Here, the reference image data RI consists of multiple consecutive digital image data frames that are related to each other and set according to the shooting scene. These reference image data frames are captured, for example, in high-performance HDR imaging or night scene mode, and have slightly different exposures, frame rates, and focus.

[0037] The super-resolution processing unit 11a expands the resolution of each reference image data RI (each single frame) to higher resolution image data (super-resolution image data) based on the trained machine learning model (super-resolution model) 1L stored in the memory unit 12. In other words, the super-resolution processing unit 11a in this embodiment performs single-frame super-resolution using individual image data.

[0038] Although a detailed description of the learning process of the machine learning model (super-resolution model) 1L in the super-resolution processing unit 11a is omitted here, in recent years, learning-based deep learning, which uses a multi-layered neural network, has demonstrated high recognition accuracy, particularly in the field of image recognition. Even in multi-layered deep learning, image processing is performed using a convolutional neural network (CNN) that uses multiple convolutional and pooling layers to extract input features. For example, in supervised learning, the neural network is trained using a large number of pairs of low-resolution images and their corresponding true high-resolution images. The loss function of the machine learning model (super-resolution model) is designed to minimize the difference between the generated high-resolution image and the true high-resolution image.

[0039] The alignment processing unit 11b aligns multiple super-resolution image data processed by the super-resolution processing unit 11a to achieve a more accurate position. Normally, the continuous reference image data input to the image processing unit 11 is slightly misaligned due to camera shake or other factors. The alignment processing unit 11 uses image recognition technology to perform alignment in order to accurately superimpose multiple super-resolution image data processed by the super-resolution processing unit 11a.

[0040] Specifically, the alignment processing unit 11b does not use only integer coordinates in pixel units, but instead uses existing decimal coordinate pixel values, which express the position between pixels more precisely with decimal values, to perform interpolation processing to estimate the value of new data (decimal coordinate pixel values), thereby moving the pixels to a more accurate position and performing alignment. Note that this alignment method can be implemented simply by adding software when the terminal device 1 is a smartphone, thus improving accuracy while keeping costs down, but it places a load on processing speed, computing resources, and memory capacity.

[0041] The oversampling unit 11c is a processing unit that performs compression processing for resizing image data, and is, for example, a low-pass filter. Generally, if the reference image data RI is directly aligned using decimal coordinates after oversampling, blurring occurs. On the other hand, as in the image processing device 11 according to this embodiment, if the super-resolution image generated by the super-resolution processing unit 11a is aligned by the alignment processing unit 11b, and then oversampled by the oversampling unit 11c, there is an advantage that blurring hardly occurs. As a result, noise is suppressed and color reproduction is stabilized, resulting in a higher quality composite image.

[0042] Regarding the processing procedures of the alignment processing unit 11b and the oversampling unit 11c, (a) the alignment of the super-resolution image data is performed in the alignment processing unit 11b and then resized (reduced) in the oversampling unit 11c, or (b) the resized (reduced) of the super-resolution image data is performed in the oversampling unit 11c and then the alignment is performed in the alignment processing unit 11b. In case (b), since the reduced image is used for alignment in the alignment processing unit 11b, the accuracy of the final composite image data will be lower than in case (a).

[0043] The synthesis unit 11d extracts and synthesizes the desired information from multiple super-resolution image data that have been aligned in the alignment processing unit 11b, and generates a final single composite image data. For example, the pixel values ​​of each aligned super-resolution image are weighted averaged according to their contribution, and the pixel-level weights for all frames are calculated. The images are then moved pixel by pixel at the target image location to align and synthesize them. As a result, a higher resolution and more natural image can be synthesized. Furthermore, if the super-resolution image data is compressed in the oversampling unit 11c, the synthesis unit 11d extracts and synthesizes the desired information from multiple compressed image data after oversampling, and generates a final single composite image data.

[0044] Next, a specific example of multi-frame synthesis in the image processing device 11 according to this embodiment will be described with reference to Figures 3 to 5.

[0045] Figure 3 shows an example of the multi-frame synthesis process of the image processing device 11, where the target magnification of the synthesized image data generated after multi-frame synthesis is the same as the input reference image data, and the input reference image data is subjected to single-frame super-resolution to twice its original size, then aligned, and finally returned to the original size.

[0046] Specifically, as shown in Figure 3(a), consecutive reference image data 301a to 301d are input to the super-resolution processing unit 11a. Note that this figure illustrates the case where there are four reference image data 301a to 301d, but this is merely an example, and the number of images is not limited to this.

[0047] Then, as shown in Figure 3(b), the super-resolution processing unit 11a generates super-resolution image data 302a to 302d for each single frame (reference image data) by enlarging the reference image data 301a to 301d by a factor of two at a higher resolution, based on the super-resolution model. Next, as shown in Figure 3(c), the alignment processing unit 11b aligns the super-resolution image data 302a to 302d, moving the pixels to their more precise positions. Then, as shown in Figure 3(d), the oversampling unit 11c resizes the aligned image data 303 to the same size as the reference image data 301a to 301d, and finally, as shown in Figure 3(e), the synthesis unit 11d generates a single, higher-precision synthesized image data 305.

[0048] Figure 4 shows an example in the multi-frame synthesis process of the image processing device 11, where the target magnification of the synthesized image data generated after multi-frame synthesis is twice that of the input reference image data, and the input reference image data is subjected to single-frame super-resolution to twice its original size before alignment.

[0049] Specifically, as shown in Figure 4(a), consecutive reference image data 401a to 401d are input to the super-resolution processing unit 11a. Then, as shown in Figure 4(b), the super-resolution processing unit 11a generates super-resolution image data 402a to 402d, which are obtained by doubling the resolution of the reference image data 401a to 401d based on the super-resolution model, by performing super-resolution on each single frame (reference image data). Then, as shown in Figure 4(c), the alignment processing unit 11b aligns the super-resolution image data 402a to 402d, moving the pixels to their positions more accurately. Finally, as shown in Figure 4(d), the synthesis unit 11d generates a single, higher-precision synthesized image data 404. Note that in the case of the multi-frame synthesis process shown in this figure, degradation due to the enlargement of the input reference image data is suppressed, but degradation due to the movement of decimal coordinates during alignment remains.

[0050] Figure 5 shows an example in the multi-frame synthesis process of the image processing device 11, where the target magnification of the synthesized image data generated after multi-frame synthesis is twice the input reference image data (target magnification), the reference image data is subjected to single-frame super-resolution to four times (target magnification or higher), then aligned, and then reduced to half size (i.e., returned to the target magnification).

[0051] Specifically, as shown in Figure 5(a), consecutive reference image data 501a to 501d are input to the super-resolution processing unit 11a. Then, as shown in Figure 5(b), the super-resolution processing unit 11a generates super-resolution image data 502a to 502d by performing super-resolution on each single frame (reference image data) based on the super-resolution model, which is a higher resolution version of the reference image data 501a to 501d, magnified four times (magnified beyond the target magnification). Next, as shown in Figure 5(c), the alignment processing unit 11b aligns the super-resolution image data 502a to 502d, moving the pixels to their more precise positions. Then, as shown in Figure 5(d), the oversampling unit 11c resizes the aligned image data 503, compressing it to twice the size of the reference image data 501a to 501d, i.e., half the size. Finally, as shown in Figure 5(e), the synthesis unit 11d generates a single, higher-precision synthesized image data 505 at the final target magnification (2x). Furthermore, in the case of multi-frame synthesis shown in this figure, degradation due to enlargement and degradation due to alignment are suppressed, and high-quality synthesized image data can be generated.

[0052] As described above, the present invention is an image processing apparatus 11 that performs multi-frame synthesis processing using a plurality of consecutive reference image data, comprising: a super-resolution processing unit 11a that expands the resolution of each reference image data RI to a higher resolution super-resolution image data; an alignment processing unit 11b that aligns the plurality of super-resolution image data processed by the super-resolution processing unit 11a; and a synthesis unit 11d that generates a final single synthesized image data from the plurality of super-resolution image data after alignment by the alignment processing unit 11b. Furthermore, it comprises an oversampling unit 11c that performs resizing and compression processing of the super-resolution image data, and the synthesis unit 11d generates a final single synthesized image data from the compressed image data after compression processing by the oversampling unit 11c.

[0053] With this configuration, the image processing device 11 can further suppress the degradation of the composite image that occurs in two processing steps in multi-frame synthesis, particularly image resizing and image data movement, and generate a higher quality composite image.

[0054] Here, if the reference image data and the composite image data are at the same size before and after the multi-frame synthesis process, the super-resolution processing unit 11a can enlarge the reference image data using single-frame super-resolution, and then oversample it during alignment to suppress blurring caused by the movement of fractional pixels.

[0055] On the other hand, when the composite image data before and after multi-frame synthesis processing involves an enlargement of the target magnification of the reference image data, single-frame super-resolution up to the target magnification of multi-frame synthesis suppresses blurring due to enlargement, and single-frame super-resolution at magnifications exceeding the target magnification can further suppress blurring due to the movement of fractional pixels. Furthermore, super-resolution data exceeding the target magnification can be returned to the target magnification in the oversampling unit 11c to prevent blurring.

[0056] (Variation 1) Next, a modification 1 of the image processing apparatus 11 according to an embodiment of the present invention will be described with reference to Figures 6 to 8. In the image processing apparatus 11 according to this modification 1, as shown in Figure 6, an enlargement processing unit 11e is provided instead of a super-resolution processing unit 11a. This enlargement processing unit 11e simply performs a process of enlarging each pixel of the reference image data by an integer multiple without interpolation. Furthermore, the image processing apparatus 11 according to this modification 1 may also include a composite image reduction unit 11f that further reduces the generated composite image data.

[0057] Next, a specific example of multi-frame synthesis in the image processing device 11 according to this modified example 1 will be explained with reference to Figures 7 and 8.

[0058] Figure 7 shows an example in the multi-frame synthesis process of the image processing device 11, where the target magnification of the synthesized image data generated after multi-frame synthesis is twice that of the input reference image data, and the input reference image data is enlarged to twice its original size without interpolation before alignment.

[0059] Specifically, as shown in Figure 7(a), the magnification processing unit 11e generates magnified image data 702a to 702d for each single frame (reference image data) by magnifying the consecutive reference image data 701a to 701d by a factor of two, pixel by pixel, without interpolation. Then, as shown in Figure 7(b), the alignment processing unit 11b performs alignment of the magnified image data 702a to 702d, including the movement of a fraction of a pixel, to move the pixels to their positions more accurately. Finally, as shown in Figure 7(c), the synthesis unit 11d generates a single, higher-precision synthesized image data 703.

[0060] Figure 8 shows an example in the multi-frame synthesis process of the image processing device 11 according to this modified example 1, in which the target magnification of the synthesized image data generated after multi-frame synthesis is the same as the input reference image data, and the input reference image data is enlarged to 2 times without interpolation, then aligned, and then reduced to 1 / 2 times by the image reduction unit 11f.

[0061] Specifically, as shown in Figure 8(a), the magnification processing unit 11e generates magnified image data 802a to 802d for each single frame (reference image data) by magnifying each pixel of the reference image data 801a to 801d by a factor of 2 without interpolation. Then, as shown in Figure 8(b), the alignment processing unit 11b performs alignment of the magnified image data 802a to 802d, including the movement of a fraction of a pixel, to move the pixels to their positions more accurately. Finally, as shown in Figures 8(c) and (d), the synthesis unit 11d generates a single, higher-precision synthesized image data 803, which is further reduced by half in the image reduction unit 11f to generate the final synthesized image data 804. In this case, the multi-frame synthesis shown in Figure 7 is the same process as the case with a 2x synthesis target up to a certain point (up to Figure 8(c)), but by further reducing it by half at the end, the quality of the final synthesized image can be improved compared to synthesizing with a 1:1 target without generating magnified images from the beginning.

[0062] As shown in Modification 1, when performing interpolation-free scaling of consecutive reference image data, blurring associated with the scaling of the reference image data does not occur. However, since the scaling processing unit 11e in Modification 1 simply performs image scaling, noise is superimposed on the high-frequency side of the portion exceeding the resolution of the original reference image data. This noise can be eliminated to some extent by the overall processing of multi-frame synthesis.

[0063] (Modification 2) Next, a modified example 2 of the multi-frame synthesis in the image processing device 11, in which an enlarged frame is generated in real time from consecutive frames, will be described with reference to Figures 9 and 10. For example, in games and the like, when frames are generated in real time, multiple frames may be processed by repeatedly synthesizing only the previous frame and the current frame. In that case, the super-resolution applied in the processing according to the present invention is only applied to the previous frame.

[0064] As a conventional example, as shown in Figure 9(a), the "previous frame (901)" enlarged by multi-frame compositing and the "current frame (902)" before processing (enlargement) are combined to generate the "processed current frame (903)" enlarged by multi-frame compositing.

[0065] In this example, first, as shown in Figure 9(b), the "previous frame (901)" is aligned with the "current frame (902)" using a separately input motion vector to generate the "previous frame aligned past frame (901a)". Then, as shown in Figure 9(c), this "aligned past frame (901a)" and the "current frame (902)" are combined to generate the "processed current frame (903)". More precisely, the "current frame (902)", the "previous frame aligned past frame (901a)", and other information are input together into the neural network to generate an enlarged "processed current frame (903)". In the subsequent processing in this example, the enlarged "processed current frame (903)" (i.e., synonymous with "previous frame (901)") and the next "current frame (902)" are combined in the same way, and this recursive synthesis is repeated.

[0066] Figure 10 shows an example of applying the present invention to the example in Figure 9. First, as shown in Figure 10(a), a single-frame super-resolution process is applied to the enlarged "previous frame (1001)" to generate the "previous frame with super-resolution enlargement (1001a)". Next, using a separately input motion vector, only the "previous frame with super-resolution enlargement (1001a)" is aligned with the "current frame (1002)" to generate the "previous frame with super-resolution enlargement and alignment (1001b)". Finally, this is reduced to its original size to generate the "previous frame with alignment and reduced to original size (1001c)".

[0067] Then, as shown in Figure 10(b), the "previous frame (1001c) that has been scaled down to its original size and aligned to the previous frame" and the "current frame (1002)" are combined to generate the "processed current frame (1003)". In this case, since alignment is only performed on the "previous frame (1001)", the process involves applying super-resolution processing to this frame, then aligning and scaling it down.

[0068] (Variation 3) Next, a third modified example of anti-aliasing processing by multi-frame synthesis in the image processing device 11 will be described with reference to Figures 11 and 12.

[0069] A conventional example is shown in Figure 11, where the "previous frame (1101)" and the "current frame (1102)" before processing (enlargement) are aligned and combined to generate the "processed current frame (1103)". Essentially, denoising of the same frame size is performed. In the case of anti-aliasing, the "previous frame (1101)" is aligned using motion vectors, and the "current frame (1102)" is aligned to correct any positional discrepancies.

[0070] Figure 12 shows an example of applying the present invention to the example in Figure 11. First, the "enlarged past frame (1201) from the previous frame" and the "current frame (1202)" before processing (enlargement) are used to prepare the "enlarged current frame (1202a)" by enlarging the "current frame (1202)" without super-resolution or integer interpolation. Then, using a separately input motion vector, the "enlarged past frame (1201) from the previous frame" and the "enlarged current frame (1202a)" are aligned to generate the "enlarged current frame after alignment and synthesis (1202b)". Finally, this is reduced to its original size to generate the "current frame after alignment and synthesis (1202c)".

[0071] As in this example, by combining the data in an enlarged state during the compositing process and then returning it to its original size after compositing, it is possible to use the data of the previous frame before returning it to its original size, thereby enlarging only the "current frame 1202" before alignment and compositing. In this way, as with other embodiments and modifications, alignment can be done at a larger size, so alignment blurring does not occur, and the enlargement process only needs to be performed on the "current frame 1202". After the compositing process, the composite data before reduction (i.e., the "enlarged current frame after alignment and compositing (1202b)") can be carried over as the "enlarged past frame from the previous frame (1201)", and the compositing with the next current frame can be repeated.

[0072] Alternatively, the image processing device 11 may be configured to function as a web server, providing the above-described machine learning model functionality to a web client device equipped with a display unit and a communication unit. In this case, the communication unit 13 is used to receive requests from the web client device and transmit processing results.

[0073] Furthermore, the present invention is not limited to the configuration of the above embodiment, and various modifications are possible without changing the spirit of the invention. In addition, in order to achieve the objective of the present invention, the present invention can be implemented as an image processing method in which characteristic components included in the image processing apparatus are used as steps, or as a program that includes these characteristic steps. The program can be stored in ROM or the like, or distributed via a recording medium such as a USB memory or a communication network.

[0074] Furthermore, the present invention can also be realized as a computer system that transmits input data to an image processing device or computer program and receives and utilizes output data from the image processing device or computer program. This system is a processing system that utilizes data obtained from a machine learning model trained through the above-described process and can provide various services. The devices used in this system are image processing devices equipped with a display unit and a communication unit, or information processing devices that can send and receive information with a computer, such as so-called PCs, smartphones, mobile terminals, and game devices. [Explanation of Symbols]

[0075] 11 Image Processing Device 11a Super-resolution processing unit 11b Alignment processing unit 11c Oversampling section 11d Synthesis section 11e Enlarged Processing Unit

Claims

1. An image processing device that performs multi-frame synthesis processing using multiple consecutive reference image data, A super-resolution processing unit that expands the resolution of the aforementioned reference image data to a higher resolution super-resolution image data for each of the aforementioned reference image data, The super-resolution processing unit includes an alignment processing unit that performs alignment of the super-resolution image data processed by the super-resolution processing unit, An image processing apparatus characterized by comprising: a synthesis unit that generates a final single composite image data using the super-resolution image data after alignment in the alignment processing unit.

2. The aforementioned image processing device further, The super-resolution image data is provided with an oversampling unit that performs resizing and compression processing, The image processing apparatus according to claim 1, characterized in that the synthesis unit generates a final single synthesized image from the compressed image data after compression processing in the oversampling unit.

3. If the reference image data and the composite image data are at the same size before and after the multi-frame compositing process, The super-resolution processing unit enlarges each of the reference image data to a higher resolution super-resolution image data, The image processing apparatus according to claim 2, characterized in that the oversampling unit performs compression processing to return the super-resolution image data aligned by the alignment processing unit back to its original magnification.

4. If the composite image data before and after the multi-frame compositing process involves an increase in the target magnification of the reference image data, The super-resolution processing unit enlarges each of the reference image data to a higher resolution super-resolution image data with a magnification equal to or greater than the target magnification. The image processing apparatus according to claim 1, characterized in that the oversampling unit performs compression processing to return the super-resolution image data aligned by the alignment processing unit back to the target magnification.

5. An image processing device that performs multi-frame synthesis processing using multiple consecutive reference image data, The aforementioned reference image data includes an enlargement processing unit that performs an integer multiplier on each pixel without interpolation, and The aforementioned magnification processing unit includes a positioning processing unit that aligns the positions of a plurality of magnified image data that have been magnified, An image processing apparatus characterized by comprising a synthesis unit that generates a final single composite image from a plurality of enlarged image data after alignment in the alignment processing unit.

6. An image processing method that performs multi-frame synthesis processing using multiple consecutive reference image data, A super-resolution processing step that expands the resolution of the aforementioned reference image data to a higher resolution super-resolution image data for each of the said reference image data, The super-resolution processing step includes a positioning process that performs positioning of the super-resolution image data processed in the super-resolution processing step, An image processing method characterized by comprising a synthesis step of generating a final single composite image data using the super-resolution image data after alignment in the alignment processing step.

7. An image processing method that performs multi-frame synthesis processing using multiple consecutive reference image data, The process involves scaling each pixel of the aforementioned reference image data by an integer multiple without interpolation, and The aforementioned enlargement processing step includes an alignment processing step that aligns the positions of a plurality of enlarged image data that have been enlarged in the enlargement processing step, An image processing method characterized by including a synthesis step of generating a final single composite image from a plurality of enlarged image data after alignment in the alignment processing step.

8. A program executed by an image processing device that performs multi-frame synthesis processing using multiple consecutive reference image data, A super-resolution processing step that expands the resolution of the aforementioned reference image data to a higher resolution super-resolution image data for each of the said reference image data, The super-resolution processing step includes a positioning process that performs positioning of the super-resolution image data processed in the super-resolution processing step, A program characterized by including a synthesis step that generates a final single composite image using the super-resolution image data after alignment in the alignment processing step.

9. A program executed by an image processing device that performs multi-frame synthesis processing using multiple consecutive reference image data, The process involves scaling each pixel of the aforementioned reference image data by an integer multiple without interpolation, and The aforementioned enlargement processing step includes an alignment processing step that aligns the positions of a plurality of enlarged image data that have been enlarged in the enlargement processing step, A program characterized by including a synthesis step of generating a final single composite image from a plurality of enlarged image data after alignment in the alignment processing step.

Images