Training and upscaling large size images

By dividing large images into sub-pictures, padding, and upsampling, the method addresses memory challenges in ML-based upscaling, enabling efficient processing of large images within practical memory constraints.

JP7596455B2Active Publication Date: 2024-12-09SONY GROUP CORP +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023109105
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-06-14
Filing Date
2023-07-03
Publication Date
2024-12-09
Estimated Expiration
2039-10-17

AI Technical Summary

Technical Problem

Traditional ML-based image upscaling processes face memory size challenges as target image size increases, especially in deep neural networks, requiring a method independent of input/output image size to keep memory within practical limits.

Method used

The method involves dividing large-sized input images into small-sized sub-pictures, enlarging each sub-picture with targeted padding, upsampling, cropping, and concatenating the sub-pictures to generate an output image, using ML-based upscaling techniques.

Benefits of technology

This approach reduces memory requirements by processing images based on sub-picture size and neural network complexity, rather than the entire image size, effectively handling large images without exceeding practical memory limits.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007596455000001
    Figure 0007596455000001
  • Figure 0007596455000002
    Figure 0007596455000002
  • Figure 0007596455000003
    Figure 0007596455000003
Patent Text Reader

Abstract

To provide methods and systems that maintain a required memory size within a practical range even when an objective image size is large when performing super-resolution upscaling of an image using machine learning (ML).SOLUTION: The method includes: dividing a larger-sized input image into multiple smaller-sized subpictures; enlarging each subpicture of the multiple smaller-sized subpictures using objective padding pixels; upscaling each subpicture using an ML-based upscaler; trimming the enlarged and upscaled subpictures to upscaling sizes equal to an original size of each subpicture multiplied by an upscaling factor; repeating enlargement, upscaling, and trimming for the multiple subpictures; and concatenating the multiple trimmed subpictures to generate an output image.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. §119(e) of co-pending U.S. Provisional Patent Application No. 62 / 747,453, entitled "Machine-Learning Based Single Image Upscaling Application for Motion Pictures," filed October 18, 2018. The disclosures of the above related applications are incorporated herein by reference.

[0002]

[0002] The present disclosure relates to training and upscaling images, and more particularly, to training and upscaling large size input images. [Background technology]

[0003]

[0003] Machine learning (ML) can be used to perform super-resolution upscaling of images. However, conventional ML-based training and upscaling processes have some problems. For example, the memory size required for ML-based training and upscaling increases exponentially as the target image size increases, especially in deep neural networks with many layers. In order to keep the required memory size within a practical range while using advanced neural network-based training and upscaling processes, a method that is independent of input / output image size is desired. Summary of the Invention [Problem to be solved by the invention]

[0004]

[0004] The present disclosure provides for training and upscaling large size input images. [Means for solving the problem]

[0005]

[0005] In one implementation, a method for training and upscaling a large-sized input image is disclosed, which includes: dividing the large-sized input image into a plurality of smaller-sized sub-pictures, upscaling each of the plurality of smaller-sized sub-pictures using target padding pixels to generate an upscaled sub-picture, upscaling each sub-picture using an ML-based upscaler to generate an upscaled and upscaled sub-picture, cropping the upscaled and upscaled sub-picture to an upscaling size equal to an original size of each sub-picture multiplied by an upscaling factor, repeating the upscaling, upscaling, and cropping for the plurality of sub-pictures, and concatenating the plurality of cropped sub-pictures to generate an output image.

[0006]

[0006] In one implementation, expanding each subpicture includes determining how to pad extra pixels around each subpicture and adding a small region of the extra pixels around each subpicture. In one implementation, adding a small region includes determining a type for each of the small regions. In one implementation, a first type of the small regions includes the small regions having pixels that are entirely within an active picture area of ​​the large size image. In one implementation, the method further includes expanding each subpicture in the direction of the small region when the small region is determined to be of the first type. In one implementation, a second type of the small regions includes the small regions having pixels that are not within an active picture area of ​​the large size image but are on an edge of each subpicture. In one implementation, the method further includes expanding each subpicture using a butterfly image on the edge when the small region is determined to be of the second type. In one implementation, a third type of the sub-region includes the sub-region having pixels that are not within an active picture area of ​​the larger size image but are on a corner of each sub-picture. In one implementation, the method further includes expanding each sub-picture using a butterfly image on the corner when the sub-region is determined to be of the third type.

[0007]

[0007] In another implementation, a large size image upscaling system is disclosed, which includes an image divider configured to divide a large size input image into a plurality of sub-pictures, a sub-picture expansion unit configured to expand each sub-picture of the plurality of sub-pictures by adding extra pixels around each sub-picture to generate an expanded sub-picture, and send the expanded sub-pictures to a neural network for training and upscaling the expanded sub-pictures, a sub-picture cropping unit configured to receive the expanded and upscaled sub-pictures and crop them to a size corresponding to a size of the original sub-picture multiplied by an upscaling factor, and an image filling unit configured to concatenate all the cropped sub-pictures of the plurality of sub-pictures to reconstruct an output image.

[0008]

[0008] In one implementation, the subpicture expansion unit adds the extra pixels around each subpicture by adding a small region of the extra pixels around each subpicture. In one implementation, adding the small region includes determining a type for each of the small regions. In one implementation, a first type of the small regions includes the small regions having pixels that lie entirely within an active picture area of ​​the large size input image. In one implementation, the subpicture expansion unit is further configured to expand each subpicture in the direction of the small region when the small region is determined to be of the first type. In one implementation, a second type of the small regions includes the small regions having pixels that lie on an edge of each expanded subpicture rather than within an active picture area of ​​the large size input image. In one implementation, the subpicture expansion unit is further configured to expand each subpicture using a butterfly image on the edge when the small region is determined to be of the second type. In one implementation, a third type of the small regions includes the small regions having pixels that are not within an active picture area of ​​the large size input image but are on a corner of each expanded sub-picture, and in one implementation, the sub-picture expansion unit is further configured to expand each sub-picture using a butterfly image on the corner when the small region is determined to be of the third type.

[0009]

[0009] In yet another implementation, a non-transitory computer-readable storage medium storing a computer program for training and upscaling a large-sized input image is disclosed, the computer program including executable instructions for causing a computer to split the large-sized input image into a plurality of smaller-sized sub-pictures, upscale each of the plurality of smaller-sized sub-pictures using target padding pixels to generate an upscaled sub-picture, upscale each sub-picture using an ML-based upscaler to generate an upscaled and upscaled sub-picture, crop the upscaled and upscaled sub-picture to an upscaled size equal to an original size of each sub-picture multiplied by an upscaling factor, repeat the executable instructions for causing the computer to upscale, upscale, and crop the plurality of sub-pictures, and concatenate the plurality of cropped sub-pictures to generate an output image.

[0010]

[0010] In one implementation, the executable instructions that cause a computer to enlarge each subpicture include executable instructions that cause the computer to determine how to pad extra pixels around each subpicture and to add a small area of ​​the extra pixels.

[0011]

[0011] Other features and advantages will become apparent from the present specification, which illustrates by way of example embodiments of the present disclosure.

[0012]

[0012] Details of the present disclosure, both as to its structure and operation, can be gleaned in part by studying the accompanying drawings, in which like parts are designated with like reference numerals and in which: [Brief description of the drawings]

[0013] [Figure 1A]FIG. 2 illustrates a large-sized input image that is divided into six smaller-sized sub-pictures. [Figure 1B] FIG. 2 illustrates the processing of sub-pictures and the formation of an output image from the sub-pictures. [Diagram 2] FIG. 1C is a detailed flow diagram illustrating the expansion process (described with respect to FIG. 1B) according to one implementation of the present disclosure. [Diagram 3] 1 is a flowchart illustrating a process for training and upscaling large size images according to one implementation of the present disclosure. [Figure 4] 1 is a detailed flowchart illustrating a padding process according to one implementation of the present disclosure. [Diagram 5] 1 is a detailed flowchart illustrating a trimming process according to one implementation of the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating a large size image upscaling system according to one implementation of the present disclosure. [Figure 7A] 1 is a diagram of a large size image upscaling system and a user according to one implementation of the present disclosure. [Figure 7B] FIG. 2 is a functional block diagram illustrating a computer system hosting a large size image upscaling application according to one implementation of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014]

[0022] As described above, conventional ML-based training and upscaling processes have some problems. For example, as the target image size increases, especially in deep neural networks with many layers, the memory size required for ML-based training and upscaling increases exponentially. In order to keep the required memory size within a practical range while using advanced neural network-based training and upscaling processes, a method that is independent of input / output image size is desired.

[0015]

[0023] To address the above problems of the conventional process, some implementations of the present disclosure provide for (1) splitting a large-sized input image into multiple smaller-sized sub-pictures; (2) stretching each sub-picture by target padding pixels around the sub-picture; (3) upscaling the stretched sub-pictures; (4) cropping image areas from the stretched and upscaled sub-pictures; and (5) concatenating the cropped sub-picture images to reconstruct an upscaled output image.

[0016]

[0024] After reading these descriptions, it will become apparent how to implement the present disclosure in various implementations and applications. Although various implementations of the present disclosure are described herein, it should be understood that these implementations are presented by way of example only and not by way of limitation. Thus, detailed descriptions of various implementations should not be construed as limiting the scope or breadth of the present disclosure.

[0017]

[0025] 1A and 1B form a flow diagram showing a process for training and upscaling large size images according to one implementation of the present disclosure. In one implementation, large size images include high definition (HD) and higher images. For example, HD images can be defined as having a resolution of 1920x1080 pixels, while higher resolution images such as 4K can be defined as having a resolution of 3840x2160 pixels. Therefore, HD images or 4K images can be classified as large size images, which cannot be processed as one single input image into a complex neural network layer for training.

[0018]

[0026] The implementation shown in Figure 1A shows a large sized input image 100 that is divided into six smaller sized sub-pictures 110, 120, 130, 140, 150, 160. Figure 1B shows the processing of the sub-pictures and the formation of an output image from the sub-pictures. Figure 1B shows the process for two of the six sub-pictures. However, the remaining four sub-pictures can be processed similarly to the process for the two sub-pictures shown.

[0019]

[0027] Note that how small the small size sub-pictures need to be depends on the memory size and complexity of the neural network used for training. That is, a larger memory size allows processing with larger size sub-pictures, while a more complex neural network requires smaller size sub-pictures. In one example, the maximum size of a sub-picture that can be sent for training is 64x64 pixels or 128x128 pixels.

[0020]

[0028] In the implementation shown in Figure 1B, each of the sub-pictures 130, 160 is processed separately. First, each sub-picture is expanded by the target padding pixels around the sub-picture using a expansion process 170. This expansion process 170 is shown in Figure 2 and described in detail in the description of Figure 2.

[0021]

[0029] In the implementation shown in Figure 1B, each sub-picture is padded (by an upscaling process 170) to form an upscaled sub-picture 132, 162. The upscaled sub-picture 132, 162 is then upscaled (by a neural network-based upscaling process 180). Thus, in the upscaling process 180 of Figure 1B, the required memory size is dictated by the size of each sub-picture and the complexity of the neural network, rather than the size of the entire input image.

[0022]

[0030] 1B shows the enlarged and upscaled sub-pictures 134, 164. Each of the enlarged and upscaled sub-pictures 134, 164 is cropped to generate the original image area of ​​the sub-pictures 136, 166. The cropped sub-picture images 116, 126, 136, 146, 156, 166 are concatenated to reconstruct the upscaled output image 190.

[0023]

[0031] Figure 2 is a detailed flow diagram illustrating the upscaling process 170 (described with respect to Figure 1B) according to one implementation of the present disclosure. The implementation shown in Figure 2 illustrates upscaling one subpicture 130 with additional padding / pixel regions A, B, C to generate an upscaled subpicture 132. Although Figure 2 illustrates the upscaling process only for subpicture 130, the remaining subpictures can be upsized using the same process described here for subpicture 130.

[0024]

[0032] In one implementation, the additional padding / pixel area to be added to the subpicture 130 includes eight sub-areas (e.g., two sides, top and bottom, and four corners). In this implementation, the eight sub-areas are categorized into three types of pad designs (e.g., A, B, C).

[0025]

[0033] In one example, a type A pad design is used for sub-regions that exist within the active picture area 100. Thus, the type A pad design is formed by expanding the image area in four directions. In FIG. 2, a type A pad design is used for the left sub-region, the bottom sub-region, and the bottom left corner sub-region. Thus, the expanded sub-picture 132 illustrates the formation of those sub-regions (i.e., the left sub-region, the bottom sub-region, and the bottom left corner sub-region) by expanding the image area in four directions.

[0026]

[0034] In another example, a type B pad design is used for sub-regions that are not within the active picture area 100 but are on the edges (e.g., left, right, top, or bottom). Thus, the type B pad design is formed using butterfly images on the edges. In Figure 2, a type B pad design is used for the top and right sub-regions that are not within the active picture area 100 and are not in any of the corners. Thus, the enlarged sub-picture 132 shows the use of butterfly images on the edges to form those sub-regions (i.e., the top and right sub-regions).

[0027]

[0035] In another example, a type C pad design is used for sub-regions that are not within the active picture area 100 but are at the corners (e.g., the top left corner, the top right corner, the bottom left corner, or the bottom right corner). Thus, the type C pad design is formed using butterfly images on the corner points. In FIG. 2, a type C pad design is used for the top left corner sub-region, the top right corner sub-region, and the bottom right corner sub-region that are not within the active picture area 100 but are at the corners. Thus, the enlarged sub-picture 132 illustrates the use of butterfly images on the corner points to form those sub-regions (i.e., the top left corner sub-region, the top right corner sub-region, and the bottom right corner sub-region).

[0028]

[0036] 3 is a flowchart illustrating a process 300 for training and upscaling a large size image according to one implementation of the present disclosure. In the implementation shown in FIG. 3, a large size input image is divided into multiple smaller size sub-pictures in block 310. FIG. 1A shows an example of a large size input image 100 that is divided into six smaller size sub-pictures 110, 120, 130, 140, 150, 160.

[0029]

[0037] At block 320, each subpicture is expanded using the target padding pixels around the subpicture. In one implementation, the expansion process includes adding extra padding / pixels around each subpicture to generate the expanded subpicture. Details of how the extra padding / pixels are added to each subpicture are shown in the flow diagram of FIG. 2 and the flowchart of FIG. 4.

[0030]

[0038] In one implementation, in block 330, each padded subpicture is sent to an ML-based upscaling processor (e.g., a neural network) to generate a padded upscaled subpicture. In block 340, each padded upscaled subpicture is cropped to the subpicture's original size multiplied by the upscaling factor. If in block 350 it is determined that there are more subpictures to be processed, the process 300 proceeds to block 320 to process the additional subpictures. In block 360, all cropped subpictures are concatenated to reconstruct the output image.

[0031]

[0039] 4 is a detailed flow chart illustrating the padding process (block 320 of FIG. 3) according to one implementation of the present disclosure. In the implementation shown in FIG. 4, we first determine how to pad pixels around a sub-picture in block 400. In one implementation, we group the pixels to be padded around a sub-picture into multiple sub-regions (types A, B, and C).

[0032]

[0040] In the implementation shown in Figure 4, the pixels to be padded are grouped into three small regions. In block 410, a check is made to determine whether each small region to be padded around the sub-picture is of a first type (e.g., type A in Figure 2) that exists within the active picture area (e.g., 100 in Figure 1A). Then, in block 412, if the small region exists within the active picture area, the image area of ​​the sub-picture is expanded in the direction of the small region.

[0033]

[0041] On the other hand, if the sub-region is determined to be outside the active picture area but is on an edge (e.g., type B in FIG. 2) in block 420, the butterfly image is used on the edges to pad the sub-picture in block 422. As mentioned above, type B padding design is used for the top and right sub-regions that are not within the active picture area and are not in any of the corners.

[0034]

[0042] On the other hand, if the sub-region is determined to be outside the active picture area but is in a corner (e.g., type C in FIG. 2) in block 430, the butterfly images are used on the corner points to pad the sub-picture in block 432. As mentioned above, type C padding design is used for the top left corner sub-region, the top right corner sub-region, and the bottom right corner sub-region that are not in the active picture area but are in a corner.

[0035]

[0043] If, at block 440, it is determined that there are more sub-regions to process, then, at block 450, the process proceeds to the next sub-region and returns to block 410. Otherwise, the process ends.

[0036]

[0044] Although blocks 410, 420, 430 of process 320 in FIG. 4 are arranged in a particular order, these blocks may be performed in any order without affecting the outcome of process 320.

[0037]

[0045] FIG. 5 is a detailed flow chart illustrating the cropping process (block 340 in FIG. 3) according to one implementation of the present disclosure. In the implementation shown in FIG. 5, in block 500, an enlarged and upscaled subpicture is received. In block 510, an original size and an upscaling factor of the subpicture are determined. Then, in block 520, an upscaling size is determined by multiplying the original size by the upscaling factor. In block 530, the subpicture is cropped to the upscaling size. In block 540, the cropped subpicture is output.

[0038]

[0046] 6 is a block diagram illustrating a large size image upscaling system 600 according to one implementation of the present disclosure. In the implementation illustrated in FIG. 6, the large size image upscaling system 600 includes an image splitter 610, a sub-picture expander 620, a sub-picture cropper 640, and an image filler 650.

[0039]

[0047] 6, the image divider 610 is configured to divide a large-sized input image into multiple smaller-sized sub-pictures. In one implementation, the large-sized input image is divided into multiple smaller equally-sized sub-pictures. The sub-picture expander 620 is configured to expand the sub-pictures by adding extra padding / pixels around the sub-pictures to generate enlarged sub-pictures.

[0040]

[0048] In the implementation shown in Figure 6, the upscaled sub-pictures are sent to an ML-based upscaling processor (e.g., neural network 630) to generate upscaled sub-pictures. The sub-picture cropping unit 640 is configured to crop the upscaled sub-pictures to a size corresponding to the size of the original sub-picture before the upscaling process multiplied by the upscaling factor. The image filling unit 650 is configured to concatenate all the cropped sub-pictures to reconstruct the output image.

[0041]

[0049] In one implementation, the large size image upscaling system 600 is a system implemented entirely in hardware, including one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate / logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. In another implementation, the large size image upscaling system 600 is implemented as a combination of hardware and software.

[0042]

[0050] 7A is a diagram of a large size image upscaling system 700 and a user 702 according to one implementation of the present disclosure. The user 702 uses the computer system 700 to implement an application for frame processing in the upscaling process.

[0043]

[0051] The computer system 700 stores and executes the large size image upscaling application 790 of Figure 7B. Additionally, the computer system 700 can be in communication with a software program 704. The software program 704 can include software code for the large size image upscaling application. The software program 704 can be loaded onto an external medium, such as a CD, DVD, or storage drive, as described further below.

[0044]

[0052] Further, the computer system 700 can be connected to a network 780. The network 780 can be connected in a variety of different architectures, such as a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, the network 780 can communicate with a server 785 that coordinates engines and data used in large size image upscaling applications. The network can also be different types of networks. For example, the network 780 can be the Internet, a local area network or any variation of a local area network, a wide area network, a metropolitan area network, an intranet or extranet, or a wireless network.

[0045]

[0053] 7B is a functional block diagram illustrating a computer system 700 hosting a large image upscaling application 790 according to one implementation of the present disclosure. The controller 710 is a programmable processor that controls the operation of the computer system 700 and its components. The controller 710 loads instructions (e.g., in the form of a computer program) from the memory 720 or an on-board controller memory (not shown) and executes these instructions to control the system. In its execution, the controller 710 provides a software system to the large image upscaling application 790, enabling, for example, the creation and configuration of engines and data extractors within the large image upscaling application. Alternatively, this service can be implemented as a separate hardware component in the controller 710 or the computer system 700.

[0046]

[0054] Memory 720 temporarily stores data for use by other components of computer system 700. In one implementation, memory 720 is implemented as RAM. In one implementation, memory 720 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0047]

[0055] Storage 730 stores data temporarily or long term for use by other components of computer system 700. For example, storage 730 stores data used by large size image upscaling application 790. In one implementation, storage 730 is a hard disk drive.

[0048]

[0056] The media device 740 accepts removable media and reads and / or writes data to the inserted media. In one implementation, for example, the media device 740 is an optical disk drive.

[0049]

[0057] User interface 750 includes components for receiving user input from and presenting information to a user 702 of computer system 700. In one implementation, user interface 750 includes a keyboard, a mouse, audio speakers, and a display. Controller 710 uses the input from user 702 to coordinate the operation of computer system 700.

[0050]

[0058] I / O interface 760 includes one or more I / O ports to connect to corresponding I / O devices, such as external storage or supplemental devices (e.g., a printer or PDA). In one implementation, the ports of I / O interface 760 include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another implementation, I / O interface 760 includes a wireless interface for communicating wirelessly with external devices.

[0051]

[0059] Network interface 770 includes wired and / or wireless network connections, such as an RJ-45 supporting an Ethernet connection or a "Wi-Fi" interface (including but not limited to 802.11).

[0052]

[0060] Computer system 700 includes additional hardware and software typical of a computer system (e.g., power, cooling, operating system), although these components are not specifically shown in Figure 7B for simplicity. In other implementations, different configurations of computer systems can be used (e.g., different bus or storage configurations or multi-processor configurations).

[0053]

[0061] The description of the disclosed implementations is provided to enable any person skilled in the art to make or use the disclosure. Many modifications of these implementations will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other implementations without departing from the spirit or scope of the disclosure. For example, although the above description refers to the problem of upscaling large size images, the upscaling difficulties may also include the difficulties of upscaling images with a larger bit depth (e.g., 16-bit depth vs. 8-bit depth). Thus, the disclosure is not intended to be limited to the implementations shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0054]

[0062] Various implementations of the present disclosure may be realized in the form of electronic hardware, computer software, or a combination of these technologies. Some implementations include one or more computer programs executed by one or more computing devices. In general, a computing device includes one or more processors, one or more data storage components (e.g., volatile or non-volatile memory modules and persistent optical and magnetic storage devices, such as hard disk drives and floppy disk drives, CD-ROM drives, and magnetic tape drives), one or more input devices (e.g., game controllers, mice, and keyboards), and one or more output devices (e.g., display devices).

[0055]

[0063] A computer program typically comprises executable code that is stored on a persistent storage medium and copied into memory at run time. At least one processor executes the code by retrieving program instructions from memory in a predetermined order. During execution of the program code, the computer receives data from input and / or storage, performs operations on the data, and provides the resulting data to output and / or storage.

[0056]

[0064] Those skilled in the art will appreciate that the various exemplary modules and method steps described herein can be implemented as electronic hardware, software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary modules and method steps have been described herein generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Additionally, the grouping of functions within a module or step is for ease of description. Certain functionality may be moved from one module or step to another without departing from the present disclosure.

[0057]

[0065] Not all features of each of the above embodiments are necessarily required in a particular implementation of the present disclosure. Furthermore, it should be understood that the description and drawings presented herein are representative of the subject matter broadly intended by the present disclosure. Furthermore, it should be understood that the scope of the present disclosure fully includes other implementations that may become apparent to those skilled in the art, and therefore, the scope of the present disclosure is not limited by anything other than the scope of the appended claims. [Explanation of symbols]

[0058] 100 Larger size input image / active picture area 110,120,130,140,150,160 Subpictures 116,126,136,146,156,166 Cropped subpicture images 132,162 Enlarged Subpictures 134,164 Enlarged and upscaled subpictures 170 Enlargement Process 180 Upscaling Process 190 upscaled output image 300 A process for training and upscaling large size images 310 Divide the input image into smaller sub-pictures 320 Enlarge each subpicture by adding extra padding around it. 330 ML-based upscaling applied 340 Crop each enlarged and upscaled subpicture to the subpicture's original size multiplied by the scaling factor. 350 Are there more subpictures to process? 360 Concatenate all cropped subpictures to reconstruct the output image 400 Determine how to pad a small region around a subpicture 410 Is the small region within the active picture area? 412 Expanding to the direction of the small region 420 Edge? 422 Use butterfly images on edges to pad small regions 430 corner? 432 Use butterfly images on corner points to pad small regions 440 Are there further sub-areas to be determined? 450 Go to the next sub-area 500 Receive the enlarged and upscaled subpicture 510 Determine the original size and scaling factor of a subpicture 520 Determine the upscaling size by multiplying the original size by the scaling factor Crop 530 subpicture to upscaling size 540 Output cropped subpicture 600 Large Size Image Upscaling System 610 Image division unit 620 Subpicture Extension 630 Neural Networks 640 Subpicture Cropping Section 650 Image Filling Section 700 Large size image upscaling system / computer system 702 users 704 Software Programs 710 Controller 720 Memory 730 Storage 740 Media Devices 750 User Interface 760 I / O Interface 770 Network Interface 780 Network 785 Server 790 Large size image upscaling application

Claims

1. 1. A method for training and upscaling an input image, the method comprising: Dividing the input image into a number of sub-pictures; stretching each sub-picture of the plurality of sub-pictures using target padding pixels to generate a stretched sub-picture; upscaling each upscaled subpicture using a machine learning based upscaler to generate an upscaled subpicture; cropping the enlarged and upscaled sub-pictures to an upscaled size equal to the original size of each sub-picture multiplied by an upscaling factor; repeating the enlarging, upscaling, and cropping for the plurality of sub-pictures; concatenating the plurality of cropped sub-pictures to generate an output image; Including, The step of enlarging each subpicture comprises: determining how to pad extra pixels around each sub-picture; adding a small region of said extra pixels around each sub-picture; Including, The step of adding a sub-region includes a step of determining a type for each of the sub-regions; The first type of sub-region is the sub-region having pixels lying entirely within an active picture area of ​​the input image; A method comprising:

2. Furthermore, expanding each sub-picture in the direction of the sub-region when the sub-region is determined to be of the first type; Including, 2. The method according to claim 1 .

3. The second type of sub-region is the sub-regions having pixels that are not within the active picture area of ​​the input image but are on the edge of each sub-picture, 2. The method according to claim 1 .

4. Furthermore, when said sub-regions are determined to be of said second type, expanding each sub-picture using butterfly images on said edges by forming each sub-picture including at least a first and a second sub-region by connecting the first and second sub-regions by copying and mirroring the first sub-region onto the second sub-region along a line of symmetry formed by the edge; Including, 4. The method according to claim 3 .

5. The third type of sub-region is the sub-regions having pixels that are not within the active picture area of ​​the input image but are on the corners of each sub-picture, 2. The method according to claim 1 .

6. Furthermore, when said sub-regions are determined to be of said third type, expanding each sub-picture using a butterfly image on said corners by forming each sub-picture including at least a first and a second sub-region by connecting the first and second sub-regions by copying and mirroring the first sub-region onto the second sub-region on a symmetric point formed by a corner point of the corner; Including, 6. The method according to claim 5 .

7. An image upscaling system comprising: an image divider configured to divide an input image into a plurality of sub-pictures; a subpicture expansion unit configured to expand each subpicture of the plurality of subpictures by adding extra pixels around each subpicture to generate an expanded subpicture, a sub-picture expansion unit for sending the expanded sub-picture to a neural network for training and upscaling the expanded sub-picture; a sub-picture cropping unit configured to receive the enlarged and upscaled sub-picture and crop it to a size corresponding to a size of the original sub-picture multiplied by an upscaling factor; an image filler configured to concatenate all the cropped sub-pictures of the plurality of sub-pictures to reconstruct an output image; Including, the subpicture extension unit adds the extra pixels around each subpicture by adding a small region of the extra pixels around each subpicture; Adding a sub-region includes determining a type for each of the sub-regions; The first type of sub-region is the sub-region having pixels lying entirely within an active picture area of ​​the input image; A system characterized by:

8. 8. The system of claim 7, wherein the sub-picture expansion unit is further configured to expand each sub-picture in the direction of the sub-region when the sub-region is determined to be of the first type.

9. The second type of sub-region is the sub-regions having pixels that are not within the active picture area of ​​the input image but are on the edges of each expanded sub-picture; 8. The system according to claim 7 .

10. 10. The system of claim 9, wherein the sub-picture expansion unit is further configured to expand each sub-picture using butterfly images on the edges when the small region is determined to be of the second type.

11. The third type of sub-region is the sub-regions having pixels that are not within the active picture area of ​​the input image but are on corners of each expanded sub-picture; 8. The system according to claim 7 .

12. 12. The system of claim 11, wherein the subpicture expansion unit is further configured to expand each subpicture using a butterfly image on the corners when the small region is determined to be of the third type.

13. 1. A non-transitory computer-readable storage medium storing a computer program for training and upscaling an input image, the computer program including executable instructions that cause a computer to: Dividing the input image into a plurality of sub-pictures; stretching each sub-picture of the plurality of sub-pictures using target padding pixels to generate a stretched sub-picture; upscaling each upscaled subpicture using a machine learning based upscaler to generate an upscaled subpicture; cropping the enlarged and upscaled sub-pictures to an upscaled size equal to the original size of each sub-picture multiplied by an upscaling factor; repeating executable instructions to cause the computer to enlarge, upscale, and crop the plurality of sub-pictures; concatenating the plurality of cropped sub-pictures to generate an output image; and Let them do so, The executable instructions for causing a computer to enlarge each sub-picture include executable instructions for causing a computer to: determining how to pad extra pixels around each subpicture; adding said small region of extra pixels; and Let them do so, Adding a sub-region includes determining a type for each of the sub-regions; The first type of sub-region is the sub-region having pixels lying entirely within an active picture area of ​​the input image; A non-transitory computer-readable storage medium comprising:

Citation Information

Patent Citations

  • Image processing system, image processing apparatus, and image display device

    JP2017216598A

  • Image processing device, display, image processing method, program, and recording medium

    WO2009130820A1