Image processing system and related image processing method for image enhancement based on region control and texture synthesis

Abstract

An image processing system includes a material image generator, at least one texture generator, and an output controller. The material image generator is configured to generate a material image. The at least one texture generator is coupled to the material generator unit and configured to adjust texture characteristics of the material image to generate at least one texture image. The output controller is coupled to the at least one texture generator and configured to perform region characteristic analysis on a source image to generate analysis results, determine region weights based on the analysis results, and synthesize the at least one texture image with the source image based on the region weights to generate an output image.

Description

TECHNICAL FIELD

[0001] The present application relates to image processing, and in particular to an image processing system and related image processing method for image enhancement based on region control and texture synthesis techniques. BACKGROUND

[0002] The mid-high frequency details of compressed images or streaming images are often lost due to compression operations. Image enhancement processing is generally used to attempt to restore these lost details. Common processing methods include sharpening and deep learning image enhancement techniques. Sharpening generally increases the high-frequency details in an image, such as using a high-pass filter to enhance the texture and edge regions in the image. However, the drawback of sharpening is that if the texture and edges in the image have been completely destroyed by compression operations, there is no way to increase more details. On the other hand, deep learning is a method of training an image enhancement model by inputting a large number of images of various types, so that the model can acquire knowledge of the correlation between image content and detail texture. When the image enhancement model is subsequently faced with a source image that has been compressed, it can guess the lost details based on the image content of the source image and regenerate them. However, the drawback of this method is that the texture and details generated by deep learning are difficult to control, and unnatural artifacts are easily generated. Furthermore, this method requires high computational power. SUMMARY

[0003] Therefore, the present application provides an image enhancement processing technique based on texture synthesis and region control. The image enhancement processing of the present application has the ability to generate details, and can still achieve certain enhancement effects even if the details of the source image are completely lost. Furthermore, since the present application does not use a deep learning network to generate details, the requirement for computational resources is also relatively low. In embodiments of the present application, a material image is generated by a material image generator, and one or more texture generators are used to adjust the texture characteristics of the material image to generate a texture image. The texture in this image has a specific direction and a specific density. In embodiments of the present application, multiple texture images are generated by different settings, thereby improving the adaptability to different types of details in the source image. Subsequently, based on the analysis of the region characteristics of the source image (such as frequency, brightness, semantic segmentation, or object dynamics), the intensity of the image enhancement effect caused by the texture image is controlled in a partitioned manner when the texture image and the source image are synthesized, thereby improving the adjustability and the matching degree of the texture and the details in the source image, and achieving better and more natural image enhancement effects.

[0004] Embodiments of the present invention provide an image processing system comprising: a source image generator, at least one texture generator, and an output controller. The source image generator is used to generate source images. The at least one texture generator is coupled to the source image generator and is used to adjust the texture characteristics of the source image to generate at least one texture image. The output controller is coupled to the at least one texture generator and is used to perform regional characteristic analysis on the source image to generate analysis results, determine regional weights based on the analysis results, and synthesize the at least one texture image with the source image based on the regional weights to generate an output image.

[0005] An embodiment of the present invention provides an image processing method, the image processing method comprising: generating a source image; adjusting the texture characteristics of the source image to obtain at least one texture image; performing regional characteristic analysis on the source image to generate an analysis result; determining a region weight based on the analysis result; and compositing the at least one texture image with the source image based on the region weight to generate an output image. Attached Figure Description

[0006] Figure 1 An architecture diagram of the image processing system according to a first embodiment of the present invention is shown.

[0007] Figure 2 This illustrates how an embodiment of the present invention determines region weights based on the regional characteristics of a source image.

[0008] Figure 3 An architecture diagram of an image processing system according to a second embodiment of the present invention is shown.

[0009] Figure 4 An architecture diagram of an image processing system according to a third embodiment of the present invention is shown.

[0010] Figure 5 An architecture diagram of an image processing system according to a fourth embodiment of the present invention is shown.

[0011] Figure 6 A flowchart of an image processing method according to an embodiment of the present invention is shown.

[0012] Figure 7 This invention illustrates how to implement an image processing method using a hardware device in an embodiment of the invention. Detailed Implementation

[0013] Numerous specific details are described below to provide the reader with a thorough understanding of embodiments of the invention. However, those skilled in the art will appreciate how the invention can be implemented in the absence of one or more specific details, or by utilizing other methods, elements, or materials. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the core concepts of the invention.

[0014] The phrase "one embodiment" in this specification means that a particular feature, structure, or characteristic described in that embodiment may be included in at least one embodiment of the invention. Therefore, the phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment. Furthermore, the aforementioned particular features, structures, or characteristics may be combined in any suitable form in one or more embodiments.

[0015] Please refer to Figure 1 This figure illustrates the architecture of an image processing system according to an embodiment of the present invention. As shown, the image processing system 100 performs image enhancement processing on a source image IMG_S to generate an output image IMG_OUT. The image processing system 100 includes a source image generator 110, texture generators 120_1 to 120_2, and an output controller 130. Note that although only two texture generators 120_1 to 120_2 are shown in the illustrated embodiment, those skilled in the art should understand in the following description how the core concepts disclosed in this embodiment can be extended and applied to implement a complete image processing system with more or fewer second texture generators, and such variations still fall within the scope of the present invention.

[0016] The source image generator 110 generates a source image IMG_MA (with dimensions HxW, the same as the source image IMG_S). In one embodiment, the source image generator 110 can be a random noise generation device, which generates an image including noise with a random distribution. In some embodiments of the present invention, the random noise generation device can be implemented using hardware circuitry such as a Linear Feedback Shift Register (LFSR) or a Hardware Random Number Generator (HRNG) based on thermal noise. The source image IMG_MA generated by the source image generator 110 will be provided to texture generators 120_1 and 120_2.

[0017] Texture generators 120_1 to 120_2 adjust the texture characteristics of the source image IMG_MA, converting it into a texture with a specific shape and distribution. In one embodiment, each of texture generators 120_1 to 120_2 includes one or more filters that can be used to adjust the direction and density of noise distribution in the source image IMG_MA. Figure 1 In the illustrated embodiment, texture generators 120_1 to 120_2 consist of directional filters 122_1 to 122_2 and low-pass filters 124_1 to 124_2. The directional filters 122_1 to 122_2 can change the noise distribution direction in the source image IMG_MA, while the low-pass filters 124_1 to 124_2 can change the noise distribution density in the source image IMG_MA. Through such adjustments, texture generators 120_1 to 120_2 can generate unique texture images IMG_TXT1 and IMG_TXT2 (whose image size can be the same as the source image IMG_S, both being HxW), respectively, suitable for enhancing different types of details in the source image IMG_S. Note that in other embodiments of the invention, texture generators 120_1 to 120_2 may include more or fewer filters, or other types of filters, or a different arrangement (e.g., low-pass filters first, directional filters second). Those skilled in the art to which this invention pertains should understand that, based on the above concepts, different types or numbers of filters can be used to change the texture characteristics of the source image IMG_MA, thereby generating a specific texture image. The texture images IMG_TXT1 and IMG_TXT2 generated by texture generators 120_1 to 120_2 are provided to the output controller 130, which then combines the texture images IMG_TXT1 and IMG_TXT2 with the source image IMG_S.

[0018] The output controller 130 includes a region analysis unit 132, a weight generation unit 134, a magnification units 136_1 to 136_2, and a synthesis unit 138_1 to 138_2. The output controller 130 senses the regional characteristics of the source image IMG_S and accordingly controls the weighting of texture synthesis to achieve a good image enhancement effect. The region analysis unit 132 is used to perform partition analysis on the source image IMG_S. Figure 2As shown, the region analysis unit 132 can divide the source image IMG_S into 6x4 regions R0 to R23 and analyze the characteristics of each region. The region analysis unit 132 can analyze (but is not limited to) one or more of the following characteristics: region frequency (analyzed after converting the source image IMG_S to the frequency domain), region brightness, region semantics (i.e., region type, such as grassland, water surface, or sand, which can be determined through semantic segmentation), and region dynamics. The region analysis unit 132 quantifies the obtained region frequency, region brightness, region semantics, and region dynamics to obtain the analysis results. The analysis results generated by the region analysis unit 132 are provided to the weight generation unit 134.

[0019] Based on this analysis result, the weight generation unit 134 generates region weight A corresponding to texture image IMG_TX1 and region weight B corresponding to texture image IMG_TX2. Region weights A and B are as follows: Figure 2 As shown, each region includes 6x4 weight coefficients A0-A23 and B0-B23, indicating the composite intensity of texture images IMG_TXT1 and IMG_TXT2 relative to the source image IMG_S. For example, weight coefficient A11 in region weight A indicates the intensity to be used when compositing the corresponding region of texture image IMG_TXT1 into region R11 of the source image IMG_S, and weight coefficient B15 in region weight B indicates the intensity to be used when compositing the corresponding region of texture image IMG_TXT2 into region R15 of the source image IMG_S. It should be noted that the number of partitions and the number of weight coefficients in region weights A and B are not limitations of this invention, and other possibilities exist in other embodiments of this invention.

[0020] Furthermore, the texture distribution direction and density of texture images IMG_TXT1 and IMG_TXT2 themselves may make them suitable for enhancing the details of different types of image content. For example, texture image IMG_TXT1 may be relatively suitable for enhancing dark details, while texture image IMG_TXT2 may be relatively suitable for enhancing bright details; or texture image IMG_TXT1 may be relatively suitable for enhancing grass details, while texture image IMG_TXT2 may be relatively suitable for enhancing water surface details; or texture image IMG_TXT1 may be relatively suitable for enhancing the details of dynamic objects, while texture image IMG_TXT2 may be relatively suitable for enhancing the details of static objects. After obtaining the analysis results generated by the region analysis unit 132 for the region characteristics of the source image IMG_S, the weight generation unit 134 can determine the region weights A and B according to the characteristics of texture images IMG_TXT1 and IMG_TXT2, thereby aggravating or mitigating the influence of texture images IMG_TXT1 and IMG_TXT2 on specific regions in the source image IMG_S. For example, the weight of a texture image whose texture characteristics are suitable for a specific region is increased for that specific region (adaptive detail enhancement); the weight of a texture image whose texture characteristics are not suitable for a specific region is decreased for that specific region (adaptive detail reduction); the weight of all texture images for a specific region is increased, and the weight of all texture images for a specific region is decreased. Once the weight generation unit 134 determines the region weights A and B, the amplification units 136_1 to 136_2 and the synthesis units 138_1 to 138_2 can synthesize the texture images IMG_TXT1 and IMG_TXT2 with the source image IMG_S based on the weight coefficients A0 to A23 and B0 to B23, thereby generating the output image IMG_OUT.

[0021] As explained above, texture images IMG_TXT1 and IMG_TXT2 affect the adaptability of the image processing system 100 to source images with different content. Therefore, in other embodiments of the present invention, the image processing system 100 can have more texture generators to generate more texture images with different texture distribution directions and densities, thereby better restoring the details of specific image content. Furthermore, in Figure 3 and Figure 4 The illustrated embodiments provide architectures for different texture generators, wherein... Figure 3In the illustrated embodiment, the filter parameters of the directional filter 122_3 and / or low-pass filter 124_3 in the texture generator 120_3 are determined based on the analysis results of the region analysis unit 132. For example, when the region analysis unit 132 analyzes the semantics of a specific region in the source image IMG_S, the filter parameter database 126 outputs the corresponding filter parameters to the directional filter 122_3 and / or low-pass filter 124_3 according to the category index corresponding to this specific region semantics, to generate the texture image IMG_TXT. In this embodiment, since the texture image IMG_TXT generated by the texture generator 120_3 is directly adaptable to the source image IMG_S, it is possible to restore different types of details in the source image IMG_S without the need for a multi-path texture generator. Figure 4 In the illustrated embodiment, the texture generator 120_4 can even be implemented using a convolutional neural network. Similarly, the texture image IMG_TXT generated by the texture generator 120_4 is directly adaptable to the source image IMG_S, thus eliminating the need for other texture generators.

[0022] On the other hand, in other embodiments of the present invention, the material image generator can be implemented by a pattern capture device, such as... Figure 5 In the illustrated embodiment, pattern extractor 112 extracts a pattern with a specific frequency from the source image IMG_S, generates a material image IMG_MA based on this pattern, and then the subsequent texture generators 120_1 to 120_2 generate a texture image, which is then composited with the source image IMG_S. The image extractor 112 may include a Sobel filter or a Discrete Cosine Transform unit to extract the portion of the source image IMG_S with a specific frequency as the material image IMG_MA.

[0023] Figure 6 A flowchart of an image processing method according to an embodiment of the present invention is shown. As shown, the image processing method of the present invention includes the following steps:

[0024] S310: Generate source images;

[0025] S320: Adjust the texture properties of the source image to generate a texture image;

[0026] S330: Perform regional characteristic analysis on the source image to generate analysis results;

[0027] S340: Determine the regional weights based on the analysis results; and

[0028] S350: Based on region weights, the texture image is synthesized with the source image to generate the output image.

[0029] Since the principles and specific details of the above steps have been described in detail in the previous embodiments, they will not be repeated here. It should be noted that the above process can be improved by adding other additional steps or making appropriate changes and adjustments to better achieve image enhancement processing and further enhance its image enhancement effect. Furthermore, all operations in the foregoing embodiments of the present invention can be achieved through… Figure 7 The device 400 shown is used to implement this. The storage unit 410 (e.g., non-volatile or volatile memory) in the device 400 can be used to store source code, instructions, variables, or data. The hardware processing unit 420 (e.g., a general-purpose processor) in the device 400 can execute the source code and instructions stored in the storage unit 410, and refer to the variables or data therein to perform all the operations described in the foregoing embodiments.

[0030] In summary, the image enhancement processing provided by this invention possesses the ability to generate details, thus achieving a certain enhancement effect even when the details of the source image are completely lost. Furthermore, since no deep learning network is used to generate details, the computational resource requirements are relatively low. In embodiments of this invention, a source image generator or pattern extractor is used to generate a source image, and one or more texture generators are used to adjust the texture characteristics of the source image to generate a texture image. In embodiments of this invention, multiple texture images are generated using different settings, thereby improving adaptability to different types of details in the source image. Subsequently, based on the regional characteristic analysis of the source image (e.g., frequency, brightness, semantic segmentation, or object dynamics), the intensity of the image enhancement effect caused by the texture image is controlled regionally during the synthesis of the texture image and the source image, thereby improving adjustability and the matching degree between the texture and the details in the source image, achieving a better and more natural image enhancement effect.

[0031] Embodiments of the present invention can be implemented using hardware, software, firmware, and combinations thereof. With a suitable instruction execution system, embodiments of the present invention can be implemented using software or firmware stored in memory and a corresponding instruction execution processor. In terms of hardware, any of the following technologies or combinations thereof can be applied: individual operational logic having logic gates capable of performing logical functions based on data signals; application-specific integrated circuits (ASICs) having suitable combinational logic gates; programmable gate arrays (PGAs); or field-programmable gate arrays (FPGAs), etc.

[0032] The flowcharts and blocks in the specification illustrate the architecture, functionality, and operation achievable by systems, methods, and computer software products based on various embodiments of the present invention. In this regard, each block in the flowchart or functional block diagram may represent a module, segment, or portion of source code, including one or more executable instructions for implementing a specified logical function. Furthermore, each block in the functional block diagram and / or flowchart, and combinations of blocks, can be substantially implemented by a dedicated hardware system that performs the specified function or action, or a combination of dedicated hardware and computer program instructions. These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable medium implement the function / action specified by the blocks in the flowchart and / or functional block diagram.

[0033] The above description is only a preferred embodiment of the present invention. Equivalent changes and modifications made according to the claims of the present invention are all within the scope of the present invention.

[0034] Figure Labels

[0035] 100 Image Processing System

[0036] 110 Source Image Generator

[0037] 112 Pattern Capturer

[0038] Texture generators 120_1~120_4

[0039] Directional filters 122_1~122_3

[0040] 124_1~124_3 Low-pass filters

[0041] 126 Filter Parameter Database

[0042] 130 Output Controller

[0043] 132 Regional Analysis Units

[0044] 134 weight generation units

[0045] 136_1~136_2 Amplification Unit

[0046] 138_1~138_2 Synthesis Unit

[0047] Image source: IMG_S

[0048] IMG_OUT output image

[0049] IMG_MA source images

[0050] IMG_TXT, IMG_TXT1, IMG_TXT2 texture images

Claims

1. An image processing system for enhancing a source image, comprising: A source image generator is used to generate a source image that is different from the source image but has the same size as the source image, wherein the source image is an image with randomly distributed noise; At least one texture generator, coupled to the material image generator, is used to adjust the texture properties of the material image to generate at least one texture image; An output controller, coupled to the at least one texture generator, is configured to perform regional feature analysis on a source image to generate analysis results, determine regional weights corresponding to the at least one texture image based on the analysis results, and synthesize the at least one texture image with the source image based on the regional weights to generate an output image, wherein the regional weights include weight coefficients corresponding to each region, which are used to indicate the intensity to be used when synthesizing the corresponding region in the at least one texture image to the corresponding region in the source image.

2. The image processing system of claim 1, wherein the source image generator comprises: A random noise generation apparatus for generating the source image including random noise, wherein the random noise generation apparatus includes a linear feedback shift register or a hardware random number generator based on thermal noise.

3. The image processing system of claim 1, wherein the at least one texture generator comprises: A directional filter is used to perform directional filtering on the source image to generate a directionally filtered image; as well as A low-pass filter, coupled to the directional filter, is used to perform low-pass filtering on the directional filtered image to generate the texture image.

4. The image processing system of claim 3, wherein the at least one texture generator further includes a filter parameter database, the filter parameter database providing one or more sets of specific filter parameters to at least one of the directional filter and the low-pass filter for filtering processing based on the analysis results.

5. The image processing system of claim 1, wherein the at least one texture generator comprises: A convolutional neural network is used to process the source image to generate the texture image.

6. The image processing system of claim 1, wherein the output controller comprises: The region analysis unit is used to divide the source image into NxM regions and determine multiple region characteristics of each of the NxM regions to obtain the analysis results. A weight determination unit, coupled to the region analysis unit, is used to determine multiple weight coefficients corresponding to the NxM regions respectively based on the analysis results, wherein the region weights are composed of the multiple weight coefficients; The region characteristics mentioned therein include one or more characteristics of region frequency, region brightness, region semantics, and object dynamics.

7. An image processing method for enhancing a source image, comprising: Generate source images, which are images with randomly distributed noise, different from the source images but having the same size as the source images; Adjust the texture properties of the source image to generate at least one texture image; The source image is subjected to regional characteristic analysis to generate analysis results; Based on the analysis results, determine the region weights corresponding to the at least one texture image; as well as According to the region weights, the at least one texture image is synthesized with the source image to generate an output image, wherein the region weights include weight coefficients corresponding to each region, which are used to indicate the intensity to be used when synthesizing the corresponding region in the at least one texture image to the corresponding region in the source image.

8. The image processing method of claim 7, wherein the step of generating the source image includes: The source image including random noise is generated using a random noise generation device, wherein the random noise generation device includes a linear feedback shift register or a hardware random number generator based on thermal noise.

Images