Method and device for improving visual fidelity of image

Abstract

The invention relates to the technical field of real-time three-dimensional image processing, and provides a method and a device for improving the visual fidelity of an image. According to the method, on the basis of the assumption that original pixels are located between two adjacent surfaces in a source image, the positions where the original pixels are located are classified, and different intersection filters are used for processing the original pixels meeting corresponding classification standards; after the intersection filter responds, an amplification coefficient corresponding to the decomposed source image features in at least one dimension is determined according to the processing logic of the intersection filter, original pixels are amplified based on the amplification coefficient, and a good enough approximate estimate of the amplified target image is obtained; according to the image super-resolution algorithm, the specific parameterization of the image is realized, the support of a filter is reduced to the maximum extent, the image quality and the calculation complexity are balanced to the maximum extent in the limited fixed function hardware prediction, and the problems that the image super-resolution algorithm in the prior art is not suitable for hardware and is poor in effect are solved.

Description

Technical Field

[0001] This invention relates to the field of real-time three-dimensional image processing technology, and in particular to a method and apparatus for improving the visual fidelity of images. Background Technology

[0002] Image super-resolution algorithms are image processing techniques that improve image resolution without sacrificing image quality, resulting in clearer and more stunning content presentation.

[0003] In the field of real-time 3D image processing, these algorithms improve application performance by reducing the number of pixels that must be processed per frame during rendering, while avoiding the disadvantages of increasing the processing time per pixel. These algorithms can be implemented in software; for example, by executing one or more post-processing shader instructions on a graphics processing unit (GPU); or by using hardware acceleration to implement the algorithm partially or entirely in fixed-function circuitry; or by employing machine learning methods and architectures with inherently programmable elements.

[0004] Existing image super-resolution algorithms are often not suitable for implementation entirely in fixed-function circuits, thus requiring various hardware and software resources and being highly impractical in actual production. On the other hand, algorithms that can be implemented entirely in fixed circuits often have poor image processing performance; that is, after increasing the resolution of the source image, problems such as image quality degradation or image distortion often occur.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and apparatus for improving the visual fidelity of images. The purpose is to provide a single-image super-resolution algorithm that, by implementing specific parameterization of the image, minimizes the support of filters and, within a limited fixed-function hardware budget, maximizes the balance between image quality and computational complexity, thereby solving the problem that existing image super-resolution algorithms are not suitable for hardware and have poor performance.

[0007] The present invention adopts the following technical solution: In a first aspect, the present invention provides a method for improving the visual fidelity of an image, comprising: Based on the location of the original pixels in the source image and the classification criteria of the intersection filters, determine the intersection filter used to process the original pixels; Based on the selected intersecting filter, determine the amplification factor in at least one dimension; The original pixels are magnified according to the magnification factor to obtain multiple target pixels; the multiple target pixels are combined to generate a target image.

[0008] Furthermore, the intersection filter includes a contour filter, a diagonal intersection filter, an axis-aligned intersection filter, a surface filter, and a filament filter; Determining the amplification factor in at least one dimension based on the selected intersecting filter includes: When the intersection filter is a contour filter, diagonal intersection filter, axis-aligned intersection filter, or filament filter, a parameterized coverage representation of the coverage area of ​​the original pixel is determined; based on the parameterized coverage representation and the baseline vector, the corresponding magnification factor is determined; When the intersecting filter is a surface filter, the baseline vector is scaled to obtain the amplification factor of the surface filter.

[0009] Furthermore, determining the intersection filter used to process the original pixels based on the position of the original pixels in the source image and the classification criteria of the intersection filter includes: When the position of the original pixel in the source image meets the assumption of the first classification criterion, the original pixel is processed using the axis-aligned intersection filter; wherein, the first classification criterion is that the image block where the original pixel is located only includes horizontal and vertical edges, and the horizontal and vertical edges are axis-aligned; When the position of the original pixel in the source image does not meet the assumption of the first classification criterion but meets the assumption of the second classification criterion, the original pixel is processed using the diagonal intersection filter; wherein, the second classification criterion is that the image block where the original pixel is located only includes the forward edge and the backward edge of the diagonal, and the forward edge intersects with the backward edge; When the position of the original pixel in the source image does not conform to the assumptions of the first classification criterion and the second classification criterion, the parameterized overlay representation corresponding to the original pixel is determined, and the intersection filter for processing the original pixel is determined based on the parameterized overlay representation.

[0010] Further, when the position of the original pixel in the source image does not conform to the assumptions of the first classification criterion and the second classification criterion, determining the parameterized cover representation corresponding to the original pixel, and determining the intersection filter for processing the original pixel based on the parameterized cover representation includes: Based on the assumption that the original pixel is located between two adjacent surfaces in the source image, a parameterized overlay representation corresponding to the original pixel is obtained; When the parameterized overlay representation is less than a preset normal value, the original pixel is processed using the filament filter; When the parameterized overlay representation is greater than or equal to a preset normal value, the contour filter is used to process the original pixel; When the original pixel is located in a position in the source image that cannot be classified as being processed by either a filament filter or a contour filter, the original pixel is processed by the surface filter.

[0011] Furthermore, the step of magnifying the original pixel according to the magnification factor to obtain multiple target pixels includes: Construct a complementary relationship between each eigenvalue decomposition coefficient and multiple target pixels; reconstruct the complementary relationship, and determine the expression for each target pixel based on the source image and the eigenvalue decomposition coefficients representing the multiple target pixels; The amplification factor of the selected intersecting filter is used as a reliable substitute for the eigenvalue coefficients in the corresponding dimension of the expression. Based on the reliable substitution, the color values ​​of the corresponding multiple target pixels are determined according to the color values ​​of the original pixels.

[0012] Furthermore, based on the kernel size of the intersecting filter, all the original pixels in the source image are processed using the intersecting filter according to the pipeline processing volume; Wherein, the pipeline processing capacity is a multiple of the proportional coefficient, and the filter core size is smaller than the pipeline processing capacity.

[0013] Furthermore, the method also includes: When the original pixels of the 1 / 2k row to be enlarged are located at the top of the source image, the currently processed image block is aligned k pixels with the source image; multiple currently processed image blocks are vertically aligned with each other; the currently processed image block copies adjacent pixels to expand the boundary of the source image; When the k rows of original pixels to be magnified are located vertically inside the source image, the currently processed image block is aligned with the source image by k pixels; multiple currently processed image blocks are vertically aligned with each other; the boundaries of the left and / or right edges of the magnified target image are repeated.

[0014] Furthermore, the intersecting filter uses rational univariate functions and trigonometric functions.

[0015] Secondly, the present invention also provides an apparatus for improving image visual fidelity, used to implement the method for improving image visual fidelity described in the first aspect, wherein the apparatus for improving image visual fidelity includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method for improving image visual fidelity as described in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method for improving image visual fidelity described in the first aspect.

[0017] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in a memory, performing a method for improving the visual fidelity of an image as described in the first aspect.

[0018] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer or processor, cause the computer or processor to perform a method for improving the visual fidelity of an image as described in the first to fourth aspects and any one thereof.

[0019] In a sixth aspect, a system for improving the visual fidelity of an image is provided, comprising an apparatus for improving the visual fidelity of an image as described in the second aspect, and using a method for improving the visual fidelity of an image as described in the first aspect to perform interaction with the apparatus for improving the visual fidelity of an image as described in the second aspect.

[0020] Unlike existing technologies, the present invention has at least the following beneficial effects: This invention classifies the positions of original pixels in the source image and then amplifies the original pixels by providing minimal filter support. It only uses parameterized representations to determine the pixel boundaries of the color values ​​referenced by the amplified original pixels, so a good approximation output can be obtained with a single calculation. It uses mathematical calculations that do not involve iteration, and within a limited fixed-function hardware budget, it maximizes the balance between image quality and computational complexity. It can all be embedded in hardware, breaking through the industrial application bottleneck of single-image super-resolution algorithms. It is highly practical and can achieve high clarity and color fidelity. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0022] Figure 1This is a flowchart illustrating a method for improving image visual fidelity according to an embodiment of the present invention; Figure 2 This is a specific example of a source image provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of obtaining a target pixel by magnifying an original pixel according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a baseline vector provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a coupling vector provided in an embodiment of the present invention; Figure 6 This is a flowchart illustrating step 10 provided in an embodiment of the present invention; Figure 7 This is a flowchart illustrating step 101 provided in an embodiment of the present invention; Figure 8 This is a flowchart illustrating step 20 provided in an embodiment of the present invention; Figure 9 This is a flowchart illustrating step 30 provided in an embodiment of the present invention; Figure 10 This is a specific example of an original pixel located on a single straight edge between two surfaces, provided by an embodiment of the present invention; Figure 11 This is a schematic diagram of a parameterized direction estimation provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of a surface probe provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of a first basis vector provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of a second basis vector provided in an embodiment of the present invention; Figure 15 This is a schematic diagram illustrating the division of the current circular region according to an embodiment of the present invention; Figure 16 This is a schematic diagram of another way of dividing the current circular region according to an embodiment of the present invention; Figure 17 This is a schematic diagram of a mounting limit provided in an embodiment of the present invention; Figure 18 This is an embodiment of the present invention that provides a method for each vertex in A diagram illustrating the order of intersections when scanning from 1 / 2 to 0; Figure 19 This is a schematic diagram showing the coverage of the first axial approximation method provided in the embodiments of the present invention; Figure 20This is a schematic diagram showing the coverage of the second axial approximation method provided in the embodiments of the present invention; Figure 21 This is a schematic diagram showing the coverage of the third axial approximation method provided in the embodiments of the present invention; Figure 22 This is a schematic diagram showing the coverage of the fourth axial approximation method provided in the embodiments of the present invention; Figure 23 This is an approximation based on axial approximation provided in an embodiment of the present invention. For different coverage areas of Relationship diagram; Figure 24 This is a schematic diagram showing the coverage of the first method for deriving the diagonal approximation provided in this embodiment of the invention; Figure 25 This is a schematic diagram showing the coverage of the second method for deriving the diagonal approximation provided in this embodiment of the invention; Figure 26 This is a schematic diagram showing the coverage of the third method for deriving the diagonal approximation provided in this embodiment of the invention; Figure 27 This is a schematic diagram showing the coverage of the fourth method for deriving the diagonal approximation provided in this embodiment of the invention; Figure 28 This is a schematic diagram showing the coverage of the fifth method for deriving the diagonal approximation provided in this embodiment of the invention; Figure 29 This is an approximation based on diagonal approximation provided in an embodiment of the present invention. For different coverage areas of Relationship diagram; Figure 30 This is a schematic diagram of a positive probe and a negative probe provided in an embodiment of the present invention; Figure 31 This is a schematic diagram of a contour vector provided in an embodiment of the present invention; Figure 32 This is a schematic diagram illustrating how an unmodulated first horizontal gradient parameter generates different segmentation behaviors based on different edge directions, according to an embodiment of the present invention. Figure 33 This is a schematic diagram of an unmodulated first vertical gradient parameter, a first horizontal gradient parameter, and a parameterized direction estimation provided in an embodiment of the present invention; Figure 34 This invention provides a target image magnified twice by bilinear interpolation, as provided in an embodiment of the invention. Figure 35 This is a target image magnified twice by processing with a contour filter and a surface filter, provided by an embodiment of the present invention; Figure 36 This is a schematic diagram illustrating a specific example of a pixelated region of another source image provided in an embodiment of the present invention; Figure 37 This is a schematic diagram of a target image using a universal filter response provided in an embodiment of the present invention; Figure 38 This is a schematic diagram of a target image using two filter responses provided in an embodiment of the present invention; Figure 39 This is a specific example of an original pixel located on the diagonal provided in an embodiment of the present invention; Figure 40 This is a schematic diagram of a southeast intermediate vector provided in an embodiment of the present invention; Figure 41 This is a schematic diagram of a northeast intermediate vector provided in an embodiment of the present invention; Figure 42 This is a schematic diagram of a target image using a universal filter response provided in an embodiment of the present invention; Figure 43 This is a schematic diagram of another target image using two filter responses provided in an embodiment of the present invention; Figure 44 This is a schematic diagram of a target image using three filter responses provided in an embodiment of the present invention; Figure 45 This is a specific example of an original pixel in an axis-aligned state provided by an embodiment of the present invention; Figure 46 This is a schematic diagram illustrating a degradation scenario provided by an embodiment of the present invention; Figure 47 This is a schematic diagram illustrating another degradation scenario provided by an embodiment of the present invention; Figure 48 This is a flowchart illustrating another method for improving image visual fidelity provided in an embodiment of the present invention; Figure 49 This is a schematic diagram of an additional alignment strategy provided in an embodiment of the present invention; Figure 50 This is a schematic diagram of another additional alignment strategy provided in an embodiment of the present invention; Figure 51 This is a schematic diagram of an alignment configuration provided by an embodiment of the present invention; Figure 52 This is a schematic diagram of a set of pixelated regions of a source image and a target image provided in an embodiment of the present invention; Figure 53 This is the target image magnified based on rendering target technology in the first set of specific examples provided in the embodiments of the present invention; Figure 54This is the target image magnified based on the super-resolution algorithm of the present invention in the first set of specific examples provided in the embodiments of the present invention; Figure 55 This is the target image magnified based on bilinear interpolation and sharpening techniques in the first set of specific examples provided in the embodiments of the present invention; Figure 56 This is a specific example of the second set of magnified target images provided in the embodiments of the present invention; Figure 57 This is a specific example of the third set of magnified target images provided in the embodiments of the present invention; Figure 58 This is a set of specific examples of a magnified target image generated using bilinear interpolation and the super-resolution algorithm of this invention, provided by the embodiments of the present invention; Figure 59 This is a specific example of the fifth group of magnified target images provided in the embodiments of the present invention; Figure 60 This is a specific example of the sixth group of magnified target images provided in the embodiments of the present invention; Figure 61 This is a specific example of the seventh group of magnified target images provided in the embodiments of the present invention; Figure 62 This is a specific example of the eighth group of magnified target images provided in the embodiments of the present invention; Figure 63 This is a specific example of the ninth group of magnified target images provided in the embodiments of the present invention; Figure 64 This is a schematic diagram of the architecture of a device for improving the visual fidelity of images provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0025] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0026] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0027] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] like Figure 1 As shown, this embodiment of the invention provides a method for improving the visual fidelity of images, including: Step 10: Determine the intersection filter to process the original pixels based on the location of the original pixels in the source image and the classification criteria of the intersection filter.

[0029] The concept of custom objects in the embodiments of the present invention will be explained below: The source image refers to the image whose resolution needs to be increased, and the original pixels refer to the pixels that make up the source image; a specific example of a source image is... Figure 2 As shown, Figure 2 Each square in the diagram represents an original pixel. In this embodiment of the invention, the source image is magnified by a scaling factor of 2 to obtain a target image magnified by a factor of 2. The target image is composed of all the target pixels obtained after magnification.

[0030] In this embodiment of the invention, 4×4 original pixels in the source image are processed each time. Hereinafter, the 4×4 original pixels processed each time are referred to as an image patch. A possible specific implementation of an image patch is as follows: Figure 2 As shown in the gray area; in an optional embodiment, a 4×4 sliding window can be used to process the source image; multiple image blocks can be processed in parallel to improve efficiency.

[0031] like Figure 3 The image on the left is a specific example of an input image patch, such as... Figure 3The image on the right shows the corresponding target pixels after the original 2×2 pixels in the middle of the image block are magnified by 2 times. Specifically, the original pixel marked "p11" is magnified into target pixels "p00", "p01", "p10", and "p11", and these four target pixels form the corresponding local region image; the original pixel marked "p12" is magnified into target pixels "p02", "p03", "p12", and "p13", and these four target pixels form the corresponding local region image; the original pixel marked "p21" is magnified into target pixels "p20", "p21", "p30", and "p31", and these four target pixels form the corresponding local region image; the original pixel marked "p22" is magnified into target pixels "p22", "p23", "p32", and "p33", and these four target pixels form the corresponding local region image; all the local region images obtained after magnifying the source image constitute the final target image.

[0032] It is worth noting that in this embodiment of the invention, each filter used to process the source image is based on a 3×3 size filter kernel, that is, 3×3 pixels in the source image are processed at a time. Hereinafter, the portion of the source image processed based on the filter kernel size is referred to as the current input block. This design aims to limit computational complexity and memory requirements. On the other hand, feature classification, i.e., the process of selecting and controlling filters, is inherently nonlocal and presents the same memory challenges as the filters themselves. This embodiment of the invention reduces the complexity of feature classification by describing it as a function of the parameters of neighboring pixels generated by a single filter kernel. The coupling of these parameters effectively expands the support for filters and also makes them suitable for parallel implementation, thereby scaling the number of output filtered results with the number of original pixels included in the feature classification operation. This is a very simple alignment method where feature classification computation and results should be generalizable to all included neighboring pixels; that is, the original pixels in the source image can be batch-processed in parallel, and the output target pixels can be generated in consecutive blocks, where the effective footprint of each corresponding input block is the union of the 3×3 filter kernels of its individual pixels. As follows... Figure 11As shown, the block-based approach also allows for limiting increased memory requirements by restricting the input block size to 2×2 raw pixels (considering that each raw pixel corresponds to a 4×4 footprint on an aligned 3×3 filter kernel). Unlike designing the filter kernel itself, finding a compromise is sufficient, or at least possible, where only a single adjacent set of horizontal or vertical (or diagonal) parameters is needed to generate sufficient classification results; it should be noted that the block-based approach is also a natural choice for regular convolutional kernels, as the blocks will correspond to some local bandwidth reduction buffers that cannot be further simplified in computation unless some explicit separation is made to the filter kernel.

[0033] The intersecting filter refers to the filter used to process the source image in this embodiment of the invention. This invention is based on a local feature classification heuristic algorithm, using multiple filter responses for different situations to amplify the local features in the source image by a scaling factor of 2. It is worth noting that other super-resolution algorithms in the prior art (such as AI-based algorithms) often do not directly identify the features of various special cases in the source image, but rather learn them through continuous learning based on the amount of data. This iterative learning method requires consuming several rows and columns of pixel information for each image processing, consuming excessive computational resources, making it unsuitable for hardware implementation or resulting in poor quality of the generated target image. This invention is a hand-designed algorithm for customized fixed-function hardware. The intersecting filter uses rational univariate functions and trigonometric functions, based on the characteristics of RGB (Red, Green, Blue) encoded images, and uses a purely mathematical approximation method to directly identify the features of various special cases in the source image, greatly reducing the computational load and resource consumption. This allows the method of this embodiment to be implemented very simply in hardware, representing a significant breakthrough in practicality.

[0034] The basic idea behind processing the source image in this invention is based on the following assumption: the position of the original pixel in the source image can be modeled as being between two adjacent surfaces. Based on this assumption, the position of the original pixel is classified into two cases: one between two surfaces, according to a corresponding classification standard. To obtain a higher quality target image, this invention also provides a supplementary processing scheme based on this assumption and for special cases involving a single straight edge (such as when aligned on an axis or diagonally), as well as special processing for discontinuous edges in the source image that are not applicable to this assumption. For each position case, some simple parameters are used to represent the boundaries in the source image. Based on the boundary division, the pixel (i.e., color value) at the corresponding position in the expanded target image is determined. By ensuring that these parameters are approximate values ​​that are valid in most cases, the computational complexity is reduced. The expression obtained from the parameters determines how to perform image magnification.

[0035] Step 20: Determine the amplification factor in at least one dimension based on the selected cross-linked filter.

[0036] The magnification factor in this embodiment of the invention is based on improving the resolution of the original pixels. The relevant definitions and specific improvement methods are explained below: It is a vector representing pixels, where i represents the horizontal pixel coordinate value and j represents the vertical pixel coordinate value; where the physical width of the image is w, the physical height is h, the pixel width is W, the pixel height is H, and the color value of each original pixel is the uniform average value over a rectangle centered on that original pixel, the physical width of which is... Physical height is It should be noted that although this may not produce optimal results from the perspective of sampling theory, this embodiment of the invention uses this method to define pixels in order to provide a universal solution for real-time 3D image processing.

[0037] Based on the above definitions, with an amplification factor of 2, this embodiment of the invention obtains a sufficiently good approximation of the target image based on Haar wavelet transform. Since the transformation matrix of the Haar wavelet transform is a sparse matrix, the computational cost is relatively small, making it very suitable for fast computation in real-time 3D image processing scenarios. This embodiment of the invention approximates the input signal (i.e., the source image) step by step, gradually decomposing the source image into sub-signals of different frequencies (i.e., local region images composed of target pixels). Specifically, for the input source image, it is split into 4 regional images, each sub-signal consisting of a continuous segment of the input signal, and these sub-signals constitute different frequency components of the input signal. An image with the same pixel width of 2W and pixel height of 2H can be obtained from the source image. It is converted from the source image. The image is divided into four quarters (four magnified local region images). The target pixels that make up each local region image are obtained by calculating the corresponding magnification factor. This process will be explained in detail below when using each intersection filter to respond.

[0038] Step 30: Magnify the original pixel according to the magnification factor to obtain multiple target pixels; combine the multiple target pixels to generate a target image.

[0039] like Figure 3 As shown, in this embodiment of the invention, the original pixels in the source image are processed sequentially based on image blocks of size 4×4, and each original pixel is enlarged into four target pixels. The target image is composed of all the target pixels obtained after enlargement.

[0040] This invention classifies the positions of original pixels based on the assumption that they are located between two adjacent surfaces in the source image. Different intersection filters are used to process original pixels that meet the corresponding classification criteria. After the intersection filter responds, it determines the amplification coefficient of at least one dimension corresponding to the decomposed features of the source image according to its own processing logic, and amplifies the original pixel based on this coefficient, obtaining a sufficiently good approximation of the amplified target image. Since it only uses parameterized representations to determine the pixel boundaries of the color values ​​referenced by the amplified original pixels, a good approximation output can be obtained with a single calculation. Because it uses mathematical calculations that do not involve iteration, it can be entirely embedded in hardware, breaking through the industrial application bottleneck of single-image super-resolution algorithms. It is highly practical and can achieve high clarity and color fidelity.

[0041] Before providing a detailed description of the method for improving image visual fidelity according to embodiments of the present invention, the relevant definitions in the embodiments of the present invention are explained as follows: This invention defines a set of baseline vectors, as follows: ; ; For example, such as Figure 4 As shown, in , and In the schematic diagram, All represent the dark gray pixels in the middle; in In the schematic diagram, This represents the light purple pixel on the right. Indicates the dark purple pixel on the left; in In the schematic diagram, This indicates the light purple pixels at the top. Indicates the dark purple pixel below; in In the schematic diagram, This represents the light blue pixel in the upper right corner. This represents the dark purple pixel in the lower right corner. This represents the light blue pixel in the lower left corner. This represents the dark purple pixel in the upper left corner.

[0042] Axis-aligned intersection filters and diagonal intersection filters rely on a common set of coupling vectors to determine the relevant coverage value corresponding to the original pixel. The following is an explanation of the custom coupling vector: Based on the original pixels in the source image, calculate a set of positive / negative coupling vectors; such as Figure 5 As shown, this invention represents the corresponding positive / negative coupling vectors on a set of 3×3 filter kernels, wherein the relative position directions (east, south, west, and north) of the original pixels are determined according to... Figure 5 The direction of the top-left coordinate axis is defined; the expressions for the positive / negative coupling vectors are as follows: ; Axis-aligned intersection filters and diagonal intersection filters also rely on an error threshold to meet certain classification criteria. The specific formula for the error threshold depends on how the source image was generated. A specific example of an error threshold based on sRGB (standard Red Green Blue) encoding is as follows: ; Where C is the error constant, and N is the precision of each color channel. It is a nonlinear transfer function. It is the color value of the red channel corresponding to each pixel. It is the color value of the green channel corresponding to each pixel. C is the color value of the blue channel corresponding to each pixel; in optional embodiments, C can be 1 / 4, 1 / 2, or 1; when based on the sRGB encoded source image, N is 8. It is approximately equal to 2.4.

[0043] Below is a set of definitions for the error coupling vector for each original pixel: ; It should be noted that, for a given original pixel, each negative coupling vector corresponds to the positive coupling vector of its neighboring pixels (i.e., adjacent original pixels), and vice versa; the error coupling vector is similar. Therefore, for specific implementations that use shared memory to process multiple original pixels in parallel, it is necessary to compute many asymptotic and arbitrary derivations of half of the vectors.

[0044] The characteristics of axis-aligned intersection filters and diagonal intersection filters are also reflected in the fact that corresponding error tolerances are defined for the horizontal baseline vector and the vertical baseline vector, as detailed below: ; in, To provide an error tolerance for the horizontal baseline vector, This is the error tolerance for the vertical baseline vector.

[0045] This invention employs different intersection filter responses for original pixels located at different positions in the source image; the intersection filters include contour filters, diagonal intersection filters, axis-aligned intersection filters, surface filters, and filament filters; the classification criteria for the original pixel positions in this invention are explained below, such as... Figure 6 As shown, step 10 includes: Step 101: When the position of the original pixel in the source image meets the assumption of the first classification criterion, the original pixel is processed using the axis-aligned intersection filter; wherein, the first classification criterion is that the image block where the original pixel is located only includes horizontal and vertical edges, and the horizontal and vertical edges are axis-aligned.

[0046] Step 102: When the position of the original pixel in the source image does not meet the assumption of the first classification criterion, but meets the assumption of the second classification criterion, the original pixel is processed using the diagonal intersection filter; wherein, the second classification criterion is that the image block where the original pixel is located only includes the forward edge and the backward edge of the diagonal, and the forward edge intersects with the backward edge.

[0047] The following section will provide a detailed explanation of the applicable scenarios and processing logic of axis-aligned intersection filters and diagonal intersection filters.

[0048] Step 103: When the position of the original pixel in the source image does not conform to the assumptions of the first classification criterion and the second classification criterion, determine the parameterized overlay representation corresponding to the original pixel, and determine the intersection filter for processing the original pixel based on the parameterized overlay representation.

[0049] To illustrate the process of processing using contour filters and surface filters, such as Figure 7 As shown, step 103 includes: Step 1031: Based on the assumption that the position of the original pixel in the source image is between two adjacent surfaces, obtain the parameterized overlay representation corresponding to the original pixel.

[0050] Step 1032: When the parameterized overlay representation is less than a preset normal value, the original pixel is processed using the filament filter.

[0051] The default normal value can be 0.

[0052] Step 1033: When the parameterized overlay representation is greater than or equal to a preset normal value, the original pixel is processed using the contour filter.

[0053] Step 1034: When the position of the original pixel in the source image cannot be classified as either using a filament filter or a contour filter, the original pixel is processed using the surface filter.

[0054] In this embodiment of the invention, for original pixels that do not conform to the first and second classification criteria, a general contour filter is used for processing. During this process, a parameterized coverage representation of the coverage area of ​​the corresponding original pixel is first calculated. When the parameterized coverage representation is greater than or equal to 0, it indicates that the original pixel conforms to the assumption that it is located between two adjacent surfaces, and the processing logic of the contour filter continues to respond to the original pixel. When the parameterized coverage representation is less than 0, it indicates that the original pixel does not conform to the assumption that it is located between two adjacent surfaces, and as a special case in the process of using the contour filter, a filament filter is activated to respond to the original pixel. If neither the contour filter nor the filament filter is applicable to the original pixel, a surface filter is directly used for default amplification. The processing logic of the surface filter, filament filter, and contour filter will be described in detail below.

[0055] like Figure 8 As shown, step 20 includes: Step 201: When the intersection filter is a contour filter, a diagonal intersection filter, an axis-aligned intersection filter, or a filament filter, determine the parameterized coverage representation of the coverage area of ​​the original pixel; based on the parameterized coverage representation and the baseline vector, determine the corresponding amplification factor.

[0056] Step 202: When the intersecting filter is a surface filter, the baseline vector is scaled to obtain the amplification factor of the surface filter.

[0057] In summary, the embodiments of the present invention select intersecting filters in the following order: axis-aligned intersecting filter, diagonal intersecting filter, filament filter, profile filter, and surface filter.

[0058] In this embodiment of the invention, corresponding intersection filters are used to process the original pixels that meet various classification criteria differently in order to obtain the best approximation of the target image.

[0059] After applying different intersecting filters to each original pixel in the source image, the amplification coefficient of the corresponding intersecting filter in the corresponding dimension is obtained. The original pixels are then amplified based on the amplification coefficients in each dimension of each original pixel. The process is as follows: The eigenvalues ​​are Haar coefficients; for example Figure 9 As shown, step 30 includes: Step 301: Construct a complementary relationship between each eigenvalue decomposition coefficient and multiple target pixels; reconstruct the complementary relationship, and determine the expression of each target pixel based on the source image and the eigenvalue decomposition coefficients representing the multiple target pixels.

[0060] Step 302: Use the amplification factor of the selected intersection filter as a reliable substitute for the eigenvalue coefficients in the corresponding dimension of the expression.

[0061] Step 303: Based on the trusted substitution, determine the color values ​​of the corresponding multiple target pixels according to the color values ​​of the original pixels.

[0062] By combining multiple local region images composed of the above four magnified target pixels, the magnified target image is obtained.

[0063] Because image super-resolution algorithms require high detail reproduction, they often involve complex calculations, making it inconvenient to fully integrate them into hardware, thus significantly reducing their practical value in actual industrial production scenarios. To strike a balance between image quality and computational complexity to meet users' demands for higher visual fidelity, this invention minimizes the support for amplification filters. To illustrate the processing logic of surface filters and contour filters, this embodiment of the invention, based on the aforementioned methods for improving image visual fidelity, also provides a method for local feature amplification that preserves detail, specifically including: Step 40: Model the original pixels in the source image located between two adjacent surfaces to obtain contour parameter estimates; obtain the neighboring pixels adjacent to the original pixels from the surface, model the neighboring pixels to obtain surface parameter estimates.

[0064] The concept of a custom object in the method for enhancing local features while preserving details according to embodiments of the present invention will be explained below: The source image refers to the image whose resolution needs to be increased, and the original pixels refer to the pixels that make up the source image; a specific example of a source image is... Figure 2 As shown, Figure 2Each square in the diagram represents an original pixel. In this embodiment of the invention, the source image is magnified by a scaling factor of 2 to obtain a target image magnified by a factor of 2. The target image is composed of all the target pixels obtained after magnification. It should be noted that in the local feature magnification method for preserving details in this embodiment of the invention, the target pixel refers only to the pixel obtained by magnifying the original pixels in the source image that are suitable for surface filters and contour filters.

[0065] In this embodiment of the invention, 4×4 original pixels in the source image are processed each time. Hereinafter, the 4×4 original pixels processed each time are referred to as an image patch. A possible specific implementation of an image patch is as follows: Figure 2 As shown in the gray area; in an optional embodiment, a 4×4 sliding window can be used to process the source image; multiple image blocks can be processed in parallel to improve efficiency.

[0066] This invention is based on a local feature classification heuristic algorithm, using multiple filter responses to amplify local features in the source image by a scaling factor of 2. To achieve high-fidelity detail, the positional relationships between original pixels in the source image are abstracted into two surfaces and the original pixels located between them. For example... Figure 10 The image shown is a pixelated region representation of a source image provided by an embodiment of the present invention. Figure 10 The white and black parts represent smooth regions in the source image. Smooth regions refer to parts of the source image with continuous surface texture and no excessive detail. The pixel values ​​in these parts have small gradients, are relatively uniform and continuous. Figure 10 The black dashed box contains three adjacent color areas. Figure 10 The area where white and black meet represents a relatively discontinuous region in the source image where pixel value changes have a large gradient. A specific instance of a modelable local feature includes two adjacent surfaces (such as...). Figure 10 The middle arrow points to surfaces 1 and 2), and the single straight edge formed by the original pixels located between them (such as...). Figure 10 (As shown in the dashed bar box indicated by the middle arrow) Figure 10 The document provides two sets of concrete instances: one set consisting of surfaces 1 and 2 indicated by red arrows, and a single straight edge indicated by a red dashed box; the other set consisting of surfaces 1 and 2 indicated by black arrows, and a single straight edge indicated by a black dashed box. Surface 1 is... Figure 10 Surface 1 is a continuous smooth region located to the left of the single straight edge, and Surface 2 is a continuous smooth region located to the right of the single straight edge. The single straight edge is composed of multiple relatively continuous pixels located between Surface 1 and Surface 2.

[0067] Based on the above modeling assumptions, surface filters can be used to perform default upsampling on pixels on the surface (i.e., smooth regions) to determine surface parameter estimates; by combining surface filters and contour filters, local features of a single straight edge and its surrounding area can be extracted and preserved to determine contour parameter estimates.

[0068] To preserve local features in the source image as much as possible in the target image, a hybrid response using surface filters and contour filters is required. The relevant operations are explained below. In this embodiment of the invention, the contour filter is applied to pixels located on the boundary line between two adjacent smooth regions, such as... Figure 10 As shown in the red dashed box. Contour filters can construct a single straight edge, but they don't work well in all smooth areas, so they need to be used in conjunction with surface filters; the two are used together. In cases such as... Figure 10 When performing local feature magnification on the single straight edge shown (i.e., magnifying the pixels within the dashed box), not only the pixels on the single straight edge but also their neighboring pixels are considered to comprehensively determine the color value reference of the magnified target pixel. Pixels in surfaces 1 and 2 whose distance difference from the single straight edge is less than a preset value are selected as neighboring pixels and used to estimate the mixed response with the contour parameters generated by the single straight edge. The preset value is determined by those skilled in the art based on the current input block size and other factors in the specific application scenario. Contour filters and surface filters are the core of the image super-resolution algorithm of this invention. Only by achieving accurate local magnification of details in the source image based on these two filters can the visual fidelity of the target image be ensured.

[0069] Inspired by the box filtering method, this invention derives the reconstruction function of the contour filter. The design idea of ​​the contour filter is: for the original pixels between two adjacent surfaces, some simple parameters are used to represent the boundaries in the source image. Based on the boundary division, the pixels (i.e., color values) at the corresponding positions in the expanded target image are determined. By ensuring that these parameters are approximate values ​​that are effective in most cases, the computational complexity is reduced. The expression obtained based on the parameters is used to determine how to perform upsampling processing.

[0070] Step 50: Based on the contour parameter estimation and the corresponding surface parameter estimation, determine the magnification factor in at least one dimension of the original pixel.

[0071] In this embodiment of the invention, the amplification factor in the corresponding dimension is determined based on the eigenvalue decomposition coefficients; the eigenvalue decomposition coefficients can be Haar coefficients, which refer to the coefficients obtained after decomposing the image signal using Haar wavelets.

[0072] This invention obtains a reliable approximation of the target image based on the Haar wavelet transform method, with the amplification factor used to characterize this reliable approximation. With an amplification factor of 2, this embodiment of the invention obtains a sufficiently good approximation of the target image based on the Haar wavelet transform. Since the transformation matrix of the Haar wavelet transform is a sparse matrix, the computational cost is relatively small, making it very suitable for fast computation in real-time 3D image processing scenarios. This embodiment of the invention approximates the input signal (i.e., the source image) step by step, gradually decomposing the source image into sub-signals of different frequencies (i.e., local region images composed of target pixels). Specifically, for the input source image, it is split into four 1 / 4-fold local region images, each sub-signal consisting of a continuous segment of the input signal, and these sub-signals constitute different frequency components of the input signal. An image with the same pixel width of 2W and pixel height of 2H can be obtained from the source image. It is converted from the source image. Decomposed into (four magnified local region images) One-quarter of ), the specific expression is as follows: ; More Haar coefficients can be obtained by determining the complementary set of Haar coefficients:

[0073] in, , and For Haar coefficients, the complementary set of Haar coefficients refers to all other coefficients in the Haar wavelet transform besides the directly given Haar coefficients.

[0074] Based on the above expression, it is possible to base it on the source image. Reconstructed magnified image The details are as follows:

[0075]

[0076]

[0077]

[0078] The main function of this invention is to estimate the source image. All target pixels after magnification Or, to put it another way, to find a credible alternative, denoted as The Haar coefficients in the target image are denoted as... , and Calculate according to the following expression: ; Among them, such as Figure 3 As shown, , corresponding to a single original pixel m is the x-coordinate offset of the four magnified pixels in a 2×2 pixel matrix, and n is the y-coordinate offset of the four magnified pixels in a 2×2 pixel matrix. This represents a combination of contour filters and surface filters; , and These are the Haar coefficients (i.e., the eigenvalue coefficients). These are the magnification factors in the horizontal, vertical, and tilt dimensions, respectively.

[0079] Step 60: Magnify the original pixel according to the magnification factor to obtain multiple target pixels; combine the multiple target pixels to generate a target image.

[0080] The corresponding generated target pixels are as follows: ; ; ; ; By combining the above four target pixels, a magnified local region image is obtained, which is then used to generate the target image. .

[0081] This invention is based on the assumption that the location of a local feature in the source image can be modeled as two adjacent surfaces and the original pixel located between them. The original pixels corresponding to the surfaces undergo default upsampling processing, while the original pixels located between them are modeled using parametric methods to obtain a sufficiently good approximation of the magnified pixel. By combining the processing results of the adjacent surfaces and the original pixels located between them, a magnification factor representing a reliable substitute for the magnified local feature is obtained. A magnified image with good approximation effect can be generated through a single calculation. Furthermore, since this invention only determines the pixel boundary of the color value referenced by the magnified original pixel in a parametric form, and is entirely based on mathematical calculations, avoiding the use of iterative algorithms, it can be entirely embedded in hardware, making it highly practical and achieving high clarity and color reproduction.

[0082] In existing technologies, many solutions using AI (Artificial Intelligence) algorithms (or similar AI algorithms) inevitably involve iterative calculations. These solutions are often based on exhaustive methods, repeatedly iterating to determine relevant parameters, resulting in a massive computational workload. This invention flexibly balances the required accuracy of the target image with the quality of the result. Since it processes typical sRGB (standard Red Green Blue) images, each channel is 8 bits, there are no strict accuracy requirements. Both the software and hardware accuracy requirements are quite low. Therefore, this invention uses an approximation method, introducing a certain loss in the calculation to simplify the actual logical implementation and reduce costs. This invention uses a fixed function for approximate calculations, which is less costly than a more accurate implementation in terms of complexity. This greatly expands the applicability of the method for enhancing local features while preserving detail.

[0083] Before providing a detailed description of the local feature magnification method of the present invention, the relevant definitions in the embodiments of the present invention are explained as follows: The default upsampling operation performed on the surface filter is described below. In step 40, modeling the neighboring pixels to obtain surface parameter estimates includes: Step 401a: Reduce the horizontal baseline vector by a first preset factor to obtain the surface horizontal parameters of the neighboring pixels; reduce the vertical baseline vector by a first preset factor to obtain the surface vertical parameters of the neighboring pixels; reduce the tilted baseline vector by a second preset factor to obtain the surface tilt parameters of the neighboring pixels.

[0084] Here, the first preset multiple and the second preset multiple both represent magnification factors. The first preset multiple can be 8 and the second preset multiple can be 64.

[0085] The surface level parameters are: The surface vertical parameters are: The surface tilt parameters are: .

[0086] Step 402a: Use the surface horizontal parameter, the surface vertical parameter, and the surface tilt parameter as the surface parameter estimates of the neighboring pixels in the corresponding dimensions.

[0087] After the surface filter responds, it ultimately outputs a series of amplification coefficients in the corresponding dimensions.

[0088] The contour filter of the present invention will now be described in detail: The design philosophy of the contour filter is as follows: for an original pixel located on a single straight edge between two surfaces, determining how to upsample it only requires calculating the aerial coverage from one side to the other (i.e., the parameterized coverage representation below). This results in an expression that typically depends on at least two factors: parameterized direction estimation and the calculation of the parameterized coverage representation. Since both calculations are complex, this invention provides an approximation that is effective in most cases, improving practicality and ensuring the target image approximates the optimal result as closely as possible (i.e., guaranteeing that the image quality does not degrade relative to the source image). This invention is based on fundamental trigonometry, but only constructs the corresponding expression using explicit methods, without involving iterative convergence, making it suitable for hardware implementation.

[0089] Specifically, such as Figure 11 The diagram shows the parameterized direction estimation and corresponding parameterized coverage representation under three different conditions. The parameterized edge work performed by the contour filter is divided into two main parts: (1) determining the direction the edge faces, i.e., determining the parameterized direction estimation; and (2) determining the location of the edge relative to the pixel, i.e., determining the parameterized coverage representation. In step 40, the process of modeling the original pixels located between two adjacent surfaces in the source image to obtain the contour parameter estimation includes: Step 401b: Map the symbol information of adjacent pixel pairs in the current input block to the corresponding datum to construct the parameterized orientation estimate of the original pixel; wherein the original pixel is located at the center of the current input block, and the datum is based on the original pixel as its origin.

[0090] The current input block is the filtering kernel of the source image processed by the filter (such as a contour filter) in this embodiment of the invention, and its size is 3×3 pixels. Figure 4 As shown, for ease of calculation, a set of baseline vectors in each dimension are defined with the current input block as the reference; where the pixel in the center is gray to represent the original pixel, the pixels other than the original pixel are the neighboring pixels of the original pixel, and these neighboring pixels surround the original pixel.

[0091] An octant refers to eight parts in two-dimensional space divided by two sets of mutually perpendicular coordinate axes (X1 axis, Y1 axis; X2 axis, Y2 axis). Each part represents a specific spatial region, defined by the positive and negative directions of the coordinate axes. The method for dividing the octant and mapping symbolic information to the corresponding octant in this embodiment will be described in detail below. The symbolic information refers to the boundary of the color values ​​referenced when magnifying the original pixel, and the symbolic information is as follows: Figure 12 The symbols "+", "0" and "-" are shown on the middle pixel blocks.

[0092] Step 402b: Obtain the parameterized overlay representation of the original pixel based on the parameterized direction estimation.

[0093] Step 403b: Determine the corresponding contour parameter estimates based on the parameterized coverage representation and the parameterized direction estimation.

[0094] This invention divides the current input block into eight directional datums in two-dimensional space, enabling the complete encoding of the symbolic information required for reference by the original pixel and allowing the direction facing the edge to be mapped to a single datum. This results in a sufficiently good approximation of the magnified original pixel, reduces the complexity of parameter estimation, independently determines the parameterized direction estimation, and computes the parameterized overlay representation.

[0095] It should be noted that the contour filter of this embodiment of the invention is based on dividing the original pixel region into datums and mapping the symbols to a single datum. It then performs direction estimation for axisymmetric and diagonal boundaries, as well as coverage calculation, to obtain the amplification factor and ultimately achieve parameterized boundary approximation. However, the algorithm of the contour filter of this embodiment of the invention is not limited to datum region division in eight directions. For example, quadrilateral region division can handle axisymmetric boundaries, and more region divisions than datums (e.g., 16-sided) can handle boundaries located between axisymmetric and diagonal boundaries; see the following description for details.

[0096] To illustrate the process of determining the parameterized direction estimate, step 401b includes: Step 4011: Based on the average value of adjacent pixel pairs in the current input block, synthesize multiple surface probes; wherein, the adjacent pixel pairs are composed of two adjacent pixels in the current input block, excluding the original pixel.

[0097] To estimate color values ​​and edge orientation, eight surface probes were synthesized, such as... Figure 12 As shown. Eight surface probes. The expression is as follows: ; Among them, such as Figure 12 middle As shown, its corresponding adjacent pixel pair is the light blue pixel located at the upper right corner of the original pixel and the light blue pixel adjacent to it on the right; as Figure 12 middle As shown, the corresponding adjacent pixel pair is the magenta pixel located at the lower right corner of the original pixel and the magenta pixel adjacent to its right; and so on, using all the neighboring pixels of the original pixel, the surface probes are synthesized in pairs to facilitate parameterization of each direction around the original pixel.

[0098] It should be noted that, Figure 12 Each 3×3 pixel region (i.e., to Each intersection filter processes a portion of the source image at a time; the pixel at the very center of each surface probe is the original pixel. A symbol on each pixel region indicates the pixel index offset of a neighboring pixel relative to the original pixel, as shown in the image. Figure 3 As shown in the middle left figure. For example, with Figure 3 In the left image, p11 represents the 3×3 pixel area of ​​the original pixel. Pixel p00 (top right corner of the original pixel): its index is offset by "--" relative to the original pixel's index; pixel p22: its index is offset by "++" relative to the original pixel's index; pixel p02: its index is offset by "-+" relative to the original pixel's index; pixel p12: its index is offset by "0+" relative to the original pixel's index, and so on. These symbols are only used to indicate... Figure 3 In the left image, the pixel numbers of adjacent pixels are offset relative to the pixel numbers of the original pixels.

[0099] Step 4012: Construct multiple first basis vectors based on the difference between the orientations of the surface probes; construct multiple second basis vectors based on the squared difference of the orientations of the first basis vectors.

[0100] Among them, surface probes with opposite directions, such as Figure 13 First basis vector The surface probes represented by the green and light blue pixels, or the first base vector, are shown. The surface probes represented by the medium-deep purple and magenta pixels are shown below, and so on; the corresponding first basis vectors are as follows. Figure 13 As indicated by the white arrows in the middle.

[0101] A set of four first basis vectors is constructed based on the surface probe to estimate the edge direction, as shown in the following expression: ;in, .

[0102] By differencing the squares of these first basis vectors, a pair of second basis vectors is formed, such as... Figure 14 As shown, the expression for the second basis vector, used for all subsequent direction calculations, is: ;in, The second basis vector is defined in a clockwise direction (e.g., ...). Figure 14 As shown on the right), a direction defined counterclockwise (e.g.) Figure 14 (As shown on the left side of the middle section).

[0103] Step 4013: Based on the plurality of second basis vectors, divide the current circular region located at the center of the current input block into a plurality of limits to map the symbol information to the corresponding limits; within each limit, determine the parameterized direction estimate based on the second basis vectors.

[0104] One specific instance of the current circular region is as follows: Figure 14 The second basis vector on the left The circular area enclosed by the black dashed line represents the second basis vector. The circular region is formed by rotation. The parameterized orientation estimate is obtained based on the ratio of the second basis vector, and the specific process will be explained in detail below.

[0105] To illustrate the specific process of mapping symbolic information based on the second basis vector, step 4013 includes: Step 40131: According to the relative size relationship of the plurality of second basis vectors, selectively place the first edge normal in the first interval of odd order or the first interval of even order, so as to divide the corresponding current circular region into a plurality of first intervals.

[0106] like Figure 15 As shown, the left side represents the corresponding region divided in a clockwise direction, and the right side represents the corresponding region divided in a counter-clockwise direction. The second base vector, with the center of the original pixel as its origin, divides the current circular region into eight triangular first intervals. Based on the characteristic of dividing the first intervals using the two second base vectors, the first edge normal lies in either the odd-numbered interval or the even-numbered interval depending on the magnitude of the larger second base vector. The specific expression is as follows: ; At this point, the relative magnitudes of the second basis vectors change from a clockwise direction (i.e., " ) or counterclockwise (i.e. " The expression is as follows: ; The precise meaning is the position of the edge normal's hanging point in eight directions; such as Figure 14 and Figure 15 As shown, depending on the relative magnitude of the second base vector, the corresponding edge normal lies in either the odd-numbered triangle region or the even-numbered triangle region among the eight directional limits.

[0107] Figure 15 The middle left image represents the clockwise direction, i.e. Figure 14 The right image occupies a more prominent position compared to the left image. Therefore, the corresponding... Figure 14 middle The four odd-numbered triangular regions, in Figure 15The left image uses solid colors to represent, while the corresponding... Figure 14 middle The four even-numbered triangular regions, in Figure 15 The image on the left is represented by a semi-transparent color.

[0108] Similarly, Figure 15 The middle right image shows the counter-clockwise direction, that is... Figure 14 The left image occupies a more prominent position compared to the right image. Therefore, the corresponding... Figure 14 middle The four even-numbered triangular regions, in Figure 15 The image on the right is represented by solid color, while the corresponding... Figure 14 middle The four odd-numbered triangular regions, in Figure 15 The image on the right is represented by a semi-transparent color.

[0109] Based on the sign information of their respective second basis vectors, the corresponding first intervals are further divided into horizontal intervals (e.g., Figure 16 (as shown on the left) and vertical intervals (as shown on the right) Figure 16 (As shown on the right side of the middle section). Figure 16 The middle left image represents a region where the first edge normal is a horizontal triangle. . Figure 15 The left image shows the clockwise direction, i.e. ,so It occupies the main position, and its corresponding triangular area is in Figure 16 The image on the left is represented by solid color. The corresponding triangular region is in Figure 16 The image on the left is represented by a semi-transparent color.

[0110] Similarly, Figure 15 The right figure shows , ,so It occupies the main position, and its corresponding triangular area is in Figure 16 The image on the left is represented by solid color. The corresponding triangular region is in Figure 16 The image on the left is represented by a semi-transparent color.

[0111] Similarly, we can obtain Figure 16 The rightmost image shows a triangular region where the first edge normal is perpendicular. That is, in Figure 16 The triangular area represented by the semi-transparent color in the left image is... Figure 16 The image on the right is represented by solid color.

[0112] The positional characteristics of the four wind axes represent the sign information corresponding to the largest of the two second basis vectors; a horizontal positional characteristic of the four wind axes indicates that the largest second basis vector is greater than or equal to the sign conversion value, while a vertical positional characteristic indicates that the largest second basis vector is less than the sign conversion value; where the sign conversion value is 0. That is, the four wind axes of the first edge normal. The characteristics are as follows: ; Step 40132: Based on the sum of the symbol information corresponding to the first interval, selectively place the second edge normal in the second interval with an odd order or an even order, to obtain the eight-axis line of the second edge normal, so as to divide the corresponding current circular region into multiple second intervals; wherein, the first edge normal and the second edge normal differ by a preset angle, and the eight-axis line with the position feature in a diagonal state is used to characterize that the magnitude relationship between the multiple second basis vectors and the symbol conversion value is different, and the eight-axis line with the position feature in an axis state is used to characterize that the magnitude relationship between the multiple second basis vectors and the symbol conversion value is the same.

[0113] The preset angle is determined by those skilled in the art based on the specific application scenario; the current circular area is further divided into eight second intervals in a direction different from that used to divide the first interval. The second edge normals are added together based on their symbol information, and the second edge normals are located in either odd or even intervals; the eight axes of the second edge normals... The characteristics are as follows: ; Here, "diagonal" refers to the second edge normal being more approximately diagonal (e.g., ...). Figure 17 (As shown in the two figures on the right side of the middle section), "axial" means that the normal of the second edge is more approximately in an axial state (e.g., Figure 17 (As shown in the two images on the left side of the middle section); The "" represents the logical XOR operation, whose rule is "0 for the same, 1 for different", meaning that when the second basis vectors are different, the result is 0. The relative magnitude relationship with 0, and when the second basis vector When the relative size relationship with 0 is the same (e.g., and Sometimes, or and At that time, the corresponding eight wind axes are in an axial state.

[0114] Step 40133: In each of the eight wind axes, the ratio of the smallest second base vector to the largest second base vector is determined as the direction ratio; based on the positional characteristics of the eight wind axes, the parameterized direction estimate is determined based on the direction ratio.

[0115] The meaning of the edge normals in the eight quadrants in the embodiments of the present invention is explained below: The definition of the diagonal state is: ( ) = 1, that is and ,or and .

[0116] Combination Figure 13 The direction of the first basis vector shown and Figure 17 The triangular region shown, when First basis vector at time 0 Dominant; when First basis vector As the dominant state, satisfying the diagonal state condition. and The region is defined by the first basis vector. and the first basis vector The overlapping portion of the triangular region formed by the directions, relative to Figure 17 The white area in the bottom right corner of the image Similarly, when First basis vector As the dominant force, when First basis vector As the dominant state, satisfying the diagonal state condition. and The region is defined by the first basis vector. and the first basis vector The overlapping portion of the triangular region formed by the directions, relative to Figure 17 The white area in the upper right corner of the image.

[0117] Similarly, the conditions for the state of the horizontal axis At this time, the first basis vector and the first basis vector As the dominant force, relative to Figure 17 The white area in the top left corner of the image represents the conditions for the vertical axis state. First basis vector and the first basis vector As the dominant force, relative to Figure 17 The white area in the bottom left corner of the image.

[0118] The above parameters fully encode all symbolic information for each spatial direction centered on the original pixel, and allow direction calculations to be mapped to a single datum. In this embodiment of the invention, different approximations are used to calculate the tangent depending on whether the second edge normal is closer to the axis or the diagonal. Finally, the parameterized direction estimate is determined.

[0119] Specifically, within each of the eight wind axes, the ratio of the smallest second base vector to the largest second base vector is used as the direction ratio, and the expression for the direction ratio is: ; The method for calculating tangents is as follows: .

[0120] The purpose of the contour filter is to find the edge in the original pixel region. In this embodiment of the invention, the angle of the edge is determined by the position of the edge normal within the bounding field. The position of the edge normal within the bounding field determines the approximate direction of the edge (360° / 8), while... It is an approximate estimation of the edge normal direction.

[0121] In existing technologies, to ensure the accuracy of the target image, a very costly calculation is performed on the original pixels in the source image. This involves covering all scannable original pixels in the source image and continuously trying different parameters to determine which parameter will achieve the best effect. Since the parameters to be tried often have a continuous range and can be quantified to some extent, a variety of possible different choices are obtained, and their effects need to be verified through calculation. The computational cost of these verification calculation steps is extremely high, making this iterative method very impractical in real-world scenarios, especially when implemented in hardware, as it is impossible to fully integrate it into the hardware.

[0122] To drastically reduce the complexity of estimating these parameters, this invention proposes a computational method that divides the computation of the original pixels into two steps: determining the parameterized orientation estimate and computing the parameterized overlay representation; these two steps are performed independently.

[0123] This invention provides a specific derivation process to illustrate how the invention can significantly reduce the computational complexity of estimating these parameters, as follows: Ideally, embodiments of the present invention aim to separate the determination of parameterized direction estimation from the computation of parameterized cover representation in order to simplify the contour filtering algorithm. The ratio of the increments of the second basis vectors, as defined above, typically depends on the parameterized orientation estimation. and the smaller area coverage of a pair of surfaces on the source pixel (That is, parameterized overlay representation). This relationship will be explained below, and it will be shown that the overlay-independent approximation given above provides sufficient accuracy for the output target image in most cases.

[0124] Based on the symmetry of the problem, the embodiments of the present invention are limited to That is, the first counterclockwise angular interval, so that the following specific examples do not lose generality. In the derivation process, the basis vectors... It refers to the difference in color values ​​between two surfaces.

[0125] function yes and Piecewise functions; Because the edge intersects with different pixels within different coverage areas, it will affect the scale of the second basis vector with different trigonometric function relationships. Because the order in which these intersections occur changes depending on the direction. For example... Figure 18 As shown, in There are four sub-intervals, each with a different order of intersection points: like Figure 18 As shown, the first diagram from left to right represents The second diagram illustrates the situation. The third diagram illustrates the situation. The fourth diagram illustrates the situation. The situation.

[0126] Figure 18 The corresponding number in the code indicates the coverage area of ​​each vertex. The order of intersections from 1 / 2 to 0. The intervals between each intersection point follow different trigonometric formulas. The first two intervals, where... Within the defined error range, an axial approximation method will be used for calculation; the latter two intervals, among which The calculation will be performed using the diagonal approximation method (again, the approximation method will have some error).

[0127] The following is the derivation process of the axial approximation method: The embodiments of the present invention consider each of Intervals, to distinguish and The difference between them. For example Figure 19 As shown, the first interval cover (Note that this interval will be in) (disappears at the time) The following basis vectors are obtained: ; but, ; The above expression is ideal because it does not affect... The dependence of the molecule (i.e., the smallest second basis vector) in The "+" and "-" signs change periodically to align with axial features. It's important to note that as the axial alignment of the edge normals increases, the proportion of coverage within this interval also increases; and any ratio of the aforementioned base vector sizes depends on... It can be used as an alternative parameter to determine the parameterized direction estimate.

[0128] like Figure 20 As shown, the first interval cover (Note that this interval will be in) (disappears at the time) The following basis vectors are obtained: ; but, ; When the edge overflows into the base vector and basis vectors Some will be introduced Dependence, but In this case, the basis vector and basis vectors It can be used for another clean orientation parameterization.

[0129] like Figure 21 As shown, the second interval is cover (Note that this upper bound connects to the two preceding first intervals): The following basis vectors are obtained: ; but, ; When the edge continues to overflow into the base vector and basis vectors China Times, The dependency remains unchanged, but... In the case of basis vectors and basis vectors It can be used for another concise direction parameterization.

[0130] like Figure 22 As shown, the third and last interval cover ; The following basis vectors are obtained: ; but, ; Now the edge includes all the basis vectors, therefore, accompanied by more obvious Dependency, the zero point of the minimum basis vector deviation deviates from (This means that the signs of the basis vectors correspond to different ranges) depending on Since the coverage area gradually approaches zero, if we do not consider... Use in the case of coverage The approximation becomes increasingly inaccurate as the coverage approaches zero. However, as the coverage disappears, the direction estimation of the edge normal becomes less important. Since the magnification factor used in this embodiment is 2, if the edge intersects only one of the four generated sub-pixels, the importance of the direction estimation decreases, for example, At that time. It should be noted that this means the edge direction is more accurate when it is close to the horizontal and vertical axes (which is consistent with the fact that the edge intersects the basis vector less).

[0131] The embodiments of the present invention employ the following: Axial approximation method with parameters The expression is determined based on experience and is as follows: ; like Figure 23 As shown, by plotting this approximation For different coverage areas of The relationship of true dependence is used to stitch together the above segmented relationships. Obviously, it can fit well within the relevant coverage range.

[0132] The following is the derivation of the diagonal approximation method: Consider each one in turn Coverage Intervals, to distinguish and The difference between them.

[0133] like Figure 24 As shown, the first interval cover ; The following basis vectors are obtained: ; but, ; This is the first interval in the axial case. Similarly.

[0134] like Figure 25 As shown, the second interval is cover ; The following basis vectors are obtained: ; but, ; This interval is = The time includes the last interval The same problematic zero-point deviation, in = The widest time, in = Time and = It disappears.

[0135] like Figure 26 As shown, the third interval cover (Note that this upper bound connects to the second interval mentioned earlier): The following basis vectors are obtained: ; but, ; This is different from the axial case. The final interval is similar. This interval is in... = The widest time, in = It disappears.

[0136] like Figure 27 As shown, the third interval is cover (Note that this upper bound connects to the second interval mentioned earlier): The following basis vectors are obtained: ; but ; The symmetric behavior will tend to reduce the influence of The dependency, this interval is The widest time, at this time Completely covered.

[0137] like Figure 28 As shown, the fourth and last interval cover (Note that this upper bound connects to the two preceding third intervals): The following basis vectors are obtained: ; but ; when Make this interval as wide as possible, close to In the process, it gradually transitions to symmetrical behavior.

[0138] The embodiments of the present invention employ the following: Diagonal approximation method with parameters The expression is determined based on experience, and is as follows: ; like Figure 29 As shown, by plotting this approximation For different coverage areas of The relationship of true dependence is used to stitch together the above segmented relationships. Obviously, it can fit well within the relevant coverage range.

[0139] The process of determining the contour parameterized overlay representation is described below. Step 402b includes: Step 4021: Based on the positional characteristics of the four wind axes and the eight wind axes of the first edge normal, and the fact that the first interval corresponding to the original pixel is clockwise or counterclockwise, perform linear interpolation on two adjacent surface probes with overlapping directions in the synthesized surface probes to synthesize multiple positive and negative probes estimated along the parameterized direction.

[0140] The four wind axes are used to divide the corresponding first intervals into vertical or horizontal intervals according to the symbol information of the plurality of first intervals; the positional features of the four wind axes represent the symbol information corresponding to the largest second base vector among the plurality of second base vectors; the four wind axes with horizontal positional features represent that the largest second base vector is greater than or equal to the symbol conversion value, and the four wind axes with vertical positional features represent that the largest second base vector is less than the symbol conversion value.

[0141] The sum of symbol information corresponding to the first interval is used to selectively place the second edge normal in the second interval with an odd or even order, resulting in the eight-axis line of the second edge normal, which divides the corresponding current circular region into multiple second intervals. Among them, the first edge normal and the second edge normal differ by a preset angle, and the eight-axis line with a diagonal position is used to characterize the different magnitude relationships between the multiple second basis vectors and the symbol conversion value, while the eight-axis line with an axis position is used to characterize the same magnitude relationship between the multiple second basis vectors and the symbol conversion value.

[0142] After determining the parameterized orientation estimate, a pair of surface probes with opposite orientations along the parameterized orientation estimate are synthesized by linear interpolation of adjacent (i.e., oriented coincidentally) pairs of surface probes among the eight surface probes mentioned above (i.e., those with overlapping orientations). Figure 30 (The positive and negative probes are shown). Figure 12 and Figure 30 As shown, the negative probe is based on a surface probe. and surface probe Synthesis, surface probe With surface probe Adjacent pixels have a one-pixel overlap in size; positive probes are based on surface probes. and surface probe Synthesis; Surface probe With surface probe Adjacent pixels have a one-pixel overlap in size.

[0143] First, based on the rotation direction (clockwise or counterclockwise) of the second basis vector mentioned above, and the four-wind axis and eight-wind axis constructed in the corresponding rotation direction, determine the direction for linear interpolation. Figure 12 Eight surface probes index value It is defined according to the following expression: ; in, This indicates that the eight wind axes are in an axial state. This indicates a diagonal arrangement. This indicates that the four wind axes are in a horizontal state. This indicates that the four wind axes are in a vertical state. This indicates that the corresponding interval is divided based on the second basis vector in a clockwise direction. This indicates that the corresponding interval is divided based on the second basis vector in the counterclockwise direction.

[0144] Determine the linear interpolation factor based on whether the eight wind axes are diagonal or axial. The specific expression is: .

[0145] The expression for a negative probe is: ; in," " represents modulo operation, i.e., the remainder operation; The interpolation factor is linear; the interpolation parameter is 1. The expression for a positive probe is: .

[0146] Step 4022: Based on the difference between the surface probe and the corresponding negative probe, construct the corresponding negative coupling vector; based on the difference between the surface probe and the corresponding positive probe, construct the corresponding positive coupling vector; based on the difference between the positive probe and the corresponding negative probe, construct the corresponding contour vector.

[0147] To compute the parameterized overlay representation of the original pixels, the negative coupling vector also needs to be determined. and positive coupling vector The details are as follows: ; Then, based on these two factors, determine as follows: Figure 31 The outline vector indicated by the white arrow in the middle The expression is as follows: .

[0148] Step 4023: Determine the projection of the original pixel along the axis defined by the adjacent surface probe based on the negative coupling vector, the positive coupling vector and the contour vector; obtain a parameterized overlay representation based on the projection.

[0149] The parameterized overlay representation is determined based on the projection of the original pixel along the axis defined by the adjacent surface probe; the expression for the parameterized overlay representation is: ; Where i represents the horizontal pixel coordinate, and j represents the vertical pixel coordinate. For the corresponding positive coupling vector, For the corresponding negative coupling vector, Given the corresponding contour vector, the parameterized overlay is represented as the minimum value of surface region overlap.

[0150] In this embodiment of the invention, the position of a pixel relative to a neighboring surface probe is inferred based on the color of the inner (or outer) contour edge of the original pixel.

[0151] like Figure 32 As shown, this embodiment of the invention assumes that the parameterized direction estimate is located on the first octave in the counterclockwise direction, and then modulates the amplification coefficient of the contour filter to map it onto the true octave. Specifically, step 403b includes: Step 4031: Determine the first horizontal gradient parameter and the first vertical gradient parameter based on the parameterized overlay representation and the parameterized direction estimation; determine the first tilt gradient parameter based on the first horizontal gradient parameter and the first vertical gradient parameter.

[0152] like Figure 32As shown, the unmodulated first level gradient parameters Different segmentation behaviors occur depending on the edge direction. If it is... Then two intervals need to be considered; if it is Then we need to consider three intervals.

[0153] The expression for the first level gradient parameter is: ;in, This is a parameterized overlay representation for the corresponding pixel.

[0154] like Figure 33 As shown, the first water verticality parameter without modulation Different segmentation behaviors arise depending on the edge direction, requiring consideration of three intervals. The first unmodulated vertical gradient parameter... It is unrelated to parameterized direction estimation.

[0155] The expression for the first vertical gradient parameter is: .

[0156] The expression for the first inclined gradient parameter is: .

[0157] Step 4032: Depending on whether the forward or backward edge corresponding to the first tilt gradient parameter is maximally covered, selectively use the first tilt gradient parameter as the modulated first tilt gradient parameter, or use the negative number corresponding to the first tilt gradient parameter as the modulated first tilt gradient parameter.

[0158] The tilt gradient parameter is modulated based on whether the forward or backward edge has maximum coverage (the result is a sign reversal, as the sub-pixel positions with zero coverage are flipped to the diagonal). The specific expression is: ; Step 4033: Based on the positional characteristics of the four wind axes and the fact that the first interval corresponding to the original pixel is clockwise or counterclockwise, the second horizontal gradient parameter, the second vertical gradient parameter, and the second tilt gradient parameter are obtained based on the first horizontal gradient parameter, the first vertical gradient parameter, and the modulated first tilt gradient parameter.

[0159] By inverting the first vertical gradient parameter and the implicit first tilt gradient parameter, and by swapping the first horizontal gradient parameter and the first vertical gradient parameter estimated perpendicular to the parameterization direction, a set of second gradient parameters is modulated. Its expression is: ; Step 4034: Determine the product of the second horizontal gradient parameter and the horizontal baseline vector as the contour horizontal parameter, determine the product of the second vertical gradient parameter and the vertical baseline vector as the contour vertical parameter, and determine the product of the second tilt gradient parameter and the tilt baseline vector as the contour tilt parameter; use the contour horizontal parameter, the contour vertical parameter, and the contour tilt parameter as contour parameter estimates in their respective dimensions.

[0160] The expression for the contour level parameter is: ; The expression for the profile vertical parameter is: ; The expression for the profile tilt parameter is: .

[0161] To preserve as many smooth areas as possible in the image, the responses of the contour filter and the surface filter are mixed, and step 50 includes: Step 501: Determine the maximum activation value of each original pixel based on the relationship between the parameterized overlay representation of the original pixel and the first preset threshold and the second preset threshold.

[0162] The first preset threshold can be 1 / 8, and the second preset threshold can be 1 / 16. The expression for the maximum activation value is: ; It should be noted that, It is based on the 4 original pixels in 2×2 It is obtained from the maximum value of the two corresponding original pixels, so in the calculation The corresponding calculations should be performed first. .

[0163] Step 502: Estimate and extract the corresponding maximum activation value along the parameterization direction of the second original pixel; determine the contour activation function value based on the extracted maximum activation value according to the positional features of the four wind axes and the eight wind axes, and the direction of the second basis vector corresponding to the four wind axes; wherein, the contour activation function value represents the response allocation ratio corresponding to the contour filter, and the difference between the decomposition basis and the contour activation function value represents the response allocation ratio corresponding to the surface filter.

[0164] The decomposition cardinality is 1. This represents the contour activation function value, i.e., the response allocation ratio corresponding to the contour filter; Assign a proportion to the response corresponding to the surface filter. The contour activation function value is the result of the contour activation function calculation; contour activation function The expression for , is: ; Among them, such as Figure 3 As shown, , corresponding to a single original pixel m is the x-coordinate offset of the four magnified pixels in a 2×2 pixel matrix, and n is the y-coordinate offset of the four magnified pixels in a 2×2 pixel matrix. This indicates that the eight axes corresponding to the original pixel are in an axial state. This represents the four wind axes corresponding to the original pixel. This indicates that the corresponding interval is divided based on the second basis vector in a clockwise direction. This indicates that the corresponding interval is divided based on the second basis vector in the counterclockwise direction.

[0165] Step 503: According to the response allocation ratio, based on the contour parameter estimation and the surface parameter estimation, generate a reliable feature decomposition substitution corresponding to the second original pixel.

[0166] Step 504: Determine the coefficients of the credible substitution of the feature decomposition in the corresponding dimension as the second amplification coefficient in the corresponding dimension.

[0167] The expression for the reliable substitution of the feature decomposition is: ; Among them, the horizontal magnification factor Vertical magnification factor and tilt magnification factor This is the magnification factor corresponding to the first interval region. The contour activation function value represents the surface level parameter. Surface vertical parameters and surface tilt parameters For surface parameter estimation, Contour horizontal parameters Contour vertical parameters and The profile tilt parameter is used to estimate the profile parameters.

[0168] like Figure 34 The image shown is a target image magnified twofold, generated from a source image through conventional bilinear interpolation. For the same source image, based on the assumption that the original pixels lie between two adjacent surfaces, processing it with common filters (contour filter and surface filter) yields a target image magnified twofold, as shown below. Figure 35 As shown. Comparison Figure 34 and Figure 35 You can see the effect produced by the contour filter. Figure 35 The black and white boundary lines in the middle are more Figure 34 The image is clearer; additionally, due to the effect of the surface filter, Figure 35 In the white area Figure 34 It's smoother in the middle.

[0169] When it is necessary to achieve a high resolution in the magnified target image without sacrificing the quality of the source image, it is essential to restore the local features containing various details in the source image with high quality. However, existing image super-resolution algorithms cannot specifically process two regions that are difficult to distinguish as smooth or as single-boundary intervals, resulting in poor detail restoration of the target image with such interval regions. For example, on a smooth background, a line with a width of one pixel has pixels whose adjacent pixels on both sides have the same color. Therefore, the color of a pixel on the line should not be determined by the average color of its adjacent pixels. To solve this problem, embodiments of the present invention use a filament filter to handle the exceptions that occur when using a contour filter. To illustrate the processing logic of the filament filter and the contour filter, embodiments of the present invention, based on the aforementioned method for improving image visual fidelity, also provide a method for improving the resolution of interval regions, including: Step 70: Classify the original pixels in the source image to determine the first original pixel and the second original pixel.

[0170] The concept of custom objects in the embodiments of the present invention will be explained below: The source image refers to the image whose resolution needs to be increased, and the original pixels refer to the pixels that make up the source image; a specific example of a source image is... Figure 2 As shown, Figure 2 Each square in the diagram represents an original pixel. In this embodiment of the invention, the source image is magnified by a scaling factor of 2 to obtain a target image magnified by a factor of 2. The target image is composed of all the target pixels obtained after magnification. It should be noted that in the method for improving the resolution of the interval region in this embodiment of the invention, the target pixel refers only to the pixel obtained by magnifying the original pixels in the source image that are suitable for filament filters, surface filters, and contour filters.

[0171] In this embodiment of the invention, 4×4 original pixels in the source image are processed each time. Hereinafter, the 4×4 original pixels processed each time are referred to as an image patch. A possible specific implementation of an image patch is as follows: Figure 2 As shown in the gray area; in an optional embodiment, a 4×4 sliding window can be used to process the source image; multiple image blocks can be processed in parallel to improve efficiency.

[0172] like Figure 36As shown, the second interval region can be modeled as two surfaces and a single straight edge between them. For the original pixel corresponding to the single straight edge between the two surfaces, this embodiment of the invention uses some simple parameters to represent the various boundaries in the source image. Based on the boundary division, the pixel (i.e., color value) at the corresponding position in the expanded target image is determined. By ensuring that these parameters are approximate values ​​that are valid in most cases, the computational complexity is reduced, and the expression obtained based on the parameters determines how to perform upsampling processing.

[0173] To achieve high-fidelity detail, the positional relationships between original pixels in the source image are abstracted into two surfaces and the original pixels located between them. For example... Figure 36 The image shown is a pixelated region representation of a source image provided by an embodiment of the present invention. Figure 36 The large white and black areas represent smooth regions in the source image. Smooth regions refer to parts of the source image with continuous surface texture and no excessive detail. The pixel values ​​in these parts have small gradients, are relatively uniform and continuous. Figure 36 The adjacent regions between the white and black areas represent relatively discontinuous parts of the source image where pixel value changes have significant gradients. A specific instance of a modelable local feature includes two adjacent surfaces (such as...). Figure 36 The solid arrow points to surfaces 1 and 2), and a single straight edge formed by the original pixels located between them (such as...). Figure 36 (As shown in the dashed bar box of the second interval region pointed to by the solid arrow). Surface 1 is... Figure 36 Surface 1 is a continuous smooth region located to the left of the single straight edge, and Surface 2 is a continuous smooth region located to the right of the single straight edge. The single straight edge is composed of multiple relatively continuous pixels located between Surface 1 and Surface 2.

[0174] Based on the above modeling assumptions, surface filters can be used to perform default upsampling on pixels on the surface (i.e., smooth regions) to determine surface parameter estimates; by combining surface filters and contour filters, local features of a single straight edge and its surrounding area can be extracted and preserved to determine contour parameter estimates.

[0175] To preserve local features in the source image as much as possible in the target image, a hybrid response of surface filters and contour filters is needed. The following explains the related operations. Contour filters can construct a single straight edge, but they don't work well in all smooth regions, so they need to be used in conjunction with surface filters; the two are used together. In cases such as... Figure 36 When performing local feature magnification on a single straight edge as shown (i.e., magnifying the original pixels located on that single straight edge), it is necessary to consider not only the pixels located on the single straight edge, but also the pixels in its neighborhood (such as...). Figure 36 The shadow surrounding a single straight edge within the dashed bar frame of the second interval region (shown in the image) serves as a reference for comprehensively determining the color value of the magnified target pixel. Pixels in surfaces 1 and 2 whose distance difference from the single straight edge is less than a preset value are selected as neighboring pixels and used to estimate the mixed response with the contour parameters generated by the single straight edge. The preset value is determined by those skilled in the art based on factors such as the current input block size in the specific application scenario. Contour filters and surface filters are the core of the image super-resolution algorithm of this invention. Only by achieving precise local magnification of details in the source image based on these two filters can the visual fidelity of the target image be ensured.

[0176] Inspired by the box filtering method, this invention derives the reconstruction function of the contour filter. The design idea of ​​the contour filter is: for the original pixels between two adjacent surfaces, some simple parameters are used to represent the boundaries in the source image. Based on the boundary division, the pixels (i.e., color values) at the corresponding positions in the expanded target image are determined. By ensuring that these parameters are approximate values ​​that are effective in most cases, the computational complexity is reduced. The expression obtained based on the parameters is used to determine how to perform upsampling processing.

[0177] However, since the original pixels are not necessarily continuous, some regions in the source image may not be modeled as two smooth regions adjacent by a single straight edge, nor are they smooth. Therefore, using contour filters and surface filters will yield poor results. Because the response of a contour filter relies on the assumption that the original pixel lies between two adjacent surfaces—that is, assuming the existence of two adjacent surfaces—original pixels causing discontinuities should be assumed to have a value between the two. However, some finer structures may not be suitable for this assumption, such as the width of a single pixel. Embodiments of this invention use filament filters to attempt to identify these cases, producing results similar to the response of a baseline filter, and incorporating some weighting functions (which can be adjusted by those skilled in the art according to the application without inventive effort). To apply different targeted processing methods to original pixels with different local features to ensure the fidelity of each region in the target image, it is necessary to classify the original pixels.

[0178] The first original pixel is the original pixel that cannot be modeled as located between two adjacent surfaces, and the first gap region is the region that cannot be modeled as the original pixel between two adjacent surfaces. Figure 36This document presents a concrete example of a first-interval region and a concrete example of a second-interval region. These regions contain pixels with different gray levels, meaning the color values ​​of the pixels in these regions are discontinuous, making modeling impossible based on this assumption. Instead, the original pixel that can be modeled based on this assumption is the second original pixel, and the region that can be modeled based on this assumption is the second-interval region. It should be noted that both the second original pixel and the second-interval region are approximations; the corresponding original pixels are not required to perfectly conform to the assumption (e.g., as in...). Figure 36 In the second interval region, there may be a small number of discontinuous regions.

[0179] This invention first uses a contour filter to process data based on the modeling method, and then... When the calculation determines that the corresponding hypothesis is not true, the response of the filament filter is activated as an exception to handle the special case separately. Instead of using general methods to handle special cases, special approximation processing is performed on this type of special case, thereby significantly improving the quality of the magnified target image.

[0180] Step 80: Model the first original pixel and the second original pixel respectively, determine the first magnification factor in at least one dimension corresponding to the first original pixel, and determine the second magnification factor in at least one dimension corresponding to the second original pixel.

[0181] With a magnification factor of 2, this embodiment of the invention obtains a sufficiently good approximation of the target image based on Haar wavelet transform. Since the transformation matrix of the Haar wavelet transform is a sparse matrix, the computational cost is relatively small, making it very suitable for fast computation in real-time 3D image processing scenarios. This embodiment of the invention approximates the input signal (i.e., the source image) step by step, gradually decomposing the source image into sub-signals of different frequencies (i.e., local region images composed of target pixels). Specifically, for the input source image, it is split into four 1 / 4 times the size of the local region images. Each sub-signal consists of a continuous segment of the input signal, and these sub-signals constitute different frequency components of the input signal. An image with the same pixel width of 2W and pixel height of 2H can be obtained from the source image. It is converted from the source image. It is divided into one-quarter portions (of the four magnified local region images), as shown in the following expression: ; More Haar coefficients can be obtained by determining the complementary set of Haar coefficients: ; in, , and The Haar coefficients are the coefficients obtained by decomposing the image signal using the Haar wavelet; the complementary set of the Haar coefficients refers to all other coefficients in the Haar wavelet transform besides the directly given Haar coefficients.

[0182] Based on the above expression, the source image can be reconstructed. The details are as follows: ; ; ; ; Step 90: Magnify the first original pixel according to the first magnification factor, and magnify the second original pixel according to the second magnification factor to generate the corresponding target image.

[0183] The main function of this invention is to estimate the source image. All target pixels P after magnification Or, to put it another way, to find a credible alternative, denoted as The Haar coefficients corresponding to each part of the target image are denoted as follows: , and When using a filament filter response, the corresponding expression for the first target region in the target image, magnified from the first interval region, is: ;in, Indicates a filament filter; This is the first amplification factor.

[0184] When using surface and contour filters, the corresponding expression for the second target region in the target image, magnified from the second interval region, is: ; Among them, such as Figure 3 As shown, , corresponding to a single original pixel m is the x-coordinate offset of the four magnified pixels in a 2×2 pixel matrix, and n is the y-coordinate offset of the four magnified pixels in a 2×2 pixel matrix. This represents a combination of contour filters and surface filters; , and These are the characteristic decomposition coefficients; This is the second amplification factor.

[0185] This invention obtains reliable replacements for each feature decomposition of the target image based on the Haar wavelet transform method, and the amplification factor is used to characterize the reliable replacement of the feature decomposition.

[0186] The reliable substitution based on the feature decomposition obtained from the magnification factor is taken as part of the target image, denoted as . This leads to the generation of four local region images based on the corresponding target pixels; when using contour filters and surface filters for the response, the reliable substitution is... When using contour filters and surface filters for the response, the reliable alternative is... The four local region images are as follows: ; ; ; ; By combining the above four local region images, the magnified target image is obtained. .

[0187] This invention is based on the assumption that the location of a gap region in the source image can be modeled as two adjacent surfaces and the original pixel located between them. The original pixels are classified, and then processed independently for a first gap region in the source image that is not applicable to this assumption and a second gap region that is applicable. This yields a first magnification factor for the first gap region and a second magnification factor for the second gap region. The corresponding gap regions are then magnified according to their respective magnification factors to generate corresponding reliable replacements. Specifically, this invention can generate a reliable replacement with good approximation effect for each type of gap region with a single calculation, thus achieving a high-resolution image. Furthermore, because the first gap region in the source image is directly identified during classification and treated as a special case, the magnification method used for continuous pixels is avoided, further improving the resolution of the portion of the target image magnified from the first gap region.

[0188] Before providing a detailed description of the detection and upsampling methods in the embodiments of the present invention, the relevant definitions in the embodiments of the present invention are explained as follows: To illustrate the specific response process of the contour filter, step 70 includes: Step 701: Determine the parameterized overlay representation corresponding to the original pixel.

[0189] In this context, the raw pixels currently being processed are referred to as raw pixels to be classified; the specific number of raw pixels to be classified is determined by those skilled in the art based on the specific use case and the number of pixels that can be processed in parallel each time.

[0190] The embodiments of the present invention first calculate the parameterized overlay representation based on the assumption that the position of the interval region in the source image can be modeled as two adjacent surfaces and the original pixel located between them.

[0191] Step 702: When the parameterized overlay representation corresponding to the original pixel is less than a preset normal value, the corresponding original pixel is classified as the first original pixel.

[0192] The default normal value can be 0. The corresponding expression for the judgment is: ; in, This is the parameterized overlay representation of the original pixels to be classified.

[0193] When the parameterized overlay representation corresponding to the original pixel to be classified is less than 0, the original pixel to be classified is determined to be unable to be modeled based on the assumption. That is, the original pixel to be classified is identified as the first original pixel and the interval region in which it is located is the first interval region.

[0194] Step 703: When the parameterized overlay representation corresponding to the original pixel is greater than or equal to a preset normal value, the corresponding original pixel is classified as a second original pixel.

[0195] When the parameterized overlay representation corresponding to the original pixel to be classified is greater than or equal to 0, the original pixel to be classified is determined to be suitable for modeling under this hypothesis. That is, the original pixel to be classified is identified as the second original pixel, and the interval region in which it is located is the second interval region.

[0196] The process of determining the parameterized overlay representation of the original pixels is described below, and step 701 includes: Step 7011: Map the symbol information of each adjacent pixel pair in the current input block to the corresponding saddle to construct a parameterized orientation estimate of the original pixel to be classified; wherein the original pixel to be classified is located at the center of the current input block, and the saddle is centered on the original pixel to be classified.

[0197] The current input block is the filtering kernel of the source image processed by the filter (such as a contour filter) in this embodiment of the invention, and its size is 3×3 pixels. Figure 4 As shown, for ease of calculation, a set of baseline vectors in each dimension are defined with the current input block as the reference; where the pixel in the center is gray to represent the original pixel, the pixels other than the original pixel are the neighboring pixels of the original pixel, and these neighboring pixels surround the original pixel.

[0198] An octant refers to eight parts in two-dimensional space divided by two sets of mutually perpendicular coordinate axes (X1 axis, Y1 axis; X2 axis, Y2 axis). Each part represents a specific spatial region, defined by the positive and negative directions of the coordinate axes. The method for dividing the octant and mapping symbolic information to the corresponding octant in this embodiment will be described in detail below. The symbolic information refers to the boundary of the color values ​​referenced when magnifying the original pixel, and the symbolic information is as follows: Figure 12 The symbols "+", "0" and "-" are shown on the middle pixel blocks.

[0199] Step 7012: Based on the assumption that the position of the original pixel to be classified in the source image is between two adjacent surfaces, the parameterized overlay representation of the original pixel to be classified is obtained according to the parameterized direction estimation.

[0200] In determining the parameterized direction estimate, eight surface probes are synthesized, a pair of second basis vectors is determined, and four-wind axes and eight-wind axes are constructed on the corresponding pair of second basis vectors, as detailed below. After determining the parameterized direction estimate, a pair of surface probes with opposite directions along the parameterized direction estimate are synthesized by linear interpolation of adjacent (i.e., directionally overlapping) pairs of surface probes among the eight surface probes (i.e., such as...). Figure 30 (The positive and negative probes are shown below). Figure 12 and Figure 30 As shown, the negative probe is based on a surface probe. and surface probe Synthesis, surface probe With surface probe Adjacent pixels have a one-pixel overlap in size; positive probes are based on surface probes. and surface probe Synthesis; Surface probe With surface probe Adjacent pixels have a one-pixel overlap in size.

[0201] First, based on the rotation direction of the second basis vector (clockwise or counterclockwise), and the four-wind axis and eight-wind axis constructed in the corresponding rotation direction, the linear interpolation is determined. Figure 12 Eight surface probes index value It is defined according to the following expression: ; in, This indicates that the eight wind axes are in an axial state. This indicates a diagonal arrangement. This indicates that the four wind axes are in a horizontal state. This indicates that the four wind axes are in a vertical state. This indicates that the corresponding interval is divided based on the second basis vector in a clockwise direction. This indicates that the corresponding interval is divided based on the second basis vector in the counterclockwise direction.

[0202] Determine the linear interpolation factor based on whether the eight wind axes are diagonal or axial. The specific expression is: .

[0203] The expression for a negative probe is: ; in," " represents modulo operation, i.e., the remainder operation; The interpolation factor is linear; the interpolation parameter is 1. The expression for a positive probe is: .

[0204] To compute the parameterized overlay representation of the original pixels, the negative coupling vector also needs to be determined. and positive coupling vector The details are as follows: ; Then, based on these two factors, determine as follows: Figure 31 The outline vector indicated by the white arrow in the middle The expression is as follows: .

[0205] The parameterized overlay representation is determined based on the projection of the original pixel along the axis defined by the adjacent surface probe; the expression for the parameterized overlay representation is: ; Where i represents the horizontal pixel coordinate, and j represents the vertical pixel coordinate. For the corresponding positive coupling vector, For the corresponding negative coupling vector, Given the corresponding contour vector, the parameterized overlay is represented as the minimum value of surface region overlap.

[0206] The process of determining the parameterized direction estimate is described in detail below. Step 7011 includes: Step 70111: Based on the average value of adjacent pixel pairs in the current input block, synthesize multiple surface probes; wherein, the adjacent pixel pairs are composed of two adjacent pixels in the current input block, excluding the original pixels to be classified.

[0207] This invention reduces the complexity of parameter estimation by using a sufficiently good approximation, and independently determines the parameterized direction estimation and computes the parameterized cover representation.

[0208] First, to estimate the surface color and edge orientation, eight surface detectors were synthesized. The expression is as follows: ; Among them, such as Figure 12 middle As shown, its corresponding adjacent pixel pair is the light blue pixel located at the upper right corner of the original pixel and the light blue pixel adjacent to it on the right; as Figure 12 middle As shown, the corresponding adjacent pixel pair is the magenta pixel located at the lower right corner of the original pixel and the magenta pixel adjacent to its right; and so on, using all the neighboring pixels of the original pixel, the surface probes are synthesized in pairs to facilitate parameterization of each direction around the original pixel.

[0209] Step 70112: Construct multiple first basis vectors based on the difference between the orientations of the surface probes; construct multiple second basis vectors based on the squared difference of the orientations of the first basis vectors.

[0210] Among them, surface probes with opposite directions, such as Figure 13 First basis vector The surface probes represented by the green and light blue pixels, or the first base vector, are shown. The surface probes represented by the medium-deep purple and magenta pixels are shown below, and so on; the corresponding first basis vectors are as follows. Figure 13 As indicated by the white arrows in the middle.

[0211] A set of four first basis vectors is constructed based on the surface probe to estimate the edge direction, as shown in the following expression: ;in, .

[0212] By differencing the squares of these first basis vectors, a pair of second basis vectors is formed, such as... Figure 14 As shown, the expression for the second basis vector, used for all subsequent direction calculations, is: ;in, The second basis vector is defined in a clockwise direction (e.g., ...). Figure 14As shown on the right), a direction defined counterclockwise (e.g.) Figure 14 (As shown on the left side of the middle section).

[0213] One specific instance of the current circular region is as follows: Figure 14 The second basis vector on the left The circular area enclosed by the black dashed line represents the second basis vector. The circular area formed by rotation.

[0214] Step 70113: Based on the plurality of second base vectors, the current circular region formed by the second base vectors is divided using the four-wind axis and the eight-wind axis to map the symbol information to the corresponding mounting limit.

[0215] like Figure 15 As shown, the left side represents the corresponding region divided in a clockwise direction, and the right side represents the corresponding region divided in a counter-clockwise direction. The second base vector, with the center of the original pixel as its origin, divides the current circular region into eight triangular first intervals. Based on the characteristic of dividing the first intervals using the two second base vectors, the first edge normal lies in either the odd-numbered interval or the even-numbered interval depending on the magnitude of the larger second base vector. The specific expression is as follows: ; At this point, the relative magnitudes of the second basis vectors are represented by clockwise or counterclockwise directions, and the corresponding expressions are as follows: ; Based on the sign information of their respective second basis vectors, the corresponding first intervals are further divided into horizontal intervals (e.g., Figure 16 (as shown on the left) and vertical intervals (as shown on the right) Figure 16 (As shown on the right side of the middle section). The positional characteristics of the four wind axes represent the sign information corresponding to the largest of the two second basis vectors; a four wind axis with a horizontal positional characteristic is used to indicate that the largest second basis vector is greater than or equal to the sign conversion value, and a four wind axis with a vertical positional characteristic is used to indicate that the largest second basis vector is less than the sign conversion value; where the sign conversion value is 0. That is, the four wind axes of the first edge normal. The characteristics are as follows: ; Using a direction different from that used to divide the first interval, the current circular region is further divided into eight second intervals. The second edge normals are added together based on their sign information; the second edge normals lie in either odd or even intervals. The eight wind axes of the second edge normals... The characteristics are as follows: ; Here, "diagonal" refers to the second edge normal being more approximately diagonal (e.g., ...). Figure 17 (As shown in the two figures on the right side of the middle section), "axial" means that the normal of the second edge is more approximately in an axial state (e.g., Figure 17 (As shown in the two images on the left side of the middle section); The "" represents the logical XOR operation, whose rule is "0 for the same, 1 for different", meaning that when the second basis vectors are different, the result is 0. The relative magnitude relationship with 0, and when the second basis vector When the relative size relationship with 0 is the same (e.g., Sometimes, or At that time, the corresponding eight wind axes are in an axial state.

[0216] Step 70114: Within each limit, determine the parameterized direction estimate based on the second basis vector.

[0217] The above parameters fully encode all symbolic information for each spatial direction centered on the original pixel, and allow direction calculations to be mapped to a single datum. In this embodiment of the invention, different approximations are used to calculate the tangent depending on whether the second edge normal is closer to the axis or the diagonal. Finally, the parameterized direction estimate is determined.

[0218] Specifically, within each of the eight wind axes, the ratio of the smallest second base vector to the largest second base vector is used as the direction ratio, and the expression for the direction ratio is: ; The method for calculating tangents is as follows: .

[0219] After obtaining the parameterized orientation estimate of the original pixel according to the above process, the corresponding parameterized coverage range can be determined. Then, based on whether the parameterized coverage range is greater than 0, the first original pixel located in the first interval region can be identified and treated as a special case to activate the filament filter response. The second original pixel obtained by classification is processed using general surface filter and contour filter responses to complete the targeted processing of different types of interval regions.

[0220] The process of processing the first interval region using a filament filter is described below. In step 80, determining the first magnification factor in at least one dimension corresponding to the first original pixel includes: Step 801a: When the absolute value of the parameterized overlay representation of the first original pixel is less than the third preset threshold, the absolute value is used as the amplification gradient parameter; when the absolute value is not less than the third preset threshold, the third preset threshold is used as the amplification gradient parameter.

[0221] The third preset threshold can be 1 / 8. The expression for the amplification gradient parameter of the filament filter is: .

[0222] Step 802a: The product of the horizontal baseline vector and the amplified gradient parameter is used as the horizontal parameter of the filament; the product of the vertical baseline vector and the amplified gradient parameter is used as the vertical parameter of the filament; the square of the amplified gradient parameter is determined, and the product of the inclined baseline vector and the square of the gradient is used as the inclined parameter of the filament.

[0223] Horizontal parameters of the filament: ; Wire vertical parameters: ; Wire tilt parameters: .

[0224] Step 803a: Use the horizontal parameter, vertical parameter, and tilt parameter of the filament as the first magnification factor in the corresponding dimension.

[0225] The process of processing the second interval region using contour filters and surface filters is described below. In step 80, determining the second magnification factor in at least one dimension corresponding to the second original pixel includes: Step 801b: Determine the contour parameter estimate of the second original pixel based on the parameterized overlay representation of the second original pixel and the parameterized orientation estimate.

[0226] like Figure 32 As shown, in this embodiment of the invention, the parameterized direction estimate is assumed to be located on the first octave in the counterclockwise direction, and then the amplification coefficient of the contour filter is modulated to map it onto the real octave.

[0227] The first level gradient parameter is then obtained, and its expression is: ;in, This is a parameterized overlay representation for the corresponding pixel.

[0228] like Figure 33 As shown, the first water verticality parameter without modulation Different segmentation behaviors arising from different edge directions require consideration of three intervals. The first unmodulated vertical gradient parameter... It is unrelated to parameterized direction estimation.

[0229] The expression for the first vertical gradient parameter is: .

[0230] The expression for the first inclined gradient parameter is:

[0231] The tilt gradient parameter is modulated based on whether the forward or backward edge has maximum coverage (the result is a sign reversal, because the sub-pixel positions with zero coverage are flipped to the diagonal). The specific expression for modulating the tilt gradient parameter is: ; By inverting the first vertical gradient parameter and the implicit first tilt gradient parameter, and by swapping the first horizontal gradient parameter and the first vertical gradient parameter estimated perpendicular to the parameterization direction, a set of second gradient parameters is modulated. Its expression is: ; Finally, the expression for the contour level parameter is: The expression for the profile vertical parameter is: The expression for the profile tilt parameter is: .

[0232] Step 802b: Obtain neighboring pixels adjacent to the second original pixel from the two adjacent surfaces, model the position of the neighboring pixels, and obtain surface parameter estimates.

[0233] The surface filter performs a default upsampling operation on neighboring pixels located in the smooth region, and uses the surface horizontal parameter, surface vertical parameter, and surface tilt parameter as the surface parameter estimates for the neighboring pixels in their respective dimensions; the surface horizontal parameter is: The surface vertical parameters are: The surface tilt parameters are: .

[0234] Step 803b: Based on the contour parameter estimation and the corresponding surface parameter estimation, obtain the second magnification factor in at least one dimension of the second original pixel.

[0235] Since it is important to preserve the smooth regions in the source image as much as possible in the target image, when determining the magnified target pixels corresponding to the second interval region, the contour activation function is mixed with the response of the contour filter; that is, the contour activation function is used to mix the responses of the contour filter and the surface filter. Specifically, step 803b includes: Step 8031: Determine the response allocation ratio of the contour filter and the surface filter to the second original pixel.

[0236] When discontinuous edges exist in the second interval region (these discontinuous edges are formed by the smooth region that needs to be processed by the surface filter and the region that needs to be processed by the contour filter), the contour activation function value (i.e., the response allocation ratio) is determined according to the relationship between the parameterized coverage representation and the first and second preset thresholds, so as to mix the responses of the contour filter and the surface filter to the original pixel based on the contour activation function. The first preset threshold can be 1 / 8, and the second preset threshold can be 1 / 16. The process of using the contour activation function will be explained in detail below.

[0237] Step 8032: According to the response allocation ratio, generate a reliable feature decomposition substitution corresponding to the second original pixel based on the contour parameter estimation and the surface parameter estimation.

[0238] Step 8033: Determine the coefficients of the credible substitution of the feature decomposition in the corresponding dimension as the second amplification coefficient in the corresponding dimension.

[0239] The expression for credible substitution based on eigenvalue decomposition is: ; Among them, the horizontal magnification factor Vertical magnification factor and tilt magnification factor This is the magnification factor corresponding to the second interval region. The contour activation function value represents the surface level parameter. Surface vertical parameters and surface tilt parameters For surface parameter estimation, Contour horizontal parameters Contour vertical parameters and The profile tilt parameter is used to estimate the profile parameters.

[0240] The process of using the contour activation function is described in detail below. Step 8031 ​​includes: Step 80311: Determine the maximum activation value of each of the second original pixels based on the relationship between the parameterized overlay representation of the second original pixel and the first preset threshold and the second preset threshold.

[0241] First, calculate the contour activation function for each second original pixel, i.e., the maximum activation value: ; Based on the expression for the contour activation function above, if the coverage area... Since the value is close to zero, the second original pixel is definitely not an equal mixture of its neighboring pixels (affected by quantization noise), so it is assigned a higher contour activation value (saturating it to units). When the coverage area... When approaching halfway, neighboring source pixels far from the edge must be referenced. If the area to be processed is a smooth region, embodiments of the invention expect this neighboring region to be a nearly equal blending region, covering an area of... Approximately half. Conversely, if the point to be processed is a discontinuity, embodiments of the invention will expect coverage of at least one adjacent region. It's close to zero. In other words, if we consider a pair of roughly normal edges, we only expect pixels where the edges intersect; this means that non-intersecting pixels will exhibit lower coverage. (When it is a series of discontinuous images with minimal spacing, it should be approximately equal to the coverage of the neighboring pixels.) ).

[0242] Step 80312: Estimate and extract the corresponding maximum activation value along the parameterization direction of the second original pixel; determine the contour activation function value based on the extracted maximum activation value according to the positional features of the four wind axes and the eight wind axes, and the direction of the second basis vector corresponding to the four wind axes; wherein, the contour activation function value represents the response allocation ratio corresponding to the contour filter, and the difference between the decomposition basis and the contour activation function value represents the response allocation ratio corresponding to the surface filter.

[0243] The decomposition cardinality is 1. This represents the contour activation function value, i.e., the response allocation ratio corresponding to the contour filter; This represents the response allocation ratio for the surface filter.

[0244] Contour activation function ,in It is calculated at the aligned 2×2 source pixel granularity, estimating the maximum activation value of each second original pixel along the parameterization direction (to the nearest datum). The expression for the contour activation function is: ; in, This indicates that the eight axes corresponding to the original pixel are in an axial state. This represents the four wind axes corresponding to the original pixel. This indicates that the corresponding interval is divided based on the second basis vector in a clockwise direction. This indicates that the corresponding interval is divided based on the second basis vector in the counterclockwise direction.

[0245] It is important to note that in the diagonal configuration, only one pair of comparable adjacent pixels exists, while in the horizontal and vertical configurations, there are rows and columns of active pairs, respectively.

[0246] Using contour filters and surface filters to process the source image is a general processing method for the source image in this embodiment of the invention. Based on the assumption that the original pixels are located between two adjacent surfaces, when only general contour filters and surface filters are used for processing, the resulting target image is as follows: Figure 37 As shown. To further enhance the quality of the upsampled image, contour filters and surface filters are used to process pixels that fit the hypothesis, while filament filters are used to process discontinuous pixels that do not fit the hypothesis; in contrast... Figure 37 and Figure 38 Due to the increased response of the filament filter, for example... Figure 38 For the first interval region within the middle circle, the method for improving the resolution of the interval region according to the embodiments of the present invention can restore the discontinuous black pixels located in the center according to the source image; such as Figure 37 As shown in the corresponding position, when only the contour filter and surface filter response are used (i.e., the first interval region is used as the second interval region response), the local detail features contained in the source image will be lost, and the multiple discontinuous black pixels cannot be preserved in the target image.

[0247] The source image contains a large number of original pixels arranged diagonally within image patches. Ignoring these special cases and processing them uniformly often results in poor resolution improvement. To address this issue, when the position of an original pixel in the source image does not meet the first classification criterion for using an axis-aligned intersection filter, this embodiment of the invention tests whether the original pixel meets the second classification criterion for using a diagonal intersection filter. If it does, the diagonal intersection filter is used to respond to the corresponding original pixel. To illustrate the processing logic of the diagonal intersection filter, this embodiment of the invention, based on the aforementioned method for improving image visual fidelity, provides a method for detecting and upsampling diagonal features in a single image, including: Step 100: Detect the image block based on the coverage value of the edge involved in the image block in the source image, and determine the original pixels located on the diagonal.

[0248] The concept of custom objects in the embodiments of the present invention will be explained below: The source image refers to the image that needs to be enlarged, and the original pixels refer to the pixels that make up the source image; a specific example of a source image is... Figure 2 As shown, Figure 2Each square in the diagram represents an original pixel. In this embodiment of the invention, the source image is magnified by a scaling factor of 2 to obtain a target image magnified by a factor of 2. The target image is composed of all the target pixels obtained after magnification. It should be noted that in a single-image diagonal feature detection and upsampling method according to this embodiment of the invention, the target pixel refers only to the pixel obtained by magnifying the original pixels in the source image that are suitable for a diagonal intersection filter.

[0249] To achieve high-fidelity detail, this invention abstracts the positional relationships between original pixels in the source image into two planes and a single straight edge between them. This allows for targeted upsampling of the planes (i.e., smooth regions) by default, and precise magnification of the local features of the single straight edge and its surrounding area. Unlike the aforementioned contour filter, both the diagonal intersection filter and the axis-aligned intersection filter (described later) are used to process two perpendicularly intersecting boundaries.

[0250] The single-image diagonal feature point detection and upsampling method of this invention mainly targets the special case of two straight edges located on the diagonal (including only the front and back edges of the diagonal), providing an image super-resolution algorithm to process such local details. Specifically, it provides a diagonal intersection filter specifically designed for processing aligned 2×2 image blocks in the source pixels. Figure 39 The image shows a typical example located on a diagonal: four adjacent regions divided by two diagonals intersecting at right angles, such as... Figure 39 As shown in the figure. The diagonal intersection filter of this invention is an intersection filter used to process two diagonal boundaries that intersect at right angles in a source image; as shown in the figure. Figure 39 As shown, there are two intersecting diagonal boundaries in the 4×4 pixel region of the source image. Therefore, based on the processing of one boundary in the contour filter, it is extended to process two boundaries in the 4×4 pixel region of the source image using an axis-aligned intersection filter and a diagonal intersection filter. For the axis-aligned intersection filter and the diagonal intersection filter, the condition of two intersecting boundaries on the original pixel must be met simultaneously; otherwise, it cannot be processed as an intersecting boundary.

[0251] If the condition for the axis-aligned intersection filter is not met, the diagonal intersection filter is tested. If the condition for the diagonal intersection filter is not met, a filamentary filter with a boundary line is further tested. If the condition for the filamentary filter is not met, a contour filter is further tested. If the condition for the contour filter is not met, the default surface filter is determined.

[0252] For the original pixel corresponding to a single straight edge between two planes, some simple parameters are used to represent the boundaries in the source image. Based on the boundary division, the pixel (i.e., color value) at the corresponding position in the expanded target image is determined. By ensuring that these parameters are approximate values ​​that are valid in most cases, the computational complexity is reduced. The expression obtained from the parameters is used to determine how to perform upsampling.

[0253] Diagonal intersection filters produce better quality results near the diagonal intersection edges, but perform worse elsewhere. Therefore, minimizing the possibility of false positives is crucial. A false positive refers to a pixel that should have been classified as being amplified using a filter designed for other special cases or a general filter, but is incorrectly classified as being amplified using the diagonal intersection filter; a false negative is the opposite. According to the detection method of this invention, ideally, too many false negatives should not be generated. The corresponding classification criteria will be described in detail below.

[0254] Step 110: Model the original pixels in the southeast and northeast directions to determine the magnification factor in at least one dimension.

[0255] This invention obtains a reliable substitution of the feature decomposition of the target image based on the Haar wavelet transform method, and the amplification factor is used to characterize this reliable substitution. Since the image processing scheme of this invention is based on the assumption that the relationship between the original pixels can be modeled as two planes and a single straight edge between them, the processing of details where the corresponding diagonals intersect can be achieved using diagonal intersection filters on four planes divided by two vertical boundaries.

[0256] With a magnification factor of 2, this embodiment of the invention obtains a sufficiently good approximation of the target image based on Haar wavelet transform. Since the transformation matrix of the Haar wavelet transform is a sparse matrix, the computational cost is small, making it very suitable for fast computation in real-time 3D image processing scenarios. This embodiment of the invention approximates the input signal (i.e., the source image) step by step, gradually decomposing the source image into sub-signals of different frequencies (i.e., local region images composed of target pixels). Specifically, for the input source image, it is split into 4 regional images, each sub-signal consisting of a continuous segment of the input signal, and these sub-signals constitute different frequency components of the input signal. An image with the same pixel width of 2W and pixel height of 2H can be obtained from the source image. It is converted from the source image. The image is divided into four quarters (four magnified regions), as shown in the following expression: ; More Haar coefficients can be obtained by determining the complementary set of Haar coefficients: ; in, , and For Haar coefficients, the complementary set of Haar coefficients refers to all other coefficients in the Haar wavelet transform besides the directly given Haar coefficients.

[0257] Based on the above expression, it is possible to base it on the source image. Reconstructed magnified image The details are as follows: ; ; ; ; The main function of this invention is to estimate the source image. All target pixels after magnification Or, to put it another way, to find a credible alternative, denoted as The Haar coefficients in the target image are denoted as... , and Calculate according to the following expression: ; Among them, such as Figure 3 As shown, , corresponding to a single original pixel m is the x-coordinate offset of the four magnified pixels in a 2×2 pixel matrix, and n is the y-coordinate offset of the four magnified pixels in a 2×2 pixel matrix. This represents a diagonal intersection filter; , and These are the characteristic decomposition coefficients; This is the magnification factor.

[0258] Step 120: Magnify the original pixel according to the magnification factor to obtain multiple target pixels; combine the multiple target pixels to generate a target image.

[0259] The reliable substitution based on the amplification factor As the target image The resulting four target pixels are as follows: ; ; ; ; By combining the images from the four regions above, we obtain the magnified target image. .

[0260] This invention models the local features of the original pixels in the source image in the southeast and northeast directions, and filters out regions that can be classified as being located on the diagonal. Based on parametric modeling, the amplification coefficient of the local features in the upsampled target image is obtained, and an upsampled image with a good approximation effect can be generated through a single calculation. In particular, since this invention only uses parametric representation to determine the pixel boundary of the color value referenced for upsampling the original pixels, it is entirely based on mathematical calculations that do not involve iteration, so it can all be solidified in hardware, making it highly practical and able to achieve high clarity and color reproduction.

[0261] To detect the location of original pixels in the source image and selectively process original pixels located on the diagonal using a diagonal intersection filter, this embodiment of the invention provides three tests: uniformity test, invalidity check, and sign consistency test. The comprehensive classification result obtained from the tests is used to activate the diagonal intersection filter at the granularity of aligned 2×2 image blocks in the source image, thereby using the diagonal intersection filter to complete the response to the corresponding original pixels. Specifically, step 100 includes: Step 1001: Compare each edge in the image block; based on the first amplified southeast vector, the first southeast tolerance, the second amplified southeast vector, and the second southeast tolerance, determine the absolute value of the difference in the southeast edge coverage of adjacent pixel pairs, and the southeast tolerance corresponding to the image block; based on the first amplified northeast vector, the first northeast tolerance, the second amplified northeast vector, and the second northeast tolerance, determine the absolute value of the difference in the northeast edge coverage of adjacent pixel pairs, and the northeast tolerance corresponding to the image block; determine whether each absolute value within the image block does not exceed the corresponding tolerance. This represents the first amplified vector in the southeast direction. This represents the second southeast vector after magnification. This represents the first amplified vector from the northeast direction. This represents the second amplified vector from the northeast.

[0262] When performing uniformity testing on image patches in the source image, due to the geometric characteristics of the diagonal, there is often no complete neighborhood set in the diagonal direction, so only one comparison can be performed for each edge direction.

[0263] The original pixel and its neighboring pixels in the southeast direction are identified as adjacent pixel pairs. The absolute value of the difference in the coverage range of the southeast edge of the adjacent pixel pair is taken as the coverage value difference of the edge corresponding to the original pixel and the neighboring pixel in the southeast direction in the southeast direction. The original pixel and its neighboring pixels in the northeast direction are identified as adjacent pixel pairs. The absolute value of the difference in the coverage range of the northeast edge of the adjacent pixel pair is taken as the coverage value difference of the edge corresponding to the original pixel and the neighboring pixel in the northeast direction in the northeast direction.

[0264] The difference in coverage value between adjacent pixel pairs in the northeast direction (the corresponding coverage value is rescaled according to the segmentation elimination method) is as follows: ; This allows us to determine the corresponding Northeast tolerance. Among them, the northeast tolerance is the first-order error range in the northeast direction.

[0265] Similarly, the difference in southeast edge coverage values ​​for adjacent pixel pairs in the southeast direction (the corresponding coverage values ​​are rescaled according to the segmentation elimination method) is as follows: ; Southeast tolerance .

[0266] If, in an aligned 2×2 image patch, the absolute values ​​of the differences in the coverage areas of the southeast edges of (single) adjacent pixel pairs and the absolute values ​​of the differences in the coverage areas of the northeast edges of adjacent pixel pairs do not exceed their respective first-order error ranges, then the image patch passes the uniformity test. The expression for this judgment is: .

[0267] It should be noted that without the northeast and southeast tolerances, very few image patches can pass the uniformity test; and in a space with uniform pixel error, the expression for the error boundary may be simplified, but this is not the case for images encoded with sRGB.

[0268] Step 1002: Determine the edge coverage values ​​of adjacent pixel pairs in the southeast direction and adjacent pixel pairs in the northeast direction within the image block respectively; determine whether all edge coverage values ​​exceed the error threshold in the corresponding direction.

[0269] Since each edge is compared only once, to avoid numerous false positives, it's crucial to ensure that these classification results don't easily pass due to the presence of null vectors. Therefore, all edge coverage values ​​must exceed their corresponding error thresholds; specifically, the edge coverage values ​​of adjacent pixel pairs in the northeast direction must exceed the northeast error threshold, and the edge coverage values ​​of adjacent pixel pairs in the southeast direction must exceed the southeast error threshold. For this, the image patch passes the invalidity check. The expression for this check is as follows: Among them, the southeast error threshold Northeast Error Threshold .

[0270] Step 1003: In the southeast and northeast directions, for the original pixel located at the center of the diagonal corresponding to the image block, determine whether the symbol information of the original pixel at the center is the same as that of the neighboring pixels; wherein, the color value of the magnified pixel corresponding to the original pixel at the center is determined by the color value of the neighboring pixels.

[0271] Another simple classification test to be performed is to check whether the sign information of the coverage between each center pixel position and its selected displacement neighbor pixels (i.e. which side the edge is closest to) is consistent.

[0272] Specifically, the expression for determining whether the symbol information in the southeast direction is the same is: ; The expression for determining whether the symbol information in the corresponding northeast direction is the same is: ; When the corresponding symbol information is the same in both the southeast and northeast directions, the original pixel at the center has the same symbol information as its neighboring pixels. Therefore, the image block passes the symbol consistency test. The corresponding expression for this judgment is: .

[0273] Step 1004: When the absolute values ​​do not exceed the corresponding tolerance, all edge coverage values ​​exceed the error threshold in the corresponding direction, and the symbol information is the same, the original pixel is located on the diagonal.

[0274] When the uniformity test, invalidity test, and sign consistency test all pass, the original pixels are classified as special cases located on the diagonal, and the diagonal intersection filter response is used to achieve the best upsampling effect; for the original pixels Classification results The expression is as follows: .

[0275] The following describes the special processing scheme of the diagonal intersection filter for the original pixels. Step 110 includes: Step 1101: Determine the color value of the magnified pixel corresponding to the original pixel based on the color values ​​of the neighboring pixels in the southeast and northeast directions of the original pixel.

[0276] The key difference between original pixels located diagonally and other special cases based on two axis-aligned intersection boundaries (e.g., axis-aligned intersection) is that the calculation of the overlay value automatically deviates from the pixel center. To determine the corresponding parameterized overlay representation and prevent image quality degradation, it is necessary to select neighboring pixels based on displacement as a reference for the upsampled color value. Because in the diagonal state, if there is no difference between the original pixel and its neighboring pixels, it is inconvenient to borrow surface probes from neighboring pixels in the original 2×2 pixels along the diagonal direction for selection (for original pixels in an axis-aligned intersection state, this can be done). To avoid the difference between the original pixel and its neighboring pixels being 0 in the diagonal direction, we can select adjacent pixels in the 2×2 pixels that are offset diagonally from the original pixel to calculate the pixel color difference and overlay value, i.e., the pixel... and pixels Offset between, or pixels and pixels Offset between.

[0277] It should be noted that in the embodiments of the present invention, an axis-aligned intersection filter is first used, in which the calculation of the coverage value will conditionally deviate from the pixel center; while in the subsequent diagonal intersection filter, the calculation of the coverage value will unconditionally deviate from the pixel center.

[0278] Step 1102: Determine the parameterized overlay representation corresponding to the original pixel based on the color value of the magnified pixel.

[0279] In the process of determining the parameterized overlay representation in this embodiment of the invention, the importance of the estimated value of the parameterized overlay representation is first evaluated based on the error threshold in the corresponding direction: if the color difference exceeds the given error threshold, the corresponding parameterized overlay representation is given based on the corresponding projection; otherwise, it is forced to zero.

[0280] Step 1103: Obtain the amplification factor based on the parameterized overlay representation.

[0281] After the diagonal intersecting filter responds, it ultimately outputs a series of amplification coefficients.

[0282] Specifically, step 1101 includes: Step 11011: In the northeast direction, when the color value of the original pixel is closer to the color value of the northeast neighboring pixel of the original pixel, the color value of the northeast neighboring pixel is used as the color value of the magnified pixel; when the color value of the original pixel is closer to the color value of the southwest neighboring pixel of the original pixel, the color value of the southwest neighboring pixel is used as the color value of the magnified pixel.

[0283] like Figure 40 As shown, to simplify subsequent expressions, this embodiment of the invention defines an intermediate parameter vector, namely the Northeast intermediate vector. In the northeast direction, the color value of the magnified pixel is represented by the northeast center vector; The expression for the northeast intermediate vector is: ; Where i represents the horizontal pixel coordinate, and j represents the vertical pixel coordinate. This is the first amplified vector from the northeast. for The corresponding first northeast tolerance, This is the second amplified vector from the northeast. for The corresponding second northeast tolerance, It is a vertical negative coupling vector. It is a horizontal negative coupling vector. It is the vertical negative error coupling vector. This is the vertical positive error coupling vector. The horizontal positive error coupling vector, It is a horizontal positive coupling vector. It is a vertical positive coupling vector. The horizontal negative error coupling vector, The northeast positive coupling vector, The northeast negative coupling vector. The southeast positive coupling vector, The southeast negative coupling vector, This is the northeast positive error coupling vector. This represents the negative error coupling vector to the northeast. It should be noted that in all embodiments of this invention, z represents the southeast direction and w represents the northeast direction.

[0284] Step 11012: In the southeast direction, when the color value of the original pixel is closer to the color value of the southeast neighboring pixel of the original pixel, the color value of the southeast neighboring pixel is used as the color value of the magnified pixel; when the color value of the original pixel is closer to the color value of the northwest neighboring pixel of the original pixel, the color value of the northwest neighboring pixel is used as the color value of the magnified pixel.

[0285] like Figure 41 As shown, similarly, this embodiment of the invention also defines another intermediate parameter vector, namely the southeast intermediate vector. Its expression is: .

[0286] To determine the parameterized coverage representations in the southeast and northeast directions, the parameterized coverage representations include a southeast coverage representation and a northeast coverage representation; step 202 includes: Step 11021: Determine whether the color value of the magnified pixel exceeds the northeast error threshold; if it exceeds the northeast error threshold, then determine the projection of the original pixel along the axis defined by the northeast neighboring pixels as the northeast coverage representation corresponding to the original pixel; if it does not exceed the northeast error threshold, then the northeast coverage representation is a preset invalid value.

[0287] The default invalid value is 0. The expression for the above judgment implemented within the diagonal intersection filter can be: .

[0288] Step 11022: Determine whether the color value of the magnified pixel exceeds the southeast error threshold; if it exceeds the southeast error threshold, then determine the projection of the original pixel along the axis defined by the southeast neighboring pixels as the southeast coverage representation corresponding to the original pixel; if it does not exceed the southeast error threshold, then the southeast coverage representation is a preset invalid value.

[0289] The expression for the above judgment implemented within the diagonal intersection filter can be: .

[0290] To determine the corresponding magnification factor, step 203 includes: Step 11031: Determine the southeast amplified gradient parameter based on the southeast coverage representation; determine the northeast amplified gradient parameter based on the northeast coverage representation; the expression for the northeast amplified gradient parameter is: ;in, This indicates coverage of the Northeast.

[0291] The southeast amplification gradient parameters are determined in the same way, and their expression is as follows: .

[0292] Step 11032: Based on the difference between the k-fold southeast coverage representation and the southeast amplified gradient parameter, obtain the initial southeast tilt gradient parameter; based on the difference between the k-fold northeast coverage representation and the northeast amplified gradient parameter, obtain the initial northeast tilt gradient parameter; when the first amplified southeast vector is greater than or equal to a preset invalid value, use the negative of the initial southeast tilt gradient parameter as the target southeast tilt gradient parameter; when the first amplified southeast vector is less than a preset invalid value, use the initial southeast tilt gradient parameter as the target southeast tilt gradient parameter; when the first amplified northeast vector is greater than or equal to a preset invalid value, use the negative of the initial northeast tilt gradient parameter as the target northeast tilt gradient parameter; when the first amplified northeast vector is less than a preset invalid value, use the initial northeast tilt gradient parameter as the target northeast tilt gradient parameter.

[0293] The expression for the initial southeast slope gradient parameters is: ; The expression for the initial northeast slope gradient parameters is: ; The expression for the target's southeast slope gradient parameter is: ; The expression for the target's northeast slope gradient parameter is: .

[0294] Step 11033: Based on whether the first amplified northeast vector is less than a preset invalid value, selectively perform linear interpolation on the southeast positive coupling vector and the southeast negative coupling vector based on the northeast coverage representation to obtain the southeast edge vector; based on whether the first amplified southeast vector is less than a preset invalid value, selectively perform linear interpolation on the northeast positive coupling vector and the northeast negative coupling vector based on the southeast coverage representation to obtain the northeast edge vector.

[0295] Southeast edge vector From the relative reference vector and Utilizing the coverage area of ​​the northeastern edge, and in accordance with The relevant symbol information is used to perform linear interpolation between them, thereby approximating the central mean. The specific expression is as follows: ; Similarly, the expression for the northeast edge vector is: .

[0296] Step 11034: The product of the northeast amplified gradient parameter and the northeast edge vector is used as the first amplified intermediate value; the product of the southeast amplified gradient parameter and the southeast edge vector is used as the second amplified intermediate value; the product of the target northeast tilt gradient parameter and the northeast edge vector is used as the third amplified intermediate value; the product of the target southeast tilt gradient parameter and the southeast edge vector is used as the fourth amplified intermediate value; the sum of the first amplified intermediate value and the second amplified intermediate value is used as the diagonal horizontal parameter; the difference between the first amplified intermediate value and the second amplified intermediate value is used as the diagonal vertical parameter; the sum of the third amplified intermediate value and the fourth amplified intermediate value is used as the diagonal tilt parameter; the diagonal vertical parameter, the diagonal horizontal parameter, and the diagonal tilt parameter are used as amplification coefficients in their respective dimensions.

[0297] The expression for the magnification factor is as follows: ; ; .

[0298] Based on the assumption of a single straight edge between the two planes, when only contour filters and surface filters are used for processing, the resulting upsampled image is as follows: Figure 42 As shown; to further enhance the quality of the upsampled image, a filament filter is used on top of the contour filter and surface filter to process discontinuous pixels that do not conform to the assumption of a single straight edge between two planes. The specific processing area is shown in the figure. Figure 43 The image features within the middle circle are shown; further, in Figure 43 The upsampled image generated by adding the diagonal intersection filter mentioned above is as follows: Figure 44 As shown, the corresponding pixels within the circle visually exhibit a diagonal intersection feature, which is used in... Figure 42 and Figure 43 Based on the above three types of filters, and combining the responses of the diagonal intersection filter pairs in the embodiments of the present invention, the following is obtained: Figure 44 The targeted processing results shown are in Figure 44 The black and white intersecting area within the middle circle is filtered using a diagonal intersection filter. Figure 44 The black and white intersection boundary in the middle is compared to Figure 43 The corresponding position is clearer, with fewer gray error pixels.

[0299] It should be noted that the detection and upsampling of diagonal features of a single image in this invention are not limited to use in single image super-resolution algorithms. The relevant technical features can be transferred to video super-resolution algorithms, etc. Those skilled in the art can implement this invention by referring to the embodiments of this invention without creative effort.

[0300] The source image contains a large number of original pixels that are axially aligned in the horizontal and vertical directions. If the characteristics of these special cases are ignored and they are processed uniformly, the final resolution improvement effect is often poor. To solve this problem, embodiments of the present invention use an axis-aligned intersection filter to respond to the corresponding original pixels. To illustrate the processing logic of the axis-aligned intersection filter, embodiments of the present invention, based on the aforementioned method for improving image visual fidelity, also provide a method for determining and amplifying axis-aligned edge pixels, including: Step 130: Based on the coverage value of the edges involved in the image patch in the source image, detect the image patch and determine the original pixels whose edges are in axis-aligned state.

[0301] The concept of custom objects in the embodiments of the present invention will be explained below: The source image refers to the image that needs to be enlarged, and the original pixels refer to the pixels that make up the source image; a specific example of a source image is... Figure 2 As shown, Figure 2 Each square in the diagram represents an original pixel. In this embodiment of the invention, the source image is magnified by a scaling factor of 2 to obtain a target image magnified by a factor of 2. The target image is composed of all the target pixels obtained after magnification. It should be noted that, in one method for determining and magnifying axis-aligned edge pixels in this embodiment of the invention, the target pixel refers only to the pixel obtained by magnifying the original pixels in the source image that are suitable for an axis-aligned intersection filter.

[0302] To achieve high-fidelity detail, this invention abstracts the positional relationships between original pixels in the source image into two planes and a single straight edge between them. This allows for targeted upsampling of the planes (i.e., smooth regions) by default, and precise magnification of the local features of the single straight edge and its surrounding area. The method for determining and magnifying axis-aligned edge pixels in this invention primarily addresses the special case of two straight edges being axis-aligned, providing an image super-resolution algorithm for processing such local details. Specifically, it provides an axis-aligned intersection filter specifically designed for processing aligned 2×2 image blocks in the source pixels. Figure 45 The image shows a typical example of an axis-aligned configuration: the surface is divided into four adjacent surfaces by two intersecting boundaries at right angles.

[0303] For the original pixels corresponding to the two straight edges between the four planes, some simple parameters are used to represent the boundaries in the source image. Based on the boundary division, the pixels (i.e., color values) at the corresponding positions in the expanded target image are determined. By ensuring that these parameters are approximate values ​​that are valid in most cases, the computational complexity is reduced. The expression obtained from the parameters is used to determine how to perform upsampling.

[0304] Axis-aligned intersection filters produce good quality results near the edges of axis-aligned intersection, but poor results elsewhere. Therefore, minimizing the possibility of false positives is crucial, and ideally, avoiding the introduction of excessive false negatives as well. False positives refer to original pixels that should have been classified as amplified using filters designed for other special cases or general applications, but are incorrectly classified as being amplified using an axis-aligned intersection filter; false negatives are the opposite. The corresponding classification criteria will be elaborated in detail below.

[0305] Step 140: Model the original pixels in the horizontal and vertical directions to determine the magnification factor in at least one dimension.

[0306] This invention obtains a reliable substitution of the feature decomposition of the target image based on the Haar wavelet transform method, and the amplification factor is used to characterize this reliable substitution. Since the image processing scheme of this invention is based on the assumption that the relationship between the original pixels can be modeled as two planes and a single straight edge between them, the processing of the corresponding axis-aligned intersection details can be achieved using axis-aligned intersection filters on four planes divided by two vertical boundaries.

[0307] With a magnification factor of 2, this embodiment of the invention obtains a sufficiently good approximation of the target image based on Haar wavelet transform. Since the transformation matrix of the Haar wavelet transform is a sparse matrix, the computational cost is relatively small, making it very suitable for fast computation in real-time 3D image processing scenarios. This embodiment of the invention decomposes the input signal (i.e., the source image) into sub-signals of different frequencies (i.e., local region images composed of target pixels) by progressively approximating the input signal (i.e., the source image). Specifically, for the input source image... The image is divided into four 1 / 4-fold regions, each consisting of a continuous segment of the input signal. These sub-signals constitute different frequency components of the input signal. An identical image with a pixel width of 2W and a pixel height of 2H can be derived from the source image. It is converted from the source image. The image is divided into four quarters (four magnified regions), as shown in the following expression: ; More Haar coefficients can be obtained by determining the complementary set of Haar coefficients: ; in, , and For Haar coefficients, the complementary set of Haar coefficients refers to all other coefficients in the Haar wavelet transform besides the directly given Haar coefficients.

[0308] Based on the above expression, it is possible to base it on the source image. Reconstructed magnified image The details are as follows: ; ; ; ; The main function of this invention is to estimate the source image. All target pixels after magnification Or, to put it another way, to find a credible alternative, denoted as The Haar coefficients in the target image are denoted as... , and Calculate according to the following expression: ; Among them, such as Figure 3 As shown, , corresponding to a single original pixel m is the x-coordinate offset of the four magnified pixels in a 2×2 pixel matrix, and n is the y-coordinate offset of the four magnified pixels in a 2×2 pixel matrix. Indicates an axis-aligned intersecting filter; , and These are the characteristic decomposition coefficients; This is the magnification factor.

[0309] Step 150: Magnify the original pixel according to the magnification factor to obtain multiple target pixels; combine the multiple target pixels to generate a target image.

[0310] The reliable substitution based on the amplification factor As the target image The resulting four target pixels are as follows: ; ; ; ; By combining the images from the four regions above, we obtain the magnified target image. .

[0311] This invention models the local features of the original pixels in the source image in both horizontal and vertical directions, and filters out regions that can be classified as axis-aligned. Based on parametric modeling, the magnification factor of the local features in the magnified target image is obtained, and a magnified image with a good approximation effect can be generated through a single calculation. Since this invention only uses parametric representation to determine the pixel boundary of the color value referenced by the magnified original pixel, it is based entirely on mathematical calculations that do not involve iteration, and can be completely embedded in hardware, making it highly practical and capable of achieving high clarity and color reproduction.

[0312] To detect the location of original pixels in the source image and to specifically process original pixels in an axis-aligned state using an axis-aligned intersection filter, this embodiment of the invention provides three tests: uniformity test, invalidity test, and discontinuity test. The comprehensive classification result obtained from the tests is used to activate the axis-aligned intersection filter at the granularity of aligned 2×2 image blocks in the source image, and then the axis-aligned intersection filter is used to complete the response to the corresponding original pixels. Specifically, step 130 includes: Step 1301: Determine whether the second coverage value of each edge in the image block is consistent.

[0313] First, a uniformity test is performed on the image patches in the source image: the second coverage value of each edge of the image patch is compared, and it is determined whether the second coverage value of the edge in each vertical and horizontal direction exceeds the corresponding tolerance. This process will be explained in detail below.

[0314] Step 1302: Determine the difference in the first coverage value of adjacent pixel pairs in the vertical direction and adjacent pixel pairs in the horizontal direction within the image block, respectively.

[0315] Difference in the first coverage value of the vertical boundary for:

[0316] Difference in the first coverage value of the horizontal boundary for:

[0317] The image patch is not an axis-aligned intersection point if the difference in the first coverage value of any one pixel in the adjacent pixel pair is greater than the error threshold in the corresponding direction; otherwise, the image patch is an axis-aligned intersection point. Uniformity Test for:

[0318] To avoid the uniformity test potentially passing when the color difference between the original pixel and its neighboring pixels is 0, this embodiment of the invention uses an invalidity check to examine whether the color difference between adjacent pixel pairs within an aligned 2×2 image block is lower than the error threshold in the corresponding direction.

[0319] The expression for the vertical error threshold is: ; The expression for the horizontal error threshold is: .

[0320] If only one first coverage value is below the threshold, or only one pair of first coverage values ​​on the opposite diagonal are below the error threshold in the corresponding direction, then the image patch fails the invalidity test; that is, the image patch is not considered an axis-aligned intersection. In the axis-aligned intersection filter, the specific expression for the above judgment is: .

[0321] Step 1303: Along the normal of each edge, determine whether adjacent pixel pairs constitute a true continuous edge.

[0322] Axis-aligned intersection filters cannot locally distinguish between discontinuous and smooth regions in an image; however, by moving along the normal of each edge, it can be expected that there is at least a small overlap value between adjacent pixel pairs (compared to a smooth ramp, where each value is likely to be close to an equal blend value in its neighborhood).

[0323] Discontinuity tests are used to determine Whether all horizontal and vertical overlays within an aligned 2×2 image patch constitute a true continuous edge, the expression for determining whether adjacent pixel pairs constitute a true continuous edge is as follows: ; in, This represents the vertical coverage of the target. This represents the target level coverage.

[0324] Step 1304: When the image block is an axis-aligned intersection, the second coverage value of each edge is consistent, and the adjacent pixel pairs form a true continuous edge, the original pixel is in an axis-aligned state.

[0325] When the uniformity test, invalidity test, and discontinuity test all pass, the original pixel is classified as a special case located on the axis alignment, and the axis alignment intersection filter response is used to achieve the best upsampling effect; for the original pixel Classification results The expression is as follows: .

[0326] The specific process of the uniformity test is described below. Step 1301 includes: Step 13011: Determine the original pixel and the neighboring pixels in the corresponding direction as adjacent pixel pairs, and take the absolute value of the difference in the coverage of the adjacent pixel pairs as the coverage value difference of the edges corresponding to the original pixel and the neighboring pixels in the corresponding direction in the corresponding direction.

[0327] The original pixel and its neighboring pixels in the horizontal direction are identified as adjacent pixel pairs. The absolute value of the difference in the coverage of the adjacent pixel pairs is used as the coverage value difference on the vertical edge corresponding to the original pixel and the neighboring pixels in the horizontal direction. The original pixel and its neighboring pixels in the vertical direction are identified as adjacent pixel pairs. The absolute value of the difference in the coverage of the adjacent pixel pairs is used as the coverage value difference on the horizontal edge corresponding to the original pixel and the neighboring pixels in the vertical direction.

[0328] It should be noted that in the embodiments of the present invention, x represents the horizontal direction and y represents the vertical direction.

[0329] In the axis-aligned intersection filter scheme, the vertical boundary is determined by the color values ​​of the two pixels adjacent to the boundary, that is, the two pixels to the left and right of the vertical boundary. Similarly, the horizontal boundary is determined by the color values ​​of the two pixels adjacent to the boundary, that is, the two pixels above and below the horizontal boundary. The value of .

[0330] The expression for the difference in coverage value in the vertical direction is: .

[0331] The expression for the tolerance corresponding to the difference in coverage value in the vertical direction is: ; Where i represents the horizontal pixel coordinate, and j represents the vertical pixel coordinate. The horizontal baseline vector, This is the vertical positive error coupling vector. It is a horizontal positive coupling vector. It is the vertical negative error coupling vector. It is a horizontal negative coupling vector.

[0332] Similarly, the expression for the difference in coverage value in the horizontal direction is: ; The corresponding tolerance expression is: .

[0333] Step 13012: Along each edge within the image block, determine whether the coverage value difference exceeds the corresponding tolerance; when the coverage value difference does not exceed the corresponding tolerance, the second coverage value of the corresponding edge is consistent.

[0334] The expression for achieving this judgment in the axis-aligned intersection filter is: ; It should be noted that without vertical and horizontal tolerances, very few image patches can pass the uniformity test; and while the expression for the error boundary may be simplified in a space with uniform pixel error, this is not the case for images encoded in sRGB.

[0335] The following describes the special processing scheme of the axis-aligned intersection filter for the original pixels. Step 140 includes: Step 1401: Determine the initial coverage representation corresponding to the original pixel and the initial coverage representation corresponding to the neighboring pixels of the original pixel in the image block based on the color values ​​of the original pixel relative to the neighboring pixels in the horizontal and vertical directions.

[0336] In a typical case, there are at most four distinct surfaces within each image patch. To maintain the accuracy of the parameterized overlay representation and avoid averaging neighboring pixels (which can be more sensitive to noise in the source image), the position of the vertical edge within (or outside) the source pixel is inferred from its color value relative to the horizontal neighboring pixels.

[0337] Step 1402: Obtain the target coverage representation of the original pixel based on the initial coverage representation of the original pixel and the initial coverage representation of the neighboring pixels.

[0338] Step 1403: Obtain the magnification factor based on the target coverage representation.

[0339] After the axis-aligned intersecting filter responds, it ultimately outputs a series of amplification coefficients.

[0340] Specifically, the initial coverage representation includes an initial vertical coverage representation and an initial horizontal coverage representation; step 1401 includes: Step 14011: Based on the color value of the original pixel relative to the corresponding horizontal neighboring pixel, determine whether the color value of the vertical edge corresponding to the original pixel exceeds the vertical error threshold; if it exceeds the vertical error threshold, determine the projection of the original pixel along the axis defined by the horizontal neighboring pixel as the initial vertical coverage representation corresponding to the original pixel; if it does not exceed the vertical error threshold, the initial vertical coverage representation is a preset invalid value.

[0341] The default invalid value is 0.

[0342] The expression for achieving the above judgment in the axis-aligned intersection filter is: .

[0343] Step 14012: Based on the color value of the original pixel relative to the corresponding vertical neighboring pixel, determine whether the color value of the horizontal edge corresponding to the original pixel exceeds the horizontal error threshold; if it exceeds the horizontal error threshold, determine the projection of the original pixel along the axis defined by the vertical neighboring pixel as the initial horizontal coverage representation corresponding to the original pixel; if it does not exceed the horizontal error threshold, the initial horizontal coverage representation is a preset invalid value.

[0344] The expression for achieving the above judgment in the axis-aligned intersection filter is: .

[0345] The embodiments of the present invention evaluate the initial coverage representation based on the corresponding error thresholds in the horizontal and vertical directions: if the color difference exceeds the corresponding error threshold, the initial coverage representation in the corresponding direction is the projection representation of the original pixel along its horizontal or vertical neighboring pixels, or in this case, the vertical distance between the edge and the nearest pixel edge (or in this case, the horizontal distance between the edge and the nearest pixel edge).

[0346] like Figure 46 As shown, since a source pixel may contain a pair of intersecting edges, the quality of the parametric overlay representation may degrade (i.e., the parametric overlay representation degrades); where the left side is the source image, the middle schematic shows its processed degradation, and the right schematic shows its unprocessed degradation; the horizontal and vertical neighborhood color differences of the top-left pixel (within an aligned 2×2 image patch) are both zero, although the coverage is half in both directions. According to the parametric overlay representation, this can be understood as the case of 0 divided by 0, which is undefined. To address this problem, the target overlay representation includes a target vertical overlay representation and a target horizontal overlay representation; step 1402 includes: Step 14021: When the color difference between the two sides of the vertical edge corresponding to the original pixel exceeds the vertical error threshold, the initial vertical coverage representation is determined as the target vertical coverage representation; when the color difference between the two sides of the vertical edge corresponding to the original pixel does not exceed the vertical error threshold, the initial vertical coverage representation of the vertical neighboring pixels of the original pixel in the image block is determined as the target vertical coverage representation.

[0347] As long as the color difference value of the neighboring pixels is lower than the corresponding error threshold, the coverage estimate (i.e., the initial horizontal coverage or the initial vertical coverage) will move along the intersecting edge.

[0348] The expression for the target vertical parameterization is: .

[0349] Step 14022: When the color difference between the two sides of the horizontal edge corresponding to the original pixel exceeds the horizontal error threshold, the initial horizontal coverage representation is determined as the target horizontal coverage representation; when the color difference between the two sides of the horizontal edge corresponding to the original pixel does not exceed the horizontal error threshold, the initial horizontal coverage representation of the horizontal neighboring pixels of the original pixel in the image block is determined as the target horizontal coverage representation.

[0350] Similarly, the expression for the parameterized representation of the target level is: .

[0351] The target coverage representation includes a target vertical coverage representation and a target horizontal coverage representation; step 1403 includes: Step 14031: Determine the target vertical coverage representation as the vertical amplification gradient parameter; determine the target horizontal coverage representation as the horizontal amplification gradient parameter; determine the product of the target horizontal coverage representation and the target vertical coverage representation as the tilt amplification gradient parameter.

[0352] The expression for the amplification gradient parameter is: .

[0353] Step 14032: Use the product of the vertical amplification gradient parameter and the horizontal baseline vector as the axis-aligned vertical parameter; use the product of the horizontal amplification gradient parameter and the vertical baseline vector as the axis-aligned horizontal parameter; use the product of the tilt amplification gradient parameter and the tilt baseline vector as the axis-aligned tilt parameter; use the axis-aligned vertical parameter, the axis-aligned horizontal parameter, and the axis-aligned tilt parameter as amplification coefficients in their respective dimensions.

[0354] The expression for the axis alignment horizontal parameter is: ; The expression for the axis alignment vertical parameter is: ; The expression for the axis alignment tilt parameter is: .

[0355] This invention provides a specific example to illustrate the importance of properly handling degradation, as follows: Among them, the magnification factor is aligned with the horizontal magnification factor. Vertical magnification factor aligned with axis Defined as 0, axis alignment skew parameter The value depends on the two overlay values ​​and may not be 0 (e.g., as shown in the image). Figure 46 (Specific examples shown).

[0356] In such Figure 47 In the specific example shown, the four magnified sub-pixels are as follows: ; ; ; .

[0357] The axis alignment horizontal parameter is: ; Where, assuming , ; The axis alignment vertical parameter is: ; Where, assuming , ; The axis alignment tilt parameter is: ; Similar results also apply to other specific instances (e.g., when the intersection is located in any of the other three sub-pixels).

[0358] The core of the single-image super-resolution algorithm of this invention lies in the fact that the computation used is more complex than that used in the basic filter, because the goal is to implement the algorithm in hardware. The operation of this invention is less costly than using a standard instruction set. This invention is basically developed around contour filters, by comparing projection parameters... (i.e., parametric overlay representation) effectively classifies the location region of the original pixel, such as a single straight edge (i.e., edge) and a surface (i.e., smooth region), two perpendicularly intersecting edges and the four surface regions they divide, as well as corresponding special cases.

[0359] The following will describe three key features of the single-image super-resolution algorithm of this invention: First, the number of filters used for amplification processing has been minimized.

[0360] The purpose of this invention is to insert a super-resolution algorithm into a display pipeline, allowing for upscaling with only an input image patch, similar to upscaling algorithms running in a shader. Contour filters can construct edges, but they don't work well in all smooth domains; therefore, they cannot be used alone and must be used in conjunction with surface filters. Contour filters and surface filters are two basic intersecting filters; other intersecting filters are built for special cases and are optional features used for specific purposes.

[0361] Secondly, the additional alignment strategy gives the super-resolution algorithm of this invention a hardware advantage.

[0362] The following explains the combination of surface filters and profile filters: Based on the kernel sizes of the two intersecting filters, all original pixels in the source image are processed using the two intersecting filters according to the pipeline processing volume. The pipeline processing volume is a multiple of a scaling factor, and the kernel size is smaller than the pipeline processing volume. The scaling factor is the factor by which the source image is magnified. Since this embodiment of the invention magnifies the source image by a scaling factor of 2, the kernel size, which is the kernel of each intersecting filter, is specifically 3×3; the pipeline processing volume is 4×4, which is the size of the image patch.

[0363] The granularity of image patches is crucial for classifying the locations of raw pixels. When processing raw pixels using the contour filter alone, only a 3×3 image patch needs to be processed at a time, acquiring only local features (i.e., the 3×3 pixels corresponding to the filter kernel). Based on this, a surface filter is used to examine and compare the coverage values ​​of neighboring raw pixels. The algorithm, based on the contour activation function, generates corresponding weights for each 3×3 pixel it processes, and then compares the side-by-side 4×4 image patches pairwise. That is, it combines the contextual information of the area surrounding the original pixel to determine whether it lies on a discontinuous edge, a continuous edge, or a smooth region. Therefore, the combined 4×4 calculation actually shares data and influences each other in subsequent classification, performing four sets of 3×3 calculations in parallel.

[0364] Because the size of an image patch corresponds to the processing capacity of the hardware pipeline, it means that the original pixels in the horizontal direction of the source image can be processed sequentially in a 4×4 size. Currently, for GPU pipelines, processing image patches in 4×4 size makes it easier to coordinate with the processing of other GPU pipelines, thus achieving a more ideal overall processing efficiency. For example, if other algorithms within the GPU process images in an 8×8 size, then the super-resolution algorithm of this embodiment can work in conjunction with it.

[0365] The following describes the block-based buffer compression scheme of this invention: The processing method of this invention is fundamentally different from other commonly used super-resolution image processor (ISP) algorithms. The embodiments of this invention allow for parallel and automatic processing; while the block sequences in the usual ISP algorithm are ordered rather than processed in parallel; the embodiments of this invention do not require excessive reuse of data, which makes this invention applicable to most use cases.

[0366] In all cases, the embodiments of this invention require two-pixel alignment. Based on the above processing scheme combining surface filters and contour filters, the embodiments of this invention also provide an additional alignment strategy, which eliminates the need for more than four blocks when using small frames to buffer and compress the block size. For example... Figure 48 As shown, specifically, in step 1601, as Figure 49 As shown, when the 1 / 2kth row of original pixels to be enlarged is located at the top of the source image, the currently processed image block is aligned k pixels with the source image; multiple currently processed image blocks are vertically aligned with each other; the currently processed image block copies adjacent pixels to expand the boundary of the source image. Specifically, at the top of the image, one row of pixels is enlarged, generating two rows of output pixels; to avoid needing to read more than four compressed blocks, the input image block should be aligned two pixels with the source image; the input image block is vertically aligned to avoid reading a second row of compressed blocks, and the image boundary is expanded by copying adjacent pixels.

[0367] The following is based on Figure 49 Taking the topmost image as an example, the processing of source image edges in this embodiment of the invention will be explained: The green area in this image represents the source image edge. The 4×4 pixel block is aligned with the top edge of the source image by two pixels. Therefore, when processing the 4×4 pixel block in the scan line direction, only the two rows of green pixels need to be read. If the 4×4 pixel block is not aligned with the top edge of the source image by two pixels, it cannot be guaranteed that only two rows of pixels need to be read each time when processing the 4×4 pixel block in the scan line direction.

[0368] like Figure 50As shown, in step 1602, when the k rows of original pixels to be magnified are located vertically inside the source image, the currently processed image block is aligned k pixels with the source image; multiple currently processed image blocks are vertically aligned with each other; the boundaries of the left and / or right edges of the magnified target image are repeated. Specifically, within the vertical interior of the image, two rows of pixels are magnified, producing four rows of output pixels; to avoid needing to read more than four compressed blocks, the input image block should be aligned two pixels with the source image; the input image blocks are vertically aligned to avoid reading more than two rows of compressed blocks; boundary repetition is only necessary for the left and right edges of the image.

[0369] The following is based on Figure 50 For example, the following describes the processing of pixels within the source image in an embodiment of the present invention: In this case, the image edges are only on the left and right sides; for example... Figure 51 As shown in (a), the 4×4 pixel blocks are vertically aligned, so when processing the inner 4×4 pixel blocks, it is only necessary to read no more than four adjacent pixel blocks, that is, the four adjacent pixel blocks at the top, bottom, left, and right. Figure 51 As shown in (b), if the 4×4 pixel blocks are not vertically aligned, more than four adjacent pixel blocks need to be read when processing the inner 4×4 pixel blocks; that is, more than one pixel block needs to be read on a certain adjacent boundary. Figure 51 In (b), the 4×4 pixel block with the blue border corresponds to the two 4×4 pixel blocks with the red and yellow borders that need to be read for processing its upper edge.

[0370] The following explains the role of the image patch progression footprint and the additional alignment strategy: Essentially, when calculating the parameterized overlay representation of each original pixel, classification is performed using a 3×3 filtering kernel to obtain local information and neighborhood context information. Specifically, this classification involves examining the work done by multiple neighboring pixels surrounding the original pixel, using this as a reference for the enlarged color value of the original pixel. Therefore, ideally, there wouldn't be an asymmetric alignment strategy like the additional alignment strategy provided in this embodiment. Instead, a strategy of "examining neighboring pixels to the left and right of the original pixel" is adopted. This calculation is not performed in a 4×4 size, but rather in a 5×5 size. This increases the amount of data that needs to be read, reducing the advantage from a hardware implementation perspective. This embodiment does not simultaneously reference two neighboring pixels, so fundamentally, the super-resolution algorithm of this embodiment is asymmetric, only searching to the right or left, not simultaneously searching to both sides (i.e., only looking at the right or left, not both sides). A fundamental feature of this asymmetric super-resolution algorithm design is constrained contextual feature classification. This is what distinguishes this invention from machine learning methods (such as neural networks). This invention identifies features and attempts to process the source image with a very limited footprint. Therefore, the alignment region constructed according to the additional alignment strategy in this embodiment is a hybrid. This embodiment is based on the idea of ​​parameterized filtering and is a method of balancing full AI feature classification with more localized classification.

[0371] Thirdly, the super-resolution algorithm of this invention is applicable to fixed hardware implementations.

[0372] All formulas involved in the embodiments of this invention have been determined. Since they are fixed in hardware, minimal adjustments are required, resulting in low hardware implementation costs. Algebraic expressions (e.g., dot products) are equivalent to a considerable number of instructions in software. Because images are composed of three-channel color values, the required precision of the target image is relatively low. Therefore, the methods of the embodiments of this invention can be implemented directly in fixed-function hardware at a low cost, and the cost of performing a series of operations is also relatively low.

[0373] In existing technologies, many solutions using AI algorithms (or similar AI algorithms) inevitably involve iterative calculations. These solutions are often based on exhaustive methods, repeatedly iterating to determine relevant parameters, resulting in a massive computational workload. This invention flexibly balances the required accuracy of the target image with the quality of the result. Since it processes typical sRGB (standard Red Green Blue) images, each channel is 8 bits, there is no mandatory high-precision requirement. Both the software and hardware accuracy requirements are relatively low. Therefore, this invention uses approximation, introducing a certain loss in the calculation to simplify the actual logic implementation and reduce costs. This invention uses a fixed function for approximate calculations, which is less complex and less costly than a more accurate implementation. This significantly expands the applicability of the method for enhancing local features while preserving detail.

[0374] This invention primarily uses single-variable functions, specifically rational functions or trigonometric functions. Therefore, these functions can be directly implemented in hardware at a low cost. Some transfer functions, and / or arbitrary functional relationships between parameters representing the normal direction of the contour edge, can be implemented using lookup tables or by obtaining inexpensive approximations implemented in hardware, fundamentally requiring no more cost than performing other arithmetic operations. However, approximating or implementing these using standard instruction sets is extremely costly. It should be noted that implementing the methods of this invention in software necessitates performing complex algebraic sequences, which increases the corresponding cost.

[0375] Generally, despite the additional computational complexity (e.g., quadratic scaling of multiplication and addition operations in linear filters), wider filter kernels offer greater expressive power due to their ability to implement additional degrees of freedom and better capture non-local image structures. The main overhead that embodiments of the invention aim to limit in hardware is memory; both bandwidth and buffering need to be limited relative to a given target throughput. Larger filter kernels require more input data to be processed per unit output; this typically translates to more bandwidth, or alternatively, more buffering due to overlapping filter kernels and shared data between adjacent instances. Special problems may arise when data cannot be retrieved at a basic granularity (e.g., in the case of block-based encoding of image data), causing efficient filter support to be amplified to the nearest boundary. While multi-channel filtering algorithms can reduce some computational complexity by using smaller intermediate filter kernels, they introduce additional memory bandwidth between image passes.

[0376] Based on the above considerations, the calculation properties of the embodiments of the present invention are as follows: Given the limited filtering footprint of intersecting filters, it is important to maximize the functionality of different filtering kernels if the goal is to compete with more complex solutions. For example, a software implementation may be able to perform operations within a given instruction set under a given performance budget, while a fixed-function hardware implementation, considering the accuracy trade-offs for a particular application, can leverage customized operations and functions to facilitate direct evaluation (i.e., the method of the embodiments of the present invention) and acknowledge its efficiency when the required accuracy is met. Since image data is typically encoded in sRGB, where only 8 bits of precision are needed per channel (potentially 12 bits or more after conversion to a linear color space), and given the lack of higher resolution data, it is acceptable that enlarged images are inherently noisy or error-prone. These two considerations explain why the method of the embodiments of the present invention has a relatively low-precision data path compared to standard floating-point or even some integer instruction sets. In addition to these error requirements, many trigonometric functions and rational single-variable functions can be implemented directly in hardware (e.g., directly in the form of lookup tables) at a much higher efficiency than using programmable techniques. Finally, there is a significant reuse of computational parameters between adjacent pixels, which can be naturally implemented in hardware using parallel pipelines.

[0377] The following are several specific examples illustrating that the super-resolution algorithm of the present invention can achieve better results: First, a set of pixelated region images using different intersection filter responses are provided to illustrate the effects of each intersection filter.

[0378] A pixelated region in a source image that is magnified using contour filters and surface filters, such as Figure 52 As shown in (a) in the figure; as Figure 52 (b) in the middle is based on, for example Figure 52 The set of target pixels obtained after adding an axisymmetric intersecting filter to the pixelated region shown in (a) to improve resolution.

[0379] The following provides several specific examples to demonstrate the differences in processing performance between the super-resolution algorithm of this invention and other similar algorithms in the prior art: The first set of specific examples illustrates the application scenarios of image super-resolution algorithms in computer graphics rendering: such as Figure 53 The image shown is a target image based on the render target; as shown... Figure 54 The image shown is a magnified target image based on the super-resolution algorithm of this invention; as shown... Figure 55 The following is based on Figure 54 The obtained target image; specifically, in Figure 54Based on the image shown, image sharpening technology is added and mapped to the target screen size to obtain the following result: Figure 55 The target image shown. Based on super-resolution algorithms, rendered images can be created at small sizes (e.g., Figure 53 ) after super-resolution image magnification (e.g., Figure 54 To generate high-resolution, large-size target images (e.g., Figure 55 This reduces the workload of computer graphics rendering while ensuring the quality of the target image.

[0380] Specific examples from groups two to five Figures 56 to 59 As shown; in Figures 56 to 59 In the image, the images on the left are all magnified target images based on bilinear interpolation, while the images on the right are all magnified target images based on the super-resolution algorithm of this invention.

[0381] Images generated using the super-resolution algorithm of this invention. Figure 56 and Figure 57 All contour boundaries in the image generated on the right using the super-resolution algorithm of this invention are clearer than those in the image generated on the left using bilinear interpolation.

[0382] Figure 58 and Figure 59 In the image on the right side, generated using the super-resolution algorithm of this invention, the magnified contour boundaries are clearer than those in the image on the left side generated using bilinear interpolation. Figure 58 Image (a) is the image generated using bilinear interpolation. Figure 58 Image (b) is an image generated using the super-resolution algorithm of this embodiment of the invention. The super-resolution algorithm of this embodiment of the invention performs well in certain specific regions, for example, Figure 58 (c) in the middle is Figure 58 A magnified view of part (a) in the diagram. Figure 58 (d) in the middle is Figure 58 A magnified view of part (b) in the diagram, compared to Figure 58 The magnified image in (c) is shown below. Figure 58 The magnified image in (d) is clearer. Figure 59 The image of the withered grass in the lower left corner, when magnified, shows that the contour boundaries of the image generated using the super-resolution algorithm of this invention are clearer than those generated using bilinear interpolation.

[0383] Specific examples of groups six and seven Figure 60 and Figure 61 As shown; in Figure 60 and Figure 61Among them, the images on the left side are all target images enlarged based on the ISP algorithm, and the images on the right side are all target images enlarged based on the super-resolution algorithm of the embodiments of the present invention. For Figure 60 a partial image of the Chinese text 'C:\Users\..' and Figure 61 a partial image of the upper-middle leaves, Figure 60 and Figure 61 the images on the right side in

[0384] are clearer than the images on the left side. Figure 62 As shown in Figure 62 the eighth group of specific examples, the image on the left side is the target image enlarged based on the AI algorithm, and the image on the right side is the target image enlarged based on the super-resolution algorithm of the embodiments of the present invention. The quality of the target image enlarged based on the super-resolution algorithm of the embodiments of the present invention on the right side is similar to that of the target image enlarged based on the AI algorithm on the left side, but the workload required for using the super-resolution algorithm is much smaller than that of the AI algorithm.

[0385] As shown in Figure 63 the ninth group of specific examples, the image on the left side is the target image enlarged based on the technology of the Advanced RISC Machine (ARM) architecture, and the image on the right side is the target image enlarged based on the super-resolution algorithm of the embodiments of the present invention. Figure 63 For the partial image of the Chinese text 'Directory:', the right side is clearer than the left side.

[0386] As Figure 64 shown in Figure 64 is a schematic structural diagram of the device for improving the visual fidelity of an image according to the embodiments of the present invention. The device for improving the visual fidelity of an image in this embodiment includes one or more processors 21 and a memory 22. Among them,

[0387] the processor 21 and the memory 22 can be connected through a bus or other means, Figure 64 and here, the connection through a bus is taken as an example.

[0388] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the method for improving the visual fidelity of an image in this embodiment. The processor 21 executes the method for improving the visual fidelity of an image by running the non-volatile software programs and instructions stored in the memory 22. [[ID=,38]]

[0389] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0390] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the method for improving image visual fidelity in the above embodiments, for example, performing the above-described... Figure 1 , Figures 6 to 9 and Figure 48 The steps shown.

[0391] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 64 One of the processors 21 can enable the one or more processors to execute the method for improving image visual fidelity in the specific embodiments of the present invention, for example, to perform the above-described... Figure 1 , Figures 6 to 9 and Figure 48 The steps shown can also be implemented. Figure 64 The various modules and units described above; or the method for improving image visual fidelity in specific embodiments of the present invention, for example, performing the above-described... Figure 1 , Figures 6 to 9 and Figure 48 The steps shown can also be implemented. Figure 64 The aforementioned modules and units.

[0392] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0393] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0394] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for improving the visual fidelity of an image, characterized in that, include: Based on the location of the original pixels in the source image and the classification criteria of the intersection filters, determine the intersection filter used to process the original pixels; Based on the selected intersecting filter, determine the amplification factor in at least one dimension; The original pixel is magnified according to the magnification factor to obtain multiple target pixels; The multiple target pixels are combined to generate a target image.

2. The method for improving image visual fidelity according to claim 1, characterized in that, The intersection filters include contour filters, diagonal intersection filters, axis-aligned intersection filters, surface filters, and filament filters; Determining the amplification factor in at least one dimension based on the selected intersecting filter includes: When the intersection filter is a contour filter, diagonal intersection filter, axis-aligned intersection filter, or filament filter, a parameterized coverage representation of the coverage area of ​​the original pixel is determined; based on the parameterized coverage representation and the baseline vector, the corresponding magnification factor is determined; When the intersecting filter is a surface filter, the baseline vector is scaled to obtain the amplification factor of the surface filter.

3. The method for improving image visual fidelity according to claim 2, characterized in that, The step of determining the intersection filter used to process the original pixels based on the position of the original pixels in the source image and the classification criteria of the intersection filters includes: When the position of the original pixel in the source image meets the assumption of the first classification criterion, the original pixel is processed using the axis-aligned intersection filter; wherein, the first classification criterion is that the image block where the original pixel is located only includes horizontal and vertical edges, and the horizontal and vertical edges are axis-aligned; When the position of the original pixel in the source image does not meet the assumption of the first classification criterion but meets the assumption of the second classification criterion, the original pixel is processed using the diagonal intersection filter; wherein, the second classification criterion is that the image block where the original pixel is located only includes the forward edge and the backward edge of the diagonal, and the forward edge intersects with the backward edge; When the position of the original pixel in the source image does not conform to the assumptions of the first classification criterion and the second classification criterion, the parameterized overlay representation corresponding to the original pixel is determined, and the intersection filter for processing the original pixel is determined based on the parameterized overlay representation.

4. The method for improving image visual fidelity according to claim 3, characterized in that, When the position of the original pixel in the source image does not conform to the assumptions of the first classification criterion and the second classification criterion, the parameterized cover representation corresponding to the original pixel is determined, and the intersection filter for processing the original pixel is determined based on the parameterized cover representation, including: Based on the assumption that the original pixel is located between two adjacent surfaces in the source image, a parameterized overlay representation corresponding to the original pixel is obtained; When the parameterized overlay representation is less than a preset normal value, the original pixel is processed using the filament filter; When the parameterized overlay representation is greater than or equal to a preset normal value, the contour filter is used to process the original pixel; When the original pixel is located in a position in the source image that cannot be classified as being processed by either a filament filter or a contour filter, the original pixel is processed by the surface filter.

5. The method for improving image visual fidelity according to claim 1, characterized in that, The step of magnifying the original pixel according to the magnification factor to obtain multiple target pixels includes: Construct a complementary relationship between each eigenvalue decomposition coefficient and multiple target pixels; reconstruct the complementary relationship, and determine the expression for each target pixel based on the source image and the eigenvalue decomposition coefficients representing the multiple target pixels; The amplification factor of the selected intersecting filter is used as a reliable substitute for the eigenvalue coefficients in the corresponding dimension of the expression. Based on the reliable substitution, the color values ​​of the corresponding multiple target pixels are determined according to the color values ​​of the original pixels.

6. The method for improving image visual fidelity according to claim 1, characterized in that, Based on the kernel size of the intersecting filter, all the original pixels in the source image are processed using the intersecting filter according to the pipeline processing capacity; Wherein, the pipeline processing capacity is a multiple of the proportional coefficient, and the filter core size is smaller than the pipeline processing capacity.

7. The method for improving image visual fidelity according to claim 6, characterized in that, Also includes: When the original pixels of the 1 / 2k row to be enlarged are located at the top of the source image, the currently processed image block is aligned k pixels with the source image; The multiple currently processed image blocks are vertically aligned with each other; the currently processed image blocks copy adjacent pixels to extend the boundaries of the source image; When the k rows of original pixels to be magnified are located vertically inside the source image, the currently processed image block is aligned with the source image by k pixels; multiple currently processed image blocks are vertically aligned with each other; the boundaries of the left and / or right edges of the magnified target image are repeated.

8. The method for improving image visual fidelity according to claim 1, characterized in that, The intersecting filter uses rational univariate functions and trigonometric functions.

9. An apparatus for improving the visual fidelity of an image, characterized in that, The apparatus for improving image visual fidelity includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to implement the method for improving image visual fidelity as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the method for improving the visual fidelity of an image as described in any one of claims 1 to 8.

Images