Image optimization method and device

Abstract

Provided are an image optimization method and device. The image optimization method comprises: receiving multiple image frames including a target frame; determining an unsharp mask for each of the multiple image frames; determining a result of averaging the unsharp masks of the multiple image frames as a multi-frame optimized unsharp mask for the target frame; and optimizing the target frame based on the multi-frame optimized unsharp mask for the target frame. Through the above image optimization method, the noise introduced by the unsharp mask is reduced, and the target frame presents a better sharpening effect.

Description

Technical Field

[0001] The present invention relates to image optimization, and more particularly to an image optimization method and device. Background Art

[0002] Unsharp Mask (also known as unsharp masking) technology is a commonly used image enhancement technology in digital image processing. Unsharp Mask technology blurs the original image (or smoothes it) and then performs a pixel-by-pixel difference operation with the original image, and then sums it with the original image to achieve the purpose of increasing the high-frequency components in the image and enhancing the image contour.

[0003] However, the traditional unsharp masking technology will enhance all detail signals, which will cause strong edge-related noise and some texture-like noise to be enhanced, thereby reducing the image enhancement effect. Summary of the invention

[0004] The object of the present invention is to provide an image optimization method and device that can reduce the noise introduced by the unsharp mask and make the target frame present a better sharpening effect.

[0005] According to one embodiment of the present invention, an image optimization method includes: receiving multiple image frames including a target frame; determining an unsharp mask for each of the multiple image frames; determining a result of averaging the unsharp masks of the multiple image frames as a multi-frame optimized unsharp mask for the target frame; and optimizing the target frame based on the multi-frame optimized unsharp mask for the target frame.

[0006] Optionally, the plurality of image frames are a plurality of continuous image frames, the target frame is a first frame of the plurality of image frames, and the number of the plurality of image frames is determined based on a frame rate.

[0007] Optionally, the step of determining an unsharp mask for each of the multiple image frames includes: generating a blurred image frame corresponding to each image frame by performing smoothing filtering on each image frame; and determining the difference between each image frame and the blurred image frame as an unsharp mask for each image frame.

[0008] Optionally, the step of determining the result of averaging the unsharp masks of the multiple image frames as the multi-frame optimized unsharp mask of the target frame includes: determining a similarity weight corresponding to each unsharp mask, wherein the greater the similarity between each unsharp mask and the unsharp mask of the target frame, the greater the similarity weight; performing weighted averaging on the unsharp masks of the multiple image frames based on the determined similarity weights; and determining the result of performing the weighted averaging as the multi-frame optimized unsharp mask of the target frame.

[0009] Optionally, the step of determining the similarity weight corresponding to each unsharp mask includes: determining at least one weight corresponding to each unsharp mask based on at least one similarity between each unsharp mask and the unsharp mask of the target frame; and determining the similarity weight corresponding to each unsharp mask by combining the at least one weight of each unsharp mask.

[0010] Optionally, by combining the at least one weight of each unsharp mask, the step of determining the similarity weight corresponding to each unsharp mask includes: determining the result of multiplying each element in a set consisting of the at least one weight of each unsharp mask and 1 as the similarity weight corresponding to each unsharp mask.

[0011] Optionally, the at least one similarity of each unsharp mask includes at least one of overall pixel difference similarity, pixel mean difference cosine difference similarity and frame interval similarity, and the at least one weight corresponding to each unsharp mask includes at least one of an overall pixel difference weight corresponding to the overall pixel difference similarity, a pixel mean difference cosine difference weight corresponding to the pixel mean difference cosine difference similarity and a frame interval weight corresponding to the frame interval similarity.

[0012] Optionally, the overall pixel difference similarity of each unsharp mask corresponds to the sum of pixel differences calculated based on the pixel differences between each pixel of each unsharp mask and the corresponding pixel of the unsharp mask of the target frame, the closer the pixel difference sum is to 0 or the closer the pixel difference sum is to the average of the pixel difference sums of the unsharp masks of the multiple image frames, the greater the overall pixel difference weight. The pixel mean difference cosine difference similarity of each unsharp mask corresponds to the angle between the resultant vector corresponding to the pixel mean difference vector of each pixel of each unsharp mask and the resultant vector corresponding to the pixel mean difference vector of the unsharp mask of the target frame, the pixel mean difference vector of each pixel is determined based on the difference between the pixel value of each pixel and the pixel mean of the unsharp mask including the pixel and the position of each pixel in the unsharp mask including the pixel, the smaller the angle, the greater the pixel mean difference cosine difference weight. The frame interval similarity of each unsharp mask corresponds to the frame interval between the image frame corresponding to each unsharp mask and the target frame, and the smaller the frame interval is, the greater the frame interval weight is.

[0013] Optionally, based on the multi-frame optimized unsharp mask of the target frame, the step of optimizing the target frame includes: determining the sum of the multi-frame optimized unsharp mask of the target frame and the target frame as the optimization result of the target frame.

[0014] According to an embodiment of the inventive concept, an image optimization device includes a processor configured to execute the image optimization method as described above.

[0015] According to an embodiment of the inventive concept, a computer-readable storage medium stores instructions, which, when executed by a processor, enable the processor to perform the image optimization method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or other aspects of the present disclosure will become clearer and more easily understood through the following detailed description in conjunction with the accompanying drawings.

[0017] Figure 1 is a flowchart illustrating an image optimization method for optimizing a target frame according to an embodiment of the inventive concept.

[0018] Figure 2 is a flowchart illustrating steps of determining an unsharp mask for each image frame according to an embodiment of the inventive concept.

[0019] Figure 3 is a flowchart illustrating steps of determining a multi-frame optimized unsharp mask of a target frame according to an embodiment of the inventive concept.

[0020] Figure 4 is a flowchart illustrating steps of determining a similarity weight corresponding to each unsharp mask according to an embodiment of the inventive concept.

[0021] Figure 5 is an example graph illustrating an overall pixel difference weight according to an embodiment of the inventive concept.

[0022] Figure 6 is an example graph illustrating pixel mean difference cosine difference weights according to an embodiment of the inventive concept.

[0023] Figure 7 is an example graph illustrating a frame interval weight according to an embodiment of the inventive concept.

[0024] Figure 8 is a block diagram illustrating an image optimizing apparatus for optimizing a target frame according to an embodiment of the inventive concept.

[0025] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0026] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, the order of operations described herein is only an example, and is not limited to those orders set forth herein, but can be changed as it will be clear after understanding the disclosure of the application, except for the operations that must occur in a specific order. In addition, for increased clarity and conciseness, the description of features known after understanding the disclosure of the application can be omitted.

[0027] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0028] Throughout the specification, when a component is described as being "connected to" or "coupled to" another component, the component may be directly "connected to" or "coupled to" the other component, or there may be one or more other components between them. Conversely, when an element is described as being "directly connected to" or "directly coupled to" another element, there may be no other elements between them. Similarly, similar expressions (e.g., "between" versus "immediately between" and "adjacent to" versus "immediately adjacent to") should also be interpreted in the same manner. As used herein, the term "and / or" includes any one of the relevant listed items or any combination of any two or more of the relevant listed items.

[0029] Although terms such as "first", "second" and "third" may be used to describe various components, components, regions, layers or parts herein, these components, components, regions, layers or parts are not limited by these terms. Instead, these terms are only used to distinguish one component, component, region, layer or part from another component, component, region, layer or part. Therefore, without departing from the teachings of the examples described herein, the first component, first component, first region, first layer or first part referred to in the examples may also be referred to as the second component, second component, second region, second layer or second part.

[0030] The terms used herein are only used to describe various examples and will not be used to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the stated features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0031] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art based on the understanding of the disclosure of the present application and the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant field and the disclosure of the present application, and should not be interpreted in an idealized or overly formal way. The use of the term "may" with respect to an example or embodiment herein (e.g., with respect to what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0033] Figure 1 is a flowchart illustrating an image optimization method for optimizing a target frame according to an embodiment of the inventive concept.

[0034] Reference Figure 1 In step S100, a plurality of image frames including a target frame are received. For example, the first frame to the Nth frame including the target frame may be received, where N represents the number of the plurality of image frames.

[0035] In one example, the plurality of image frames may be a plurality of image frames ordered in time sequence. For example, the plurality of image frames may be a plurality of continuous image frames. For example, the first frame to the Nth frame may be N continuous image frames. However, it should be understood that the inventive concept is not limited thereto. For example, in another example, the plurality of image frames may be a plurality of non-continuous or discrete image frames. For example, the first frame to the Lth frame may be N non-continuous or discrete image frames, where L is a positive integer greater than N.

[0036] In one example, the target frame may be the first frame in the time sequence of the plurality of image frames. For example, the target frame may be the first frame of the first frame to the Nth frame. That is, the plurality of image frames including the target frame may include the target frame as the first frame of the plurality of image frames and N-1 subsequent frames of the plurality of image frames that are located after the first frame. However, it should be understood that the inventive concept is not limited thereto.

[0037] In one example, the number of multiple image frames may be determined based on the frame rate. For example, the number of multiple image frames may be equal to the frame rate. However, it should be understood that the inventive concept is not limited thereto. For example, in another example, the number of multiple image frames may be a preset value independent of the frame rate. Hereinafter, a method for determining the number of multiple image frames will be further described when describing Equation 14. In addition, the frame rate may be the frame rate of a system of consecutive image frames. However, it should be understood that the meaning of the frame rate is not limited thereto, and it may be other frame rates other than the frame rate of a system of consecutive image frames.

[0038] In step S200, an unsharp mask (UM, also known as unsharp masking) of each image frame among the multiple image frames is determined. Various methods may be used to determine the unsharp mask of each image frame. Hereinafter, embodiments of step S200 will be described in detail with reference to Figure 2 Detailed description of the embodiments of step S200.

[0039] Figure 2 is a flowchart showing steps of determining an unsharp mask of each image frame according to an embodiment of the inventive concept.

[0040] Referring to Figure 2 , in step S210, a blurred image frame (or referred to as a blurred signal) corresponding to each image frame may be generated by performing smoothing filtering on each image frame. For example, a blurred image frame corresponding to the first frame as a target frame among the multiple image frames (hereinafter, simply referred to as the first blurred image frame) may be generated by performing smoothing filtering on the first frame;...; a blurred image frame corresponding to the i-th (where 1 < i < N) frame among the multiple image frames may be generated by performing smoothing filtering on the i-th frame (hereinafter, simply referred to as the i-th blurred image frame);...; a blurred image frame corresponding to the N-th frame as the last frame among the multiple image frames may be generated by performing smoothing filtering on the N-th frame (hereinafter, simply referred to as the N-th blurred image frame). In addition, various methods (such as Gaussian smoothing filtering, etc.) may be used to implement the smoothing filtering, and the present application does not make specific limitations thereon.

[0041] In step S220, the difference between each image frame and the blurred image frame may be determined as the unsharp mask of each image frame. For example, the difference between the first frame as the target frame among the multiple image frames and the first blurred image frame may be determined as the unsharp mask of the first frame as the target frame (hereinafter referred to as the first unsharp mask); ...; the difference between the i-th frame and the i-th blurred image frame among the multiple image frames may be determined as the unsharp mask of the i-th frame (hereinafter referred to as the i-th unsharp mask); ...; the difference between the N-th frame as the last frame among the multiple image frames and the N-th blurred image frame may be determined as the unsharp mask of the N-th frame (hereinafter referred to as the N-th unsharp mask). Here, the difference between each image frame and the blurred image frame may represent the difference between the pixel matrix of each image frame and the pixel matrix of the blurred image frame, that is, the difference between the pixel value at each pixel position in each image frame and the pixel value at the corresponding pixel position of the blurred image frame. For example, step S220 may be represented by the following formula 1.

[0042] UM n =F n -Fa n Formula 1

[0043] Among them, F n Indicates the nth frame, Fa n represents the nth blurred image frame, UM n represents the nth unsharp mask, n represents the index corresponding to the image frame and n=1, ..., N. However, it should be understood that the above formula 1 is only a non-limiting example of calculating the unsharp mask, and various modifications can be made to the above formula 1 without departing from the spirit and scope of the inventive concept of the present application.

[0044] Return to reference Figure 1 In step S300, the result of averaging the unsharp masks of the multiple image frames is determined as the multi-frame optimized unsharp mask (MFOUM) of the target frame. For example, the result of averaging the first unsharp mask to the Nth unsharp mask can be determined as the multi-frame optimized unsharp mask of the target frame. In addition, various methods (e.g., various statistical averaging methods) can be used to achieve averaging, and the present application does not make specific restrictions on this. In the following, reference will be made to Figure 3 An embodiment of step S300 is described in detail.

[0045] In step S400, the target frame is optimized based on the multi-frame optimized unsharp mask of the target frame. In one example, the sum of the multi-frame optimized unsharp mask of the target frame and the target frame can be determined as the optimization result of the target frame (e.g., an optimized sharpened image). Here, the sum of the multi-frame optimized unsharp mask and the target frame can represent the sum of the pixel matrix of the multi-frame optimized unsharp mask and the pixel matrix of the target frame, that is, the sum of the pixel value at each pixel position in the multi-frame optimized unsharp mask and the pixel value at the corresponding pixel position of the target frame. For example, step S220 can be represented by the following formula 2.

[0046] OF=F t +MFOUM Formula 2

[0047] Among them, F t represents the target frame, MFOUM represents the multi-frame optimized unsharp mask of the target frame, OF represents the optimization result of the target frame, and t represents the index corresponding to the target frame. However, it should be understood that the above formula 2 is only a non-limiting example of calculating the optimization result of the target frame, and various modifications can be made to the above formula 2 without departing from the spirit and scope of the inventive concept of the present application.

[0048] Through the above steps S100 to S400, the target frame is optimized using a multi-frame optimized unsharp mask that is the result of averaging the unsharp masks of multiple image frames including the target frame, instead of optimizing the target frame using the unsharp mask of the target frame, so that the energy of edge noise and texture-like noise in the target frame can be dispersed or offset, while protecting the original signal of the mask. Ultimately, the noise introduced by the unsharp mask is reduced, and the target frame is made to present a better sharpening effect by using the multi-frame optimized unsharp mask and superimposing it with the original signal of the target frame.

[0049] Figure 3 is a flowchart illustrating steps of determining a multi-frame optimized unsharp mask of a target frame according to an embodiment of the inventive concept.

[0050] Reference Figure 3 In step S310, a similarity weight corresponding to each unsharp mask may be determined. For example, a first similarity weight corresponding to the first unsharp mask to an Nth similarity weight corresponding to the Nth unsharp mask may be determined respectively.

[0051] In one example, the greater the similarity between each unsharp mask and the unsharp mask of the target frame, the greater the similarity weight; the smaller the similarity between each unsharp mask and the unsharp mask of the target frame, the smaller the similarity weight. Figure 4 An embodiment of step S310 is described in detail.

[0052] In step S320, weighted averaging may be performed on the unsharp masks of the multiple image frames based on the determined similarity weights. For example, weighted averaging may be performed on the first unsharp masks to the Nth unsharp masks of the multiple image frames based on the determined first similarity weights to the Nth similarity weights. Therefore, through step S320, a weighted average result may be obtained. Although the averaging step performed on the unsharp masks of the multiple image frames is described in step S320 by taking weighted averaging as an example, those skilled in the art should understand that various statistical averaging methods (e.g., arithmetic mean, geometric mean, etc.) may be used to implement the averaging performed on the unsharp masks of the multiple image frames.

[0053] In step S330, the result of performing the weighted average may be determined as the multi-frame optimized unsharp mask of the target frame. For example, step S320 and step S330 may be represented by the following formula 3.

[0054]

[0055] Among them, W n Represents the nth unsharp mask UM n However, it should be understood that the above formula 3 is only a non-limiting example of calculating a multi-frame optimized unsharp mask, and various modifications may be made to the above formula 3 without departing from the spirit and scope of the inventive concept of the present application.

[0056] Figure 4 is a flowchart illustrating steps of determining a similarity weight corresponding to each unsharp mask according to an embodiment of the inventive concept. Figure 5 is an example graph illustrating an overall pixel difference weight according to an embodiment of the inventive concept. Figure 6 is an example graph illustrating pixel mean difference cosine difference weights according to an embodiment of the inventive concept. Figure 7 is an example graph illustrating a frame interval weight according to an embodiment of the inventive concept.

[0057] Reference Figure 4 , step 310 may include step S311 and step S312.

[0058] In step S311, at least one weight corresponding to each unsharp mask may be determined based on at least one similarity between each unsharp mask and the unsharp mask of the target frame. For example, at least one weight (such as at least one of a first weight, a second weight, a third weight, etc.) corresponding to the first unsharp mask of the first frame may be determined based on at least one similarity (such as at least one of a first similarity, a second similarity, a third similarity, etc.) between the first unsharp mask of the first frame and the first unsharp mask of the first frame as the target frame;...; at least one weight (such as at least one of a first weight, a second weight, a third weight, etc.) corresponding to the i-th unsharp mask of the i-th frame may be determined based on at least one similarity (such as at least one of a first similarity, a second similarity, a third similarity, etc.) between the i-th unsharp mask of the i-th frame and the first unsharp mask of the first frame as the target frame;...; at least one weight (such as at least one of a first weight, a second weight, a third weight, etc.) corresponding to the N-th unsharp mask of the N-th frame may be determined based on at least one similarity (such as at least one of a first similarity, a second similarity, a third similarity, etc.) between the N-th unsharp mask of the N-th frame and the first unsharp mask of the first frame as the target frame.

[0059] In one example, at least one similarity may respectively correspond to at least one weight. For example, the first similarity among at least one similarity may correspond to the first weight among at least one weight;...; the j-th (where 1 < j < L, and L is the number of at least one similarity) similarity among at least one similarity may correspond to the j-th weight among at least one weight;...; the L-th similarity among at least one similarity may correspond to the L-th weight among at least one weight. Although the above example describes a one-to-one correspondence relationship between similarity and weight, those skilled in the art should understand that the present invention is not limited thereto. For example, multiple similarities may correspond to one weight, or one similarity may correspond to multiple weights.

[0060] In one example, at least one similarity of each unsharp mask may include at least one of a global pixel difference similarity, a pixel mean difference cosine similarity, and a frame interval similarity. Correspondingly, in this example, at least one weight corresponding to each unsharp mask may include at least one of a global pixel difference weight corresponding to the global pixel difference similarity, a pixel mean difference cosine weight corresponding to the pixel mean difference cosine similarity, and a frame interval weight corresponding to the frame interval similarity.

[0061] According to an embodiment of the inventive concept, the overall pixel difference similarity of each unsharp mask may correspond to the sum of pixel differences calculated based on the pixel differences between each pixel of each unsharp mask and the corresponding pixel of the unsharp mask of the target frame. Therefore, the overall pixel difference weight corresponding to each unsharp mask may be determined based on the sum of pixel differences between each unsharp mask and the unsharp mask of the target frame.

[0062] In one example, the closer the sum of pixel differences is to 0 or the closer the sum of pixel differences is to the average of the sum of pixel differences of unsharp masks of multiple image frames, the greater the overall pixel difference weight, and the closer the sum of pixel differences is to 0 or the farther the sum of pixel differences is from the average of the sum of pixel differences of unsharp masks of multiple image frames, the smaller the overall pixel difference weight. That is to say, on the one hand, from the perspective of the value of the sum of pixel differences, the smaller the difference in pixel values ​​between the image frame (or the unsharp mask of the image frame) and the target frame (or the unsharp mask of the target frame), the closer the sum of pixel differences is to 0, and the greater the overall pixel difference weight, and the greater the difference in pixel values ​​between the image frame (or the unsharp mask of the image frame) and the target frame (or the unsharp mask of the target frame), the closer the sum of pixel differences is to 0, and the smaller the overall pixel difference weight; on the other hand, from the perspective of the degree of fluctuation of the distribution of multiple sums of pixel differences corresponding to multiple image frames, the smaller the degree of fluctuation of the distribution, the closer the sum of pixel differences corresponding to the value is to the average sum of pixel differences, and the greater the overall pixel difference weight, and the greater the degree of fluctuation of the distribution, the farther the sum of pixel differences corresponding to the value is from the average sum of pixel differences, and the smaller the overall pixel difference weight.

[0063] Various methods may be used to construct the overall pixel difference weight, which is not specifically limited in the present application. In one example, the overall pixel difference weight of each unsharp mask may be calculated by the following formulas 4 to 6.

[0064]

[0065]

[0066]

[0067] Among them, A n represents the overall pixel difference coefficient of the nth unsharp mask, V n，m represents the pixel value of the mth pixel of the nth unsharp mask, V t，m represents the pixel value of the mth pixel of the unsharp mask of the target frame, M represents the number of pixels of the unsharp mask, Represents the average value of the overall pixel difference coefficient of the unsharp mask of multiple image frames, Wa n represents the overall pixel difference weight of the nth unsharp mask. Therefore, A nreflects the sum of pixel differences of the nth unsharp mask, It reflects the average value of the sum of pixel differences of the unsharp mask of multiple image frames. According to formula 4 and formula 6, Wa n ∈[0, 1] and when A n =0 or When, Wa n can be a maximum value of 1. Therefore, the weight Wa n The unsharp mask with a large difference in pixel mean from the unsharp mask of the target frame can be suppressed. The curve graph of formula 6 is as follows Figure 5 shown.

[0068] However, it should be understood that the above formulas 4 to 6 are only a non-limiting example of calculating the overall pixel difference weight, and various modifications may be made to the above formulas 4 to 6 without departing from the spirit and scope of the inventive concept of the present application.

[0069] According to an embodiment of the inventive concept, the pixel mean difference cosine difference similarity of each unsharp mask may correspond to the angle between the resultant vector corresponding to the pixel mean difference vector of each pixel of each unsharp mask and the resultant vector corresponding to the pixel mean difference vector of each pixel of the unsharp mask of the target frame. Here, the pixel mean difference vector of each pixel may be determined based on the difference between the pixel value of each pixel and the pixel mean of the unsharp mask including the pixel and the position of each pixel in the unsharp mask including the pixel. Therefore, the pixel mean difference cosine difference weight corresponding to each unsharp mask may be determined based on the above angle.

[0070] In one example, the smaller the angle, the larger the pixel mean difference cosine difference weight, and the larger the angle, the smaller the pixel mean difference cosine difference weight. Various mathematical methods can be used to construct the pixel mean difference cosine difference weight, and this application does not make specific limitations on this. In one example, the pixel mean difference cosine difference weight of each unsharp mask can be calculated by the following formulas 7 to 12.

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077] Among them, μ nRepresents the average pixel value of the nth unsharp mask, μ t represents the average value of the pixel value of the unsharp mask of the target frame and the calculation formula is similar to Formula 7, σ n The standard deviation of the pixel values ​​of the nth unsharp mask, σ t The standard deviation of the pixel values ​​of the unsharp mask of the target frame is calculated similarly to Formula 8, represents the pixel mean difference vector of the mth pixel of the nth unsharp mask, represents the pixel mean difference vector of the mth pixel of the unsharp mask of the target frame, represents the sum of all pixel mean difference vectors of the nth unsharp mask, represents the sum of all pixel mean difference vectors of the unsharp mask of the target frame, ε represents a small constant set to avoid the denominator being zero, and α n The resultant vector representing the nth unsharp mask With the resultant vector The angle between them, Formula 11 is a simplified form of Formula 10, Wb n represents the pixel mean difference cosine difference weight of the nth unsharp mask. Therefore, the resultant vector of the nth unsharp mask is and / or resultant vector It reflects the difference between the pixel value of each pixel and the pixel mean value of the unsharp mask including the pixel, as well as the position of each pixel in the unsharp mask including the pixel. According to formula 12, Wb n ∈[0, 1] and Wb n Characterizes the resultant vector With the resultant vector Similarity, and when the resultant vector With the resultant vector When the same, α n =0,Wb n can be a maximum value of 1. Therefore, the weight Wb n The unsharp mask with a signal structure that is greatly different from the unsharp mask of the target frame can be suppressed. The curve graph of formula 12 is as follows Figure 6 shown.

[0078] However, it should be understood that the above formulas 7 to 12 are only a non-limiting example of calculating the pixel mean difference cosine difference weight, and various modifications may be made to the above formulas 7 to 12 without departing from the scope of the inventive concept of the present application.

[0079] According to an embodiment of the inventive concept, the frame interval similarity of each unsharp mask may correspond to the frame interval between the image frame corresponding to each unsharp mask and the target frame. Therefore, the frame interval weight corresponding to each unsharp mask may be determined based on the frame interval.

[0080] In one example, the smaller the frame interval, the greater the frame interval weight, and the larger the frame interval, the smaller the frame interval weight. Various methods can be used to construct pixel mean difference cosine difference weights, and this application does not make specific limitations on this. In one example, the frame interval weight of each unsharp mask can be calculated by the following formula 13.

[0081]

[0082] Among them, Wc n represents the frame interval weight of the nth unsharp mask, C is a constant between 0 and 1, and FPS represents the frame rate. According to formula 13, Wc n ∈[C, 1] and Wc n Characterizes the length of the time interval between the nth frame and the target frame. The size of the constant C determines Wc n In one example, the constant C as the convergence accuracy may be 0.01 or less. However, the present application is not limited thereto, and the constant C may be a constant greater than 0.01. Therefore, the weight Wc n The unsharp mask with a long time interval from the unsharp mask of the target frame can be suppressed. The curve graph of formula 13 is as follows Figure 7 shown.

[0083] However, it should be understood that the above formula 13 is only a non-limiting example of calculating the frame interval weight, and various modifications may be made to the above formula 13 without departing from the spirit and scope of the inventive concept of the present application.

[0084] In addition, as mentioned above Figure 1 As described above, the number of multiple image frames can be determined based on the frame rate. For example, when the target frame is the first frame (ie, t=1) and the FPS is 30, according to Formula 13, when n approaches the FPS, Wc n The convergence is C. However, in practical applications, using FPS image frames for calculating multi-frame optimized unsharp masks will result in serious time consumption, so it is necessary to reduce the number of multiple image frames N. In a preferred example, the number of multiple image frames N can be determined based on the frame rate, the frame number threshold and the accuracy C, as shown in the following formula 14.

[0085]

[0086] Where C is a constant between 0 and 1 and determines Wc n The convergence accuracy of FPS is , FPS represents the frame rate, and D represents the frame number threshold. Therefore, the number N of multiple image frames can be directly determined to be the same as the FPS, or the number N of multiple image frames can be preferably determined based on Formula 14.

[0087] However, it should be understood that the above formula 14 is only a non-limiting example of calculating the number of multiple image frames, and various modifications may be made to the above formula 14 without departing from the spirit and scope of the inventive concept of the present application.

[0088] According to the above formula 14, the convergence accuracy C and the frame number threshold D are used to jointly determine the number of unsharp masks to be fused, thereby reducing the consumption of the system while ensuring the optimization quality.

[0089] Return to reference Figure 4 , in step S312, the similarity weight corresponding to each unsharp mask can be determined by combining the at least one weight of each unsharp mask. For example, the similarity weight corresponding to the first unsharp mask (hereinafter referred to as the first similarity weight) can be determined by combining at least one weight of the first unsharp mask (for example, at least one of the first weight, the second weight, the third weight, etc.); ...; the similarity weight corresponding to the i-th unsharp mask (hereinafter referred to as the i-th similarity weight) can be determined by combining at least one weight of the i-th unsharp mask (for example, at least one of the first weight, the second weight, the third weight, etc.); ...; the similarity weight corresponding to the N-th unsharp mask (hereinafter referred to as the N-th similarity weight) can be determined by combining at least one weight of the N-th unsharp mask (for example, at least one of the first weight, the second weight, the third weight, etc.).

[0090] In addition, in one example, the result of multiplying each element in the set consisting of at least one weight of each unsharp mask and 1 (e.g., continuous multiplication) can be determined as the similarity weight corresponding to each unsharp mask. For example, the result of multiplying each element in the first set consisting of at least one weight of the first unsharp mask (e.g., at least one of the first weight, the second weight, the third weight, etc.) and 1 can be determined as the first similarity weight corresponding to the first unsharp mask; ...; the result of multiplying each element in the i-th set consisting of at least one weight of the i-th unsharp mask (e.g., at least one of the first weight, the second weight, the third weight, etc.) and 1 can be determined as the i-th similarity weight corresponding to the i-th unsharp mask; ...; the result of multiplying each element in the N-th set consisting of at least one weight of the N-th unsharp mask (e.g., at least one of the first weight, the second weight, the third weight, etc.) and 1 can be determined as the N-th similarity weight corresponding to the N-th unsharp mask. That is to say, when at least one weight of the unsharp mask includes only one weight, the one weight can be directly used as the similarity weight of the unsharp mask; when at least one weight of the unsharp mask includes multiple weights, the result of multiplying each of the multiple weights can be used as the similarity weight of the unsharp mask.

[0091] In addition, although the above example describes at least one weight of each unsharp mask by taking "at least one of the first weight, the second weight, the third weight, etc." as an example, the inventive concept is not limited thereto. In another example, at least one weight of each unsharp mask used in the step of the similarity weight corresponding to each unsharp mask may be different from each other. For example, the result of multiplying each element in the first set consisting of at least one weight of the first unsharp mask (e.g., the first weight) and 1 can be determined as the first similarity weight corresponding to the first unsharp mask; ...; the result of multiplying each element in the i-th set consisting of at least one weight of the i-th unsharp mask (e.g., the first weight and the second weight) and 1 can be determined as the i-th similarity weight corresponding to the i-th unsharp mask; ...; the result of multiplying each element in the N-th set consisting of at least one weight of the N-th unsharp mask (e.g., the first weight, the second weight, and the third weight) and 1 can be determined as the N-th similarity weight corresponding to the N-th unsharp mask.

[0092] In addition, as described above, when the target frame is the first frame, since the first unsharp mask of the first frame must have the greatest similarity with itself, the first similarity weight corresponding to the first frame must also be the maximum value when the similarity is greater and the similarity weight is greater. Therefore, in this case, the first similarity weight corresponding to the first frame can be directly set to the maximum value, and the determination / calculation of the similarity weight, at least one similarity and at least one weight for the first frame as the target frame in steps S310, S311 and S312 are omitted, respectively, thereby saving computing resources and improving computing efficiency.

[0093] For example, in the case where the at least one weight includes an overall pixel difference weight, a pixel mean difference cosine difference weight, and a frame interval weight, step S312 may be represented by the following formula 15.

[0094] W n =Wa n ×Wb n ×Wc n Formula 15

[0095] According to formula 15, the weight Wa n The unsharp mask with a large difference in pixel mean from the unsharp mask of the target frame can be suppressed, and the weight Wb n The unsharp mask with a large signal structure difference from the unsharp mask of the target frame can be suppressed, and the weight Wc nThe unsharp mask that is separated from the unsharp mask of the target frame by a long time interval can be suppressed. Therefore, in the fusion process, if invalid frames are caused by environmental changes, jitter or too long frame acquisition time, the fusion optimization algorithm proposed in the present application can automatically suppress the weights of these invalid frames, playing a role similar to automatic frame screening. However, it should be understood that the above formula 15 is only a non-limiting example of calculating multi-frame optimized unsharp masks, and various modifications can be made to the above formula 15 without departing from the spirit and scope of the inventive concept of the present application.

[0096] Through step 310, step 311, step 312, step 320 and step 330, N unsharp masks of the target frame and the N-1 frames thereafter are extracted respectively, and the N unsharp masks are fused into a multi-frame optimized unsharp mask using at least one of the overall pixel difference weight, the pixel mean difference cosine difference weight and the frame interval weight, so that at least one of the overall pixel difference weight, the pixel mean difference cosine difference weight and the frame interval weight jointly determines the fusion weight of each mask, playing a role similar to automatic frame screening. Through this multi-frame fusion optimization method, the energy of edge noise and texture-like noise can be further dispersed or offset, while further protecting the original signal of the mask. Therefore, the noise introduced by the unsharp mask is further reduced, and the target frame is made to present a better sharpening effect by using the multi-frame optimized unsharp mask to be superimposed with the original signal of the target frame.

[0097] Figure 8 is a block diagram illustrating an image optimizing apparatus for optimizing a target frame according to an embodiment of the inventive concept.

[0098] Reference Figure 8 , the image optimization device 1000 for optimizing a target frame according to an embodiment of the inventive concept may include a processor 1100. The processor 1100 may execute the image optimization method for optimizing a target frame as described above. The processor 1100 may include (but is not limited to) a central processing unit (CPU), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), a system on chip (SoC), a microprocessor, an application specific integrated circuit (ASIC), etc.

[0099] The image optimization method for optimizing the target frame according to the embodiment of the present disclosure can be written as a computer program and stored on a computer-readable storage medium. When the computer program is executed by a processor, the image optimization method for optimizing the target frame as described above can be implemented. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. In one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0100] According to the image optimization method for optimizing the target frame of the present invention, the target frame is optimized using a multi-frame optimized unsharp mask that is the result of averaging the unsharp masks of multiple image frames including the target frame, instead of using the unsharp mask of the target frame to optimize the target frame, so that the energy of edge noise and texture-like noise in the target frame can be dispersed or offset, while protecting the original signal of the mask. Ultimately, the noise introduced by the unsharp mask is reduced, and the target frame is made to present a better sharpening effect by using the multi-frame optimized unsharp mask and superimposing it with the original signal of the target frame.

[0101] While the invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. An image optimization method, comprising: receiving a plurality of image frames including a target frame; determining an unsharp mask for each image frame in the plurality of image frames; Determine a result of averaging the unsharp masks of the multiple image frames as a multi-frame optimized unsharp mask of the target frame; Based on the multi-frame optimized unsharp mask of the target frame, the target frame is optimized. The step of determining the result of averaging the unsharp masks of the multiple image frames as the multi-frame optimized unsharp mask of the target frame comprises: Determine a similarity weight corresponding to each unsharp mask, wherein the greater the similarity between each unsharp mask and the unsharp mask of the target frame, the greater the similarity weight; performing weighted averaging on the unsharp masks of the plurality of image frames based on the determined similarity weights; The result of performing weighted averaging is determined as the multi-frame optimized unsharp mask of the target frame. The step of determining the similarity weight corresponding to each unsharp mask comprises: Determining at least one weight corresponding to each unsharp mask based on at least one similarity between each unsharp mask and an unsharp mask of the target frame; determining a similarity weight corresponding to each unsharp mask by combining the at least one weight of each unsharp mask, Among them, the at least one similarity of each unsharp mask includes at least one of the overall pixel difference similarity, the pixel mean difference cosine difference similarity and the frame interval similarity, and the at least one weight corresponding to each unsharp mask includes at least one of the overall pixel difference weight corresponding to the overall pixel difference similarity, the pixel mean difference cosine difference weight corresponding to the pixel mean difference cosine difference similarity and the frame interval weight corresponding to the frame interval similarity.

2. The image optimization method according to claim 1, wherein: The plurality of image frames are a plurality of continuous image frames, the target frame is the first frame of the plurality of image frames, wherein the number of the plurality of image frames is determined based on a frame rate, The step of determining an unsharp mask for each of the plurality of image frames comprises: By performing smoothing filtering on each image frame, a blurred image frame corresponding to each image frame is generated; The difference between each image frame and the blurred image frame is determined as an unsharp mask for each image frame.

3. The image optimization method according to claim 1, wherein: The step of determining a similarity weight corresponding to each unsharp mask by combining the at least one weight of each unsharp mask comprises: A result of multiplying each element in a set consisting of the at least one weight of each unsharp mask and 1 is determined as a similarity weight corresponding to each unsharp mask.

4. The image optimization method according to claim 1, wherein: The overall pixel difference similarity of each unsharp mask corresponds to the sum of pixel differences calculated based on the pixel differences between each pixel of each unsharp mask and the corresponding pixel of the unsharp mask of the target frame. The closer the pixel difference sum is to 0 or the closer the pixel difference sum is to the average of the pixel difference sums of the unsharp masks of the multiple image frames, the greater the overall pixel difference weight is. Among them, the pixel mean difference cosine difference similarity of each unsharp mask corresponds to the angle between the resultant vector corresponding to the pixel mean difference vector of each pixel of each unsharp mask and the resultant vector corresponding to the pixel mean difference vector of each pixel of the unsharp mask of the target frame, and the pixel mean difference vector of each pixel is determined based on the difference between the pixel value of each pixel and the pixel mean of the unsharp mask including the pixel and the position of each pixel in the unsharp mask including the pixel, and the smaller the angle is, the greater the pixel mean difference cosine difference weight is. The frame interval similarity of each unsharp mask corresponds to the frame interval between the image frame corresponding to each unsharp mask and the target frame, and the smaller the frame interval is, the greater the frame interval weight is.

5. The image optimization method according to claim 1, wherein: Based on the multi-frame optimized unsharp mask of the target frame, the steps of optimizing the target frame include: The sum of the multi-frame optimized unsharp mask of the target frame and the target frame is determined as the optimization result of the target frame.

6. An image optimization device, comprising a processor, wherein the processor is configured to execute the image optimization method according to any one of claims 1 to 5.

7. A computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the image optimization method according to any one of claims 1 to 5.

Images