Sharpening processing method and apparatus, and device and storage medium
The target prediction model predicts the optimal sharpening intensity, and the sharpening intensity label is determined using the performance distortion index value, which solves the problem of excessive bandwidth consumption of sharpening processing in the prior art, and achieves a balance between object quality and bandwidth consumption.
Patent Information
- Application Number
- PCT/CN2024/094919
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2024-05-23
- Publication Date
- 2024-11-28
AI Technical Summary
Existing sharpening processing technologies are difficult to effectively balance the quality and bandwidth consumption of image or video objects. Excessive sharpening intensity may lead to a decrease in object quality and an increase in bandwidth consumption.
Using the target prediction model, trained based on sample objects and sharpening intensity labels, predict the best sharpening intensity, determine the sharpening intensity label through the rate distortion performance index value, and perform sharpening processing on this basis to improve the quality of the object At the same time, reduce bandwidth consumption.
The effect of significantly reducing bandwidth consumption without reducing object quality is achieved. By accurately predicting the optimal sharpening intensity, the performance indicators of sharpening processing are optimized.
Smart Images

Figure CN2024094919_28112024_PF_FP_ABST
Abstract
Description
Sharpening processing method, device, equipment and storage medium
[0001] This application claims priority to Chinese Patent Application No. 202310594642.8 filed on May 24, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field
[0002] The embodiments of the present disclosure relate to a sharpening processing method, apparatus, device and storage medium. Background Art
[0003] With the rapid development of computer technology, sharpening of objects such as images and videos is often required to improve their quality. However, increasing the sharpening intensity does not guarantee a continuous improvement in quality. Using too high a sharpening intensity can lead to deterioration of object quality. Furthermore, the bit information contained in the object increases with increasing sharpening intensity, resulting in increased bandwidth consumption when transmitting the object. Therefore, a sharpening method that effectively balances object quality and bandwidth consumption is urgently needed.
[0004] Summary of the Invention
[0005] The present disclosure provides a sharpening processing method, apparatus, device, and storage medium to effectively balance object quality and bandwidth consumption, thereby improving object quality while reducing bandwidth consumption.
[0006] In a first aspect, an embodiment of the present disclosure provides a sharpening processing method, comprising:
[0007] Obtain the target object to be sharpened;
[0008] Inputting the target object into a target prediction model to predict sharpening intensity, wherein the target prediction model is pre-trained based on sample objects and sharpening intensity labels corresponding to the sample objects, and the sharpening intensity labels are determined based on rate-distortion performance index values corresponding to a plurality of preset sharpening intensities;
[0009] The target object is sharpened based on the target sharpening strength output by the target prediction model.
[0010] In a second aspect, the present disclosure further provides a sharpening processing device, comprising:
[0011] A target object acquisition module, used to acquire the target object to be sharpened;
[0012] a sharpening intensity prediction module, configured to input the target object into a target prediction model to predict the sharpening intensity, wherein the target prediction model is pre-trained based on sample objects and sharpening intensity labels corresponding to the sample objects, and the sharpening intensity labels are determined based on rate-distortion performance index values corresponding to a plurality of preset sharpening intensities;
[0013] A sharpening processing module is used to perform sharpening processing on the target object based on the target sharpening intensity output by the target prediction model.
[0014] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:
[0015] one or more processors;
[0016] a storage device for storing one or more programs,
[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the sharpening processing method as described in any one of the embodiments of the present disclosure.
[0018] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the sharpening processing method as described in any one of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.
[0020] FIG1 is a schematic flow chart of a sharpening processing method provided by an embodiment of the present disclosure;
[0021] FIG2 is a schematic flow chart of another sharpening method provided by an embodiment of the present disclosure;
[0022] FIG3 is an example of a rate-distortion curve corresponding to a sample video involved in an embodiment of the present disclosure;
[0023] FIG4 is a diagram illustrating an example of determining a first performance indicator value according to an embodiment of the present disclosure;
[0024] FIG5 is a diagram illustrating an example of determining a second performance indicator value according to an embodiment of the present disclosure;
[0025] FIG6 is a diagram showing an example of fitting of first performance indicator values corresponding to a plurality of preset sharpening intensities according to an embodiment of the present disclosure;
[0026] FIG7 is a diagram showing an example of fitting of second performance indicator values corresponding to a plurality of preset sharpening intensities according to an embodiment of the present disclosure;
[0027] FIG8 is a schematic flow chart of another sharpening processing method provided by an embodiment of the present disclosure;
[0028] FIG9 is a diagram illustrating an example of the architecture of a target prediction model according to an embodiment of the present disclosure;
[0029] FIG10 is a schematic structural diagram of a sharpening processing device provided by an embodiment of the present disclosure; and
[0030] FIG11 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0032] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0033] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.
[0034] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0035] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".
[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0037] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.
[0038] Figure 1 is a schematic flow diagram of a sharpening method provided by an embodiment of the present disclosure. This embodiment is applicable to sharpening objects such as videos or images, and is particularly applicable to sharpening objects such as videos or images prior to encoding and compression. The method can be performed by a sharpening device, which can be implemented in software and / or hardware. Optionally, the device can be implemented in an electronic device, such as a mobile terminal, a PC, or a server.
[0039] As shown in FIG1 , the sharpening processing method specifically includes the following steps:
[0040] S110: Obtain a target object to be sharpened.
[0041] The target object may be an object whose quality needs to be improved through sharpening. For example, the target object may be, but is not limited to, a target image or a target video. Sharpening effectively improves the quality of the object by emphasizing texture, reducing blur, and directing user attention to certain areas.
[0042] Specifically, if the object to be sharpened is an image, a target image to be sharpened can be obtained so as to improve the image quality through sharpening. If the object to be sharpened is a video, a target video to be sharpened can be obtained so as to improve the video quality through sharpening. For example, based on the shooting scene corresponding to each frame of the video image in the original video, the original video can be divided into one or more target videos to be sharpened. Each frame of the video image in each target video corresponds to the same shooting scene, so that the target video in each shooting scene can be sharpened separately, thereby ensuring the video sharpening effect and further improving the video quality.
[0043] S120. Input the target object into a target prediction model to predict the sharpening intensity, wherein the target prediction model is pre-trained based on sample objects and sharpening intensity labels corresponding to the sample objects, and the sharpening intensity labels are determined based on rate-distortion performance index values corresponding to multiple preset sharpening intensities.
[0044] Among them, the preset sharpening intensity can be a plurality of discrete sharpening intensities pre-set based on business needs. For example, the sharpening intensity range can be divided based on the sharpening intensity interval to obtain a plurality of preset sharpening intensities. For example, when the sharpening intensity interval is 0.5, the preset sharpening intensity may include: 0, 0.5, 1.0, 1.5, 2.0, 2.5 and 3.0. Among them, when the preset sharpening intensity is 0, it means no sharpening is performed, which can be used as a reference sharpening intensity to measure the sharpening effect of the non-zero preset sharpening intensity, that is, to measure the quality of the object after sharpening and the bandwidth consumption compared to the unsharpened case. The smaller the sharpening intensity interval in the preset sharpening intensity, the more accurate the determination of the sharpening intensity label.
[0045] The sharpening strength label refers to the optimal sharpening strength that effectively improves object quality while reducing bandwidth consumption. The rate-distortion performance index value can be used to evaluate the bandwidth consumption or change in object quality after object sharpening. Bandwidth consumption can be characterized by the bit information of the object after sharpening. For example, if the sharpened object is a video, the object bit information can refer to the video bitrate, that is, the number of bits per second (bps) transmitted per unit time. If the sharpened object is an image, the object bit information can refer to the pixel depth, that is, the number of bits per pixel (bpp). The higher the video bitrate or pixel depth, the greater the bandwidth consumed when transmitting the video or image. Object quality can be characterized by quality assessment indicators. For example, quality assessment indicators may include Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), or Visual Multimethod Assessment Fusion (VMAF). It should be noted that lower bit information indicates higher quality assessment index values, indicating higher performance of the object after sharpening.
[0046] Among them, each preset sharpening intensity corresponds to a rate-distortion performance index value, which is used to characterize the performance of the object after being sharpened using the preset sharpening intensity. The rate-distortion performance index value may include: a first performance index value or a second performance index value. Among them, the first performance index value may refer to the bit information change of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same quality. For example, the first performance index value refers to: when the unsharpened object (that is, the reference sharpening intensity is 0) and the object sharpened using the preset sharpening intensity have the same quality (that is, the same quality assessment index value), the bit information saved by the object after being sharpened using the preset sharpening intensity compared to the unsharpened object (that is, the saved bandwidth consumption). For example, the first performance index value can be characterized by BD-Rate (Bjontegaard Delta-Rate).
[0047] The second performance indicator information may refer to a change in a quality assessment indicator of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same bit information. For example, the second performance indicator value may refer to the improvement in the quality assessment indicator (i.e., improved object quality) of an unsharpened object (i.e., a reference sharpening intensity of 0) compared to an object sharpened with the preset sharpening intensity when the two objects have the same bit information (i.e., the same bandwidth consumption). For example, when the quality assessment indicator is VMAF, the second performance indicator value may be characterized using BD-VMAF.
[0048] The target prediction model may be a neural network model for predicting the optimal sharpening intensity. The network architecture of the target prediction model may be any regression prediction network architecture. The sample object and the target object are of the same type. For example, the sample object may be a sample image or a sample video. The target prediction model trained using the sample images can accurately predict the optimal sharpening intensity of the target image. The target prediction model trained using the sample video can accurately predict the optimal sharpening intensity of the target video.
[0049] Specifically, the optimal sharpening intensity corresponding to the sample object can be determined in advance based on the rate-distortion performance index value of the sample object at each preset sharpening intensity, and the optimal sharpening intensity can be used as the sharpening intensity label for model training. For example, the rate-distortion performance index value corresponding to each preset sharpening intensity can be compared, and the preset sharpening intensity with the highest performance can be used as the sharpening intensity label. Alternatively, curve fitting can be performed based on all rate-distortion performance index values corresponding to all preset sharpening intensities to obtain a fitting curve between the preset sharpening intensity and the rate-distortion performance index value, and the sharpening intensity with the highest performance in the fitting curve can be determined as the sharpening intensity label. Based on the sample object and the sharpening intensity label corresponding to the sample object, the prediction model to be trained is trained so that the target prediction model after the training has the ability to predict the optimal sharpening intensity. For example, the training process of the target prediction model is as follows: the sample object is input into the prediction model to be trained, and based on the output of the prediction model, the sample sharpening intensity corresponding to the sample object is obtained, and based on the loss function, the training error is determined according to the sample sharpening intensity and the sharpening intensity label, and the training error is back-propagated to the prediction model, and the model parameters in the prediction model are adjusted until the preset convergence conditions are reached, such as when the number of iterations is equal to the preset number, or when the training error changes tend to be stable, the prediction model training is determined to be completed, and the prediction model after training is used as the target prediction model. In actual use, the target object to be sharpened is input into the trained target prediction model. The target prediction model can perform network processing on the input target object, predict the optimal sharpening intensity, and output it. For example, the target prediction model can extract high-frequency feature information and ordinary image feature information from the input target object, and predict the sharpening intensity based on the extracted characteristic information, thereby realizing frequency-assisted sharpening intensity prediction and further improving the prediction accuracy of sharpening intensity.
[0050] S130 , performing sharpening processing on the target object based on the target sharpening strength output by the target prediction model.
[0051] The target sharpening intensity may refer to the optimal sharpening intensity of the target object, that is, the sharpening intensity that improves the quality of the target object while reducing bandwidth consumption. Specifically, the server may use the optimal target sharpening intensity output by the target prediction model to sharpen the target object. After sharpening, the sharpened target object is compressed and encoded, and the encoded object is sent to the client, thereby ensuring the quality of the sharpened target object while reducing bandwidth consumption.
[0052] The technical solution of the embodiment of the present disclosure determines the sharpening intensity label corresponding to the sample object in advance based on the rate-distortion performance index values corresponding to multiple preset sharpening intensities. The rate-distortion performance index value can be used to reflect the performance of the object quality and bandwidth consumption after using the preset sharpening intensity, so that the optimal sharpening intensity that can effectively balance the object quality and bandwidth consumption can be obtained based on the rate-distortion performance index value and used as the sharpening intensity label. By training based on the sample objects and the sharpening intensity labels corresponding to the sample objects, a target prediction model that can predict the optimal sharpening intensity can be obtained. By inputting the target object to be sharpened into the target prediction model to predict the sharpening intensity, and based on the optimal target sharpening intensity output by the target prediction model, the target object is sharpened, thereby improving the quality of the target object while reducing bandwidth consumption.
[0053] FIG2 is a flowchart illustrating another sharpening method provided by an embodiment of the present disclosure. This embodiment, based on the aforementioned embodiments, further describes the process of determining a sharpening intensity label corresponding to a sample object. Explanations of terms that are identical or corresponding to those in the aforementioned embodiments are omitted here.
[0054] As shown in FIG2 , a sharpening processing method specifically includes the following steps:
[0055] S210: Determine a first rate-distortion curve of the sample object at a reference sharpening intensity, and a second rate-distortion curve at each preset sharpening intensity.
[0056] Among them, the reference sharpening intensity can be a sharpening intensity used as a comparison benchmark. The object quality and bandwidth consumption at each preset sharpening intensity are compared with the object quality and bandwidth consumption at the reference sharpening intensity, so as to obtain the sharpening intensity with the highest performance among all preset sharpening intensities. For example, the reference sharpening intensity can be 0, so as to measure the change in object quality and bandwidth consumption at each sharpening relative to the object quality and bandwidth consumption when not sharpening. The reference sharpening intensity may be included in all preset sharpening intensities, or may not be included.
[0057] Among them, the rate-distortion curve may refer to a curve of changes between bit information and quality assessment index values, such as an RD curve. When the sample object is a video object, the bit information is the number of bits transmitted per unit time (bps), that is, the video bitrate. When the sample object is a sample image, the bit information is the number of bits corresponding to a single pixel (bpp), that is, the pixel depth. It should be noted that the lower the bit information, the higher the quality assessment index value, indicating that the performance of the object after sharpening is higher. The first rate-distortion curve may refer to a rate-distortion curve obtained after sharpening the sample object using a reference sharpening intensity. The second rate-distortion curve may refer to a rate-distortion curve obtained after sharpening the sample object using a preset sharpening intensity. The preset sharpening intensity corresponds one-to-one to the second rate-distortion curve.
[0058] Specifically, the process for determining the first rate-distortion curve corresponding to the reference sharpening intensity and the process for determining the second rate-distortion curve corresponding to each preset sharpening intensity are identical; only the sharpening intensities used are different. For example, after sharpening a sample object using the reference sharpening intensity or the preset sharpening intensity, the sharpened sample object is compressed and encoded multiple times using multiple compression rates to obtain multiple bits of information and corresponding multiple quality assessment index values. A corresponding rate-distortion curve can then be drawn based on all the bits of information and all the quality assessment index values.
[0059] Exemplarily, determining the first rate-distortion curve of the sample object at a reference sharpening intensity may include: sharpening the sample object based on the reference sharpening intensity, and performing multiple compression encoding on the sharpened sample object to obtain multiple coded objects corresponding to the sample object; performing quality assessment on each coded object to obtain a quality assessment index value corresponding to each coded object; and determining the first rate-distortion curve of the sample object at the reference sharpening intensity based on the bit information and quality assessment index value corresponding to each coded object.
[0060] Specifically, if the reference sharpening intensity is 0, sharpening is not required, and the sample object can be directly compressed and encoded multiple times using different compression rates to obtain the encoded object after each encoding. By performing quality assessment and bit calculation on each encoded object, the quality assessment index value and bit information corresponding to each encoded object are obtained. With the bit information as the horizontal coordinate and the quality assessment index value as the vertical coordinate, the coordinate points corresponding to each encoded object are drawn, and the coordinate points are connected to obtain the first rate-distortion curve of the sample object under the reference sharpening intensity. Referring to Figure 3, when the sample object is a sample video, the corresponding bit information is the bit rate, and the quality assessment index value is VMAF, five different compression rates, namely 21, 24, 27, 30 and 33 (equivalent to five different encoders), are used to compress and encode the sample object sharpened using the reference sharpening intensity of 0 five times, so that five different coordinate points can be obtained and connected, and then the first rate-distortion curve corresponding to the reference sharpening intensity of 0 can be obtained.
[0061] Exemplarily, determining the second rate-distortion curve of the sample object at each preset sharpening intensity may include: sharpening the sample object based on each preset sharpening intensity, and performing multiple compression encoding on the sharpened sample object to obtain multiple coded objects corresponding to the sample object; performing quality assessment on each coded object to obtain a quality assessment index value corresponding to each coded object; and determining the second rate-distortion curve of the sample object at each preset sharpening intensity based on the bit information and quality assessment index value corresponding to each coded object.
[0062] Specifically, similar to the reference sharpening intensity, for each preset sharpening intensity, the sample object is sharpened using the preset sharpening intensity, and the sharpened sample object is compressed and encoded multiple times using different compression rates to obtain the encoded object after each encoding. By performing quality assessment and bit calculation on each encoded object, the quality assessment index value and bit information corresponding to each encoded object are obtained. With the bit information as the horizontal axis and the quality assessment index value as the vertical axis, the coordinate points corresponding to each encoded object are plotted, and the coordinate points are connected to obtain the second rate-distortion curve of the sample object under the preset sharpening intensity. Referring to Figure 3, when the sample object is a sample video, the corresponding bit information is the bit rate, and the quality assessment index value is VMAF, five different compression rates, namely 21, 24, 27, 30 and 33 (equivalent to five different encoders), are used to compress and encode the sample object sharpened using each preset sharpening intensity five times, so that five different coordinate points can be obtained and connected to obtain the second rate-distortion curve corresponding to each preset sharpening intensity.
[0063] S220: Determine a rate-distortion performance indicator value of each preset sharpening intensity relative to a reference sharpening intensity based on the first rate-distortion curve and the second rate-distortion curve.
[0064] The rate-distortion performance indicator value may include: a first performance indicator value or a second performance indicator value. The first performance indicator value may refer to a bit information change of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same quality. The second performance indicator information may refer to a quality assessment indicator change of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same bit information.
[0065] It should be noted that a negative value for the first performance indicator indicates that, while maintaining the same quality, bit information is reduced and performance is improved. A positive value for the first performance indicator indicates that, while maintaining the same quality, bit information is increased and performance is degraded. A positive value for the second performance indicator indicates that, while maintaining the same bit information, quality and performance are improved. A negative value for the second performance indicator indicates that, while maintaining the same bit information, quality and performance are degraded.
[0066] Specifically, for each preset sharpening strength, the area between the first rate-distortion curve corresponding to the reference sharpening strength and the second rate-distortion curve corresponding to the preset sharpening strength can be calculated within the horizontal coordinate overlapping range (such as intmin-intmax) (that is, the integral difference within the horizontal coordinate overlapping range), and the area is divided by the horizontal coordinate length corresponding to the horizontal coordinate overlapping range (that is, the difference between intmax and intmin), and the division result is used as the rate-distortion performance index value corresponding to the preset sharpening strength. Referring to Figure 4, when the rate-distortion performance index value is the first performance index value, the first rate-distortion curve corresponding to the reference sharpening intensity 0 in Figure 3 and the second rate-distortion curve corresponding to the preset sharpening intensity 0.5 are transformed to obtain two transformed curves with VMAF as the horizontal coordinate and log (bit rate) as the vertical coordinate, and the area of the two curves within the horizontal coordinate overlapping range intmin-intmax is calculated (such as the shaded part in Figure 4), and the area is divided by the horizontal coordinate length corresponding to the horizontal coordinate overlapping range. The division result is the first performance index value corresponding to the preset sharpening intensity 0.5, that is, BD-Ratesharp0.5=-0.3252, which means that when the preset sharpening intensity 0.5 has the same quality as the reference sharpening intensity 0, it will save 32.52% of the bit rate.
[0067] Referring to Figure 5, when the rate-distortion performance index value is the second performance index value, the first rate-distortion curve corresponding to the reference sharpening intensity of 0 and the second rate-distortion curve corresponding to the preset sharpening intensity of 0.5 in Figure 3 are transformed to obtain two transformed curves with log (bit rate) as the horizontal coordinate and VMAF as the vertical coordinate, and the area of the two curves within the horizontal coordinate overlapping range intmin-intmax is calculated (such as the shaded part in Figure 5), and the area is divided by the horizontal coordinate length corresponding to the horizontal coordinate overlapping range. The division result is the second performance index value corresponding to the preset sharpening intensity of 0.5, that is, BD-VMAFsharp0.5=2.8295, which means that when the preset sharpening intensity of 0.5 and the reference sharpening intensity of 0 have the same bit rate, the quality assessment index value will be increased by 2.8295.
[0068] It should be noted that if the transformation curve corresponding to the preset sharpening intensity is below the transformation curve corresponding to the reference sharpening intensity, the calculated rate-distortion performance index value is negative. If the transformation curve corresponding to the preset sharpening intensity is above the transformation curve corresponding to the reference sharpening intensity, the calculated rate-distortion performance index value is positive.
[0069] S230: Determine a sharpening intensity label corresponding to the sample object based on the rate-distortion performance index value corresponding to each preset sharpening intensity.
[0070] Specifically, the rate-distortion performance index values corresponding to each preset sharpening intensity can be compared, and the preset sharpening intensity with the highest performance can be used as the sharpening intensity label. Alternatively, curve fitting can be performed based on all rate-distortion performance index values corresponding to all preset sharpening intensities to obtain a fitting curve between the preset sharpening intensity and the rate-distortion performance index value, and the sharpening intensity with the highest performance in the fitting curve can be determined as the sharpening intensity label. For example, referring to Figure 6, when the rate-distortion performance index value is the first performance index value, curve fitting is performed on the first performance index value corresponding to each preset sharpening intensity obtained using the method of Figures 3 and 4 to obtain a fitting curve between the sharpening intensity and the first performance index value. Since the smaller the first performance index value, the more bit rate is saved, the sharpening intensity corresponding to the minimum value of the first performance index in the fitting curve can be used as the sharpening intensity label corresponding to the sample object.
[0071] For another example, referring to FIG7 , when the rate-distortion performance index value is the second performance index value, a curve fitting is performed on the second performance index value corresponding to each preset sharpening intensity obtained using the methods of FIG3 and FIG5 , obtaining a fitting curve between the sharpening intensity and the second performance index value. Since a larger second performance index value indicates higher video quality, the sharpening intensity corresponding to the maximum value of the second performance index in the fitting curve can be used as the sharpening intensity label corresponding to the sample object.
[0072] Exemplarily, S230 may include: if the rate-distortion performance index value is a first performance index value, then determining the preset sharpening intensity corresponding to the minimum first performance index value as the sharpening intensity label corresponding to the sample object; if the rate-distortion performance index value is a second performance index value, then determining the preset sharpening intensity corresponding to the maximum first performance index value as the sharpening intensity label corresponding to the sample object.
[0073] Specifically, referring to the rate-distortion performance index value corresponding to each preset sharpening intensity in Figure 6 or Figure 7, the preset sharpening intensity of 0.5 is the sharpening intensity that saves the most bit rate and has the highest quality compared to the reference sharpening intensity of 0, so the preset sharpening intensity of 0.5 can be directly used as the sharpening intensity label corresponding to the sample object.
[0074] S240: Perform training based on the sample objects and the sharpening intensity labels corresponding to the sample objects to obtain a target prediction model.
[0075] S250: Input the target object to be sharpened into the target prediction model to predict the sharpening intensity.
[0076] S260: Sharpening the target object based on the target sharpening strength output by the target prediction model.
[0077] The technical solution of the embodiment of the present disclosure can accurately determine the rate-distortion performance index value of each preset sharpening intensity relative to the reference sharpening intensity based on the first rate-distortion curve of the sample object at the reference sharpening intensity and the second rate-distortion curve at each preset sharpening intensity, and based on the rate-distortion performance index value corresponding to each preset sharpening intensity, the optimal sharpening intensity can be screened out as the sharpening intensity label, thereby improving the accuracy of the model in predicting the optimal sharpening intensity, thereby improving the object quality while reducing bandwidth consumption.
[0078] FIG8 is a flow chart illustrating another sharpening method provided by an embodiment of the present disclosure. In this embodiment, based on the aforementioned embodiments, a target prediction model may include an image feature extraction sub-model, a high-frequency feature extraction sub-model, a feature fusion sub-model, and a regression prediction sub-model. The sharpening intensity prediction process is described in detail for this target prediction model. Explanations of terms that are identical or corresponding to those in the aforementioned embodiments are omitted here.
[0079] As shown in FIG8 , a sharpening processing method specifically includes the following steps:
[0080] S310: Obtain a target object to be sharpened.
[0081] S320: Input the target object into the image feature extraction sub-model in the target prediction model to extract image features and obtain extracted image feature information.
[0082] The image feature information may refer to complex feature information including bit information and quality information. The image feature extraction sub-model may include a lightweight multi-layer convolutional neural network to achieve efficient extraction of image feature information.
[0083] Specifically, referring to FIG9 , the target object can be input into the image feature extraction submodel in the target prediction model for efficient extraction of complex features, and the image feature extraction submodel can output the extracted image feature information containing bit information and quality information.
[0084] S330: Input the target object into the high-frequency feature extraction sub-model in the target prediction model to extract high-frequency features and obtain extracted high-frequency feature information.
[0085] The high-frequency feature information may refer to edge feature information within the target object. When sharpening the target object, the edge information within the target object is sharpened to make the edge information clearer. Edge information, as reflected in the spectrum, is equivalent to high-frequency information.
[0086] Specifically, referring to Figure 9 , the target object can also be input into the high-frequency feature extraction sub-model within the target prediction model for high-frequency feature extraction. The high-frequency feature extraction sub-model then outputs the extracted high-frequency feature information containing edge information. By extracting this high-frequency feature information, edge information within the target object can be obtained, allowing the model to more accurately predict the optimal sharpening intensity based on this edge information.
[0087] For example, referring to FIG9 , the high-frequency feature extraction sub-model may include: a discrete transformation layer and a feature extraction layer. The discrete transformation layer is a module for performing discrete cosine transform (DCT) and inverse discrete cosine transform (IDCT) on the target object to extract high-frequency components. The feature extraction layer may extract high-frequency features and dimensionally transform the extracted high-frequency component information so that the extracted high-frequency feature information has the same dimension as the image feature information, facilitating subsequent feature fusion.
[0088] Exemplarily, referring to FIG9 , S330 may include: inputting the target object into a discrete transformation layer for performing discrete cosine transform DCT and inverse discrete cosine transform IDCT to obtain high-frequency component information in the target object; inputting the high-frequency component information into a feature extraction layer for performing high-frequency feature extraction to obtain extracted high-frequency feature information.
[0089] S340: Input the image feature information and the high-frequency feature information into the feature fusion sub-model in the target prediction model to perform feature fusion, and obtain fused target feature information.
[0090] Specifically, referring to FIG9 , the feature fusion sub-model can perform feature fusion on the input image feature information and high-frequency feature information based on any fusion method. For example, the input image feature information and high-frequency feature information can be weighted summed, and the obtained summation result can be used as the target feature information.
[0091] Exemplarily, S340 may include: inputting the image feature information and the high-frequency feature information into a feature fusion sub-model to perform weighted fusion of residual connections to obtain fused target feature information.
[0092] Specifically, the feature fusion sub-model can be a residual connection sub-model, so that weighted fusion can be performed through residual connections, thereby reducing overfitting and preventing gradient vanishing. For example, the feature fusion sub-model can multiply the input image feature information and high-frequency feature information, and add the multiplication result with the image feature information, and the obtained addition result is used as the fused target feature information, thereby achieving weighted fusion of residual connections.
[0093] S350: Input the target feature information into the regression prediction sub-model in the target prediction model to perform regression prediction of the sharpening intensity, and obtain the predicted target sharpening intensity.
[0094] The regression prediction sub-model may be a nonlinear regression network for predicting the final sharpening intensity. Specifically, referring to FIG9 , the target feature information may be input into the regression prediction sub-model for regression prediction of the optimal sharpening intensity, and the predicted target sharpening intensity may be output.
[0095] S360: Sharpening the target object based on the target sharpening intensity output by the target prediction model.
[0096] The technical solution of the embodiment of the present disclosure adds a high-frequency feature extraction sub-model to the target prediction model, so that the extracted high-frequency feature information and image feature information can be used to more accurately predict the optimal sharpening intensity, thereby ensuring a balance between object quality and bandwidth consumption.
[0097] FIG10 is a schematic structural diagram of a sharpening processing device provided by an embodiment of the present disclosure. As shown in FIG10 , the device specifically includes: a target object acquisition module 410 , a sharpening intensity prediction module 420 and a sharpening processing module 430 .
[0098] Among them, the target object acquisition module 410 is used to obtain the target object to be sharpened; the sharpening intensity prediction module 420 is used to input the target object into the target prediction model to predict the sharpening intensity, wherein the target prediction model is pre-trained based on the sample object and the sharpening intensity label corresponding to the sample object, and the sharpening intensity label is determined based on the rate-distortion performance index value corresponding to multiple preset sharpening intensities; the sharpening processing module 430 is used to sharpen the target object based on the target sharpening intensity output by the target prediction model.
[0099] On the basis of the above technical solution, the device further includes:
[0100] a rate-distortion curve determining module, configured to determine a first rate-distortion curve of the sample object at a reference sharpening intensity, and a second rate-distortion curve at each preset sharpening intensity;
[0101] a rate-distortion performance index value determining module, configured to determine a rate-distortion performance index value of each preset sharpening intensity relative to the reference sharpening intensity based on the first rate-distortion curve and the second rate-distortion curve;
[0102] The sharpening strength label determination module is used to determine the sharpening strength label corresponding to the sample object based on the rate-distortion performance index value corresponding to each preset sharpening strength.
[0103] Based on the above technical solutions, the rate-distortion curve determination module is specifically used to:
[0104] Based on a reference sharpening intensity, the sample object is sharpened, and the sharpened sample object is compressed and encoded multiple times to obtain multiple coding objects corresponding to the sample object; quality assessment is performed on each coding object to obtain a quality assessment index value corresponding to each coding object; based on the bit information corresponding to each coding object and the quality assessment index value, a first rate-distortion curve of the sample object under the reference sharpening intensity is determined.
[0105] Based on the above technical solutions, when the sample object is a video object, the bit information corresponding to the encoding object is the number of bits transmitted per unit time; when the sample object is a sample image, the bit information corresponding to the encoding object is the number of bits corresponding to a single pixel.
[0106] On the basis of the above technical solutions, the rate-distortion performance index value includes: a first performance index value or a second performance index value;
[0107] The first performance indicator value refers to a bit information change amount of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same quality;
[0108] The second performance indicator information refers to a quality assessment indicator change of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same bit information.
[0109] Based on the above technical solutions, the sharpening intensity label determination module is specifically used to:
[0110] If the rate-distortion performance index value is the first performance index value, determining the preset sharpening intensity corresponding to the minimum first performance index value as the sharpening intensity label corresponding to the sample object;
[0111] If the rate-distortion performance index value is the second performance index value, the preset sharpening intensity corresponding to the maximum first performance index value is determined as the sharpening intensity label corresponding to the sample object.
[0112] On the basis of the above technical solutions, the target prediction model includes: an image feature extraction sub-model, a high-frequency feature extraction sub-model, a feature fusion sub-model and a regression prediction sub-model;
[0113] The sharpening strength prediction module 420 includes:
[0114] An image feature information extraction unit, configured to input the target object into the image feature extraction sub-model to extract image features and obtain extracted image feature information;
[0115] A high-frequency feature information extraction unit is used to input the target object into the high-frequency feature extraction sub-model to perform high-frequency feature extraction and obtain extracted high-frequency feature information;
[0116] a feature information fusion unit, configured to input the image feature information and the high-frequency feature information into the feature fusion sub-model for feature fusion, and obtain fused target feature information;
[0117] The sharpening intensity prediction unit is used to input the target feature information into the regression prediction sub-model to perform regression prediction of the sharpening intensity and obtain the predicted target sharpening intensity.
[0118] On the basis of the above technical solutions, the high-frequency feature extraction sub-model includes: a discrete transformation layer and a feature extraction layer;
[0119] The high-frequency feature information extraction unit is specifically used to:
[0120] The target object is input into the discrete transformation layer for discrete cosine transformation and inverse discrete cosine transformation to obtain high-frequency component information in the target object; the high-frequency component information is input into the feature extraction layer for high-frequency feature extraction to obtain extracted high-frequency feature information.
[0121] On the basis of the above technical solutions, the feature information fusion unit is specifically used to: input the image feature information and the high-frequency feature information into the feature fusion sub-model for weighted fusion of residual connections to obtain the fused target feature information.
[0122] The sharpening processing device provided in the embodiment of the present disclosure can execute the sharpening processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of executing the sharpening processing method.
[0123] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.
[0124] Figure 11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to Figure 11 below, it shows a schematic diagram of the structure of an electronic device (such as the terminal device or server in Figure 11) 500 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in Figure 11 is only an example and should not bring any limitations to the functions and scope of use of the embodiments of the present disclosure.
[0125] As shown in FIG11 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.
[0126] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG11 shows the electronic device 500 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0127] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0128] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0129] The electronic device provided by the embodiment of the present disclosure and the sharpening processing method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0130] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the sharpening processing method provided in the above embodiment is implemented.
[0131] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0132] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.
[0133] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0134] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device is enabled to: obtain a target object to be sharpened; input the target object into a target prediction model to predict the sharpening intensity, wherein the target prediction model is pre-trained based on sample objects and sharpening intensity labels corresponding to the sample objects, and the sharpening intensity labels are determined based on rate-distortion performance index values corresponding to multiple preset sharpening intensities; and sharpen the target object based on the target sharpening intensity output by the target prediction model.
[0135] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0137] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."
[0138] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0139] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0140] According to one or more embodiments of the present disclosure, a sharpening method is provided, comprising:
[0141] Obtain the target object to be sharpened;
[0142] Inputting the target object into a target prediction model to predict sharpening intensity, wherein the target prediction model is pre-trained based on sample objects and sharpening intensity labels corresponding to the sample objects, and the sharpening intensity labels are determined based on rate-distortion performance index values corresponding to a plurality of preset sharpening intensities;
[0143] The target object is sharpened based on the target sharpening strength output by the target prediction model.
[0144] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0145] Optionally, determining a sharpening intensity label corresponding to the sample object based on rate-distortion performance indicator values corresponding to a plurality of preset sharpening intensities includes:
[0146] determining a first rate-distortion curve of the sample object at a reference sharpening intensity and a second rate-distortion curve at each preset sharpening intensity;
[0147] Determining a rate-distortion performance index value of each preset sharpening intensity relative to the reference sharpening intensity based on the first rate-distortion curve and the second rate-distortion curve;
[0148] Based on the rate-distortion performance index value corresponding to each preset sharpening strength, a sharpening strength label corresponding to the sample object is determined.
[0149] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0150] Optionally, determining a first rate-distortion curve of the sample object at a reference sharpening intensity includes:
[0151] sharpening the sample object based on a reference sharpening strength, and performing multiple compression encoding on the sharpened sample object to obtain multiple encoded objects corresponding to the sample object;
[0152] Performing quality assessment on each coding object to obtain a quality assessment index value corresponding to each coding object;
[0153] Based on the bit information corresponding to each coding object and the quality assessment index value, a first rate-distortion curve of the sample object at a reference sharpening intensity is determined.
[0154] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0155] Optionally, when the sample object is a video object, the bit information corresponding to the coding object is the number of bits transmitted per unit time;
[0156] When the sample object is a sample image, the bit information corresponding to the coding object is the number of bits corresponding to a single pixel.
[0157] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0158] Optionally, the rate-distortion performance indicator value includes: a first performance indicator value or a second performance indicator value;
[0159] The first performance indicator value refers to a bit information change amount of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same quality;
[0160] The second performance indicator information refers to a quality assessment indicator change of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same bit information.
[0161] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0162] Optionally, determining the sharpening intensity label corresponding to the sample object based on the rate-distortion performance index value corresponding to each preset sharpening intensity includes:
[0163] If the rate-distortion performance index value is the first performance index value, determining the preset sharpening intensity corresponding to the minimum first performance index value as the sharpening intensity label corresponding to the sample object;
[0164] If the rate-distortion performance index value is the second performance index value, the preset sharpening intensity corresponding to the maximum first performance index value is determined as the sharpening intensity label corresponding to the sample object.
[0165] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0166] Optionally, the target prediction model includes: an image feature extraction sub-model, a high-frequency feature extraction sub-model, a feature fusion sub-model and a regression prediction sub-model;
[0167] Inputting the target object into a target prediction model to predict the sharpening intensity includes:
[0168] Inputting the target object into the image feature extraction sub-model to extract image features and obtain extracted image feature information;
[0169] Inputting the target object into the high-frequency feature extraction sub-model to extract high-frequency features and obtain extracted high-frequency feature information;
[0170] Inputting the image feature information and the high-frequency feature information into the feature fusion sub-model to perform feature fusion to obtain fused target feature information;
[0171] The target feature information is input into the regression prediction sub-model to perform regression prediction of the sharpening intensity to obtain the predicted target sharpening intensity.
[0172] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0173] Optionally, the high-frequency feature extraction sub-model includes: a discrete transformation layer and a feature extraction layer;
[0174] Inputting the target object into the high-frequency feature extraction sub-model to extract high-frequency features and obtain extracted high-frequency feature information includes:
[0175] Inputting the target object into the discrete transformation layer to perform discrete cosine transformation and inverse discrete cosine transformation to obtain high-frequency component information in the target object;
[0176] The high-frequency component information is input into the feature extraction layer to extract high-frequency features, thereby obtaining extracted high-frequency feature information.
[0177] According to one or more embodiments of the present disclosure, a sharpening processing method is provided, further comprising:
[0178] Optionally, inputting the image feature information and the high-frequency feature information into the feature fusion sub-model to perform feature fusion to obtain fused target feature information includes:
[0179] The image feature information and the high-frequency feature information are input into the feature fusion sub-model for weighted fusion of residual connections to obtain fused target feature information.
[0180] According to one or more embodiments of the present disclosure, there is provided a sharpening processing device, comprising:
[0181] A target object acquisition module, used to acquire the target object to be sharpened;
[0182] a sharpening intensity prediction module, configured to input the target object into a target prediction model to predict the sharpening intensity, wherein the target prediction model is pre-trained based on sample objects and sharpening intensity labels corresponding to the sample objects, and the sharpening intensity labels are determined based on rate-distortion performance index values corresponding to a plurality of preset sharpening intensities;
[0183] A sharpening processing module is used to perform sharpening processing on the target object based on the target sharpening intensity output by the target prediction model.
[0184] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.
[0185] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0186] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.
Claims
1. A sharpening method, include: Acquire the target object to be sharpened; Inputting the target object into a target prediction model to predict the sharpening strength, wherein the target prediction model is pre-trained based on sample objects and sharpening strength labels corresponding to the sample objects, and the sharpening strength labels are determined based on rate-distortion performance indicator values corresponding to a plurality of preset sharpening strengths; The target object is sharpened based on the target sharpening strength output by the target prediction model.
2. The sharpening method according to claim 1, in, Determining a sharpening intensity label corresponding to the sample object based on rate-distortion performance indicator values corresponding to a plurality of preset sharpening intensities includes: Determine a first rate-distortion curve of the sample object at a reference sharpening intensity, and a second rate-distortion curve at each preset sharpening intensity; Determining a rate-distortion performance index value of each preset sharpening intensity relative to the reference sharpening intensity based on the first rate-distortion curve and the second rate-distortion curve; Based on the rate-distortion performance indicator value corresponding to each preset sharpening intensity, a sharpening intensity label corresponding to the sample object is determined.
3. The sharpening method according to claim 2, in, Determining a first rate-distortion curve of the sample object at a reference sharpening intensity includes: Based on the reference sharpening strength, the sample object is sharpened, and the sharpened sample object is compressed and encoded multiple times to obtain multiple encoding objects corresponding to the sample object; Performing quality assessment on each encoding object to obtain a quality assessment index value corresponding to each encoding object; Based on the bit information corresponding to each coding object and the quality assessment index value, a first rate-distortion curve of the sample object at the reference sharpening intensity is determined.
4. The sharpening method according to claim 3, in, When the sample object is a video object, the bit information corresponding to the coding object is the number of bits transmitted per unit time; When the sample object is a sample image, the bit information corresponding to the coding object is the number of bits corresponding to a single pixel.
5. The sharpening method according to any one of claims 2 to 4, in, The rate-distortion performance indicator value includes: a first performance indicator value or a second performance indicator value; The first performance indicator value refers to a bit information change amount of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same quality; The second performance indicator information refers to a quality assessment indicator change of the preset sharpening intensity relative to the reference sharpening intensity when the preset sharpening intensity and the reference sharpening intensity correspond to the same bit information.
6. The sharpening method according to claim 5, in, The determining, based on the rate-distortion performance indicator value corresponding to each preset sharpening intensity, the sharpening intensity label corresponding to the sample object comprises: If the rate-distortion performance indicator value is the first performance indicator value, determining the preset sharpening intensity corresponding to the minimum first performance indicator value as the sharpening intensity label corresponding to the sample object; If the rate-distortion performance indicator value is the second performance indicator value, the preset sharpening intensity corresponding to the maximum first performance indicator value is determined as the sharpening intensity label corresponding to the sample object.
7. The sharpening method according to any one of claims 1 to 6, in, The target prediction model includes: an image feature extraction sub-model, a high-frequency feature extraction sub-model, a feature fusion sub-model and a regression prediction sub-model; Inputting the target object into a target prediction model to predict the sharpening intensity includes: Inputting the target object into the image feature extraction sub-model to extract image features and obtain extracted image feature information; Inputting the target object into the high-frequency feature extraction sub-model to extract high-frequency features, and obtaining extracted high-frequency feature information; Inputting the image feature information and the high-frequency feature information into the feature fusion sub-model to perform feature fusion to obtain fused target feature information; The target feature information is input into the regression prediction sub-model to perform regression prediction of sharpening intensity to obtain the predicted target sharpening intensity.
8. The sharpening method according to claim 7, in, The high-frequency feature extraction sub-model includes: a discrete transformation layer and a feature extraction layer; The target object is input into the high-frequency feature extraction sub-model to extract high-frequency features. Obtain the extracted high-frequency feature information, including: Inputting the target object into the discrete transformation layer to perform discrete cosine transformation and inverse discrete cosine transformation to obtain high-frequency component information in the target object; The high-frequency component information is input into the feature extraction layer to extract high-frequency features, and the extracted high-frequency feature information is obtained.
9. The sharpening method according to claim 7, in, The step of inputting the image feature information and the high-frequency feature information into the feature fusion sub-model for feature fusion to obtain fused target feature information includes: The image feature information and the high-frequency feature information are input into the feature fusion sub-model for weighted fusion of residual connections to obtain fused target feature information.
10. A sharpening processing device, include: A target object acquisition module, configured to acquire a target object to be sharpened; a sharpening strength prediction module, configured to input the target object into a target prediction model to predict the sharpening strength, wherein the target prediction model is pre-trained based on sample objects and sharpening strength labels corresponding to the sample objects, and the sharpening strength labels are determined based on rate-distortion performance index values corresponding to a plurality of preset sharpening strengths; The sharpening processing module is configured to perform sharpening processing on the target object based on the target sharpening intensity output by the target prediction model.
11. An electronic device, include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the sharpening processing method as described in any one of claims 1-8.
12. A storage medium containing computer executable instructions, in, When the computer executable instructions are executed by a computer processor, they are used to perform the sharpening processing method as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Image sharpening processing method and system thereof
CN110136073A
Optimization method and device for image shot in weak light environment
CN114004974A
Target detection method and device and readable storage medium
CN115100500A
Video sharpening method, and training method and device of sharpening intensity prediction model
CN116033093A
Sharpening processing method and device, equipment and storage medium
CN116630190A
