A method and apparatus for video conference document sharing clarification processing

Abstract

Embodiments of the present application provide a method for video conference shared document sharpening processing. The method comprises: extracting a luminance component based on YUV format video image data; performing denoising based on the luminance component to obtain a denoised image; performing edge information extraction on the denoised image; obtaining a sharpening intensity based on the edge information; performing foreground and background segmentation on the denoised image; and performing sharpening on the denoised image based on the sharpening intensity and foreground information. Through the sharpening processing of the shared document in the video conference, the user obtains better visual effects, and the user has a better experience and better meets the user's needs. In addition, embodiments of the present application provide a device for video conference shared document sharpening processing.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video communication technology, and more specifically, the embodiments of the present invention relate to a method and apparatus for clarifying shared documents in video conferencing. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Currently, internet-based video communication technology is widely used. Screen sharing is one of the most frequently used functions in video conferencing, and it is an indispensable feature in daily meetings and remote training. However, due to lossy compression during data transmission, text in the shared document at the receiving end often exhibits blurry edges and ghosting, resulting in indistinct fonts. To improve the user's visual experience, the shared image needs to be sharpened and enhanced, making the shared image appear more defined and the fonts clearer.

[0004] Due to lossy compression during video transmission, the received shared documents or screens may contain artifacts. Directly sharpening these artifacts will amplify them, resulting in a degraded visual quality. Furthermore, since shared documents primarily focus on text, which is characterized by sharp edge changes, traditional sharpening methods can cause significant overshoot, resulting in noticeable white borders around the text in certain areas and severely degrading image quality.

[0005] This invention designs a method and apparatus for real-time enhancement of shared documents in video conferencing. The method involves smoothing and denoising the received document while preserving text edges; segmenting the foreground and background based on the denoised image, where the foreground is the part to be enhanced (e.g., text, image edges); obtaining high-frequency components from the denoised image, dividing the detail region and text edge region according to a set threshold, and obtaining the sharpening intensity pixel-by-pixel based on the foreground image; performing edge sharpening on the denoised image; and combining the inter-frame invariance of the shared document's same page to achieve real-time enhancement of the shared screen and document in video conferencing, providing users with a better visual experience. Summary of the Invention

[0006] The purpose of this invention is to provide a feasible real-time document sharpening solution for shared documents in video conferencing. Document sharpening is achieved through edge-preserving denoising, foreground / background segmentation, sharpening and enhancing high-frequency foreground components, and maintaining frame-invariant relationships within the same page of a document during video conferencing. Current technologies lack dedicated real-time sharpening enhancement techniques for shared screens and documents, and applying existing sharpening techniques directly to shared screens and documents results in noticeable overshoot, producing prominent white borders around text in certain areas. Therefore, an improved real-time document sharpening technology is urgently needed to address these problems.

[0007] In this context, embodiments of the present invention aim to provide a document clarification processing method and apparatus.

[0008] In a first aspect of the present invention, a method for enhancing the clarity of shared video conferencing documents is provided, comprising: extracting a luminance component based on the YUV format video image data; denoising the luminance component to obtain a denoised image; extracting edge information from the denoised image; obtaining a sharpening intensity based on the edge information; segmenting the foreground and background of the denoised image; and sharpening the denoised image based on the sharpening intensity and the foreground information.

[0009] In one embodiment of the present invention, the step of extracting the luminance component based on the text region includes: extracting the luminance component of the region to be processed based on each frame of image data.

[0010] In another embodiment of the present invention, the step of denoising based on the luminance component pair to obtain a denoised image includes: performing a smoothing filter on the luminance component using a bilateral filter to obtain the denoised image; the bilateral filter includes a spatial distance parameter and a grayscale distance parameter.

[0011] In another embodiment of the present invention, the step of obtaining edge information from the denoised image includes: obtaining a low-frequency image based on the denoised image; the difference between the denoised image and the low-frequency image is the edge information.

[0012] In another embodiment of the present invention, obtaining a low-frequency image based on the denoised image includes: performing Gaussian blur processing on the denoised image to obtain the low-frequency image.

[0013] In another embodiment of the present invention, obtaining the sharpening intensity based on the acquired edge information includes: dividing the edge into a white edge and / or a black edge based on the positive and negative information of the edge; setting different enhancement coefficients for the white edge and / or the black edge; and obtaining the sharpening intensity based on the enhancement coefficients and the edge intensity.

[0014] In another embodiment of the present invention, obtaining the sharpening intensity based on the enhancement coefficient and the edge intensity includes: when the denoised image is divided into white edges, when the edge intensity is greater than zero and less than a first threshold, the sharpening intensity is zero; when the edge intensity is greater than or equal to the first threshold and less than a second threshold, the enhancement coefficient is a first enhancement coefficient, and the sharpening intensity is the product of the first enhancement coefficient and the edge intensity; when the edge intensity is greater than or equal to the second threshold and less than a third threshold, the enhancement coefficient is a second enhancement coefficient, and the sharpening intensity is the product of the second enhancement coefficient and the edge intensity; when the edge intensity is greater than or equal to the third threshold and less than a fourth threshold, the enhancement coefficient is a third enhancement coefficient, and the sharpening intensity is the product of the third enhancement coefficient and the edge intensity; when the edge intensity is greater than or equal to the fourth threshold, the sharpening intensity is the fourth threshold.

[0015] In another embodiment of the present invention, obtaining the sharpening intensity based on the enhancement coefficient and the edge intensity includes: when the denoised image is divided into black edges, when the edge intensity is greater than a fifth threshold and less than zero, the sharpening intensity is zero; when the edge intensity is greater than a sixth threshold and less than or equal to the fifth threshold, the enhancement coefficient is a fourth enhancement coefficient, and the sharpening intensity is the product of the fourth enhancement coefficient and the edge intensity; when the edge intensity is greater than a seventh threshold and less than or equal to the sixth threshold, the enhancement coefficient is a fifth enhancement coefficient, and the sharpening intensity is the product of the fifth enhancement coefficient and the edge intensity; when the edge intensity is greater than an eighth threshold and less than or equal to the seventh threshold, the enhancement coefficient is a sixth enhancement coefficient, and the sharpening intensity is the product of the sixth enhancement coefficient and the edge intensity; when the edge intensity is less than or equal to the eighth threshold, the sharpening intensity is the eighth threshold.

[0016] In another embodiment of the present invention, the segmentation of foreground and background in the denoised image includes: performing grayscale value statistics on pixels with sharpening intensity greater than a ninth threshold within a given window; using the median of the statistics as the background value of the region; if the absolute value of the difference between the current pixel value and the background value is greater than a preset value, then the pixel is a foreground point; otherwise, it is a background point; pixels with sharpening intensity not greater than the ninth threshold are defaulted to being foreground points that need to be enhanced.

[0017] In another embodiment of the present invention, the step of sharpening the denoised image based on the sharpening intensity and foreground information includes: setting the sharpening intensity corresponding to each pixel in the background to zero; and adding each pixel of the denoised image to its corresponding sharpening intensity value to achieve sharpening of the denoised image.

[0018] In another embodiment of the present invention, the method further includes: determining whether the current frame image of the video is the first frame image; performing the above-described sharpening process on the first frame image; dividing the non-first frame image into N-square regions; determining whether the content of each region has changed compared to the corresponding region of the previous frame; if the region has changed, performing the above-described sharpening process on the region; if the region has not changed, directly using the sharpening result of the previous frame to sharpen the region.

[0019] In a second aspect of the present invention, an apparatus for de-duplicating video conferencing shared documents is provided. The apparatus includes: an extraction module for extracting a luminance component based on YUV format video image data; a denoising module for denoising the luminance component to obtain a denoised image; an edge information acquisition module for acquiring edge information from the denoised image; a sharpening intensity acquisition module for acquiring a sharpening intensity based on the edge information; a segmentation module for segmenting the denoised image into foreground and background; and a sharpening module for sharpening the denoised image based on the sharpening intensity and foreground information.

[0020] In one embodiment of this application, the extraction module includes a unit for extracting the luminance component of the region to be processed based on each frame of image data.

[0021] In another embodiment of this application, the denoising module includes: a module for smoothing the luminance component using bilateral filtering to obtain the denoised image; the bilateral filtering includes a spatial distance parameter and a grayscale distance parameter.

[0022] In another embodiment of this application, the edge information acquisition of the denoised image includes: acquiring a low-frequency image based on the denoised image; the difference between the denoised image and the low-frequency image is the edge information.

[0023] In another embodiment of this application, the module for obtaining a low-frequency image based on the denoised image includes: a unit for performing Gaussian blur processing on the denoised image to obtain the low-frequency image.

[0024] In another embodiment of this application, the sharpening intensity acquisition module includes: a unit for dividing the edge into a white edge and / or a black edge based on the positive and negative information of the edge; a unit for setting different enhancement coefficients for the white edge and / or the black edge; and a unit for acquiring the sharpening intensity based on the enhancement coefficients and the edge intensity.

[0025] In another embodiment of this application, the module for obtaining the sharpening intensity based on the enhancement coefficient and the edge intensity includes: a unit for setting the sharpening intensity to zero when the edge intensity is greater than zero and less than a first threshold when the denoised image is divided into white edges; a unit for setting the sharpening intensity to the product of the first enhancement coefficient and the edge intensity when the edge intensity is greater than or equal to the first threshold and less than a second threshold; a unit for setting the sharpening intensity to the product of the second enhancement coefficient and the edge intensity when the edge intensity is greater than or equal to the second threshold and less than a third threshold; a unit for setting the sharpening intensity to the product of the third enhancement coefficient and the edge intensity when the edge intensity is greater than or equal to the third threshold and less than a fourth threshold; and a unit for setting the sharpening intensity to the fourth threshold when the edge intensity is greater than or equal to the fourth threshold.

[0026] In another embodiment of this application, the module for obtaining the sharpening intensity based on the enhancement coefficient and the edge intensity includes: a unit for setting the sharpening intensity to zero when the edge intensity is greater than a fifth threshold and less than zero, and a unit for setting the sharpening intensity to the product of the fourth enhancement coefficient and the edge intensity when the edge intensity is greater than a sixth threshold and less than or equal to the fifth threshold; a unit for setting the sharpening intensity to the product of the fifth enhancement coefficient and the edge intensity when the edge intensity is greater than a seventh threshold and less than or equal to the sixth threshold; a unit for setting the sharpening intensity to the product of the sixth enhancement coefficient and the edge intensity when the edge intensity is greater than an eighth threshold and less than or equal to the seventh threshold; and a unit for setting the sharpening intensity to the eighth threshold when the edge intensity is less than or equal to the eighth threshold.

[0027] In another embodiment of this application, the segmentation module includes: a unit for performing grayscale value statistics on pixels with sharpening intensity greater than a ninth threshold within a given window; a unit for using the statistical median as the background value of the region; a unit for determining if the pixel is a foreground point if the absolute value of the difference between the current pixel value and the background value is greater than a preset value, and otherwise as a background point; and a unit for defaulting to designating pixels with sharpening intensity not greater than the ninth threshold as foreground points that need enhancement.

[0028] In another embodiment of this application, the sharpening module includes a unit for sharpening and enhancing the foreground of the denoised image based on the segmentation results of the background and foreground and the sharpening intensity.

[0029] In another embodiment of this application, the apparatus further includes: a module for determining whether the current frame image of the video is the first frame image; a module for performing the sharpening process on the first frame image; a module for dividing non-first frame images into N-square regions; a module for determining whether the content of each region has changed compared to the corresponding region of the previous frame; a module for performing the sharpening process on the region if the region has changed; and a module for directly sharpening the region using the sharpening result of the previous frame if the region has not changed.

[0030] According to an embodiment of the present invention, a document sharpening processing method and apparatus are used to smooth and denoise a received document, while preserving the edges of the text during denoising; high-frequency components are obtained based on the denoised image, and detail regions and text edge regions are divided according to a set threshold, and sharpening intensity is obtained pixel by pixel in combination with the foreground image; foreground and background are segmented based on the denoised image, wherein the foreground is the part that needs to be enhanced (such as text); edge sharpening is performed on the denoised image by combining the sharpening intensity and foreground information; and real-time sharpening of shared screens and documents is achieved by combining the inter-frame invariance of shared documents on the same page in video conferencing, so as to provide users with better visual effects and a better user experience. Attached Figure Description

[0031] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0032] Figure 1 A flowchart illustrating a method for implementing document clarity processing according to an embodiment of the present invention is shown schematically.

[0033] Figure 2 A flowchart illustrating a method for clarifying shared documents in a video conference according to another embodiment of the present invention is shown schematically.

[0034] Figure 3 A flowchart illustrating a method for implementing document clarification processing according to yet another embodiment of the present invention is shown.

[0035] Figure 4 A schematic diagram illustrating text edge features and their sharpening according to another embodiment of the present invention is shown.

[0036] Figure 5 A schematic diagram of an apparatus for implementing document clarity processing according to an embodiment of the present invention is shown.

[0037] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0038] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0039] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0040] According to an embodiment of the present invention, a method and apparatus for clarifying shared documents in video conferencing are proposed.

[0041] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview

[0043] The inventors have discovered that existing video image sharpening technologies have the following drawbacks: there is no real-time sharpening enhancement technology specifically designed for shared screens and documents in video conferencing; and directly applying existing sharpening enhancement technologies to shared screens and documents can lead to obvious overshoot, resulting in noticeable white borders around text in certain areas, which does not better meet user needs.

[0044] To overcome the problems existing in the prior art, the present invention provides a document sharpening processing method and apparatus. The method includes the following steps: extracting luminance components based on YUV format video image data; denoising based on the luminance components to obtain a denoised image; extracting edge information from the denoised image; obtaining a sharpening intensity based on the acquired edge information; segmenting the foreground and background based on the denoised image; and sharpening the denoised image based on the sharpening intensity.

[0045] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0046] Application Scenarios Overview

[0047] The embodiments of the present invention can be applied to sharing documents on the screen during video conferencing; however, those skilled in the art will fully understand that the applicable scenarios of the embodiments of the present invention are not limited by any aspect of this framework.

[0048] Exemplary methods

[0049] The following section provides examples and references. Figure 1-5 This document describes a method for clarifying shared documents in video conferencing according to an exemplary embodiment of the present invention. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the invention, and the embodiments of the invention are not limited in any way. Rather, the embodiments of the invention can be applied to any applicable scenario.

[0050] refer to Figure 1 and Figure 3 The diagram schematically illustrates a method flowchart for implementing document clarification processing according to an embodiment of the present invention. The method includes the following steps:

[0051] S100 Extracts luminance components from YUV format video image data.

[0052] As an example, the luminance component of the region to be processed is extracted based on each frame of image data. Extracting the luminance component avoids the influence of color components and other elements present in the document on subsequent document sharpening.

[0053] S110. Denoise is performed based on the luminance component to obtain a denoised image.

[0054] As an example, a bilateral filtering method is used to smooth the luminance component to obtain the denoised image, wherein the bilateral filtering includes spatial distance parameters and grayscale distance parameters. Specifically, when sharing documents via screen sharing, the document sharing uses video encoding technology, which transmits lossily compressed data to the remote end, where it is then decoded and displayed. During this process, the lossy compression of the video transmission results in artifacts in the received shared screen and document. During sharpening enhancement, these artifacts are amplified, leading to a degraded visual experience; therefore, smoothing and denoising are necessary first. Furthermore, sharpening the edges is required for screen and document clarity; therefore, bilateral filtering is chosen in the denoising process to preserve the image edges.

[0055] Parameter settings for bilateral filtering spatial distance and grayscale distance: Considering that smoothing filtering can cause image blurring, the spatial domain filter diameter of bilateral filtering should not be too large, generally 3-5. Considering the high contrast between text pixels and background pixels, the grayscale distance in the color domain can be a larger value, generally 50-80.

[0056] S120. Extract edge information from the denoised image.

[0057] As an example, a low-frequency image is obtained based on the denoised image, and the difference between the denoised image and the low-frequency image is the edge information, wherein the low-frequency image is obtained by performing Gaussian blur processing on the denoised image. Specifically, for example, a radius of r (r takes values ​​from 1 to 3) is used to perform a Gaussian blur processing on the denoised image I. F Gaussian blurring is performed to obtain the low-frequency image I. L The two-dimensional Gaussian distribution function is:

[0058]

[0059] In the formula, x and y represent the distances of the current pixel from the center pixel in the X and Y directions, respectively, and r = x 2 +y 2 σ represents the standard deviation of the normal distribution.

[0060] Denoising Image I F (i,j) and low-frequency image I L The difference between (i,j) is the edge strength I. H (i,j), that is

[0061] I H (i,j)=I F (i,j)-I L (i,j),

[0062] In the formula, h(i,j) represents the Gaussian filter kernel.

[0063] Obtaining high-frequency components from a denoised image rather than the original image can effectively remove the influence of noise and avoid amplifying the noise, especially in flat textured areas that are very sensitive to noise.

[0064] S130. Obtain the sharpening intensity based on the edge information.

[0065] As an example, since the human eye has different sensitivities to brightness changes in low-brightness and high-brightness areas, the denoised image is divided into white edges and / or black edges based on the positive and negative information of the edge intensity. An edge is considered white when its intensity is greater than zero, and black when its intensity is less than zero. Different enhancement coefficients are set for the white edges and / or the black edges. The sharpening intensity is obtained based on the enhancement coefficients and the edge intensity, and the denoised image is sharpened based on the sharpening intensity.

[0066] Specifically, when the denoised image is divided into white edges, when the edge intensity is greater than zero and less than the first threshold, the sharpening intensity is zero at this time; when the edge intensity is greater than or equal to the first threshold and less than the second threshold, the enhancement coefficient is the first enhancement coefficient, and the sharpening intensity is the product of the first enhancement coefficient and the edge intensity at this time; when the edge intensity is greater than or equal to the second threshold and less than the third threshold, the enhancement coefficient is the second enhancement coefficient, and the sharpening intensity is the product of the second enhancement coefficient and the edge intensity at this time; when the edge intensity is greater than or equal to the third threshold and less than the fourth threshold, the enhancement coefficient is the third enhancement coefficient, and the sharpening intensity is the product of the third enhancement coefficient and the edge intensity at this time; when the edge intensity is greater than or equal to the fourth threshold, the sharpening intensity is the fourth threshold at this time.

[0067] More specifically, according to the detail level of the image, set thresholds to classify black edges and white edges respectively and perform different amplitude enhancements:

[0068] White edges:

[0069] 1) 0 < I H (i, j) < WT0: The detailed texture of noise / text, no enhancement required, I H ′(i,j) = 0

[0070] 2) WT0 ≤ I H (i, j) < WT1: Larger text detail area, set a smaller enhancement coefficient w k1 , I H ′(i,j) = w k1 *I H (i,j)

[0071] 3) WT1 ≤ I H (i, j) < WT2: Relatively blurred text edge, set a larger enhancement coefficient w k2 , I H ′(i,j) = w k2 *I H (i,j)

[0072] 4) WT2 ≤ I H (i, j) < WT3: Relatively clear text edge, set a smaller enhancement coefficient w k3 , I H ′(i,j) = w k3 *I H (i,j)

[0073] 5) I H (i, j) ≥ WT3: Clear text edge area, need to suppress the enhancement amplitude, I H ′(i,j) = wT3

[0074] Where wT0 is the first threshold, with an optimal value range of 2-4; wT1 is the second threshold, with a value range of 4-7; wT2 is the third threshold, with a value range of 10-15; and wT3 is the fourth threshold, with a value range of 30-40. k1 The first coefficient, with a value ranging from 0.5 to 1, w k2 The second coefficient, with a value range of 1.5-2, w k3 This is the third coefficient, with a value range of 0.7-1. H ′(i,j) represents the sharpening intensity.

[0075] Black edge:

[0076] 1)bT0 H (i, j) < 0: Noise / text detail texture, no enhancement needed, I H ′(i,j)=0

[0077] 2)bT1 H (i, j) ≤ bT0: Larger text detail areas, set a smaller enhancement coefficient b k1 ,I H ′(i,j)=b k1 *I H (i,j)

[0078] 3)bT2 H (i, j) ≤ bT1: For relatively blurry text edges, set a larger enhancement coefficient b. k2 ,I H ′(i,j)=b k2 *I H (i,j)

[0079] 4) bT3 H (i, j) ≤ bT2: Relatively clear text edges, with a smaller enhancement coefficient b. k3 I H ′(i,j)=b k3 *I H (i,j)

[0080] 5)I H (i, j) ≤ bT3: Clear text edge regions require suppression of enhancement amplitude. H ′(i,j)=bT3

[0081] Where bT0 is the fifth threshold, with an optimal value range of -2 to 4; bT1 is the sixth threshold, with a value range of -4 to 10; bT2 is the seventh threshold, with a value range of -18 to 25; bT3 is the eighth threshold, with a value range of -40 to 50; and the fourth enhancement coefficient b... k1 The value range is 0.5-1, the fifth b​​​​k2 The value range is 1.5-2, the sixth b k3 The value range is 0.7-1. Note the value of w when choosing this value. k k .

[0082] The sharpening intensity value of each pixel can be obtained by performing the above operations.

[0083] S140. Segment the foreground and background based on the denoised image.

[0084] When sharing screens and documents, users focus on the text areas in images, and for text sharpening illustrations, such as... Figure 4 As shown, Figure 4 Taking black text as an example, white text is the opposite of black text. For example... Figure 4 Directly sharpening the background using the Unsharp Mask (USM) method will over-sharpen the background edges, resulting in noticeable white edges. Figure 4 As shown in c, the sharpening process described in this application only enhances the text portion (i.e., the foreground portion) (i.e., bolds the text), while avoiding sharpening the background edges. To achieve this effect, the background and foreground of the text image need to be segmented. A key characteristic of text images is that the grayscale value fluctuations in the background and foreground regions are very small. Through analysis of the grayscale statistical features of a large number of text images, text images exhibit definite grayscale differentiation characteristics within a certain region: the grayscale value distribution of the background and the grayscale value distribution of the foreground (text pixels) are located in two different grayscale value regions. The value with the absolute dominance in the grayscale histogram can be identified as the background value; that is, the more pixels in a grayscale range in the histogram, the more pixels that are considered background pixels.

[0085] Specifically, for pixels with a sharpening intensity greater than the ninth threshold, grayscale values ​​are statistically analyzed within a given window; the median of the statistical values ​​is used as the background value for that region; if the absolute value of the difference between the current pixel value and the background value is greater than a preset value, then the pixel is the foreground point; pixels with a sharpening intensity not greater than the ninth threshold are defaulted to being the foreground point.

[0086] ​More specifically, based on the acquired sharpening intensity, pixels with a sharpening intensity greater than the set threshold DiffT are statistically analyzed within a defined window range (window diameter 11-21). The median of the statistical analysis is used as the background value, thus achieving foreground and background segmentation for that pixel: if the absolute value of the difference between the current pixel value and the background value is greater than a preset value T (generally 10-15), the pixel is considered a foreground pixel; otherwise, it is considered a background pixel. Pixels with a sharpening intensity not greater than the set threshold DiffT are considered foreground pixels that need enhancement by default. By segmenting the foreground and background, the text portion (i.e., the portion that needs sharpening enhancement) can be determined. By further sharpening the already sharpened text portion, the text in the document can be made clearer.

[0087] S150. Sharpen the denoised image based on the sharpening intensity and foreground information.

[0088] The denoised image is sharpened based on the sharpening intensity value of each pixel calculated above. A higher sharpening intensity value indicates a higher degree of sharpening, while a lower sharpening intensity value indicates a lower degree of sharpening.

[0089] The sharpening intensity value I of the background region pixels. H Set (i,j) to 0, and then use the sharpening intensity value I obtained from the above-mentioned denoised image pixels and the corresponding pixels. H Adding (i,j) together will produce a clearer document.

[0090] like Figure 2 As shown in another embodiment, in a video conference, the shared document maintains the invariance of image content between frames for a certain period of time. However, when the document is turned, the image content changes abruptly. To simplify the operation process and improve sharpening efficiency, when the current frame image is received, it is determined whether the current frame image is the first frame image. If it is confirmed that the current frame image is the first frame image, the sharpening processing flow described in S100-S150 is directly executed on the image. If it is determined that the current frame image is not the first frame image, the current frame image is divided into n*n regions. The image content of each region is compared with the corresponding region of the previous frame. If the content of the region has changed, the sharpening processing flow of S100-S150 is executed on the region to achieve sharpening enhancement of the region; otherwise, the sharpening result of the region is the same as the sharpening result of the previous frame, and the sharpening result of the corresponding region of the previous frame can be used directly. This avoids repeatedly sharpening the same frame image multiple times, thus improving sharpening efficiency.

[0091] This application achieves real-time sharpening of the received shared screen and document by smoothing and denoising the image while preserving the edges of the text; it obtains high-frequency components from the denoised image, divides the detail region and text edge region according to a set threshold, and obtains sharpening intensity pixel by pixel by combining the foreground image; it segments the foreground and background based on the denoised image, where the foreground is the part that needs to be enhanced (such as text); it sharpens the edges of the denoised image; and it combines the inter-frame invariance of the shared document to achieve real-time sharpening of the shared screen and document, providing users with a better visual experience.

[0092] Exemplary device

[0093] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 5 A schematic diagram of an apparatus for implementing document declarative processing according to an exemplary embodiment of the present invention is provided. The apparatus includes the following modules:

[0094] Extraction module 500 is used to extract the luminance component based on YUV format video image data.

[0095] As an example, the luminance component of the region to be processed is extracted based on each frame of image data. Extracting the luminance component avoids the influence of color components and other elements present in the document on subsequent document sharpening.

[0096] The denoising module 510 is used to denoise based on the luminance component to obtain a denoised image.

[0097] As an example, a bilateral filtering method is used to smooth the luminance component to obtain the denoised image, wherein the bilateral filtering includes spatial distance parameters and grayscale distance parameters. Specifically, when sharing documents via screen sharing, the document sharing uses video encoding technology, which transmits lossily compressed data to the remote end, where it is then decoded and displayed. During this process, the lossy compression of the video transmission results in artifacts in the received shared screen and document. During sharpening enhancement, these artifacts are amplified, leading to a degraded visual experience; therefore, smoothing and denoising are necessary first. Furthermore, sharpening the edges is required for screen and document clarity; therefore, bilateral filtering is chosen in the denoising process to preserve the image edges.

[0098] Parameter settings for bilateral filtering spatial distance and grayscale distance: Considering that smoothing filtering can cause image blurring, the spatial domain filter diameter of bilateral filtering should not be too large, generally 3-5. Considering the high contrast between text pixels and background pixels, the grayscale distance in the color domain can be a larger value, generally 50-80.

[0099] The edge information acquisition module 530 is used to extract edge information from the denoised image.

[0100] As an example, a low-frequency image is obtained based on the denoised image, and the difference between the denoised image and the low-frequency image is the edge intensity, wherein the low-frequency image is obtained by performing Gaussian blur processing on the denoised image. Specifically, for example, a radius of r (r takes values ​​from 1 to 3) is used to perform a Gaussian blur processing on the denoised image I. F Gaussian blurring is performed to obtain the low-frequency image I. L The two-dimensional Gaussian distribution function is:

[0101]

[0102] In the formula, x and y represent the distances of the current pixel from the center pixel in the X and Y directions, respectively, and r = x 2 +y 2 σ represents the standard deviation of the normal distribution.

[0103] Denoising Image I F (i,j) and low-frequency image I L The difference between (i,j) is the edge strength I. H (i,j), that is

[0104] I H (i,j)=I F (i,j)-I L (i,j),

[0105] In the formula, h(i,j) represents the Gaussian filter kernel.

[0106] Obtaining high-frequency components from a denoised image rather than the original image can effectively remove the influence of noise and avoid amplifying the noise, especially in flat textured areas that are very sensitive to noise.

[0107] The sharpening intensity acquisition module 540 is used to acquire sharpening intensity based on the acquired edge intensity.

[0108] As an example, since the human eye has different sensitivities to brightness changes in low-brightness and high-brightness areas, the denoised image is divided into white edges and / or black edges based on the positive and negative information of the edge intensity. An edge is considered white when its intensity is greater than zero, and black when its intensity is less than zero. Different enhancement coefficients are applied to the white edges and / or the black edges, and the sharpening intensity is obtained based on these enhancement coefficients and the edge intensity.

[0109] Specifically, when the denoised image is divided into white edges, when the edge intensity is greater than zero and less than the first threshold, the sharpening intensity is zero at this time; when the edge intensity is greater than or equal to the first threshold and less than the second threshold, the enhancement coefficient is the first enhancement coefficient, and the sharpening intensity is the product of the first enhancement coefficient and the edge intensity at this time; when the edge intensity is greater than or equal to the second threshold and less than the third threshold, the enhancement coefficient is the second enhancement coefficient, and the sharpening intensity is the product of the second enhancement coefficient and the edge intensity at this time; when the edge intensity is greater than or equal to the third threshold and less than the fourth threshold, the enhancement coefficient is the third enhancement coefficient, and the sharpening intensity is the product of the third enhancement coefficient and the edge intensity at this time; when the edge intensity is greater than or equal to the fourth threshold, the sharpening intensity is the fourth threshold at this time.

[0110] More specifically, according to the detail level of the image, thresholds are set to classify black edges and white edges respectively and perform different amplitude enhancements:

[0111] White edges:

[0112] 1) 0 < I H (i, j) < WT0: The detailed texture of noise / text, no enhancement is required, I H ′(i, j) = 0

[0113] 2) WT0 ≤ I H (i, j) < WT1: Larger text detail area, set a smaller enhancement coefficient w k1 , I H ′(i, j) = w k1 *I H (i, j)

[0114] 3) WT1 ≤ I H (i, j) < WT2: Relatively blurred text edges, set a larger enhancement coefficient w k2 , I H ′(i, j) = w k2 *I H (i, j)

[0115] 4) WT2 ≤ I H (i, j) < WT3: Relatively clear text edges, set a smaller enhancement coefficient w k3 , I H ′(i, j) = w k3 *I H (i, j)

[0116] 5) I H (i, j) ≥ WT3: Clear text edge area, need to suppress the enhancement amplitude, I H ′(i, j) = wT3

[0117] Where wT0 is the first threshold, with an optimal value range of 2-4; wT1 is the second threshold, with a value range of 4-7; wT2 is the third threshold, with a value range of 10-15; and wT3 is the fourth threshold, with a value range of 30-40. k1 The first coefficient, with a value ranging from 0.5 to 1, w k2 The second coefficient, with a value range of 1.5-2, w k3 This is the third coefficient, with a value range of 0.7-1. H ′(i,j) represents the sharpening intensity.

[0118] Black edge:

[0119] 1)bT0 H (i, j) < 0: Noise / text detail texture, no enhancement needed, I H ′(i,j)=0

[0120] 2)bT1 H (i, j) ≤ bT0: Larger text detail areas, set a smaller enhancement coefficient b k1 ,I H ′(i,j)=b k1 *I H (i,j)

[0121] 3)bT2 H (i, j) ≤ bT1: For relatively blurry text edges, set a larger enhancement coefficient b. k2 ,I H ′(i,j)=b k2 *I H (i,j)

[0122] 4) bT3 H (i, j) ≤ bT2: Relatively clear text edges, with a smaller enhancement coefficient b. k3 I H ′(i,j)=b k3 *I H (i,j)

[0123] 5)I H (i, j) ≤ bT3: Clear text edge regions require suppression of enhancement amplitude. H ′(i,j)=bT3

[0124] Where bT0 is the fifth threshold, with an optimal value range of -2 to 4; bT1 is the sixth threshold, with a value range of -4 to 10; bT2 is the seventh threshold, with a value range of -18 to 25; bT3 is the eighth threshold, with a value range of -40 to 50; and the fourth enhancement coefficient b... k1 The value range is 0.5-1, the fifth b​​​​k2 The value range is 1.5-2, the sixth b k3 The value range is 0.7-1. Note the value of w when choosing this value. k k .

[0125] The sharpening intensity value of each pixel can be obtained by performing the above operations.

[0126] The segmentation module 540 segments the foreground (text edges to be enhanced) and background of the denoised image.

[0127] When sharing screens and documents, users focus on the text areas in images, and for text sharpening illustrations, such as... Figure 4 As shown, Figure 4 Taking black text as an example, white text is the opposite of black text. For example... Figure 4 Directly sharpening the background using the Unsharp Mask (USM) method will over-sharpen the background edges, resulting in noticeable white edges. Figure 4 As shown in c, the sharpening process described in this application only enhances the text portion (i.e., the foreground portion) (i.e., bolds the text), while avoiding sharpening the background edges. To achieve this effect, the background and foreground of the text image need to be segmented. A key characteristic of text images is that the grayscale value fluctuations in the background and foreground regions are very small. Through analysis of the grayscale statistical features of a large number of text images, text images exhibit definite grayscale differentiation characteristics within a certain region: the grayscale value distribution of the background and the grayscale value distribution of the foreground (text pixels) are located in two different grayscale value regions. The value with the absolute dominance in the grayscale histogram can be identified as the background value; that is, the more pixels in a grayscale range in the histogram, the more pixels that are considered background pixels.

[0128] Specifically, for pixels with a sharpening intensity greater than the ninth threshold, grayscale values ​​are statistically analyzed within a given window; the median of the statistical values ​​is used as the background value for that region; if the absolute value of the difference between the current pixel value and the background value is greater than a preset value, then the pixel is the foreground point; pixels with a sharpening intensity not greater than the ninth threshold are defaulted to being the foreground point.

[0129] ​More specifically, based on the acquired sharpening intensity, pixels with a sharpening intensity greater than the set threshold DiffT are statistically analyzed within a defined window range (window diameter 11-21). The median of the statistical analysis is used as the background value, thus achieving foreground and background segmentation for that pixel: if the absolute value of the difference between the current pixel value and the background value is greater than a preset value T (generally 10-15), the pixel is considered a foreground pixel; otherwise, it is considered a background pixel. Pixels with a sharpening intensity not greater than the set threshold DiffT are considered foreground pixels that need enhancement by default. By segmenting the foreground and background, the text portion (i.e., the portion that needs sharpening enhancement) can be determined. Further sharpening and enhancing the already sharpened text portion can make the text in the document clearer.

[0130] The sharpening module 550 is used to sharpen the denoised image based on the sharpening intensity and the foreground information.

[0131] The denoised image is sharpened based on the sharpening intensity value of each pixel calculated above. A higher sharpening intensity value indicates a higher degree of sharpening, while a lower sharpening intensity value indicates a lower degree of sharpening.

[0132] The sharpening intensity value I of the background region pixels. H Set (i,j) to 0, and then use the sharpening intensity value I obtained from the above-mentioned denoised image pixels and the corresponding pixels. H Adding (i,j) together will produce a clearer document.

[0133] like Figure 2 As shown in another embodiment, the shared screen and document maintain the invariance of image content between frames within a certain time. When the document is turned, the image content changes abruptly. To simplify the operation process and improve sharpening efficiency, when the current frame image is received, it is determined whether the current frame image is the first frame image. If it is confirmed that the current frame image is the first frame image, the sharpening processing flow described in module 500-550 is directly executed on the image. If it is determined that the current frame image is not the first frame image, the current frame image is divided into n*n regions, and the image content of each region is judged to be different from that of the corresponding region in the previous frame. If the content of the region has changed, the sharpening processing flow described in module 500-550 is executed on the region to achieve sharpening enhancement of the region; otherwise, the sharpening result of the region is the same as the sharpening result of the previous frame, and the sharpening result of the corresponding region in the previous frame can be used directly. This avoids repeatedly sharpening the same frame image multiple times, thus improving sharpening efficiency.

[0134] It should be noted that although several units / modules or sub-units / modules of a document clarification processing apparatus are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0135] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0136] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for clarifying shared documents in video conferencing, characterized in that, include: Extracting luminance components from YUV format video image data; Denoising is performed based on the brightness component to obtain a denoised image; Edge information is extracted from the denoised image; The sharpening intensity is obtained based on the edge information; The denoised image is segmented into foreground and background; The denoised image is sharpened based on the sharpening intensity and foreground information; The current frame image of the shared document in the video conference is determined to be the first frame image. If it is the first frame image, perform the above-mentioned sharpening process; If it is not the first frame image, then divide it into multiple regions; Determine whether the content of each region has changed compared to the corresponding region in the previous frame; If the area changes, the above-described clarification process is repeated for that area. If the region remains unchanged, the sharpening result from the previous frame is used directly to sharpen that region.

2. The video conferencing shared document enhancement method according to claim 1, characterized in that, The extraction of luminance components based on the YUV format video image data includes: The luminance component of the pixels in the region to be processed is extracted based on the image data of each frame.

3. The method for clarifying shared documents in video conferencing according to claim 1, characterized in that, The step of denoising based on the luminance component to obtain a denoised image includes: The luminance component is smoothed using a bilateral filtering method to obtain the denoised image; The bilateral filtering includes spatial distance parameters and grayscale distance parameters.

4. The method for clarifying shared documents in video conferencing according to claim 1, characterized in that, The step of obtaining edge information from the denoised image includes: A low-frequency image is obtained based on the denoised image; The difference between the denoised image and the low-frequency image is the edge information.

5. The video conferencing shared document enhancement method according to claim 4, characterized in that, The step of obtaining the low-frequency image based on the denoised image includes: The low-frequency image is obtained by performing Gaussian blur processing on the denoised image.

6. The method for clarifying shared documents in video conferencing according to claim 1, characterized in that, The step of obtaining the sharpening intensity based on the edge information includes: Based on the positive and negative information of the edge, the edge is divided into a white edge and / or a black edge; Different enhancement coefficients are applied to the white edge and / or the black edge; The sharpening intensity is obtained based on the enhancement coefficient and the edge information.

7. The video conferencing shared document enhancement method according to claim 6, characterized in that, The step of obtaining the sharpening intensity based on the enhancement coefficient and the edge information includes: When the denoised image is divided into white edges, and the edge information is greater than zero and less than the first threshold, the sharpening intensity is zero. When the edge information is greater than or equal to the first threshold and less than the second threshold, the enhancement coefficient is the first enhancement coefficient, and the sharpening intensity is the product of the first enhancement coefficient and the edge information. When the edge information is greater than or equal to the second threshold and less than the third threshold, the enhancement coefficient is the second enhancement coefficient, and the sharpening intensity is the product of the second enhancement coefficient and the edge information. When the edge information is greater than or equal to the third threshold and less than the fourth threshold, the enhancement coefficient is the third enhancement coefficient, and the sharpening intensity is the product of the third enhancement coefficient and the edge information. When the edge information is greater than or equal to the fourth threshold, the sharpening intensity is the fourth threshold.

8. The video conferencing shared document enhancement method according to claim 6, characterized in that, The step of obtaining the sharpening intensity based on the enhancement coefficient and the edge information includes: When the denoised image is divided into black edges, and the edge information is greater than the fifth threshold and less than zero, the sharpening intensity is zero. When the edge information is greater than the sixth threshold and less than or equal to the fifth threshold, the enhancement coefficient is the fourth enhancement coefficient, and the sharpening intensity is the product of the fourth enhancement coefficient and the edge information. When the edge information is greater than the seventh threshold and less than or equal to the sixth threshold, the enhancement coefficient is the fifth enhancement coefficient, and the sharpening intensity is the product of the fifth enhancement coefficient and the edge information. When the edge information is greater than the eighth threshold and less than or equal to the seventh threshold, the enhancement coefficient is the sixth enhancement coefficient, and the sharpening intensity is the product of the sixth enhancement coefficient and the edge information. When the edge information is less than or equal to the eighth threshold, the sharpening intensity is the eighth threshold.

9. The method for clarifying shared documents in video conferencing according to claim 1, characterized in that, The segmentation of the foreground and background in the denoised image includes: Perform grayscale value statistics on pixels with sharpening intensity greater than the ninth threshold within a given window; The statistical median was used as the background value for this region. If the absolute value of the difference between the current pixel value and the background value is greater than a preset value, then the pixel is a foreground pixel; otherwise, it is a background pixel. Pixels with a sharpening intensity no greater than the ninth threshold are considered as foreground pixels that need to be enhanced.

10. The method for clarifying shared documents in video conferencing according to claim 1, characterized in that, The denoised image is sharpened based on the sharpening intensity and foreground information: Set the sharpening intensity of each pixel in the background to zero; The denoised image is sharpened by adding each pixel of the denoised image to its corresponding sharpening intensity value.

11. A video conferencing shared document enhancement processing device, characterized in that, include: The extraction module is used to extract the luminance component based on YUV format video image data; A denoising module is used to denoise based on the luminance component to obtain a denoised image; An edge information extraction module is used to extract edge information from the denoised image; A sharpening intensity acquisition module is used to acquire the sharpening intensity based on the edge information; The segmentation module is used to segment the foreground and background of the denoised image; A sharpening module is used to sharpen the denoised image based on the sharpening intensity and foreground information; The device further includes: A module for determining whether the current frame image of the video conferencing shared document is the first frame image; This module is used to perform a sharpening process on the first frame of the image; A module used to divide non-first frame images into multiple regions; A module used to determine whether the content of each region has changed compared to the corresponding region in the previous frame; A module for performing the sharpening process on the region if the region changes; This module is used to sharpen the region directly using the sharpening result of the previous frame if the region has not changed.

12. The video conferencing shared document enhancement processing device according to claim 11, characterized in that, The extraction module includes: A unit used to extract the luminance component of the region to be processed based on each frame of image data.

13. The video conferencing shared document enhancement processing device according to claim 11, characterized in that, The noise reduction module includes: A unit for smoothing the luminance component using bilateral filtering to obtain the denoised image; The bilateral filtering includes spatial distance parameters and grayscale distance parameters.

14. The video conferencing shared document enhancement processing device according to claim 11, characterized in that, The module for acquiring edge information from the denoised image includes: A unit for acquiring a low-frequency image based on the denoised image; The difference between the denoised image and the low-frequency image is the edge information.

15. The video conferencing shared document enhancement processing device according to claim 14, characterized in that, The module for obtaining a low-frequency image based on the denoised image includes: A unit for performing Gaussian blur processing on the denoised image to obtain the low-frequency image.

16. The video conferencing shared document enhancement processing device according to claim 11, characterized in that, The module for obtaining sharpening intensity includes: Units used to divide edges into white edges and / or black edges based on the positive and negative edge information; A unit for setting different enhancement coefficients for the white edge and / or the black edge; A unit for obtaining the sharpening intensity based on the enhancement coefficient and the edge information.

17. The video conferencing shared document enhancement processing device according to claim 16, characterized in that, The module for obtaining the sharpening intensity based on the enhancement coefficient and the edge information includes: When the denoised image is divided into white edges, the unit is used to make the sharpening intensity zero when the edge information is greater than zero and less than a first threshold. The unit is used when the edge information is greater than or equal to a first threshold and less than a second threshold, the enhancement coefficient is a first enhancement coefficient, and the sharpening intensity is the product of the first enhancement coefficient and the edge information. The unit is used when the edge information is greater than or equal to the second threshold and less than the third threshold, the enhancement coefficient is the second enhancement coefficient, and the sharpening intensity is the product of the second enhancement coefficient and the edge information. The unit is used when the edge information is greater than or equal to the third threshold and less than the fourth threshold, the enhancement coefficient is the third enhancement coefficient, and the sharpening intensity is the product of the third enhancement coefficient and the edge information. A unit used to sharpen the edge information to the fourth threshold when the edge information is greater than or equal to the fourth threshold.

18. The video conferencing shared document enhancement processing device according to claim 16, characterized in that, The module for obtaining the sharpening intensity based on the enhancement coefficient and the edge information includes: When the denoised image is divided into black edges, a unit is used where the sharpening intensity is zero when the edge information is greater than the fifth threshold and less than zero. The unit is used when the edge information is greater than the sixth threshold and less than or equal to the fifth threshold, the enhancement coefficient is the fourth enhancement coefficient, and the sharpening intensity is the product of the fourth enhancement coefficient and the edge information. The unit is used when the edge information is greater than the seventh threshold and less than or equal to the sixth threshold, the enhancement coefficient is the fifth enhancement coefficient, and the sharpening intensity is the product of the fifth enhancement coefficient and the edge information. The unit is used when the edge information is greater than the eighth threshold and less than or equal to the seventh threshold, the enhancement coefficient is the sixth enhancement coefficient, and the sharpening intensity is the product of the sixth enhancement coefficient and the edge information. A unit used when the edge information is less than or equal to the eighth threshold, at which point the sharpening intensity is the eighth threshold.

19. The video conferencing shared document enhancement processing device according to claim 11, characterized in that, The segmentation module includes: A unit used to perform grayscale value statistics on pixels with sharpening intensity greater than the ninth threshold within a given window; The unit used to use the statistical median as the background value for the region; A unit used to determine if a pixel is a foreground pixel if the absolute value of the difference between the current pixel value and the background value is greater than a preset value, and otherwise a background pixel. This is used to define pixels with a sharpening intensity no greater than the ninth threshold as foreground points that need to be enhanced by default.

20. The video conferencing shared document enhancement processing apparatus according to any one of claims 11-18, characterized in that, The sharpening module includes: A unit used to set the sharpening intensity of each pixel in the background to zero; This unit is used to add each pixel of the denoised image to its corresponding sharpening intensity value to sharpen the denoised image.

Images