Encoder-optimized text rendering

Abstract

To provide a method and device for rendering a text in a raster image suitable for block-wise transform coding.SOLUTION: A method 100 for rendering a text character string in a digital raster image suitable for block-wise transform coding comprises: obtaining a partition of an image area into coding blocks for the block-wise transform coding (110); representing a text character string as a plurality of graphical elements from a typeface which are arranged according to a tentative layout in the image area, the tentative layout defining at least position, orientation and size of each graphical element (114); modifying the tentative layout by applying a spectrum sparsening operation to at least one nonempty coding block, thereby obtaining a modified layout (116); and rendering a digital raster image of the graphical elements arranged according to the modified layout (118).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of two-dimensional image data generation. In particular, the present disclosure presents a technique for rendering text in a raster image suitable for block-based transform coding.

Background Art

[0002] Thanks to significant progress in the field of digital image coding, it is possible to represent an input image as a bitstream with a very low bitrate, from which the image can be reconstructed without significantly degrading its visual quality. In some use cases, the input image is fixed, e.g., recorded by a camera or pre-synthesized. And the digital image coding process can usually only control the bitrate of the bitstream by applying various data compression techniques including irreversible and non-lossy compression. In other use cases, the input image can be subject to modifications within certain limits. This applies particularly when the input image is synthesized (rendered) simultaneously with image coding, and the synthesis is to meet the rendering specifications of the end user leaving some aspects undetermined. For example, the rendering specifications can define the geometric shapes and positions of some three-dimensional objects, but do not specify their colors and textures and / or do not specify the lighting of the scene. In other words, the person or device responsible for rendering has the freedom to select the above colors, textures, or lighting during rendering, and each selection is acceptable to the end user. The end user may be a person viewing the reconstructed image (e.g., a consumer), a processor performing optical character recognition (OCR) on the reconstructed image, or actually the system owner for which the image coding is performed instead.

[0003] The inventors recognized that this degree of freedom can be used to improve the performance of image coding processes. In particular, the inventors discovered untapped potential in the field of text rendering. [Overview of the Initiative]

[0004] One object of this disclosure is to make available a method for rendering text strings within a digital raster image so that the digital raster image is suitable for block-unit transformation encoding. A further object is to make available a text rendering method for generating a digital raster image having a sparse spectrum. A further object is to make available such a method that satisfies one or more layout constraints. A further object is to make available a text rendering method suitable for generating overlay text that is encoded together with a background digital image. Another further object is to propose devices and computer programs having these capabilities.

[0005] At least some of these objectives are achieved by the present invention as defined by the independent claims. The dependent claims relate to advantageous embodiments.

[0006] A first aspect of this disclosure provides a method for rendering a text string in a digital raster image suitable for block-based transformation coding. The text string is predetermined with respect to an entity performing the method, i.e., the text string may be received from an end user (see the above description) or created by a running software application, and such entity shall not modify the text string. The method includes: obtaining a partition of an image region into coding blocks for block-based transformation coding; representing the text string as a plurality of graphic elements from a typeface arranged according to a provisional layout in the image region, wherein the provisional layout defines, represents, at least the position, orientation, and size of each graphic element; modifying the provisional layout by applying a spectral sparsification operation to at least one non-empty coding block to obtain the modified layout; and rendering a digital raster image of the graphic elements arranged according to the modified layout.

[0007] A spectral sparsification operation (or, in other words, a spectral sparsification operation) makes the spectrum of a coded block in a digital raster image sparser than if the spectral sparsification operation were not performed. That is, a spectral sparsification operation is likely to reduce the number of non-zero conversion coefficients that occur when applied to a coded block. This means that a digital raster image can be encoded at a slightly lower bitrate. Recognizing that spectral sparsification is an important accelerator for efficient image encoding, we have developed a class of advantageous spectral sparsification operations that are non-destructive when applied to text strings. More precisely, the spectral sparsification operation modifies the provisional layout (subject to optional layout constraints) but preserves the text string. The method according to the first aspect is a way of bringing the text rendering process and the subsequent image encoding process closer together, thereby leveraging the synergistic effects between these processes. In this way, the encoded image with the rendered text string utilizes the amount of available communication bandwidth or memory space in the system more efficiently.

[0008] The spectral sparsification operations further specified below can be characterized as model-based open-loop operations that do not presuppose interaction with subsequent image encoding processes. The spectral sparsification method is model-based in the sense that, based on the inventors' extensive experience, its effects on digital raster images can be accurately predicted. Model-based open-loop methods enable economical use of processing resources and execution time, thereby ensuring that the method according to the first embodiment is suitable for important mass-market use cases. Spectral sparsification operations are not appropriately characterized as trial-and-error methods. In contrast, model-free closed-loop methods for achieving compatible bitrate reductions are likely to be more computationally expensive. Closed-loop methods may include, for example, iterative searches, each iteration including (1) rendering a text string in a new layout, (2) encoding an image, and (3) evaluating the change in size of the encoded image until a sufficient size is reached. Iterative searches are likely to be less efficient than the methods proposed herein because they are not guided by experience of how the spectrum of a digital image responds to layout modifications. For example, a considerable number of encoding operations (2) may be required for each image. Similarly, poor performance can be expected from a model-free approach where a text string is rendered against a large number of randomly generated layouts, and the best-performing layout is selected based on the size of the encoded image.

[0009] In some embodiments, spectral sparsification operations include one or more of the following: rotation of graphic elements, tilting of graphic elements, isotropic or anisotropic rescaling of entire graphic elements, isotropic or anisotropic rescaling of a portion of graphic elements, translation of graphic elements, typeface modification, typeface replacement, and contrast modification. The inventors have discovered specific guidelines for use with these subtypes of spectral sparsification operations, which are described in detail below.

[0010] In particular, spectral sparsification operations can consist of rigid body transformations of graphic elements, such as rotation, tilt, or translation.

[0011] In some embodiments, the method further includes obtaining one or more layout constraints specifying typeface, maximum range, minimum range, and / or orientation.

[0012] In some embodiments, block-unit transformation coding may include projection of a doubly periodic function onto an orthogonal basis, followed by a round-to-zero operation.

[0013] In some embodiments, graphic elements include glyphs such as characters.

[0014] A second aspect of this disclosure relates to devices configured to perform the methods of the first aspect. These devices may have different primary purposes, such as image or video editing, image or video content management, video subtitling, image or video playback, word processing, or other authoring tools. Furthermore, devices may be designed for specific use cases, such as indoor or outdoor video surveillance, having an auto-annotation function for overlaying text strings. Devices in the second aspect of this disclosure generally share the effects and benefits of the first aspect, and they may be embodied with a comparable degree of technical modification.

[0015] This disclosure further relates to a computer program that includes instructions for causing a computer to perform the methods described above. The computer program may be stored in or distributed in a data carrier. As used herein, “data carrier” may be a transient data carrier, such as a modulated electromagnetic wave or light wave, or a non-transient data carrier. Non-transient data carriers include volatile and non-volatile memories, such as magnetic, optical, or solid-state type persistent and non-persistent storage media. Still within the scope of “data carrier,” such memories may be fixedly mounted or portable.

[0016] In general, all terms used in the claims should be interpreted according to their ordinary meanings in the art unless otherwise explicitly stated herein. All references to “one (a / an) element, apparatus, component, means, step, etc.” should be broadly interpreted as referring to at least one example of an element, apparatus, component, means, step, etc. unless otherwise specifically stated. The steps of any method disclosed herein do not need to be performed in the exact order disclosed unless otherwise specified.

[0017] The following describes the aspects and embodiments with reference to the attached drawings. [Brief explanation of the drawing]

[0018] [Figure 1] This is a flowchart of a method for rendering text strings within a digital image. [Figure 2] This is a flowchart of a method for providing an encoded digital image with a rendered text string. [Figure 3] This diagram shows a framework for specifying specific layout constraints. [Figure 4] This is a block diagram of a device suitable for performing the methods shown in Figures 1 and 2. [Figure 5]A diagram showing a spectral sparsification operation including rotation of graphic elements. [Figure 6] A diagram showing a spectral sparsification operation including rotation of graphic elements. [Figure 7] A diagram showing a spectral sparsification operation including tilting of graphic elements. [Figure 8] A diagram showing a spectral sparsification operation including local and / or anisotropic rescaling of graphic elements. [Figure 9] A diagram showing a spectral sparsification operation including local and / or anisotropic rescaling of graphic elements. [Figure 10] A diagram showing a spectral sparsification operation including local and / or anisotropic rescaling of graphic elements. [Figure 11] A diagram showing an encoding block of a raster image. [Figure 12] A diagram showing an encoding block of a raster image. [Figure 13] A diagram showing a spectral sparsification operation including translation of graphic elements. [Figure 14] A diagram showing a spectral sparsification operation including translation of graphic elements. [Figure 15] A diagram showing a spectral sparsification operation including translation of graphic elements. [Figure 16] A diagram showing a spectral sparsification operation including translation of graphic elements. [Figure 17] A diagram showing a spectral sparsification operation including various font corrections. [Figure 18] A diagram showing a spectral sparsification operation including various font corrections. [Figure 19] A diagram showing a spectral sparsification operation including various font corrections. [Figure 20] A diagram showing a spectral sparsification operation including various font corrections. [Figure 21]This figure shows the orthogonal basis of a double periodic function for each 8x8 pixel coding block. [Modes for carrying out the invention]

[0019] Next, aspects of the present disclosure are described more fully below with reference to the accompanying drawings illustrating specific embodiments of the invention. However, these embodiments may be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided as examples to ensure that the present disclosure is thorough and complete and fully conveys to those skilled in the art the scope of all aspects of the invention. Throughout the description, similar numbers refer to similar elements.

[0020] Referring to Figure 1, a method 100 for rendering text strings in a digital raster image suitable for block-based transformation encoding is described here.

[0021] A raster image is understood to include a matrix or grid of pixels, where the pixels are preferably square or rectangular. The pixels form an image region 310 (Figure 3). A raster image may also be a block-format pixel buffer image. An image with vector-format graphic elements is not a raster image.

[0022] A text string is understood to be an ordered sequence of characters selected from a predetermined character table, such as the Unicode table (ISO / IEC 10646). Characters can represent letters, syllables, ideograms, modifiers, marks, numbers, punctuation marks, mathematical symbols, currency symbols, separators, ligatures, and more.

[0023] Furthermore, transform coding, in a broad sense, is a double periodic function It is understood that this involves projecting image data related to the orthogonal basis of TIFF0007882828000001.tif5170. TIFF0007882828000002.tif14170 Here, TIFF0007882828000003.tif5170 is a conversion coefficient, TIFF0007882828000004.tif4170 is image data for pixel (n1,n2). It should be noted that the constraints of TIFF0007882828000005.tif5170 may correspond to a single period or a constant value for the lowest (k1,k2) pair. In particular, the basis can consist of real-valued double periodic harmonics (e.g., DCT, DST, DFT, wavelet transform). The transformation coefficients calculated in the projection operation constitute a discrete representation of the spectrum of the image data. The encoded image may include the transformation coefficients after non-destructive data compression (e.g., entropy, Huffman, Lempel-Ziv, run-length encoding, binary or non-binary arithmetic coding, e.g., context-adaptive variable-length coding, CAVLC, context-adaptive binary arithmetic coding, CABAC) and / or other processing steps. In particular, the transformation coefficients may undergo rounding operations to zero, e.g., quantization operations. When conversion coefficients are supplied to one of the aforementioned non-destructive data compression techniques, rounding is important because zero conversion coefficients are not typically encoded as a number of zero values ​​("0.0"), but can be omitted to occupy significantly less space in the image bitstream than non-zero conversion coefficients. The intended conversion encoding is block-by-block conversion in the sense that the processing of an encoded block is independent of the processing of other encoded blocks.

[0024] Optionally, the transformation coefficients of a coded block can be encoded predictively, particularly by intra-frame predictive ("intra-predictive") coding. According to such predictive coding techniques, which are well known as such, the transformation coefficients of one coded block are expressed incrementally with reference to one or more previous or subsequent coded blocks. This results in efficient data compression, especially when the image depicts a natural scene with a high degree of spatial autocorrelation. The inventors have recognized that data compression is generally more important when the spectrum of coded blocks is sparse, i.e., when they contain a large proportion of zero transformation coefficients.

[0025] In the first step 110 of Method 100, a partition 320 of the image region 310 is acquired. The partition 320 may be acquired as a predetermined partition based on instructions from the end user (e.g., according to prior agreement or standard specifications), or it may be generated by the entity performing Method 100. The partition 320 defines a set of coding blocks such that each point of the image region 310 belongs to a coding block. Similarly, the combination of coding blocks is equal to the image region 310. In Figure 3, the partition 320 is shown by dotted lines separating the coding blocks. The same dotted line notation is used to show the boundaries of coding blocks in Figures 5, 6, 10, and 13-19. When the image is a frame of a video sequence, the coding blocks may be macroblocks in the sense of the ITU-T H.26x video coding standard, and can be transform blocks, predictive blocks, or blocks having both uses. Note that Figure 3 is a simplified diagram for illustrative purposes only. In typical use cases at the time of the present invention, it is common to use much finer partitions. For example, macroblocks within a video frame can be 4x4 pixels, 8x8 pixels, 16x16 pixels, 32x32 pixels, or 64x64 pixels.

[0026] Figure 21 includes 64 plots in the (n1,n2) plane of real-valued biperiodic function bases determined for the 8x8 pixel case as discrete cosine functions. The top line in Figure 21 of TIFF0007882828000006.tif10170 is, ρ 0,0 (n1,n2), ρ 1,0 (n1,n2), ρ 2,0 (n1,n2), ..., ρ 7,0 (n1,n2) Includes plot of, ρ 7,7 The plots up to (n1,n2) can be seen in the lower right corner. In Figure 21, white represents, This represents TIFF0007882828000007.tif5170, and the black color is, This represents TIFF0007882828000008.tif5170. Coding blocks corresponding to a single basis function or a linear combination of a few basis functions are relatively inexpensive to encode (i.e., they will have a sparser spectrum) than coding blocks with a more complex appearance and / or higher informational content that must be formed by a larger number of basis functions. Based on the examination in Figure 21, coding blocks with equidistant horizontal or vertical lines are, for example, basis functions k1=0 or k2=0. It may be recognized that encoding using TIFF0007882828000009.tif4170 is likely to be inexpensive. Encoding blocks that mainly have diagonal elements are generally, for example, k1=k2 Encoding using TIFF0007882828000010.tif4170 is also inexpensive. According to the Nyquist-Shannon sampling theorem, TIFF0007882828000011.tif5170 can be seen as a complete basis for an 8x8 pixel coded block.

[0027] The rendered text string constitutes the input data to Method 100. That is, the text string is predetermined from the perspective of the entity performing the Method, i.e., the text string may be received from an end user or automatically created by a software application, and the entity shall not modify the text string under normal conditions. Method 100 may optionally accept one or more layout constraints as input. The layout constraints obtained in the optional second step 112 specify the maximum range 332 (Figure 3) of the rendered text string, the minimum range 334 of the rendered text string, and / or the orientation α of the rendered text string. The layout constraints may further specify one or more acceptable typefaces. The text string should be rendered according to these layout constraints, which may limit the effectiveness of the spectral sparsification operation performed.

[0028] In the third step 114, the text string is represented as multiple graphic elements from a typeface arranged according to a provisional layout within the image area 310. The provisional layout defines at least the position, orientation, and size of each graphic element. Graphic elements may include glyphs, particularly glyphs representing characters. It is emphasized that graphic elements in this sense refer to specific representations of characters in this typeface, i.e., concrete shapes, rather than abstract characters. Furthermore, the characters and graphic elements of the text string do not necessarily have a one-to-one relationship; they may be one-to-many or many-to-one. Graphic elements can be represented as vector graphics, such as scalable lines, curves, or polygons.

[0029] In the next step 116, the temporary layout is modified by applying spectral sparsification operations. Spectral sparsification operations can be applied to a single coding block at a time, a group of coding blocks at a time (e.g., adjacent coding blocks), or the entire image region 310. Spectral sparsification operations are preferably limited to non-empty coding blocks, i.e., such coding blocks containing at least one graphic element or a portion of a graphic element. As described in detail below, spectral sparsification operations may include rotating a graphic element (step 116.1), tilting a graphic element (step 116.1), isotropic or anisotropic rescaling of a complete graphic element (step 116.2), isotropic or anisotropic rescaling of a portion of a graphic element (step 116.2), translating a graphic element (step 116.3), font correction (step 116.4), font replacement or contrast correction (step 116.4), or various combinations thereof.

[0030] The output of step 116 is a modified layout, which forms the basis for a fifth step 118 in which a digital raster image of the graphic elements is rendered according to the modified layout. Rendering may include rasterization, i.e., the transformation of vector graphics as a matrix of pixels. Rasterization may include performing a line drawing algorithm or a curve drawing algorithm. Alternatively, if typefaces represent the graphic elements as bitmaps, rendering in step 118 may include combining such bitmaps into an output digital raster image using their size and position according to the modified layout.

[0031] Rendering can be performed separately for the entire image region or as a single operation for each encoded block.

[0032] Step 118 may further include pre-processing steps before actual rendering. In the pre-processing steps, several groups of graphic elements arranged according to the modified layout are joined, for example, by deforming (expanding) parts of the graphic elements toward each other and / or by adding ligatures or connectors. The resulting joined appearance may be essential for certain non-Latin scripts, including Arabic, and can be used as an alternative for Latin scripts to resemble cursive.

[0033] As shown in Figure 2, the text rendering method 100 described above may be embedded in a method 200 for providing an encoded digital image of a text string. Such a method 200 may include rendering a digital raster image of a text string by performing method 100. After this, a block-by-block transform coding operation 210 is applied to each block of the above partition 320 of the image region. The transform coding may include a projection 210.1 with respect to an orthogonal basis of a double periodic function (see Equation 1 above) followed by a rounding operation 210.2 to zero. The output of method 200 is an encoded digital image, which can be represented as a set of transform coefficient values ​​having a predetermined coding.

[0034] Embodiments of Method 200 can be particularly adapted to text overlay use cases. In such embodiments, a background image is acquired (e.g., received from an end user, recorded by a camera) and combined with the digital raster image rendered in step 118. For example, pixels representing rendered graphic elements may be replaced with corresponding pixels in the background image, and the complements of these pixels are treated as transparent, i.e., the background image is unaffected here. The block-by-block transformation coding operation 210 is applied to the combined image. Optionally, pixels representing rendered graphic elements may be overlaid with a configurable degree of transparency so that the background image is partially visible through the text. It is understood that the background image and the digital raster image may need to be fitted to each other before being combined, for example, by rescaling, cropping, or expanding operations.

[0035] Alternatively, a text string may be displayed on top of a background image (step 114), and a spectral sparsification operation (step 116) may be applied to the combination of the background image and the text string according to a hypothetical layout. This allows the spectral sparsification operation to leverage synergies with the background image, for example, by making the placed graphic element resemble the background image in some encoding blocks, resulting in a sparser spectrum and therefore less expensive encoding. For example, a successful spectral sparsification operation may identify layout modifications within an encoding block such that the spectrum of the overlaid graphic element leaves approximately the same zero values ​​for the conversion coefficients as the spectrum of the background image in the same encoding block.

[0036] Methods 100 and 200 can be executed by a general-purpose computer. In particular, methods 100 and 200 can be executed by a device having the basic functional structure shown in Figure 4. As shown, device 400 includes a processing circuit 414, a memory 410, and an external interface 418. An internal data bus 416 facilitates communication between these components. The memory 410 may be suitable for storing a computer program 412 having instructions to implement either method 100 or 200. The external interface 418 may be a communication interface that enables device 400 to communicate with similar devices (not shown) operated by consumers or video content authors (e.g., recording devices) via a wide-area or local area network 420. Furthermore, device 400 can communicate with a host computer 430 such as a server for storing raw or encoded image data, fonts, etc., or networked ("cloud") processing resources that can be utilized by device 400 as needed. It is understood that the host computer 430 can be configured to offload the computationally demanding operations of the device 400 within methods 100 and 200.

[0037] Figures 5 and 6 illustrate spectral sparsification behavior, including rotation of graphic elements.

[0038] The above rotations can be determined to reduce the number of intrinsic line orientations within the encoding block. In the left half of Figure 5, the symbol " / " (forward slash) is nearly parallel to the left stroke of the letter "A", so it can be expected that these graphic features can be encoded using the same or partially the same basis function. However, in the right half of Figure 5, the symbol " / " and the left stroke of the letter "A" have different orientations and require more conversion coefficients to have non-zero values. Taking this into consideration, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the right half of Figure 5 to be more similar to the left half of Figure 5.

[0039] The above rotation can be defined so that the orientation of the lines aligns with a vertical or horizontal axis, or with the diagonals of these axes. The vertical or horizontal axis corresponds to the axis of the pixel matrix of the rendered digital raster image, which is how the basis functions are parameterized (n1, n2 variables). In the right half of Figure 6, the letter "L" is not aligned with the axis of the coding block, where the axis corresponds to the boundary of the coding block containing the letter "L" drawn with a dotted line. The unaligned letter "L" is costly to encode because it cannot be represented as a combination of a few basis functions. The orientation in the left half of Figure 6 is likely more preferable from the viewpoint of block-by-block transformation coding. For this reason, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the right half of Figure 6 to be more similar to the left half of Figure 6.

[0040] Figure 7 illustrates spectral sparsification behavior, including tilting graphic elements. A rightward tilt may correspond to characters that gradually become italicized in some typefaces. Conversely, tilt may be used to deitalize one or more characters to some extent, thereby allowing their strokes (e.g., crossbars, shoulders, bowls, stems) to align better with the vertical, horizontal, or diagonal axis. Tilt can contribute to reducing the number of intrinsic line orientations within a coding block, aligning line orientations with the vertical or horizontal (or diagonal) axis, and / or reducing the number of intrinsic vertical or horizontal distances within a coding block. The “distance” within a coding block may be the thickness of a stroke or the length of separation (space, gap). Furthermore, it should be noted that because the basis function is periodic, the transform coding effectively samples the periodic expansion of the coding block, and therefore the spectral sparsification behavior should be designed to minimize the number of intrinsic distances in the periodic expansion of the coding block. This is illustrated below with reference to Figure 12.

[0041] Figures 8 and 9 illustrate spectral sparsification behavior, including local rescaling and / or anisotropic rescaling of graphic elements.

[0042] Figure 8 shows how the x-height of the lowercase letter "h" can be changed by local vertical rescaling, which makes it possible to limit the number of intrinsic vertical distances. In Figure 8, no horizontal rescaling is applied. Note that the central shape in Figure 8 has a single vertical distance, since the x-height is approximately half the cap height, as long as the letter "h" occupies a complete encoding block vertically. In some cases, the left and right shapes in Figure 8 contain at least two intrinsic vertical distances (i.e., x-height and ascender height) and therefore require a larger number of non-zero conversion factors. Accordingly, some embodiments of Method 100 include performing a spectral sparsification operation that can modify the layout to more closely resemble the central shape in Figure 8.

[0043] Figure 9 shows the effect of a horizontal-only overall rescaling, which is an example of anisotropic rescaling. Such rescaling can affect the number of intrinsic distances within a coded block. Anisotropic rescaling also involves combinations of horizontal and vertical rescaling behavior due to different factors.

[0044] Figure 10 shows an example in which a rescaling operation can be used for the purpose of spectral sparsification. The left half of Figure 10 corresponds to a hypothetical layout, where the lowercase letter "o" and the digit "0" (zero) occupy a common encoding block. The letter "o" is slightly smaller and narrower than the digit "0". The right half of Figure 10 corresponds to the output of the spectral sparsification operation, where the complete graphic element representing the digit "0" is isotropically downscaled to have the same height as the letter "o". As part of the spectral sparsification operation, it is further ensured that the horizontal space between the letter "o" and the digit "0" is approximately equal to the width of the letter "o", which can be achieved by translating the letter "o" or the digit "0" horizontally. As a result, the three horizontal distances indicated by the arrows are approximately equal, as are the two vertical distances. Based on these considerations, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the left half of Figure 10 to be more similar to the right half of Figure 10. Note that since the conversion encoding is block-based, it is not necessary to consider adjacent characters "N" and digits "5" within adjacent encoding blocks.

[0045] To illustrate the effect of spectral sparsification on the pixel level, Figures 11 and 12 show 8x8 pixel encoded blocks of a raster image. In the left half of Figure 11, lines have a uniform horizontal thickness of one dark pixel, and they are separated horizontally by three bright pixels. For the reasons described above, a horizontal periodic expansion of the encoded blocks is considered, and the outer light pixels are combined to form a total separation of 2+1=3 light pixels. Conceptually, the vertical boundaries of the encoded blocks are "glued together," and the horizontal boundaries can be considered similarly. In contrast, in the right half of Figure 11, two bright pixels, one dark pixel, three bright pixels, and two dark pixels are identified horizontally. The right half of Figure 11 is more costly to encode because it has a larger number of intrinsic horizontal distances. Therefore, some embodiments of Method 100 include performing spectral sparsification to modify the layout according to the right half of Figure 11 to be more similar to the left half of Figure 11.

[0046] In the first coded block of Figure 12, the upper portion contains constant image data, and the lower portion contains a graphic element resembling a forward slash, with horizontal distances of 1 and 7. (It is reminiscent of the relevance of distance in periodic expansion.) In the second coded block of Figure 12, the central upper portion contains constant image data, and the lower portion contains two copies of the graphic element, with horizontal distances of 1 and 3. This suggests that the respective costs of coding the first and second coded blocks of Figure 12 are approximately equal. Therefore, it is beneficial to gather the two copies of the graphic element in the same coded block rather than using a layout located in two different coded blocks. Furthermore, it is useful to consider a third coded block of Figure 12, which contains horizontal distances of 1 and 7 across its entire vertical range. The third coded block can be expected to have even more zero-value transformation coefficients than the first and second coded blocks of Figure 12. The third coded block of Figure 12 can be obtained from the second coded block by translating the graphic element on the right upwards until it aligns with the graphic element on the left.

[0047] Taking these considerations into account, some embodiments of Method 100 include performing a spectral sparsification operation that involves translating one or more graphic elements so that geometrically similar graphic elements are grouped into one coding block and / or geometrically dissimilar graphic elements are separated into different coding blocks. This may reduce the number of intrinsic line orientations and / or intrinsic vertical or horizontal distances within the coding block.

[0048] Figures 13, 14, 15, and 16 illustrate spectral sparsification operations involving the translation of graphic elements. As described above, the purpose of such translations may be to group geometrically similar graphic elements together in each coding block and / or to separate geometrically dissimilar graphic elements. These expressions can be understood in the usual sense. Alternatively, in some embodiments, the identification of “geometrically similar” and “geometrically dissimilar” graphic elements may be systematized to identify spectrally similar and spectrally dissimilar graphic elements. This is achieved by caching spectra (transformation coefficients) from previous block-unit transformation codings of various graphic elements. Spectra may be cached in full or in a simplified form, and a useful simplified format may describe which transformation coefficients are non-zero in each spectrum. For example, two graphic elements may be considered spectrally similar if they have the same or nearly identical set of non-zero transformation coefficients, and spectrally dissimilar otherwise. Spectrally similar graphic elements are grouped together, and spectrally dissimilar graphic elements are separated into different coding blocks.

[0049] To avoid overly close (non-global) optimization of graphic element placement, such spectral sparsification operations, including translation of graphic elements, are preferably applied to a search window of multiple adjacent coding blocks at a time. Within the search window, the number of possible redistributions of graphic elements to different coding blocks is evaluated, and the favorable one is selected. Among the various possible redistributions of graphic elements, the more suitable ones can satisfy one or more of the following criteria: a smaller total number of non-zero conversion coefficients, a smaller proportion of high-frequency coefficients (larger values ​​for k1 and k2), and smaller variation in the configuration of the set of non-zero conversion coefficients (with respect to the intra-predictive scan order) between consecutive coding blocks. Obviously, characters in a text string are not reordered under the above redistribution, but their sequence is preserved. However, to limit the complexity of the computation, the window should not be too wide, as is the case when an entire long text string is handled in a single spectral sparsification operation. For example, a search window can contain 2 to 10 coding blocks in the write direction, such as 3, 4, or 5 coding blocks. It is also possible to use a sliding search window.

[0050] Figure 13 illustrates the use of translation to group geometrically similar graphic elements into each coding block. In the left half of Figure 13, the left coding block is shared by fragments of the letter "F" and the character "\" (backslash), while the right coding block contains one partial copy and two complete copies of the character "\". However, since the fragments of the letter "F" and the character "\" are geometrically dissimilar, this may not be optimal for coding purposes. It is even less optimal because the character "\" occupies two coding blocks. In a periodic expansion of coding blocks where two vertical boundaries are assumed to be "glued to each other", this corresponds to a large discontinuity that is costly to represent with periodic basis functions. On the right side of Figure 13, the three copies of the character "\" are translated to the right, so all three are included in the right coding block, while the letter "F" is included alone in the left coding block. Since the left coding block has only two inherent line orientations and the right coding block has only one inherent line orientation, this translation can be expected to reduce the total number of resulting non-zero conversion coefficients. Furthermore, translation also reduced the number of coding blocks occupied by the leftmost character "\". Taking this into consideration, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the left half of Figure 13 to be more similar to the right half of Figure 13.

[0051] Figure 14 illustrates the use of translation to separate geometrically dissimilar graphic elements into coding blocks. In the left half of Figure 14, the graphic element representing the character "P" is contained within one coding block. Because the bowl and stem of this character are geometrically dissimilar, they must be represented by a relatively large number of basis functions, i.e., at a relatively high cost. This can be improved, for example, by splitting the character into one "D"-like part and one "I"-like part, and translating these into two different coding blocks, as shown in the right half of Figure 14, where the separation between the parts is exaggerated for visibility. This demonstrates that characters and graphic elements in a text string can have a one-to-many relationship. Alternatively, the character "P" could be split into one "I"-like part and one part resembling a horizontally flipped "C," so that they are placed on two different coding blocks. With this in mind, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the left half of Figure 14 to be more similar to the right half of Figure 14.

[0052] Figure 15 illustrates the use of translation to rearrange graphic elements within a coding block in order to reduce the number of inherent vertical distances within the coding block. In the left half of Figure 15, two graphic elements in the form of an umlaut dot are positioned close to a third graphic element representing a lowercase "o". Thus, the coding block has a large number of vertical distances. In contrast, in the right half of Figure 15, the umlaut dot is translated upward and positioned above the "o" at approximately the same height as the "o" itself. As a result, the vertical distance corresponding to the thickness of the dot and the "o" are approximately equal. Furthermore, the space corresponding to the height of the "o" (x height, lower vertical double arrow), the vertical space from the two dots of the umlaut to the top of the "o" (upper vertical double arrow), and the complete character within the coding block are all considered. The sum of the upper and lower free spaces of TIFF0007882828000012.tif6163 constitutes a triplet of approximately equal space. (It is recalled that distance is of paramount importance in the periodic expansion of coded blocks, and in the case of vertical distance, vertical periodic expansion is considered.) Therefore, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the left half of Figure 15 to be more similar to the right half of Figure 15.

[0053] Figure 16 illustrates the use of translation to rearrange graphic elements within a coding block in order to reduce the number of intrinsic vertical and horizontal distances within the coding block. The left half of Figure 16 is identical to the left half of Figure 15. In the right half of Figure 16, the umlaut dot is not only translated upward as in Figure 15, but is also further spaced out by horizontal translation so that it has a horizontal spacing approximately the same as the width of the "o". In the right half of Figure 16, not only the number of intrinsic vertical distances but also the number of intrinsic horizontal distances is minimized. This may be advantageous to some extent from the perspective of later predictive coding. Translation may represent an even greater advantage when in-frame predictive coding is applied. Therefore, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the left half of Figure 16 to be more similar to the right half of Figure 16.

[0054] Starting from the right half of Figure 16, yet another spectral sparsification can be obtained by converting umlaut dots to umlaut overbars, which may be accepted as an allograph by some readers. To explain this mathematically, the following pixel representation of the right half of Figure 16 is considered. The DCT spectrum of such an encoded block, TIFF0007882828000013.tif34170, has the following appearance, where * represents a non-zero transformation coefficient. TIFF0007882828000014.tif34170 When an umlaut dot is replaced with an umlaut overbar, it is as follows: TIFF0007882828000015.tif34170 Next, the DCT spectrum is, The filename becomes TIFF0007882828000016.tif34170. Since the number of non-zero conversion coefficients decreases by 3, it can be seen that the coding block can be represented more compactly digitally.

[0055] As described above, step 118 may include a preprocessing step of joining one or more graphic elements. Joining can be potentially useful in the context of spectral sparsification by translation, which can improve the visual appearance of the modified layout. For example, joining can make uneven spaces between characters appear more uniform and therefore less visible. Another application is to satisfy layout constraints that specify a minimum range 334 (Figure 3) of a text string within the image region 310. For example, the characters of a text string representing a word can be distributed across the minimum range 334, but are still recognized as a word thanks to ligatures. Joining characters typically adds negligible coding effort and can even be beneficial. In the particular case of Arabic script, such joining can be achieved by so-called kashida, i.e., by extending a portion of a graphic element, or by inserting a glyph (tatwil) that functions as a ligature. For further details, please refer to the research paper MJEBenatia et al., "Arabic text justification. Survey of historical methods of Arabic text justification, and a recommended algorithm," TUGboat, vol.27(2006), no.2, pp. 137-146.

[0056] Figures 17, 18, 19, and 20 illustrate spectral sparsification operations, including various typeface modifications or substitutions. Generally speaking, the typeface modifications and substitutions considered simplify the geometric shapes of graphic elements. This can contribute to reducing the number of inherent line orientations within a coding block, aligning line orientations with vertical or horizontal (or diagonal) axes, and / or reducing the number of inherent vertical or horizontal distances within a coding block. Note that if Method 100 is implemented in accordance with layout constraints specifying typefaces or a list of acceptable typefaces, the typeface substitutions under consideration must be limited accordingly.

[0057] Figure 17 illustrates a typeface modification in which a graphic element representing the capital letter "E" in a serif typeface (left half of Figure 17) is converted to an allograph for a non-serif "E" (right half). In other words, the conversion simplifies the terminal shape of the graphic element by removing the serif. The typeface modification further uniformizes the width of the different strokes on the letter "E". The graphic element in the right half of Figure 17 has fewer inherent line orientations and / or fewer inherent distances, and is therefore likely to be less expensive to encode. With this in mind, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the left half of Figure 17 to be more similar to the right half of Figure 17. Similarly, the spectral sparsification operation can replace an "E" from a serif typeface with an "E" from a sans-serif typeface, particularly one from a sans-serif typeface with a more uniform stroke width.

[0058] Figure 18 illustrates typeface modification in which typeface weights within a coding block are homogenized. Exemplary typeface weights include extra-thin, thin, medium, bold, and extra-bold. In the left half of Figure 18, three graphic elements representing the lowercase letter "a" have three different weights, corresponding to numerous inherent vertical and horizontal distances within the coding block. In the right half, the weights are nearly homogenized by typeface substitution or appropriate rescaling, which is advantageous for economical conversion coding. Therefore, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the left half of Figure 18 to more closely resemble the right half of Figure 18.

[0059] Referring to Figure 17, it has been shown that the width of each of the different strokes on a graphic element can be made uniform. Figure 19 shows a typeface modification that makes the width of a particular stroke on a graphic element uniform. In the right half of Figure 19, graphic elements from a typeface with non-uniform stroke weights are used to form the text string "abc". In the left half of the figure, a typeface with uniform stroke weights is used. Furthermore, the small ears and serifs seen in the right half of Figure 19 are absent in the left half. Both of these modifications tend to make the arrangement of graphic symbols in the left half cheaper to encode. Therefore, some embodiments of Method 100 include performing a spectral sparsification operation that modifies the layout according to the right half of Figure 19 to be more similar to the left half of Figure 19.

[0060] Figure 20 illustrates typeface modification that can alter the aperture of a graphic element. More precisely, Figure 20 shows three graphic elements representing the lowercase letter "e" with respect to the length of the lower open arc. The spacing from the vertex to the crossbar can be changed without the graphic element losing its significance as the letter "e". This fact can be used to control the number of intrinsic horizontal and / or vertical distances within a coding block. Taking these considerations into account, some embodiments of Method 100 include performing a spectral sparsification operation that transforms the aperture of similar graphic elements with respect to the variation between the three graphic elements in Figure 20.

[0061] Figure 21 is described above.

[0062] As numerous examples have demonstrated, the inventors propose a method for arranging graphic elements constituting a given text string in relation to an encoded block pattern in a manner that facilitates efficient transform encoding. More specifically, the inventors have developed a toolbox of operations for changing a provisional layout of graphic elements to a modified layout that has a sparser spectrum and can therefore be represented more compactly. These operations leave the text string intact and thus preserve its communicative importance, and are generally not visually noticeable to non-expert viewers. The discreteness of the spectral sparsification operations can be further ensured by implementing one of the embodiments in which layout constraints are accepted and observed.

[0063] The aspects of this disclosure are primarily described above with reference to several embodiments. However, as will be readily apparent to those skilled in the art, other embodiments not disclosed above are equally possible within the scope of the invention, as defined by the appended claims. For example, while the examples primarily relate to characters from the Latin alphabet, those skilled in the art will understand that the techniques disclosed herein are readily applicable to other scripts such as Greek, Cyrillic, Arabic, and ideographic scripts.

Claims

1. A method for rendering text strings in a digital raster image suitable for block-unit transformation encoding, Obtaining partitions of image regions into encoding blocks for the aforementioned block-unit transformation encoding, Representing the text string as a plurality of graphic elements from a typeface arranged according to a provisional layout within the image area, wherein the provisional layout defines at least the position, orientation, and size of each of the plurality of graphic elements, and representing the text string. The provisional layout is modified by applying a spectral sparsification operation to at least one non-empty coding block, thereby obtaining the modified layout. Rendering digital raster images of the plurality of graphic elements arranged according to the modified layout. Includes, The spectral sparsification operation described above is: Rotating or tilting graphic elements reduces the number of unique line orientations within a coding block. Reducing the number of intrinsic vertical or horizontal distances within a coding block by isotropic or anisotropic rescaling of the entire graphic element, or by isotropic or anisotropic rescaling of a portion of the graphic element. By translating graphic elements, the number of unique line orientations and / or unique vertical or horizontal distances within a coding block is reduced, and To reduce the number of inherent line orientations within a coding block, align line orientations with vertical, horizontal, or diagonal axes, and / or reduce the number of inherent vertical or horizontal distances within a coding block by modifying or replacing the typeface of graphic elements. Including one or more of the following: method.

2. The method according to claim 1, wherein the rotation or tilt of the graphic element aligns the orientation of the line with the vertical axis or the horizontal axis.

3. The method according to claim 1, wherein the translation groups geometrically similar graphic elements together and / or separates geometrically dissimilar graphic elements for each coding block.

4. The method according to claim 3, wherein the translation groups spectrally similar graphic elements and / or separates spectrally dissimilar graphic elements for each encoded block based on at least one cached spectrum from a previous block-unit transform encoding of the graphic elements.

5. The method according to claim 1, wherein the translation rearranges graphic elements within a single coding block.

6. The method according to claim 1, wherein the translation reduces the number of coding blocks occupied by the graphic element.

7. The method according to claim 1, wherein the typeface modification or typeface replacement simplifies the geometric shape of the graphic element.

8. The method according to claim 1, wherein the typeface modification or typeface replacement includes one or more of the following: simplifying the terminal shape of a graphic element; removing ears or serifs on a graphic element; equalizing the width of strokes on a graphic element; equalizing the width of multiple strokes on a graphic element; equalizing the typeface weight in an encoding block; and equalizing the opening of one or more graphic elements.

9. A device for rendering text strings in a digital raster image suitable for block-unit transformation coding, comprising a processor, and configured to perform the method described in claim 1.

10. A non-temporary computer storage medium for storing instructions for performing the method according to claim 1, for rendering text strings in a digital raster image suitable for block-unit transformation encoding when executed on a device having processing power.

Images