Text-based collaborative rendering systems, methods, computer devices, media, and program products

Abstract

This disclosure provides a collaborative rendering system, method, computer device, medium, and program product based on text attributes, relating to the field of computer technology, particularly to image processing, video encoding and decoding technology, etc. The implementation scheme includes: a text attribute detection unit configured to detect text attribute information in an image to be transmitted; a text pixel segmentation unit configured to remove pixels of the text to obtain a background image; an image restoration unit configured to fill the removed text pixels to generate a restored background image; a first data encoding unit configured to encode the restored background image and text attribute information respectively to generate an image bitstream and a text attribute data stream; a first data decoding unit configured to perform decoding operations to obtain the restored background image and text attribute information; and a first image rendering unit configured to render the restored background image based on the text attribute information.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the fields of image processing and video encoding / decoding technology, and especially to a text-based collaborative rendering system, method, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of cloud computing technology, cloud rendering services are becoming increasingly popular. This type of service typically executes the rendering logic of the application interface in the cloud, and then transmits the rendered image to the client as a video stream for display.

[0003] However, in weak network environments, high latency, or low bitrate networks, mainstream video encoders are prone to compression corruption when compressing high-frequency information such as text on the interface, often resulting in blurred outlines and poor readability of text displayed on the client interface. Although existing optical character recognition (OCR) technology can extract text content, it cannot recognize information such as font, font size, and layout, causing the terminal to be unable to accurately reproduce the visual effect of the original text. Therefore, how to ensure clear display of text on the client interface under limited bandwidth has become a pressing technical challenge in the field of cloud rendering. Summary of the Invention

[0004] This disclosure provides a text-based collaborative rendering system, method, computer device, computer-readable storage medium, and computer program product.

[0005] According to one aspect of this disclosure, a text attribute-based collaborative rendering system is provided, comprising: a text attribute detection unit configured to detect text attribute information of text in an image to be transmitted based on a neural network for text detection, wherein the text attribute information includes first information associated with the position and pose of the text in the image to be transmitted, second information associated with the layout of the text in the image to be transmitted, and third information associated with the font appearance of the text; a text pixel segmentation unit configured to remove pixels of text from the image to be transmitted based on the text attribute information to obtain a background image; an image inpainting unit configured to fill the pixels of the removed text in the background image to generate an inpainted background image; and a first data encoding unit configured to perform a first data encoding on the inpainted background image. An encoding operation is performed to generate an image bitstream, and a second encoding operation is performed on text attribute information to generate a text attribute data stream. In the first encoding operation, the timestamp information of the repaired background image is added to the image bitstream, and in the second encoding operation, the timestamp information of the text attribute information is added to the text attribute data stream. A first data decoding unit is configured to perform a first decoding operation on the image bitstream to obtain the repaired background image, and simultaneously perform a second decoding operation on the text attribute data stream to obtain text attribute information. The repaired background image and the text attribute information each have their own timestamp information. A first image rendering unit is configured to render the repaired background image based on the timestamp information and according to the text attribute information at the same timestamp as the repaired background image.

[0006] In some embodiments, the text pixel segmentation unit includes: a text positioning unit configured to determine a text region corresponding to text in an image to be transmitted based on first information in the text attribute information; a text removal unit configured to perform binarization processing on the text region to form a cutout region based on the contrast between the text line portion in the text region and the background portion of the image to be transmitted; and an edge replacement unit configured to replace a first pixel constituting the edge contour of the text located in the cutout region with a second pixel, wherein the second pixel includes pixels of the background portion adjacent to the edge contour of the text, and the range of the region corresponding to the second pixel is greater than or equal to the range of the region corresponding to the first pixel.

[0007] In some embodiments, the image restoration unit includes: a pixel selection unit configured to select a pixel located in the edge contour of the cutout region as a target pixel; and a pixel filling unit configured to fill the target pixel based on pixels located in the background portion adjacent to the target pixel.

[0008] In some embodiments, the first data encoding unit includes: an information encoding unit configured to perform a first sub-encoding operation on first information to generate a first information data stream, perform a second sub-encoding operation on second information to generate a second information data stream, and perform a third sub-encoding operation on third information to generate a third information data stream; and a data stream fusion unit configured to fuse the first information data stream, the second information data stream, and the third information data stream to generate a text attribute data stream.

[0009] In some embodiments, the first image rendering unit includes: a vector text generation unit configured to generate parameterized vector text based on text attribute information obtained via a second decoding operation; a text bitmap generation unit configured to rasterize the parameterized vector text to generate a bitmap for restoring the text; and an image overlay unit configured to overlay the bitmap onto the restored background image.

[0010] In some embodiments, the text attribute-based collaborative rendering system is applied to a server and a client, wherein the text attribute-based collaborative rendering system further includes: a first data sending unit configured to send an image bitstream and a text attribute data stream from the server; and a first data receiving unit configured to receive an image bitstream and a text attribute data stream from the client.

[0011] In some embodiments, the image to be transmitted is a frame in a video. The text attribute-based collaborative rendering system further includes: a change determination unit configured to compare the text attribute information of the text of the current image to be transmitted in the video with the text attribute information of the text of the previous frame to determine whether the text attribute information of the text of the image to be transmitted has changed; a second data encoding unit configured to perform a first encoding operation on the repaired background image to generate an image bitstream in response to determining that the text attribute information has not changed; a second data sending unit configured to send the image bitstream from the server; a second data receiving unit configured to receive the image bitstream from the client; a second data decoding unit configured to perform a first decoding operation on the image bitstream to obtain the repaired background image; and a second image rendering unit configured to render the repaired background image according to the text attribute information of the text of the previous frame.

[0012] According to one aspect of this disclosure, a text attribute-based collaborative rendering method is provided, comprising: detecting text attribute information of text in an image to be transmitted based on a neural network for text detection, wherein the text attribute information includes first information associated with the position and pose of the text in the image to be transmitted, second information associated with the layout of the text in the image to be transmitted, and third information associated with the font appearance of the text; removing pixels of the text from the image to be transmitted based on the text attribute information to obtain a background image; filling the removed text pixels in the background image to generate a repaired background image; and performing a first encoding operation on the repaired background image to generate an image bitstream. Simultaneously, a second encoding operation is performed on the text attribute information to generate a text attribute data stream. In the first encoding operation, the timestamp information of the repaired background image is added to the image bitstream. In the second encoding operation, the timestamp information of the text attribute information is added to the text attribute data stream. A first decoding operation is performed on the image bitstream to obtain the repaired background image. At the same time, a second decoding operation is performed on the text attribute data stream to obtain the text attribute information. The repaired background image and the text attribute information each have their own timestamp information. Based on the timestamp information, the repaired background image is rendered according to the text attribute information that is at the same timestamp as the repaired background image.

[0013] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the methods provided above in this disclosure.

[0014] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods described above.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above.

[0016] According to one or more embodiments of this disclosure, it is possible to ensure that the client clearly displays text content even when network transmission bandwidth is limited.

[0017] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 This is a schematic diagram illustrating the structure of a text attribute-based collaborative rendering system according to an exemplary embodiment.

[0020] Figure 2 This is a schematic diagram illustrating the structure of a text pixel segmentation unit according to an exemplary embodiment of the present disclosure.

[0021] Figure 3 This is a schematic diagram illustrating text removal according to an exemplary embodiment.

[0022] Figure 4 This is a schematic diagram illustrating edge replacement according to an exemplary embodiment.

[0023] Figure 5 This is a structural block diagram illustrating an image restoration unit according to an exemplary embodiment of the present disclosure.

[0024] Figure 6 This is a schematic diagram illustrating the structure of a first data encoding unit according to an exemplary embodiment of the present disclosure.

[0025] Figure 7 This is a block diagram illustrating the structure of a first image rendering unit according to an exemplary embodiment of the present disclosure.

[0026] Figure 8 This is a schematic diagram illustrating image rendering according to an exemplary embodiment.

[0027] Figure 9 This is a block diagram illustrating the structure of a collaborative rendering system according to another exemplary embodiment of the present disclosure.

[0028] Figure 10 This is a block diagram illustrating the structure of a collaborative rendering system according to another exemplary embodiment of the present disclosure.

[0029] Figure 11 This is a flowchart illustrating a text attribute-based collaborative rendering method according to an exemplary embodiment.

[0030] Figure 12 An example configuration of a computer device that can be used to implement the methods described herein is shown. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0033] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0034] In related technologies, during image or video transmission, text is typically encoded in pixel form along with the background image and transmitted over a network to the terminal for decoding and display.

[0035] First, during transmission, in order to ensure real-time transmission, video encoding technology usually compresses the image. However, text information in the user interface is a high-frequency image signal with high edge contrast. When the network environment is unstable or bandwidth resources are limited, details are easily lost, resulting in blurry, jagged, or artifact-like text displayed on the terminal, which seriously affects the user's visual experience.

[0036] Secondly, as text is a typical vector graphic information, although existing optical character recognition technology can extract text content, it cannot simultaneously and accurately identify information such as font, font size, color, and layout. As a result, the terminal cannot restore the visual effect consistent with the original text based on the extracted results, making it difficult to meet users' demand for high-definition images.

[0037] To address this, embodiments of this disclosure propose a more effective collaborative rendering scheme based on text attributes. By extracting the attribute information of the text in the image on the server side, encoding it and the repaired image separately, and transmitting them to the client; after decoding on the client side, vector text is generated based on the text attribute information and rendered on the background image, thus achieving accurate restoration of the text content.

[0038] Therefore, the method of this disclosure, through the collaborative processing mechanism of server-side extraction of text attributes and client-side rendering of text onto a background image based on text attribute parameters, can ensure that text content can still be accurately displayed on the client interface even when network transmission bandwidth is limited.

[0039] Figure 1 This is a schematic diagram illustrating the structure of a text attribute-based collaborative rendering system according to an exemplary embodiment.

[0040] like Figure 1 As shown, the collaborative rendering system includes a text attribute detection unit, a text pixel segmentation unit, an image restoration unit, a first data encoding unit, a first data decoding unit, and a first image rendering unit.

[0041] like Figure 1 As shown, the image to be transmitted can be input into the text attribute detection unit to extract text attribute information. Simultaneously, the text portion of the image to be transmitted is removed by the text pixel segmentation unit to generate a background image without text. The background image is then repaired by the image restoration unit to generate a restored background image. The extracted text attribute information and the restored image are then encoded by the first data encoding unit to generate an image bitstream and a text attribute data stream. The encoded data is then decoded in the first data decoding unit to generate the restored background image and text attribute information. Finally, the restored image is rendered in the first image rendering unit based on the text attribute information to generate a rendered image, thereby accurately restoring the original image to be transmitted.

[0042] The text attribute detection unit is configured to detect text attribute information of text in the image to be transmitted based on a neural network for text detection, wherein the text attribute information includes first information associated with the position and pose of the text in the image to be transmitted, second information associated with the layout of the text in the image to be transmitted, and third information associated with the font appearance of the text.

[0043] In the example, the image to be transmitted can refer to a frame in a video stream generated on the server and sent to the client over the network. This image can include text and a background image. First, the image to be transmitted can be input into a text detection neural network. This network can be based on a lightweight optical character recognition (OCR) architecture to recognize text content and attributes. During training, training data can be generated from fully automated synthetic data, rendering random text on a random background image. This text covers various font libraries, sizes, rotations, perspective distortions, affine distortions, and spacing attributes. The goal is to enable the neural network not only to recognize text content but also to detect various attribute information of the text.

[0044] In the example, text attribute information includes not only the text content but also a comprehensive description of the text's form within the image, which is divided into three categories. The first category refers to data describing the precise position and orientation of the text in the image coordinate system, specifically including vertex coordinates, center point coordinates, and rotation angle. The second category refers to data describing the text's paragraph structure and arrangement rules, including text orientation, character spacing, line spacing, and alignment (left alignment, centering, etc.), describing the text's layout. The third category refers to data describing the text's style, including font name, font size, color value, and whether it includes shadows or strokes, describing the text's appearance. By combining these three categories of information, a structured description of the text's overall state within the image can be achieved.

[0045] In this example, the text attribute information is essentially parametric information extracted from the original image pixels using a neural network. The text is not encoded as pixels during transmission; instead, it is converted into parametric information describing its position, orientation, layout, and font appearance features, allowing the text to exist independently of the original image pixels. Therefore, regardless of image degradation during transmission, the text can be transmitted completely as parametric text attribute information. At the receiving end, the text is rendered and presented based on this text attribute information, thus avoiding text blurring caused by image compression in weak network, high latency, or low bitrate environments.

[0046] In the example, the image to be transmitted can be input into a pre-trained text detection neural network, which analyzes the pixel distribution and features of the image layer by layer. By analyzing the text regions in the image, the position and pose data of the text in the image can be determined, thus outputting the first piece of information; by analyzing the relative relationships of the characters in the text, the second piece of information reflecting the text layout can be output; and by recognizing the font features of the text, the third piece of information describing the appearance of the text can be output. Therefore, splitting the text attribute information into these three different information types can ensure that the original text can be accurately reproduced during subsequent client-side rendering.

[0047] The text pixel segmentation unit is configured to remove text pixels from the image to be transmitted based on text attribute information to obtain a background image.

[0048] In the example, the text region in the image to be transmitted can be located based on the information describing the position and orientation of the text in the text attribute information. Within these specific regions, the pixels of each character in the text can be identified, and these pixels can be removed from the image to be transmitted, thus obtaining a background image with the text removed. In this way, the image to be transmitted can be split into a background image without text and text attribute information, which facilitates the separate processing of the background image and text attribute information in subsequent steps.

[0049] The image inpainting unit is configured to fill the pixels of the removed text in the background image to generate a repaired background image.

[0050] In the example, after processing by the text pixel segmentation unit, a background image without text is obtained. This background image contains missing areas due to text removal, which need to be filled to visually match the surrounding background area, thus obtaining the repaired background image. By filling in the removed pixels, blank areas in the background image can be avoided, thereby ensuring the integrity of the background image.

[0051] The first data encoding unit is configured to perform a first encoding operation on the repaired background image to generate an image bitstream, and simultaneously perform a second encoding operation on the text attribute information to generate a text attribute data stream. In the first encoding operation, the timestamp information of the repaired background image is added to the image bitstream, and in the second encoding operation, the timestamp information of the text attribute information is added to the text attribute data stream.

[0052] In the example, the first and second encoding operations can be encoding processes performed on different data types. For the repaired background image, it can be encoded using video coding standards (such as H.264 and H.265) to generate an image bitstream that conforms to the video transmission protocol; for text attribute data, it can be encoded into data packets of a specific format, i.e., a text attribute data stream, for easy subsequent transmission.

[0053] In the example, due to the different encoding algorithm complexities of the image bitstream and the text attribute datastream, and the potential buffering delay during network transmission, the background image and text may become out of sync when displayed on the client interface (for example, the background image has switched to the next frame, but the text content remains in the previous frame). Therefore, during the encoding operation, it is necessary to add the timestamp information of the background image and text attribute information to the corresponding image bitstream and text attribute datastream respectively. This will ensure that the background image and text attribute information maintain a correspondence in the time dimension and prevent the text from being misaligned with the background image.

[0054] The first data decoding unit is configured to perform a first decoding operation on the image bitstream to obtain the repaired background image, and simultaneously perform a second decoding operation on the text attribute datastream to obtain text attribute information. The repaired background image and text attribute information each have their own timestamp information.

[0055] In the example, the first and second decoding operations can be decoding processes performed on different data streams. For the image stream, a video decoder (e.g., an H.264 / H.265 decoder) can be used to restore the continuous image stream data into frame-by-frame images, i.e., the repaired background images. For the text attribute data stream, parsing processing can be performed to extract text attribute information. Simultaneously, during the decoding process, the decoding unit can parse the embedded timestamps from the frame headers of the image stream and extract the timestamps corresponding to the images from the packet headers of the text attribute data stream, thus ensuring that each decoded background image and each set of text attribute information corresponds one-to-one in time.

[0056] The first image rendering unit is configured to render the repaired background image based on timestamp information and according to the text attribute information that is at the same timestamp as the repaired background image.

[0057] In the example, the timestamp information of the repaired background image can be determined first, and based on this timestamp, the text attribute information at the same timestamp can be determined. Subsequently, since the text attribute information is a comprehensive description of the text's form in the image, rendering processing can be performed on the repaired background image according to the text attribute information, so that the text is drawn and superimposed onto the corresponding background image according to the form described by the text attribute information. By matching and rendering based on timestamp information, the temporal consistency of the rendering process can be ensured, ultimately restoring the original text on the background image.

[0058] Therefore, by separating the text from the image and encoding and transmitting it separately from the background image as text attribute information, blurring or distortion caused by network latency or low bitrate during encoding or transmission can be avoided. By adding timestamps to the background image and text attribute information respectively, and matching and rendering based on the timestamps at the receiving end, the temporal consistency between the text and the background image can be ensured, thereby accurately reproducing the original text on the background image. Simultaneously, this allows the background image to obtain a higher bitrate allocation, improving bandwidth utilization efficiency.

[0059] Figure 2 This is a schematic diagram illustrating the structure of a text pixel segmentation unit according to an exemplary embodiment of the present disclosure.

[0060] In some embodiments, the text pixel segmentation unit may include a text positioning unit, a text removal unit, and an edge replacement unit.

[0061] The text localization unit is configured to determine the text region corresponding to the text in the image to be transmitted based on the first information in the text attribute information.

[0062] The text removal unit is configured to binarize the text region to form a cutout region based on the contrast between the text lines in the text region and the background of the image to be transmitted.

[0063] The edge replacement unit is configured to replace a first pixel of the edge contour of the text located in the cutout area with a second pixel, wherein the second pixel includes pixels of the background portion adjacent to the edge contour of the text, and the range of the region corresponding to the second pixel is greater than or equal to the range of the region corresponding to the first pixel.

[0064] In the example, the first piece of information in the text attribute provides data such as the vertex coordinates, center position, and rotation angle of the text box. Therefore, the position of the text region in the image to be transmitted can be determined based on this first piece of information. Subsequently, when removing text, operations can be performed only on the text region, thus accurately removing text pixels without damaging the background pixels of the image.

[0065] In the example, when performing text removal, the text area can be binarized. That is, by analyzing the luminance or chromaticity difference between the pixels of the text strokes and the surrounding background pixels, the pixels constituting the text strokes can be identified, and then these pixels can be cleared from the image and become transparent, thus leaving a blank hollowed-out area on the original image.

[0066] In the example, after binarization, when processing the transitional pixels located at the intersection of the text edge contour and the background part, these pixels are often not purely text color nor purely background color, but a mixed color of both. Therefore, these pixels will remain on the background image, that is, the first pixels. To process these first pixels remaining on the edge contour, an algorithm operation similar to morphological dilation can be adopted. Select the background pixels adjacent to the text edge contour, that is, the second pixels, and replace them with the first pixels. Since the remaining first pixels need to be completely covered, in this process, the area corresponding to the second pixels must be greater than or equal to the area corresponding to the remaining first pixels in terms of geometric area.

[0067] Therefore, by performing text removal and processing the pixels of the text edge contour, it is possible to avoid jagged marks at the edge of the hollowed-out area, which helps to make the background image more continuous and natural visually, thus providing a basis for subsequent processing of the background image; at the same time, the area corresponding to the cleared text can be reduced in space, enabling the subsequently rendered and restored text to completely cover the original cleared text area.

[0068] Figure 3 FIG. shows a schematic diagram of text removal according to an exemplary embodiment.

[0069] As Figure 3 shown, Figure 3 in the left side shows the text "one", and the slanted part is the image background part. Based on the contrast between the stroke part of the text "one" and the image background part, the pixels of the text "one" can be removed to obtain Figure 3 the image on the right side in FIG., that is, the background image after removing the text. The gray area in this background image represents the hollowed-out area formed after the removal operation.

[0070] Figure 4 FIG. shows a schematic diagram of edge replacement according to an exemplary embodiment.

[0071] As Figure 4 shown, still taking the text "one" in Figure 3 as an example, after the text removal operation, as Figure 4As shown on the left side of [Figure], the small squares in the figure represent pixels. There will be some pixels of the text line part remaining at the position of the edge contour of the text. For example, pixel 410. Therefore, an algorithm operation similar to morphological dilation can be adopted. Select pixel 420 in the background image adjacent to pixel 410 and replace it with pixel 410. The same replacement operation is also performed on the remaining pixels of the edge contour of the text "one". Finally, a background image without text can be obtained, that is, as Figure 4 shown on the right side of [Figure].

[0072] It can be understood that Figure 4 only a small number of pixels are taken as examples for illustrative purposes. However, the embodiments of the present disclosure are not limited thereto, and the number of pixels in the hollowed-out area obtained after the segmentation operation is not limited.

[0073] Figure 5 is a structural block diagram showing an image restoration unit according to an exemplary embodiment of the present disclosure.

[0074] In some embodiments, the image restoration unit may include a pixel selection unit and a pixel filling unit.

[0075] The pixel selection unit is configured to select pixels located in the edge contour of the hollowed-out area as target pixels.

[0076] The pixel filling unit is configured to fill the target pixels based on the pixels located in the background part adjacent to the target pixels.

[0077] In the example, image restoration means that the pixels in the hollowed-out area need to be filled so that the background image is visually coherent and complete. The target pixels can refer to the blank pixels that need to be filled and processed within the edge contour of the hollowed-out area. When filling, for the selected target pixels, the pixels located in the background part adjacent to them in the spatial position can be obtained, and the target pixels can be filled and processed based on the pixel characteristics of this part of the background pixels. Such filling and processing can be to copy the pixel values of the background pixels to the target pixels, or to generate pixel values for replacing the target pixels according to the pixel values of multiple adjacent background pixels, so that the display effect of the target pixels is coordinated with the surrounding background.

[0078] Therefore, by adopting this filling strategy based on adjacent background pixels, it can be ensured that the generated filling content is more natural and coordinated in visual texture.

[0079] Figure 6 is a schematic structural diagram showing a first data encoding unit according to an exemplary embodiment of the present disclosure.

[0080] In some embodiments, the first data encoding unit may include an information encoding unit and a data stream fusion unit.

[0081] The information encoding unit is configured to perform a first sub-encoding operation on first information to generate a first information data stream, perform a second sub-encoding operation on second information to generate a second information data stream, and perform a third sub-encoding operation on third information to generate a third information data stream.

[0082] The data stream fusion unit is configured to fuse the first information data stream, the second information data stream, and the third information data stream to generate a text attribute data stream.

[0083] In the example, since the text attribute information contains different types of information, corresponding encoding strategies can be applied to these three different information types. For example, for the first information, which represents the text position and posture with continuously changing values, differential quantization encoding can be performed to record the relative displacement and generate the first information data stream; for the second information, which represents the text layout, compact structured encoding can be used to generate the second information data stream; and for the third information, which represents the font appearance with a fixed format, a style index table can be constructed for dictionary encoding.

[0084] In the example, after completing their respective sub-encoding operations, these three information data streams can be merged, for example, packaged and compressed according to a preset format, and integrated into a unified text attribute data stream.

[0085] Therefore, by encoding different types of text attribute information separately and then fusing them later, we can improve encoding efficiency and avoid interference between different types of information during the encoding process.

[0086] Figure 7 This is a block diagram illustrating the structure of a first image rendering unit according to an exemplary embodiment of the present disclosure.

[0087] In some embodiments, the first image rendering unit may include a vector text generation unit, a text bitmap generation unit, and an image overlay unit.

[0088] The vector text generation unit is configured to generate parameterized vector text based on text attribute information obtained through a second decoding operation.

[0089] The text bitmap generation unit is configured to rasterize parameterized vector text to generate a bitmap for reconstructing the text.

[0090] The image overlay unit is configured to overlay a bitmap onto the repaired background image.

[0091] In the example, after the decoding operation, text attribute information for comprehensively describing the form of the text in the image can be obtained. Therefore, based on the text attribute information, parametric modeling of the structure and appearance of the text can be performed, and the text attribute information can be converted into a set of vectorized text, which describes the geometric contour and overall form of the text in parametric form.

[0092] In the example, the rasterization operation may refer to converting text in vector form into a bitmap image composed of discrete pixels. Therefore, the rasterization operation can be performed on the parameterized vector text to convert the vector text into a corresponding bitmap, that is, generating a bitmap for restoring the text. Finally, according to the position of the text in the background image indicated in the text attribute information, this text bitmap is overlaid on the background image that has been decoded and repaired, so as to achieve the effect of restoring the original image to be transmitted.

[0093] Therefore, by first generating parameterized vector text, then rasterizing it and overlaying it on the repaired background image, the text can be restored to the background image in a reconstructed manner, thus avoiding damage to text details caused by video stream compression transmission.

[0094] Figure 8 FIG. is a schematic diagram showing image rendering according to an exemplary embodiment.

[0095] As Figure 8 shown, still taking the text "one" and the background image in Figure 3 and Figure 4 as an example, after the decoding operation, text attribute information for comprehensively describing the form of the text "one" in the image can be obtained. The text attribute information is converted into a set of vectorized text and rasterized into a bitmap 810. According to the position of the text "one" in the background image indicated in the text attribute information, the bitmap 810 is overlaid on the repaired background image 820 and rendered into an image 830, so as to achieve the effect of restoring the original image to be transmitted.

[0096] Figure 9 FIG. is a block diagram showing the structure of a collaborative rendering system according to another exemplary embodiment of the present disclosure.

[0097] In some embodiments, the collaborative rendering system is applied to the server side and the client side, and the collaborative rendering system may further include a first data sending unit and a first data receiving unit.

[0098] The first data sending unit is configured to send an image code stream and a text attribute data stream from the server side.

[0099] The first data receiving unit is configured to receive an image code stream and a text attribute data stream from the client side.

[0100] In this example, the collaborative rendering system can refer to a system capable of collaboratively completing data transmission and processing between the server and client. On the server side, a text attribute detection unit, a text pixel segmentation unit, an image restoration unit, and a first data encoding unit can be configured to extract text attribute information, segment the text and background image, restore the background image, and encode the background image and text attribute information. On the client side, a first data decoding unit and a first image rendering unit can be configured to decode the data and render the background image based on the text attribute information.

[0101] In this example, since data transmission is required between the server and the client, the collaborative rendering system may also include a first data sending unit and a first data receiving unit. The first data sending unit can be configured on the server side to send the encoded image bitstream and text attribute data stream to the client; the first data receiving unit can be configured on the client side to receive the image bitstream and text attribute data stream sent from the server side.

[0102] Therefore, by sending image bitstreams and text attribute data streams on the server side and having them received by the client, image data and text attribute data can be transmitted in independent data formats over the network, thereby improving the stability of the data transmission process.

[0103] Figure 10 This is a block diagram illustrating the structure of a collaborative rendering system according to another exemplary embodiment of the present disclosure.

[0104] In some embodiments, the image to be transmitted is a frame in a video, and the collaborative rendering system may further include: a change determination unit configured to compare the text attribute information of the text of the current image to be transmitted in the video with the text attribute information of the text of the previous frame image to determine whether the text attribute information of the text of the image to be transmitted has changed; a second data encoding unit configured to perform a first encoding operation on the repaired background image to generate an image bitstream in response to determining that the text attribute information has not changed; a second data sending unit configured to send the image bitstream from the server; a second data receiving unit configured to receive the image bitstream from the client; a second data decoding unit configured to perform a first decoding operation on the image bitstream to obtain the repaired background image; and a second image rendering unit configured to render the repaired background image according to the text attribute information of the text of the previous frame image.

[0105] In the example, during video transmission, the text portion of the video may remain unchanged, with only the background image changing. Encoding and transmitting the text attribute information for each frame would increase the transmission bitrate. Therefore, to improve transmission efficiency, the change determination unit can compare the text attribute information of the current image to be transmitted with that of the previous frame. If the comparison results show a match, it can be determined that the text portion has not changed. At this point, the second data encoding unit will only perform the first encoding operation on the repaired background image to generate an image stream, without encoding the text attribute information of the current image to be transmitted. Subsequently, the second data sending unit sends the image stream from the server to the client.

[0106] In the example, the second data receiving unit receives the image bitstream sent from the server, and the second data decoding unit decodes it to obtain the repaired background image. During the rendering operation of the second image rendering unit, if it detects that there is no corresponding text attribute information in the currently decoded data, it directly calls the text attribute information of the text from the previously cached image. Subsequently, parameterized vector text is generated based on this text attribute information, and rasterization is performed to generate a bitmap. Finally, according to the position of the text in the background image indicated by the text attribute information, the bitmap is overlaid on the repaired background image.

[0107] Therefore, in video transmission, by performing text change detection on the image to be transmitted, if the text portion remains unchanged, the text attribute information of that text will not be transmitted, which can reduce the transmission bitrate and thus improve transmission efficiency.

[0108] Figure 11 This is a flowchart illustrating a text attribute-based collaborative rendering method 1100 according to an exemplary embodiment.

[0109] like Figure 11 As shown, method 1100 includes steps S1101, S1102, S1103, S1104, S1105 and S1106.

[0110] In step S1101, text attribute information of the text in the image to be transmitted is detected based on a neural network for text detection. The text attribute information includes first information associated with the position and orientation of the text in the image to be transmitted, second information associated with the layout of the text in the image to be transmitted, and third information associated with the font appearance of the text.

[0111] In step S1102, based on the text attribute information, the pixels of the text are removed from the image to be transmitted to obtain the background image.

[0112] In step S1103, the pixels of the removed text in the background image are filled to generate the repaired background image.

[0113] In step S1104, a first encoding operation is performed on the repaired background image to generate an image bitstream, and a second encoding operation is performed on the text attribute information to generate a text attribute data stream. In the first encoding operation, the timestamp information of the repaired background image is added to the image bitstream, and in the second encoding operation, the timestamp information of the text attribute information is added to the text attribute data stream.

[0114] In step S1105, a first decoding operation is performed on the image bitstream to obtain the repaired background image, and a second decoding operation is performed on the text attribute data stream to obtain text attribute information. The repaired background image and text attribute information each have their own timestamp information.

[0115] In step S1106, the repaired background image is rendered based on the timestamp information and the text attribute information that is at the same timestamp as the repaired background image.

[0116] According to the text attribute-based collaborative rendering method of embodiments of this disclosure, text and background images can be processed separately, so that text is no longer transmitted in pixel form, but is independently encoded and transmitted in the form of text attribute information. By rendering vector text based on text attribute information on the client side, the text area can be unaffected by compression loss, maintaining high readability under bandwidth constraints; at the same time, the background image can obtain a higher bitrate allocation, thereby improving bandwidth utilization efficiency.

[0117] It should be understood that Figure 11 Steps S1101, S1102, S1103, S1104, S1105, and S1106 shown in the reference can be used with reference to Figure 1 The text attribute detection unit, text pixel segmentation unit, image inpainting unit, first data encoding unit, first data decoding unit, and first image rendering unit in the described collaborative rendering system correspond to each other. Therefore, the operations, features, and advantages described above for the collaborative rendering system also apply to method 1100. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0118] While specific functions have been discussed above with reference to specific units, it should be noted that the functions of the units discussed herein may be divided into multiple units, and / or at least some functions of multiple units may be combined into a single unit. The specific unit performing an action discussed herein includes the specific unit itself performing the action, or alternatively, the specific unit calling or otherwise accessing another component or unit that performs the action (or performs the action in conjunction with the specific unit). Therefore, a specific unit performing an action may include the specific unit performing the action itself and / or another unit that performs the action, called or otherwise accessed by the specific unit.

[0119] It should also be understood that this document can describe various techniques within the general context of software or program units. The above regarding... Figure 1 The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuit. For example, these units can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0120] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0121] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0122] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0123] In the following text, combined with Figure 12 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0124] Figure 12 An example configuration of a computer device 1200 that can be used to implement the methods described herein is shown.

[0125] Computer device 1200 can be a variety of different types of devices. Examples of computer device 1200 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0126] Computer device 1200 may include at least one processor 1202, memory 1204, communication interfaces(s) 1206, display device 1208, other I / O devices 1210, and one or more mass storage devices 1212, which are capable of communicating with each other, such as via system bus 1214 or other suitable connections.

[0127] Processor 1202 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 1202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 1202 may be configured to acquire and execute computer-readable instructions stored in memory 1204, mass storage device 1212, or other computer-readable media, such as program code of operating system 1216, program code of application program 1218, program code of other program 1220, etc.

[0128] Memory 1204 and mass storage device 1212 are examples of computer-readable storage media for storing instructions that are executed by processor 1202 to perform the various functions described above. For example, memory 1204 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 1212 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 1204 and mass storage device 1212 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by processor 1202 as a specific machine configured to perform the operations and functions described in the examples herein.

[0129] Multiple programs may be stored on mass storage device 1212. These programs include operating system 1216, one or more application programs 1218, other programs 1220, and program data 1222, and they may be loaded into memory 1204 for execution. Examples of such application programs or program units may include computer program logic (e.g., computer program code or instructions) of the methods described herein and / or other embodiments described herein.

[0130] Although Figure 12 The data is illustrated as being stored in memory 1204 of computer device 1200, but the operating system 1216, application program 1218, other programs 1220, and program data 1222, or portions thereof, may be implemented using any form of computer-readable medium accessible by computer device 1200. As used herein, "computer-readable medium" includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0131] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program units, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program units, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0132] One or more communication interfaces 1206 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces include near field communication (NFC) interfaces. Communication interface 1206 facilitates communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 1206 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0133] In some examples, a display device 1208, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 1210 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0134] The techniques described herein can be supported by these various configurations of computer device 1200 and are not limited to specific examples of the techniques described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from computer device 1200. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 1200 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computer device 1200 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. A collaborative rendering system based on text attributes, characterized in that, The system includes: The text attribute detection unit is configured to detect text attribute information of text in an image to be transmitted based on a neural network for text detection, wherein the text attribute information includes first information associated with the position and orientation of the text in the image to be transmitted, second information associated with the layout of the text in the image to be transmitted, and third information associated with the font appearance of the text. A text pixel segmentation unit is configured to remove pixels of the text from the image to be transmitted based on the text attribute information to obtain a background image; An image restoration unit is configured to fill the pixels of the removed text in the background image to generate a restored background image; The first data encoding unit is configured to perform a first encoding operation on the repaired background image to generate an image bitstream, and simultaneously perform a second encoding operation on the text attribute information to generate a text attribute data stream. In the first encoding operation, the timestamp information of the repaired background image is added to the image bitstream, and in the second encoding operation, the timestamp information of the text attribute information is added to the text attribute data stream. A first data decoding unit is configured to perform a first decoding operation on the image bitstream to obtain the repaired background image, and simultaneously perform a second decoding operation on the text attribute datastream to obtain the text attribute information. The repaired background image and the text attribute information each carry their respective timestamp information. The first image rendering unit is configured to render the repaired background image based on the timestamp information and the text attribute information that is at the same timestamp as the repaired background image.

2. The system according to claim 1, characterized in that, The text pixel segmentation unit includes: The text localization unit is configured to determine the text region corresponding to the text in the image to be transmitted based on the first information in the text attribute information. A text removal unit is configured to binarize the text region to form a cutout region based on the contrast between the text line portion of the text region and the background portion of the image to be transmitted; and An edge replacement unit is configured to replace a first pixel constituting the edge contour of the text located in the cutout area with a second pixel, wherein the second pixel includes pixels of the background portion adjacent to the edge contour of the text, and the range of the region corresponding to the second pixel is greater than or equal to the range of the region corresponding to the first pixel.

3. The system according to claim 2, characterized in that, The image restoration unit includes: The pixel selection unit is configured to select pixels located in the edge contour of the cutout region as target pixels; and A pixel filling unit is configured to fill the target pixel based on pixels located in the background portion that are adjacent to the target pixel.

4. The system according to any one of claims 1 to 3, characterized in that, The first data encoding unit includes: An information encoding unit is configured to perform a first sub-encoding operation on the first information to generate a first information data stream, perform a second sub-encoding operation on the second information to generate a second information data stream, and perform a third sub-encoding operation on the third information to generate a third information data stream; and The data stream fusion unit is configured to fuse the first information data stream, the second information data stream, and the third information data stream to generate the text attribute data stream.

5. The system according to any one of claims 1 to 3, characterized in that, The first image rendering unit includes: The vector text generation unit is configured to generate parameterized vector text based on the text attribute information obtained via the second decoding operation; A text bitmap generation unit is configured to rasterize the parameterized vector text to generate a bitmap for reconstructing the text; and An image overlay unit is configured to overlay the bitmap onto the repaired background image.

6. The system according to any one of claims 1 to 3, characterized in that, The system is applied to both the server and client sides, and further includes: A first data sending unit is configured to send the image bitstream and the text attribute datastream from the server; and The first data receiving unit is configured to receive the image bitstream and the text attribute datastream from the client.

7. The system according to claim 1, characterized in that, The image to be transmitted is a frame from a video, and the system further includes: The change determination unit is configured to compare the text attribute information of the text of the current image to be transmitted in the video with the text attribute information of the text of the previous frame image to determine whether the text attribute information of the text of the image to be transmitted has changed. The second data encoding unit is configured to perform the first encoding operation on the repaired background image to generate the image bitstream in response to determining that the text attribute information has not changed. The second data sending unit is configured to send the image bitstream from the server. The second data receiving unit is configured to receive the image bitstream from the client; A second data decoding unit is configured to perform the first decoding operation on the image bitstream to obtain the repaired background image; and The second image rendering unit is configured to render the repaired background image based on the text attribute information of the text in the previous frame image.

8. A collaborative rendering method based on text attributes, characterized in that, The method includes: The text attribute information of the text in the image to be transmitted is detected based on a neural network for text detection, wherein the text attribute information includes first information associated with the position and pose of the text in the image to be transmitted, second information associated with the layout of the text in the image to be transmitted, and third information associated with the font appearance of the text. Based on the text attribute information, the pixels of the text are removed from the image to be transmitted to obtain a background image; The pixels of the removed text in the background image are filled to generate a repaired background image; The repaired background image is subjected to a first encoding operation to generate an image bitstream, and the text attribute information is subjected to a second encoding operation to generate a text attribute data stream. In the first encoding operation, the timestamp information of the repaired background image is added to the image bitstream, and in the second encoding operation, the timestamp information of the text attribute information is added to the text attribute data stream. A first decoding operation is performed on the image bitstream to obtain the repaired background image, and a second decoding operation is performed on the text attribute datastream to obtain the text attribute information. The repaired background image and the text attribute information each carry their respective timestamp information. Based on the timestamp information, the repaired background image is rendered according to the text attribute information that is at the same timestamp as the repaired background image.

9. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of claim 8.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of claim 8.

Images