Real-time conversion of macroblocks to directed distance fields to improve

Abstract

An apparatus and method for performing efficient video transmission are disclosed. In various implementations, a computing system includes a transmitter that transmits a video stream over a network to a receiver. Prior to encoding a video frame, the transmitter identifies a first set of one or more macroblocks of the video frame comprising text. The transmitter replaces pixel color information with pixel distance information for the first set of one or more macroblocks. The transmitter inserts an indication in the metadata information that identifies the first set of one or more macroblocks and specifies color values for pixels in the first set of one or more macroblocks. The transmitter encodes the video frame and transmits it together with the metadata information to the receiver. The receiver uses the metadata information to reproduce the original pixel color and maintain text definition of an image to be depicted on the display device.

Description

Background Art

[0001] Description of Related Technology

[0002] Video processing algorithms are complex and include many different functions. Computing systems use advanced processors to meet high computing requirements. Video processing complexity increases with the increase of display resolution. In addition, in the consumer market, high-definition video encoding applications are growing rapidly. In addition, with the reduction of available data bandwidth and the occurrence of real-time processing, video processing becomes more complex. For example, desktop streaming media services have become very common and include services that allow users to access various contents provided on remote servers in real time through networks (such as the Internet). The content is presented in a virtual computing environment on the user's computing device, such as a laptop computer, a desktop computer, etc. Remote Desktop Services (RDS) provides a replicated view of the working desktop environment that is usually provided on the user's working desktop computer, but is now presented on the user's remote computing device. Remote applications are hosted or run on a virtual machine (VM), but it looks as if these applications are running on the user's remote computing device as local applications.

[0003] A video game (or game) streaming service is another example of a service that provides real-time presentation of content on a user's remote computing device, where the content is updated in real time based on user input. A live streaming service that broadcasts live content of a user performing a demonstration is yet another example of such a real-time service. In order to avoid jumps or glitches in the video content on a monitor or screen of a user's remote computing device, a video encoder increases the video compression ratio before transmitting the video frames to the user's remote computing device over a network.

[0004] Specific areas of the video frame corresponding to the image to be viewed on the screen of the user's remote computing device often include blocks of text. For example, video frames corresponding to applications for remote desktop services often include text. The visual clarity of these specific areas of the video frame (such as specific macroblocks) decreases as the video compression ratio increases. As the compression provided by the video encoder increases, the text in these specific macroblocks may also appear blurry due to the lack of sharp contrast at the edges of the text. If sub-pixel anti-aliasing techniques are used, the resulting compressed pixel values ​​may cause additional color to appear at the edges of the text, which further reduces visual clarity.

[0005] In light of the foregoing, methods and systems for performing efficient video transmission that preserves text clarity are desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is a generalized block diagram of a video processing system.

[0007] Figure 2is a general block diagram of a streaming desktop.

[0008] Figure 3 It is a generalized block diagram of a computing system.

[0009] Figure 4 is a generalized block diagram of a method for performing efficient video transmission while preserving text clarity.

[0010] Figure 5 is a generalized block diagram of a method for supporting efficient video transmission while maintaining text clarity.

[0011] Although the present invention is susceptible to various modifications and alternative forms, specific implementations are shown by way of example in the drawings and described in detail herein. However, it should be understood that the drawings and detailed description thereof are not intended to limit the invention to the specific forms disclosed, but on the contrary, the invention is to cover all modifications, equivalents and alternatives within the scope of the invention defined by the appended claims. DETAILED DESCRIPTION

[0012] In the following description, many specific details are set forth to provide a thorough understanding of the present invention. However, it will be appreciated by those of ordinary skill in the art that the present invention may be practiced without these specific details. In some cases, well-known circuits, structures, and techniques are not shown in detail to avoid obscuring the present invention. In addition, it should be understood that for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, the size of some elements is enlarged relative to other elements.

[0013] Various systems, devices and methods for performing efficient video transmission are disclosed. In various specific implementations, a computing system includes a transmitter that transmits a video stream to a receiver via a network such as the Internet. In a specific implementation, the transmitter is a remote server, and the receiver is a user computing device that communicates with the remote server via the network. The transmitter supports the receiver to perform streaming services, such as desktop streaming services, live streaming services, video game streaming services, etc. When the user executes the streaming service on the receiver, the streaming service provides real-time updates of video content based on input from the user. In some specific implementations, the remote server includes a circuit system of an accelerated processing unit (APU), which includes a plurality of different integrated circuits, each of which provides different functions. For example, the APU includes at least a general processing unit, such as a central processing unit (CPU), wherein a plurality of cores are capable of executing instructions of a general instruction set architecture (ISA). The APU also includes a parallel data processing unit, such as a graphics processing unit (GPU), which includes a circuit system of one or more processor cores having a single instruction multiple data (SIMD) parallel architecture. The APU also includes a video encoder for compressing video frames of images to be depicted on a display device of the user computing device.

[0014] The hardware (such as circuit) of the transmitter receives the rendering source data of the image. For example, based on the command generated by the CPU, the GPU renders the video frame corresponding to the image. The video encoder encodes (i.e., compresses) the rendered video frame by dividing the video frame into macroblocks and generates a compressed bitstream. As used herein, a "bitstream" is a sequence of bits. Before the transmitter transmits the compressed bitstream to the receiver over the network, the video encoder stores the generated (and compressed) bitstream in a buffer. However, before the video encoder compresses the macroblock, one or more CPUs, GPUs, or another integrated circuit in the CPU of the transmitter identifies which one or more macroblocks of the video frame are completely composed of rendered text. These one or more macroblocks provide the first group of one or more macroblocks of the video frame. If such macroblocks do not exist in the video frame, the transmitter processes the video without using the following additional steps and continues to the next video frame. The circuit of the transmitter uses one of a plurality of techniques for detecting the first group of one or more macroblocks. Examples of these techniques are edge-based algorithms in various edge-based algorithms that rely on the calculated edge density or calculated edge gradient of the macroblock, sum the counts of these values ​​in one or more histograms, and analyze one or more histograms for text detection.

[0015] The transmitter replaces the first data representation of the first group of one or more macroblocks with a second data representation different from the first data representation. In a specific implementation, the first data representation includes pixel information, such as one or more color values ​​corresponding to one or more color components. Typically, for black and white images, a single color value corresponding to a single grayscale color component is used, where the color value is a positive integer, ranging from 0 indicating black to 255 indicating white. Therefore, a single byte (8 bits) can be used to represent a pixel. For a color image, three or more color values ​​corresponding to three or more color components. In the example, three color values ​​are used to represent a red component, a green component, and a blue component. Each color component uses bytes, and three bytes represent pixels. However, after replacing the macroblocks identified as the first group of one or more macroblocks in the previous step, the pixels in the first group of macroblocks no longer represent pixel color data. Instead, the pixels in the first group of macroblocks represent the nearest pixel distance to the surface of the identified rendered text.

[0016] In one specific implementation, the distance value in the second data representation of a pixel in a macroblock is a directed distance field (SDF) value for the pixel. The SDF value represents the shortest distance between the corresponding pixel and the glyph closest to the pixel in the macroblock. As used herein, a "glyph" is a graphic symbol used to represent a letter of the alphabet. For example, the alphabet letter "A" has a specific graphic symbol (or glyph) that changes based on a selected font, a selected text size, and selected characteristics (such as italic printing, bold printing, etc.). In another specific implementation, the value in the second data representation of a pixel in a macroblock encodes the coordinates of the nearest point on the glyph surface, as well as a one-dimensional representation of the normal vector of the surface. It is possible and conceivable that various other distance information corresponding to the pixel in the macroblock and the nearest glyph can be used in the second data representation.

[0017] The circuit of the transmitter inserts one or more indications identifying the first group of one or more macroblocks in the metadata information corresponding to the video frame. The transmitter also inserts indications specifying the color values ​​of the pixels in the first group of one or more macroblocks in the metadata information. Generally speaking, rendered text only includes a foreground color (the color of the glyph) and a background color. In this case, only two color values ​​are transmitted for each macroblock. In another specific implementation, additional color values ​​are transmitted together with information describing which areas of the macroblock use which color values ​​to illustrate multi-color text. The video encoder compresses (encodes) the macroblocks of the video frame. Therefore, the transmitter encodes one or more macroblocks including pixel color information, and additionally encodes one or more macroblocks in which the pixel color information is replaced by pixel distance information. However, the video encoder does not receive metadata information or any information identifying the first group of one or more macroblocks. Therefore, the video encoder is unaware of which macroblocks include pixel color information and which macroblocks include pixel distance information.

[0018] Afterwards, the transmitter transmits the metadata information and the encoded macroblocks of the video frame to a receiver with a display device. The receiver includes a decoder for decoding (decompressing) the encoded macroblocks of the video frame. After decoding the macroblocks of the video frame, the receiver converts the pixel distance information into pixel color information. To this end, the receiver uses the metadata information to identify the macroblocks for which the pixel color information is replaced by the pixel distance information. As mentioned above, the metadata information also includes at least a foreground color value and a background color value. Figures 1 to 5 Further details of efficient video transmission while maintaining text clarity are provided in the following description of .

[0019] refer to Figure 1 , a generalized block diagram of a video processing system 100 is shown. The video processing system 100 (or system 100) includes at least a transmitter 110 and a receiver 160, which are capable of communicating with each other via a network 112 with a limited bandwidth connection. The transmitter 110 and the receiver 160 represent any type of communication device and / or computing device. For example, in various specific implementations, the transmitter 110 and / or the receiver 160 are one of a mobile phone (or smart phone), a tablet computer, a desktop computer, a laptop computer, a server computer (or server), a game console connected to a television, or other types of computing devices or communication devices. In various specific implementations, the transmitter 110 and the receiver 160 include a network interface (not shown) that supports one or more communication protocols for data and message transmission over the network 112. The transmitter 110 transmits video information, such as rendering information corresponding to a video frame 140 (or frame 140), to the receiver 160. While frame 140 depicts a picture of a lake overlaid with text data 142, in other examples, frame 140 includes information of various visual information, such as a scene from a sporting event, a scene from a video game, a virtual desktop corresponding to a remote desktop connection, a live broadcast from a particular user, etc.

[0020] In some specific implementations, the transmitter 110 supports the receiver 160 to perform streaming services, such as desktop streaming services, live streaming services, video game streaming services, etc. When the user executes the streaming service on the receiver 160, the streaming service provides real-time updates of video content (such as frame 140) based on the input from the user. The network 140 includes multiple switches, routers, cables, wireless transmitters and the Internet for transmitting messages and data. Therefore, in some specific implementations, the network interface of the transmitter 110 and the receiver 160 supports the hypertext transfer protocol (HTTP) for cross-world wide web communication. In other specific implementations, the network interface of the transmitter 110 and the receiver 160 supports the transmission control protocol (TCP), the user datagram protocol (UDP), etc. It should also be noted that in some specific implementations, one or more wireless links are used to transmit data between the transmitter 110 and the receiver 160. For ease of illustration, clock sources such as a phase-locked loop (PLL), an interrupt controller, a communication structure, a data bus, a power controller, a memory controller, an interface for input / output (I / O) devices, etc. are not shown in the system 100 for each of the transmitter 110 and the receiver 160.

[0021] The circuitry of the transmitter 110 provides any number and type of processors and memory devices. In a specific implementation, the circuitry of the transmitter 110 includes an accelerated processing unit (APU) 120 on a motherboard, which includes a general-purpose central processing unit (CPU) 122, a graphics processing unit (GPU) 124, and a video encoder 126 (or encoder 126). In another specific implementation, the CPU 122 is in a package on the motherboard, and one or more slots (sockets) on the motherboard include a video graphics card with GPU 124 and one or more additional GPUs. In yet other specific implementations, the transmitter 110 includes one or more of an application-specific integrated circuit (ASIC), a field programmable array (FGPA), other types of parallel data processors, etc. The memory 116 uses one or more of various random access memories (RAM), hard disk drives (HDDs), solid-state drives (SSDs), etc. In a similar manner, the circuitry of the receiver 160 provides any number and type of processors and memory devices as described above to implement the processor 170 and the memory 180.

[0022] CPU 122 uses one or more processor cores with circuits for executing instructions according to a predefined general instruction set in various predefined general instruction sets. GPU 124 uses multiple parallel execution channels in a single instruction multiple data word (SIMD) micro-architecture. In one example, one or more of the operating system scheduler executed by CPU 122 and the command processor in the GPU schedules commands on the SIMD channel. In some specific implementations, GPU 124 includes a pixel processing pipeline. In other specific implementations, the pixel processing pipeline is located outside the GPU 124. One or more of the SIMD channel and the pixel processing pipeline performs pixel value calculation, vertex transformation and other graphics operations, such as color management, environment adaptive pixel (AAP) modification, dynamic backlight control (DPB), panel gamma correction and dithering.

[0023] The video encoder 126 encodes (i.e., compresses) the video stream before transmitting it to the receiver 160. As used herein, "compression level" is also referred to as "compression amount" or "compression ratio". The greater the compression level or compression ratio, the greater the compression amount and the less video information included. Similarly, the smaller the compression level or compression ratio, the less compression amount and the greater the video information included. In various specific implementations, the video encoder 126 (or encoder 126) is implemented using any suitable combination of hardware and / or software (such as firmware).

[0024] The encoder 126 generates bits in a bitstream and stores them in a buffer. As used herein, a "bitstream" is a sequence of bits. In some cases, the output bitstream of the encoder is measured as a "bitrate," i.e., the number of bits generated or processed per unit time. For example, in some designs, the bitrate is expressed in kilobits per second (kbps) or megabits per second (mbps). As used herein, each of the terms "macroblock" and "block" is used to refer to a group of pixels. For example, in one specific implementation, a block is a group of 16x16 continuous pixels that form a square in the displayed image. In other specific implementations, blocks (or macroblocks) of other shapes and / or other sizes are used.

[0025] The encoder 126 receives uncompressed, rendered video information and generates bits in the form of a bitstream in a compressed format that conforms to a standard video compression specification. Examples of compression specifications or standards are various proprietary custom-designed codecs, MPEG-2 Part 2, MPEG-4 Part 2, H.264 (MPEG-4 Part 10), H.265 (High Efficiency Video Coding for Supporting 4K Video Compression), Theora, RealVideo RV40, VP9, ​​and AV1. The compression provided by the encoder 126 is typically lossy, so the output compressed video information lacks some of the information present in the original, rendered, and uncompressed video information. The video information is typically divided into frames, and frames are sometimes divided into macroblocks or blocks.

[0026] Due to the lossy nature of compression, the encoder 126 determines which information in the raw, rendered, and uncompressed video information to remove while minimizing visual quality degradation of the image depicted on a display device (not shown) of the receiver 160 as viewed by a user. For example, the encoder 126 determines which areas of the block or frame video information to compress at a higher compression ratio and which areas to compress at a lower compression ratio. In addition, the compression algorithm keeps track of the amount of data used to represent the video, which is determined by the bit rate, and also keeps track of the storage level of the buffer storing the compressed video information to avoid underflow conditions and overflow conditions.

[0027] For the above reasons, the encoder 126 faces many challenges to support the compression of received and rendered video information while achieving a target compression ratio, minimizing the latency of video transmission, preventing overflow and underflow of the buffer storing output data, and maximizing the user's subjective image quality on the display device. In addition, the encoder 126 typically removes higher frequency information from the video frame corresponding to the image in order to maintain the target bit rate. The sharp contrast edges of the text are an example of this type of high-frequency information. Due to video compression, the visual clarity of these edges is reduced. However, replacing the pixel color information of the macroblocks including the text with pixel distance information (such as SDF values) reduces the high-frequency information. These macroblocks no longer include sharp differences in value, but include smoothly changing values. In the example, the transmitter 110 replaces pixel color values ​​such as (142, 48, 96) with pixel distance values ​​such as (178, 178, 178). The steps for performing such replacement are further described below with respect to the CPU 122 and the APU 120.

[0028] In various implementations, the transmitter 110 transmits the frame 140 as a plurality of macroblocks, which are depicted by dashed lines placed on the frame 140. In the illustration, the frame 140 includes 20 macroblocks. However, it is possible and contemplated that the frame 140 includes any number of macroblocks based on design requirements. As previously described, the frame 140 includes a single image having one or more objects, such as an object of a lake, clouds in the sky, and the sun, and a text box 142 including one or more glyphs, such as alphabet letters used to spell the words "TEXT" and "HERE".

[0029] The CPU 122 converts the instructions of the parallel data function call of the application into commands that can be executed by the GPU 124. The GPU 124 stores the rendered video information of the frame 140 in one or more of the local buffer and the memory 116. Before the video encoder 126 compresses the rendered macroblocks, one or more of the CPU 122, GPU 124, or another integrated circuit of the transmitter 110 identifies a first group of one or more macroblocks of the frame 140 that correspond to an image including text such as text box 142. In the illustrated implementation, four of the 20 macroblocks of the frame 140 include text information. These four macroblocks are highlighted by the shaded squares in the metadata information 144 (or metadata 144). The circuitry of the transmitter 110 uses one of a variety of techniques for detecting the first group of one or more macroblocks. Examples of these techniques are one of various edge-based algorithms that rely on computing edge density or computing edge gradients of macroblocks, summing the counts of these values ​​in one or more histograms, and analyzing one or more histograms for text detection.

[0030] The transmitter 110 generates a second set of one or more macroblocks by replacing a first data representation of the first set of one or more macroblocks with a second data representation that is different from the first data representation. In a specific implementation, the first data representation includes pixel color information, such as one or more color values ​​for each pixel of the macroblock corresponding to one or more color components. The second data representation does not use the color values ​​of the pixels in the macroblock. Instead, the second data representation includes a distance value for each pixel in the macroblock.

[0031] In one example, a particular rendered pixel includes a pixel color value (142, 48, 96). In a specific implementation, each of these pixel color values ​​is an 8-bit value corresponding to a red value, a green value, and a blue value, respectively. Based on the 8 bits used to represent the color value, these color values ​​range from 0 to 255. The particular pixel is located in a macroblock within one of the four shaded squares corresponding to the macroblock containing the text. CPU 122 or other circuits of APU 120 determine a pixel distance value, which represents the shortest distance between the corresponding pixel and the nearest letter of the text in the macroblock. In a specific implementation, the pixel distance value uses a value of 0.5 to indicate that the pixel is on the surface of the nearest letter, a value of 0 indicates that the pixel is located at the farthest distance from the surface but within the letter, and a value of 1 indicates that the pixel is located at the farthest distance from the surface and outside the letter. Other values ​​and ranges are also possible and conceivable.

[0032] In some implementations, circuitry of the APU 120, such as a CPU, generates a pixel distance value by scaling the maximum color value by a ratio of the pixel distance value to the maximum pixel distance value. In other words, the APU 120 generates the pixel distance value using the expression (maximum color value) x ((pixel distance) / (maximum pixel distance)). When the CPU 122 or other circuitry of the APU 120 determines that the pixel distance value is 0.7 using a range of 0 to 1, where 0.5 is on the surface of the letter, the APU 120 generates a pixel distance value of 255 x (0.7 / 1), or 178. The APU 120 now represents the pixel with a pixel distance value of (178, 178, 178) instead of representing the pixel with a pixel color value of (142, 48, 96). The APU 120 also updates the metadata 144 with an indication that the macroblock that includes the pixel uses a pixel distance value instead of a pixel color value. The APU 120 also updates the metadata 144 with an indication of both the foreground color and the background color. The foreground color is the color of text, such as one or more glyphs in a particular macroblock. The background color is the color of the background behind the text in a particular macroblock. The metadata 144 is later used by the receiver 160. However, the encoder 126 does not receive the metadata 144, and the encoder 126 is unaware of which macroblocks include pixel color information and which macroblocks include pixel distance information.

[0033] In various specific implementations, the pixel distance value in the second data representation of the pixel is the directed distance field (SDF) value of the pixel. The SDF value represents the shortest distance between the corresponding pixel and the glyph closest to the pixel in the macroblock. As used herein, a "glyph" is a graphic symbol used to represent a letter of the alphabet. For example, the alphabet letter "A" has a specific graphic symbol (or glyph) that changes based on a selected font, a selected text size, and selected characteristics (such as italic printing, bold printing, etc.). In another specific implementation, the value in the second data representation of a pixel in a macroblock is the shortest distance between the pixel and a normal vector that starts at the nearest point on the glyph closest to the pixel in the macroblock. It is possible and conceivable that various other distance information corresponding to the pixel in the macroblock and the nearest glyph can be used in the second data representation.

[0034] Circuitry of transmitter 110 inserts one or more indications in metadata 144 corresponding to frame 140 that identify a first group of one or more macroblocks indicated by shaded squares. The transmitter also inserts indications in metadata 144 that specify color values ​​for pixels in the first group of one or more macroblocks. For a particular macroblock of the first group corresponding to one of the shaded squares in metadata 144, transmitter 110 inserts indications of both a foreground color and a background color. The foreground color is the color of text, such as one or more glyphs in the particular macroblock. The background color is the color of the background behind the text in the particular macroblock. Encoder 126 of transmitter 110 compresses (encodes) the macroblocks of frame 140. Thus, encoder 126 compresses one or more macroblocks that include pixel color information, and additionally compresses one or more macroblocks that include pixel distance information. Encoder 126 is unaware of which macroblocks include pixel color information and which macroblocks include pixel distance information.

[0035] The sender then transmits the metadata 144 and the encoded (compressed) macroblocks of the frame 140 to a receiver 160 with a display device. In a specific implementation, the sender transmits the metadata 144 and the encoded macroblocks of the frame 140 as streaming media data 130 via the network 112. The sender 110 transmits the streaming media data 130 to the receiver 160 based on a user request received from the receiver 160. The data cycle from the receiver 160 begins with the user request generated by the receiver 160 and ends with the receiver 160 receiving the streaming media data 130 to be decoded and displayed, and the data cycle occurs in real time. Combined with the limited communication bandwidth between the sender 110 and the receiver 160, the processing capabilities of both the sender 110 and the receiver 160 need to be efficient.

[0036] Receiver 160 includes a decoder 172 for decoding (decompressing) the encoded macroblocks of frame 140. Similar to encoder 126, decoder 172 does not receive metadata 144, and decoder 172 is unaware of which macroblocks include pixel color information and which macroblocks include pixel distance information. Receiver 160 uses metadata 144 to perform decoding. For example, receiver 160 uses metadata 144 to identify a third group of one or more macroblocks. After decoding the macroblocks of frame 140, receiver 160 converts the pixel distance information of one or more macroblocks into pixel color information. To this end, receiver 160 uses the pixel color information stored in metadata 144. Receiver 160 uses metadata 144 to identify pixels within a macroblock that includes text. For the pixels of the previous example having pixel distance values ​​(178, 178, 178), receiver 160 uses the foreground color information and background color information stored in metadata 144 to convert these values ​​into pixel color values ​​(142, 48, 96).

[0037] Using the above steps, the receiver 160 is able to generate a frame 150 having four macroblocks, which include text information of the text box 152. By encoding and sending pixel distance information as described above, the system 100 is able to provide text information with increased clarity and sharp edges, which allows the user to easily distinguish the alphabet letters or other symbols presented in the image of the frame 150 on the display device (screen or monitor) of the receiver 160. It should be noted that the text information in the text boxes 142 and 152 may include various types of symbols with a symbol foreground color and a single background color. These symbols are usually provided by the user through the computer keyboard. Examples of symbols are various types of alphabets, punctuation marks, mathematical operators, parentheses, square brackets, etc.

[0038] Now turn to Figure 2 , a generalized block diagram of a portion of a streaming desktop 200 delivered with efficient video delivery is shown. The portion shown includes a user interface 260, which is provided as a graphical user interface (GUI). In various specific implementations, the user interface 260 is presented on a display device of a user's computing device after being sent from a remote server that supports Remote Desktop Services (RDS). The user interface 260 is presented by a general application for generally showing user interface components. For ease of illustration, other components of the streaming desktop 200 are not shown. For example, in various specific implementations, the streaming desktop 200 additionally includes a taskbar, one or more icons of applications placed on the background of the streaming desktop 200, other open windows corresponding to other applications, one or more objects of images or scenes presented as a background when a solid color background is not used, and the like.

[0039] Similar to ( Figure 1), the streaming desktop 200 provides an example of an image being streamed in real time from a sender to a receiver. In a specific implementation, the receiver is a user's computing device and the sender is a remote server accessed via a network such as the Internet. Figure 1 The streaming desktop desktop 200 includes a plurality of text information areas, such as a text box 142 of a frame 140 of the streaming desktop desktop 200. Therefore, when the frame corresponding to the streaming desktop desktop 200 is divided into macroblocks, some of the macroblocks include text information. For example, the macroblocks including the text information include at least macroblocks including the texts "directory 210", "file 211", "directory 220", "back", "drawing", "project", "update", "clear", "address", "forward", values ​​"100%" to "70%", "August 22" in the calendar, dates from "Monday" to "Sunday" in the calendar, dates from "1" to "31" in the calendar, and the like.

[0040] As described above, the transmitter identifies a first group of macroblocks, stores the identification information of these macroblocks in metadata information, stores pixel color information (foreground color, background color) in metadata information, and generates a second group of macroblocks using pixel distance information instead of pixel color information. Therefore, the transmitter encodes one or more macroblocks including pixel color information, and additionally encodes one or more macroblocks including pixel distance information. Afterwards, the transmitter transmits the metadata information and the encoded macroblocks of the video frame to a receiver having a display device.

[0041] Similar to other user interfaces such as other GUIs, the user interface 210 includes features such as drop-down menus, navigation fields, address bars, etc. It should be noted that Figure 2The naming conventions depicted in (e.g., "directory 210," "file 211," etc.) are provided for ease of discussion. In a typical implementation, names that reflect those used in common business or other environments (e.g., documents, spreadsheets, tasks, discussion topics, and user-defined directories and / or file names, etc.) are used as desired. User interface 210 includes a "drawing" pane 202 and a "project" pane 204. Within drawing pane 202, a hierarchical structure of directories and files is shown to allow user navigation. The shown hierarchy includes directories 210, 220, 220, 240, and 250. Directory 210 includes files 211-214. The user is able to select the contents of one or more of directories 210-250 and files 211-214 via a move button and add them to project pane 204. Project pane 204 includes at least a chart and a calendar. Any additions are stored by selecting an update button, and any additions or modifications are removed by selecting a clear button. In other implementations, the user interface 210 includes other types of buttons, drop-down menus, etc. that present information in another manner (eg, a pie chart, a three-dimensional graph, etc.) or provide other types of functionality.

[0042] Now turn to Figure 3 , a schematic diagram of a computing system 300 is shown. In the illustrated implementation, computing system 300 includes multiple client devices 350, 352, and 354, a network 340, servers 320A-320D, and a data storage device 330 including a copy of an application 332. As shown, server 320A includes a processor 322 that accesses a memory 324 to process tasks; and a processor 326 that accesses a memory 328 to process tasks. Although three client devices 350, 352, and 354 are shown, any number of client devices access applications running on servers 320A-320D. For example, server 320A stores application 360 in memory 324, which is a copy of application 332 stored in data storage device 330. As shown, client device 354 includes hardware, such as circuits, of a processor 356 and a decoder 358. Processor 356 executes instructions of a computer program. Decoder 358 decodes encoded video frame information received from one or more of servers 320A-320D via network 340. Client devices 350 and 352 also include circuitry similar to processor 356 and decoder 358 of client device 354. Examples of client devices 350, 352, and 354 are laptop computers, smart phones, game consoles connected to a television, tablet computers, desktop computers, or others.

[0043] For ease of illustration, clock sources such as phase-locked loops (PLLs), interrupt controllers, communication structures, power controllers, memory controllers, interfaces for input / output (I / O) devices, etc. are not shown in computing system 300. It should also be noted that the number and size of components of computing system 300 are not shown. Figure 3 The number of subcomponents of the components shown in (such as within client 320) may vary depending on the specific implementation. The number of each component / subcomponent may be more or less than the number shown for computing system 300.

[0044] In various specific implementations, server 320A supports a corresponding client device in client devices 350-354 to access streaming media services, such as application 332, and remote request execution streaming media services. Examples of streaming media services are desktop streaming media services, live streaming media services, video game streaming media services, etc. When a user accesses streaming media services on a corresponding client device in client devices 350-354, streaming media services provide real-time updates of video content based on input from the user. In various specific implementations, client devices 350, 352 and 354 include a network interface (not shown) that supports one or more communication protocols for data and messages to be transmitted through network 340. Network 340 includes multiple switches, routers, cables, wireless transmitters and the Internet for transmitting messages and data. Therefore, the network interface of client device 350 supports one or more of hypertext transfer protocol (HTTP), transmission control protocol (TCP), user datagram protocol (UDP) or another protocol for cross-world wide web communication. In some specific implementations, an organization center (not shown) maintains application 332. In addition to communicating with client devices 350, 352, and 354 via network 340, the organization center also communicates with data storage 330 to store and retrieve data. Data storage 330 includes one or more of various hard disk drives and solid state drives for data storage. Through user authentication, users can access resources through the organization center to update user profile information, access purchase history or other access content, and download content.

[0045] In various implementations, processors 322 and 326 have the same configuration as previously described for ( Figure 1 The encoder 366 has the same functionality as previously described for the CPU 122 and GPU 124. Figure 1The processor 122 converts the instructions of the parallel data function call of the application 360 into commands 362 that can be executed by the processor 326. The encoder 366 can compress the rendered macroblocks. The servers 320A to 320D include various server types, such as database servers, computing servers, application servers, file servers, mail servers, etc. In various specific implementations, the servers 320A to 320D and the client devices 350, 352 and 354 operate with a client-server architecture model. In various specific implementations, the application 332 is a type of parallel data application among various types of parallel data applications.

[0046] Application 332 includes instructions to support parallel data algorithms. In a specific implementation, application 332 includes algorithms for a graphics shader program that directs the processor how to render pixels to control shading effects. In addition, application 332 may also include pixel interpolation algorithms for geometric transformations. Pixel interpolation obtains new pixel values ​​at arbitrary coordinates from existing data. Application 332 may also include instructions for directing the SIMD core of processor 326 (after conversion by processor 322) to perform general matrix-to-matrix multiplication (GEMM) operations when rendering macroblocks of a video frame.

[0047] In some implementations, application 332 (and its copy 360) is a user-requested application that is a specific video game application accessed through network 340 that provides content in real time to a user operating one of client devices 350-354. In other implementations, application 332 is a user-requested application that provides remote desktop connections for a business environment. In other implementations, application 332 is a user-requested application that provides a live stream of another user's broadcast content, such as a cooking show, movie reviews, clothing or other products, etc. In each of these implementations, server 320A provides rendered and compressed macroblocks of video frames to a corresponding one of client devices 350-354 in real time.

[0048] In various specific implementations, the server 320A implements the previously described Figure 1) transmitter 110 described steps. For example, the circuit of server 320A receives rendered source data of a video frame of an image to be depicted on a display device corresponding to one of the client devices 350-354. Before encoding, server 320A identifies a first group of one or more macroblocks of the video frame including text. Server 320A generates a second group of one or more macroblocks by replacing the pixel color information of the first group of one or more macroblocks with pixel distance information. Server 320A inserts an indication identifying the first group of one or more macroblocks in the metadata information. Server 320A also inserts an indication specifying the color value of the pixel in the first group of one or more macroblocks in the metadata information. Server 320A encodes the video frame. Therefore, server 320A encodes one or more macroblocks including pixel color information, and additionally encodes one or more macroblocks including pixel distance information. Server 320A transmits the encoded macroblocks together with the metadata information to the corresponding one of the client devices 350-354.

[0049] refer to Figure 4 , shows a generalized block diagram of a method 400 for performing efficient video transmission with text clarity preserved. For discussion purposes, this specific implementation (and Figure 5 However, in other implementations, some steps occur in a different order than shown, some steps are performed simultaneously, some steps are combined with other steps, and some steps are not present.

[0050] In various implementations, a computing system includes a transmitter that transmits a video stream to a receiver via a network such as the Internet. In a specific implementation, the transmitter is a remote server, and the receiver is a user computing device that communicates with the remote server via the network. In some implementations, the remote server includes a circuit system of an accelerated processing unit (APU), which includes a plurality of different integrated circuits, each of which provides different functions. For example, the APU includes at least a general processing unit, such as a central processing unit (CPU), wherein a plurality of cores are capable of executing instructions of a general instruction set architecture (ISA). The APU also includes a parallel data processing unit, such as a graphics processing unit (GPU), which includes a circuit system of one or more processor cores with a single instruction multiple data (SIMD) parallel architecture. The CPU converts the instructions of the parallel data function call of the application into commands that can be executed by the GPU. The GPU renders a video frame based on the command and stores the rendered video frame in a buffer. The circuit of the transmitter divides the rendering source data of the video frame into a plurality of macroblocks (block 402).

[0051] The transmitter selects a macroblock from a plurality of macroblocks (block 404). The transmitter determines whether the selected macroblock includes text. For example, the transmitter uses one of a plurality of techniques for detecting text within the selected macroblock. Examples of these techniques are one of various edge-based algorithms that rely on calculating edge density or calculating edge gradients of the macroblock, summing counts of these values ​​in one or more histograms, and analyzing one or more histograms for text detection. If the transmitter does not detect text within the selected macroblock (the "no" branch of conditional block 406), the transmitter maintains a current first data representation of the macroblock in the video frame (block 408). In a specific implementation, the first data representation includes color pixel information. In an example, a particular rendered pixel includes a pixel color value (212, 56, 180). In a specific implementation, each of these pixel color values ​​is an 8-bit value corresponding to a red value, a green value, and a blue value, respectively. Based on the 8 bits used to represent the color value, these color values ​​range from 0 to 255. If the transmitter has not reached the last macroblock of the video frame (the "NO" branch of conditional block 410), control flow of method 400 returns to block 404 where the transmitter selects a macroblock from the plurality of macroblocks.

[0052] If the transmitter detects text within the selected macroblock (the "yes" branch of conditional block 406), the transmitter inserts an indication of the color of the specified macroblock in the metadata corresponding to the video frame (box 412). For example, the transmitter inserts an indication of both the foreground color of the text (such as one or more glyphs in the macroblock) and the background color of the macroblock. The transmitter replaces the first data representation of the macroblock with the second data representation of the macroblock in the frame (box 414). For example, the transmitter generates a second macroblock including the second data representation of the selected first macroblock. In a specific implementation, the second data representation includes pixel distance information, such as a directed distance field (SDF) value of the pixel in the macroblock. In a specific implementation, the pixel distance value uses a value of 50 to indicate that the pixel is on the surface of the nearest glyph, a value of 0 to indicate that the pixel is at the farthest distance from the surface but within the glyph, and a value of 100 to indicate that the pixel is at the farthest distance from the surface and outside the glyph. Other values ​​and ranges are also possible and contemplated.

[0053] The transmitter inserts an indication in the metadata information corresponding to the video frame that the macroblock is represented using the second data (box 416). Thus, the transmitter identifies a set of one or more macroblocks of the video frame that includes text information and now uses the second data representation. In some specific implementations, the transmitter generates a pixel distance value by scaling the maximum color value with the ratio of the pixel distance value to the maximum pixel distance value. In other words, the transmitter generates the pixel distance value using the expression (maximum color value) x ((pixel distance) / (maximum pixel distance)). When the transmitter determines that the pixel distance value is 36 using a range of 0 to 100, where 50 is on the surface of the glyph, the transmitter generates a pixel distance value of 255x (36 / 100) or 92. The transmitter now represents the pixel with a pixel distance value (92,92,92) instead of a pixel color value (24,208,88). The transmitter also updates the metadata 144 with an indication that the macroblock including the pixel uses a pixel distance value instead of a pixel color value, as well as the foreground and background colors of the macroblock.

[0054] If the transmitter has reached the last macroblock of the video frame (the "yes" branch of conditional block 410), the transmitter compresses (encodes) multiple macroblocks of the video frame (box 418). For example, the APU of the transmitter also includes a video encoder for compressing the video frame of an image to be depicted on a display device of the receiver (such as a user's computing device). Therefore, the transmitter encodes one or more macroblocks including pixel color information and additionally encodes one or more macroblocks including pixel distance information. However, the encoder does not receive metadata, and the encoder is unaware of which macroblocks include pixel color information and which macroblocks include pixel distance information. The transmitter transmits the multiple compressed macroblocks of the video frame to the receiver for display (box 420). The transmitter also transmits the metadata information to the receiver (box 422). In some specific implementations, the metadata information is stored within the multiple compressed macroblocks of the video frame. In other specific implementations, the metadata information is stored separately from the multiple compressed macroblocks.

[0055] refer to Figure 5 , a generalized block diagram of a method 500 for supporting efficient video transmission that maintains text clarity is shown. In various specific implementations, a computing system includes a transmitter that transmits a video stream to a receiver via a network such as the Internet. The receiver receives rendered and compressed macroblocks of a video frame (block 502). The receiver also receives metadata information specifying data representations and colors of particular macroblocks of the video frame (block 504). Circuitry of the receiver, such as a decoder, decodes (decompresses) the macroblocks (block 506). Similar to the encoder of the transmitter, the decoder of the receiver does not receive metadata indicating pixel distance values, and the decoder is unaware of which macroblocks include pixel color information and which macroblocks include pixel distance information.

[0056] The receiver selects a macroblock from a plurality of macroblocks (block 508). In some implementations, metadata information is stored within a plurality of compressed macroblocks of a video frame. In other implementations, metadata information is stored separately from the plurality of compressed macroblocks. Using the metadata information, the receiver determines whether the selected macroblock includes text. If so, the metadata information includes pixel color information for the selected macroblock. For example, the metadata information includes an identifier for the selected macroblock and an indication of both a foreground color of the text (such as one or more glyphs in the macroblock) and a background color of the macroblock. The selected macroblock does not include pixel color information. Instead, as previously described, the selected macroblock includes one of a plurality of types of distance information corresponding to a pixel in the selected macroblock and a nearest glyph.

[0057] If the receiver determines that the selected macroblock has pixel color information specified in the metadata information (the "yes" branch of conditional block 510), the receiver uses the metadata information to replace the data representation of the macroblock with another data representation that includes the pixel color information. For example, the received version of the selected macroblock currently uses a data representation that includes pixel distance information, such as a directed distance field (SDF) value of the pixels in the macroblock. The receiver converts the pixel distance information into pixel color information (block 512). If the receiver determines that the selected macroblock does not have pixel color information specified in the metadata information (the "no" branch of conditional block 510), block 512 of method 500 is skipped.

[0058] If the receiver has not yet reached the last macroblock of the video frame (the "No" branch of conditional block 514), the control flow of method 500 returns to block 508, where the receiver selects a macroblock from the plurality of macroblocks. However, if the receiver has reached the last macroblock of the video frame (the "Yes" branch of conditional block 514), the receiver passes the decoded video frame to the display device (block 516).

[0059] It should be noted that one or more of the above-mentioned specific implementations include software. In such specific implementations, the program instructions of the implementation method and / or mechanism are transmitted or stored on a computer-readable medium. Many types of media configured to store program instructions are available and include hard disks, floppy disks, CD-ROMs, DVDs, flash memory, programmable ROMs (PROMs), random access memories (RAMs) and various other forms of volatile or non-volatile storage devices. Generally speaking, a computer-accessible storage medium includes any storage medium that can be accessed by a computer during use to provide instructions and / or data to a computer. For example, a computer-accessible storage medium includes such as magnetic or optical media, such as storage media such as disks (fixed or removable), tapes, CD-ROMs or DVD-ROMs, CD-Rs, CD-RWs, DVD-Rs, DVD-RWs or blue-rays. The storage medium also includes volatile or non-volatile storage media, such as RAM (e.g., synchronous dynamic RAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM, low power DDR (LPDDR2, etc.) SDRAM, Rambus DRAM (RDRAM), static RAM (SRAM), etc.), ROM accessible via a peripheral device interface (such as a universal serial bus (USB) interface, etc.), flash memory, non-volatile memory (e.g., flash memory). The storage medium includes micro-electromechanical systems (MEMS), and storage media accessible via communication media such as a network and / or a wireless link.

[0060] In addition, in various specific implementations, the program instructions include a behavioral level description or a register transfer level (RTL) description of the hardware functionality in a high-level programming language such as C or a design language (HDL) such as Verilog, VHDL, or a database format such as GDS II stream format (GDSII). In some cases, the description is read by a synthesis tool, which synthesizes the description to produce a netlist including a list of gates from a synthesis library. The netlist includes a set of gates, which also represents the functionality of the hardware comprising the system. The netlist is then placed and threaded to produce a data set describing the geometry to be applied to the mask. The mask is then used in various semiconductor manufacturing steps to produce a semiconductor circuit or circuit corresponding to the system. Alternatively, the instructions on the computer-accessible storage medium are a netlist (with or without a synthesis library) or a data set as desired. In addition, the instructions are used by the computer-accessible storage medium such as and Mentor Such vendors' hardware-based types of emulators are used for emulation purposes.

[0061] Although the above embodiments have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to encompass all such variations and modifications.

Claims

1. A device, comprising: A circuit, the circuit being configured as: identifying a first group of one or more blocks of a plurality of blocks of a video frame that include text information; replacing the first data representation of the first set of one or more blocks with a second data representation different from the first data representation; as well as The first set of one or more blocks is compressed using the second data representation. 2 . The apparatus of claim 1 , wherein the circuit is further configured to transmit the first set of one or more blocks to a receiver having a display device via a network.

3. The apparatus of claim 1, wherein the circuitry is further configured to insert, in metadata corresponding to the video frame, a first plurality of indications specifying colors used in the first set of one or more blocks.

4. The apparatus of claim 3, wherein the circuit is further configured to insert a second plurality of indications in the metadata, the second plurality of indications identifying the first group of one or more blocks as blocks of the video frame using the second data representation. 5 . The apparatus of claim 4 , wherein the circuit is further configured to communicate the metadata to a receiver having a display device via a network.

6. The apparatus of claim 4, wherein the circuit is further configured to compress one or more of the plurality of blocks of the video frame using the first data representation in response to the one or more blocks not including text information. The apparatus of claim 1 , wherein the second data representation comprises signed distance field information for pixels.

8. A method comprising: storing rendered source data of a video frame via a memory circuit for viewing on a display device; Receiving the video frame by circuitry of an accelerated processing unit; identifying, by the circuitry, a first group of one or more blocks of a plurality of blocks of a video frame that include text information; replacing, by the circuitry, the first data representation of the first set of one or more blocks with a second data representation different from the first data representation; as well as The first set of one or more blocks is compressed by the circuitry using the second data representation.

9. The method of claim 8, further comprising communicating, by the circuitry, the first set of one or more blocks to a receiver having a display device via a network.

10. The method of claim 8, further comprising inserting, by the circuitry, in metadata corresponding to the video frame, a first plurality of indications specifying colors used in the first set of one or more blocks.

11. The method of claim 10, further comprising inserting, by the circuitry, in the metadata, a second plurality of indications identifying the first set of one or more blocks as blocks using the second data representation.

12. The method of claim 11, further comprising communicating, by the circuitry, the metadata to a receiver having a display device via a network.

13. The method of claim 11, further comprising compressing, by the circuit, one or more of the plurality of blocks of the video frame using the first data representation in response to the one or more blocks not including text information. The method of claim 8 , wherein the second data representation comprises signed distance field information for pixels.

15. A computing system, comprising: a memory including circuitry configured to store rendered source data of a video frame for viewing on a display device; An accelerated processing unit, the accelerated processing unit comprising a circuit, wherein the circuit is configured to: Receiving the video frame; identifying a first group of one or more blocks of a plurality of blocks of the video frame that include text information; replacing the first data representation of the first set of one or more blocks with a second data representation different from the first data representation; as well as The first set of one or more blocks is compressed using the second data representation.

16. The computing system of claim 15, wherein the circuitry is further configured to communicate the first set of one or more blocks to a receiver having a display device via a network.

17. The computing system of claim 15, wherein the circuitry is further configured to insert, in metadata corresponding to the video frame, a first plurality of indications specifying colors used in the first set of one or more blocks.

18. The computing system of claim 17, wherein the circuitry is further configured to insert a second plurality of indications in the metadata, the second plurality of indications identifying the first set of one or more blocks as blocks using the second data representation.

19. The computing system of claim 18, wherein the circuit is further configured to communicate the metadata to a receiver having a display device via a network.

20. The computing system of claim 15, wherein the second data representation comprises signed distance field information for pixels.

Images