System and method for redirecting audio and video data streams in a display-server computing system

Abstract

A system operable on a data communications network to operate on audio / video data includes a server computer communicatively coupled to the network. The server computer includes an application unit that runs an application. The application is accessible to a user device communicatively coupled to the network. The application unit requires audio / video data to run the application. The system also includes an audio / video server computer communicatively coupled to the network. The audio / video server computer transmits audio / video data over the network in response to a request. The user device includes at least a buffer, a decoder, and an output device. The server computer issues a demand request over the network to the user device to receive audio / video data for operation of the application unit. The demand request includes an identifier of the audio / video data and a network address of the audio / video server computer. The user device, upon receiving the demand request, issues a request over the network to the audio / video server computer in communication, and the audio / video server computer communicates the audio / video data over the network to the user device in communication. The user device reformats the audio / video data and communicates the reformatted audio / video data to the server computer. The server computer reverses the reformatted audio / video data to obtain substantially the original audio / video data for operation of the application unit.

Description

[0001] Cross-reference to related applications

[0002] This application enjoys priority to the following application, which has at least one inventor in common with it, and is incorporated herein by reference: U.S. Provisional Patent Application No. 62 / 646,429, filed March 22, 2018, entitled “System and Method for Redirecting Audio and Video Data Streams in a Display-Server Computing System”. Technical Field

[0003] This invention relates generally to computing systems, and more specifically to display-server systems and communication networks. Background Technology

[0004] End-user computing software programs and their operation (i.e., applications) are traditionally processed (i.e., run) on the end-user device itself, such as a personal computer running applications on a personal computer (PC), a smartphone running applications on a smartphone, a game console running game applications on a controller, and so on. A different approach is to run end-user applications on servers located in data centers. These server-run applications are transmitted to the end-user device, such as those described, or even a display device with limited processing power, via a communication connection between the end-user device and the server. Such a display-server system is described in U.S. Patent No. 8,700,723. When the application running on the server receives audio and video from an external source, this audio and video data must be transmitted from the server to the end-user display device. The required network bandwidth includes the bandwidth used for data transmission to the server and the bandwidth used for data transmission from the server to the end-user device. In effect, the bandwidth requirement doubles.

[0005] Reference Figure 1A non-exclusive example of a conventional system 100 includes an application unit 105 that runs directly on a device 106 (such as an end-user computer, smartphone, controller, or other device). Device 106 is communicatively connected to network 102. System 100 describes in detail a typical use case of a non-monitor-server system where device 106 requires significant processing and storage to operate the application unit. The application unit 105 running on device 106 requests the ability to retrieve audio / video data from an audio / video server 101 communicatively connected to network 102. Device 106 sends a request 103 for audio / video information on network 102 to audio / video server 101. Audio / video server 101 responds by transmitting the required audio / video data 113 across network 102 to device 106. Application unit 105 of device 106 receives the audio / video data 113 and stores it in a buffer 112 of device 106. When there is sufficient audio / video data in buffer 112, application unit 105 issues instruction 107 to decoder 110 of device 106 to begin decoding. The decoder output is displayed on display 109.

[0006] In a monitor-server computing system, a server computer operates an application unit. When audio and / or video data is needed to operate the application unit, the server computer typically contacts an audio / video server computer via a network and requests the audio / video data. The audio / video data is then sent by the audio / video server computer to the server computer operating the application unit via the network. The server computer operating the application unit must then send the audio / video data used for application unit operation to user devices, such as monitors, personal computers, smartphones, controllers, or other devices. The user device communicates with the server computer, and the server computer transmits the audio / video data associated with the application unit being operated on the server computer. In this traditional monitor-server computing system, bandwidth requirements are critical because the audio / video data must be retrieved by the server computer and transmitted to the user device.

[0007] Providing more efficient and optimized audio and video data streams to end-user devices (such as personal computers, smartphones, controllers, or even displays or other devices with limited processing power) would be a significant advancement in this field. Summary of the Invention

[0008] One embodiment of the present invention is a system comprising: a server computer running an application unit on the server computer; a user device communicatively connected to the server computer; and another server computer capable of transmitting audio and video data. Both the server computer and the user device include decoders for decrypting the audio and video data.

[0009] The server computer running the application unit requires audio / video data from another server computer (typically, this other server computer is different from the server / computer running the application unit, although in embodiments, the same server may host the application unit and provide the audio / video data). A request for this audio / video data, along with an identifier for the audio / video server computer, is sent by the server computer running the application unit to the user equipment. The user equipment forwards this request to the audio / video server computer. The audio / video server computer responds to the user equipment with the required audio / video data. The user equipment receives the audio / video data and stores it in a buffer. The audio / video data is also reformatted by the user equipment to include only the data required by the server computer running the application unit to operate the application unit, excluding the data required by the decoder. This significantly reduces the size of the reformatted audio / video data. The reformatted data is then transmitted by the user equipment to the server computer running the application unit for operating the application unit.

[0010] The server computer of the application unit receives reformatted audio / video data, converts it back to a standard format, and then transmits the audio / video data to the application unit. For the application unit operating on the server computer, this audio / video data is indistinguishable from the original audio / video data sent from the audio / video server computer to the user device. The application unit running on the server computer stores this audio / video data in a buffer and uses it to operate normally. When there is sufficient audio / video data in the buffer, the application unit sends a command to the decoder of the server computer to begin decoding. The decoding command is forwarded to the user device, such as a display device, and then the user device's decoder begins decoding the audio / video from the user device's local buffer. Therefore, the audio / video data used by the user device's decoder is the actual audio / video data sent from the audio / video server, and the amount of audio / video data sent by the user device to the server is very limited. Attached Figure Description

[0011] The invention has been illustrated in the accompanying drawings by way of example rather than limitation, and similar reference numerals in the drawings indicate similar elements, wherein:

[0012] Figure 1 A system according to certain embodiments of the present invention is shown, which can operate on a data communication network of a server computer running an application accessed by a user or a display system of a conventional display-server system;

[0013] Figure 2 A system according to certain embodiments of the present invention is illustrated, which can operate on a data communication network of a server computer running an application accessed by a user or display system, wherein it is the user or display system, not the server computer, that obtains audio / video data from an audio / video server computer, and the user or display system transmits reformatted audio / video data to the server computer for use by the running application; and

[0014] Figure 3 Exemplary audio / video data streams according to certain embodiments of the present invention are shown, including reformulated audio / video data streams and reconstructed audio / video data streams, the data streams being transmitted to and used by a user or display device and a server computer, the server computer running an application accessed by the user or display device. Detailed Implementation

[0015] Reference Figure 2 One non-exclusive example embodiment of system 200 includes a server computer 215. Server computer 215 includes, for example, applications communicatively connected to application unit 224 via a data communication network. By way of example, and not limitation, server computer 215 includes a processor 223, a cache 222, and a reformatting module 214. Application unit 224 also includes a decoder. Application unit 224 runs on server computer 215.

[0016] The server computer 215 of application unit 224 is connected to one or more networks 202, 213 in a communication manner. The networks can be the same or different networks, and in each case, they can include a combination of communication networks such as wired and / or wireless data networks.

[0017] Audio / video server 201 is communicatively connected to one or more networks 202, 213. In this example embodiment, network 202 may be the same as or different from network 213. As a non-exclusive example, audio / video server 201 may include a processor, a cache, and memory. Audio / video server 201 transmits data representing audio and video information via network 202, which can be used by application unit 224 of server computer 215 and by user equipment 204, such as a display, personal computer, smartphone, controller, or other device, for output display or audio.

[0018] User equipment 204 is connected to audio / video server 201 via network 202 and to server computer 215 via network 213. Networks 202 and 213 may be wholly or partially the same or different. As a non-exclusive example, user equipment 204 includes a control / processor 206, a buffer 205, a decoder 210, a display 209 or other output device, and a reformatting module 211. User equipment 204 may include other or alternative elements, if applicable. User equipment 204 may, but does not necessarily, have limited storage and processing capabilities. In this example embodiment, application unit 224 operating on server computer 215 is accessed by user equipment 204 via network 213 to provide running applications for user equipment 204.

[0019] In operation, server computer 215 controls application unit 224 in a controlled manner for access by user equipment 204. During this operation, application unit 224 requires audio / video data. This audio / video data can be obtained from audio / video server 201. Server computer 215 sends a request 217, along with an identifier of audio / video server 201, to user equipment 204 via network 213 in a communicative manner. User equipment 204 receives request 217 via processor 206. User equipment 204 forwards request 217 to audio / video server 201 via network 202 in a communicative manner. Audio / video server 201 responds to user equipment 204 via network 202 with the required audio / video data 203. User equipment 204 receives the audio / video data 203 and stores it in buffer 205. The audio / video data 203 is further reformatted 211 by the user equipment 204 to include only the data required for the operation of the application unit 224, excluding the data required by the decoder of the server computer 215. The size of the reformatted audio / video data 211 is significantly smaller than the original audio / video data received by the user equipment 204 from the audio / video server 201. The reformatted data 212 is then transmitted by the user equipment 204 to the server computer 215 via network 213.

[0020] Server computer 215 receives the reformatted data 212 from user equipment 204 via network 213. Server computer 215 processes the reformatted data 212 by converting it back to the standard format of the original audio / video data 203 received by user equipment 204. Reformatting module 214 of server computer 215 receives the reformatted data 212. Reformatting module 214 converts the reformatted data 212 back to the standard format of audio / video data 220, which is substantially the same as the original audio / video data 203. The converted data from reformatting module 214, i.e., audio / video data 220, is transmitted by reformatting module 214 to application unit 224 of server computer 215. Audio / video data 220 is indistinguishable to application unit 224 from the original audio / video data 203 sent from audio / video server 201. Application unit 224 stores audio / video data 220 in buffer 222, and then the audio / video data 220 is used for operations performed by application unit 224. When there is enough audio / video data 220 in buffer 222, application unit 224 issues instruction 218 to the decoder of server computer to begin decoding audio / video data 220. Decoding instruction 218 is forwarded by server computer 215 to user equipment 204 on network 213, and then decoder 210 of user equipment 204 decodes the raw audio / video data 203 stored in local buffer 205 of user equipment. The output of decoder 210 is displayed on display 209 as audio output, or otherwise output by user equipment 204 depending on the nature of audio / video data 203.

[0021] Therefore, the audio / video data 203 used by decoder 210 is the actual audio / video data 203 sent by audio / video server 201 to user equipment 204. The audio / video data 212 sent by user equipment 204 to server computer 215 is significantly smaller than the original audio / video data 203. This reduces the bandwidth requirements for communication.

[0022] Combination Figure 2 and refer to Figure 3 A non-exclusive example embodiment of system 300 includes three data formats for the audio / video data in the foregoing example. These three data formats include received audio / video data stream 203, transmitted audio / video stream 212, and reconstructed audio / video data stream 220. Received audio / video data stream 203 is received by user equipment 204 from audio / video server 201.

[0023] User equipment 204 reformatts the received audio / video data stream 203 into a transmitted audio / video data stream 212. Reformatting involves directly copying information data 303 and header data 304 to the transmitted audio / video data 212. Then, decoder data 306 is converted into a cookie 305 (web cookie) with a length 307. This process is repeated for each segment of the received audio / video data stream 203.

[0024] The transmitted audio / video data 212 is received by the server computer 215 from the user equipment 204. The server computer 215 reformatts 214 the transmitted audio / video data 212 from the user equipment 204 into a reconstructed audio / video data stream 220. The reformatting includes directly copying information data 310, header data 311, and cookie data 312 into the reconstructed audio / video data stream 220. Then, using length 313, that amount of padding data 316 is created in the reconstructed audio / video data stream 220. This process is repeated for each segment of the transmitted audio / video data 212.

[0025] The length 301 of the received audio / video data stream 203 is exactly the same as the length 301 of the reconstructed audio / video data stream 220. The length 308 of the transmitted audio / video data stream 212 is much smaller than that of the received audio / video data stream 203.

[0026] In the foregoing specification, the invention has been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the invention as set forth in the following claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the invention.

[0027] Benefits, other advantages, and solutions to problems have been described above with reference to specific embodiments. However, the benefits, advantages, solutions to problems, and devices, connections, and elements that may lead to or make more apparent any benefit, advantage, or solution should not be considered essential, claimed, or necessary features or elements of any or all claims. As used herein, the terms “comprising,” “including,” or any other variation are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

Claims

1. A system operating on a data communication network, comprising: Server computers connected to the network according to communication methods; The application unit of the server computer requires audio / video data in a first format for operation; Display devices connected to the server computer via a network using a communication method; The reformatting unit of the display device; and An audio / video server is connected to the display device via a network using a communication method. The audio / video server has a unique network identifier. The audio / video server transmits raw audio / video data in a first format to the display device. The raw audio / video data has the following format: raw information data, first raw header data, first decoder data, second raw header data, second decoder data, third raw header data, and third decoder data. The application unit sends a request for raw audio / video data in a first format, as well as the network identifier of the audio / video server, to the display device via the network. The display device, based on the network identifier, directly sends a request for raw audio / video data in the first format to the audio / video server at the network address via the network. The audio / video server directly sends raw audio / video data in a first format related to a request from the display device to the display device via a network; The reformatting unit of the display device converts the original audio / video data in the first format into reformatted audio / video data in the second format. The reformatted audio / video data in the second format has a smaller byte size than the original audio / video data in the first format. The data format of the reformatted audio / video data is as follows: reformatting information data, first reformatting header data, first reformatting network cookie, first reformatting length, second reformatting header data, second reformatting network cookie, second reformatting length, third reformatting header data, third reformatting network cookie, and third reformatting length. The original information data and original header data are directly copied into reformatting information data and reformatting header data, and the decoder data is converted into reformatting network cookie and reformatting length. The display device transmits reformatted audio / video data in a second format to the application unit of the server computer via a network; and The server computer reconstructs the second-format reformatted audio / video data received from the display device into a first-format reconstructed audio / video data. The reconstructed audio / video data has the following format: reconstructed information data, first reconstructed header data, first reconstructed network cookie, first padding data, second reconstructed header data, second reconstructed network cookie, second padding data, third reconstructed header data, third reconstructed network cookie, and third padding data. The reformatted information data, reformatted header data, and reformatted network cookie are directly copied as reconstructed information data, reconstructed header data, and reconstructed network cookie, and padding data is created by using the reformatted length. In this way, the reconstructed audio / video data corresponds to the original audio / video data.

2. The system according to claim 1, wherein, The application unit operates using reconstructed audio / video data in a first format converted from reformatted audio / video data in a second format. This reconstructed audio / video data is indistinguishable to the application unit from the original audio / video data sent from the audio / video server.

3. The system according to claim 1, wherein, The display device decodes the raw audio / video data in the first format and outputs the raw audio / video data in the first format via the display device's monitor or another output device.

4. A system capable of operating on a data communication network to manipulate audio / video data of a first format, comprising: A server computer connected to a network via a communication method, the server computer including an application unit connected to the network via a communication method and running an application, the application unit requiring audio / video data in a first format to run the application; An audio / video server connected to a network by a communication method, the audio / video server having a unique network identifier for itself, the audio / video server responding to requests and transmitting audio / video data in a first format over the network; and User equipment connected to a network according to a communication method, the user equipment includes at least a buffer, a decoder, and an output device; The server computer sends a request to the user equipment via the network to receive raw audio / video data in a first format for operating the application unit. The request includes the identifier of the raw audio / video data and the network address of the audio / video server. Upon receiving a request, the user equipment directly sends a request to the audio / video server via the network using the network identifier. The audio / video server then directly transmits the original audio / video data in the first format to the user equipment via the network. The original audio / video data has the following data format: original information data, first original header data, first decoder data, second original header data, second decoder data, third original header data, and third decoder data. A reformatting device of a user equipment converts the original audio / video data of the first format into reformatted audio / video data of the second format with a reduced byte size. The data format of the reformatted audio / video data is as follows: reformatting information data, first reformatting header data, first reformatting network cookie, first reformatting length, second reformatting header data, second reformatting network cookie, second reformatting length, third reformatting header data, third reformatting network cookie, and third reformatting length. The original information data and original header data are directly copied as reformatting information data and reformatting header data, and the decoder data is converted into reformatting network cookie and reformatting length. In this process, the user equipment transmits the reformulated audio / video data of the second format to the server computer via a network; and The server computer's reverse formatting unit converts the second-formatted reformatted audio / video data into the first-formatted reconstructed audio / video data for use by the application unit. The reconstructed audio / video data has the following data format: reconstructed information data, first reconstructed header data, first reconstructed network cookie, first padding data, second reconstructed header data, second reconstructed network cookie, second padding data, third reconstructed header data, third reconstructed network cookie, and third padding data. The reformatted information data, reformatted header data, and reformatted network cookie are directly copied as reconstructed information data, reconstructed header data, and reconstructed network cookie, and padding data is created using the reformatted length. The reconstructed audio / video data is indistinguishable from the original audio / video data sent from the audio / video server to the application unit.

5. The system according to claim 4, further comprising: The user equipment decoder converts the reformulated audio / video data of the second format into output through the user equipment. and The server computer's decoder converts the reconstructed audio / video data of the first format for operation on the application unit.

6. The system according to claim 4, wherein, The requirement is to reduce bandwidth for displaying or otherwise outputting reformatted audio / video data in a second format by a user device, and to transmit the reformatted audio / video data in the second format to a server computer for operation of the application unit.

7. A method for redirecting audio and video data streams in a display-server computing system, comprising: The server computer transmits a request for raw audio / video data in a first format to a user device that is connected to the server computer via a network through a data communication network. The request includes a network identifier of the audio / video server connected to the user device via a network. The user equipment directly accesses the audio / video server via the network based on the network identifier to obtain the original audio / video data in the first format. The original audio / video data has the following data format: original information data, first original header data, first decoder data, second original header data, second decoder data, third original header data, and third decoder data. The user equipment reformats the original audio / video data in the first format into reformatted audio / video data in a second format with a reduced byte size. The reformatted audio / video data has the following data format: reformatted information data, first reformatted header data, first reformatted network cookie, first reformatted length, second reformatted header data, second reformatted network cookie, second reformatted length, third reformatted header data, third reformatted network cookie, and third reformatted length. The original information data and original header data are directly copied into reformatted information data and reformatted header data, and the decoder data is converted into reformatted network cookie and reformatted length. The user equipment sends the reformulated audio / video data in the second format over the network for the server computer to receive; The reformatted audio / video data in the second format received by the server computer is reverse-formatted back into reconstructed audio / video data in the first format. The reconstructed audio / video data has the following data format: reconstructed information data, first reconstructed header data, first reconstructed network cookie, first padding data, second reconstructed header data, second reconstructed network cookie, second padding data, third reconstructed header data, third reconstructed network cookie, and third padding data. Specifically, the reformatted information data, reformatted header data, and reformatted network cookie are directly copied as reconstructed information data, reconstructed header data, and reconstructed network cookie, and padding data is created using the reformatted length. The application unit is run by the server computer, and the application unit operates on the reconstructed audio / video data in the first format.

8. The method according to claim 7, in, Transmission is a response to the application unit; The operation includes the application unit processing the reconstructed audio / video data in the first format, and the result is the same as the original audio / video data.

9. The method according to claim 8, further comprising: The user equipment stores the raw audio / video data of the first format in a buffer; The user equipment decodes the raw audio / video data of the first format in the buffer; as well as The user equipment outputs, via an output device, reformatted audio and / or video represented by reformatted audio / video data in the second format.

10. The method of claim 9, further comprising: The server computer stores the reconstructed audio / video data in the buffer, obtained through reverse reformatting and in the first format. The server computer decodes the reconstructed audio / video data of the first format in the buffer; and In response to decoding, the application unit is run by the server computer using the reconstructed audio / video data in the first format as input.

Images