Block-based multimedia packet assignment using multisource coding

Abstract

Methods and apparatus to perform block-based packet-assignment for delivering multimedia. According to an example embodiment, an end-user device receives descriptions (i) identifying blocks of multimedia content encoded using a rateless error correction code and stored at a plurality of network sources and (ii) specifying network conditions for the corresponding plurality of network paths. The end-user device communicates to the network sources a packet assignment plan for transmission of packets corresponding to different blocks from the network sources to the end-user device. The packet assignment plan is determined based on the received plurality of descriptions and an optimization performed with an objective function including a component representing an estimated distortion of the received multimedia content and another component representing a cost of delivering the packets. The end-user device reconstructs the multimedia content using the blocks assembled from corresponding sets of packets transmitted in accordance with the communicated packet assignment plan.

Description

BLOCK-BASED MULTIMEDIA PACKET ASSIGNMENT USING MULTISOURCE CODING 1. Cross-Reference to Related Applications

[0001] This application claims priority to US provisional application 63 / 613,654, filed 21 December 2023, which is incorporated herein by reference in its entirety. 2. Field of the Disclosure

[0002] Various example embodiments relate to multimedia streaming and, more specifically but not exclusively, to video streaming with multiple sources. 3. Background

[0003] Multipath routing is a technique that allows for network communications to be sent over multiple paths. Multipath routing stands in contrast to single-path routing where a single path from a source to a destination is used to deliver a data stream. With multiple paths, any number of selectable routes can beneficially be used to achieve performance enhancements, connection stability, and / or security improvements. BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS

[0004] Various embodiments provide efficient and effective solutions directed at optimizing the quality of received multimedia content in a multimedia delivery system that includes multiple network sources connected to an end-user device via multiple network paths. Some embodiments are directed at optimizing the delivery of a single multimedia content to the end-user device. Some other embodiments are directed at optimizing parallel delivery of multiple multimedia contents to the end-user device. For each category, algorithmic solutions corresponding to several specific sub- scenarios and / or use cases are disclosed. For example, in the multiple multimedia contents category, solutions corresponding to two different optimization objectives are presented: (1) substantially minimizing the combined amount of distortion for the received multiple multimedia contents and (2) substantially minimizing the largest individual content distortion among the received multiple multimedia contents. A foundational algorithm serving as a building block for a plurality of different scenarios, optimization objectives, and use cases is provided. With the adoption ofmultimedia partition, multiple network sources, and rateless code, the disclosed embodiments can beneficially be used to improve the end-user viewing experience.

[0005] According to an example embodiment, provided is a block-based packet-assignment method for delivering multimedia, the method comprising: receiving, at an end-user device, a plurality of descriptions identifying blocks of a multimedia content stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicating from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the multimedia content from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on an optimization with an objective function including a first component representing an estimated distortion of the multimedia content when received by the end-user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstructing the multimedia content using the blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

[0006] According to another example embodiment, provided is a non-transitory computer- readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the above method.

[0007] According to yet another example embodiment, provided is an apparatus to perform block-based packet assignment for delivering multimedia, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: receive, at an end-user device, a plurality of descriptions identifying blocks of a multimedia content stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicate from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the multimedia content from theplurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on an optimization with an objective function including a first component representing an estimated distortion of the multimedia content when received by the end-user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstruct the multimedia content using the blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

[0008] According to yet another example embodiment, provided is a block-based packet- assignment method for delivering multiple multimedia, the method comprising: receiving, at an end- user device, a plurality of descriptions identifying blocks of a plurality of multimedia contents stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicating from the end- user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the plurality of multimedia contents from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on an optimization with an objective function including a first component representing an estimated distortion of the plurality of multimedia contents when received by the end-user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstructing each of the plurality of multimedia contents using blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

[0009] According to yet another example embodiment, provided is a non-transitory computer- readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the above method.

[0010] According to yet another example embodiment, provided is an apparatus to perform block-based packet assignment for delivering multiple multimedia, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least onememory and the program code are configured to, with the at least one processor, cause the apparatus at least to: receive, at an end-user device, a plurality of descriptions identifying blocks of a plurality of multimedia contents stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicate from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the plurality of multimedia contents from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on optimization with an objective function including a first component representing estimated distortion of the plurality of multimedia contents when received by the end-user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstruct each of the plurality of multimedia contents using blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

[0011] According to yet another example embodiment, provided is a method of distributing blocks of a multimedia content for storage at a plurality of network sources, the method comprising: sorting the blocks of the multimedia content into different groups having different respective priorities; and distributing the sorted blocks over the plurality of network sources based on an evaluation of a cost to store at different ones of the network sources, a cost of delivery therefrom to an end-user device, and a relative contribution to quality of user experience during playback of the multimedia content at the end-user device.

[0012] According to yet another example embodiment, provided is a non-transitory computer- readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other aspects, features, and benefits of various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings, in which:

[0014] FIG.1 an example of using multimedia partitions for video encoding.

[0015] FIG. 2 is a block diagram illustrating application of a multimedia partition concept to a 3D model according to one example.

[0016] FIGS.3-5 are block diagrams illustrating stages of content delivery from multiple network sources to a single end-user device according to one example.

[0017] FIG. 6 is a block diagram illustrating a multimedia delivery system that can be used to deliver prioritized multisource media to an end-user device according to one example.

[0018] FIG. 7 is a block diagram illustrating a multimedia delivery system that can be used to deliver prioritized multisource media to an end-user device according to another example.

[0019] FIG. 8 is a block diagram illustrating a first scenario corresponding to a Single Multimedia Content (SMC) embodiment according to some examples.

[0020] FIGS.9-14 are block diagrams illustrating operations of a fast greedy algorithm applied to the SMC transmission according to one embodiment.

[0021] FIG.15 is a block diagram illustrating a second scenario corresponding to the SMC embodiment according to some examples.

[0022] FIG.16 is a block diagram illustrating an example outcome of a fast greedy algorithm applied to the SMC transmission in the scenario illustrated in FIG.15.

[0023] FIG. 17 is a block diagram illustrating a third scenario corresponding to the SMC embodiment according to some examples.

[0024] FIG.18 is a block diagram illustrating a scenario corresponding to a Multiple Multimedia Content (MMC) embodiment according to some examples.

[0025] FIGS.19-24 are block diagrams illustrating an algorithm applied to MMC transmission according to one example.

[0026] FIG.25 is a block diagram of an example computing device configured to various operations in accordance with various embodiments.

[0027] FIG.26 is a flowchart illustrating a block-based packet-assignment method for delivering multimedia according to some examples. DETAILED DESCRIPTION

[0028] Some embodiments are directed to a joint video and network coding solution employing multiple network sources (e.g., content delivery network “CDN,” peers, distributed network storage, and the like) to deliver a video stream to the end device in the “last mile” setup. In one example, such delivery is adapted to a multi-path scenario in which different paths connecting one or more network sources to the destination node have different respective characteristics, such as bandwidth, packet loss rate, financial cost, etc. In one approach, the multimedia content is partitioned into multiple blocks with different priorities and received-multimedia quality impact. With joint consideration of both each network path’s condition and each multimedia block’s information, we can apply different strengths of rateless coding for each block in different network paths to enable the unequal error protection (UEP). In some examples, the rateless coding is implemented using random linear network coding (RLNC). The disclosed framework lends itself to optimization directed at achieving nearly maximum received-multimedia quality subject to the network constraint(s).

[0029] Some features of the disclosed embodiments are inspired by an observation that, in at least some examples, a conventional multimedia bitstream exhibits several drawbacks. For example, a required bit rate may be relatively high, which can disadvantageously cause network congestion and / or playback jitter. In many cases, multimedia compression exploits self-redundancy inside the corresponding multimedia. Removing the redundancy often translates into a decoding dependency. Consequently, losing one portion of the bitstream may cause next portions of the bitstream undecodable, thereby disrupting the playback of the content. Such characteristics of conventional multimedia bitstream tend to detrimentally impact the final user experience.

[0030] The above-indicated challenges may be tackled in a number of different ways. For example, from the multimedia coding side, increasing the compression efficiency and removing decoding dependency can be leveraged. In one example, a multimedia content can be partitioned into multiple blocks and each block can be coded independently. Perception for each block can be different, thus priority / quality measure for each block can be defined. From the network side, adding multiple network sources to exploit multi-path diversity and circumvent single-path failureand appending a forward error correction (FEC) coding mechanism to reduce the impact of packet loss may be useful. With recent advances in rateless coding, such as the use of random linear network coding (RLNC), the system can be configured to prepare one set of coded packets to handle many different network conditions. In most cases, this feature can be more attractive than the use of conventional FEC coding under which parity packets typically need to be re-encoded when network conditions change. In some examples, by combining the above-outlined design freedom, a multimedia block can be sent from multiple network sources with different levels of error protection. Formally, when the end user has the corresponding source and network information, the resource allocator can optimize how to protect and transmit each block to reach an optimal perceived quality, subject to the network conditions and availability of content at different network sources.

[0031] Some embodiments disclosed herein address the above-indicated scenarios to optimize end-user’s experience. Various options include sending single or multiple multimedia content(s), e.g., video(s), from multiple network sources to an end user. For each scenario, we address several different sub-scenarios with different objective functions. We formulate each sub-scenario as an optimization problem (e.g., minimizing a total amount of distortion or minimizing maximal distortion) and provide a corresponding solution to achieve approximately highest overall received multimedia quality. Multimedia Content Partition

[0032] Multimedia content often entails a high bit rate, and the corresponding bitstream may exhibit relatively strong decoding dependency. In many use cases, a conventional coding structure may present challenges to implementing computation optimization (such as parallelism) and received-quality optimization (e.g., dealing with unreliable network conditions). In some examples, the multimedia content is partitioned into blocks, and each block is encoded separately or independently to satisfy a selected set of goals, typically with some associated penalty in terms of the coding efficiency (e.g., a slightly diminished R-D performance). In different embodiments, different types of multimedia partitions can be used. For illustration purposes and without any implied limitations, some examples are described below in reference to tile-type multimedia partitions. Based on the provided description, a person of ordinary skill in the pertinent art will be able to make and use other examples in which other types of multimedia partitions are used.

[0033] One example of multimedia partitions is referred to as “tile.” One tile example is described in the HEVC standard. Other tile examples and definitions can also be used in various embodiments. The following description of a particular type of tile is provided merely for illustration purposes, without limiting various embodiments to the described tiles. Based on the provided description, a person of ordinary skill in the pertinent art will be able to use other types of tiles without any undue experimentation.

[0034] One tile example (e.g., representing one block) was originally proposed in video coding to facilitate the use of parallelism for increasing the encoding / decoding speed. When used together with the temporal motion constrained tile set SEI (MCTS SEI), the tile partition can be implemented such that an individual tile has no dependency on other tiles. The corresponding coding constraints include: (1) not allowing motion prediction and compensation outside the current tile boundary and (2) not allowing loop-filtering across tile boundary. At the decoder, tiles can be selected and re- assembled to another conforming bitstream. Therefore, the tile partition can also be used as an effective tool for robust video transmission, e.g., to overcome packet loss, and as a vehicle to deliver more versatile viewing experiences, e.g., including 360-degree videos.

[0035] A light assembler / transcoder for tile re-assembly is needed to rewrite parameter sets (e.g., VPS - video parameter set; SPS - sequence parameter set; and PPS - picture parameter set) and update the slice header because one or all of the slice address related syntax elements (including first_slice_segment_in_pic_flag and slice_segment_address) would typically need to have different values.

[0036] Independent tiles can also be used to enable dynamic tiling, e.g., as described in R. Skupin, Y. Sanchez, C. Hellge and T. Schierl, “Tile Based HEVC Video for Head Mounted Displays,” 2016 IEEE International Symposium on Multimedia (ISM), San Jose, CA, USA, 2016, pp.399-400, doi: 10.1109 / ISM.2016.0089, which is incorporated herein by reference in its entirety. Each tile can be encoded independently. At the decoder, received tiles can be selected and assembled to have one conforming HEVC bitstream to re-use the existing hardware / software of the HEVC decoder.

[0037] It is noted that the file format corresponding to the “.NAL” extension (ISO / IEC 14496- 15) allows a “tile” sub track, which specifies how to store a tiled HEVC bitstream into multiple tiletracks. This feature allows tile adaptation in the pertinent file format (such as mp4) to be very flexible.

[0038] FIG.1 an example of using multimedia partitions for video encoding. In the example shown, a captured video frame (100) is partitioned into nine tiles (110ij) arranged in three rows and three columns, where i=1, 2, 3 and j=1, 2, 3. Each tile (110ij) is encoded independently. A region of interest (ROI) includes the tiles (1101,2, 1102,2) and may be treated as having a higher priority due to its visual importance. In various additional examples, different numbers of tiles (110ij) can be used.

[0039] The multimedia partition concept illustrated in FIG.1 can be extended to an assembled image, e.g., containing multiple individual images. In some examples, an image can be partitioned into several blocks, and each block can be resampled to a different respective resolution and processed with a different respective image-processing operator. Parts of the assembled image can be encoded and decoded independently.

[0040] FIG.2 is a block diagram illustrating application of the above-explained “block” concept to a 3D neural network model (200) according to one example. In some applications, the 3D model (200) is used to represent a 3D scene which is defined by the cubic range (outlined by the black line boundaries 212. In the example shown in the left portion of FIG.2, the 3D model (200) uses a single relatively large neural network. Using the above-mentioned block concept, the 3D model (200) is partitioned (216) into 64 smaller 3D models (210ijk), where i=1, 2, 3, 4; j=1, 2, 3, 4; and k=1, 2, 3, 4. Each of the smaller 3D models (210ijk) is implemented with a corresponding Neural Radiance Field (NeRF) employing a respective multilayer perceptron (MLP). Each MLP can be decoded and rendered independently. Depending on the context and / or preferences, each MLP may be given a different respective priority, e.g., based on its perceptual impact on the 3D model (200). Rateless Coding

[0041] Conventional FEC coding typically uses a fixed coding rate between the source packets and parity packets. Once the coding rate is determined and the parity packets are generated, the error protection strength is set and cannot adapt to varying channel conditions. In contrast, a rateless code can generate a variable number of coded packets. As long as the number of successfully received packets is no less than the number of source packets, the receiver will be able to recover the source information. The transmitted number of coded packets may depend on the real-time channelconditions, and there is no need to re-encode the FEC-coded stream or prepare multiple versions of the FEC-coded stream with different protection strengths. Representative examples of rateless codes include, but are not limited to, Luby transform (LT) codes, Raptor codes, rateless fountain codes, rateless spinal codes, and RLNC codes. For illustration purposes and without any implied limitations, some examples are described below in reference to RLNC codes. Based on the provided description, a person of ordinary skill in the pertinent art will be able to make and use other examples in which other types of rateless codes are used.

[0042] The RLNC code, which provides only one representative example of rateless coding, is described in Dejan Vukobratovic and Vladimir Stankovic, “Unequal Error Protection Random Linear Coding Strategies for Erasure Channels,” IEEE TRANSACTIONS ON COMMUNICATIONS, VOL.60, NO.5, MAY 2012, pp. 1243-1252, which is incorporated herein by reference in its entirety. RLNC achieves the rateless property by applying a random linear combination of source packets with coefficients randomly selected from a given finite field GF(2q).For example, by given the source message ^ = [^^, ^^, … , ^^^^], a set of randomly selected elementfrom GF(2q) as ^(^) = [^(^)^ , ^(^)^ , … ,new coded packet is constructed as(1)The resulting coded packet has the same length as the source packet L. Each coded packet has the information for the coefficients (^) to facilitate the decoding. Note that it is also feasible to use a pseudo random number generator to signal the coefficients, and we just need to transmit the random-generator seed.

[0043] We can generate multiple ^(^)with different random. At the decoder, as long as we receive C packets, one can organize those C coefficients and the received signal into the following matrix / vector form:An abbreviated form of Eq. (2) is:^ = ^^ (3)One can use a Gaussian elimination method to recover ^. Also note that M can be much larger than C.Multiple Network Sources to Single End-User -- Scenario 1

[0044] FIGS.3-5 are block diagrams illustrating delivery of content from multiple network sources (3100, 3101, 310N-1) to a single end-user device (302) according to one example. Each of the network sources (3100, 3101, 310N-1) uses one respective communication path of a plurality paths (3200, 3201, 320N-1) to the end-user device (302). Three different stages (300, 400, 500) of information flow between the network sources (3100, 3101, 310N-1) and the end-user device (302) are separately illustrated in FIGS.3-5, respectively.

[0045] Referring to FIG. 3, during the stage (300), each of the network sources (3100, 3101, 310N-1) provides a respective description of the multimedia content and network conditions to the end-user device (302). Upon receiving the descriptions from the network sources (3100, 3101, 310N-1), the end-user device (302) operates to perform resource-allocation optimization, e.g., to determine how to protect the source packets and further determine which specified packets are to be transmitted from which of the network sources. In other words, during the stage (300), the end-user device (302) determines the packet assignment for various ones of the network sources (3100, 3101, 310N-1).

[0046] Referring to FIG. 4, during the stage (400), the end-user device (302) operates to communicate relevant portions of the packet assignment plan determined during the stage (300) to different ones of the network sources (3100, 3101, 310N-1).

[0047] Referring to FIG. 5, during the stage (500), the network sources (3100, 3101, 310N-1) send through their corresponding paths (320n), to the end-user device (302), the packets specified in the packet assignment plan received from the end-user device (302) during the stage (400). In various examples, the sent packets also contain the decoding information used in network coding and how to assemble the multimedia blocks back to the original format. Note that, since a rateless code is used for transmitting the packets, there is no need to apply different FEC codes for different network conditions. Instead, the corresponding network source (310n) can generate a plurality of coded packets and then select a certain number of coded packets to send based on the current condition of the corresponding path (320n). Multiple Network Sources to Single End-User -- Scenario 2

[0048] FIG.6 is a block diagram illustrating a multimedia delivery system (600) that can be used to deliver prioritized multisource media to an end-user device (602) according to one example. In the example shown, the system (600) includes an edge network (620) and a core network (640) connected to the end-user device (602) as indicated in FIG.6. The edge network (620) includes an edge server (610) configured to receive and decode prioritized multisource blocks or tiles of content from the edge network (620) and the core network (640). The edge server (610) then operates to translate the decoded blocks or tiles into a single-path flow with dynamically generated coded error- protection packets and further operates to transmit a resulting data stream to the end-user device (602).

[0049] As an illustration, FIG. 6 shows ten blocks of content (block 0, block 1, …, block 9) that are being delivered to the end-user device (602), with each such block having 100 coded or some combination of coded and systematic symbols. The blocks (block 0, block 1, …, block 9) are distributed over four different queues representing four different respective priority levels. The blocks from the different priority queues are applied to a plurality (650) of network-based just-in- time (JiT) encoders (P1, P2, P3, P4), with each of the priority queues being fed into a corresponding one of the encoders (P1, P2, P3, P4). A plurality of encoded data streams generated by the encoders (P1, P2, P3, P4) are then routed via different respective paths through the core network (640) towards the edge network (620), the edge server (610), and the end-user device (602).

[0050] In some examples, the above-described concepts can be adapted for video on demand (VOD) applications. In such examples, the encoded data streams are routed via different paths through the core network and cached / stored in a content delivery network (CDN) or functional equivalent thereof. The client then makes requests to the CDNs to download the packets per its packet assignment scheme and based on one or more pertinent parameters such as cost, quality, etc. In such examples, suitable non-JiT processes can be used. In some embodiments, a multimedia file / data content is encoded into variants, which are distributed across CDNs ahead of time. Such embodiments may also benefit from at least some features described below in reference to FIG.7.

[0051] FIG.7 is a block diagram illustrating a multimedia delivery system (700) that can be used to deliver prioritized multisource media to a plurality of end-user devices (702) according to another example. The system (700) includes an edge network (720) and a core network (740) connected to the end-user devices (702) as indicated in FIG. 7. The system (700) also includesmultisource encoders (P1, P2, P3, P4) configured to pre-code blocks of content (block 0, block 1, …, block 9) variously prioritized using four different priority levels. The resulting multisource packets (750) generated by the multisource encoders (P1, P2, P3, P4) are then variously distributed over various devices of the core network (740) and the edge network (720), which are chosen based on the evaluation of the cost to store and / or deliver versus provided performance. The distributed packets are then used to deliver the corresponding multimedia to the plurality of end-user devices (702), e.g., as described in more detail below.

[0052] Let us assume that there are N media sources in the system (700) that are available / reachable via one or more protocols and by one or more multimedia clients, such as the end-user devices (702). Each of the N media sources is characterized by a respective (e.g., unique) set / combination of costs associated with transmitting / delivering multimedia traffic (such as packet(s) / stream(s)) in response to a request from one or more clients. Examples of such costs include, but are not limited to, the operating cost per Gigabyte of multimedia data stored and / or egressed from a source, performance related costs including various suitable metrics, such as sustained egress capacity in bits-per-second over a finite timeframe / location, response time, availability, reliability, and the like.

[0053] Based on all of the cost / performance metrics or a selected subset thereof, a performance / cost parameter w is computed and associated with each of the N media sources in the system (700). In one example, the value of w can be represented in Cartesian form, in which a first component (e.g., the y term) represents the normalized operating cost, and a second component (e.g., the x term) represents the normalized performance of the source node. In some examples, the value of the performance / cost w for each source is updated, e.g., with frequency corresponding to a chosen length of the time interval between updates. In one example, the performance / cost parameter value for each network media source can be pre-determined (prior to use by the multimedia clients) and signaled. In another example, the performance / cost parameter value is determined independently, e.g., on-the-fly, by each multimedia client making requests to one or more available sources. In yet another example, a hybrid approach can be utilized, in which the initial cost / value parameter value for each source is communicated to the client via an MPEG Dynamic Adaptive Streaming over HTTP (DASH) or HTTP Live Streaming (HLS) manifest and subsequent performance / cost indices are computed locally by the corresponding multimedia client. Some embodiments may benefit from at least some features disclosed in one or more of the followingdocuments: (1) ISO / IEC 14496-12:2020, Information technology -- Coding of audio-visual objects - - Part 12: ISO base media file format; (2) ISO / IEC 14496-15:2021, Information technology -- Coding of audio-visual objects -- Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format; (3) ISO / IEC 23000-19:2018, Information technology — Multimedia application format (MPEG-A) -- Part 19: Common media application format (CMAF) for segmented media, and (4) ISO / IEC 23009-1(2019), Information Technology -- Dynamic Adaptive Streaming over HTTP (DASH) -- Part 1: Media Presentation Description and Segment Formats. Each of these documents is incorporated herein by reference in its entirety.

[0054] In some examples, based on the respective performance / cost value associated with each of the N active media sources (across the cloud and / or networks (720, 740)), the one or more clients (702) operate to continuously optimize the operating cost (e.g., via linear programming) and reliability while maintaining a target quality of experience with respect to the multimedia rendered on the clients. This approach beneficially reduces the burden of dynamically inserting redundant information into every active stream, which may cause inefficiency and implementation and operational complexities on the sender / server side.

[0055] In some examples, the workflow implemented in the system (700) includes three stages. Stage 1: partitioning and encoding a multimedia content into a prioritized set / sequence of blocks and / or tiles, e.g., as explained above in the subsection entitled “Multimedia Content Partition.” Stage 2: a rateless coder (e.g., xCD-1, RLNC, RaptorQ) is introduced to generate Y multisource representations for the prioritized (source) coded blocks or tiles from Stage 1. In some examples, an intermediate step includes partitioning the source coded tile data into groupings of smaller data units (e.g., symbols) for the rateless coder to properly generate N functionally equivalent copies for distribution. Herein, the term “groupings” refers to constraining the rateless coder to operate over some desired number of symbols called a generation or a block. In one example, the number of symbols in a generation or block can be equivalent to the size (in bytes) of an HLS or MPEG DASH segment or a Group of Pictures (GOP), or similar demarcation, etc.Stage 3: each of the prioritized multisource representations is distributed to a plurality of sources across the network (e.g., CDN PoPs, Peers, Storage, etc.) according to a desired strategy.

[0056] In one example, Y multisource representations are generated for each prioritized source coded tile. In another example, Y multisource representations are generated for prioritized groupings of source coded tiles. Note that, in at least some examples, each of the Y representations is considered unique and functionally equivalent (as noted above), thereby enabling efficient recovery of the source coded block or tile information when one or more multimedia clients (702) are simultaneously requesting multisource media, from the N sources over multiple (non- synchronous) network pathways including multi-access pathways.

[0057] In an example, one unique set of multisource representations for high priority tile(s) may be distributed and cached in a network media source when the value of w exceeds a selected threshold value (e.g., where the network source offers high performance but comes with a high operating cost) while another set of unique multisource representations for the same high priority tile(s) can be distributed and cached in a different network media source when the value of w falls below the selected threshold value (e.g., where the network source, in this case, offers lower performance at a lower operating cost).

[0058] The use of the rateless coding for generating multiple sets of functionally equivalent representations of the prioritized video tiles enables a client (702) to be free of scheduling downloads from multiple sources without downloading redundant information. Additionally, some examples may improve reliability without the coding overhead often associated with inserting protection packets into a single source stream.

[0059] By coupling the multisource coding with prioritized video tiles and distributing those coded tiles across various network nodes with a plurality of cost / performance values enables a client (702) to improve its ability to independently optimize operational cost vs. quality. In other words, the client (702) has a substantially larger operating envelope of sources to choose from during operation, which can be chosen without fear of downloading redundant information or complexities associated with switching and scheduling downloads from each source.

[0060] The system (700) illustrated in FIG.7 is a representative example of a deployed topology in which the output of the content partitioning / tiling step (Stage 1) described above generates prioritized (and, in some cases, downsampled) blocks of partitioned / tiled video data representing a user / system defined time interval. Each of the prioritized blocks is indexed to mark the coded symbol boundaries needed for executing the multisource coding process. In the example shown, there are four multisource encoder processes run by the multisource encoders (P1, P2, P3, P4). These encoder processes utilized are mapped to four priority levels corresponding to the earlier portioning / tiling process. In general, if four priority levels are generated, then up to four multisource encoder processes can be used. Each of the priority mapped multisource encoders (P1, P2, P3, P4) can generate one or more multisource coded variants, e.g., using a JiT process as illustrated in FIG.6 or an offline process as illustrated in FIG.7. In different examples, the number of coded variants for each priority level can be fixed or dynamically varied, with the choices being dependent on the downstream distribution topology of the corresponding system. Other considerations include information-source (e.g., CDN, storage, peers, etc.) attributes, such as cost, performance (e.g., historical and / or current throughput, historical and / or current throughput variance), location (e.g., geo, data center, ISP, etc.), source diversity, access network type, client- server round trip times, and the like.

[0061] The coded variants (750) generated by the multisource encoders (P1, P2, P3, P4) represent functionally equivalent copies of the prioritized sources created during the partitioning / tiling step (Stage 1), with the multisource coded symbols in each block (in a variant) being useful for recovery of source data. In some examples, each of the multisource coded variants can be generated in a deterministic manner (using a predefined set of coding coefficients) or by randomly generating the coding coefficients to minimize redundant information.

[0062] In some examples, the operations illustrated in FIGS. 3-5 can be included into Stages 1- 3, respectively. For example, in Stage 1, the network source provides the multimedia content information and network conditions to the end device (702), e.g., as illustrated in FIG. 3. Upon receiving the information, the end device (702) will perform resource allocation optimization to determine how to protect the source packets and which network source is going to transmit the specified packets. In other words, in Stage 2, the end device (702) determines the packet assignment for all network sources. After optimizing the resources, the packet assignment plan is sent to each network source, e.g., as illustrated in FIG.4. The network sources receive the packet assignmentplan and send the specified packets through their own paths to the end device in Stage 3, e.g., as illustrated in FIG.5. Example Embodiment: Single Multimedia Content

[0063] In this section, we consider an example use case in which the end-user device (302, 602, 702) requests to transmit one multimedia content (MC), e.g., a 2D or 3D video, from multiple network sources (NSs). We will first present the adopted notations for discussed objects / quantities and example mathematical constructs that can be used to compute pertinent probabilities. We then consider several use cases and propose the corresponding practically acceptable (e.g., nearly optimal) solutions for each of the use cases.

[0064] Herein, we consider N network sources (abbreviated as NS) and N transmission paths from each NS to the end user. Let!represent the nthNS. Denote the bandwidth (in terms of the number of packets) for the nthpath as "!, the packet loss rate (indicated by a value from the interval

[0001] ) as #!, and the financial cost to send one packet as $!. The overall cost of packets to carryfrom all NSs is expressed as follows:The considered multimedia content is split into K blocks, e.g., as described above in the subsection entitled “Multimedia Content Partition.” Denote the kthblock as '^, for k=0, 1, …, K-1. The number of source packets for each block is denoted as (^. Denote the incremental multimedia distortion reduction as )^when we can recover block k. If we cannot successfully recover block k, then the distortion reduction )^is )^=0.

[0065] In each NS, for each block, we will generate the coded packet using RLNC (discussed above) to provide error protection. Note that the coded packets in different NSs are preferably different to fully utilize the properties of the rateless code. Denote the number of packets for the kthblock sent by the nthNS as *^!. We can also arrange all {*^!} together as a matrix +, which represents the packet assignment plan. Note that a packet can be lost or become invalid owing to late arrival. Denote the number of packets successfully received for the kthblock sent by the nthNS as ,^!. The {,^!} elements can similarly be arranged to form a corresponding matrix .. The matrixelements of the matrices A and R are subject to the following inequality:

[0066] We can use a binary variable, 3^!, to represent whether the block k is stored at the nthNS. In one convention, 3 = 1 means that the kth block th^! is stored in the n NS, whereas 3^! = 0indicates that the kthblock is not stored in the nthNS. A matrix, 6, can be built to represent the overall storage distribution for K blocks in N network sources. A vector, 7^, is constructed to represent the storage assignment for the block k over all NSs:Denote the number of coded packets sent from all NSs for the block k as 8^:Owing to packet loss or delay greater than a deadline, the actual number of coded packets successfully received from all NSs for block k is 9^:The total number of coded packets sent from all NSs for the kth block is :^:: = (7 );^ ^ 8^ (9)For the kthblock to be successfully decoded, we need to at least successfully receive (^packets,where:(^ ≤ (7^);9^ ≤ (7^);8^ = :^ (10)

[0067] For a given 8^and packet assignment for the block k, the probability that the end userreceived exactly λ packets from all N paths is #(8^ , =):One can apply the de Moiver-Laplace theorem to approximate a binomial distribution of Eq. (11) with a normal distribution as follows:where \^!(*^!) = *^! ∙ #! (13)]P^! (*^!) = *^! ∙ #! ∙ (1 − #!) (14)

[0068] The sum of N normally distributed independent variables with means {\^!(*^!)} andvariances {]P^! (*^!)} follows a normal distribution with the mean and variance expressed as:Using the above, we can approximate #(8^ , =) as follows:Using this approximation, one can express the probability that the end device (602, 702) receives at least (^packets from all paths with the packet assignment 8^(where the total number of packets for block k is :^) as follows:whereThe approximation of ^(8^ , (^) given by Eqs. (18), (19) provides a convenient and fast way toapproximately compute the block recovery probability without extensive computations.

[0069] The expected distortion reduction for block k under packet assignment plan 8^ is)i^(8^) = ^(8^, (^))^ (20)For all blocks in all paths, we have the packet assignment plan + and the expected distortion is)i(+):where )!jkkrepresents the picture quality without any packet received. It is a constant and, in some cases, it can be omitted in the formulation of the optimization process. Under this packetassignment plan, the total network cost $(+) can be expressed as:

[0070] Using the above notations, derivations, and approximations, the problem of finding an optimal packet assignment plan + can be formulated as follows: Problem Formulation (P1) The objective function of this optimization problem can be formulated to minimize the distortion with a weighting factor (m) to reduce the cost by finding the optimal packet assignment plan +: n(+) = )i(+) + m ∙ $(+) (23)Subject to bandwidth constraint: ∑l^^^^^ *^! ≤ "! ∀2 (24)Note that the solution to the problem (23)-(24) is NP hard. We thus describe several approximate solutions to this problem for several different scenarios.

[0071] FIG.8 is a block diagram illustrating a first scenario corresponding to the Single Multimedia Content (SMC) embodiment according to some examples. For illustration purposes, the first scenario is described in reference to a system configuration previously illustrated in FIGS.3-5. A person of ordinary skill in the pertinent art will readily understand that the first scenario can also be realized in other system configurations, such as those of the systems (600, 700) illustrated in FIGS.6, 7.

[0072] In the example shown, each of network sources (NS0, NS1, …, NS(N−1)) contains copies of all pertinent K blocks (8020, …, 802K-2, 802K-1) of the SMC. As a result, all matrixelements in the above-described matrix 6 are 3^! = 1. Finding the optimal solution to theproblem P1, Eqs. (23)-(24), is NP-hard. In one example, an acceptable approximation to the optimal solution is obtained via a fast greedy (sub-optimal) algorithm described below.

[0073] To facilitate our discussion of the fast greedy algorithm, we will sort the network sources (3100, 3101, 310N-1) according to the packet loss rate ascendingly and reorder the NS indices {n}such that:#^ ≤ #^ ≤ #^ ⋯ ≤ #&^^ (25)We also sort the multimedia blocks (802k) according to the priority, or more specifically, according to the distortion reduction measurement in the descending order and reorder the block indices {k}such that:)^ ≥ )^ ≥ )P ≥ ⋯ ≥ )l^^ (26)Observing the final-distortion equation Eq. (21), one can see that it is a summation of block recovery rate multiplied by the distortion reduction from each block. Accordingly, to provide higher utilization of the packets, it is preferrable when a higher distortion-reduction block (802k) is sent through a path (320n) characterized by a lower packet loss rate.

[0074] FIGS.9-14 are block diagrams illustrating operations of a fast greedy algorithm (900) applied to the SMC transmission according to one embodiment. The algorithm (900) includes forming a 1D packet array (902) by concatenating the packets corresponding to the above-described packet-loss-rate sorted NS indices in an algorithm block (900a). The packet array (902) is depicted in FIG.9. We use a notation according to which the combined set of packets is denoted as q^. The packets from the network source (NS0) have the indices q^to qrL^^; the packets from the network source (NS1) have the indices qrLto qrLsrJ^^; the packets from the network source (NS2) have the indices qrLsrJto qrLsrJsrS^^, and so on. In the packet array (902), the lower the packet index, the smaller the packet-loss rate.

[0075] To facilitate further discussion of the algorithm (900), we define a packet counter indext with initial value as 0. Various operations of the algorithm (900) directed at finding anapproximate solution to the problem P1, Eqs. (23)-(24), are further illustrated in FIGS.10-14. An initialization step of the algorithm (900) is performed in accordance with Eqs. (27), (28) as follows:t = 0 (27)*^! = 0 ∀1, 2 (28)The algorithm (900) further includes three additional blocks (900b, 900c, 900d) of operations illustrated in FIGS.10-14. More specifically, the block (900b) includes a baseline packet assignment illustrated in FIG.10. The block (900c) includes a batch packet assignment illustrated in FIGS.11 and 12. The block (900d) includes a sequential individual packet assignment illustrated in FIGS.13 and 14.

[0076] Referring to FIG. 10, operations of the block (900b) include assigning (^packets with indices from q^to q`L^^to block 0, denoted as '^. The block '^is needed to obtain the minimum (baseline) quality of the delivered content. No playback can be performed at the end device (302) if the block '^is not delivered. Note that, in some examples, the number (^may be larger than $^. In such examples, multiple NSs are used to carry the packets of the block '^.

[0077] Operations of the block (900c) will now be described. Some of the operations of the block (900c) are illustrated in FIGS. 11 and 12. Operations of the (900c) are directed at determining, for the kthblock in the remaining of K−1 blocks, whether to send (^packets of the kthblock or use the corresponding packet budget to increase error protection of the blocks '^to '^^^. Note that, since the decoding of the k-th block requires at least (^packets, sending a number of packets that is smaller than the number (^is not going to help with the perceived quality of the received content at the end device (302).

[0078] In one example, operations of the block (900c) include: For k = 1 to K−1, •Compute the current available packet budget % − t.• The number representing the packet budget is updated q = miny(^ , % − tz.• (A) Compute an overall cost if assigning { packets to the kthblock. Update {*^!} and the corresponding matrixThe corresponding cost value is computed as• (B) Compute an overall cost by sequentially assigning each packet among the { packets to the block k’, where 0 ≤ 1} < 1. For j = 0 : 1 : q-1. The computations include:o (B1) For each block 1}, assign packet q^s^to this block. Update matrix +(^)(^)X^^^^Z(^^) and compute the cost value as no (B2) Select the block (1′(^)∗) with a lowest cost and assign q^s^to block 1′(^)∗.oRepeat (B1), (B2) for each of the q packets. Denote the corresponding matrix A as+(^)(^)(^) and cost value• From the sets of operations (A) and (B), we have two cost values and select the configuration with the lowest cost, e.g., as follows:•If the selection is to send the block k, (i.e., ^#d∗ == ^), then the algorithm (900) performsthe packet assignment and move to the next block by incrementing the k value to k=k+1. This intermediate outcome of the block (900c) is illustrated in FIG. 11 using the block '^as an example. •If the optimal setting is to add more coded packets to the previous blocks (i.e., ^#d∗ == '),then the algorithm (900) performs the packet assignment but remains at the block '^, i.e., the k value is not incremented. The above-described procedure is then repeated. This intermediate outcome of the block (900c) is illustrated in FIG.12 using the block '^as an example. •Update *^! and set t = t + {.• If t ≥ %, then all packets have been assigned. Terminate the method (900).

[0079] In some cases, after the above-described operations of the block (900c) are completed,there might remain (% − t) unassigned packets. This final outcome of the block (900c) is illustratedin FIG. 13. After this outcome of the block (900c), the algorithm (900) operates to sequentially determine the assignment for each unassigned packet in the block (900d). In one example, operations of the block (900d) include:For i = (% − t) : 1 : (% − 1),• Operations of the block (900d) are configured to determine which block '^will get the packet slot q^. •For the current packet assignment of each block k, assign q^ to the block k to have the matrix+(P)(^)(^) and compute the corresponding cost value• For all blocks'^, the algorithm computes K cost values and then selects the block with the lowest cost, e.g., as follows: 1(P)∗ = *,q (P)(^)(^)^^y^ m,^,…in,l^^z nX+ Z (31)• Assign q^to block 1(P)∗. •Update *^! and set t = t + 1• Repeat the above operations of the block (900d) for the updated t. An example final outcome of the algorithm (900) for cases including operations of the block (900d) is illustrated in FIG. 14.

[0080] FIG.15 is a block diagram illustrating a second scenario corresponding to the Single Multimedia Content (SMC) embodiment according to some examples. In the example shown, each of the network sources (NS0, NS1, …, NS(N−1)) stores different respective (e.g., unique) blocks (802k) of the SMC. An individual block (802k) is stored in a corresponding single network source (NSn). However, some of the network sources (NS0, NS1, …, NS(N−1)) may have multiple respective blocks (802k) stored therein. As an example, two network sources (NS0, NS(N−1)) are shown in FIG.15 as storing a single corresponding block, with the network source (NS0) storing the block (8020) and the network source (NS(N−1)) storing the block (802K-1). The network source (NS1) is shown in FIG.15 as storing more than one block, illustratively the blocks (8021, 8022).

[0081] The packet assignment algorithm applicable to the scenario illustrated in FIG. 15 can also be applied to a special case in which there are multiple network sources storing a copy of the same block(s). To adapt this case to the scenario illustrated in FIG. 15, we merge those multiple network sources into a single combined network source. This combined network source is then given a single index and is treated by the packet assignment algorithm as a single network source analogous to the other network sources having respective unique blocks of SMC stored therein.

[0082] In one approach, the packet assignment problem corresponding to the scenario illustrated in FIG.15 can be decomposed into N sub-problems. Each of the N sub-problems is then solved individually according to the number of stored blocks for each of the network sources (NS0, NS(N−1)). For example: • For the network sources storing only one respective block (802k) of the SMC, the solution is straightforward: the corresponding entire path (320n) is used to carry all coded packets for that one block. • For the network sources storing more than one block, the packet assignment problem becomes: how to assign the number of packets to each block to achieve an optimal result.

[0083] For the latter type of problem, one can reuse the algorithm (900) after relatively small modifications thereof, e.g., including different (e.g., simpler) settings. For example, since in this case N=1, we can redefine the index n and treat the corresponding network source as NS0 and also redefine the stored block indices k to 0 through K−1, where the meaning of K here is changed to mean the number of stored unique blocks in the corresponding NS. After such redefinition, the above-described algorithm (900) can be used to solve the packet assignment problem for the scenario illustrated in FIG. 15.

[0084] FIG.16 is a block diagram illustrating an example outcome of the algorithm (900) in the scenario illustrated in FIG. 15. The packet assignment illustrated in FIG. 16 is qualitatively similar packet assignment illustrated in FIG.14 for a different scenario. One difference between these two packet assignments is that, in the packet assignment illustrated in FIG. 16, all scheduled packets correspond to a single network source (illustratively NS0) whereas, in the packet assignment illustrated in FIG.14, the scheduled packets correspond to multiple network sources. This difference is apparent from a comparison of the upper bars in FIGS.14 and 16, which indicate the corresponding network sources for different sets of scheduled packets.

[0085] FIG.17 is a block diagram illustrating a third scenario corresponding to the Single Multimedia Content (SMC) embodiment according to some examples. In the example shown, some of the blocks of the SMC are stored in multiple ones (but not all) of the network sources (NS0, NS1, …, NS(N−1)). As an example, two network sources (NS0, NS1) are shown in FIG.17 as storing respective copies of the block (8020). Another two network sources (NS1, NS(N−1)) are shown in FIG.17 as storing respective copies of the block (8021). The network source (NS0) does not have acopy of the block (8021) stored in the network sources (NS1, NS(N−1)). The network source NS(N−1) does not have a copy of the block (8020) stored in the network sources (NS0, NS1). Neither of the network sources (NS0, NS1) has a copy of the block (802K-1) stored in the network source (NS(N−1)).

[0086] In one approach to the packet assignment problem corresponding to the scenario illustrated in FIG.17, the corresponding packet assignment algorithm is configured to attainsuccessful packet decoding for each block (802k), namely:Let us denote the set of network sources containing the block k as ^^. In each ^^, a first step is to sort the network sources according to the packet loss rate in the ascending order. In other words, the network sources listed closer to the beginning of sorted set have progressively lower packet loss rates. We also denote the set of blocks stored in the network source NSn asTo facilitate the discussion, in each network source NSn, we define a packet counter index t!with the initial value of zero.

[0087] In the following, we describe a 4-step algorithm that can be used to solve the packet assignment problem for the scenario illustrated in FIG. 17. This algorithm is referred to as Algorithm 2 or A2. Operations of the algorithm (A2) are grouped into four blocks of operations referred to as STEP 0, STEP 1, STEP 2, and STEP 3, respectively.

[0088] STEP 0: Initialization. t! = 0 (33)*^! = 0 ∀1, 2 (34)

[0089] STEP 1: Baseline Assignment. For the block 0, • Assign (^packets from ^^. Note that (^might be larger than the number of packets supported in the first network source in ^^. In such cases, block 0 needs more than one network source to transmit. • Update *^!. • Update packet counter (t!) for each network source in ^^.• Each network source has the number of unassigned packets expressed as ($! − t!).

[0090] STEP 2: Assign New Block and Maintain Priority Order. For k = 1 : 1 : K−1, (2A) Assign (^packets from ^^if there is enough packet space. • Compute packet budget• If enough space, then assign (^packets from ^^o Update *^!o Update packet counter (t!) for each NS in ^^. oEach NS has number of unassigned packets as ($! − t!).• Else go to (2B) (2B) Maintain Priority Order for blocks 0 to k, i.e. ^(8^, (^) ≥ ^(8^, (^) ≥ ⋯ ≥ ^(8^, (^) (37)We assign new packets from block k-1 back to block 0. For each block i, we add new packets such that ^(8^, (^s^) < ^(8^, (^s^). After assigning packets for block 0 ~ k, we maintain thepriority order (37). •Compute ^(8^ , (^)• j = k • For i = k-1 : -1 : 0 o If we still have packet budget to assign packet for block i, i.e.owhile^ assigning one new packet for block i, update^compute ^(8^, (^s^)^ update t!At the end of STEP 2, collect all network sources that have packets assigned as set Ω.

[0091] STEP 3: Assign Packets of Largest Distortion Reduction Block in each network source. There might be some unassigned packets for each network source from previous steps. For thenetwork source NSn, the algorithm (A2) operates to identify the common blocks in both setsΩ Then, the algorithm (A2) operates to assign those ($! − t!) packets to the blockhavingthe largest distortion reduction )^ among Ω ∪ ^!. By doing so, we increase the successful packetrecovery rate for the impactful blocks. Example Embodiment: Multiple Multimedia Contents

[0092] In this section, we consider an example use case in which the end-user device (302, 602, 702) requests to transmit two or more multimedia contents (MCs) from multiple network sources. The number of different multimedia contents in the network sources is denoted as U. For illustration purposes and without any implied limitations, some examples provided below correspond to U=2, i.e., two different multimedia contents represented by the index u. In general, the index u takes values from o to U−1.

[0093] FIG.18 is a block diagram illustrating a scenario corresponding to the Multiple Multimedia Content (MMC) embodiment according to some examples. For illustration purposes, this scenario is described for U=2 in reference to a system configuration previously illustrated in FIGS.3-5. A person of ordinary skill in the pertinent art will readily understand that this scenario can also be realized in other system configurations, such as those of the systems (600, 700) illustrated in FIGS.6, 7. In the example shown, each of network sources (NS0, NS1, …, NS(N−1)) contains copies of all pertinent blocks (802u,0, …, 802u,K-2, 802u,K-1) of the MMC, where u=0, 1.

[0094] To introduce the U dimension in addition to the previously introduced N and K dimensions, we add the subscript, u, to the notations to represent the uthmultimedia content. For the uthmultimedia content, we denote the kthblock as 'j^, the corresponding distortion measurement for that block as )j^, and the corresponding number of source packets for that block as (j^. Wealso denote the number of packets for the kthblock of the uthmultimedia content by the nthnetworksource NSn as ^j^!. We collect all {^j^!} together to form a 3D matrix ^ (^ × ^ × ^). Thepacket assignment for the uth multimedia content is a 2D matrix +^ (^ × ^). The packet assignmentfor the block of the uth multimedia content is a 2D matrix 8^^ (^ × 1).

[0095] The expected quality for the uth multimedia content can be expressed as follows:where )j,!jkkrepresents the picture quality without any packets received. Two possible overall objectives for a corresponding packet assignment algorithm are discussed in more detail below. Objective 1 (O1) is to approximately minimize the overall distortion of the received MMC. Objective 2 (O2) is to approximately minimize the maximum observed distortion of each individual multimedia content of the received MMC.

[0096] For the objective (O1), an example case that we consider involves minimizing the overall received distortion subject to the bandwidth constraint by finding an optimal packet assignment ^.In this case, the total expected quality can be expressed as:The corresponding total network cost is $(^). The packet assignment problem for thecorresponding algorithm to solve is formulated as an optimization problem as follows:

[0097] Since the goal is to minimize the overall distortion of the received MMC, we can collectall ^ × ^ source blocks {'j^} together and sort them according to )j^ in descending order. Wecan vectorize the block index and rename each block’s index according to the priority derived fromthis order as follows:The MMC algorithm can then be configured to treat those ^ × ^ blocks as blocks representing asingle multimedia content (SMC). Accordingly, the corresponding packet assignment problem can thereafter be solved using substantially the same approach as that applied above to the SMC embodiment. For example, the problem formulation can be in accordance with Eqs. (23)-(24) presented for the problem formulation (P1). The corresponding problem can then be solved, e.g., using a suitable embodiment of the algorithm (900) as described above.

[0098] For the objective (O2), an example case that we consider involves minimizing the maximum observed distortion of each individual multimedia content of the received MMC. We assume that the system continuously uses up all bandwidth from all network sources. Under this assumption, the network cost remains constant and can be removed from the objective function used for optimization.

[0099] In one example, the corresponding optimization problem can be formulated as follows:We will group together the blocks from the MMCs which reduce to similar amount of distortion into one superblock under perfect network condition. By doing so, we can build a single super multimedia content with multiple super-blocks. Then, we can solve the problem (43) using the same approach as that presented for the problem formulation (P1). For example, this problem can be solved using a suitable embodiment of the algorithm (900). Such an embodiment involving a two- step algorithm is described in more detail below.

[0100] We will now explain the superblock building process according to one example. Let us denote as )£j^the amount of distortion when we receive block 0 to block k for the multimedia content u under perfect network conditions, i.e., without any packet losses. The distortion )£j^canbe expressed as follows:There is a total of (^ × ^) of us among all {)£j^} as)£¤Eh = max y)j,!jkk ∀2z. We also use a counter {j^ to count the block k in the MMC u to indicatewhether or not the block k is already included in at least one superblock. The initial values of all {{ } are 0. We can create T super-blocksj^. Let us denote the collection of blocks in the superblock as ¥band the number of blocks in the tthsuperblock as ¦b. To facilitate our discussion, we also denote the accumulated number of blocks from 0thsuperblock to thesuperblock as ¦§b,i.e.:

[0101] In one example, we build the superblocks one by one by sequentially packing ¦bblocks into the tthsuperblock. A main objective during the packing is to ensure that the distortions in all U MMCs are similar. We arrange each MMC’s {)£j^} in the number line and set the target distortion (which is gradually adjusted to become smaller and smaller) to find how many blocks from allMMCs to include in the superblock to ensure that the distortion in each MMC is similar and near the target distortion. By moving the target distortion toward a small value, we build the superblock sequentially, and the blocks in each MMC will provide similar distortion.

[0102] In one example, the above-outlined procedure is implemented based on the following Algorithm (A3): Pseudocode for Algorithm (A3) Initialization Set all ¥b are empty set ∅, d ∈ [0, © − 1]¥ª = ∅t = 0; Set target distortion )« = max y)j,!jkkzSelect ∆«while( t < T ) / / find all the blocks having the distortion no smaller than )«as set ¥«¥« = y(­, 1)|)« ≤ )£j^zIf( |¥«| ≥ ¦§b) / / collect enough blocks¥b = ¥«\¥ª / / the new superblock only contains new blocks¥ª = ¥« / / update the accumulated block set¦b = |¥b| / / update the correct number of blocks for superblock t¦§b = ∑b^^^ ¦b / / update the correct accumulated number of blockst = t + 1 / / move to next superblock else )« = )« − ∆«end end

[0103] FIGS.19-24 are block diagrams illustrating the algorithm (A3) according to one example. FIG.19 shows the overall configuration of the multimedia contents handled by thealgorithm (A3) in the presented example. FIGS.20-24 illustrate progression of the algorithm (A3) toward lower distortion.

[0104] Referring to FIG. 19, in the example shown, there are three MMCs (MC0, MC1, MC2). Each of the MMCs (MC0, MC1, MC2) has five blocks 'j^. The respective values of u and k are indicated for each of the blocks 'j^inside the corresponding rectangle representing the block. The blocks 'j^are positioned along the “distortion” axis in accordance with the exhibited distortion amounts. The distortion amount reduces from left to right upon sequential addition of more received blocks 'j^for each of the MMCs (MCu). Note that the respective values of )j,!jkkmay ormay not be different for different MMCs. In the example shown in FIG. 19, the respective values of)j,!jkk are different. The designed number of blocks in each superblock is ¦^ = 4, ¦^ = 4, ¦P = 4,and ¦± = 3.

[0105] Referring to FIG. 20, an initial target distortion (TDini) is set to the largest )j,!jkkamong the MMCs (MC0, MC1, MC2). In the example shown, the largest )j,!jkkis exhibited by the multimedia content (MC1). From that position, the algorithm (A3) is for progressively smaller target distortion values to select blocks 'j^from all of the MMCs (MC0, MC1, MC2) for different superblocks. The number of such blocks in each of such superblocks is set by the values of as ¦§b. In the example illustrated by FIGS.21-24, the designed number of blocks in different ones of thesuperblocks is ¦^ = 4, ¦^ = 4, ¦P = 4, and ¦± = 3, respectively.

[0106] FIG. 21 is a block diagram illustrating the selection of four (¦^ = 4) blocks 'j^ for afirst superblock (SB0). More specifically, the algorithm (A3) moves the TD line gradually from the initial position (TDini) toward lower distortion (i.e., to the right in the depiction of FIG. 21) until thenumber of whole blocks 'j^ to the left of the TD line meets the constraint ¦^ = 4. The position ofthe TD line at that point is denoted as (TD0). The four blocks 'j^located between the TD line positions (TDini) and (TD0) are the blocks B00, B01, B10, and B11. These blocks 'j^are selected forthe 0th superblock. In mathematical terms, the first superblock ¥^ is ¥^ = y00, 01, 10, 11z.

[0107] FIG. 22 is a block diagram illustrating the selection of four (¦^ = 4) blocks 'j^ for asecond superblock (SB1). More specifically, the algorithm (A3) moves the TD line gradually from the position (TD0) toward lower distortion (i.e., to the right in the depiction of FIG.22) until the number of the whole, previously unselected blocks 'j^to the left of the TD line meets theconstraint ¦^ = 4. The position of the TD line at that point is denoted as (TD1). The fourpreviously unselected blocks 'j^located to the left from the TD line position (TD1) are the blocksB02, B12, B20, and B21. As such, the second superblock= y02, 12, 20, 21z.

[0108] FIG. 23 is a block diagram illustrating the selection of four (¦P = 4) blocks 'j^ for athird superblock (SB2). More specifically, the algorithm (A3) moves the TD line gradually from the position (TD1) toward lower distortion (i.e., to the right in the depiction of FIG.23) until the numberof the whole, previously unselected blocks 'j^ to the left of the TD line meets the constraint ¦P =4. The position of the TD line at that point is denoted as (TD2). The four previously unselectedblocks 'j^located to the left from the TD line position (TD2) are the blocks B03, B13, B22, and B23.As such, the third superblock ¥P is ¥P = y03, 13, 22, 23z.

[0109] FIG. 24 is a block diagram illustrating the final partition of the MMCs (MC0, MC1, MC2) into the four superblocks (SB0, SB1, SB2, SB3) for the considered example. The final fourthsuperblock (SB3) has three blocks 'j^ (¦± = 3). As such, ¥± = y04, 14, 24z. Upon completion ofthe partitioning of the MMCs (MC0, MC1, MC2) into the superblocks (SB0, SB1, SB2, SB3), the latter are subjected to packet assignment processing in accordance with the algorithm (900). Example Metadata

[0110] In this subsection, we describe example metadata that can be used to assist the corresponding system with efficient handling of various above-described processes and algorithms. The provided description is structured in accordance with the above-described three stages (300, 400, 500) of information exchange between the network sources (3100, 3101, 310N-1) and the end- user device (302, 602, or 702).

[0111] A first set of metadata relates to resource allocation and is provided from the network sources (3100, 3101, 310N-1) to the end-user device (302, 602, or 702) during the stage (300). In one example, the first set of metadata includes: • Metadata describing multimedia content: o Number of source packets in each block y(^z. o Distortion information for each block y)^z and initial distortion )!jkk. • Metadata describing network conditions: o Bandwidth for each path y"!z.o Packet loss rate for each path y#!z. o Financial cost for each path y$!z.

[0112] A second set of metadata relates to packet assignment and is provided from the end-user device (302, 602, or 702) to the network sources (3100, 3101, 310N-1) during the stage (400). In one example, the second set of metadata includes: • Packet assignment plan for single content: +. • Packet assignment plan for multiple content: ^.

[0113] A third set of metadata relates to block decoding and is provided from the network sources (3100, 3101, 310N-1) to the end-user device (302, 602, or 702) during the stage (500). In one example, the third set of metadata includes: • Packet group and ID: an identification for each packet to indicate which block of which content the packet belongs to. • RLNC decoding coefficients and ID: each block has its own RLNC configuration, and each packet belonging to a different block communicates the RLNC decoding coefficients and which block it belongs to. Example Hardware

[0114] FIG. 25 is a block diagram of an example computing device (2500) configured to perform at least some operations of the above-described methods and procedures in accordance with various embodiments. For example, in some embodiments, the computing device (2500) may perform at least some operations of the network source (310n) or the end-user device (302). In various embodiments, a node of the pertinent system may be implemented by a single computing device (2500) or by multiple computing devices (2500).

[0115] The computing device (2500) of FIG.25 is illustrated as having a number of components, but any one or more of these components may be omitted or duplicated, as suitable for the application and setting. In some embodiments, some or all of the components included in the computing device (2500) may be attached to one or more motherboards and enclosed in a housing. In some embodiments, some of those components may be fabricated onto a single system-on-a-chip (SoC) (e.g., the SoC may include one or more processing devices (2502) and one or more storage devices (2504)). Additionally, in various embodiments, the computing device (2500) may notinclude one or more of the components illustrated in FIG. 25, but may include interface circuitry for coupling to the one or more components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High-Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other appropriate interface). For example, the computing device (2500) may not include a display device (2510), but may include display device interface circuitry (e.g., a connector and driver circuitry) to which an external display device (2510) may be coupled.

[0116] The computing device (2500) includes a processing device (2502) (e.g., one or more processing devices). As used herein, the term “processing device” refers to any device or portion of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. In various embodiments, the processing device (2502) may include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), server processors, or any other suitable processing devices.

[0117] The computing device (2500) also includes a storage device (2504) (e.g., one or more storage devices). In various embodiments, the storage device (2504) may include one or more memory devices, such as random-access memory (RAM) devices (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive-bridging RAM (CBRAM) devices), hard drive-based memory devices, solid-state memory devices, networked drives, cloud drives, or any combination of memory devices. In some embodiments, the storage device (2504) may include memory that shares a die with the processing device (2502). In such an embodiment, the memory may be used as cache memory and include embedded dynamic random-access memory (eDRAM) or spin transfer torque magnetic random-access memory (STT-MRAM), for example. In some embodiments, the storage device (2504) may include non-transitory computer readable media having instructions thereon that, when executed by one or more processing devices (e.g., the processing device (2502)), cause the computing device (2500) to perform any appropriate ones of the methods disclosed herein or portions of such methods.

[0118] The computing device 2500 further includes an interface device (2506) (e.g., one or more interface devices (2506)). In various embodiments, the interface device (2506) may include one ormore communication chips, connectors, and / or other hardware and software to govern communications between the computing device (2500) and other computing devices. For example, the interface device (2506) may include circuitry for managing wireless communications for the transfer of data to and from the computing device (2500). The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data via modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not. Circuitry included in the interface device 2506 for managing wireless communications may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards, Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). In some embodiments, circuitry included in the interface device (2506) for managing wireless communications may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. In some embodiments, circuitry included in the interface device (2506) for managing wireless communications may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, circuitry included in the interface device 2506 for managing wireless communications may operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. In some embodiments, the interface device (2506) may include one or more antennas (e.g., one or more antenna arrays) configured to receive and / or transmit wireless signals.

[0119] In some embodiments, the interface device (2506) may include circuitry for managing wired communications, such as electrical, optical, or any other suitable communication protocols. For example, the interface device (2506) may include circuitry to support communications in accordance with Ethernet technologies. In some embodiments, the interface device (2506) may support both wireless and wired communication, and / or may support multiple wired communicationprotocols and / or multiple wireless communication protocols. For example, a first set of circuitry of the interface device (2506) may be dedicated to shorter-range wireless communications such as Wi- Fi or Bluetooth, and a second set of circuitry of the interface device (2506) may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some other embodiments, a first set of circuitry of the interface device (2506) may be dedicated to wireless communications, and a second set of circuitry of the interface device (2506) may be dedicated to wired communications.

[0120] The computing device (2500) also includes battery / power circuitry (2508). In various embodiments, the battery / power circuitry (2508) may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device (2500) to an energy source separate from the computing device (2500) (e.g., to AC line power).

[0121] The computing device (2500) also includes a display device (2510) (e.g., one or multiple individual display devices). In various embodiments, the display device (2510) may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0122] The computing device (2500) also includes additional input / output (I / O) devices (2512). In various embodiments, the I / O devices (2512) may include one or more data / signal transfer interfaces, audio I / O devices (e.g., microphones or microphone arrays, speakers, headsets, earbuds, alarms, etc.), audio codecs, video codecs, printers, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, etc.), image capture devices (e.g., one or more cameras), human interface devices (e.g., keyboards, cursor control devices, such as a mouse, a stylus, a trackball, or a touchpad), etc.

[0123] Depending on the specific embodiment, various components of the interface devices (2506) and / or I / O devices (2512) can be configured to output suitable signals, receive suitable signals, and receive and output data streams. In some examples, the interface devices (2506) and / or I / O devices (2512) include one or more analog-to-digital converters (ADCs) for transforming received analog signals into a digital form suitable for operations performed by the processing device (2502) and / or the storage device (2504). In some additional examples, the interface devices (2506) and / or I / O devices (2512) include one or more digital-to-analog converters (DACs) for transforming digital signals provided by the processing device (2502) and / or the storage device(2504) into an analog form suitable for being communicated over the corresponding communication channels. Example Workflow

[0124] FIG. 26 is a flowchart illustrating a block-based packet-assignment method (2600) for delivering multimedia according to some examples. The block-based packet-assignment method (2600) includes receiving, at the end-user device (602, 702), a plurality of descriptions in a processing block (2602). The descriptions identify blocks of the multimedia content stored at a plurality of network sources connected to the end-user device (602, 702) via a plurality of network paths (see, e.g., FIGS.6-7). Each of the blocks is encoded using a rateless error correction code. The plurality of descriptions also specifies the network conditions for the plurality of network paths and, in some cases, a respective quality measurement for each block.

[0125] The block-based packet-assignment method (2600) also includes communicating from the end-user device to the plurality of network sources a packet assignment plan in a processing block (2604). The packet assignment plan includes a plan for transmission of packets corresponding to different blocks of the multimedia content from the plurality of network sources to the end-user device (602, 702). The packet assignment plan is determined based on the received plurality of descriptions and further based on an optimization with an objective function. In some examples, the objective function includes: (i) a first component representing an estimated distortion of the multimedia content when received by the end-user device, and (ii) a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device.

[0126] The block-based packet-assignment method (2600) also includes reconstructing the multimedia content in a processing block (2606). The multimedia content is reconstructed using the corresponding blocks assembled from the pertinent sets of packets received at the end-user device from the plurality of network sources. The network sources transmit such sets of packets in accordance with the packet assignment plan communicated thereto in the above-described processing block (2604).

[0127] According to an example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS. 1-26, provided is a block- based packet-assignment method for delivering multimedia, comprising: receiving, at an end-userdevice, a plurality of descriptions identifying blocks of a multimedia content stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicating from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the multimedia content from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on optimization with an objective function including a first component representing estimated distortion of the multimedia content when received by the end- user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstructing the multimedia content using the blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

[0128] In some embodiments of the above method, each of the blocks represents a respective tile of a corresponding video frame.

[0129] In some embodiments of any of the above methods, the plurality of descriptions also specifies a respective quality measurement for each of the identified blocks.

[0130] In some embodiments of any of the above methods, each of the blocks represents a respective partition of a corresponding portion of the multimedia content.

[0131] In some embodiments of any of the above methods, at least one of the respective partitions is a tile of a video frame.

[0132] In some embodiments of any of the above methods, a first set of tiles of the corresponding video frame has a first priority; and wherein a second set of tiles of the corresponding video frame has a different second priority, the first and second priorities being assigned based on relative visual importance of the first and second sets of tiles in the corresponding video frame.

[0133] In some embodiments of any of the above methods, each of the blocks represents a corresponding Neural Radiance Field that models a respective partition of a 3D scene.

[0134] In some embodiments of any of the above methods, the rateless error correction code comprises a Random Linear Network Code (RLNC).

[0135] In some embodiments of any of the above methods, the rateless error correction code comprises a code selected from the group consisting of a Luby transform code, a Raptor code, a rateless fountain code, a rateless spinal code, and an RLNC.

[0136] In some embodiments of any of the above methods, the blocks of the multimedia content have been sorted into different groups having different respective priorities and distributed over the plurality of network sources based on an evaluation of a cost to store at different ones of the network sources, a cost of delivery therefrom to the end-user device, and a relative contribution to quality of user experience during playback of the multimedia content at the end-user device.

[0137] In some embodiments of any of the above methods, the blocks of the multimedia content have been encoded using a plurality of priority mapped multisource encoders to generate one or more respective multisource coded variants of each of the blocks.

[0138] In some embodiments of any of the above methods, each of the network sources has stored therein copies of each of the blocks.

[0139] In some embodiments of any of the above methods, each of the blocks is stored at a corresponding single one of the network sources.

[0140] In some embodiments of any of the above methods, at least one of the network sources has two or more of the blocks stored therein.

[0141] In some embodiments of any of the above methods, at least one of the blocks has a respective copy thereof stored at each network source of a subset of the plurality of network sources including two or more of the network sources.

[0142] In some embodiments of any of the above methods, a copy of at least one of the blocks is stored in multiple ones but not all of the network sources.

[0143] In some embodiments of any of the above methods, determining the packet assignment plan comprises: generating a sorted list of network sources by sorting the plurality of network sources based on respective packet loss rates determined from the network conditions specified inthe plurality of descriptions; generating a sorted list of blocks by sorting the blocks according to respective contributions to distortion reduction of the reconstructed multimedia content; performing a baseline packet assignment using an order of the network sources in the sorted list of network sources and further using an order of the blocks in the sorted list of blocks; and iteratively performing a batch packet assignment based on a remaining packet budget by deciding whether to send a respective baseline number of packets of a next block from the sorted list of blocks or to add one or more additional packets to one or more previously assigned blocks from the sorted list of blocks, said deciding being performed based on respective changes to the objective function.

[0144] In some embodiments of any of the above methods, determining the packet assignment plan further comprises: with the remaining packet budget being greater than zero, sequentially performing individual packet assignment for unassigned packets by ruling which of the blocks a next packet slot is assigned to, said ruling being based on respective changes to the objective function imparted by individual ones of the unassigned packets.

[0145] In some embodiments of any of the above methods, determining the packet assignment plan comprises: for each of the blocks, determining a respective group of the network sources having stored therein a copy of the block; and performing packet assignments for the packet assignment plan on a group-by-group basis while following priority order of the groups of the network sources.

[0146] A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the above methods.

[0147] According to another example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS. 1-26, provided is an apparatus to perform block-based packet assignment for delivering multimedia, comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: receive, at an end-user device, a plurality of descriptions identifying blocks of a multimedia content stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths;communicate from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the multimedia content from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on optimization with an objective function including a first component representing estimated distortion of the multimedia content when received by the end-user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstruct the multimedia content using the blocks thereof assembled from corresponding sets of packets received at the end- user device after the transmission performed in accordance with the communicated packet assignment plan.

[0148] According to yet another example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS.1-26, provided is a block-based packet-assignment method for delivering multiple multimedia, comprising: receiving, at an end-user device, a plurality of descriptions identifying blocks of a plurality of multimedia contents stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicating from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the plurality of multimedia contents from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on optimization with an objective function including a first component representing estimated distortion of the plurality of multimedia contents when received by the end-user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstructing each of the plurality of multimedia contents using blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

[0149] In some embodiments of the above method, the plurality of descriptions also specifies a respective quality measurement for each of the identified blocks.

[0150] In some embodiments of the above method, each of the blocks represents a respective tile of a corresponding video frame.

[0151] In some embodiments of any of the above methods, each of the blocks represents a respective partition of a corresponding portion of the multimedia content.

[0152] In some embodiments of any of the above methods, the rateless error correction code comprises a code selected from the group consisting of a Luby transform code, a Raptor code, a rateless fountain code, a rateless spinal code, and a Random Linear Network Code (RLNC).

[0153] In some embodiments of any of the above methods, a first set of tiles of the corresponding video frame has a first priority; and wherein a second set of tiles of the corresponding video frame has a different second priority, the first and second priorities being assigned based on relative visual importance of the first and second sets of tiles in the corresponding video frame.

[0154] In some embodiments of any of the above methods, each of the blocks represents a corresponding Neural Radiance Field that models a respective partition of a 3D scene.

[0155] In some embodiments of any of the above methods, the rateless error correction code comprises a Random Linear Network Code (RLNC).

[0156] In some embodiments of any of the above methods, the blocks of the plurality of multimedia contents have been sorted into different groups having different respective priorities and distributed over the plurality of network sources based on an evaluation of a cost to store at different ones of the network sources, a cost of delivery therefrom to the end-user device, and a relative contribution to quality of user experience during playback of the plurality multimedia content at the end-user device.

[0157] In some embodiments of any of the above methods, the blocks of the plurality of multimedia contents have been encoded using a plurality of priority mapped multisource encoders to generate one or more respective multisource coded variants of each of the blocks.

[0158] In some embodiments of any of the above methods, each of the network sources has stored therein copies of each of the blocks.

[0159] In some embodiments of any of the above methods, the optimization with the objective function is configured to minimize combined distortion of the plurality of multimedia contents when received by the end-user device subject to a constraint associated with the cost of delivering the packets.

[0160] In some embodiments of any of the above methods, the optimization with the objective function is configured to minimize a maximum distortion in a set of individual distortions for the plurality of multimedia contents when received by the end-user device subject to a constraint associated with the cost of delivering the packets.

[0161] In some embodiments of any of the above methods, determining the packet assignment plan comprises: for each one of the plurality of multimedia contents, arranging the blocks thereof according to respective distortion values; and packing the blocks of the plurality of multimedia contents into a sequence of superblocks, with each of the superblocks including at least two of the blocks corresponding to different ones of the plurality of multimedia contents, wherein each of the superblocks includes a respective predetermined number of blocks of the plurality of multimedia contents selected based on a respective target distortion value, with each next superblock of the sequence being characterized by a lower target distortion value than a corresponding preceding superblock of the sequence.

[0162] In some embodiments of any of the above methods, determining the packet assignment plan comprises: for each superblock, generating a respective sorted list of network sources by sorting a respective subset of the plurality of network sources based on respective packet loss rates determined from the network conditions specified in the plurality of descriptions; generating a respective sorted list of blocks by sorting the blocks of the superblock according to respective contributions to distortion reduction of the reconstructed plurality of multimedia contents; performing a baseline packet assignment using an order of the network sources in the respective sorted list of network sources and further using an order of the blocks in the respective sorted list of blocks; and iteratively performing a batch packet assignment based on a remaining packet budget by deciding whether to send a respective baseline number of packets of a next block from the respective sorted list of blocks or to add one or more additional packets to one or more previously assigned blocks from the respective sorted list of blocks, said deciding being performed based on respective changes to the objective function.

[0163] In some embodiments of any of the above methods, determining the packet assignment plan further comprises: for one or more of the superblocks, with the remaining packet budget being greater than zero, sequentially performing individual packet assignment for unassigned packets by ruling which of the blocks a next packet slot is assigned to, said ruling being based on respective changes to the objective function imparted by individual ones of the unassigned packets.

[0164] In some embodiments of any of the above methods, constructing a superblock comprises: generating a respective sorted list of network sources by sorting a respective subset of the plurality of network sources based on respective packet loss rates determined from the network conditions specified in the plurality of descriptions; generating a respective sorted list of blocks by sorting the blocks of the superblock according to respective contributions to distortion reduction of the reconstructed plurality of multimedia contents; performing a baseline packet assignment using an order of the network sources in the respective sorted list of network sources and further using an order of the blocks in the respective sorted list of blocks; and iteratively performing a batch packet assignment based on a remaining packet budget by deciding whether to send a respective baseline number of packets of a next block from the respective sorted list of blocks or to add one or more additional packets to one or more previously assigned blocks from the respective sorted list of blocks, said deciding being performed based on respective changes to the objective function.

[0165] In some embodiments of any of the above methods, constructing the superblock further comprises: with the remaining packet budget being greater than zero, sequentially performing individual packet assignment for unassigned packets by ruling which of the blocks a next packet slot is assigned to, said ruling being based on respective changes to the objective function imparted by individual ones of the unassigned packets.

[0166] In some embodiments of any of the above methods, constructing a tthsuperblock comprises: sequentially packing a predetermined number of blocks into the tthsuperblock, with the packing being configured to have distortions of constituent blocks within a selected range; arranging distortions of each Multiple Multimedia Content (MMC) in a descending order and setting a respective target distortion to find how many blocks from the MMCs to include in the superblock such that respective distortions in each MMC are within the selected range centered on the respective target distortion; and reducing the target distortion to sequentially build the superblock.

[0167] A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the above methods.

[0168] According to yet another example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS.1-26, provided is an apparatus to perform block-based packet assignment for delivering multiple multimedia, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: receive, at an end-user device, a plurality of descriptions identifying blocks of a plurality of multimedia contents stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicate from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the plurality of multimedia contents from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on optimization with an objective function including a first component representing estimated distortion of the plurality of multimedia contents when received by the end- user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstruct each of the plurality of multimedia contents using blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

[0169] According to yet another example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS. 1-26, provided is a method of distributing blocks of a multimedia content for storage at a plurality of network sources, the method comprising: sorting the blocks of the multimedia content into different groups having different respective priorities; and distributing the sorted blocks over the plurality of network sources based on an evaluation of a cost to store at different ones of the network sources, a cost of delivery therefrom to an end-user device, and a relative contribution to quality of user experience during playback of the multimedia content at the end-user device.

[0170] In some embodiments of the above method, the method further comprises encoding the blocks of the multimedia content using a plurality of priority mapped multisource encoders to generate one or more respective multisource coded variants of each of the blocks.

[0171] In some embodiments of any of the above methods, a subset of the network sources has stored therein copies of each of the blocks.

[0172] In some embodiments of any of the above methods, each of the blocks is stored at a corresponding single one of the network sources.

[0173] In some embodiments of any of the above methods, at least one of the network sources has two or more of the blocks stored therein.

[0174] In some embodiments of any of the above methods, at least one of the blocks has a respective copy thereof stored at each network source of a subset of the plurality of network sources including two or more of the network sources.

[0175] In some embodiments of any of the above methods, a copy of at least one of the blocks is stored in multiple ones but not all of the network sources.

[0176] A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the above methods.

[0177] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.

[0178] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would beapparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0179] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0180] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

[0181] While this disclosure includes references to illustrative embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the disclosure, which are apparent to persons skilled in the art to which the disclosure pertains are deemed to lie within the principle and scope of the disclosure, e.g., as expressed in the following claims.

[0182] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit.

[0183] Some embodiments can be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented invention(s). Some embodiments can also be embodied in the form of program code, for example, stored in a non- transitory machine-readable storage medium including being loaded into and / or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as a computer or a processor, the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0184] Unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about” or “approximately” preceded the value or range.

[0185] The use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate the interpretation of the claims. Such use is not to be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0186] Although the elements in the following method claims, if any, are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence.

[0187] Reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments necessarily mutually exclusive of other embodiments. The same applies to the term “implementation.”

[0188] Unless otherwise specified herein, the use of the ordinal adjectives “first,” “second,” “third,” etc., to refer to an object of a plurality of like objects merely indicates that different instances of such like objects are being referred to, and is not intended to imply that the like objects so referred-to have to be in a corresponding order or sequence, either temporally, spatially, in ranking, or in any other manner.

[0189] Unless otherwise specified herein, in addition to its plain meaning, the conjunction “if” may also or alternatively be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” which construal may depend on the corresponding specific context. For example, the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event].”

[0190] Also, for purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energy is allowed to be transferred between two or more elements, and the interposition of one or more additional elements is contemplated, although not required. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements.

[0191] As used herein in reference to an element and a standard, the term compatible means that the element communicates with other elements in a manner wholly or partially specified by the standard and would be recognized by other elements as sufficiently capable of communicating with the other elements in the manner specified by the standard. The compatible element does not need to operate internally in a manner specified by the standard.

[0192] The functions of the various elements shown in the figures, including any functional blocks labeled as “processors” and / or “controllers,” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory(ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

[0193] As used in this application, the terms “circuit,” “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0194] It should be appreciated by those of ordinary skill in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0195] “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification is intended to introduce some example embodiments, with additional embodiments being described in“DETAILED DESCRIPTION” and / or in reference to one or more drawings. “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

CLAIMS What is claimed is:

1. A block-based packet-assignment method for delivering multimedia, the method comprising: receiving, at an end-user device, a plurality of descriptions identifying blocks of a multimedia content stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicating from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the multimedia content from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on an optimization with an objective function including a first component representing an estimated distortion of the multimedia content when received by the end-user device and a second component representing a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstructing the multimedia content using the blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

2. The method of claim 1, wherein the plurality of descriptions also specifies a respective quality measurement for each of the identified blocks.

3. The method of claim 1 or 2, wherein each of the blocks represents a respective partition of a corresponding portion of the multimedia content.

4. The method of claim 3, wherein at least one of the respective partitions is a tile of a video frame.

5. The method of claim 3, wherein a first set of tiles of the corresponding video frame has a first priority; andwherein a second set of tiles of the corresponding video frame has a different second priority, the first and second priorities being assigned based on relative visual importance of the first and second sets of tiles in the corresponding video frame.

6. The method of claim 1 or 2, wherein each of the blocks represents a corresponding Neural Radiance Field that models a respective partition of a 3D scene.

7. The method of any preceding claim, wherein the rateless error correction code comprises a code selected from the group consisting of a Luby transform code, a Raptor code, a rateless fountain code, a rateless spinal code, and a Random Linear Network Code (RLNC).

8. The method of any preceding claim, wherein the blocks of the multimedia content have been sorted into different groups having different respective priorities and distributed over the plurality of network sources based on an evaluation of a cost to store at different ones of the network sources, a cost of delivery therefrom to the end-user device, and a relative contribution to quality of user experience during playback of the multimedia content at the end-user device.

9. The method of any one of claims 1 to 7, wherein the blocks of the multimedia content have been encoded using a plurality of priority mapped multisource encoders to generate one or more respective multisource coded variants of each of the blocks.

10. The method of claim 1, wherein each of the network sources has stored therein copies of each of the blocks.

11. The method of claim 1, wherein each of the blocks is stored at a corresponding single one of the network sources.

12. The method of claim 11, wherein at least one of the network sources has two or more of the blocks stored therein.

13. The method of claim 1, wherein at least one of the blocks has a respective copy thereof stored at each network source of a subset of the plurality of network sources including two or more of the network sources.

14. The method of claim 1, wherein a copy of at least one of the blocks is stored in multiple ones but not all of the network sources.

15. The method of any preceding claim, wherein determining the packet assignment plan comprises: generating a sorted list of network sources by sorting the plurality of network sources based on respective packet loss rates determined from the network conditions specified in the plurality of descriptions; generating a sorted list of blocks by sorting the blocks according to respective contributions to distortion reduction of the reconstructed multimedia content; performing a baseline packet assignment using an order of the network sources in the sorted list of network sources and further using an order of the blocks in the sorted list of blocks; and iteratively performing a batch packet assignment based on a remaining packet budget by deciding whether to send a respective baseline number of packets of a next block from the sorted list of blocks or to add one or more additional packets to one or more previously assigned blocks from the sorted list of blocks, said deciding being performed based on respective changes to the objective function.

16. The method of claim 15, wherein said determining the packet assignment plan further comprises: with the remaining packet budget being greater than zero, sequentially performing individual packet assignment for unassigned packets by ruling which of the blocks a next packet slot is assigned to, said ruling being based on respective changes to the objective function imparted by individual ones of the unassigned packets.

17. The method of claim 15, wherein said determining the packet assignment plan comprises: for each of the blocks, determining a respective group of the network sources having stored therein a copy of the block; andperforming packet assignments for the packet assignment plan on a group-by-group basis while following priority order of the groups of the network sources.

18. The method of any preceding claim, wherein the multimedia content comprises a plurality of multimedia contents, and the reconstructing comprises reconstructing each of the plurality of multimedia contents.

19. The method of claim 18, wherein the optimization with the objective function is configured to minimize combined distortion of the plurality of multimedia contents when received by the end- user device subject to a constraint associated with the cost of delivering the packets.

20. The method of claim 18, wherein the optimization with the objective function is configured to minimize a maximum distortion in a set of individual distortions for the plurality of multimedia contents when received by the end-user device subject to a constraint associated with the cost of delivering the packets.

21. The method of claim 18, wherein determining the packet assignment plan comprises: for each one of the plurality of multimedia contents, arranging the blocks thereof according to respective distortion values; and packing the blocks of the plurality of multimedia contents into a sequence of superblocks, with each of the superblocks including at least two of the blocks corresponding to different ones of the plurality of multimedia contents, wherein each of the superblocks includes a respective predetermined number of blocks of the plurality of multimedia contents selected based on a respective target distortion value, with each next superblock of the sequence being characterized by a lower target distortion value than a corresponding preceding superblock of the sequence.

22. The method of claim 21, wherein said determining the packet assignment plan comprises: for each superblock, generating a respective sorted list of network sources by sorting a respective subset of the plurality of network sources based on respective packet loss rates determined from the network conditions specified in the plurality of descriptions;generating a respective sorted list of blocks by sorting the blocks of the superblock according to respective contributions to distortion reduction of the reconstructed plurality of multimedia contents; performing a baseline packet assignment using an order of the network sources in the respective sorted list of network sources and further using an order of the blocks in the respective sorted list of blocks; and iteratively performing a batch packet assignment based on a remaining packet budget by deciding whether to send a respective baseline number of packets of a next block from the respective sorted list of blocks or to add one or more additional packets to one or more previously assigned blocks from the respective sorted list of blocks, said deciding being performed based on respective changes to the objective function.

23. The method of claim 22, wherein said determining the packet assignment plan further comprises: for one or more of the superblocks, with the remaining packet budget being greater than zero, sequentially performing individual packet assignment for unassigned packets by ruling which of the blocks a next packet slot is assigned to, said ruling being based on respective changes to the objective function imparted by individual ones of the unassigned packets.

24. The method of claim 22, wherein constructing a superblock comprises: generating a respective sorted list of network sources by sorting a respective subset of the plurality of network sources based on respective packet loss rates determined from the network conditions specified in the plurality of descriptions; generating a respective sorted list of blocks by sorting the blocks of the superblock according to respective contributions to distortion reduction of the reconstructed plurality of multimedia contents; performing a baseline packet assignment using an order of the network sources in the respective sorted list of network sources and further using an order of the blocks in the respective sorted list of blocks; and iteratively performing a batch packet assignment based on a remaining packet budget by deciding whether to send a respective baseline number of packets of a next block from the respectivesorted list of blocks or to add one or more additional packets to one or more previously assigned blocks from the respective sorted list of blocks, said deciding being performed based on respective changes to the objective function.

25. The method of claim 24, wherein the constructing the superblock further comprises: with the remaining packet budget being greater than zero, sequentially performing individual packet assignment for unassigned packets by ruling which of the blocks a next packet slot is assigned to, said ruling being based on respective changes to the objective function imparted by individual ones of the unassigned packets.

26. The method of claim 21, wherein constructing a tthsuperblock comprises: sequentially packing a predetermined number of blocks into the tthsuperblock, with the packing being configured to have distortions of constituent blocks within a selected range; arranging distortions of each Multiple Multimedia Content (MMC) in a descending order and setting a respective target distortion to find how many blocks from the MMCs to include in the superblock such that respective distortions in each MMC are within the selected range centered on the respective target distortion; and reducing the target distortion to sequentially build the superblock.

27. A method of distributing blocks of a multimedia content for storage at a plurality of network sources, the method comprising: sorting the blocks of the multimedia content into different groups having different respective priorities; and distributing the sorted blocks over the plurality of network sources based on an evaluation of a cost to store at different ones of the network sources, a cost of delivery therefrom to an end-user device, and a relative contribution to quality of user experience during playback of the multimedia content at the end-user device.

28. The method of claim 27, further comprising encoding the blocks of the multimedia content using a plurality of priority mapped multisource encoders to generate one or more respective multisource coded variants of each of the blocks.

29. The method of claim 27, wherein a subset of the network sources has stored therein copies of each of the blocks.

30. The method of claim 27, wherein each of the blocks is stored at a corresponding single one of the network sources.

31. The method of claim 30, wherein at least one of the network sources has two or more of the blocks stored therein.

32. The method of claim 27, wherein at least one of the blocks has a respective copy thereof stored at each network source of a subset of the plurality of network sources including two or more of the network sources.

33. The method of claim 27, wherein a copy of at least one of the blocks is stored in multiple ones but not all of the network sources.

34. An apparatus to perform block-based packet assignment for delivering multimedia, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: receive, at an end-user device, a plurality of descriptions identifying blocks of a multimedia content stored at a plurality of network sources connected to the end-user device via a plurality of network paths, each of the blocks being encoded using a rateless error correction code, the plurality of descriptions also specifying network conditions for the plurality of network paths; communicate from the end-user device to the plurality of network sources a packet assignment plan for transmission of packets corresponding to different blocks of the multimedia content from the plurality of network sources to the end-user device, the packet assignment plan being determined based on the received plurality of descriptions and further based on an optimization with an objective function including a first component representing an estimated distortion of the multimedia content when received by the end-user device and a second componentrepresenting a cost of delivering the packets from the plurality of network sources to the end-user device; and reconstruct the multimedia content using the blocks thereof assembled from corresponding sets of packets received at the end-user device after the transmission performed in accordance with the communicated packet assignment plan.

35. The apparatus of claim 34, wherein the plurality of descriptions also specifies a respective quality measurement for each of the identified blocks.

36. The apparatus of claim 34 or 34, wherein the multimedia content comprises a plurality of multimedia contents, and at least one memory and the program code are configured to, with the at least one processor, cause the apparatus to reconstruct each of the plurality of multimedia contents.

37. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of any of claims 1 to 33.

Images