Personalized Sensitivity Measurement and Playback Factors for Adaptive and Personalized Media Encoding and Delivery - Patent application

Abstract

A method for delivering media to a playback device includes outputting first test media for viewing by a first user. The method further includes receiving a first user input associated with a first perception of the first test media by the first user and indicating a first personalized quality of experience for the first user with respect to the first test media. The method further includes generating a first personalized sensitivity profile based on the first user input, the first personalized sensitivity profile including one or more viewing characteristics of the first user, and determining first media parameters based at least in part on the first personalized sensitivity profile. The first media parameters are determined to increase efficiency of media delivery to the first playback device over a network while preserving the first personalized quality of experience for the first user.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 056,942, filed July 27, 2020, and U.S. Provisional Patent Application No. 62 / 882,068, filed August 2, 2019, both of which are incorporated by reference in their entireties.

[0002] This application relates generally to delivering visual media over a network to a user device for display by the user device for viewing by the user. Summary of the Invention [Means for solving the problem]

[0003] Various aspects of the present disclosure relate to apparatus, systems, and methods that provide personalized, adaptive media encoding and delivery based on playback-side information that is often collected without separate sensors.

[0004] In one aspect of the disclosure, a method for delivering media to a playback device is provided. The method may include outputting, with a first playback device, a first test media to be viewed by a first user during a first test measurement session. The method may further include receiving a first user input from the first user. The first user input may be related to a first perception of the first test media by the first user and may indicate a first personalized quality of experience of the first user with respect to the first test media. The method may further include generating, with one or more electronic processors, a first personalized sensitivity profile including one or more viewing characteristics of the first user based on the first user input. The method may further include determining, with the one or more electronic processors, first media parameters based at least in part on the first personalized sensitivity profile. The first media parameters may be determined to increase efficiency of media delivery to the first playback device over a network while preserving the first personalized quality of experience of the first user. The method may further include providing a first output media to the first playback device over the network in accordance with the first media parameters, the first output media being configured to be output using the first playback device.

[0005] In another aspect of the disclosure, an electronic computing device is provided that may include a first playback device including a display. The display may be configured to output media to a first user. The electronic computing device may also include one or more electronic processors communicatively coupled to the display. The one or more electronic processors may be configured to output, using the first playback device, a first test media to be viewed by the first user during a first test measurement session. The one or more electronic processors may be further configured to receive a first user input from the first user. The first user input may be related to a first perception of the first test media by the first user and may indicate a first personalized quality of experience of the first user with respect to the first test media. The one or more electronic processors may be further configured to generate a first personalized sensitivity profile based on the first user input, the first personalized sensitivity profile including one or more viewing characteristics of the first user. The one or more electronic processors may be further configured to determine a first media parameter based at least in part on the first personalized sensitivity profile. The first media parameters may be determined to increase efficiency of media delivery to the first playback device over a network while preserving a first personalized quality of experience for the first user. The one or more electronic processors may be further configured to provide a first output media to the first playback device over the network in accordance with the first media parameters. The first output media may be configured to be output using the first playback device.

[0006] In another aspect of the disclosure, a method is provided for displaying a hybrid image on a playback device. The method may include determining, with one or more electronic processors of an electronic computing device, a first value of a media parameter supported by a media server and a network configured to stream media. The method may further include determining, with the one or more electronic processors, a second value of the media parameter supported by the media server and the network. The method may further include at least one of generating and selecting, with the one or more electronic processors, the hybrid image based on the first value of the media parameter and the second value of the media parameter, such that the hybrid image includes a first interpretation corresponding to the first value of the media parameter and a second interpretation corresponding to the second value of the media parameter. The method may further include displaying the hybrid image on a display of the playback device.

[0007] Other aspects of the embodiments will become apparent by consideration of the detailed description and accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example adaptive bitrate (ABR) based media encoding and delivery system according to embodiments described herein. [Diagram 2] FIG. 2 illustrates a portion of an adaptive bitrate (ABR)-based media encoding and delivery system configured to identify individualized / personalized viewing characteristics during a test measurement session, according to some embodiments described herein. [Diagram 3] FIG. 3 illustrates a portion of an adaptive bitrate (ABR)-based media encoding and delivery system configured to identify individualized / personalized viewing characteristics during a test measurement session, according to some embodiments described herein. [Figure 4]FIG. 4 illustrates a block diagram of a generic objective model being transformed into a personalized objective model via model transformation according to embodiments described herein. [Diagram 5] FIG. 5 illustrates a graph including two different contrast sensitivity functions (CSFs) showing example relationships between contrast sensitivity and spatial frequency, according to embodiments described herein. [Figure 6] FIG. 6 illustrates a modified adaptive bitrate (ABR) based media encoding and delivery system according to an embodiment described herein. [Figure 7] FIG. 7 is a hardware block diagram of the playback system of FIG. 6 according to an embodiment described herein. [Figure 8] FIG. 8 is a block diagram of the media server of FIG. 6 according to an embodiment described herein. [Figure 9] FIG. 9 illustrates a flow chart of a method for delivering media to the playback system of FIG. 6 according to an embodiment described herein. [Figure 10ABC] 10A-10C show example hybrid images of three different sizes according to embodiments described herein. [Figure 10DE] 10D-10E show example hybrid images of three different sizes according to embodiments described herein. [Figure 11] 11A and 11B show graphs of an example optimal ABR ladder estimation using hybrid images as test media in a multi-stage binary tree search during a test measurement session according to embodiments described herein. [Figure 12] 12A and 12B show example bandwidths of a network when using existing streaming methods to stream media versus the method of FIG. 9, according to embodiments described herein. [Figure 13A]FIG. 13A illustrates another example of how the method of FIG. 9 enables more users / subscribers to stream media on a fixed capacity network without adversely affecting QoE, according to embodiments described herein. [Figure 13B] FIG. 13B illustrates another example of how the method of FIG. 9 allows more users / subscribers to stream media on a fixed capacity network without adversely affecting QoE, according to embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Visual media (e.g., images, videos, etc.) can be distributed over one or more communication networks to many different kinds of playback systems / devices (e.g., televisions, computers, tablets, smartphones, etc.) for viewing by users. In the visual media distribution chain, adaptive bitrate (ABR) streaming enables improved network resource management through adaptively selecting bitrate and resolution on the media ladder based on network conditions, playback buffer state, shared network capacity, and other factors affected by the network. Besides ABR streaming, other media distribution methods (which may also include encoding methods or source encoding methods) can be used as well to control one or more media parameters of an upstream video encoder / transcoder / translator, such as bitrate, frame rate, resolution, etc.

[0010] However, to date, media delivery methods such as ABR streaming have not considered additional factors, such as factors related to the playback system / device, factors related to the user's viewing capabilities, and factors related to the environment in which the user views visual media, to further improve network resource management. Rather, these factors are typically assumed to be ideal and uniform across different users / environments when performing content processing, encoding, delivery, decoding, and post-processing, despite the existence of diverse viewing conditions and variations in human visual performance that greatly affect the viewer's actual Quality of Experience (QoE).

[0011] For example, short-distance viewing may make a user more sensitive in distinguishing between low-resolution and high-resolution video content. Also, different viewers may have different visual sensitivity due to factors including, but not limited to, refractive errors (even when wearing corrective lenses), accumulation of floaters in the vitreous humor, age-related changes in color absorption by the crystalline lens, cataracts or macular degeneration, etc. For example, a user / viewer's contrast sensitivity may decrease due to increasing refractive error, increasing disease, and / or aging. Additionally, an individual's personal QoE may vary from place to place and from time to time, especially in a mobile environment.

[0012] Detection of these visual sensitivity factors for each user / viewer helps estimate personalized QoE in real-world end-to-end systems and provides opportunities to improve QoE and further improve media delivery efficiency. For example, a media delivery system can save bandwidth by sending a custom filtered version of the same video to match the user / viewer's eyesight or viewing distance from the TV, while maintaining each user / viewer's personalized QoE.

[0013] Some studies have proposed using various sensors to collect playback-side factors for the purpose of selecting optimal bit rates and resolutions in media streaming or feeding back information to media pre-processing, encoding and post-processing. However, the approach of using various sensors to collect playback-side information is insufficient and impractical for many playback systems (e.g., televisions). Such sensors are insufficient because they do not measure the user's inherent visual acuity or sensitivity. This is impractical because it would be an excessive burden to motivate and coordinate television manufacturers across the consumer display industry to equip televisions with the necessary sensors and metadata protocols. For mobile devices, which already have many available sensors, this burden is less, but user privacy is still a concern, especially when the sensors collect visual information about the user. Another problem with the existing approach of using sensors to collect playback-side information is that different models / brands of televisions have their own upscaling and post-processing algorithms, and users may adjust various television settings, such as brightness, contrast or motion smoothing, to suit their preferences.

[0014] To address the above technical challenges, the methods, devices, and systems described herein include a new mechanism or protocol for sharing parameters related to playback device characteristics and personalized visual sensitivity factors with an upstream device configured to control the transmission of visual media to the playback device. The methods, devices, and systems described herein provide personalized and adaptive media delivery based on playback side information, often collected without the use of separate sensors. In addition, the collected playback side information may indicate personalized QoE for different users and / or different viewing environments. The methods, devices, and systems described herein further improve network resource management / media delivery efficiency while maintaining personalized QoE for each user.

[0015] Figure 1 shows an example adaptive bitrate (ABR) based media encoding and delivery system 100. System 100 includes a media server 105 that provides media to playback systems 110 (i.e., playback devices) over a network 115. Although Figure 1 shows a single playback system 110, the media server 105 may be configured to simultaneously stream the same or different media to additional playback systems 110.

[0016] The playback system 110 may include many different types of playback systems, such as a television, a tablet, a smartphone, a computer, etc. In some embodiments, the playback system 110 includes a buffer / decoder 120 and a playback renderer 125. The buffer / decoder 120 may receive media from the server 105 over the network 115. The buffer / decoder 120 may buffer the received media and decode the received media for output by the playback renderer 125. The buffer / decoder 120 may include an electronic processor (e.g., a microprocessor, microcontroller, or other suitable processing device) of the playback system 110, as described in more detail below with respect to FIG. 7. The playback renderer 125 may include an output device configured to display images and / or video. For example, the playback renderer 125 may include a light emitting diode (LED) display and / or a touch screen display, as described in more detail below with respect to FIG. 7. The playback system 110 is located within the environment 130. A user 135 may also be located within the environment 130 and may view the media output by the playback system 110.

[0017] 1, in some embodiments, the media server 105 includes an ABR ladder 137 implemented by an electronic processor of the media server 105. In some embodiments, the media server 105 receives one or more ABR requests 140 from the playback system 110 and coordinates bitrate / quality decisions for ABR streaming of media from the media server 105 to the playback system 110 over the network 115. For example, the playback system 110 retrieves from memory and / or utilizes a stored generic objective model 145 (e.g., device type, display resolution, GEO-based startup resolution or more comprehensive models such as ITU-T P.1203) and uses the generic objective model 145 to observe / measure performance information related to the streaming session and playback, such as network connectivity metrics, media player buffer state, codec, bitrate, initial load delay and stall events, along with playback device 110 information, such as display resolution, screen size, playback device type, etc. In some embodiments, packet header information and partial / full bitstream parsing may also be used to collect streaming session and playback related performance information. The streaming and playback information is used to generate a general Quality of Experience (QoE) estimate of the media streaming and playback of the media. This QoE estimate may be used by the playback system 110 to influence the ABR requests 140. For example, the playback system 110 periodically determines each ABR request 140 for a media segment based on locally generated bandwidth estimates, buffer sizes, round trip times, etc., with the aim of maintaining seamless playback. That is, the general objective model 145 may be configured to enable the playback system 210 to control the media streaming based on at least one of the availability of resources of the network 115 and playback system parameters.

[0018] In some cases, the playback system 110 simultaneously requests two or more segments that represent the same period in the media but are encoded at different bit rates from the ABR ladder 137. Such a strategy can be inefficient and often results in the playback system 110 requesting more data than it needs for seamless playback. Such a strategy can also result in the playback system 110 requesting a resolution / bit rate / frame rate combination from the ABR ladder 137 that provides a quality of media that cannot be perceived by the user 135. That is, existing ABR selection logic that attempts to increase the delivered resolution / bit rate / frame rate beyond the sensitivity threshold of the user 135 does not translate to an increase in QoE for the user 135. In addition to not translating to an increase in QoE for the user 135, the requested resolution / bit rate / frame rate combination may use more network resources (e.g., more bandwidth) and / or result in additional costs to the user 135 (e.g., if the service provider of the media server 105 charges based on the amount of data provided to the playback system 110).

[0019] The above-mentioned problems with existing ABR selection logic are caused by the general objective model 145 not taking into account personalized QoE when determining the ABR request 140. For example, the general objective model 145 may not take into account individualized / personalized visual characteristics such as the lighting in the environment 130, the viewing distance of the user 135 (i.e., the distance between the user 135 and the playback renderer 125), the visual sensitivity and ability of the user 135's eyes based on, for example, spatial frequency, etc. Rather, existing ABR selection techniques assume that these characteristics are the same for each environment 130 and each user 135, when in reality they may vary significantly between environments and / or users and affect the QoE of the user 135 to perceive the media being displayed by the playback system 110.

[0020] Although FIG. 1 and its corresponding description refer to ABR ladder 137 and ABR selection techniques, ABR-related media distribution is merely an example intended to represent a general media distribution methodology that may be implemented by upstream devices such as media server 105 and network 115. ABR ladder 137 and ABR selection techniques are also used throughout this application as exemplary media distribution methods with respect to additional figures (see, for example, FIGS. 2, 3, and 11A-11B). However, the features disclosed herein may be applied to any one of a number of different media distribution methods (which may also include encoding or source encoding methods) that are not based on ABR ladder 137 or ABR selection techniques. That is, the features described herein may be used with other media distribution methods other than ABR streaming that may also be used to control any media parameters of an upstream video encoder / transcoder / translator, such as bit rate, frame rate, resolution, etc. Additionally or alternatively, the features described herein may be used with encoding or source encoding methods used to encode / process media before it is streamed. These encoding and source encoding methods may be generally referred to herein as media delivery methods. In some embodiments, the media parameters include parameters that affect media delivery from the media server 105 to the playback system 210 over the network 115 (i.e., upstream media parameters) (see Figures 2, 3, and 6). In some embodiments, the media parameters additionally or alternatively include playback system parameters (i.e., downstream parameters), such as brightness and / or contrast settings of the playback system 210.

[0021] 2 and 3 show portions of adaptive bitrate (ABR)-based media encoding and delivery systems 200 and 300 configured to identify individualized / personalized viewing characteristics during a test measurement session, according to some embodiments. The test measurement session may be used to address the above challenges of ABR selection logic (and / or other media delivery methods) by further taking into account personalized viewing characteristics with (see FIG. 3) or without (see FIG. 2) separate sensors to gather additional information.

[0022] Figure 2 includes a playback system 210 that includes several components similar to the playback system 110 of Figure 1. For example, the playback system 210 includes a buffer / decoder 120 and a playback renderer 125. Although not shown in Figure 2, the playback system 210 may be communicatively coupled to a media server via a network, similar to the media server 105 and network 115 shown in Figure 1. However, instead of including the general objective model 145 shown in Figure 1, the playback system 210 may generate personalized (i.e., individualized) sensitivity profiles (PSPs) 215 for a number of different users 135 and / or environments 130. These personalized sensitivity profiles 215 may be used to provide ABR requests (or other requests regarding other media delivery methods) to a media server that streams media to the playback system 210. In the following, the personalized sensitivity profile 215 is described as being generated by the playback system 210 (e.g., an electronic processor of the playback system 210), however, in some embodiments, the generation and storage of the personalized sensitivity profile 215 may additionally or alternatively be performed by an electronic processor in a media server, an electronic processor in a remote cloud computing cluster, or a combination thereof, as described in further detail herein.

[0023] To generate the personalized sensitivity profile 215, the playback system 210 conducts a test measurement session in which user responses 220 to test media 225 are collected from the user 135. During the test measurement session, the sensitivity of the user 135 is measured given their typical viewing conditions and environment. As an example, the user 135 may be sitting in a typical viewing position, such as a sofa in a living room, which represents typical viewing conditions in terms of viewing distance, viewing angle, ambient brightness, and playback system characteristics and settings. The playback system 210 then guides the user 135 through a test measurement session that measures the audiovisual sensitivity of the user 135 in the environment 130 by following instructions provided by the playback system 210. In the session, the user 135 may be requested to make one or more selections using a remote control device according to a sequence of images and / or videos presented by the playback system 210. For example, the playback system 210 may display multiple images and request the user 135 to select the image that is the clearest / best in focus for the user 135. As another example, the playback system 210 may display an image with multiple interpretations depending on the visual capabilities of the user 135 and the user's viewing distance, and ask the user to select the interpretation that is most dominant / obvious to the user.

[0024] From the test measurement session, the playback system 210 and / or the media server 105 may determine personalized viewing characteristics of the user 135 and / or the environment 130. For example, the user responses 220 received during the test measurement session may indicate system factors such as playback system characteristics, playback parameter settings, post-processing algorithms of the playback system 210 (usually owned by the device manufacturer). As another example, the user responses 220 received during the test measurement session may indicate environmental factors such as viewing distance, viewing angle, ambient brightness, ambient noise, user expectations, etc. In some embodiments, user expectations refer to conscious or subconscious psychological aspects of the user 135 that may affect the user's perceived QoE. For example, the user 135 may have a higher expectation level for media associated with a paid subscription than for other media such as a free video-on-demand service. As yet another example of personalized viewing characteristics of the user 135 and / or the environment 130, the user responses 220 received during the test measurement session may indicate human factors such as the user's 135 sensory / visual sensitivity and ability, age, gender, etc. As shown in Figure 2, in some embodiments, the playback system 210 may further provide playback system information 230 (e.g., device type, display resolution, etc.) in a manner similar to that described above with respect to the system 100 of Figure 1. The playback system 210 may use aggregated information 235 including the playback system information 230 and user responses 220 to the test media 225 to identify a personalized sensitivity profile 215 for a particular user 135 and / or environment 130.

[0025] As shown in FIG. 2, the personalized sensitivity profile 215 may include one or more of a user identification, other personal information (i.e., individualized viewing characteristics of the user 135 as identified by the user responses 220), playback system information 230, and a geographic location, weather information, and dates and times at which the test media 225 was presented to the user 135 during one or more measurement sessions. In the example personalized sensitivity profile 215 shown in FIG. 2, each personalized sensitivity profile 215 is associated with a user and may include multiple sub-profiles for different environments in which the user 135 participated in a test measurement session (e.g., different rooms in the user's home, different time periods, different weather information (e.g., sunny vs. cloudy), etc.). In other embodiments, each personalized sensitivity profile 215 may be associated with a user and a particular environment such that each user may have multiple personalized sensitivity profiles 215, each corresponding to a different environment in which the user 135 participated in a test measurement session.

[0026] Although the personalized viewing characteristics described above are not explicitly collected by a separate sensor (e.g., a sensor measuring the distance between the playback device 210 and the user 135), the system 200 can identify / estimate one or more of these characteristics based on the user responses 220 to the test media 225 during a test measurement session. Thus, in some embodiments, personalized viewing characteristic information can be collected from the user 135 and the environment 130 without the use of a separate explicit sensor. In other embodiments, a separate explicit sensor may be used to provide additional information (see, e.g., FIG. 3).

[0027] 3 illustrates a portion of an adaptive bitrate (ABR) based media encoding and delivery system 300 similar to the portion of the system 200 of FIG. 2. However, the system 300 further includes one or more environmental sensors 305. For example, the playback system 210 may include or be communicatively coupled to a brightness / luminance sensor (e.g., an integrated sensor, a smart home sensor communicatively coupled to the playback system 210, etc.) that measures ambient light in the environment 130. The environmental sensors 305 may include other smart home sensors located in the environment 130 that indicate, for example, whether lights are on / off, whether curtains covering windows are open / closed, etc. The environmental sensors 305 may be configured to identify the time of day and the geographic location of the playback system 210, for example, for purposes of identifying daytime hours. The environmental sensing data 310 from the environmental sensors 305 may be included in the aggregated information 235 or, as shown in FIG. 3, in a personalized sensitivity profile 215. In some embodiments, the environmental sensing data 310 included in the aggregated information 235 includes user-provided information such as the level of social activity in the environment 130 and / or viewing distance.

[0028] Collecting user responses 220 to test media 225 during a test measurement session implicitly accounts for many personalized viewing characteristics that may otherwise be difficult, impractical, and / or intrusive to collect explicitly using sensors. In addition, in some situations, data collected explicitly using sensors may not allow for an accurate determination of the QoE of the user 135. Thus, systems 200 and 300 provide numerous potential advantages and benefits.

[0029] One exemplary benefit concerns user diversity. Two different users may have identical environmental characteristics (e.g., viewing distance, brightness, screen size, etc.). However, these two different users may have significantly different viewing capabilities, e.g., due to differences in refractive error, age, and / or eye disease. Thus, using only physical context / environment characteristics to determine ABR requirements (or requirements for other media delivery methods) may result in different levels of QoE for these different users. In some embodiments, the personalized sensitivity profile 215 of the system 200, 300 takes these differences in personalized viewing capabilities into account when determining the ABR requirements to prevent and / or mitigate degradation of each user's personalized QoE.

[0030] Another exemplary benefit may be the ease of implementation of systems 200 and 300. In some embodiments, no additional sensors need to be added to the playback system 210. Similarly, in some embodiments, user privacy is protected because specific details of the user's environment may not be explicitly measured and recorded. Rather, in some embodiments, a global assessment of the user response 220, which implicitly includes only more detailed factors (such as the user's viewing distance and viewing ability), is used to generate an ABR request to the media server 105. That is, the ABR request is based on a composite measurement that is not based on a collection of independent attribute measurements from separate sensors. Rather, the ABR request in systems 200 and 300 is a global and implicit measurement of the user's QoE that reflects the combined effect of many factors that are difficult or even impossible to collect explicitly.

[0031] Yet another exemplary benefit relates to enhanced personalized content for the user 135. In addition to enabling a more holistic and accurate estimation of the end user's QoE, the systems 200 and 300 enable improvements to the media / content played to the user 135. Specifically, portions of a video frame that have too little or too low contrast (i.e., spatial frequencies or contrasts that exceed the user's measured contrast sensitivity function (CSF)) to be perceived by the user can be enhanced by the playback system 210. Examples of such improvements may include cropping and enlarging the frame and / or applying local contrast adjustments to ensure that salient parts of a scene are visible to the user 135 (i.e., within the user's measured CSF). Such improvements may enhance the user's viewing experience by helping the user 135 follow the media / content and maintain engagement with the media / content being viewed.

[0032] Referring again to FIG. 1, the generic objective model 145 for generating the ABR request 140 may include any one of a number of models to estimate QoE. For example, to build the ABR ladder 137 (or another media delivery method), typically one generic model is used that includes multiple versions of bitrate / quality of media segments of the reference source media by analyzing audio / video signals and optimizing coding efficiency. Another model is used to select the appropriate bitrate / quality decision for ABR streaming based on network conditions and loading of the playback device. However, none of the illustrated types of models take into account the user's personalized viewing characteristics. The personalized sensitivity profile 215 may be applied to any type of generic model 145 to build a personalized objective model 405 from the generic objective model 145.

[0033] FIG. 4 shows a block diagram of the generic objective model 145 being transformed into a personalized objective model 405 via model transformation 410. The transformation may occur according to one or more goals desired to be realized in the control and management of media delivery. For example, if the goal is to conserve bandwidth for media streaming by selecting the minimum bitrate in the ABR ladder 137 without degrading the personalized QoE for each user 135, the system 200, 300 may identify the JND (Just Noticeable Difference) of the image / video in the resolution-bitrate grid space. The system 200, 300 may then transform the generic objective model 145 into a personalized objective model 405 for each user 135 / playback system 210 based on the personalized JND of the user 135 identified based on the personalized sensitivity profile 215 of the user 135 in the environment 130 in which the user 135 is using the playback system 210.

[0034] As another example, for more sophisticated streaming management involving real-time or non-real-time pre-processing, decoding, transcoding or transrating in the loop, the system 200, 300 may estimate personalized psychometric functions such as spatial contrast sensitivity, temporal contrast sensitivity and spatio-temporal contrast sensitivity for achromatic and chromatic contrast stimuli to build a personalized objective model 405. For example, FIG. 5 shows a graph including two different contrast sensitivity functions (CFS) showing an example relationship between contrast sensitivity 505 and spatial frequency 510. The solid curve shows an ideal CSF 515. The dashed curve shows an example user CSF 520 of a user 135 determined based on the user response 220 to a test media 225 during a given test measurement session. As shown in FIG. 5, the user CSF 520 is transformed and scaled compared to the ideal CSF 515 due to, for example, a longer viewing distance in the environment 130 than the ideal viewing environment, a lower brightness of the TV screen than the ideal viewing environment, and the user 135 being myopic / nearsighted. 5, compared to the ideal CSF 515, the user 135 has a smaller range of perceptible differences in contrast sensitivity 505 versus spatial frequency 510. That is, the visual capabilities of the user 135 are not as sensitive as those of an ideal user, e.g., due to the environmental and personal conditions discussed above. For example, the user 135 may not be able to distinguish differences in contrast when the spatial frequency 510 is above the second value 530.

[0035] Thus, providing higher quality streaming of media that may enable an ideal user with ideal contrast sensitivity to experience improved QoE in an ideal environment may not actually result in improved QoE for the example user 135 having the user CSF 520 shown in Figure 5. As such, providing this higher quality streaming would impose a sacrifice on the system 200, 300 (e.g., more bandwidth would be used since the media would be streamed using a higher bit rate), and this sacrifice made by the system 200, 300 would essentially be wasted since no improvement in QoE would be achieved for the user 135 viewing the streamed media.

[0036] To assist the system 200, 300 in controlling the streaming of media from the media server 105, the user CSF 520 is an example of data included in the personalized sensitivity profile 215 that is used to convert the generic objective model 145 into the personalized objective model 405 (see FIG. 4). For example, the generic objective model 145 may use data corresponding to the ideal CSF 515 or another generic CSF that is not personalized to the user's viewing environment and personal visual capabilities. On the other hand, the personalized objective model 405 may use the user CSF 520 that has been scaled and transformed to be personalized to the user's viewing environment and personal visual capabilities. The user CSF 520 may be used in combination with other factors and / or algorithms included in the generic objective model 145 to create the personalized objective model 405. As noted above in the example of streaming higher quality media than user 135 can perceive, the personalized objective model 405 may be utilized during video encoding, transcoding, and / or transrating to improve coding efficiency and improve network efficiency (e.g., by reducing bandwidth) without affecting the personalized QoE of user 135. For example, higher quality media may be streamed to a more sensitive user 135 who can perceive the higher quality media, while lower quality media may be streamed to a less sensitive user 135 who cannot perceive the difference in quality between the lower quality media and the higher quality media.

[0037] 5 references the contrast sensitivity of user 135, the generic objective model 145 may include values ​​and / or functions for other types of viewing characteristics (e.g., temporal degradation related to video frame rate, quantization degradation, etc.) that may be personalized based on user responses 220 to test media 225. For example, generic media parameter values ​​and / or functions characterizing the generic temporal degradation model may be replaced or retrained to generate personalized values ​​and / or functions in the personalized objective model 405. Additionally or alternatively, generic algorithms used to identify media parameters such as ABR requests 140 (or requests for other media delivery methods) may have coefficients adjusted to generate the personalized objective model (POM) 405.

[0038] FIG. 6 illustrates a modified adaptive bitrate (ABR) based media coding and delivery system 600. The system 600 is similar to the system 100 of FIG. 1, but instead of the general objective model 145, it includes a personalized objective model (POM) 405 that is used to generate the ABR request 140 (or a request for other media delivery methods). In some embodiments, the system 600 includes at least one of the portions of the systems 200, 300 illustrated in FIG. 2 and FIG. 3. For example, as illustrated in FIG. 6, the personalized objective model 405 takes into account the playback system information 230, the environmental sensing information 310, and the personalized viewing characteristics information from the PSP 215, which are determined based on the user responses 220 to the test media 225 as described above.

[0039] Additional data sources that may be used by the POM 405 to generate ABR requests 140 (or requests for other media delivery methods) include real-time media player state information, including, but not limited to, buffer size, playback state, player performance characteristics, etc. Another data source that may be used by the POM 405 includes real-time network performance estimates, such as throughput measured by the playback system 210, throughput measured from sensors located in the network 115, congestion notifications, delays, packet loss rates, etc. Other data sources that may be used by the POM 405 include content metadata including bit rate, resolution, frame rate, bit depth per sample, chroma sampling, source encoding method (including levels and profiles), color space, supplemental enhancement message (SEI), composition playlist, group of pictures (GOP) size, instantaneous decoding refresh (IDR) frames, maximum frame average light level (MaxFALL), maximum content light level (MaxCLL), electro-optical transfer function (EOTF), language, service type, scene description (including boundary information), number of audio channels, audio sample rate, audio sample bit depth, audio service type, digital signature method, SCTE35 message, caption data, program volume, regulatory information, rating information, etc. In some embodiments, the additional data sources described herein may be referred to as media parameters.

[0040] Another data source that may be used by the POM 405 includes network operator policy parameters including maximum allowed bit rate per downstream and / or upstream channel or channel equivalent, spatial resolution, frame rate, etc. This example data source may enable network-wide and session-to-session optimization. Another data source that may be used by the POM 405 includes playback environment sensor information 310 as described above (e.g., ambient brightness level, ambient audio noise level, number of people viewing the streamed content, distance of each viewer from the screen, etc.). Another data source that may be used by the POM 405 includes auxiliary mobile device information such as distance from the main playback system 210, mobile device sensor information, etc. Another data source that may be used by the POM 405 includes real-time user / viewer preferences that may be entered by the user 135 and stored by the memory of one of the devices included in the system 600.

[0041] FIG. 7 is a hardware block diagram of a playback system 210 (i.e., a playback device) according to one example embodiment. As mentioned above, the playback system 210 may include many different types of playback systems, such as a television, a tablet, a smartphone, a computer, etc. In the illustrated embodiment, the playback system 210 includes a first electronic processor 705 (e.g., a microprocessor or other electronic device). The first electronic processor 705 includes an input / output interface (not shown) and is electrically coupled to a first memory 710, a first network interface 715, an optional microphone 720, a speaker 725, and a display 730. In some embodiments, the playback system 210 includes fewer or additional components in a configuration different from that shown in FIG. 7. For example, the playback system 210 may not include a microphone 720. As another example, the playback system 210 may include one or more additional input devices, such as a computer mouse and / or a keyboard, that receive input from a user of the playback system 210. As yet another example, playback system 210 may include environmental sensors, such as an ambient light sensor and / or a location tracking device (e.g., a Global Positioning System (GPS) receiver). In some embodiments, playback system 210 performs functions in addition to those described below.

[0042] The first memory 710 may include read only memory (ROM), random access memory (RAM), other non-transitory computer readable media, or a combination thereof. The first electronic processor 705 is configured to receive instructions and data from the first memory 710 and, among other things, execute the instructions. In particular, the first electronic processor 705 executes the instructions stored in the first memory 710 to perform the methods described herein.

[0043] The first network interface 715 transmits and receives data to and from the media server 105 via the network 115. In some embodiments, the first network interface 715 includes one or more transceivers for wireless communication with the media server 105 and / or the network 115. Alternatively or additionally, the first network interface 715 may include a connector or port for receiving a wired connection to the media server 105 and / or the network 115, such as an Ethernet cable. The first electronic processor 705 may receive one or more data streams (e.g., video streams, audio streams, image streams, etc.) via the network 115 through the first network interface 715. The first electronic processor 705 may output one or more data streams received from the media server 105 via the first network interface 715 via the speaker 725, the display 730, or a combination thereof. In addition, the first electronic processor 705 may return data generated by the playback system 210 via the network 115 to the media server 105 via the first network interface 715. For example, the first electronic processor 705 may identify and send the ABR request 140 mentioned earlier in this specification to the media server 105. The media server 105 may then send one or more media streams to the playback system 210 in accordance with the ABR request 140 from the playback system 210.

[0044] The display 730 is configured to display images, video, text, and / or data to the user 135. The display 730 may be a liquid crystal display (LCD) screen or an organic light emitting display (OLED) screen. In some embodiments, a touch-sensitive input interface may be incorporated into the display 730, allowing the user 135 to interact with the content provided on the display 730. In some embodiments, the display 730 includes a projector or a future developed display technology. In some embodiments, the speaker 725 and the display 730 are referred to as output devices that provide media streams and other information to the user 135 of the playback system 210. In some embodiments, the microphone 720, computer mouse, and / or keyboard or touch-sensitive display are referred to as input devices that receive input from the user 135 of the playback system 210.

[0045] FIG. 8 is a block diagram of a media server 105 according to one example embodiment. In the illustrated example, the media server 105 includes a second electronic processor 805 electrically connected to a second memory 810 and a second network interface 815. These components are similar to the similarly named components of the playback system 210 described above with respect to FIG. 7 and function similarly as described above. In some embodiments, the second network interface 815 transmits and receives data to and from the playback system 210 over the network 115. In some embodiments, the media server 105 includes fewer or additional components in a configuration different from that shown in FIG. 8. For example, the media server 105 may include an additional display, such as a touch screen, to allow a back-end user to reprogram settings or rules of the media server 105. In some embodiments, the media server 105 performs functions other than those described below.

[0046] 7 and 8 show separate block diagrams of the playback system 210 and the media server 105, in some embodiments the media server 105, one or more playback systems 210, a remote cloud computing cluster communicating over or forming part of the network 115, or a combination thereof, are referred to as an electronic computing device performing the functions described herein. For example, the electronic computing device may be a single electronic processor (e.g., the second electronic processor 805 of the media server 105) or multiple electronic processors located within the media server 105. In other embodiments, the electronic computing device includes multiple electronic processors distributed across different devices. For example, the electronic computing device is implemented in one or more of the first electronic processor 705 of the playback system 210, the second electronic processor 805 of the media server 105, and one or more electronic processors located in a remote cloud computing cluster communicating over or forming part of the network 115 or in one or more other devices located remotely. In some embodiments, the remote cloud computing cluster includes a software-defined network (SDN) / network function virtualization (NFV) enabled access network.

[0047] In some embodiments, the device implementing the POM 405 may determine the goals and capabilities of the POM 405. For example, an implementation of the POM 405 in the playback system 210 allows for distributed operation in the absence of network operator or other control signals. On the other hand, an implementation of the POM 405 in the media server 105 and / or the network 115 (e.g., as a network virtualization function (NVF) located on a software-defined network (SDN) node) may simplify the deployment of network-wide QoE optimization and other network operator policies (e.g., optimization of network services to desired subscriber QoE, edge / access network capacity targets, or a combination of both).

[0048] One or more of the hardware components of the playback system 210 shown in Figure 7 implement and / or configure the functional components of the playback system 210 shown in Figures 2, 3, and 6. For example, the first electronic processor 705 (or multiple first electronic processors 705 of the playback system 210) may function as one or more of the buffer / decoder 120 and the playback renderer 125. The first electronic processor 705 may also determine the personalized sensitivity profile 215, the personalized objective model 405, and the ABR request 140.

[0049] In some embodiments, one or more personalized sensitivity profiles (PSPs) 215 of one or more users and environments are stored in a first memory 710 of the playback system 210. The first memory 710 may store additional information such as general playback system information 230 of the playback system 210 (e.g., screen size, product identification number, etc.). In some embodiments, the one or more personalized sensitivity profiles (PSPs) 215 of one or more users and environments are also or alternatively stored in a second memory 810 of the media server 105 and / or in a memory of a remote cloud computing cluster that communicates through or forms part of the network 115. In some embodiments, the cloud storage of a user's PSP 215 enables a secure link to the user's wired / wireless Internet Service Provider (ISP) or network-delivered media account (e.g., cable TV). Such a link may be useful for network operators to utilize individual PSPs 215 to generate more efficient media distribution across a portion of their subscriber base with PSPs 215 associated with those accounts, as described in more detail herein.

[0050] FIG. 9 illustrates a flow chart of a method 900 for delivering media to the playback system 210, according to one example embodiment. The method 900 is described as being performed by an electronic computing device that includes one or more of the electronic processors described previously herein. Although certain operations are described as being performed by an electronic processor of a particular device, in other embodiments, electronic processors of other devices may perform the same operations. Although a particular sequence of processing steps, message receipt, and / or message transmission is illustrated by way of example in FIG. 9, the timing and sequence of such steps, receipt, and transmission may be varied as appropriate without negating the objectives and advantages of the examples described in detail throughout the remainder of this disclosure.

[0051] At block 905, the media delivery method 900 is initiated. In some embodiments, the media delivery method 900 is initiated by the first electronic processor 705 of the playback system 210 in response to a user 135 turning on the playback system 210 and / or requesting that a data stream be output by the playback system 210.

[0052] In response to the media delivery method 900 being initiated, at block 910, one or more electronic processors of the electronic computing device retrieve a stored personalized sensitivity profile (PSP) 215 associated with at least one of the user 135, the playback system 210, and the environment 130 in which the playback system 210 is located. For example, the stored PSP 215 may have been generated based on a previous test measurement session of the playback system 210. In some embodiments, the stored PSP 215 may be used by the electronic computing device to provide output media to the playback system 210 without performing a new test measurement session. For example, if the playback system 210 is being used in an environment 130 that has already undergone a test measurement session by the same user 135, the electronic computing device may use the stored PSP 215 corresponding to the environment 130 and the user 135. In some embodiments, the electronic computing device determines that the current environment and user have already undergone a test measurement session by comparing playback system information, environment information, and / or user information (e.g., user login information received by the playback system 210) with stored information of the PSP 215. For example, the electronic computing device may identify an identification number of the playback system 210, one or more characteristics of the environment 130 (e.g., time of day, amount of ambient light, location of the playback system 210, etc.), and the identity of the user 135. If this identification information matches a PSP 215 already stored in one of the memories of the electronic computing device, the electronic computing device may control the provision of output media to the playback system 210 and the display of the output media according to the corresponding previously stored PSP 215 without proceeding to block 915 to perform a new test measurement session.

[0053] Alternatively, the method 900 may continue to block 915 where the playback system 210 is controlled to perform a new test measurement session. Here, the new test measurement session may be a full session or an abbreviated session based on prior knowledge of one or more stored PSPs 215. For example, if one or more of the above-mentioned types of identification information do not match the stored PSPs 215, the electronic computing device may perform a new test measurement session and generate a new PSP 215, as described in more detail below. In some embodiments, the new test measurement session is initiated by the user 135 (e.g., via a user input to an input device of the playback system 210). In some embodiments, the electronic computing device may determine that at least one characteristic of the previously stored PSP 215 has changed (e.g., a power outage, a change in Internet Protocol (IP) address, a change in WiFi signal strength, a newly detected peripheral device being coupled to the playback system 210, a change in detected ambient light, a change in the detected location of the playback system 210, etc.). In response, the electronic computing device may instruct the playback system 210 to recommend that the user 135 participate in a new test measurement session. For example, the playback system 210 may determine (e.g., based on a time of day measurement based on data received from the environmental sensor 305, etc.) that the user 135 is watching the playback system 210 at night instead of during the day. As another example, the media server 105 may determine that a new playback system 210 not associated with a previously stored PSP 215 is connected to the network 115. In response, the media server 105 may send a request to the playback system 210 to recommend that the user 135 participate in a test measurement session to generate a PSP 215.

[0054] Methods for conducting the test measurement session may include an initial set-top box (STB), digital media adapter (DMA), mobile device, or other playback system 210 setup step, integration as a third party application running on the playback system 210, or as a cloud service hosting both the test media 225 and / or the PSP 215. As previously described herein, results of the test measurement session may be stored locally on the playback system 210 and / or remotely as part of a cloud service enabling cross-platform and cross-service compatibility.

[0055] At block 915, the electronic computing device outputs the test media 225 to be viewed by the user 135. The test media 225 may be generated by the first electronic processor 705 of the playback system 210 or may be generated by the media server 105 and then received by the playback system 210. In some embodiments, the test media 225 is generated to measure a user's sensitivity / quality of experience (QoE). For example, at block 920, the electronic computing device receives user input (i.e., user response 220) from the user 135. The user input is related to a perception of the test media 225 by the user 135 and indicates a first personalized QoE of the user 135 with respect to the test media 225.

[0056] In some embodiments, the electronic computing device determines the user sensitivity / QoE of the user 135 by generating an optotype visual acuity measurement using test media 225 including a Snellen chart or an open ring pattern. In some embodiments, the electronic computing device also or alternatively determines the user sensitivity / QoE of the user 135 by generating a contrast sensitivity function (CSF) measurement using differently oriented sine wave gratings (see, for example, FIG. 5). In some embodiments, the CSF measurement may include a rapid CSF method using test media 225 including band-pass filtered Sloan letters. In some embodiments, the electronic computing device also or alternatively determines the user sensitivity / QoE of the user 135 by displaying the test media 225 in the form of an interactive game played by the user 135. In some embodiments, the electronic computing device also or alternatively determines the user sensitivity / QoE of the user 135 by performing a user sensitivity measurement based on a series of images or video material displayed as the test media 225.

[0057] In some embodiments, the electronic computing device may display the test media 225 in the form of a hybrid image. In some embodiments, the hybrid image is a still image that tends to have different interpretations depending on the user's viewing ability and environmental factors. For example, human viewers lose the ability to see image details as the viewing distance increases, resulting in an inability to distinguish between high-resolution and low-resolution video. In some embodiments, the hybrid image is a still image that results in two or more different interpretations to a human user that vary as a function of spatial frequency range and / or viewing distance. Based on the user response 220 to the displayed hybrid image, the electronic computing device may estimate the dominant and non-dominant spatial frequency ranges of the user 135 in the media viewing environment 130 without the use of explicit sensors.

[0058] To create a hybrid image, two different source images may be processed differently such that a particular spatial frequency range predominates for each image. For example, a first source image may be low-pass filtered and a second source image may be high-pass filtered. The low-pass filtered source image may then be combined (i.e., superimposed) with the high-pass filtered source image to create a hybrid image. Because the sensitive region of a given image in spatial frequency moves from lower to higher frequencies as the viewing distance of the user 135 decreases, a human user perceives a high-pass filtered source image more easily at a shorter viewing distance than at a longer viewing distance. Conversely, a human user perceives a low-pass filtered source image more easily at a longer viewing distance than at a shorter viewing distance. That is, either the low-pass filtered source image or the high-pass filtered source image is perceived as dominant by the user 135 depending on one or more viewing characteristics of the user 135.

[0059] 10A-10C show examples of hybrid images 1000 of three different sizes, with FIG. 10A showing the largest size image 1000 and FIG. 10C showing the smallest size image 1000. In the illustrated example, a first source image containing a dog's face 1005 is low-pass filtered (see low-pass filtered source image 1050 in FIG. 10D) and combined with a high-pass filtered second source image 1010 containing a cat's face 1010 (see high-pass filtered source image 1060 in FIG. 10E). As shown in FIG. 10A-10C, due to the high-pass filtering of the second source image, the cat's face 1010 is easier for a human user to perceive in the larger version of FIG. 10A than in the smaller versions of FIG. 10B and FIG. 10C. That is, due to the high pass filtering (see FIG. 10E), the hybrid image 1000 contains only the fine details of the second source image, including the face of the cat 1010, which are easier to perceive when the image 1000 is large (i.e., close / short viewing distance). Conversely, the dog's face 1005 is easier for a human user to perceive in FIG. 10A and FIG. 10B than in FIG. 10C. That is, due to the low pass filtering (see FIG. 10D), the hybrid image 1000 contains only the coarse details of the first source image, including the face of the dog 1005, which are easier to perceive when the image 1000 is small (i.e., long viewing distance). To aid the viewer's visual interpretation of FIG. 10A-FIG. 10C, FIG. 10A-FIG. 10E has the nose 1015 of the cat 1010 labeled in each figure and the nose 1020 of the dog 1005 labeled in each figure.

[0060] Although the generation of the hybrid image has been described above as including low-pass and high-pass filtered different source images, in some embodiments the hybrid image is additionally or alternatively generated using different band-pass filters. In some embodiments, varying the size of the source images results in a magnification or reduction in the spatial frequency domain. Thus, varying the size of the source images in combination with filtering is another method for generating a hybrid image.

[0061] At block 915, by displaying a series / plurality of hybrid images as test media 225 during a test measurement session, the electronic computing device may be able to identify viewing characteristics of the user 135 and environmental factors associated with the playback system 210. In some embodiments, the electronic computing device may change the size of the hybrid image displayed by the playback system 210. For example, the electronic computing device may change the size of the hybrid image until the user response 220 indicates that the user's perception of the hybrid image has changed from a first perception of the first source image to a second perception of the second source image. Based on the size of the hybrid image that was displayed at the time the user response 220 was received and based on the resolution and screen size of the playback system 210, the electronic computing device may be able to identify an estimated viewing distance of the user 135, an estimated CSF of the user 135, etc.

[0062] In some embodiments, the electronic computing device may vary the cutoff frequency of the low-pass and high-pass (or band-pass) filters of each source image used to create the hybrid image. For example, the electronic computing device may receive a first user input associated with a first perception of the first hybrid image by the user. In response, the electronic computing device may generate a second hybrid image using filters where at least one filter's cutoff frequency is based on the first user input associated with the first perception of the first hybrid image (see, e.g., FIGS. 11A and 11B). The electronic computing device then controls the playback system 210 to output the second hybrid image for viewing by the user 135.

[0063] In some embodiments, the electronic computing device may determine the cutoff frequency of the spatial filter (and / or other characteristics used to generate the hybrid image, such as the size of the hybrid image being displayed) based on media parameters and / or playback system parameters supported by the media server and network 115. For example, the electronic computing device may determine the cutoff frequency of the spatial filter together with the available video resolution in the ABR ladder 137 of the media server 105 (or together with available values ​​of other media parameters based on another media delivery method utilized by the media server 105). As another example, the electronic computing device may determine the cutoff frequency based on the available bit rate of the media server 105 / network 115, the available frame rate of the media server 105 / network 115, the device type of the playback system 210, the screen size of the display 730 of the playback system 210, and / or other parameters / attributes described herein above.

[0064] In some embodiments, the electronic computing device may identify a first value of a media parameter supported by the media server 105 and the network 115. The electronic computing device may also identify a second value of a media parameter supported by the media server 105 and the network 115. The electronic computing device may then generate and / or select a hybrid image based on the first value of the media parameter and the second value of the media parameter, such that the hybrid image includes a first interpretation corresponding to the first value of the media parameter and a second interpretation corresponding to the second value of the media parameter (see, e.g., FIGS. 11A and 11B). The electronic computing device may then control the playback system 210 to display the hybrid image on the display 730.

[0065] In some embodiments, the electronic computing device displays additional hybrid images based on user responses 220 to the previously displayed hybrid images, as described herein above. For example, the electronic computing device may generate and / or select the second hybrid image based on the first value of the media parameter and the third value of the media parameter (determined to be supported by the media server 105 and the network 115), such that the second hybrid image includes a third interpretation corresponding to the third value of the media parameter and a fourth interpretation corresponding to the first value of the media parameter.

[0066] In some embodiments, the hybrid image described in the examples above is generated by the electronic computing device by overlaying source images as described herein above. In other embodiments, the electronic computing device may retrieve a previously generated and stored hybrid image having characteristics corresponding to values ​​of the media parameters determined to be supported by the media server 105 and network 115.

[0067] During the test measurement session, the electronic computing device may receive user input from the user 135 using an input device of the playback device. The user input indicates that a first interpretation of the hybrid image is perceived by the user 135 when the hybrid image is displayed on the display 730. Based on the user input, the electronic computing device may determine that the user 135 is more sensitive to a first value of the media parameter (e.g., a first spatial frequency range, viewing distance, resolution, and / or the like) than a second value of the media parameter (e.g., a second spatial frequency range, viewing distance, resolution, and / or the like). In some embodiments, the electronic computing device generates a personalized sensitivity profile 215 of the viewing characteristics of the user 135 based on the determination that the user 135 is more sensitive to the first value of the media parameter. The personalized sensitivity profile 215 may include the first value of the media parameter. In some embodiments, the media server 105 may provide output media to the playback system of the user 135 via the network 115 in accordance with the personalized sensitivity profile 215, as described herein above.

[0068] Continuing with the example immediately above, the electronic computing device may determine, based on the user input, at least one of the subset of spatial frequencies of the hybrid image to which the user 135 is most sensitive (i.e., the contrast of the hybrid image) and the sizing of the hybrid image to which the user 135 is most sensitive. In some embodiments, the viewing characteristics of the personalized sensitivity profile 215 generated by the electronic computing device include at least one of the subset of spatial frequencies of the hybrid image to which the user is most sensitive and the sizing of the hybrid image to which the user is most sensitive.

[0069] Using hybrid images generated or selected based on media parameters and / or playback system parameters during a test measurement session, as shown in the above examples, the electronic computing device can, for example, identify how different media-centric parameters affect a user's personalized QoE. For example, the electronic computing device can identify how different video resolutions in the ABR ladder 137 affect a user's personalized QoE. That is, based on the user responses 220 to the test media 225, the electronic computing device estimates the range of dominant spatial frequencies that affect the user's perception and invisible spatial frequencies. This perception information can be used to improve the efficiency of media coding and delivery, as described herein. For example, the lowest video resolution in the ABR ladder 137 can be identified as below which the user 135 begins to experience quality degradation compared to a full resolution video.

[0070] 11A and 11B show graphs of an example of optimal ABR ladder estimation using hybrid images as test media 225 in a multi-step binary tree search during a test measurement session. In some embodiments, the electronic computing device obtains the available video resolutions (e.g., 360p, 540p, 720p, and 1080p) of media streaming of the media server 105 and the network 115 from a manifest file. According to the video resolutions, the electronic computing device determines the cutoff frequencies of low-pass and high-pass filters applied to the source images A and B to create the hybrid image. The vertical dotted lines shown in FIG. 11A and 11B drawn along the rows represent the upper frequency limits of the corresponding video resolutions in the ABR ladder 137. For example, the spectral content of a 540p video can only be used up to the second vertical dotted line from the left. The top graphs 1105, 1155 of Figures 11A and 11B, respectively, show the estimated contrast sensitivity function (CSF) as a function of spatial frequency [cycles per pixel] for a user 135 at a particular viewing distance. As shown in Figure 11A, the peak 1110 of sensitivity for user 135 lies between the 540p and 720p points.

[0071] The middle graph 1115 of FIG. 11A shows the sensitivity of the first hybrid image created from the filtered source images A and B. As shown by the graph 1115, the electronic computing device may first set the spatial frequency at which changes in human perception of the source images A and B of the first hybrid image may occur to 540p. Based on the user response 220 to the first hybrid image (i.e., the test media 225), the electronic computing device identifies whether the source image A or B is more perceptually dominant to the user 135. If the user response 220 indicates that the source image B is more dominant, the electronic computing device may generate a second hybrid image represented by the bottom graph 1120 of FIG. 11A. As shown by the bottom graph 1120, the electronic computing device may set the spatial frequency at which changes in human perception of the second source images A and B of the second hybrid image may occur to 720p based on the user response 220 of selecting the source image B from the display of the first hybrid image represented by the middle graph 1115. Through displaying multiple hybrid images dynamically / adaptively adjusted based on user responses 220, the electronic computing device is configured to narrow the dominant frequency range perceptible by the user 135. Based on the user responses 220 received throughout the test measurement session, the electronic computing device may be configured to determine an estimated CSF 1125, as shown in the top graph 1105 of Figure 11A. Although Figure 11A shows only two iterations of the graph representing the displayed hybrid images, in some embodiments, the electronic computing device displays additional hybrid images (i.e., test media 225) and receives corresponding additional user responses 220 during the test measurement session.

[0072] FIG. 11B is a similar example to FIG. 11A, but for a different user 135, environment 130 and / or playback system 210 (e.g., the same user 135 and playback device 210, but at a greater viewing distance than the example of FIG. 11A). Graphs 1155, 1160 and 1165 generally correspond to graphs 1105, 1115 and 1120, respectively, of FIG. 11A, with values ​​adjusted for different viewing situations as described above. As shown by graph 1160, the electronic computing device may first set the spatial frequency at which changes in human perception of source images A and B of the first hybrid image may occur to 540p. Based on the user response 220 to the first hybrid image (i.e., test media 225), the electronic computing device identifies whether source image A or B is more perceptually dominant to the user 135. Unlike the example shown in FIG. 11A, if the user response 220 indicates that source image A is more dominant, the electronic computing device may generate a second hybrid image represented by the bottom graph 1165 of FIG. 11B. As shown by the bottom graph 1165, the electronic computing device may set the spatial frequency at which human perception changes may occur for the second source images A and B in the second hybrid image to 360p based on the user response 220 of selecting source image A from the display of the first hybrid image represented by the middle graph 1160. As described above with respect to FIG. 11A, the electronic computing device may continue to display the hybrid image and receive the user response 220 to identify an estimated CSF 1170, as shown in the top graph 1155 of FIG. 11B.

[0073] 11A and 11B, the CSFs 1125 and 1170 differ from each other due to differences in one or more of the user 135, the environment 130, and / or the playback system 210. For example, the CSF 1170 in FIG. 11B has a sensitivity peak 1175 at a lower resolution than the sensitivity peak 1110 of the CSF 1125 in FIG. 11A. As another example, because the overall range of the CSF 1170 in FIG. 11B is smaller than that of the CSF 1125 in FIG. 11A, a user in FIG. 11A can distinguish different resolutions greater than about 540p, while a user in FIG. 11B cannot distinguish different resolutions greater than about 540p.

[0074] As is evident from the above description, the CSFs 1125 and 1170 are personalized CSFs based on user responses 220 received by the electronic computing device in response to the displayed hybrid images and / or other test media 225. The personalized CSFs identified by the electronic computing device are similar to the CSF 520 shown in FIG. 5 and described earlier herein. That is, instead of using general ABR logic (e.g., the ideal CSF 515 of FIG. 5) to control media streaming from the media server 105 to the playback system 210, the electronic computing device may use personalized bitrate / resolution decision rules to do so. In some embodiments, the electronic computing device generates a new personalized ABR ladder based on the user responses 220 to the test media 225 during the test measurement session.

[0075] In some embodiments, the one or more stored PSPs 215 may affect characteristics of the test media 225 output by the playback system 210 during the test measurement session. In some embodiments, the electronic computing device retrieves a previously stored personalized sensitivity profile (PSP) 215 and generates the test media 225 based on one or more viewing characteristics included in the previously stored PSP 215. In some embodiments, to retrieve the previously stored PSP 215, the electronic computing device identifies characteristics of the current / ongoing test measurement session including at least one of the characteristics of the user 135, the characteristics of the first playback system 210, and the characteristics of the environment 130 in which the user 135 is viewing the first playback system 210. The electronic computing device may then identify a previously stored PSP 215 from the multiple previously stored PSPs 215 based on the previously stored PSP 215 including one or more of the same characteristics as the characteristics of the current / ongoing test measurement session.

[0076] For example, the electronic computing device may determine that the stored PSP 215 contains information about the same user 135, but the current playback system 210 and / or the current environment 130 are different from the stored playback system 210 and / or the environment 130 (e.g., the same user is watching television on a different television in a different room in the home). Even though the characteristics of the stored PSP 215 do not exactly match the current situation, the electronic computing device may still begin outputting test media 225 during a test measurement session using one or more media parameters of the stored PSP 215 as a baseline. That is, rather than outputting the test media 225 randomly or according to a general model, the electronic computing device may output test media 225 that is filtered or otherwise modified according to the stored PSP 215 (such as a hybrid image). In some circumstances, outputting the test media 225 based on the media parameters contained in the stored PSP 215 may shorten the duration of the test measurement session and / or improve the measurement accuracy that best suits the current situation. For example, when a contrast sensitivity function (CSF) is generated by an electronic computing device to define personalized sensitivity information, typically several tens of measurements are required to accurately estimate the media parameters of the CSF in one test measurement session. However, if the electronic computing device starts the test measurement session from a starting point that has already been measured for a user 135 in a different environment 130 and / or by a different playback system 210 (or for another common attribute other than having a common user 135), the number of measurements required to accurately estimate the media parameters in the current situation can be reduced compared to the typical amount. That is, the spatial frequency and contrast of the current stimulus for CSF measurement in the current test measurement session can be adjusted according to the user response 220 of the previous test media 225 and the estimate of the CSF from the previously stored PSP 215.

[0077] Similarly, in some embodiments, when generating a PSP 215 to estimate and optimize QoE, a single PSP 215 may be estimated from multiple stored PSPs 215 or selected to closely match other identified attributes (e.g., location, demographic, viewing device make / model, screen size, etc.). For example, if the electronic computing device detects a change in the user 135, environment 130, and / or playback device 210 and the user 135 chooses not to participate in a new test measurement session, the electronic computing device may generate an estimated PSP 215 based on multiple stored PSPs 215 for similar users 135, environments 130, and / or playback devices 210.

[0078] In the case of multiple viewers (each with a unique or unknown PSP) for a single playback system 210 (e.g., a television in a home with multiple users / viewers), the electronic computing device may select a single PSP 215 based on many different criteria. For example, if the goal of the system 600 is to minimize the risk of perceived QoE degradation for any user, the electronic computing device may select the most sensitive PSP 215 from a group of PSPs 215 corresponding to each of the multiple viewers. In this example, the electronic computing device attempts to ensure that even the most sensitive user viewing the display 730 does not experience a degradation in QoE. Assuming that the most sensitive user does not experience a degradation in QoE, a less sensitive user viewing the same display 730 will not experience a degradation in QoE either, since they are less sensitive to changes in image / video quality than the most sensitive user. In some embodiments, system 600 may reduce the number of PSP candidates for a given playback system 210 (e.g., a television in a home with multiple users / viewers) based on user presence information derived, for example, from other applications (e.g., a smart home application) or GPS information from a personal mobile device.

[0079] At block 925, the electronic computing device determines whether it has gathered enough information to complete the personalized sensitivity profile 215. As discussed above, this information may be gathered from current user responses 220 to current test media 225 (at block 920) and / or retrieved from a previously stored PSP 215 (at block 910). In FIG. 9, block 910 is shown with a dashed line to indicate that block 910 is optional and may not occur in some implementations of method 900. That is, in some circumstances, the electronic computing device may generate (at block 930) a PSP 215 based on received user responses 220 to test media 225 without retrieving a previously stored PSP 215.

[0080] Conversely, although blocks 915, 920, and 925 are not shown in dashed lines in FIG. 9, in some circumstances blocks 915, 920, and 925 may not be performed by the electronic computing device. That is, the electronic computing device may not perform a test measurement session in some circumstances, but instead rely solely on one or more stored PSPs 215 to generate the PSP 215 used for the current media session. For example, in retrieving (at block 910) the stored PSPs 215 for the user 135 and / or the environment 130, the electronic computing device may determine that one of the stored PSPs 215 corresponds to the user 135, the environment 130, and the playback device 210. Thus, at block 930, the electronic computing device may utilize the corresponding previously stored PSP 215 as the PSP 215 for the current media session of the user 135 on the playback device 210 in the environment 130. In this situation, there is no need for the electronic computing device to perform a test measurement session. This is because the display characteristics of the current media delivery session were pre-stored in PSP 215 during the previous test measurement session.

[0081] As another example where the electronic computing device does not perform a test measurement session (i.e., does not perform blocks 915, 920, and 925 of FIG. 9 ), as described above, if the electronic computing device detects a change in the user 135, the environment 130, and / or the playback device 210, and the user 135 chooses not to participate in a new test measurement session, the electronic computing device may generate an estimated PSP 215 based on multiple stored PSPs 215 with similar users 135, environments 130, and / or playback devices 210. For example, if the stored PSPs 215 are associated with the same user, the electronic computing device may adjust one or more characteristics of the stored PSP 215 based on a known change in display size or other display characteristics between the playback system 210 associated with the stored PSP 215 and the playback system 210 currently being used by the user 135. Similarly, the electronic computing device may adjust one or more characteristics of the stored PSP 215 based on a known change in the environment of the user 135. For example, based on sensor data from the environmental sensors 305, the electronic computing device may determine that the current environment 130 is darker than the environment 130 associated with the stored PSP 215. In other embodiments, instead of generating an estimated PSP 215, the electronic computing device may retrieve and use a stored PSP 215 that includes similar characteristics to the identified and / or known characteristics of the user 135, the environment 130, and / or the playback device 210. For example, the electronic computing device may retrieve a stored PSP 215 for the user 135, even if the stored PSP 215 is for a different environment 130 and / or a different playback system 210.

[0082] Returning to the discussion of block 925, if the electronic computing device determines that more information is desired to complete the PSP 215 (e.g., to more accurately complete the CSF as shown in FIGS. 5, 11A and 11B), the method 900 returns to block 915 and continues to output the test media 225 and receive user input (i.e., user responses 220) responsive to the test media 225. If the electronic computing device determines that sufficient information has been collected to complete the PSP 215, the method proceeds to block 930.

[0083] At block 930, the electronic computing device generates a personalized sensitivity profile (PSP) of one or more viewing characteristics of the user based on the user input. For example, the electronic computing device generates a personalized CSF 520 that is transformed and / or scaled from the ideal CSF 515, as shown in FIG. 5. Additionally or alternatively, algorithms used to identify streaming parameters, such as an ABR request 140 (or a request for another media delivery method), may have coefficients adjusted and stored in the PSP 215. In some embodiments, the electronic computing device generates a personalized ABR ladder (or another personalized media delivery method) to be included in the PSP 215.

[0084] At block 935, the electronic computing device identifies media parameters based at least in part on the PSP 215. For example, the electronic computing device determines values ​​of the media parameters (e.g., values ​​of one or more of segment size, bit rate, resolution, frame rate, and another media parameter that affects the operation of a video encoder / transcoder / translator associated with the media server 105 and / or the network 115). At block 940, the media server 105 provides output media to the playback system 210 over the network 115 in accordance with the media parameters. The output media is configured to be output to the playback system 210 (e.g., images / video configured to be output on the display 730 of the playback system 210).

[0085] To determine the media parameters (i.e., values ​​of the media parameters) at block 935, the electronic computing device may perform a translation of the general objective ABR logic model to a personalized objective ABR logic model (POM 405) as shown in and described earlier in this specification with respect to FIG. 4. In some embodiments, the POM 405 may be implemented within a streaming system that supports adaptive bitrate delivery, such as a television, a set-top box, a digital media adapter, or a mobile device, as shown in FIG. 6. In this example, the POM 405 is utilized by the ABR request logic 140 of the playback system 210 to improve the selection of encoded video and / or audio segments based on, but not limited to, segment size, bit rate, resolution, frame rate, codec, etc., to match or provide the closest match among the available coded segments to the PSP 215 of the user 135. For example, algorithms defining a general objective ABR logic model (or another general media delivery method) used to determine streaming parameters such as ABR request 140 (or a request for another media delivery method) may have their coefficients adjusted / personalized according to information stored in PSP 215.

[0086] As previously described herein, the ABR ladder 137 of FIG. 6 represents a collection of available audio and video segments across a range of bit rates, resolutions, frame rates, etc. Also, as previously described herein, existing media delivery systems / methods (whether ABR is enabled or not) are inefficient, often leading to the playback system 210 requesting more data than is needed for seamless playback and / or requesting values ​​of media parameters (e.g., resolution / bit rate / frame rate combinations) that exceed the PSP 215 of the user 135. That is, existing media delivery logic that attempts to increase the delivered resolution / bit rate / frame rate beyond the sensitivity threshold of the user 135 does not translate into improved QoE for the user 135. Because the disclosed POM-based media delivery method translates into more efficient delivery, it translates into reduced delivery costs for over-the-top (OTT) services, as these services pay per gigabyte extracted from their content delivery network (CDN) vendors.

[0087] In some embodiments, at block 935, the electronic computing device selects values ​​for one or more media parameters (e.g., a resolution / bit rate / frame rate combination) that result in the streamed media being within the range of the sensitivity perception of the user 135. For example, the electronic computing device may use CSF 1125 of FIG. 11A to control the ABR request 140 to the media server 105 to request a resolution of the streamed media of about 720p because CSF 1125 indicates that the first user's sensitivity peak 1110 is about 720p. On the other hand, the electronic computing device may use CSF 1170 of FIG. 11B to control the ABR request 140 to the media server 105 to request a resolution of the streamed media of about 540p, which is a lower resolution, because CSF 1170 indicates that the second user's sensitivity peak 1175 is about 540p. In the two examples immediately above, the first media parameter (i.e., 540p resolution) and the second media parameter (i.e., 720p resolution) are determined such that providing the first output media to the first playback device 210 of the second user reduces a first usage of the network 115 resources to be less than a second usage of the network 115 resources associated with providing the second output media to the second playback device 210 of the first user. However, despite the difference in resource usage in streaming the output media to the two playback devices 210, the first percentage of the first personalized QoE of the first user remains at approximately the same level as the second percentage of the second personalized QoE of the second user (see FIG. 12B and Table 1B).

[0088] Although the method 900 has been described above with respect to a media session for a single playback system 210 or two playback systems 210, in some embodiments the method 900 may be performed with respect to additional playback systems 210. For example, the method 900 may be used to determine a PSP 215 for each of a plurality of playback systems 210 that receive media streams from a particular node on the network 115. The electronic computing device may improve / optimize one or more media parameters (e.g., encoding and delivery parameters) of each media stream provided to each of the plurality of playback systems 210 to improve / optimize the media streams from the network 115 collectively / as a whole.

[0089] For example, mobile wireless and broadband network operators can utilize the disclosed POM-based media distribution and encoding method 900 to add additional capacity to existing access networks without trading off end user / viewer QoE. In some embodiments, the method 900 provides network operators with a new way to reduce the rate of capital investment required to increase the capacity of their networks. FIG. 1B shows exemplary bandwidth and QoE statistics for network 115 when streaming media using existing streaming methods and when streaming media using method 900. Servers using existing methods are referred to as existing servers, whereas servers using method 900 are referred to as xCD servers (i.e., experience coding and delivery servers).

[0090] FIG. 12A shows a chart 1205 for a network with no bandwidth cap. The upper curve 1210 represents the bandwidth used by media streamed by an existing server. The lower curve 1215 represents the bandwidth used by media streamed by an xCD server using method 900. As shown in FIG. 12A, the bandwidth used by media streamed by an xCD server using method 900 is approximately 20% less than the bandwidth used by the same media streamed by an existing server. In addition, as shown in Table 1A below, which corresponds to FIG. 12A, all viewers (e.g., both high sensitivity users (i.e., close viewers) and low sensitivity users (i.e., distant viewers) remain at 100% (i.e., perfect QoE).

[0091] [Table 1A]

[0092] 12A and Table 1A, the method 900 results in more efficient media delivery without degrading user QoE for a population of users each receiving a unicast session based on a respective PSP 215. This increased efficiency and reduced bandwidth without degrading user QoE is a result of the system 600 reducing the bitrate, resolution, etc. of the streamed media based on the PSP 215 of users 135 who do not experience an improvement in QoE when the bitrate, resolution, etc. are increased beyond a certain point. That is, in some embodiments, the method 900 may be aimed at delivering the highest perceivable quality of media to each user (i.e., personalized media content delivery) without delivering a higher quality of media to a particular user than the particular user can perceive.

[0093] FIG. 12B shows a chart 1250 for a network with a capped bandwidth / fixed network capacity (e.g., about 60 Mbps). Curve 1255 represents the bandwidth used by media streamed by an incumbent server. Curve 1260 represents the bandwidth used by media streamed by an xCD server using method 900. Unlike the curves in FIG. 12A, curves 1255 and 1260 in FIG. 12B use approximately the same bandwidth over time. In some embodiments, the bandwidth used by media streamed by an xCD server using method 900 may be approximately 1% less than the bandwidth used by the same media streamed by an incumbent server. However, as shown by Table 1B below, which corresponds to FIG. 12B, the use of method 900 by an xCD server over a fixed-capacity network link results in a more uniform degradation of QoE across high and low sensitivity users compared to an incumbent server implementing a conventional ABR segment selection approach.

[0094] [Table 1B]

[0095] For example, Table 1B shows that in a capped bandwidth network, when media is streamed by an incumbent server, high sensitivity users (i.e., users closest to their respective playback systems 210) experience a QoE degradation of about 40%. In contrast, in a capped bandwidth network, when media is streamed by an incumbent server, low sensitivity users (i.e., users farthest from their respective playback systems 210) experience a QoE degradation of only about 20%. This difference in QoE degradation is due to the incumbent server degrading streaming quality equally for all users, even though changes in streaming quality affect different users differently.

[0096] On the other hand, the same capped bandwidth network can provide a more uniform reduction in QoE among all users of the system 600 because the user PSP 215 is used by the xCD server to more intelligently reduce streaming quality in different ways for different users. In some embodiments, the more uniform reduction in QoE results in an overall higher QoE for users of the system 600. For example, Table 1B shows that when media is streamed by the xCD server, in a capped bandwidth network, high sensitivity users (i.e., users closest to their respective playback systems 210) experience only about a 20% reduction in QoE. Similarly, when media is streamed by the xCD server, in a capped bandwidth network, low sensitivity users (i.e., users farthest from their respective playback systems 210) experience only about a 20% reduction in QoE. That is, as shown by Table 1B, the xCD server implementing the method 900 significantly improves the QoE of high sensitivity users while moderately reducing or maintaining the QoE of low sensitivity users.

[0097] 13A and 13B show another example of how the method 900 allows more users / subscribers to stream media over a fixed capacity network without adversely affecting QoE. FIGs. 13A and 13B show example graphs 1305 and 1350 of the percentage of subscribers experiencing QoE degradation (y-axis) versus the number of subscribers served (x-axis) using a fixed capacity network of 1.89 Gbit / s (4096-QAM) with an effective throughput of 88% after accounting for overhead. The graphs 1305, 1350 in FIGs. 13A and 13B assume a 50% split between high sensitivity (e.g., 3 image heights (3H) away from each playback system 210) and low sensitivity (e.g., 6 image heights (6H) away from each playback system 210) viewers.

[0098] Graphs 1305, 1350 show that as users / subscribers are added to a fixed capacity network, the video resolution must be reduced (as an example) when the number of users / subscribers reaches a certain threshold 1310, 1355. However, similar to the above examples with respect to Figures 12A and 12B and Tables 1A and 1B, uniformly reducing the resolution for all users / subscribers (as shown in Figure 13A) will result in uneven QoE depending on whether the user / subscriber is watching at 3H or 6H. For example, if the same resolution reduction is implemented for all media streams, a user watching at 3H will generally perceive a larger degradation in QoE than a user watching at 6H. This difference in QoE degradation between different types of users is illustrated by the 3H curve 1315 and the 6H curve 1320 shown in Figures 13A and 13B. The shaded area 1325 between the 3H curve 1315 and the 6H curve 1320 shows the non-uniformity of QoE degradation between high and low sensitivity users / subscribers. For example, for a network serving 1200 users, 60% of the users at 3H (i.e., high sensitivity users) will experience degradation in video quality, but only about 2% of the users at 6H (i.e., low sensitivity users).

[0099] Figure 13A shows a first QoE curve 1330 illustrating the QoE experienced by different users / subscribers when using a streaming management method that similarly adapts video coding bit rate and resolution based on the viewing distance of all users / subscribers. On the other hand, Figure 13B shows a second QoE curve 1360 illustrating the QoE experienced by different users / subscribers when the method 900 of Figure 9 is used to personalize the resolution degradation for different users as additional users are added to the network. As mentioned above, both Figures 13A and 13B assume a 50% split in the viewing distances of 3H and 6H among users / subscribers.

[0100] By using the method 900, the electronic computing device controlling the media parameters understands (e.g., based on information stored in each user's PSP 215) which users, at what distances, and how each user's QoE is affected by a reduction in resolution. The electronic computing device executing the method 900 can then allocate bitrate / resolution combinations to achieve equal average QoE for both groups of users (i.e., high sensitivity users and low sensitivity users). This improvement is illustrated by the difference between the QoE curve 1330 in FIG. 13A and the QoE curve 1360 in FIG. 13B. For example, the threshold 1310 in FIG. 13A, at which any user experiences a reduction in QoE, is for a network serving about 600 users. In contrast, the threshold 1355 in FIG. 13B is about doubled to about 1200 users. That is, a network executing the method 900 can serve about twice as many users as the existing network without any users experiencing a reduction in QoE.

[0101] Returning to FIG. 9 , in some embodiments, the electronic computing device repeats blocks 935 and 940, as indicated by the dashed arrows in the diagram. The repetition of these blocks enables real-time tracking of sensor outputs in the playback environment 130 (e.g., environmental sensors 305) and / or network 115 during the media delivery service, and correspondingly enables dynamic / adaptive adjustment of the personalized objective model 405 for various use cases. For example, the system 600 may determine that the room in which the playback system 210 is located has become dark since the user 135 began watching television (e.g., the sun has gone down, the user 135 has closed the curtains in the room, etc.). In response to this determination, the system 600 may adjust the personalized objective model 405, for example, by selecting another stored PSP 215 that includes ambient light characteristics that are more similar to the characteristics of the currently darker room.

[0102] As previously described herein, the ABR ladder 137 and ABR selection methods referred to herein are merely example methods that may be used by the system 600 to control media delivery from the media server 105 to the playback system 210 over the network 115. In other embodiments, other methods may be used to dynamically adjust parameters of a video encoder / transcoder / translator (i.e., media parameters), such as the bit rate and / or resolution of the coded media being streamed. As with the ABR-related methods in many examples, these other media delivery methods have their media parameters adjusted based on one or more PSPs 215 to optimize the media delivery, as described herein. In some embodiments, the media delivery method is an upstream media delivery method implemented by the media server 105 and / or the network 115 (i.e., upstream of the playback system 210).

[0103] It should be understood that the application of the present embodiment is not limited to the detailed configuration and arrangement of components described herein or illustrated in the accompanying drawings. The embodiments may be practiced or performed in various ways. It should also be understood that the phraseology and terminology used herein is for purposes of description and should not be regarded as limiting. The use of "including" or "having" and variations thereof means including additional items in addition to the items listed thereafter and their equivalents. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled" and variations thereof are used in the broadest sense to encompass both direct and indirect mounting, connecting, supporting, and coupling.

[0104] In addition, it should be understood that the embodiments may include hardware, software, and electronic components or modules, which for purposes of explanation may be shown and described as if most of the components were implemented solely in hardware. However, those skilled in the art will recognize, based on reading this detailed description, that in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored in a non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and / or an application-specific integrated circuit ("ASIC"). Thus, multiple hardware and software-based devices and multiple different structural components may be utilized to implement the embodiments. For example, the "server" and "computing device" described herein may include one or more electronic processors, one or more computer-readable media modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0105] Various features and advantages are set forth in the following claims.

Claims

1. 1. A method for delivering media to a playback device, the method comprising: outputting, by a first playback device, a first test media for viewing by a first user during a first test measurement session; receiving a first user input from the first user, the first user input relating to a first perception of the first test media by the first user and indicating a first personalized quality of experience of the first user with respect to the first test media; generating, with one or more electronic processors, a first personalized sensitivity profile based on the first user input, the first personalized sensitivity profile including one or more viewing characteristics of the first user and / or the environment in which the first user is viewing the first playback device; determining, with the one or more electronic processors, first media parameters based at least in part on the first personalized sensitivity profile to enhance efficiency of media delivery over a network to the first playback device while preserving a first personalized quality of experience for the first user; providing, by a media server over the network, a first output media to the first playback device in accordance with the first media parameters, the first output media being configured to be output using the first playback device; Including, the first test media includes a plurality of hybrid images, the first user input includes a plurality of user inputs, each of the first user inputs being received in response to a respective hybrid image of the plurality of hybrid images, each of the plurality of hybrid images being a still image that tends to have different interpretations depending on the viewing capabilities of the first user and environmental factors or a still image that results in two or more different interpretations to the first user that vary as a function of spatial frequency range and / or viewing distance; Determining the first media parameter comprises: retrieving, with the one or more electronic processors, from a memory, a generic objective model configured to control media streaming based on at least one of resource availability of the network and playback system parameters; converting, using the one or more electronic processors, the generic objective model into a personalized objective model using the first personalized sensitivity profile; providing, by the media server, the first output media to the first playback device over the network in accordance with the personalized objective model; A method comprising:

2. 2. The method of claim 1 , wherein providing the first output media to the first playback device in accordance with the first media parameters results in the network using reduced bandwidth to provide the first output media to the first playback device without degrading a first personalized quality of experience of the first user.

3. outputting, by a second playback device, a second test media for viewing by a second user during a second test measurement session; receiving a second user input from the second user, the second user input relating to a second perception of the second test media by the second user and indicating a second personalized quality of experience of the second user with respect to the second test media, the second personalized quality of experience indicating that the second user is more sensitive to degradation in media quality than the first user of the first playback device; generating, with the one or more electronic processors, a second personalized sensitivity profile based on the second user input, the second personalized sensitivity profile including one or more viewing characteristics of the second user and / or an environment in which the second user is viewing the second playback device; determining, with the one or more electronic processors, second media parameters for the second playback device based at least in part on the second personalized sensitivity profile; and providing, with the one or more electronic processors, a second output media to the second playback device over the network in accordance with the second media parameters; Further comprising: the first media parameters and the second media parameters are determined such that providing the first output media to the first playback device results in a first usage of resources of the network that is lower than a second usage of resources of the network for providing the second output media to the second playback device; 3. The method of claim 1 or 2, wherein a first percentage of a first personalized quality of experience for the first user is maintained at approximately the same level as a second percentage of a second personalized quality of experience for the second user.

4. 4. The method of claim 3, wherein the one or more electronic processors include at least one of a first electronic processor of the first playback device, a second electronic processor of the second playback device, and a third electronic processor associated with the network or the media server.

5. using said one or more electronic processors, low pass filtering the first source image to produce a low pass filtered source image; high-pass filtering the second source image to produce a high-pass filtered source image; overlaying the low-pass filtered source image and the high-pass filtered source image on one another to create the hybrid image; generating each hybrid image by 5. A method according to claim 1, wherein either the low-pass filtered source image or the high-pass filtered source image is perceived as dominant by the first user depending on the one or more viewing characteristics of the first user and / or the environment.

6. Outputting the first test media includes: outputting, by the first playback device, a first hybrid image to be viewed by the first user; receiving the first user input from the first user, the first user input relating to a first perception of the first hybrid image by the first user; outputting, by the first playback device, a second hybrid image for viewing by the first user, wherein a cutoff frequency of at least one of the low pass filtering and the high pass filtering used to create the second hybrid image is based on the first user input associated with the first perception of the first hybrid image by the first user; The method of claim 5 , comprising:

7. Outputting the first test media includes: retrieving, with the one or more electronic processors, from a memory a previously stored personalized sensitivity profile; generating, with the one or more electronic processors, the first test media based on one or more viewing characteristics contained in the previously stored personalized sensitivity profile; The method of any one of claims 1 to 6, comprising:

8. Retrieving the previously stored personalized sensitivity profile includes: using the one or more electronic processors to identify characteristics of the first test measurement session, including at least one of a characteristic of the first user, a characteristic of the first playback device, and a characteristic of the environment in which the first user is viewing the first playback device; identifying, with the one or more electronic processors, from a plurality of previously stored personalized sensitivity profiles, the previously stored personalized sensitivity profile based on the previously stored personalized sensitivity profile including one or more of the same characteristics as the characteristics of the first test measurement session; The method of claim 7, comprising:

9. 9. The method of claim 1, wherein converting the generic objective model to the personalized objective model includes at least one of converting and scaling an ideal contrast sensitivity function (CSF) used by the generic objective model to create a personalized CSF based on the first personalized sensitivity profile.

10. 10. The method of claim 1, wherein the one or more viewing characteristics of the first user and / or the environment include at least one of a viewing distance between the first user and a display of the first playback device, lighting in the environment from which the first user is viewing the display, and visual sensitivity of the first user's eyes.

11. determining environmental conditions using an environmental sensor in the environment in which the playback device is located; 11. The method of claim 1, wherein generating the first personalized sensitivity profile including the one or more viewing characteristics of the first user and / or the environment comprises generating, using the one or more electronic processors, the first personalized sensitivity profile such that the first personalized sensitivity profile includes the environmental conditions.

12. 1. An electronic computing device comprising: a first playback device including a display, the display configured to output media to a first user; one or more electronic processors communicatively coupled to the display; Including, the one or more electronic processors: outputting, by the first playback device, a first test media for viewing by the first user during a first test measurement session; receiving a first user input from the first user, the first user input relating to a first perception of the first test media by the first user and indicating a first personalized quality of experience of the first user with respect to the first test media; generating a first personalized sensitivity profile based on the first user input, the first personalized sensitivity profile comprising one or more viewing characteristics of the first user and / or an environment in which the first user is viewing the first playback device; determining, based at least in part on the first personalized sensitivity profile, first media parameters for increasing efficiency of media delivery to the first playback device over a network while preserving a first personalized quality of experience for the first user; providing, by a media server over the network, a first output media to the first playback device in accordance with the first media parameters, the first output media being configured to be output using the first playback device; configured to: the first test media includes a plurality of hybrid images, the first user input includes a plurality of user inputs, each of the first user inputs being received in response to a respective hybrid image of the plurality of hybrid images, each of the plurality of hybrid images being a still image that tends to have different interpretations depending on the viewing capabilities of the first user and environmental factors or a still image that results in two or more different interpretations to the first user that vary as a function of spatial frequency range and / or viewing distance; Determining the first media parameter comprises: retrieving, with the one or more electronic processors, from a memory, a generic objective model configured to control media streaming based on at least one of resource availability of the network and playback system parameters; converting, using the one or more electronic processors, the generic objective model into a personalized objective model using the first personalized sensitivity profile; providing, by the media server, the first output media to the first playback device over the network in accordance with the personalized objective model; 2. An electronic computer device comprising:

13. 13. The electronic computing device of claim 12, wherein the one or more electronic processors include at least one of a first electronic processor of the first playback device and a second electronic processor associated with the network or the media server.

Images