Personalization sensitivity measurement and playback factors for adaptive personalization media coding and delivery

Abstract

A method for delivering media to a playback device includes outputting first test media for viewing by a first user. The method additionally includes receiving first user input related to a first perception of the first test media by the first user and indicative of a first personalized quality of experience with respect to the first test media by the first user. The method additionally includes generating a first personalized sensitivity profile curve including one or more viewing characteristics of the first user based on the first user input, and determining first media parameters based at least in part on the first personalized sensitivity profile curve. The first media parameters are determined so as to improve efficiency of media delivery to the first playback device via a network while maintaining the first personalized quality of experience of the first user.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 056,942, filed July 27, 2020, and U.S. Provisional Patent Application No. 62 / 882,068, filed August 2, 2019, both of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] This application generally relates to delivering visual media to user devices over a network and displaying the visual media by the user devices for viewing by users. SUMMARY

[0004] Various aspects of the present disclosure relate to apparatuses, systems, and methods that provide personalized and adaptive media transcoding and delivery based on playback-side information that is typically collected without individual sensors.

[0005] In one aspect of the disclosure, a method for delivering media to a playback device is provided. The method can include outputting, by a first playback device, first test media for viewing by a first user during a first test measurement session. The method can additionally include receiving first user input from the first user. The first user input can relate to a first perception of the first test media by the first user and can indicate a first personalized quality of experience of the first user with respect to the first test media. The method can additionally include generating, by one or more electronic processors, a first personalized sensitivity profile curve including one or more viewing characteristics of the first user based on the first user input. The method can additionally include determining, by the one or more electronic processors, first media parameters based at least in part on the first personalized sensitivity profile curve. The first media parameters can be determined so as to improve efficiency of media delivery to the first playback device via a network while maintaining the first personalized quality of experience of the first user. The method can additionally include providing first output media to the first playback device via the network in accordance with the first media parameters. The first output media is configured to be output by the first playback device.

[0006] In another aspect of this disclosure, an electronic computing device is provided that may include a first playback device comprising a display. The display is configured to output media to a first user. The electronic computing device may further include one or more electronic processors communicatively coupled to the display. The one or more electronic processors are configured to output first test media for the first user to view during a first test measurement session via the first playback device. The one or more electronic processors are further configured to receive first user input from the first user. The first user input may relate to a first perception of the first test media by the first user and may indicate a first personalized experience quality of the first user with respect to the first test media. The one or more electronic processors are further configured to generate a first personalized sensitivity distribution curve incorporating one or more viewing characteristics of the first user based on the first user input. The one or more electronic processors are further configured to determine first media parameters at least in part based on the first personalized sensitivity distribution curve. The first media parameters may be determined to improve the efficiency of media delivery via a network to the first playback device while maintaining the first personalized experience quality of the first user. The one or more electronic processors may be further configured to provide the first output medium to the first playback device via the network according to the first media parameters. The first output medium may be configured to be output through the first playback device.

[0007] In another aspect of this disclosure, a method for displaying a mixed image on a playback device is provided. The method may include determining, via one or more electronic processors of an electronic computing device, a first value of a media parameter supported by a media server and a network configured to stream media. The method may further include determining, via the one or more electronic processors, a second value of the media parameter supported by the media server and the network. The method may further include at least one of the following generation and selection operations: generating and selecting the mixed image via the one or more electronic processors based on the first value and the second value of the media parameter, such that the mixed image contains a first interpretation corresponding to the first value of the media parameter and a second interpretation corresponding to the second value of the media parameter. The method may further include displaying the mixed image on a display of the playback device.

[0008] Other aspects of the embodiments will become apparent from the detailed description and accompanying drawings. Attached Figure Description

[0009] Figure 1 This describes an instance of adaptive bit rate (ABR) based on a media decoding and delivery system according to the embodiments described herein.

[0010] Figure 2 and 3 This section describes a portion of the adaptive bit rate (ABR) of a media decoding and delivery system configured to determine individualized / personalized viewing characteristics during a test measurement session, based on some embodiments described herein.

[0011] Figure 4 This diagram illustrates the transformation of a general target model into a personalized target model via model transformation according to the embodiments described herein.

[0012] Figure 5 The illustration includes graphs of two different contrast sensitivity functions (CSFs) illustrating the instance relationship between contrast sensitivity and spatial frequency, according to embodiments described herein.

[0013] Figure 6 This describes a modified adaptive bit rate (ABR) based on a media decoding and delivery system according to embodiments described herein.

[0014] Figure 7 According to the embodiments described herein Figure 6 Hardware block diagram of the playback system.

[0015] Figure 8 According to the embodiments described herein Figure 6 A block diagram of the media server.

[0016] Figure 9 This document describes an embodiment for delivering media to... Figure 6 The flowchart shows the method for the playback system.

[0017] Figures 10A-10C This illustrates a composite image of three different sizes of instances according to the embodiments described herein.

[0018] Figure 10D Explanation of composition Figures 10A-10C A low-pass filtered source image of a portion of a mixed image.

[0019] Figure 10E Explanation of composition Figures 10A-10C A high-pass filtered source image that is part of a mixed image.

[0020] Figure 11A and 11B This document describes a graph illustrating the instance-optimal ABR ladder estimation using a hybrid image as the test medium during a multi-step binary tree search, according to the embodiments described herein.

[0021] Figure 12A and 12BThis document describes the embodiments described herein, comparing the use of existing streaming methods and streaming media. Figure 9 The method of streaming media is used to determine the network instance bandwidth.

[0022] Figure 13A and 13B Description of embodiments according to the present document Figure 9 This method could allow more users / subscribers to stream media on a fixed-capacity network without adversely affecting another instance of QoE. Detailed Implementation

[0023] Visual media (e.g., images, videos, etc.) can be delivered to various types of playback systems / devices (e.g., televisions, computers, tablets, smartphones, etc.) for user viewing via one or more communication networks. In the visual media delivery chain, Adaptive Bit Rate (ABR) streaming allows for improved network resource management through adaptive selection of bit rate and resolution on the media based on network conditions, playback buffer status, shared network capacity, and other network-influenced factors. Besides ABR streaming, other media delivery methods (which may also include decoding or source decoding methods) can similarly be used to control one or more media parameters of the upstream video encoder / transcoder / transrater, such as bit rate, frame rate, and resolution.

[0024] However, media delivery methods such as ABR streaming have not yet considered additional factors to further improve network resource management, such as those related to the playback system / device, the user's viewing ability, and the environment in which the user is viewing the visual media. In practice, when performing content processing, decoding, delivery decoding, and post-processing, even though varying viewing conditions and changes in human visual performance can significantly impact the viewer's actual quality of experience (QoE), these factors are generally assumed to be ideal and consistent across different users / environments.

[0025] For example, short-distance viewing can make users more sensitive in distinguishing between low-resolution and high-resolution video content. Furthermore, different viewers can have different visual sensitivities due to factors including, but not limited to, refractive errors (even when wearing corrective lenses), vitreous fluid buildup, age-related changes in lens color absorption, cataracts, or macular degeneration. For example, a user's / viewer's contrast sensitivity can be attributed to decreased sensitivity due to increased refractive errors, disease, and / or age. Additionally, an individual's personal QoE can change with location and time, especially in mobile environments.

[0026] Detecting these visual sensitivity factors for each user / viewer can help estimate personalized QoE in real-world end-to-end systems and provide opportunities to improve QoE and further enhance media delivery efficiency. For example, a media delivery system can save bandwidth while maintaining personalized QoE for each user / viewer by transmitting a custom-filtered version of the same video to match the user / viewer's eye level or viewing distance from the television.

[0027] Several studies have proposed using multiple sensors to collect playback-side factors, aiming to select the optimal bit rate and resolution for media streaming, or to feed this information back to media preprocessing, encoding, and post-processing. However, using multiple sensors to collect playback-side information is insufficient and impractical for multiple playback systems (e.g., televisions). It is insufficient because such sensors do not measure the user's innate vision or sensitivity. It is impractical because the effort required to push and coordinate with television manufacturers across the consumer display industry to equip televisions with the necessary sensors and metadata protocols is too heavy. While this burden is lighter for mobile devices that already have multiple available sensors, user privacy remains an issue, especially when sensors collect information about the user's vision. Another problem with existing methods of using sensors to collect playback-side information is that different models / brands of televisions have their own proprietary upscaling and post-processing algorithms, and users can adjust various television settings, such as brightness, contrast, or motion smoothness, to suit their preferences.

[0028] To address the technical problems identified above, the methods, apparatus, and systems described herein incorporate novel mechanisms or protocols for sharing parameters related to playback device characteristics and personalized visual sensitivity factors with upstream devices configured to control the transmission of visual media to the playback device. The methods, apparatus, and systems described herein provide personalized adaptive media delivery based on playback-side information typically collected without the use of individual sensors. Furthermore, the collected playback-side information can indicate personalized QoE for different users and / or different viewing environments. The methods, apparatus, and systems described herein further improve network resource management / media delivery efficiency while maintaining personalized QoE for each user.

[0029] Figure 1 This describes an instance of Adaptive Bit Rate (ABR) based on media decoding and delivery system 100. System 100 includes a media server 105 that provides media to playback system 110 (i.e., playback device) via network 115. Although Figure 1 A single playback system 110 is shown, but the media server 105 can be configured to simultaneously stream the same or different media to an additional playback system 110.

[0030] Playback system 110 may include various types of playback systems, such as televisions, tablet computers, smartphones, computers, etc. In some embodiments, playback system 110 includes a buffer / decoder 120 and a playback presenter 125. The buffer / decoder 120 may receive media from server 105 via network 115. The buffer / decoder 120 may buffer the received media and decode the received media for output through playback presenter 125. (The following text is about...) Figure 7 In further detail, the buffer / decoder 120 may include an electronic processor (e.g., a microprocessor, microcontroller, or other suitable processing device) of the playback system 110. The playback presenter 125 may include output devices configured to display images and / or video. For example, as described below... Figure 7 In further detail, the playback presenter 125 includes a light-emitting diode (LED) display and / or a touchscreen display. The playback system 110 is located in environment 130. The user 135 is also located in environment 130 and can view the media output by the playback system 110.

[0031] like Figure 1 As illustrated, in some embodiments, media server 105 includes an ABR ladder 137 implemented by the electronic processor of media server 105. In some embodiments, media server 105 receives one or more ABR requests 140 from playback system 110 to adjust the bit rate / quality decisions for ABR streaming of media from media server 105 to playback system 110 via network 115. For example, playback system 110 may retrieve and / or utilize a stored generic target model 145 (e.g., device type, display resolution, geographic-based startup resolution, or a more comprehensive model, such as ITU-TP.1203, etc.) from memory and use the generic target model 145 to monitor / measure streaming session and playback-related performance information, such as network connectivity metrics, media player buffer status, codecs, bit rate, initial load latency and pause events, etc., and playback device 110 information, such as display resolution, screen size, playback device type, etc. In some embodiments, packet header information and partial / complete bitstream parsing may also be used to collect streaming session and playback-related performance information. Streaming and replay information is used to generate a general Quality of Experience (QoE) estimate for media streaming and media replay. This QoE estimate can be used by replay system 110 to influence ABR requests 140. For example, replay system 110 periodically determines each ABR request 140 for a media segment based on locally generated bandwidth estimates, buffer size, round-trip time, etc., with the aim of maintaining seamless replay. In other words, the general target model 145 can be configured to allow replay system 210 to control media streaming based on at least one of the resource availability of network 115 and replay system parameters.

[0032] In some cases, playback system 110 may simultaneously request two or more segments representing the same time period in the media but encoded at different bitrates in the ABR ladder 137. Such strategies can be inefficient and often result in playback system 110 requesting more data than is required for seamless playback. These strategies may also cause playback system 110 to request a resolution / bitrate / frame rate combination from the ABR ladder 137 to provide higher quality media that is imperceptible to user 135. In other words, existing ABR selection logic attempting to increase the delivered resolution / bitrate / frame rate beyond user 135's sensitivity threshold does not translate into an increase in user 135's QoE. Besides not translating into an increase in user 135's QoE, the requested resolution / bitrate / frame rate combination may utilize more network resources (e.g., more bandwidth) and / or may require user 135 to pay additional fees (e.g., where the service provider of media server 105 charges based on the amount of data provided to playback system 110).

[0033] The problem with the existing ABR selection logic described above arises because the generic target model 145 does not consider personalized QoE when determining the ABR request 140. For example, the generic target model 145 may not consider individualized viewing characteristics, such as lighting in environment 130, the viewing distance of user 135 (i.e., the distance between user 135 and the playback presenter 125), and the visual sensitivity and ability of user 135's eyes based on, for example, spatial frequency. In fact, existing ABR selection techniques assume that these characteristics are the same for every environment 130 and every user 135, when in reality these characteristics can vary greatly between different environments and / or users and affect the user 135's QoE when perceiving the media displayed by the playback system 110.

[0034] Although Figure 1 The corresponding explanation refers to ABR ladder 137 and ABR selection technology, but ABR-related media delivery is merely an example intended to illustrate a general media delivery method that can be implemented by upstream devices (e.g., media server 105 and network 115). ABR ladder 137 and ABR selection technology are also used throughout this application in relation to additional diagrams (see, for example...). Figure 2 , 3The media delivery methods described herein (and 11A-11B) are examples of media delivery methods. However, the features disclosed herein are applicable to any of several different media delivery methods (which may also include decoding methods or source decoding methods) that are not based on ABR ladder 137 or ABR selection technology. In other words, the features described herein can be used in conjunction with other media delivery methods besides ABR streaming, which can similarly be used to control any media parameters, such as bit rate, frame rate, resolution, etc., of the upstream video encoder / transcoder / rasterizer. Alternatively or additionally, the features described herein can be used in conjunction with decoding methods or source decoding methods used to decode / process media before streaming media. These decoding methods and source decoding methods are generally referred to herein as media delivery methods. In some embodiments, media parameters include those affecting the transmission from media server 105 to playback system 210 via network 115 (see [link to documentation]). Figure 2 , 3 The media delivery parameters (i.e., upstream media parameters) of playback system 210 and 6). In some embodiments, the media parameters additionally or alternatively include playback system parameters (i.e., downstream parameters), such as the brightness settings and / or contrast settings of playback system 210.

[0035] Figure 2 and 3 This section describes, according to some embodiments, a portion of the adaptive bit rate (ABR) of media decoding and delivery systems 200 and 300 configured to determine individualized / personalized viewing characteristics during a test measurement session. Figure 3 ) or not using (see Figure 2 In the case of a separate sensor collecting this additional information, personalized viewing characteristics should also be considered to address the issues raised above regarding the ABR selection logic (and / or other media delivery methods).

[0036] Figure 2 Includes playback system 210, which is related to Figure 1 The playback system 110 contains some similar components. For example, the playback system 210 includes a buffer / decoder 120 and a playback renderer 125. Although in Figure 2 Not shown in the image, but the playback system 210 can be accessed via... Figure 1 The network shown, similar to media server 105 and network 115, is communicatively coupled to the media server. However, instead of including... Figure 1As shown in the general target model 145, the playback system 210 can generate personalized (i.e., individualized) sensitivity distribution profiles (PSPs) 215 for a large number of different users 135 and / or environments 130. These personalized sensitivity distribution profiles 215 can be used to provide ABR requests (or other requests regarding other media delivery methods) to the playback system 210 for streaming media to the media server. Although the personalized sensitivity distribution profiles 215 are explained below as being generated by the playback system 210 (e.g., the electronic processor of the playback system 210), in some embodiments, as described further in detail herein, the generation and storage of the personalized sensitivity distribution profiles 215 may additionally or alternatively be performed by the electronic processor at the media server, by the electronic processor at a remote cloud computing cluster, or a combination thereof.

[0037] To generate a personalized sensitivity distribution curve 215, the playback system 210 conducts a test measurement session, during which user responses 220 to the test media 225 are collected from user 135. During the test measurement session, the sensitivity of user 135 is measured under typical viewing conditions and environments for a given user 135. For example, user 135 would be seated in a typical viewing position (e.g., sitting on a sofa in a family room, which represents typical viewing conditions in terms of viewing distance, viewing angle, ambient brightness, and playback system characteristics and settings). The playback system 210 then guides user 135 through the test measurement session to measure user 135's audiovisual sensitivity in environment 130 based on subsequent instructions provided by the playback system 210. During the session, user 135 may be asked to make one or more selections using remote controls based on a series of images and / or videos displayed by the playback system 210. For example, the playback system 210 may display multiple images and request the user to select the image that is clearest / sharpiest when presented to user 135. As another example, the playback system 210 may display an image with multiple interpretations depending on the user 135's visual ability and viewing distance, and request the user to select the interpretation that is most important / obvious to the user.

[0038] From a test and measurement session, playback system 210 and / or media server 105 can determine personalized viewing characteristics of user 135 and / or environment 130. For example, user responses 220 received during the test and measurement session can indicate system factors such as playback system characteristics, playback parameter settings, post-processing algorithms of playback system 210 (typically proprietary to the device manufacturer), etc. As another example, user responses 220 received during the test and measurement session can indicate environmental factors such as viewing distance, viewing angle, ambient brightness, ambient noise, user expectations, etc. In some embodiments, user expectations refer to conscious or subconscious psychological aspects of user 135 that can influence their perceived QoE. For example, user 135's expectation level for media associated with a paid subscription may be higher than its expectation level for other media, such as free video-on-demand services. As another example of personalized viewing characteristics of user 135 and / or environment 130, user responses 220 received during the test and measurement session can indicate human factors such as user 135's sensory acuity / visual sensitivity and ability, age, gender, etc. Figure 2 As indicated above, in some embodiments, the playback system 210 may be as described above regarding... Figure 1 The playback system 210 provides additional playback system information 230 (e.g., device type, display resolution, etc.) in a manner similar to that interpreted by system 100. The playback system 210 can use the aggregated information 235 (including the playback system information 230) and the user response 220 to the test media 225 to determine a personalized sensitivity distribution curve 215 for a particular user 135 and / or environment 130.

[0039] like Figure 2 As shown, the personalized sensitivity distribution curve 215 may include user identification, other personal information (i.e., the individualized viewing characteristics of user 135 determined by user response 220), replay system information 230, and one or more of the following: geographic location, weather information, date, and time when the test media 225 is displayed to user 135 during one or more measurement sessions. Figure 2 In the example of personalized sensitivity distribution curves 215 shown, each personalized sensitivity distribution curve 215 may be associated with a user and may include multiple sub-distribution curves for different environments in which the user 135 has participated in a test measurement session (e.g., different rooms in the user's home, different times of day, different weather information (e.g., sunny and cloudy)). In other embodiments, each personalized sensitivity distribution curve 215 may be associated with a user and a specific environment, such that each user may have multiple personalized sensitivity distribution curves 215 corresponding to different environments in which the user 135 has participated in a test measurement session.

[0040] Although the personalized viewing characteristics described above are not explicitly collected by a single sensor (e.g., a sensor measuring the distance between the playback device 210 and the user 135), the system 200 is able to determine / estimate one or more of these characteristics based on the user response 220 to the test medium 225 during a test measurement session. Therefore, in some embodiments, personalized viewing characteristic information can be collected from the user 135 and the environment 130 without using a separate explicit sensor. In other embodiments, a separate explicit sensor may be used to provide additional information (see, for example...). Figure 3 ).

[0041] Figure 3 Explanation based on Figure 2 The playback system 200 is similar to the adaptive bit rate (ABR) portion of the media decoding and delivery system 300. However, system 300 additionally includes one or more environmental sensors 305. For example, playback system 210 may include or may communicatively couple to a luminance / illuminance sensor (e.g., an integrated sensor, a smart home sensor communicatively coupled to playback system 210, etc.) that measures ambient light in environment 130. Environmental sensor 305 may include other smart home sensors located in environment 130 that indicate, for example, whether lights are on / off, whether curtains covering windows are open / closed, etc. Environmental sensor 305 may be configured to determine the time and geographic location of playback system 210 during the day, for example, for the purpose of determining the number of hours of daylight. Environmental sensing data 310 from environmental sensor 305 may be included in summarized information 235 and may be included in, for example,... Figure 3 The personalized sensitivity distribution curve 215 indicated in the figure. In some embodiments, the environmental sensing data 310 included in the summarized information 235 includes user-provided information, such as the level of social activity and / or viewing distance in the environment 130.

[0042] During the test measurement session, the collection of user responses 220 to test media 225 implicitly takes into account a variety of personalized viewing characteristics: it would be difficult, impractical, and / or unrealistic to explicitly collect these personalized viewing characteristics using sensors. Furthermore, in some cases, data explicitly collected using sensors may not allow for an accurate determination of the QoE of user 135. Therefore, systems 200 and 300 offer several potential advantages and benefits.

[0043] One example of the benefit involves user variability. Two different users may have the same environmental characteristics (e.g., viewing distance, illuminance, screen size, etc.). However, these two different users may have significantly different viewing abilities due to differences in factors such as refractive errors, age, and / or eye diseases. Therefore, determining the ABR request (or request regarding other media delivery methods) solely based on physical context / environmental characteristics can result in different levels of QoE for these different users. In some embodiments, the personalized sensitivity distribution curve 215 of systems 200, 300 takes into account these personalized viewing ability differences when determining the ABR request in order to prevent and / or mitigate a reduction in personalized QoE for each user.

[0044] Another benefit is the ease with which systems 200 and 300 can be implemented. In some embodiments, no additional sensors need to be added to playback system 210. Along a similar line, user privacy is protected because, in some embodiments, specific details of the user's environment may not be explicitly measured and recorded. In fact, in some embodiments, a holistic assessment of the user response 220 (implicitly including more detailed factors such as the user's viewing distance and viewing ability) is used to generate an ABR request to media server 105. In other words, the ABR request is based on composite measurements that are not based on a set of non-dependent attribute measurements from individual sensors. In effect, the ABR request in systems 200 and 300 is a comprehensive implicit measurement of the user's QoE, reflecting the combined effect of multiple factors that are difficult or even impossible to collect explicitly.

[0045] Another benefit involves personalized content enhancement for user 135. In addition to achieving a more comprehensive and accurate estimate of the end-user's QoE, systems 200 and 300 allow for enhancement of the media / content played back to user 135. Specifically, playback system 210 can enhance portions of video frames that are too small or too low relative to the user's perception (i.e., spatial frequency or contrast exceeding the user-measured contrast sensitivity function (CSF)). Examples of this enhancement may include cropping and enlarging frames and / or applying local contrast adjustments to ensure that prominent parts of the scene are visible to user 135 (i.e., within the user-measured CSF). This type of enhancement can improve the user's viewing experience by helping user 135 follow the media / content and maintain focus on what is being viewed.

[0046] Return to reference Figure 1The generic target model 145 that generates the ABR request 140 may contain any of a number of models that estimate the QoE. For example, a generic model is used to construct the ABR ladder 137 (or another media delivery method) typically by analyzing audio / video signals and optimizing decoding efficiency, containing multiple versions of the bit rate / quality of media segments from the reference source media. Another model is used to make appropriate bit rate / quality decisions for selecting ABR streaming based on network conditions and playback device loading. However, instance-type models do not consider the user's personalized viewing characteristics. The personalized sensitivity distribution curve 215 can be applied to any type of generic model 145 to construct a personalized target model (POM) 405 from the generic target model 145 (see [link to relevant documentation]). Figure 4 and 6 ).

[0047] Figure 4 This diagram illustrates the transformation of a general target model 145 into a personalized target model 405 via model transformation 410. The transformation can be performed based on one or more desired objectives achieved in media delivery control and management. For example, if the objective is to save media streaming bandwidth by selecting the minimum bit rate in the ABR ladder 137 without degrading the personalized QoE of each user 135, then systems 200 and 300 can determine the minimum perceptible difference (JND) of the image / video in the resolution-bit rate grid space. Systems 200 and 300 can then transform the general target model 145 into a personalized target model 405 for each user 135 / playback system 210 based on the personalized JND of user 135, such as based on the personalized sensitivity distribution curve 215 of user 135 in the environment 130 where user 135 is using playback system 210.

[0048] As another example, for more complex streaming management encompassing real-time or non-real-time preprocessing, encoding, transcoding, or transrate in the loop, systems 200 and 300 can estimate personalized psychological functions (e.g., spatial contrast sensitivity, temporal contrast sensitivity, and spatial-temporal contrast sensitivity to colorless and counter-color stimuli) to construct a personalized target model 405. For instance, Figure 5 The illustration includes graphs showing the instance relationship between contrast sensitivity 505 and spatial frequency 510. The solid line curve illustrates the ideal CSF 515. The dashed line curve illustrates the instance user CSF 520 for user 135 determined based on user response 220 to test medium 225 during a given test measurement session. Figure 5As shown, compared to the ideal CSF 515, the translated and scaled user CSF 520 is due to factors such as, for example, a longer viewing distance in environment 130 than in the ideal viewing environment, a lower brightness of the television screen than in the ideal viewing environment, and the user's hyperopia / myopia. Also as Figure 5 As shown, compared to the ideal CSF 515, user 135 has a smaller perceptible range of difference in contrast sensitivity 505 relative to spatial frequency 510. In other words, due to environmental and personal conditions as mentioned above, user 135's visual ability is not as sensitive as that of an ideal user. For example, when spatial frequency 510 increases above the second value 530, user 135 may not be able to distinguish contrast differences.

[0049] Therefore, providing higher quality media streaming to allow an increased QoE behavior that provides an ideal user experience with ideal contrast sensitivity in an ideal environment will not actually cause QoE with Figure 5 The example user 135 of user CSF 520 shown in the figure experiences an increased QoE. Therefore, if providing this higher quality streaming requires a cost to systems 200 and 300 (e.g., using more bandwidth due to the use of higher bit rate streaming media), then this cost experienced by systems 200 and 300 is essentially wasted, because the cost does not result in user 135 experiencing an improved QoE while watching the streaming media.

[0050] To assist systems 200 and 300 in controlling the streaming of media from media server 105, user CSF 520 is included in personalized sensitivity distribution curve 215 to transform the general target model 145 into a personalized target model 405 (see...). Figure 4One example of data is the general target model 145. For instance, the general target model 145 may use data corresponding to an ideal CSF 515 or data corresponding to another general CSF that is not personalized for the user's viewing environment and individual visual abilities. On the other hand, the personalized target model 405 may use a user CSF 520 that has been scaled and translated to be personalized for the user's viewing environment and individual visual abilities. The user CSF 520 may be used in combination with other factors and / or algorithms included in the general target model 145 to form the personalized target model 405. As indicated by the above example of streaming media of higher quality than that perceived by user 135, the personalized target model 405 may be utilized during video encoding, transcoding, and / or transrate to improve decoding efficiency, thereby improving network efficiency (e.g., by reducing bandwidth) without affecting the personalized QoE of user 135. For example, higher quality media may be streamed to a more sensitive user 135 who can perceive higher quality media, while lower quality media may be streamed to a less sensitive user 135 who cannot perceive the quality difference between lower and higher quality media.

[0051] Although Figure 5 The example chart shown refers to the contrast sensitivity of user 135, but the general target model 145 may include values ​​and / or functions related to other types of viewing characteristics that can be personalized based on the user response 220 to the test media 225 (e.g., temporal degradation, quantization degradation, etc. related to video frame rate). For example, the general media parameter values ​​and / or functions characterizing the general temporal degradation model may be replaced or retrained to produce personalized values ​​and / or functions in the personalized target model 405. Alternatively, the general algorithm used to determine the media parameters (i.e., streaming parameters) may have coefficients adjusted to produce the personalized target model (POM) 405.

[0052] Figure 6 This describes a modified adaptive bit rate (ABR) based on media decoding and delivery system 600. System 600 is similar to... Figure 1 System 100 includes a personalized target model (POM) 405 for generating an ABR request 140 (or a request regarding another media delivery method) instead of a generic target model 145. In some embodiments, system 600 includes Figure 2 and 3 At least one of the components of systems 200 and 300 shown. For example, such as Figure 6As shown, the personalized target model 405 takes into account replay system information 230, environmental sensing information 310, and personalized viewing characteristics information from PSP 215 as described above, such as those determined based on user response 220 to test media 225.

[0053] Additional data sources available for POM 405 to generate ABR request 140 (or a request regarding another media delivery method) include, but are not limited to, real-time media player state information, including buffer size, playback status, player performance characteristics, etc. Another data source available for POM 405 includes real-time network performance estimates, such as throughput measured by playback system 210, throughput measured from sensors located within network 115, congestion notifications, latency, packet loss rate, etc. Another data source available to POM 405 includes content metadata, including bit rate, resolution, frame rate bit depth per sample, chroma sampling, source decoding method (including ratings and profiles), color space, supplementary enhancement messages (SEI), composite playlists, group of pictures (GOP) size, instantaneous decode refresh (IDR) frames, maximum average optical level per frame (MaxFALL), maximum content optical level (MaxCLL), electro-optical transfer function (EOTF), language, type of service, scene description (including boundary information), number of audio channels, audio sampling rate, audio sample bit depth, audio type of service, digital signature method, SCTE 35 messages, caption data, program loudness, regulatory information, rating information, etc. In some embodiments, the additional data source described herein may be referred to as media parameters.

[0054] Another data source available to POM 405 includes network operator policy parameters, including the maximum permissible bit rate, spatial resolution, frame rate, etc., for each downstream and / or upstream channel or channel equivalent. This example data source allows for network-wide optimization and cross-session optimization. Another data source available to POM 405 includes playback environment sensor information 310 as explained above (e.g., ambient illuminance level, ambient audio noise level, number of people watching the streaming content, distance from each viewer's screen, etc.). Another data source available to POM 405 includes auxiliary mobile device information, such as distance from the primary playback system 210, mobile device sensor information, etc. Another data source available to POM 405 includes real-time user / viewer preferences that can be input by user 135 and stored in the memory of one of the devices included in system 600.

[0055] Figure 7This is a hardware block diagram of a playback system 210 (i.e., a playback device) according to one example embodiment. As mentioned above, the playback system 210 may include various different types of playback systems, such as televisions, tablet computers, smartphones, computers, etc. In the illustrated embodiment, the playback system 210 includes a first electronic processor 705 (e.g., a microprocessor or other electronic device). The first electronic processor 705 includes input and output interfaces (not shown) and is electrically coupled to a first memory 710, a first network interface 715, an optional microphone 720, a speaker 725, and a display 730. In some embodiments, the playback system 210 differs from... Figure 7 The configuration described herein includes fewer or additional components. For example, playback system 210 may not include microphone 720. As another example, playback system 210 may include one or more additional input devices, such as a computer mouse and / or keyboard, for receiving input from a user of playback system 210. As yet another example, playback system 210 may include environmental sensors, such as an ambient light sensor and / or a orientation tracking device (e.g., a Global Positioning System (GPS) receiver). In some embodiments, playback system 210 performs functions other than those described below.

[0056] The first memory 710 may include read-only memory (ROM), random access memory (RAM), other non-transitory computer-readable media, or combinations thereof. The first electronic processor 705 is configured to receive instructions and data from the first memory 710 and, in particular, execute the instructions. Specifically, the first electronic processor 705 executes instructions stored in the first memory 710 to perform the methods described herein.

[0057] The first network interface 715 sends data to and receives data from the media server 105 via the network 115. In some embodiments, the first network interface 715 includes one or more transceivers that wirelessly communicate with the media server 105 and / or the network 115. Alternatively or additionally, the first network interface 715 may include a connector or port for receiving wired connections to the media server 105 and / or the network 115 (e.g., Ethernet cable). The first electronic processor 705 may receive one or more data streams (e.g., video streams, audio streams, image streams, etc.) via the network 115 through the first network interface 715. The first electronic processor 705 may output one or more data streams received from the media server 105 via the first network interface 715 through a speaker 725, a display 730, or a combination thereof. Additionally, the first electronic processor 705 may transmit data generated by the playback system 210 back to the media server 105 via the network 115 through the first network interface 715. For example, the first electronic processor 705 may determine and send the ABR request 140 mentioned earlier herein to the media server 105. The media server 105 may then transmit one or more media streams to the playback system 210 in accordance with the ABR request 140 from the playback system 210.

[0058] Display 730 is configured to display images, video, text, and / or data to user 135. Display 730 may be a liquid crystal display (LCD) screen or an organic light-emitting diode (OLED) display screen. In some embodiments, a touch-sensitive input interface may also be incorporated into display 730, allowing user 135 to interact with content provided on display 730. In some embodiments, display 730 includes a projector or a future display technology. In some embodiments, speaker 725 and display 730 are referred to as output devices for presenting media streams and other information to user 135 of playback system 210. In some embodiments, microphone 720, computer mouse and / or keyboard, or touch-sensitive display are referred to as input devices for receiving input from user 135 of playback system 210.

[0059] Figure 8 This is a block diagram of a media server 105 according to one example embodiment. In the illustrated example, the media server 105 includes a second electronic processor 805 electrically connected to a second memory 810 and a second network interface 815. These components are similar to those described above regarding... Figure 7 The playback system 210 is explained as having similarly named components and operating in a manner similar to that described above. In some embodiments, the second network interface 815 sends data to and receives data from the playback system 210 via network 115. In some embodiments, the media server 105 operates in a manner different from... Figure 8The configuration described herein includes fewer or additional components. For example, media server 105 may additionally include a display such as a touchscreen to allow backend users to reprogram the settings or rules of media server 105. In some embodiments, media server 105 performs functions other than those described below.

[0060] Although Figure 7 and 8 Separate block diagrams of playback system 210 and media server 105 are shown, but in some embodiments, media server 105, one or more playback systems 210, a remote cloud computing cluster communicating via or forming part of network 115, or a combination thereof are referred to as electronic computing devices performing the functions described herein. For example, an electronic computing device may be a single electronic processor (e.g., a second electronic processor 805 of media server 105) or multiple electronic processors located within media server 105. In other embodiments, an electronic computing device includes multiple electronic processors distributed across different devices. For example, an electronic computing device may be implemented on one or more of the following: a first electronic processor 705 of playback system 210, a second electronic processor 805 of media server 105, and one or more electronic processors in one or more other devices located at a remote location or communicating via or forming part of remote cloud computing cluster 115. In some embodiments, the remote cloud computing cluster includes an access network supporting software-defined networking (SDN) / network function virtualization (NFV).

[0061] In some embodiments, the apparatus implementing POM 405 may determine the objectives and functions of POM 405. For example, implementation of POM 405 within playback system 210 allows for distributed operation in the absence of network operator or other control signals. On the other hand, implementation of POM 405 within media server 105 and / or network 115 (e.g., as a network virtualization function (NVF) located on a software-defined networking (SDN) node) simplifies network-wide QoE optimization and the deployment of other network operator strategies (e.g., optimization of network services for desired subscriber QoE, edge / access network capacity targets, or a combination of both).

[0062] Figure 7 One or more implementations and / or components of the hardware components of the playback system 210 shown herein. Figure 2 , 3The functional components of the playback system 210 shown in Figure 6. For example, a first electronic processor 705 (or a plurality of first electronic processors 705 of the playback system 210) may act as one or more of the buffer / decoder 120 and the playback presenter 125. The first electronic processor 705 may also determine the personalized sensitivity distribution curve (PSP) 215, the personalized target model (POM) 405, and the ABR request 140.

[0063] In some embodiments, one or more personalized sensitivity distribution curves (PSPs) 215 for one or more users and environments are stored in a first memory 710 of the playback system 210. The first memory 710 may store additional information, such as general playback system information 230 for the playback system 210 (e.g., screen size, product identification number, etc.). In some embodiments, one or more personalized sensitivity distribution curves (PSPs) 215 for one or more users and environments are additionally or alternatively stored in a second memory 810 of the media server 105 and / or in the memory of a remote cloud computing cluster communicating with or forming part of the network 115. In some embodiments, cloud storage of a user's PSP 215 enables secure linking to the user's wired / wireless Internet service provider (ISP) or network delivery media account (e.g., cable television). As described in more detail herein, such linking can be used by network operators to generate more efficient media delivery across a portion of their subscriber base using individual PSPs 215, where each PSP 215 is associated with an account.

[0064] Figure 9 A flowchart illustrating a method 900 for delivering media to a playback system 210 according to an example embodiment is provided. Method 900 is described as being performed by an electronic computing device including one or more electronic processors previously described herein. While some actions are interpreted as being performed by the electronic processor of a particular device, in other embodiments, the electronic processor of other devices may perform the same actions. Although, by way of example, in Figure 9 The instructions specify a particular order of processing steps, message reception, and / or message transmission, but the timing and order of such steps, reception, and transmission may vary as appropriate and will not negate the purpose and advantages of the examples detailed throughout the remainder of this disclosure.

[0065] At block 905, media delivery method 900 is initiated. In some embodiments, media delivery method 900 is initiated by a first electronic processor 705 of playback system 210 in response to user 135 activating playback system 210 and / or requesting playback system 210 to output a data stream.

[0066] In response to the initiation media delivery method 900, at block 910, one or more electronic processors of the electronic computing device retrieve a stored personalized sensitivity distribution profile (PSP) 215 relating to at least one of the user 135, the playback system 210, and the environment 130 in which the playback system 210 is located. For example, the stored PSP 215 may have been generated based on a previous test measurement session of the playback system 210. In some embodiments, the stored PSP 215 may be available to the electronic computing device for delivering output media to the playback system 210 without performing a new test measurement session. For example, when the playback system 210 is being used in an environment 130 that has already undergone a test measurement session with the same user 135, the electronic computing device may use the stored PSP 215 corresponding to the environment 130 and the user 135. In some embodiments, the electronic computing device determines that the current environment and the user have already undergone a test measurement session by comparing playback system information, environment information, and / or user information (e.g., user login information received by the playback system 210) with the stored information in the PSP 215. For example, the electronic computing device can determine the identification number of the playback system 210, one or more characteristics of the environment 130 (e.g., time of day, amount of ambient light, orientation of the playback system 210, etc.), and the identity of the user 135. If this identification information matches a PSP 215 already stored in one of the memory locations of the electronic computing device, then the electronic computing device can control the provision of output media to the playback system 210 and the display of the output media by the playback system 210 according to the previously stored corresponding PSP 215 without proceeding to the block 915 for executing a new test and measurement session.

[0067] On the other hand, method 900 may continue to block 915, wherein the playback system 210 is controlled to execute a new test measurement session. Here, the new test measurement session may be a full-length session or a reduced-length session based on prior knowledge of one or more stored PSPs 215. For example, if one or more of the identification information of the type described above does not match the stored PSPs 215, then the electronic computing device may execute a new test measurement session and generate a new PSP 215, as further explained in detail below. In some embodiments, user 135 (e.g., via user input on the input device of playback system 210) initiates a new test measurement session. In some embodiments, the electronic computing device may determine that at least one characteristic of the previously stored PSP 215 has changed (e.g., power interruption, Internet Protocol (IP) address change, WiFi signal strength change, recent detection of a peripheral device coupled to playback system 210, detected change in ambient light, detected change in orientation of playback system 210, etc.). In response, the electronic computing device may instruct the playback system 210 to suggest that user 135 participate in a new test measurement session. For example, the playback system 210 may determine that user 135 is currently watching the playback system 210 at night rather than during the day (e.g., based on time of day measurements, data received from environmental sensor 305, etc.). As another example, the media server 105 may determine that a new playback system 210 not associated with any previously stored PSP 215 has been connected to the network 115. In response, the media server 105 may send a request to the playback system 210 to suggest that user 135 participate in a test measurement session to generate a PSP 215.

[0068] The method of conducting a test measurement session may involve integrating it as a third-party application running on the playback system 210 or as a cloud service for hosting test media 225 and / or PSP 215 into a set-top box (STB), digital media adapter (DMA), mobile device, or other playback system 210. As previously mentioned herein, the results of the test measurement session may be stored locally on the playback system 210 and / or remotely as part of a cloud service to achieve cross-platform and cross-service compatibility.

[0069] At block 915, the electronic computing device outputs test media 225 for viewing by user 135. Test media 225 may be generated by a first electronic processor 705 of playback system 210 or may be received by playback system 210 after being generated by media server 105. In some embodiments, test media 225 is generated to measure user sensitivity / quality of experience (QoE). For example, at block 920, the electronic computing device receives user input (i.e., user response 220) from user 135. The user input relates to user 135's perception of test media 225 and indicates a first personalized QoE for user 135 regarding test media 225.

[0070] In some embodiments, the electronic computing device determines the user sensitivity / QoE of user 135 by generating a target acuity measurement including a Snellen chart or an open-loop pattern using test medium 225. In some embodiments, the electronic computing device additionally or alternatively determines the user sensitivity / QoE of user 135 by generating a contrast sensitivity function (CSF) measurement using sinusoidal gratings with different orientations (e.g., see...). Figure 5 In some embodiments, the CSF measurement may include a fast CSF method using test media 225, including pass-through filtered Sloan letters. In some embodiments, the electronic computing device additionally or alternatively determines the user sensitivity / QoE of user 135 by displaying test media 225 in the form of an interactive game that user 135 will play. In some embodiments, the electronic computing device additionally or alternatively determines the user sensitivity / QoE of user 135 by obtaining a user sensitivity measurement based on a set of images or video materials displayed as test media 225.

[0071] In some embodiments, the electronic computing device may display test media 225 in the form of a mixed image. In some embodiments, the mixed image is a still image that tends to have disparate interpretations depending on the user's viewing ability and environmental factors. As an example, human viewers lose their ability to see subtle details in an image as viewing distance increases, making it impossible to distinguish between high-resolution and low-resolution video. In some embodiments, the mixed image is a still image that generates two or more disparate interpretations for a human user that vary with spatial frequency range and / or viewing distance. Based on the user response 220 to the displayed mixed image, the electronic computing device can estimate the dominant and non-dominant spatial frequency ranges of the user 135 in the media viewing environment 130 without using explicit sensors.

[0072] To form a blended image, two different source images can be processed in different ways to produce a specific spatial frequency range that is dominant for each image. For example, the first source image can be low-pass filtered, and the second source image can be high-pass filtered. The low-pass filtered source image and the high-pass filtered source image can then be combined (i.e., overlapped) to form a blended image. Because the sensitive region of a given image in spatial frequencies shifts from lower frequencies to higher frequencies as the user's viewing distance decreases, a human user is more likely to perceive the high-pass filtered source image at a shorter viewing distance than at a longer viewing distance. Conversely, a human user is more likely to perceive the low-pass filtered source image at a longer viewing distance than at a shorter viewing distance. In other words, the user's perception of the low-pass filtered source image or the high-pass filtered source image is dominant depending on one or more of the user's viewing characteristics.

[0073] Figures 10A-10C This describes a blend of three instances of different sizes in an image of size 1000. Figure 10A The image with the largest size shown is 1000, and Figure 10C The smallest image size 1000 is shown. In the example shown, the first source image containing the face of dog 1005 is low-pass filtered (see [link]). Figure 10D The low-pass filtered source image 1050 and the second source image containing the high-pass filtered face of the cat 1010 (see [image source image]). Figure 10E The source image (1060) is combined after being filtered by a high-pass filter. For example, by... Figures 10A-10C As indicated, due to the high-pass filtering of the second source image, human users are in a larger... Figure 10A medium to small Figure 10B and 10C The cat's face is more easily perceived in the image. In other words, this is attributed to the high-pass filtering (see...). Figure 10E The mixed image 1000 contains only the subtle details of the second source image, including the cat 1010's face, which is more easily perceived when image 1000 is larger (i.e., at a close / short viewing distance). Conversely, human users... Figure 10C China and Belgium in Figure 10A and 10B The dog's face is more easily perceived in the image. In other words, this is attributed to the low-pass filter (see...). Figure 10D The mixed image 1000 contains only coarse details of the first source image, including the face of the dog 1005, which is more easily perceived when image 1000 is smaller (i.e., at a longer viewing distance). In order to... Figures 10A-10C Visual explanations are used to assist viewers. Figures 10A-10E It has a nose 1015 for the cat 1010 marked in each figure and a nose 1020 for the dog 1005 marked in each figure.

[0074] While the generation of a blended image is explained above as involving low-pass and high-pass filtering of different source images, in some embodiments, different band-pass filters are used additionally or alternatively to generate the blended image. In some embodiments, changes in the size of the source images cause a proportional magnification or reduction in the spatial frequency domain. Therefore, combining filtering with changes in the size of the source images is another way to generate a blended image.

[0075] By displaying a series / multiple mixed images as test media 225 at frame 915 during a test measurement session, the electronic computing device may be able to determine the viewing characteristics of user 135 and environmental factors related to playback system 210. In some embodiments, the electronic computing device may change the size of the mixed images displayed by playback system 210. For example, the electronic computing device may change the size of the mixed images until user response 220 indicates that the user's perception of the mixed images has changed from a first perception of a first source image to a second perception of a second source image. Based on the size of the mixed images displayed at the time user response 220 is received and based on the resolution and screen size of playback system 210, the electronic computing device may be able to determine the user 135's estimated viewing distance, the user 135's estimated CSF, etc.

[0076] In some embodiments, the electronic computing device may adaptively change the cutoff frequencies of the low-pass and high-pass filters (or band-pass filters) of each source image being used to form the blended image, either randomly or based on previous user responses 220 received during a test-measurement session. For example, the electronic computing device may receive first user input relating to a user's first perception of the first blended image. In response, the electronic computing device may use filters to generate a second blended image, wherein the cutoff frequency of at least one filter is based on the first user input relating to the first perception of the first blended image (e.g., see...). Figure 11A and 11B The electronic computing device can then control the playback system 210 to output a second mixed image for the user 135 to view.

[0077] In some embodiments, the electronic computing device may determine the cutoff frequency of the spatial filter (and / or another characteristic used to generate the blended image, such as the size of the blended image being displayed) based on playback system parameters and / or media parameters supported by the media server and network 115. For example, the electronic computing device may determine the cutoff frequency of the spatial filter in conjunction with the available video resolution in the ABR ladder 137 of the media server 105 (or in conjunction with the available values ​​of other media parameters based on another media delivery method being utilized by the media server 105). As another example, the electronic computing device may determine the cutoff frequency based on the available bit rate of the media server 105 / network 115, the available frame rate of the media server 105 / network 115, the device type of the playback system 210, the screen size of the display 730 of the playback system 210, and / or other parameters / attributes previously mentioned herein.

[0078] In some embodiments, the electronic computing device determines a first value of the media parameters supported by the media server 105 and the network 115. The electronic computing device may also determine a second value of the media parameters supported by the media server 105 and the network 115. The electronic computing device may then perform at least one of the following generation and selection operations: generating and selecting a blended image based on the first and second values ​​of the media parameters, such that the blended image contains a first interpretation corresponding to the first value of the media parameters and a second interpretation corresponding to the second value of the media parameters (e.g., see...). Figure 11A and 11B The electronic computing device can then control the playback system 210 to display the mixed image on the display 730.

[0079] In some embodiments, as previously described herein, the electronic computing device displays an additional blended image based on a user response 220 to a previously displayed blended image. For example, the electronic computing device may perform at least one of the following generation and selection operations: generating and selecting a second blended image based on a first value and a third value of media parameters (determined to be supported by media server 105 and network 115), such that the second blended image includes a third interpretation corresponding to the third value of the media parameters and a fourth interpretation corresponding to the first value of the media parameters.

[0080] In some embodiments, as previously described herein, the electronic computing device generates the blended image described in the above examples by overlaying source images. In other embodiments, the electronic computing device can retrieve previously generated and stored blended images by means of characteristics corresponding to values ​​of media parameters determined to be supported by media server 105 and network 115.

[0081] During a test and measurement session, the electronic computing device may receive user input from user 135 via the input device of the playback device. The user input indicates a first interpretation of the mixed image perceived by user 135 when the mixed image is displayed on display 730. Based on the user input, the electronic computing device may determine that user 135 is more sensitive to a first value of a media parameter (e.g., a first spatial frequency range, viewing distance, resolution, etc.) than to a second value of a media parameter (e.g., a second spatial frequency range, viewing distance, resolution, etc.). In some embodiments, based on the determination that user 135 is more sensitive to the first value of the media parameter, the electronic computing device generates a personalized sensitivity distribution curve 215 of the viewing characteristics of user 135. The personalized sensitivity distribution curve 215 may include the first value of the media parameter. In some embodiments, as previously explained herein, media server 105 may provide output media to user 135's playback system via network 115 according to the personalized sensitivity distribution curve 215.

[0082] Continuing with the example above, the electronic computing device may determine at least one of the following based on user input: a spatial frequency subset of the blended image to which the user 135 is most sensitive (i.e., the contrast of the blended image) and a size of the blended image to which the user 135 is most sensitive. In some embodiments, the viewing characteristics of the personalized sensitivity distribution curve 215 generated by the electronic computing device include at least one of the following: a spatial frequency subset of the blended image to which the user is most sensitive and a size of the blended image to which the user is most sensitive.

[0083] As indicated by the examples above, the use of mixed images generated or selected based on media parameters and / or playback system parameters (i.e., media-centric parameters) during a test measurement session allows the computing device to determine, for example, how different media-centric parameters affect a user's personalized QoE. For instance, the computing device can determine how different video resolutions of the ABR ladder 137 (or different values ​​of media parameters for another media delivery method) affect a user's personalized QoE. In other words, based on the user response 220 to the test media 225, the computing device estimates the dominant and invisible spatial frequency ranges affecting the user's perception. This perception information can be used to improve the efficiency of media decoding and delivery as illustrated herein. For example, the lowest video resolution in the ABR ladder 137 can be identified as the video resolution at which the user 135 begins to experience quality degradation compared to full-resolution video.

[0084] Figure 11A and 11BThis diagram illustrates the optimal ABR ladder estimation using the hybrid image as an instance of test media 225 during a multi-step binary tree search during a test measurement session. In some embodiments, the electronic computing device obtains the available video resolutions (e.g., 360p, 540p, 720p, and 1080p) of the media streaming from the media server 105 and network 115 from a manifest file. Based on the video resolution, the electronic computing device determines the cutoff frequencies of the low-pass and high-pass filters to be applied to the source images A and B to form the hybrid image. Figure 11A and 11B The vertical dashed lines drawn along the rows shown represent the upper frequency limits for the corresponding video resolution in ABR ladder 137. For example, the spectral content of 540p video may only reach the second vertical dashed line on the left. Figure 11A and 11B The top charts 1105 and 1155 in each of these figures show the speculative contrast sensitivity function (CSF) as a user 135 varies with the spatial frequency [per pixel cycle] at a specific viewing distance. Figure 11A As shown, the peak sensitivity of user 135 is between 540p and 720p.

[0085] Figure 11A Intermediate chart 1115 illustrates the sensitivity of the first mixed image generated from filtered source images A and B. As indicated by chart 1115, the electronic computing device can initially set the spatial frequency at which changes in human perception of the source images A and B of the first mixed image can occur at 540p. Based on user response 220 to the first mixed image (i.e., test medium 225), the electronic computing device determines which of source images A or B is perceptually more dominant for user 135. When user response 220 indicates that source image B is more dominant, the electronic computing device can generate a sensitivity of the first mixed image generated from the first mixed image (i.e., test medium 225). Figure 11A The bottom chart 1120 represents the second mixed image. As indicated by the bottom chart 1120, the electronic computing device can select the source image B based on the display of the first mixed image represented by the intermediate chart 1115, with the spatial frequency at which a change in human perception of the second source images A and B occurs in the second mixed image set to 720p. By displaying multiple mixed images that are dynamically / adaptively adjusted based on the user response 220, the electronic computing device is configured to narrow the range of dominant frequencies perceptible to the user 135. The electronic computing device can be configured to determine, based on the user response 220 received throughout the test measurement session, the dominant frequency range that the user 135 can perceive. Figure 11A The estimated CSF 1125 is shown in the top chart 1105. Although in Figure 11AThe diagram shows only two repetitions of the graph representing the displayed mixed image, but in some embodiments, the electronic computing device displays additional mixed images (i.e., test media 225) and receives additional corresponding user responses 220 during the test measurement session.

[0086] Figure 11B Is with Figure 11A Similar but corresponding to different users 135, environments 130, and / or playback systems 210 (e.g., the same user 135 and playback device 210 but different). Figure 11A Examples of instances (viewed from a greater distance). Figures 1155, 1160, and 1165 roughly correspond to... Figure 11A Corresponding graphs 1105, 1115, and 1120 show values ​​adjusted according to different viewing scenarios as described above. As indicated by graph 1160, the electronic computing device can initially set the spatial frequency at which changes in human perception of the source images A and B of the first mixed image can occur at 540p. Based on the user response 220 to the first mixed image (i.e., test medium 225), the electronic computing device determines which of the source images A or B is more perceptually dominant for the user 135. Unlike Figure 11A In the example shown, when the user responds to instruction 220 that source image A is more dominant, the electronic computing device can generate a... Figure 11B The bottom chart 1165 represents the second mixed image. As indicated by the bottom chart 1165, the electronic computing device can select source image A based on the display of the first mixed image represented by the intermediate chart 1160, and the spatial frequency at which a change in human perception of the second source images A and B of the second mixed image occurs is set to 360p. As mentioned above regarding Figure 11A As explained, the electronic computing device can continuously display mixed images and receive user responses 220 to determine, for example... Figure 11B The estimated CSF is 1170, as shown in the top chart 1155.

[0087] like Figure 11A and 11B As shown, CSFs 1125 and 1170 differ from each other due to differences in one or more of the user 135, environment 130, and / or playback system 210. For example, Figure 11B The peak sensitivity of CSF 1170 is 1175 and Figure 11A The CSF 1125's sensitivity peak of 1110 is at a lower resolution. As another example, Figure 11B The overall range of CSF 1170 is smaller than Figure 11A The overall range of the CSF 1125 makes Figure 11A Users can distinguish between different resolutions greater than approximately 540p, while Figure 11BUsers cannot distinguish between different resolutions greater than approximately 540p.

[0088] As is evident from the above explanation, CSFs 1125 and 1170 are personalized CSFs based on user responses 220 received by the electronic computing device in response to the displayed mixed image and / or other test media 225. The personalized CSF determined by the electronic computing device is similar to... Figure 5 The CSF 520 is shown in the diagram and previously explained herein. In other words, instead of using general ABR logic (e.g., Figure 5 The ideal CSF (515) controls the media streaming from the media server 105 to the playback system 210, which can be achieved using personalized bit rate / resolution decision rules. In some embodiments, the electronic computing device generates a new personalized ABR ladder based on the user response 220 to the test media 225 during a test measurement session.

[0089] In some embodiments, one or more stored PSPs 215 may influence the characteristics of test media 225 output by playback system 210 during a test measurement session. In some embodiments, the electronic computing device retrieves previously stored personalized sensitivity distribution curves (PSPs) 215 and generates test media 225 based on one or more viewing characteristics contained in the previously stored PSPs 215. In some embodiments, to retrieve previously stored PSPs 215, the electronic computing device determines the characteristics of the current / ongoing test measurement session, including at least one of the characteristics of user 135, the characteristics of the first playback system 210, and the characteristics of the environment 130 in which user 135 is viewing the first playback system 210. The electronic computing device can then identify a previously stored PSP 215 from a plurality of previously stored PSPs 215 based on the previously stored PSPs 215 (containing one or more of the same characteristics as the characteristics of the current / ongoing test measurement session).

[0090] For example, the electronic computing device may determine that the stored PSP 215 contains information about the same user 135, but the current playback system 210 and / or the current environment 130 differs from the stored playback system 210 and / or environment 130 (e.g., the same user is watching TV on different TVs in different rooms of their home). Even though the characteristics of the stored PSP 215 do not accurately match the current context, the electronic computing device can still use one or more media parameters of the stored PSP 215 as a baseline for starting the output of test media 225 during a test measurement session. In other words, the electronic computing device may output test media 225 filtered or otherwise modified (e.g., mixed images) according to the stored PSP 215, rather than outputting test media 225 randomly or according to a generic model. In some cases, outputting test media 225 based on the media parameters contained in the stored PSP 215 can reduce the duration of the test measurement session and / or improve measurement accuracy optimally suited to the current context. For example, if an electronic computing device generates a contrast sensitivity function (CSF) to specify personalized sensitivity information, typically dozens of measurements are needed to accurately estimate the media parameters of the CSF in a test measurement session. However, when the electronic computing device begins a test measurement session from a starting point that has already been measured for users 135 in different environments 130 and / or through different playback systems 210 (or for another common attribute besides ordinary user 135), the number of measurements required to accurately estimate the media parameters in the current context can be reduced compared to typical measurements. In other words, the spatial frequency and contrast of the current stimulus for CSF measurement in the current test measurement session can be adjusted based on the user response 220 of the previously tested media 225 and the CSF estimate from the previously stored PSP 215.

[0091] Along a similar line, in some embodiments, when generating PSP 215 to estimate and optimize QoE, a single PSP 215 can be estimated from a plurality of stored PSPs 215 or a single PSP 215 that closely matches other identified attributes (e.g., orientation, demographics, viewing device brand / model, screen size, etc.). For example, when the electronic computing device detects a change in user 135, environment 130, and / or playback device 210 and user 135 chooses not to participate in a new test measurement session, the electronic computing device can generate an estimated PSP 215 based on a plurality of stored PSPs 215 for similar user 135, environment 130, and / or playback device 210.

[0092] In use cases where a single playback system 210 has multiple viewers (each with a unique or unknown PSP) (e.g., a television in a home with multiple users / viewers), the electronic computing device can select an individual PSP 215 based on a variety of different criteria. For example, if the goal of system 600 is to minimize the risk of any user-perceived QoE degradation, the electronic computing device can select the most sensitive PSP 215 from a group of PSPs 215 corresponding to each of the multiple viewers. In this example, the electronic computing device attempts to ensure that even the most sensitive user viewing display 730 does not experience a reduced QoE. Assuming the most sensitive user does not experience a QoE reduction, it is clear that less sensitive users viewing the same display 730 will also not experience a QoE reduction because they are less sensitive to changes in image / video quality compared to the most sensitive user. In some embodiments, system 600 may reduce the number of PSP candidates for a given playback system 210 (e.g., a TV with multiple users / viewers in the home) based on user presence information extracted from other applications (e.g., smart home applications) or GPS information of a personal mobile device.

[0093] At box 925, the electronic computing device determines whether sufficient information has been collected to complete the Personalized Sensitivity Distribution Profile (PSP) 215. As explained above, this information can be collected from the current user response 220 to the current test medium 225 (at box 920) and / or retrieved from previously stored PSP 215 (at box 910). Figure 9 In the diagram, box 910 is shown in dashed lines to indicate that box 910 is optional and may not be performed in some embodiments of method 900. In other words, in some cases, the electronic computing device may generate PSP 215 (at box 930) based on the received user response 220 to test medium 225 without retrieving a previously stored PSP 215.

[0094] Conversely, although in Figure 9Boxes 915, 920, and 925 are not shown in dashed lines, but in some cases, the electronic computing device may not execute boxes 915, 920, and 925. In other words, the electronic computing device may, in some cases, not participate in the test measurement session and may instead rely solely on one or more stored PSPs 215 to generate a PSP 215 for the current media session. For example, after retrieving the stored PSPs 215 for user 135 and / or environment 130 (at box 910), the electronic computing device may immediately determine one of the stored PSPs 215 corresponding to user 135, environment 130, and playback device 210. Therefore, at box 930, the electronic computing device may utilize the corresponding previously stored PSP 215 as the PSP 215 for the current media session of user 135 on playback device 210 in environment 130. In this scenario, no electronic computing device is required to participate in the test measurement session because the viewing characteristics of the current media delivery session were previously stored in the PSP 215 during the previous test measurement session.

[0095] As an electronic computing device, it does not participate in test and measurement sessions (i.e., it does not perform...). Figure 9 In another example of frames 915, 920, and 925, as explained above, when the electronic computing device detects a change in user 135, environment 130, and / or playback device 210, and user 135 chooses not to participate in a new test and measurement session, the electronic computing device can generate an estimated PSP 215 based on a plurality of stored PSPs 215 for similar user 135, environment 130, and / or playback device 210. For example, if a stored PSP 215 is associated with the same user, the electronic computing device can adjust one or more characteristics of the stored PSP 215 based on known changes in display size or other display characteristics between the playback system 210 associated with the stored PSP 215 and the playback system 210 currently used by user 135. Similarly, the electronic computing device can adjust one or more characteristics of the stored PSP 215 based on known changes in the environment of user 135. For example, based on sensor data from environmental sensor 305, the electronic computing device can determine that the current environment 130 is darker than the environment 130 associated with the stored PSP 215. In other embodiments, instead of generating an estimated PSP 215, the electronic computing device can retrieve and use a stored PSP 215 that includes characteristics similar to the determined and / or known characteristics of the user 135, environment 130, and / or playback device 210. For example, the electronic computing device can retrieve a stored PSP 215 for the user 135, even if the stored PSP 215 is for a different environment 130 and / or a different playback system 210.

[0096] The explanation of return box 925 states that when the electronic computing device determines that more information is needed to complete PSP 215 (e.g., to complete more accurately such as...), Figure 5 , 11A When the test medium 225 (as shown in 11B) is output, method 900 continues to return to block 915 to continue outputting the test medium 225 and receiving user input (i.e., user response 220) in response to the test medium 225. When the electronic computing device determines that sufficient information has been collected to complete the PSP 215, the method proceeds to block 930.

[0097] At box 930, the electronic computing device generates a personalized sensitivity distribution (PSP) curve for one or more viewing characteristics of the user based on user input. For example, such as Figure 5 As shown, the electronic computing device generates a personalized CSF 520 translated from and / or scaled up from the ideal CSF 515. Alternatively, the algorithm for determining streaming parameters (e.g., ABR request 140 (or a request regarding another media delivery method)) may have adjusted coefficients stored in the PSP 215. In some embodiments, the electronic computing device generates a personalized ABR ladder (or another personalized media delivery method) to be included in the PSP 215.

[0098] At block 935, the electronic computing device determines media parameters at least in part based on PSP 215. For example, the electronic computing device determines values ​​for media parameters (e.g., one or more values ​​of segment size, bit rate, resolution, frame rate, another media parameter affecting the operation of the video encoder / transcoder / rasterizer associated with media server 105 and / or network 115, etc.). At block 940, media server 105 provides output media to playback system 210 via network 115 according to the media parameters. The output media is configured to be output through playback system 210 (e.g., images / videos are configured to be output on display 730 of playback system 210).

[0099] To determine the media parameters (i.e., the values ​​of the media parameters), at box 935, the electronic computing device may perform the following: Figure 4 As shown and previously mentioned in this article Figure 4 The transformation from a generic target ABR logic model to a personalized target ABR logic model (POM405) is described. In some embodiments, POM 405 can be implemented in streaming systems that support adaptive bit rate delivery (e.g., televisions, set-top boxes, digital media adapters, or similar systems). Figure 6Within the mobile device shown. In this example, the ABR request logic 140 of the playback system 210 utilizes POM 405 to improve the selection of encoded video and / or audio segments based on, but not limited to, segment size, bit rate, resolution, frame rate, codec, etc., thereby matching or providing the closest match among the available encoded segments to the PSP 215 of the user 135. For example, an algorithm defining a generic target ABR logic model (or another generic media delivery method) for determining streaming parameters (e.g., ABR request 140 (or a request regarding another media delivery method)) can adjust / personalize its coefficients based on information stored in the PSP 215.

[0100] As described earlier in this article, Figure 6 In this context, ABR ladder 137 represents a range of audio and video segments available in terms of bitrate, resolution, frame rate, etc. As previously explained herein, existing media delivery systems / methods (whether ABR-enabled or otherwise) are inefficient and typically cause playback system 210 to request more data than is needed for seamless playback and / or request values ​​exceeding those of user 135's PSP 215's media parameters (e.g., the combination of resolution / bitrate / frame rate). In other words, existing media delivery logic attempting to increase the delivered resolution / bitrate / frame rate beyond user 135's sensitivity threshold will not translate into improved QoE for user 135. The disclosed POM-based media delivery method translates into more efficient delivery and thus reduces delivery costs for over-the-top (OTT) services, which purchase data in gigabytes from their content delivery network (CDN) providers.

[0101] In some embodiments, at block 935, the electronic computing device selects values ​​for one or more media parameters (e.g., a combination of resolution / bit rate / frame rate) to generate streaming media within the sensitivity perception range of user 135. For example, the electronic computing device may use... Figure 11A The CSF 1125 controls the ABR request 140 to the media server 105 for streaming media resolution approximately 720p, because the CSF 1125 indicates that the sensitivity peak 1110 of the first user is approximately 720p. On the other hand, the electronic computing device can use... Figure 11BThe CSF 1170 controls the ABR request 140 to the media server 105 for a lower resolution streaming media resolution of approximately 540p, because the CSF 1170 indicates that the sensitivity peak 1175 of the second user is approximately 540p. In the two immediately following examples, a first media parameter (i.e., 540p resolution) and a second media parameter (i.e., 720p resolution) are determined such that the first output media provided to the first playback device 210 of the second user will reduce the first resource usage of network 115 to a level lower than the second resource usage of network 115 with respect to the second playback device 210 providing the second output media to the first user. However, despite this difference in resource usage when the output media is streamed to the two playback devices 210, the first percentage of the first personalized QoE of the first user remains at a level approximately the same as the second percentage of the second personalized QoE of the second user (see Figure 12B (and Table 1B).

[0102] While method 900 is described above with respect to a media session with a single playback system 210 or two playback systems 210, in some embodiments, method 900 may be performed with respect to additional playback systems 210. For example, method 900 may be used to determine the PSP 215 of each of a plurality of playback systems 210 receiving media streams from a particular node on network 115. A computing device may refine / optimize one or more media parameters (e.g., decoding and delivery parameters) for each media stream being supplied to each of the plurality of playback systems 210 to refine / optimize the total / overall media stream from network 115.

[0103] For example, mobile wireless network and broadband network operators can utilize the disclosed POM-based media delivery and decoding method 900 to add additional capacity to existing access networks without compromising end-user / viewer QoE. In some embodiments, method 900 provides network operators with a new method to reduce the capital investment ratio required to increase network capacity. Figure 12A and 12B Tables 1A and 1B illustrate the instance bandwidth and QoE statistics of network 115 when using existing streaming methods versus using method 900 streaming media. Servers using existing methods are referred to as existing servers, while servers using method 900 are referred to as xCD servers (i.e., experience-based decoding and delivery servers).

[0104] Figure 12A Chart 1205 illustrates a network with unlimited bandwidth. The upper curve 1210 represents the bandwidth used for streaming media to existing servers. The lower curve 1215 represents the bandwidth used for streaming media to xCD servers using Method 900. Figure 12AAs shown, the bandwidth used by the media for streaming via method 900 for the xCD server is approximately 20% less than the bandwidth used by the same media for streaming via an existing server. Additionally, as shown by the corresponding... Figure 12A As indicated in Table 1A below, the QoE remains at 100% (i.e., perfect QoE) for all viewers (e.g., both highly sensitive users [i.e., near viewers] and low-sensitive users [i.e., far viewers]).

[0105] Table 1A :

[0106]

[0107]

[0108] As by Figure 12A As indicated in Table 1A, method 900 produces more efficient media delivery without reducing the user QoE of each user group receiving a unicast session based on their respective PSP 215. This increase in efficiency and reduction in bandwidth without reducing user QoE is a result of system 600 reducing the bit rate, resolution, etc., of users 135 who do not experience an increase in QoE when the bit rate, resolution, etc., of the streaming media increases beyond a certain point. In other words, in some embodiments, method 900 may be intended to deliver media of the highest perceived quality to each user and not deliver media of a higher quality than that perceived by a particular user to any particular user (i.e., personalized media content delivery).

[0109] Figure 12B Chart 1250 illustrates a network with limited bandwidth / fixed network capacity (e.g., approximately 60 Mbps). Curve 1255 represents the bandwidth used for media streaming to existing servers. Curve 1260 represents the bandwidth used for media streaming to xCD servers using method 900. (Different from...) Figure 12A The curve, Figure 12B Curves 1255 and 1260 use approximately the same bandwidth over time. In some embodiments, the bandwidth used by media streamed by method 900 for an xCD server may be approximately 1% smaller than the bandwidth used by the same media streamed by an existing server. However, as per the corresponding... Figure 12B As indicated in Table 1B below, the xCD server using method 900 on a fixed-capacity network link achieves a more consistent reduction in QoE for both high-sensitivity and low-sensitivity users compared to existing servers implementing the conventional ABR segment selection method.

[0110] Table 1B :

[0111]

[0112] For example, Table 1B indicates that highly sensitive users (i.e., users closest to their respective playback system 210) experience approximately a 40% QoE reduction in a limited bandwidth network when streaming media through an existing server. In contrast, low-sensitive users (i.e., users furthest from their respective playback system 210) experience only approximately a 20% QoE reduction in a limited bandwidth network when streaming media through an existing server. This difference in QoE reduction is caused by the existing server reducing the streaming quality for all users in the same way, even though changes in streaming quality affect different users in different ways.

[0113] On the other hand, because the xCD server uses user PSP 215 to more intelligently reduce streaming quality differently for different users, the same limited bandwidth network can provide a more consistent QoE reduction across all users of system 600. In some embodiments, the more consistent QoE reduction results in a higher overall QoE for users of system 600. For example, Table 1B indicates that highly sensitive users (i.e., users closest to their respective playback system 210) experience only approximately 20% QoE reduction in a limited bandwidth network when streaming media through the xCD server. Similarly, low-sensitive users (i.e., users furthest from their respective playback system 210) experience only approximately 20% QoE reduction in a limited bandwidth network when streaming media through the xCD server. In other words, as indicated by Table 1B, the xCD server implementation method 900 can significantly improve the QoE for highly sensitive users while only moderately reducing or maintaining the QoE for low-sensitive users.

[0114] Figure 13A and 13B This illustrates another example of how Method 900 can allow more users / subscribers to stream media on a fixed-capacity network without adversely affecting QoE. Figure 13A and 13B Example charts 1305 and 1350 show the number of subscribers (x-axis) serving a fixed-capacity network with 88% effective throughput after accounting for overhead, and the percentage reduction in QoE experienced by subscribers (y-axis). Figure 13A and 13B Figures 1305 and 1350 assume a 50% separation between highly sensitive viewers (e.g., three picture heights [3H] away from their respective playback system 210) and low-sensitive viewers (e.g., six picture heights [6H] away from their respective playback system 210).

[0115] Charts 1305 and 1350 confirm that when users / subscribers are added to a fixed-capacity network, the video resolution must be downgraded once the number of users / subscribers reaches certain thresholds 1310 and 1355 (as examples). However, similar to the information regarding... Figure 12A and 12B The examples above in Tables 1A and 1B degrade the resolution of all users / subscribers equally (e.g.) Figure 13A As shown, the QoE varies depending on whether the user / subscriber is watching at 3H or 6H. For example, when the same resolution reduction is applied across all media streams, users watching at 3H typically perceive a greater QoE reduction compared to those watching at 6H. This difference in QoE reduction between different types of users is caused by… Figure 13A and 13B The 3H curve 1315 and 6H curve 1320 shown illustrate this. The shaded area 1325 between the 3H curve 1315 and the 6H curve 1320 illustrates the unequal QoE reduction (i.e., perceived service degradation) between high-sensitivity users / subscribers and low-sensitivity users / subscribers. For example, for a network serving 1200 users, the video quality degradation is visible to 60% of users (i.e., high-sensitivity users) at 3H, but only to about 2% of users (i.e., low-sensitivity users) at 6H.

[0116] Figure 13A The first QoE curve 1330 illustrates the QoE experienced by different users / subscribers when a streaming management method is used to adjust the video decoding bit rate and resolution in the same way based on the viewing distance of all users / subscribers. On the other hand, Figure 13B The second QoE curve 1360 indicates the value when using... Figure 9 Method 900 personalizes the QoE experience for different users / subscribers when additional users are added to the network and when resolution degradation occurs. As explained above, Figure 13A and 13B Both assume a 50% separation between 3H and 6H viewing distances among users / subscribers.

[0117] Using method 900, the electronic computing device controlling the media parameters understands which users are viewing at what distance and how a reduction in resolution will affect the QoE of each user (e.g., based on information stored in each user's PSP 215). Therefore, the electronic computing device performing method 900 can allocate bit rate / resolution combinations to achieve equal average QoE across the two user groups (i.e., high-sensitivity users and low-sensitivity users). This improvement is achieved by… Figure 13A The QoE curve in the middle is 1330 and Figure 13B The difference between the QoE curves at 1360 and 1360 is illustrated. For example, Figure 13AThe threshold of 1310 at which any user experiences a decrease in QoE is when the network serves approximately 600 users. In comparison, Figure 13B The threshold of 1355 is roughly double for approximately 1200 users. In other words, the network implementing method 900 can serve roughly twice as many users as the existing network can serve, and no user will experience a decrease in QoE.

[0118] Return to Figure 9 As indicated by the dashed arrows, in some embodiments, the electronic computing device repeats boxes 935 and 940. Repeating these boxes allows for real-time tracking of sensor outputs in the playback environment 130 (e.g., ambient sensor 305) and / or network 115 during media delivery services, and for dynamically / adaptively adjusting the personalized target model 405 to suit common use cases. For example, system 600 may determine that the room where playback system 210 is located has become darker since user 135 began watching television (e.g., due to sunset, due to user 135 drawing the curtains, etc.). In response to this determination, system 600 may adjust the personalized target model 405 by, for example, selecting a different stored PSP 215 that more closely resembles the ambient light characteristics of the now darker room.

[0119] As previously explained herein, the ABR ladder 137 and ABR selection method mentioned herein are merely one example method available to system 600 for controlling media delivery from media server 105 to playback system 210 via network 115. In other embodiments, other methods may be used to dynamically adjust video encoder / transcoder / rasterizer parameters (i.e., media parameters), such as the bit rate and / or resolution of the encoded media being streamed. Similar to the ABR-related methods included in various instances, these other media delivery methods adjust their media parameters based on one or more PSPs 215 to optimize media delivery as described herein. In some embodiments, the media delivery method is an upstream media delivery method implemented by media server 105 and / or network 115 (i.e., upstream of playback system 210).

[0120] It should be understood that the embodiments are not limited in their application to the details of the configuration and arrangement of the components set forth herein or illustrated in the accompanying drawings. The embodiments can be practiced or implemented in various ways. Furthermore, it should be understood that the wording and terminology used herein are for illustrative purposes and should not be considered restrictive. The use of “comprising,” “including,” or “having,” and variations thereof, is intended to cover the items listed thereafter and their equivalents, as well as additional items. Unless otherwise specified or limited, the terms “mounted,” “connected,” “supported,” and “coupled,” and variations thereof, are used broadly and cover direct and indirect mounting, connection, support, and coupling.

[0121] Additionally, it should be understood that embodiments may include hardware, software, and electronic components or modules, which may be shown and described for the purposes of discussion as if most components were implemented solely in hardware. However, those skilled in the art, and upon reading based on this specific embodiment, will recognize that in at least one embodiment, the electronic aspects may be implemented by software executed by one or more electronic processors (e.g., stored on a non-transitory computer-readable medium), such as microprocessors and / or application-specific integrated circuits (“ASICs”). Therefore, it should be noted that multiple hardware and software-based devices and multiple different structural components may be used to implement embodiments. For example, the “server” and “computing device” described in the specification may include one or more electronic processors, one or more computer-readable media modules, one or more input / output interfaces, and various connectors (e.g., system buses) connecting the various components.

[0122] The appended claims set forth various features and aspects.

Claims

1. A method for delivering media to a playback device, the method comprising: outputting, by a first playback device, first test media for viewing by a first user during a first test measurement session; receiving first user input from the first user, the first user input relating to a first perception of the first test media by the first user and indicating a first personalized quality of experience of the first user with respect to the first test media; generating, by one or more electronic processors, a first personalized sensitivity profile curve including one or more viewing characteristics of the first user and / or one or more viewing characteristics of an environment in which the first user is viewing the first playback device based on the first user input; determining, by the one or more electronic processors, first media parameters based at least in part on the first personalized sensitivity profile curve to improve efficiency of media delivery to the first playback device via a network while maintaining the first personalized quality of experience of the first user; and providing, by a media server, first output media to the first playback device via the network in accordance with the first media parameters, the first output media configured to be output by the first playback device, wherein the first test media includes a plurality of mixed images, and the first user input includes a plurality of user inputs, each of the first user inputs received in response to a respective mixed image of the plurality of mixed images, and wherein determining the first media parameters includes: retrieving, by the one or more electronic processors, a generic objective model configured to control media streaming from a memory based on at least one of resource availability of the network and playback system parameters; transforming, by the one or more electronic processors, the generic objective model into a personalized objective model using the first personalized sensitivity profile curve; and providing, by the media server, the first output media to the first playback device via the network in accordance with the personalized objective model.

2. The method of claim 1, wherein providing the first output media to the first playback device in accordance with the first media parameters causes the network to provide the first output media to the first playback device using reduced bandwidth without degrading the first personalized quality of experience of the first user.

3. The method of claim 1 or claim 2, further comprising: outputting, by a second playback device, second test media for viewing by a second user during a second test measurement session; receiving second user input from the second user, the second user input relating to a second perception of the second test media by the second user and indicating a second personalized quality of experience of the second user with respect to the second test media, wherein the second personalized quality of experience indicates that the second user is more sensitive to media quality degradation than the first user of the first playback device; ​ generating, by the one or more electronic processors, a second individualized sensitivity profile that includes one or more viewing characteristics of the second user and / or one or more viewing characteristics of an environment in which the second user is located while viewing the second playback device based on the second user input; determining, by the one or more electronic processors, second media parameters for the second playback device based at least in part on the second individualized sensitivity profile; providing, by the one or more electronic processors, second output media to the second playback device via the network in accordance with the second media parameters, the second output media being configured to be output by the second playback device; wherein the first media parameters and the second media parameters are determined such that providing the first output media to the first playback device reduces a first amount of resource usage of the network to less than a second amount of resource usage of the network with respect to providing the second output media to the second playback device; and wherein a first percentage of the first individualized quality of experience for the first user is maintained at approximately the same level as a second percentage of the second individualized quality of experience for the second user.

4. The method of claim 3, wherein the one or more electronic processors include at least one of a first electronic processor of the first playback device, a second electronic processor of the second playback device, and a third electronic processor associated with the network or the media server.

5. The method of claim 1 or claim 2, further comprising: generating, by the one or more electronic processors, each blended image by low-pass filtering a first source image to form a low-pass filtered source image, high-pass filtering a second source image to form a high-pass filtered source image, and overlapping the low-pass filtered source image and the high-pass filtered source image with one another to form the blended image; wherein the first user perceives the low-pass filtered source image or the high-pass filtered source image as dominant depending on the one or more viewing characteristics of the first user and / or the environment.

6. The method of claim 5, wherein outputting the first test media includes: outputting, by the first playback device, a first blended image for viewing by the first user; receiving, from the first user, the first user input related to the first perception of the first blended image by the first user; and outputting, by the first playback device, a second blended image for viewing by the first user, wherein a cutoff frequency of at least one of the low-pass filtering and the high-pass filtering used to form the second blended image is based on the first user input related to the first perception of the first blended image by the first user.

7. The method of claim 1 or claim 2, wherein outputting the first test media includes: retrieving, by the one or more electronic processors, a previously stored individualized sensitivity profile from memory; and generating, by the one or more electronic processors, the first test media based on one or more viewing characteristics included in the previously stored personalized sensitivity profile.

8. The method of claim 7, wherein retrieving the previously stored personalized sensitivity profile includes: determining, by the one or more electronic processors, characteristics of the first test measurement session, the characteristics of the first test measurement session including at least one of characteristics of the first user, characteristics of the first playback device, and characteristics of the environment in which the first user is watching the first playback device; and identifying, by the one or more electronic processors, the previously stored personalized sensitivity profile from a plurality of previously stored personalized sensitivity profiles based on the previously stored personalized sensitivity profile including one or more of the same characteristics as the characteristics of the first test measurement session.

9. The method of claim 1 or claim 2, wherein transforming the generic target model into the personalized target model includes at least one of a translation and a scaling operation to translate and scale an ideal contrast sensitivity function (CSF) used for the generic target model to form a personalized CSF based on the first personalized sensitivity profile.

10. The method of claim 1 or claim 2, wherein the one or more viewing characteristics of the first user and / or the environment include at least one of a viewing distance between the first user and a display of the first playback device, an illumination of the environment in which the first user is watching the display, and a visual sensitivity of the first user’s eyes.

11. The method of claim 1 or claim 2, further comprising: determining, by an environmental sensor in the environment in which the playback device is located, an environmental condition; wherein generating the first personalized sensitivity profile including the one or more viewing characteristics of the first user and / or the environment includes generating, by the one or more electronic processors, the first personalized sensitivity profile such that the first sensitivity profile includes the environmental condition.

12. An electronic computing device comprising: a first playback device including a display, wherein the display is configured to output media to a first user; and one or more electronic processors communicatively coupled to the display, the one or more electronic processors configured to output, by the first playback device, first test media for viewing by the first user during a first test measurement session, receive first user input from the first user, wherein the first user input relates to a first perception of the first test media by the first user and is indicative of a first personalized quality of experience of the first user with respect to the first test media, generate, based on the first user input, a first personalized sensitivity profile including one or more viewing characteristics of the first user and / or one or more viewing characteristics of an environment in which the first user is watching the first playback device, determining a first media parameter based at least in part on the first personalized sensitivity profile to improve efficiency of media delivery to the first playback device via a network while maintaining the first personalized quality of experience of the first user, and providing, by a media server, first output media to the first playback device via the network in accordance with the first media parameter, wherein the first output media is configured to be output by the first playback device, wherein the first test media includes a plurality of mixed images, and the first user input includes a plurality of user inputs, each of the first user inputs being received in response to a respective mixed image of the plurality of mixed images, and wherein determining the first media parameter includes: retrieving, by the one or more electronic processors, a generic objective model configured to control media streaming from a memory based on at least one of resource availability of the network and playback system parameters; transforming, by the one or more electronic processors, the generic objective model into a personalized objective model using the first personalized sensitivity profile; and providing, by the media server, the first output media to the first playback device via the network in accordance with the personalized objective model.

13. The electronic computing device of claim 12, wherein the one or more electronic processors include at least one of a first electronic processor of the first playback device and a second electronic processor associated with the network or the media server.

14. A method for displaying a mixed image on a playback device, the method comprising: determining, by one or more electronic processors of an electronic computing device, a first value of a media parameter supported by a media server and a network configured to stream media; determining, by the one or more electronic processors, a second value of the media parameter supported by the media server and the network; at least one of generating and selecting, by the one or more electronic processors, the mixed image based on the first value of the media parameter and the second value of the media parameter such that the mixed image includes a first interpretation corresponding to the first value of the media parameter and a second interpretation corresponding to the second value of the media parameter; displaying the mixed image on a display of the playback device; determining, by the one or more electronic processors, that a user is more sensitive to the first value of the media parameter than the second value of the media parameter based on user input; and generating, by the one or more electronic processors, a personalized sensitivity profile of a viewing characteristic of the user and / or a viewing characteristic of an environment in which the user is viewing the playback device based on the determination that the user is more sensitive to the first value of the media parameter than the second value of the media parameter, the personalized sensitivity profile including the first value of the media parameter. ​ ​ 15. The method of claim 14, wherein the media parameters include at least one of a video resolution, a bit rate, and a frame rate of a media stream from the media server via the network.

16. The method of claim 14 or claim 15, further comprising: receiving, by an input device of the playback device, the user input from the user, the user input indicating that the user perceives the first interpretation when the mixed image is displayed on the display; determining, by the one or more electronic processors, a third value of the media parameters supported by the media server and the network based on the user input; at least one of generating and selecting, by the one or more electronic processors, a second mixed image based on the first value of the media parameters and the third value of the media parameters, such that the second mixed image includes a third interpretation corresponding to the third value of the media parameters and a fourth interpretation corresponding to the first value of the media parameters; and displaying the second mixed image on the display of the playback device.

17. The method of claim 14 or claim 15, wherein at least one of the operations of generating and selecting the mixed image includes generating, by the one or more electronic processors, the mixed image by: low-pass filtering a first source image to form a low-pass filtered source image, wherein a cutoff frequency of the low-pass filtering is based on the first value of the media parameters; high-pass filtering a second source image to form a high-pass filtered source image, wherein a cutoff frequency of the high-pass filtering is based on the second value of the media parameters; and overlapping the low-pass filtered source image and the high-pass filtered source image on top of each other to form the mixed image; wherein, when the mixed image is displayed on the display, the user perceives the low-pass filtered source image or the high-pass filtered source image as dominant depending on one or more viewing characteristics of the user.

18. The method of claim 14 or claim 15, further comprising: receiving, by an input device of the playback device, the user input from the user, the user input indicating that the user perceives the first interpretation when the mixed image is displayed on the display; and providing, by the media server, output media to the playback device via the network in accordance with the personalized sensitivity profile, the output media configured to be output by the playback device.

19. The method of claim 18, further comprising: determining, by the one or more electronic processors, based on the user input, at least one of: a subset of spatial frequencies of the mixed image to which the user is most sensitive and a size determination of the mixed image to which the user is most sensitive; wherein the viewing characteristics of the personalized sensitivity profile include the at least one of: the subset of spatial frequencies of the mixed image to which the user is most sensitive and the size determination of the mixed image to which the user is most sensitive.

Images