Generating hybrid images for use when capturing personalized playback context information for the user.

Abstract

The method may include generating a hybrid image associated with a first interpretation corresponding to a first value of the media parameter and a second interpretation corresponding to a second value of the media parameter. The hybrid image may include a first visibility ratio between the first interpretation and the second interpretation. The method may include refining the hybrid image to create an improved hybrid image including a second visibility ratio different from the first visibility ratio. The method may include displaying the improved hybrid image on a display of a first playback device and receiving user input related to a first perception of the improved hybrid image by a user. The method may include determining optimized values ​​of the media parameters based at least in part on the user input and providing output media to the playback device for display to the user in accordance with the optimized values ​​of the media parameters.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority to European Patent Application No. 22160457.2 filed on 7 March 2022 and U.S. Provisional Application No. 63 / 307,566 filed on 7 February 2022, the entire contents of which are incorporated herein by reference.

[0002] [Technical field] This application relates to media. More specifically, embodiments of the present invention relate to processing, displaying and / or distributing visual media. [Background technology]

[0003] Various aspects of this disclosure relate to devices, systems, and methods for providing the delivery of visual media to user devices over a network for display by user devices for viewing by users. As described in PCT application PCT / US2020 / 044241 filed on 30 July 2020, now International Publication WO2021 / 025946, adaptive bit rate (ABR) streaming in a visual media delivery chain enables improved network resource management through adaptive selection of bitrate and resolution on a media ladder based on network conditions, playback buffer state, shared network capacity, and other network-affected factors. Its entirety is incorporated herein by reference and attached herein as Appendix B. In addition to ABR streaming, other media delivery methods (including coding methods or source coding methods) may similarly be used to control one or more media parameters of an upstream video encoder / transcoder / translator, such as bitrate, frame rate, and resolution. For example, the methods described herein are also applicable to scalable video coding (e.g., H.264 / Scalable Video Coding (SVC), H.265 / Scalable High Efficiency Video Coding (SHVC), Versatile Video Coding (VVC), Multilayer Main 10, VP9 Video Coding, and AOMedia Video 1 (AV1)), simulcasting of multiple alternative bitstreams, and reference picture resampling (RPR) coding tools in VVC for use cases including broadcast, broadband, one-to-one, and multi-party video communications.

[0004] Furthermore, as described in PCT application PCT / US2020 / 044241, it is advantageous to share parameters related to playback device characteristics and personalized visual sensitivity factors with an upstream device configured to control the transmission of visual media to the playback device. Specifically, it is often advantageous to provide personalized adaptive media delivery based on collected playback-side information without using individual sensors. Moreover, the collected playback-side information may indicate personalized quality of experience (QoE) for different users and / or different viewing environments. Thus, improvements in network resource management / media delivery efficiency may exist while maintaining personalized QoE for each user.

[0005] PCT application PCT / US2020 / 044241 also describes the use of hybrid images to collect data that may be used to estimate a user's QoE in response to user input regarding the user's perception of the displayed hybrid image. However, not all hybrid images are useful for evaluating the user's perception of the hybrid image. [Overview of the project]

[0006] Accordingly, the disclosed devices, systems, and methods aim to address the above technical problems in generating (or selecting or receiving) hybrid images that are more useful for evaluating the user's perception of hybrid images with respect to relevant values ​​of media parameters used to control the delivery of visual media to user devices on a network. In other words, the disclosed devices, systems, and methods include sensorless methods for capturing playback-side context information using hybrid images for improved media processing and delivery. The disclosure includes methods for creating hybrid images to estimate an approximate minimum resolution for approximate maximum perceived quality (e.g., approximate minimum-maximum QoE resolution), given a set of available video resolution settings for a media streaming. The disclosure also includes methods for estimating a model of playback-side context information (e.g., estimated QoE transfer function, estimated contrast sensitivity function (CSF), etc.) as a function of spatial frequency. In some embodiments, the disclosed devices, systems, and methods are used in conjunction with context / environment sensors (e.g., sensors on the playback device configured to collect context information such as ambient light information, viewing distance between the user and the playback device, time, and / or geographic location of the playback device) to capture playback-side context information using hybrid images for improved media processing and delivery.

[0007] In one embodiment of the present disclosure, a method executable by one or more electronic processors is provided. The method may include at least one of generating and selecting a hybrid image associated with a first interpretation corresponding to a first value of a media parameter and a second interpretation corresponding to a second value of the media parameter using one or more electronic processors. The hybrid image may include a first visibility ratio between the first interpretation and the second interpretation. The method may further include improving the hybrid image using one or more electronic processors to create an improved hybrid image including a second visibility ratio different from the first visibility ratio. The method may further include displaying the improved hybrid image on a display of a first playback device. The method may further include receiving a first user input from a first user using one or more electronic processors. The first user input may be related to a first perception of the improved hybrid image by the first user. The method may further include determining an optimized value of the media parameter at least partially based on the first user input using one or more electronic processors. The method may further include providing a first output media to the first playback device on a network according to the optimized value of the media parameter. The first output media may be configured to be output by the first playback device.

[0008] In another embodiment, a non - transient computer - readable storage medium storing one or more programs configured to be executed by one or more electronic processors of an electronic computing device that may include a network interface and a display is provided. The one or more programs may include instructions for performing the above method and / or any of the methods described herein.

[0009] In another embodiment, an electronic computing device may be provided that includes a network interface, a display, one or more electronic processors, and a memory storing one or more programs configured to be executed by the one or more electronic processors. The one or more programs may include instructions for performing any of the above methods and / or the methods described herein.

[0010] Other aspects of the embodiments will become apparent by considering the detailed description and the accompanying drawings.

Brief Description of the Drawings

[0011] This patent or application file includes at least one drawing created in color. Copies of this patent or patent application publication, which include color drawings, will be provided by the Patent Office upon request and payment of the necessary fees. [Figure 1] An exemplary media coding and delivery system according to an embodiment described herein is shown. [Figure 2] A block diagram of a playback device of the media coding and delivery system of FIG. 1 according to an embodiment described herein is shown. [Figure 3] A block diagram of a media server of the media coding and delivery system of FIG. 1 according to an embodiment described herein is shown. [Figure 4A] Exemplary source images that can be used to generate the exemplary hybrid images shown in FIGS. 4C - 4E according to an embodiment described herein are shown. [Figure 4B] Exemplary source images that can be used to generate the exemplary hybrid images shown in FIGS. 4C - 4E according to an embodiment described herein are shown. [Figure 4C] Exemplary hybrid images generated using the exemplary source images shown in FIGS. 4A and 4B according to an embodiment described herein are shown. [Figure 4D]This specification shows an exemplary hybrid image generated using the exemplary source images shown in Figures 4A and 4B, according to embodiments described herein. [Figure 4E] This specification shows an exemplary hybrid image generated using the exemplary source images shown in Figures 4A and 4B, according to embodiments described herein. [Figure 5] Examples of azimuthal-averaged 1-d power spectra along the radius from the origin of the source images shown in Figures 4A and 4B are shown according to embodiments described herein. [Figure 6A] The following are exemplary filter responses of a 6th-order Butterworth slow-pass filter and a 6th-order Butterworth high-pass filter used to filter the source images in Figures 4A and 4B, respectively, according to embodiments described herein. [Figure 6B] This shows the low-pass filtered image of the source image of Figure 4A according to the embodiments described herein. [Figure 6C] This shows the high-pass filtered image of the source image of Figure 4B according to the embodiment described herein. [Figure 6D] Figure 6C shows a hybrid image created by adding the filtered source image of Figure 6B to the filtered source image of Figure 6C, according to an embodiment described herein. [Figure 6E] Examples of azimuthal-averaged 1-d power spectra along the radius from the origin of the low-pass filtered source image in Figure 6B and the high-pass filtered source image in Figure 6C, according to embodiments described herein, are shown. [Figure 6F] A modified version of the graph in Figure 6E is shown, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 6G] Figure 6D shows a modified version of the hybrid image, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 7A] The following are exemplary filter responses of a 6th-order Butterworth slow-pass filter and a 6th-order Butterworth high-pass filter used to filter the source images in Figures 4A and 4B, respectively, according to embodiments described herein. [Figure 7B] This shows the low-pass filtered image of the source image of Figure 4A according to the embodiments described herein. [Figure 7C] This shows the high-pass filtered image of the source image of Figure 4B according to the embodiment described herein. [Figure 7D] Figure 7C shows a hybrid image created by adding the filtered source image of Figure 7B to the filtered source image of Figure 7C, according to the embodiments described herein. [Figure 7E] Examples of azimuthal-averaged 1-d power spectra along the radius from the origin of the low-pass filtered source image in Figure 7B and the high-pass filtered source image in Figure 7C are shown according to embodiments described herein. [Figure 7F] A modified version of the graph in Figure 7E is shown, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 7G] Figure 7D shows a modified version of the hybrid image, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 8A] The following are exemplary filter responses of a 6th-order Butterworth slow-pass filter and a 6th-order Butterworth high-pass filter used to filter the source images in Figures 4A and 4B, respectively, according to embodiments described herein. [Figure 8B] This shows the low-pass filtered image of the source image of Figure 4A according to the embodiments described herein. [Figure 8C] This shows the high-pass filtered image of the source image of Figure 4B according to the embodiment described herein. [Figure 8D] Figure 8C shows a hybrid image created by adding the filtered source image of Figure 8B to the filtered source image of Figure 8C, according to an embodiment described herein. [Figure 8E] Examples of azimuthal-averaged 1-d power spectra along the radius from the origin of the low-pass filtered source image in Figure 8B and the high-pass filtered source image in Figure 8C are shown according to embodiments described herein. [Figure 8F] A modified version of the graph in Figure 8E is shown, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 8G] Figure 8D shows a modified version of the hybrid image, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 9A] The following are exemplary filter responses of a 6th-order Butterworth slow-pass filter and a 6th-order Butterworth high-pass filter used to filter the source images in Figures 4A and 4B, respectively, according to embodiments described herein. [Figure 9B] This shows the low-pass filtered image of the source image of Figure 4A according to the embodiments described herein. [Figure 9C] This shows the high-pass filtered image of the source image of Figure 4B according to the embodiment described herein. [Figure 9D] Figure 9C shows a hybrid image created by adding the filtered source image of Figure 9B to the filtered source image of Figure 9C, according to an embodiment described herein. [Figure 9E] Examples of azimuthal-averaged 1-d power spectra along the radius from the origin of the low-pass filtered source image in Figure 9B and the high-pass filtered source image in Figure 9C are shown according to embodiments described herein. [Figure 9F]Figure 9E shows a modified version of the graph, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 9G] Figure 9D shows a modified version of the hybrid image, and the low-pass filtered image in Figure 6B and the high-pass filtered image in Figure 6C have been equalized by energy according to the embodiments described herein. [Figure 10] To illustrate the relationship between image size and cutoff frequency according to the embodiments described herein, multiple graphs of the spectral distribution of a hybrid image consisting of filtered images A and B, with the image size changed in each graph, are shown. [Figure 11] This specification shows a flowchart for generating an improved hybrid image to be displayed to user 135 in order to evaluate the user's visual ability above a test frequency (ftest), according to the embodiments described herein. [Figure 12A] An exemplary improved hybrid image generated according to the flowchart in Figure 11, based on embodiments described herein, is shown. [Figure 12B] The spectra of the low-pass filtered image and the high-pass filtered image are shown, which are combined to create the improved hybrid image shown in Figure 12A according to the embodiments described herein. [Figure 13A] An exemplary improved hybrid image generated according to the flowchart in Figure 11, based on embodiments described herein, is shown. [Figure 13B] The spectra of the low-pass filtered image and the high-pass filtered image are shown, which are combined to create the improved hybrid image shown in Figure 12A according to the embodiments described herein. [Figure 14A]An exemplary improved hybrid image generated according to the flowchart in Figure 11, based on embodiments described herein, is shown. [Figure 14B] The spectra of the low-pass filtered image and the high-pass filtered image are shown, which are combined to create the improved hybrid image shown in Figure 12A according to the embodiments described herein. [Figure 15] The following are five improved hybrid images according to the embodiments described herein, in which the gain values ​​of the high-pass filtered images increase from left to right. [Figure 16] The following are five improved hybrid images according to the embodiments described herein, in which the gain values ​​of the high-pass filtered images increase from left to right. [Figure 17] A flowchart shows a method that may be performed by an electronic computing device to capture personalized playback-side context information of a user in an environment and to generate a hybrid image for use when providing output media to a playback device according to the personalized playback-side context information, as described herein. [Figure 18A] Figure 12A shows an example of an improved hybrid image, similar to the one shown. [Figure 18B] Figure 18B shows a graph containing the user's estimated contrast sensitivity function (CSF), the first CSF-weighted power spectrum of a low-pass filtered first image, and the second CSF-weighted power spectrum of a high-pass filtered second image, according to the embodiments described herein. The graph in Figure 18B also includes the contents of the graph in Figure 12B. [Figure 19A] Figure 13A shows an illustrative image of the same improved hybrid as shown in Figure 13A. [Figure 19B]Figure 19B shows a graph containing the user's estimated contrast sensitivity function (CSF), the first CSF-weighted power spectrum of a low-pass filtered first image, and the second CSF-weighted power spectrum of a high-pass filtered second image, according to the embodiments described herein. The graph in Figure 19B also includes the contents of the graph in Figure 13B. [Figure 20A] The following is an exemplary improved hybrid image, similar to the one shown in Figure 14A. [Figure 20B] Figure 20B shows a graph containing the user's estimated contrast sensitivity function (CSF), the first CSF-weighted power spectrum of a first low-pass filtered image, and the second CSF-weighted power spectrum of a second high-pass filtered image, according to the embodiments described herein. The graph in Figure 20B also includes the contents of the graph in Figure 14B. [Figure 21] The graphs in Figures 18A, 19A, and 20A show the difference (ΔP) (y-axis) between the power spectra of the low-pass filtered first image and the high-pass filtered second image, respectively, with the gain of the high-pass filtered second image shown. This indicates that the user's perception of the improved hybrid images in Figures 18A, 19A, and 20A changes from the high-pass filtered second image to the low-pass filtered first image. The graph in Figure 21 shows the unweighted ΔP curve and the CSF-weighted ΔP curve as a function of the second scale factor (x-axis) for different improved hybrid image sizes. [Modes for carrying out the invention]

[0012] Figure 1 shows an example of a media coding and distribution system 100. The system 100 includes a media server 105 that provides media (e.g., visual media) to playback devices 110 (e.g., playback systems 110) over a network 115. Although Figure 1 shows a single playback device 110, the media server 105 may be configured to provide the same or different media to further playback devices 110 (e.g., streaming). In some embodiments, the system 100 includes further media servers 105, an environment 130 and / or users 135. For example, the system 100 may be a distributed coding, multi-source, multi-path media distribution system including multiple media servers 105 and / or caches that store media content that may be provided to one or more playback devices 110, as described in PCT application PCT / US21 / 63723 filed December 16, 2021, the full details of which are incorporated herein by reference and attached herein by reference as Appendix A.

[0013] The playback device 110 may include one or more playback devices of one or more types, such as a television, tablet, smartphone, or computer. In some embodiments, the playback device 110 includes a buffer / decoder and a playback renderer, such as those described in PCT / US2020 / 044241, filed on 30 July 2020, now International Publication WO2021 / 025946, the full contents of which are incorporated herein by reference. The playback device 110 is located in environment 130. A user 135 is also located in environment 130 and may view media output by the playback device 110.

[0014] Figure 2 is a block diagram of a playback device 110 (e.g., playback system 110) according to one exemplary embodiment. As shown, the playback device 110 includes a first electronic processor 205 (e.g., a microprocessor or other electronic device). The first electronic processor 205 includes input and output interfaces (not shown) and is electrically coupled to a first memory 210, a first network interface 215, an optional microphone 220, a speaker 225, and a display 230. In some embodiments, the playback device 110 includes fewer or more components in a configuration different from that shown in Figure 2. For example, the playback device 110 may not include a microphone 220. In another example, the playback device 110 may include one or more additional input devices, such as a computer mouse and / or keyboard, that receive input from a user 135 of the playback device 110. As yet another example, the regeneration device 110 may include environmental sensors such as an ambient light sensor and / or a position tracking device (e.g., a Global Positioning System (GPS) receiver). In some embodiments, the regeneration device 110 performs functions other than those described below.

[0015] The first memory 210 may include read-only memory (ROM), random access memory (RAM), other non-temporary computer-readable media, or a combination thereof. The first electronic processor 205 is configured to receive instructions and data from the first memory 210 and, in particular, to execute instructions. Specifically, the first electronic processor 205 executes instructions stored in the first memory 210 to perform the method described herein.

[0016] The first network interface 215 transmits data to the media server 105 over the network 115 and receives data from the media server. In some embodiments, the first network interface 215 includes one or more transceivers for wireless communication with the media server 105 and / or the network 115. Alternatively or further, the first network interface 215 may include connectors or ports for receiving wired connections to the media server 105 and / or the network 115, such as Ethernet® cables. The first electronic processor 205 may receive one or more data streams (e.g., video streams, audio streams, image streams, etc.) over the network 115 through the first network interface 215. The first electronic processor 205 may output one or more data streams received from the media server 105 through the first network interface 215 through a speaker 225, a display 230, or a combination thereof. Furthermore, the first electronic processor 205 may communicate the data generated by the playback system 110 to the media server 105 on the network 115 via the first network interface 215. For example, the first electronic processor 205 may transmit a media request from the media server 105 based on the first electronic processor 205's determination of desired media parameters based on user input received in response to the display of a hybrid image on the display 230. The media server 105 may then transmit one or more media streams to the playback device 110 in accordance with the request / determination from the playback device 110. As another example, the first electronic processor 205 may transmit data indicating user input received in response to the display of a hybrid image on the display 230 for analysis by the media server 105. The media server 105 may determine the desired media parameters for the playback device 110 and user 135 based on the user input received in response to the display of a hybrid image on the display 230.Next, the media server 105 may transmit one or more media streams to the playback device 110 according to the determination of desired media parameters for the user 135 and the playback device 110 from the playback device 110.

[0017] The display 230 is configured to display images, videos, text, and / or data to the user 135. The display 230 may be a liquid crystal display (LCD) screen or an organic light-emitting display (OLED) screen. In some embodiments, a touch-sensitive input interface may also be incorporated into the display 230, enabling the user 135 to interact with the content provided on the display 230. In some embodiments, the display 230 includes a projector or a display technology to be developed in the future. In some embodiments, the speaker 225 and the display 230 are referred to as output devices that present media streams and other information to the user 135 of the playback device 110. In some embodiments, the microphone 220, computer mouse, and / or keyboard or touch-sensitive display are referred to as input devices that receive input from the user 135 of the playback device 110. In some embodiments, the input devices of the playback device 110 may also include sensors or devices configured to detect motion-based input (e.g., movement by the user 135). For example, such sensors or devices configured to detect motion-based input may include virtual reality (VR) / augmented reality (AR) controllers, handheld remotes / wands configured to detect movements caused by user 135, headphones with head tracking of user head movements, gaze detection sensors configured to determine where user 135's eyes are looking and / or where they are focused, and so on.

[0018] Figure 3 is a block diagram of a media server 105 according to one exemplary embodiment. In the illustrated example, the media server 105 includes a second electronic processor 305 electrically connected to a second memory 310 and a second network interface 315. These components are similar in name to the components of the playback device 110 described above with respect to Figure 2 and function in the same manner as described above. In some embodiments, the second network interface 315 transmits data to the playback device 110 via the network 115 and receives data from the playback device 110. In some embodiments, the media server 105 includes fewer or more components in a configuration different from that shown in Figure 3. For example, the media server 105 may further include a display, such as a touchscreen, to allow a backend user to reprogram the settings or rules of the media server 105. In some embodiments, the media server 105 performs functions other than those described below.

[0019] Figures 2 and 3 show separate block diagrams of the playback device 110 and the media server 105. In some embodiments, the media server 105, one or more playback devices 110, a remote cloud computing cluster communicating on or forming part of the network 115, or a combination thereof, is referred to as an electronic computing device that performs the functions described herein. For example, the electronic computing device may include a single electronic processor (e.g., a second electronic processor 305 in the media server 105 or a first electronic processor 205 in the playback device 110) or multiple electronic processors located in the media server 105. In other embodiments, the electronic computing device may include multiple electronic processors distributed among different devices. For example, the electronic computing device may be implemented in one or more of the first electronic processor 205 in the playback device 110, the second electronic processor 305 in the media server 105, and one or more electronic processors located in one or more other devices located at remote locations or in a remote cloud computing cluster communicating on or forming part of the network 115. In some embodiments, the remote cloud computing cluster includes a software-defined network (SDN) / Network Function Virtualization (NFV) enabled access network.

[0020] In this specification, the method / operation is described as being performed primarily by the regeneration device 110 (in particular, the first electronic processor 205). However, it should be understood that in some embodiments, one or more of the methods / operations described herein may be performed, either further or alternatively, by other devices (e.g., any single device or combination of devices that may constitute the electronic computing device described above).

[0021] As described in PCT / US2020 / 044241, a hybrid image is a still image generated from at least two distinct source images. Hybrid images tend to have distinct interpretations depending on the user's viewing ability and environmental factors. For example, as the viewing distance increases, human viewers lose the ability to see fine details in an image, and as a result, they become unable to distinguish between high-resolution and low-resolution videos. In some embodiments, a hybrid image is a still image that generates for a human user two or more distinct interpretations (e.g., a first interpretation dominated by / based on a first source image and a second interpretation dominated by / based on a second source image) that vary as a function of spatial frequency range and / or viewing distance. Based on the user's response to the hybrid image displayed by the playback device 110, the playback device 110 may estimate the dominant and non-dominant spatial frequency ranges of the user 135 in the media viewing environment 130 without using explicit sensors.

[0022] Furthermore, as described in PCT / US2020 / 044241, in order to create a hybrid image, two different source images may be processed differently so that a specific spatial frequency range is dominant with respect to each processed image included in the hybrid image. For example, the first source image may be low-pass filtered and the second source image may be high-pass filtered. The low-pass filtered source image may then be combined with (e.g., overlaid on) the high-pass filtered source image to create a hybrid image. Since the sensitive region of a given image in spatial frequency shifts from lower frequencies to higher frequencies as the user's viewing distance decreases, a human user perceives the high-pass filtered source image more easily at shorter viewing distances than at longer viewing distances. Conversely, a human user perceives the low-pass filtered source image more easily at longer viewing distances than at shorter viewing distances. In other words, either a low-pass filtered source image or a high-pass filtered source image may be perceived as dominant by user 135, depending on one or more of user 135's viewing characteristics and / or user 135's environment 130.

[0023] Throughout this disclosure, the generation of one or more hybrid images is referred to. In some embodiments, one or more of the hybrid images are generated by an electronic computing device by overlaying a source image as described herein. In some embodiments, the electronic computing device may select and / or receive previously generated and stored hybrid images having characteristics corresponding to desired browsing / test parameter values ​​as described herein. For example, playback device 110 may select, retrieve and / or receive stored hybrid images from media server 105 and / or from another device outside playback device 110.

[0024] Figures 4A and 4B show exemplary source images used to generate the exemplary hybrid images shown in Figures 4C to 4E. In the illustrated examples, Figure 4A is source image A of a woman, and Figure 4B is source image B of a man. Each of the three hybrid images 405, 410, and 415 in Figures 4C to 4E may be generated by adding a source image A (e.g., the woman's image) that has been low-pass filtered and a source image B (e.g., the man's image) that has been high-pass filtered using different cutoff frequencies used for filtering in each hybrid image 405, 410, and 415. For example, the cutoff frequencies used by the low-pass and high-pass filters to generate hybrid image 405 in Figure 4C are lower than the cutoff frequencies used by the low-pass and high-pass filters to generate hybrid images 410 and 415 in Figures 4D and 4E. The cutoff frequencies used by the low-pass and high-pass filters to generate the hybrid image 410 in Figure 4D are between the cutoff frequencies used by the low-pass and high-pass filters to generate the hybrid images 405 and 415 in Figures 4C and 4E. The cutoff frequencies used by the low-pass and high-pass filters to generate the hybrid image 415 in Figure 4E are higher than the cutoff frequencies used by the low-pass and high-pass filters to generate the hybrid images 405 and 410 in Figures 4C and 4D. Therefore, as shown in Figures 4C to 4E, the perception (e.g., dominant interpretation) of the hybrid images 405, 410, and 415 transitions from male to female as the cutoff frequency increases (and / or as the user 135's viewing distance for a given hybrid image increases).

[0025] A technical challenge in generating hybrid images is that producing a useful hybrid image from two source images may not be achievable simply by combining either of the two source images. Rather, when generating a hybrid image, multiple factors of the source images may be considered to ensure that each of the two perceptions / interpretations associated with the hybrid image is visible in at least some viewing situations. For example, perceptual grouping modifies the effectiveness of the hybrid image, as the visual system groups ambiguous blobs at low spatial frequencies to form meaningful interpretations. According to Gestalt's rules of perception, the human eye can perceive a set of individual elements as a whole. Therefore, in a hybrid image, non-dominant image interpretations should be perceived as noise to the dominant image rather than forming an independent image perception. As another example, one way to reduce the influence of one spatial channel of one source image on the other spatial channel of the other source image is to generate the hybrid image so that it has alignment of edges and blobs contained in the two source images. As yet another example, low-pass and high-pass filters used to filter source images should not have significant overlap to avoid ambiguous interpretations between the two source images.

[0026] Furthermore, another technical issue is that it may not be possible in all viewing contexts to generate a hybrid image that is useful for evaluating the user's perception of the hybrid image with respect to relevant values ​​of media parameters used to control the delivery of visual media to the playback device 110 on the network 115, simply by selecting the cutoff frequencies of the low-pass and high-pass filters used to filter the source image to be approximately equal to the Nyquist frequencies of the available video resolutions of the media streaming application (e.g., 360p, 540p, 720p, and 1080p on a 1080p display). For example, as shown in Figures 6-9, such a hybrid image may be largely dominated by the low-pass filtered image (e.g., low-pass filtered source image A of a woman) unless the high-pass filtered image retains enough information to form a multiscale image perception. In other words, selecting any arbitrary cutoff frequency when generating a hybrid image does not always produce a useful hybrid image in which the human eye can perceive each of the two interpretations in the hybrid image, at least in some viewing situations, due to the low-pass spectral characteristics of the natural image.

[0027] Figure 5 shows examples of azimuthal-averaged 1-d power spectra along the radius from the origin for source images A and B in Figures 4A and 4B. Curve 505 represents source image A (female). Curve 510 represents source image B (male). The y-axis represents power in decibels (dB), and the x-axis represents frequency in cycles per pixel (cpp). Figure 5 illustrates the above technical problem of generating hybrid images from natural images, as the natural images in Figures 4A and 4B are dominated by these low-frequency range spectra.

[0028] Figures 6–9 show exemplary images and graphs associated with hybrid images generated using cutoff frequencies equal to the video resolution of a hypothetical video streaming application (e.g., 0.17, 0.2, 0.25, and 0.33 cycles / pixel (cpp) corresponding to the Nyquist frequencies of 360p, 432p, 540p, and 720p video on a 1080p display, respectively, according to Equation 2 below). Figures 6A, 7A, 8A, and 9A show exemplary filter responses of 6th-order Butterworth slow-pass filters (605A, 705A, 805A, 905A) and high-pass filters (610A, 710A, 810A, 910A), respectively, used to filter source image A in Figure 4A and source image B in Figure 4B based on the above cutoff frequencies.

[0029] Figures 6B, 7B, 8B, and 9B show the low-pass filtered images of source image A from Figure 4A. Figures 6C, 7C, 8C, and 9C show the high-pass filtered images of source image B from Figure 4B. Figures 6D, 7D, 8D, and 9D show hybrid images created by adding the filtered source image A from Figures 6B, 7B, 8B, and 9B to the respective filtered source image B from Figures 6C, 7C, 8C, and 9C. As shown in Figures 6D, 7D, 8D, and 9D, unlike Figures 4C-4E, all of the hybrid images in Figures 6D, 7D, 8D, and 9D appear to be perceived as low-pass filtered source image A (female) because the cutoff frequency of the high-pass filter removes components that are important to the image perception of source image B (male). Figures 6C, 7C, 8C, and 9C further emphasize this removal of components from source image B, to the point that the high-pass filtered source image B is either invisible or barely visible in Figures 6C, 7C, 8C, and 9C.

[0030] Figures 6E, 7E, 8E, and 9E show examples of azimuthal-averaged 1-d power spectra along the radius from the origin for low-pass filtered source images A (605E, 705E, 805E, 905E) in Figures 6B, 7B, 8B, and 9B, and high-pass filtered source images B (610E, 710E, 810E, 910E) in Figures 6C, 7C, 8C, and 9C, respectively. Figures 6E, 7E, 8E, and 9E reveal the degree of spectral energy imbalance between low-pass filtered source image A and high-pass filtered source image B through the 1-d power spectra. As shown in Figures 6F, 7F, 8F, and 9F, even when low-pass filtered source image A and high-pass filtered source image B are equalized in energy, the resulting hybrid images in Figures 6G, 7G, 8G, and 9G do not create a satisfactory perception / interpretation of a multiscale image. In other words, as shown by the graphs in Figures 6E, 7E, 8E, and 9E, the power of the two filtered source images in Figures 6-9 varies greatly with respect to each other, depending on spatial frequency, even when the low-pass filtered and high-pass filtered images are equalized in energy, as indicated by curves 605F, 705F, 805F, 905F and 610F, 710F, 810F, 910F, respectively, in Figures 6F, 7F, 8F, and 9F. This large spatial frequency-dependent variability in the power of the source images relative to each other makes the hybrid image less useful in many viewing situations, as the high-pass filtered image is not perceptible to the human eye.

[0031] As shown in Figures 6-9, using the Nyquist frequency corresponding to a typical video resolution in a media streaming application as the filter cutoff frequency for source images A and B results in a cutoff frequency that is too high, making it impossible to retain the perception / interpretation of the high-pass filtered source image B due to excessive loss of low-frequency components (see, for example, Figures 6C, 7C, 8C, and 9C). To address this technical problem and generate hybrid images that are more useful for evaluating the user's perception of the hybrid image with respect to relevant media parameters (e.g., the Nyquist frequency corresponding to a typical video resolution in a media streaming application), the playback device 110 may utilize the frequency scaling properties of the image spectrum as shown in Figure 10.

[0032] Graph 1005 above Figure 10 shows that a quarter of the Nyquist frequency value corresponding to a typical video resolution in a media streaming application may be a quarter of the desired value (f c Cutoff frequency (f) at / 4 c The spectral distribution of a hybrid image consisting of images A and B filtered using ) is shown. In some embodiments, one-quarter of the desired cutoff frequency value is low enough, with appropriate gain adjustment, to allow perception / interpretation of the hybrid image dominated by the high-pass filtered image B. In other words, reducing the cutoff frequency to one-quarter of the desired value may preserve the perception / interpretation of the high-pass filtered source image B so that the high-pass filtered source image B appears to user 135 under some viewing conditions. As shown in graphs 1010 and 1015 of Figure 10, the frequency scale of the spectrum expands as the image size is reduced. For example, when the hybrid image is displayed at half its original size (e.g., using half the number of pixels along each of the length and width of the hybrid image), the frequency scale of the hybrid image expands from one-quarter of the desired cutoff frequency value to half of the desired cutoff frequency value (f c / 2) is extended to. Similarly, when the hybrid image is displayed at a quarter of its original size (e.g., using a quarter of the amount of pixels along each of the length and width of the hybrid image), the frequency scale of the hybrid image is from a quarter of the value of the desired cut-off frequency to the value of the desired cut-off frequency (f c ) is extended to.

[0033] Using the above characteristics of the frequency scaling of the image spectrum shown in FIG. 10, the playback device 110 can ensure that each of the source images A and B is perceptible to the human eye under some viewing conditions, and the effective cut-off frequencies of the filtered source images A and B correspond to relevant values of the media parameters (e.g., the Nyquist frequency corresponding to a typical video resolution in a media streaming application). Next, based on user input corresponding to the displayed hybrid image, the playback device 110 can evaluate the visibility of frequency content above an arbitrary test frequency (f test )(e.g., the Nyquist frequency corresponding to a typical video resolution in a media streaming application) that corresponds to a relevant value of the media parameter. In some embodiments, Equation 1 (below) may be used to set an arbitrary test frequency (f test ). Equation 1: f test =(F c S) / S’

[0034] In some embodiments, f test represents the test frequency at which the hybrid image is configured to test whether the user 135 can perceive a frequency difference (e.g., a change in perceived quality of experience (QoE)) above the test frequency. In some embodiments, the test frequency is the initial cut-off frequency (f cThis is equivalent to ). For example, when S=S', the test frequency is equal to the initial cutoff frequency. In other words, in some embodiments, the first scaling factor and the second scaling factor may be the same (see Figures 14A-14B) or different (see Figures 12A-12B and 13A-13B).

[0035] In some embodiments, f c ∫ represents the initial cutoff frequency of the low-pass filter used to filter source image A, and the initial cutoff frequency of the high-pass filter used to filter source image B. The initial cutoff frequencies may correspond to relevant values ​​of media parameters (e.g., Nyquist frequencies corresponding to typical video resolutions in media streaming applications). For example, the initial cutoff frequencies may be 0.17, 0.2, 0.25, or 0.33 cycles / pixel (cpp), which correspond to the Nyquist frequencies of 360p, 432p, 540p, and 720p video on a 1080p display, respectively, according to Equation 2 below. Formula 2:f cpp =(0.5*f Nyquist ) / 1080

[0036] As described above in this specification, using an initial cutoff frequency corresponding to the relevant value of the media parameter to generate a hybrid image may not result in a hybrid image that includes the perception / interpretation of both the human-visible source images A and B. Therefore, in some embodiments, a first scaling coefficient S (e.g., a first coefficient) is used to scale the initial cutoff frequency to a scaled cutoff frequency (f cIt is used to scale to S). For example, the first coefficient S may be chosen such that the perception / interpretation of both filtered source images A and B is perceptible to the human eye under at least some viewing conditions. In some embodiments, the first coefficient S is less than 1 to reduce the initial cutoff frequency of the filter used to generate the hybrid image, so that the filter's cutoff frequency is low enough that the perception / interpretation of the high-pass filtered source image B is perceptible to the human eye under at least some viewing conditions. In some embodiments, the scaled cutoff frequencies of the low-pass and high-pass filters are the same. In some embodiments, the scaled cutoff frequencies of the low-pass and high-pass filters are different and may be separated by a separation value (df). In some embodiments, the frequency separation of the filters used to generate the hybrid image is the first scaled cutoff frequency f of the low-pass filter. c S-df and the second scaled frequency f of the high-pass filter c This can also be adjusted by creating S+df.

[0037] In some embodiments, S' represents a second scaling factor (e.g., a second factor) used to scale the size of the hybrid image configured to be displayed on the display 230 of the playback device 110. In some embodiments, scaling the size of the hybrid image changes the number of pixels along the length and width of the hybrid image, respectively, used by the display 230 to display the hybrid image. For example, a second scaling factor S' of 0.50 may reduce the display of both the length and width of the hybrid image to half the number of pixels that the hybrid image would otherwise have been displayed (see, for example, the size difference between Figure 12A and Figure 13A).

[0038] Using Equation 1, the scaled cutoff frequencies of the low-pass and high-pass filters used to filter source images A and B, respectively, may be set low enough to control the interpretation shown in the hybrid image (in particular, the interpretation of the high-pass filtered source image B). Furthermore, by changing the values ​​of the variables in Equation 1, the same hybrid image can be used to evaluate the visual ability of user 135 above any test frequency by simply resizing the hybrid image by a second scaling factor S' according to the desired test frequency to be tested.

[0039] Figure 11 shows the test frequency (f) calculated according to Equation 1. test To evaluate the visual capabilities of user 135 that exceed those of user 135, a flowchart 1100 is shown for generating an improved hybrid image to display to user 135. As described above in this specification, among other devices, the first electronic processor 205 of the playback device 110 may implement the functions shown in Figure 11.

[0040] As shown in the example in Figure 11, the source image A (female) in Figure 4A has a scaled cutoff frequency (f c The first image 1120 is low-pass filtered by the low-pass filter 1105 using S) to produce a low-pass filtered first image 1120. In other words, the first electronic processor 205 sets the first cutoff frequency of the low-pass filter 1105 used to filter the first image to the first scaled cutoff frequency (f) c The first cutoff frequency (f) may be configured to scale by a first coefficient (S). c ) may be selected based on a first and / or second value of media parameters corresponding to interpretations associated with hybrid images that can be perceived by the human eye under at least some viewing conditions (for example, based on the Nyquist frequency corresponding to a typical video resolution in a media streaming application).

[0041] Similarly, source image B in Figure 4B shows the scaled cutoff frequency (f c The second image 1125 is high-pass filtered by the high-pass filter 1110 using S) to produce a high-pass filtered second image 1125. In other words, the first electronic processor 205 sets the second cutoff frequency of the high-pass filter 1110 used to filter the second image to a second scaled cutoff frequency (f) c The first cutoff frequency (f) may be configured to scale by a first coefficient (S). As described above in this specification, in some embodiments, the first cutoff frequency and the second cutoff frequency may be equivalent or different from each other. Similar to the first cutoff frequency, the second cutoff frequency (f) may be configured to scale by a first coefficient (S). c ) may be selected based on a first and / or second value of a media parameter, each corresponding to an interpretation associated with a hybrid image that can be perceived by the human eye under at least some viewing conditions.

[0042] In some embodiments, in block 1115, the gain (g) of the high-pass filtered source image B is adjusted (e.g., increased) to control the desired perception / interpretation of the high-pass filtered source image B in the improved hybrid image. For example, the gain (g) may be increased to make the high-pass filtered source image B more visible to the human eye. Although not shown in Figure 11, in some embodiments, the gain of the low-pass filtered source image A may be adjusted, or alternatively, to control the desired perception / interpretation of the low-pass filtered source image A in the improved hybrid image.

[0043] In some embodiments, the scaled cutoff frequency (f c The value of S) is the cutoff frequency (f c) is the Nyquist frequency of one of the available video resolutions that is to be tested (e.g., 720p on a 1080p display, e.g., f according to Equation 2). c The scaling factor (S) is determined by setting it to be the same as =720 / 2*1080=0.333cpp). In some embodiments, the first scaling factor (S) is determined by adjusting the gain (g) of the high-pass filter used to filter the source image B, thereby scaling the cutoff frequency (f c S) may be empirically determined to be low enough to control the perception / interpretation of the hybrid image between source image A and source image B (for example, to control which of source image A or B is predominantly visible to the human eye in the hybrid image).

[0044] In block 1120, the low-pass filtered source image A(1120) and the high-pass filtered source image B(1125) are combined (e.g., overlaid on each other) to produce a scaled cutoff frequency filtered hybrid image 1135. In some embodiments, the scaled cutoff frequency filtered hybrid image 1135 includes a first interpretation provided by the low-pass filtered first image 1120, which is visible to the human eye under at least some viewing conditions, and a second interpretation provided by the high-pass filtered second image 1125, which is visible to the human eye under at least some viewing conditions.

[0045] In block 1140, the size of the scaled cutoff frequency filtered hybrid image 1135, configured to be displayed on the display 230 of the playback device 110, is scaled to a size scaled by a second scaling factor (S'). Scaling the size of the scaled cutoff frequency filtered hybrid image 1135 resizes the scaled cutoff frequency filtered hybrid image and the desired test frequency (f test ) generates an improved hybrid image designed to test the visual abilities of user 135. As described above in this specification, in some embodiments scaling the size of the scaled cutoff frequency filtered hybrid image 1135 scales the number of pixels along the length and width of the scaled cutoff frequency filtered hybrid image 1135, respectively, which are used by the display 230 to display the scaled cutoff frequency filtered hybrid image 1135.

[0046] Figures 12A, 13A, and 14A show three different exemplary improved hybrid images generated according to the flow diagram 1100 of Figure 11. Correspondingly, Figures 12B, 13B, and 14B show the spectra of the low-pass filtered image 1205, 1305, 1405 and the high-pass filtered image 1210, 1310, 1410, which are combined to create the respective improved hybrid images shown in Figures 12A, 13A, and 14A. In each of Figures 12A, 13A, and 14A, the initial cutoff frequency (f c) is set to 0.33 cycles / pixel (cpp), corresponding to a 720p video resolution on a 1080p display, based on equation 2 shown above: (0.5*720) / 1080 = 0.333cpp. In Figures 12A, 13A, and 14A, the first scaling factor (S) is set to 0.25. Thus, the scaled cutoff frequency (f) in Figures 12A, 13A, and 14A is c S) is approximately 0.0833. In the example shown in Figures 12A to 14C, the energies of the two filtered images are equalized.

[0047] In Figures 12A, 13A, and 14A, the size of the improved hybrid image is scaled differently, as indicated by the number of pixels along the length and width of each improved hybrid image. In other words, the second scaling factor (S') is different in each of Figures 12A, 13A, and 14A. For example, the second scaling factor is 1.0 in Figure 12A, which means that the size of the improved hybrid image is not adjusted after filtering and combining source images A and B. As another example, the second scaling factor is 0.50 in Figure 13A, which means that the size of the improved hybrid image is reduced by 50% after filtering and combining source images A and B. As yet another example, the second scaling factor is 0.25 in Figure 14A, which means that the size of the improved hybrid image is reduced by 25% after filtering and combining source images A and B. Using Equation 1, the test frequency (f) for each of the improved hybrid images in Figures 12A, 13A, and 14A is calculated. testThe test frequency may be determined as shown in Figures 12B, 13B, and 14B. In Figure 12A, the test frequency is 0.083, which corresponds to 180p on a 1080p display according to Equation 2: (0.5*180) / 1080 = 0.083cpp. In Figure 13A, the test frequency is 0.167, which corresponds to 360p on a 1080p display according to Equation 2: (0.5*360) / 1080 = 0.167cpp. In Figure 14A, the test frequency is 0.333, which corresponds to 720p on a 1080p display according to Equation 2: (0.5*720) / 1080 = 0.333cpp. The test frequencies for each hybrid image in Figures 12A, 13A, and 14A are represented by the vertical dashed lines 1215, 1315, and 1415 in Figures 12B, 13B, and 14B, respectively.

[0048] Each of the improved hybrid images of different sizes in Figures 12A, 13A, and 14A shows its respective test frequency (f testIt may be used to evaluate the visual ability of user 135 at a spatial frequency higher than that corresponding to ). For example, the improved hybrid image in Figure 14A may be used to evaluate the visual ability of user 135 at a level higher than 0.333 cpp (corresponding to 720p video resolution on a 1080p display). In other words, the improved hybrid image in Figure 14A may be used to determine whether user 135's eyes are able to distinguish between 720p video resolution and a higher video resolution (e.g., 1080p video resolution). For user 135 sitting at a 3H viewing distance (e.g., a viewing distance of three times the height of the display) on a 1080p display and able to see 30 cycles per degree, the visual resolution ability is approximately 1146 pixels at one picture height, and there should be a gain that allows such user 135 to perceive the improved hybrid image as an interpretation corresponding to the high-pass filtered image B (male). In other words, if user 135 can perceive the high-pass filtered image B (male) in the improved hybrid image shown in Figure 14A, then user 135 can distinguish between 720p video resolution and higher video resolutions. On the other hand, if user 135 cannot perceive the high-pass filtered image B (male) in the improved hybrid image shown in Figure 14A, then user 135 cannot distinguish between 720p video resolution and higher video resolutions. In the latter situation, since user 135 cannot perceive the difference between 720p video resolution and 1080p video resolution, presenting user 135 with media at a higher resolution than 720p (e.g., 1080p video resolution) will not result in improved quality of experience (QoE).

[0049] In some embodiments, the first electronic processor 205 is configured to display on the display 230 of the playback device 110 a plurality of first improved hybrid images, each containing at least one of (i) a high-pass filtered second image with different gain values ​​and (ii) a different size. The first user input received by the playback device 110 can then indicate whether the user 135 perceives a second interpretation of the high-pass filtered second image B (male) in one or more of the first plurality of improved hybrid images. In some embodiments, the first user input is a series / sequence of one or more user inputs (e.g., user inputs relating to each user's perception of the improved hybrid images displayed on the display 230). In some embodiments, the second user input is a series / sequence of one or more user inputs (e.g., user inputs relating to each user's perception of further improved hybrid images displayed on the display 230). The difference (i) and / or (ii) between different displayed improved hybrid images may be associated with different test frequencies for each improved hybrid image. Therefore, based on the first user input received with respect to the first multiple improved hybrid images (and / or further user input with respect to further improved hybrid images), the first electronic processor 205 may determine the details of the visual capabilities of the user 135 in the browsing environment 130.

[0050] As examples of improved hybrid images within a first set of improved hybrid images having different sizes, the improved hybrid images in Figures 12A, 13A, and 14A may be displayed on display 230 to determine whether user 135 can perceive spatial frequencies above 0.083 cpp (180p video resolution on a 1080p display), 0.167 (360p video resolution on a 1080p display), and 0.333 cpp (720p video resolution on a 1080p display), respectively. If user 135 can perceive the high-pass filtered image B (male) for any of these improved hybrid images, then user 135 can perceive spatial frequencies above the test frequency of each improved hybrid image.

[0051] As an example of improved hybrid images within a first set of improved hybrid images having different gain values ​​for the second high-pass filtered image B, the first set of improved hybrid images may include improved hybrid images that are identical except for the different gain values ​​(g) of the high-pass filtered image B. For each subsequent improved hybrid image within the set of improved hybrid images, the first electronic processor 205 may increase the gain value (g) of the high-pass filtered image B included in the improved hybrid image. For example, Figures 15 and 16 show five improved hybrid images, each with increasing gain values ​​for the high-pass filtered image B from left to right. Thus, the high-pass filtered image B (male) is not the most dominant in the leftmost improved hybrid image in each of Figures 15 and 16, but is the most dominant in the rightmost improved hybrid image in each of Figures 15 and 16. If any of the improved hybrid images shown in Figures 15 and 16 respectively contain a gain value that allows user 135 to perceive a second interpretation of the high-pass filtered image B, the first electronic processor 205 determines that user 135 can perceive spatial frequencies above the test frequency of the first multiple improved hybrid images. On the other hand, if user 135 cannot perceive a second interpretation of the high-pass filtered image B in any of the improved hybrid images shown in Figures 15 and 16 respectively (for example, even in the rightmost improved hybrid image where the gain of the high-pass filtered image B is highest), the first electronic processor 205 determines that user 135 cannot perceive spatial frequencies above the test frequency of the first multiple improved hybrid images. In some embodiments, the improved hybrid image in Figure 15 has a test frequency of 540p (f test ) may be generated to test a second scaling factor (S') of 0.5, while the improved hybrid image in Figure 16 is a test frequency (f) of 720p. test ) may be generated to test, and a second scaling factor (S') of 0.25 may be used.

[0052] In some embodiments, each improved hybrid image within a plurality of improved hybrid images is displayed simultaneously on the display 230. For example, as shown in Figures 15 and 16, the plurality of improved hybrid images are displayed in a row simultaneously. In such embodiments, the first electronic processor 205 may also control the display 230 to display a command indicating that the user 135 should select (e.g., using an input device) each improved hybrid image in which the high-pass filtered image B (male) is visible to the user 135. In some embodiments, the plurality of improved hybrid images may be displayed sequentially. In such embodiments, the display 230 may again display a command indicating that the user 135 should indicate (e.g., using an input device) whether or not the high-pass filtered image B is visible to the user 135 in each improved hybrid image being displayed. In some embodiments, the display 230 may begin by displaying the first improved hybrid image (e.g., the leftmost image in the plurality of improved hybrid images shown in Figures 15 and 16). In some embodiments, the display 230 may also display instructions indicating that user 135 should adjust the improved hybrid image (e.g., using an input device) until the high-pass filtered image B is visible to user 135.

[0053] In some embodiments, the first electronic processor 205 may adjust the displayed improved hybrid image in response to user input received via the input device of the playback device 110, thereby changing the size of the improved hybrid image and / or the gain value of the high-pass filtered image B. For example, the display 230 may include a slider bar that the user 135 can control to control how the first electronic processor 205 controls the display / generation of the improved hybrid image. In some embodiments, the first electronic processor 205 may gradually and automatically adjust the displayed improved hybrid image until the playback device 110 receives user input indicating that the user 135 perceives the high-pass filtered image B, or until the playback device 110 receives user input indicating that the user 135 does not perceive the high-pass filtered image B at any point during the adjustment of the displayed improved hybrid image. In such embodiments, the adjusted parameters of the displayed improved hybrid image may be reset after the adjusted parameters have been adjusted to a predetermined limit. For example, after Figure 26 and the rightmost improved hybrid image in Figure 26 are displayed, the displayed improved hybrid image may be reset to display the leftmost improved hybrid image in Figures 15 and 16. While multiple improved hybrid images are being displayed, the display 230 may display an indicator of how one or more parameters of the improved hybrid image have been adjusted.

[0054] Based on the user 135's visual ability to view a first plurality of improved hybrid images displayed on the display 230, as determined by the first electronic processor 205 in response to a first user input received by the playback device 110, the first electronic processor 205 may generate / select a second plurality of hybrid images to narrow down to the estimated minimum and maximum QoE resolution of the user 135 in the environment 130. In some embodiments, in response to first user input indicating that the first user perceives a second interpretation of a second image B that has been high-pass filtered within one or more of the first plurality of improved hybrid images, the first electronic processor 205 is configured to display a second plurality of improved hybrid images on the display of the playback device 110, each being smaller or larger in size than each of the first plurality of improved hybrid images. For example, a scaled cutoff frequency (f c The value of S) is the cutoff frequency (f c ) may be determined by setting it to be the same as the Nyquist frequency of the second highest available video resolution (e.g., 720p on a 1080p display, e.g., fc = 720 / 2 * 1080 = 0.333 cpp according to Equation 2). In some embodiments, the first scaling factor (S) is the scaled cutoff frequency (f c S) is empirically determined to be low enough (for example, as shown in Figure 15) to allow control over the perception / interpretation of the hybrid image between source image A and source image B through the adjustment of the gain (g) of the high-pass filter used to filter source image B.

[0055] In some embodiments, the available video resolutions for the media streaming application may be 360p, 540p, 720p, and 1080p on a 1080p display. Continuing the above example, the first electronic processor 205 may begin by testing the user's visual ability at 540p. In other words, the test frequency (f) of the first multiple improved hybrid images displayed on the display 230. test ) is 540p (for example, according to equation 2, f cpp =(0.5*540) / 1080=0.25cpp). Therefore, the second scaling factor (S') is S'=(f c Science fiction test =720S / 540 (where the first scaling factor (S) is determined empirically in advance as described above). As described above in this specification, the first multiple improved hybrid images may be displayed with an increasing gain value (g) of the high-pass filtered second image B until user input is received indicating that user 135 perceives the high-pass filtered second image B, or until user input is received indicating that user 135 does not perceive the high-pass filtered second image B (male) in any of the first multiple improved hybrid images.

[0056] In response to a first user input indicating that the first user perceives a second interpretation of a second image B that has been high-pass filtered within one or more of the first multiple improved hybrid images, the first electronic processor 205 is configured to display the second multiple improved hybrid images on the display of the playback device 110, each of which is smaller in size than each of the first multiple improved hybrid images. In other words, since there is a gain value that causes user 135 to perceive / interpret at least one of the first multiple improved hybrid images as the high-pass filtered second image B (male), the first electronic processor 205 determines that the minimum and maximum QoE resolution of user 135 in the environment 130 is higher than the test frequency (e.g., 540p in this example).

[0057] Therefore, the first electronic processor 205 generates a second multiple improved hybrid image to have a test frequency with a higher available video resolution (e.g., 720p). In some embodiments, the first electronic processor 205 sets the second scaling coefficient (S') according to Equation 1 as S'=(f c Science fiction test The size of the second plurality of hybrid images is adjusted by adjusting =720S / 720 (for example, making the second plurality of images smaller compared to the first plurality of images). The first electronic processor 205 may then repeat the display and test process to determine whether the minimum and maximum QoE resolution of user 135 in the environment 130 is higher than the second test frequency (e.g., 720p in this example). In some embodiments, the first electronic processor 205 is configured to receive a second user input from the first user 135. The second user input may indicate whether the first user 135 perceives a second interpretation of the second image B that has been high-pass filtered in one or more of the second plurality of improved hybrid images. In some embodiments, the first electronic processor 205 is configured to determine an optimized value for a media parameter (e.g., the minimum and maximum QoE resolution of user 135) based at least in part on the second user input.

[0058] For example, in response to a second user input indicating that the first user 135 perceives a second interpretation of the second image B that has been high-pass filtered within one or more of the second multiple improved hybrid images, the first electronic processor 205 may determine that the minimum and maximum QoE resolution for user 135 in environment 130 is higher than the second test frequency (e.g., 720p in this example). Thus, the first electronic processor 205 may set the video resolution of the media displayed on the playback device 110 to 1080p. On the other hand, in response to a second user input indicating that the first user 135 does not perceive a second interpretation of the second image B that has been high-pass filtered within one or more of the second multiple improved hybrid images, the first electronic processor 205 may determine that the minimum and maximum QoE resolution for user 135 in environment 130 is lower than the second test frequency (e.g., 720p in this example). Since the first electronic processor 205 has already determined that the minimum and maximum QoE resolution of the user 135 in the environment 130 is greater than 540p, the first electronic processor 205 may set the video resolution of the media displayed on the playback device 110 to 720p.

[0059] Returning to the initial video resolution test of 720p, in response to a first user input indicating that the first user does not perceive a second interpretation of the second image B that has been high-pass filtered within one or more of the first multiple improved hybrid images, the first electronic processor 205 is configured to display a third multiple improved hybrid image on the display of the playback device 110, each being larger in size than each of the first multiple improved hybrid images. In other words, since there is no gain value that causes user 135 to perceive at least one of the first multiple improved hybrid images as the second image B (male) that has been high-pass filtered, the first electronic processor 205 determines that the minimum and maximum QoE resolution for user 135 in the environment 130 is lower than the test frequency (e.g., 540p in this example).

[0060] Therefore, the first electronic processor 205 generates a third multiple improved hybrid image to have a test frequency for the next lowest available video resolution (e.g., 360p). In some embodiments, the first electronic processor 205 sets the second scaling coefficient (S') according to Equation 1 as S'=(f c Science fiction test The size of the third plurality of hybrid images is adjusted by adjusting to =720S / 360 (for example, making the third plurality of images larger compared to the first plurality of images). The first electronic processor 205 may then repeat the display and test process to determine whether the minimum and maximum QoE resolution of user 135 in the environment 130 is higher than the third test frequency (for example, 360p in this example). In some embodiments, the first electronic processor 205 is configured to receive a third user input from the first user 135. The third user input may indicate whether the first user 135 perceives a second interpretation of the second image B that has been high-pass filtered in one or more of the third plurality of improved hybrid images. In some embodiments, the first electronic processor 205 is configured to determine an optimized value for a media parameter (for example, the minimum and maximum QoE resolution of user 135) based at least in part on the third user input.

[0061] For example, in response to a third user input indicating that a first user 135 perceives a second interpretation of a second image B that has been high-pass filtered within one or more of the third set of improved hybrid images, the first electronic processor 205 may determine that the minimum / maximum QoE resolution of user 135 in the environment 130 is higher than the third test frequency (e.g., 360p in this example). Since the first electronic processor 205 has previously determined that the minimum / maximum QoE resolution of user 135 is not greater than 540p, the first electronic processor 205 may set the video resolution of the media displayed on the playback device 110 to 540p. On the other hand, in response to a third user input indicating that the first user 135 does not perceive a second interpretation of the second image B that has been high-pass filtered within one or more of the third set of improved hybrid images, the first electronic processor 205 may determine that the minimum and maximum QoE resolution of user 135 in the environment 130 is less than the third test frequency (e.g., 360p in this example). Therefore, since user 135 cannot perceive the difference between 360p video and video displayed at a higher resolution, the first electronic processor 205 may set the video resolution of the media displayed on the playback device 110 to 360p (e.g., the minimum video resolution of the display 230).

[0062] As illustrated by the above example, the first electronic processor 205 may determine optimized values ​​for media parameters for streaming output media over network 115 and / or displaying output media on playback device 110. For example, the optimized value for media parameters may be the minimum resolution for maximum perceived quality (minimum-maximum QoE resolution) personalized for the first user 135 based at least partially on the first user input, as described above herein. As another example, the optimized value for media parameters may be the value of an estimated perceived quality (QoE) transfer function personalized for the first user 135 based at least partially on the first user input, as described below. As yet another example, the optimized value for media parameters may be the bitrate or frame rate of media streaming from media server 105 on network 115, based on maximum-minimum QoE resolution or QoE transfer function. The optimized value for media parameters may also be the value of other media parameters based on minimum-maximum QoE resolution or QoE transfer function.

[0063] Figure 17 shows a flowchart of Method 1700, which may be performed by an electronic computing device to capture personalized playback-side context information of a user 135 in an environment 130 and generate a hybrid image for use when providing output media to a playback device according to the personalized playback-side context information, according to an exemplary embodiment. Method 1700 is a generalized method that may represent one or more of the specific exemplary implementations of the improved hybrid image generation described above herein. Method 1700 may be performed by one or more electronic processors of an electronic computing device. As described above herein, the electronic computing device may be a single device or a combination of multiple devices (e.g., media server 105, playback device 110, etc.).

[0064] In block 1705, one or more electronic processors of an electronic computing device perform at least one of generating and selecting a hybrid image associated with a first interpretation corresponding to a first value of a media parameter and a second interpretation corresponding to a second value of a media parameter. In some embodiments, the hybrid image includes a first visibility ratio between the first and second interpretations. As described above herein, the first interpretation may be generated based on a low-pass filtered first source image A (see, for example, Figure 4A). The second interpretation may be generated based on a high-pass filtered second source image B (see, for example, Figure 4B). While one or more electronic processors may generate a hybrid image, one or more electronic processors may further or alternatively select a previously generated hybrid image.

[0065] In some embodiments, the first interpretation may correspond to a first value of a media parameter, and as a result, if user 135 has the visual ability in the environment 130 corresponding to the first value of the media parameter, the first interpretation of the low-pass filtered first image A will be visible to user 135. Similarly, the second interpretation may correspond to a second value of a media parameter, and as a result, if user 135 has the visual ability in the environment corresponding to the second value of the media parameter, the second interpretation of the high-pass filtered second image B will be visible to user 135. For example, as described above in numerous examples herein, the first interpretation of the low-pass filtered first image A may correspond to a video resolution / spatial frequency below the test frequency (e.g., below 1080p video resolution on a 720p display), and the second interpretation of the high-pass filtered second image B may correspond to a video resolution / spatial frequency above the test frequency (e.g., above 1080p video resolution on a 720 display).

[0066] In some embodiments, the ratio between how much of a first interpretation of low-pass filtered image A is visible in the hybrid image and how much of a second interpretation of high-pass filtered image B is visible in the hybrid image is called the visibility ratio. In other words, the visibility ratio may also be a comparison of how much each of the two interpretations / perceptions in the hybrid image is visible to the human eye (e.g., to a person with 20 / 20 visual acuity and no visual impairment / disability). For example, the visibility ratio may differ for different hybrid images depending on one or more of the following: (i) the cutoff frequencies of one or both of the filters used to filter each of source images A and B, (ii) the gain of one or both of the filters used to filter each of source images A and B, and (iii) the size of the hybrid image displayed on the display 230.

[0067] In block 1710, one or more electronic processors are configured to improve the hybrid image to create an improved hybrid image that includes a second visibility ratio different from the first visibility ratio. For example, as described above herein, the hybrid image may be improved by adjusting one or more of the following: (i) the cutoff frequencies of one or both of the filters used to filter each of source images A and B, (ii) the gain of one or both of the filters used to filter each of source images A and B, and (iii) the size of the hybrid image displayed on the display 230. For example, the hybrid image may be adjusted according to Equation 1 to generate an improved hybrid image that tests a desired test frequency associated with a media parameter (e.g., video resolution) such that the high-pass filtered image B appears to the user 135 when displayed on the display 230, provided the user's visual ability is sufficiently high.

[0068] In some embodiments, after the improvement block 1710 is executed, the second visibility ratio of the improved hybrid image is closer to 1:1 than the first visibility ratio of the hybrid image. For example, the initial / unimproved hybrid image may be dominated by the low-pass filtered image A, so that even the high gain value of the high-pass filtered second image B does not allow the high-pass filtered second image B to be visible to the human eye in most viewing situations. However, the improved hybrid image may not be dominated by the low-pass filtered image A so that the high-pass filtered image B is visible to the user 135 when displayed on the display 230 if the user's visual ability is sufficiently high (e.g., the second visibility ratio is closer to 1:1 than the first visibility ratio). In some embodiments, the first visibility ratio of the initial / unimproved hybrid image is closer to 1:1 than the second visibility ratio of the improved hybrid image. In some embodiments, the first visibility ratio of the initial / unimproved hybrid image and the second visibility ratio of the improved hybrid image differ against separate reference values ​​(e.g., a default ratio value, a target ratio value which may be predetermined to provide a balanced hybrid image in which the interpretation of both source images A and B appears, for example, depending on the viewing conditions and viewing ability of a human user with a 20 / 20 visual acuity and negligence disorder or impairment).

[0069] In block 1715, one or more electronic processors control the display 230 of the playback device 110 to display the improved hybrid image. In block 1720, one or more electronic processors receive a first user input from a first user 135 via the input device of the playback device 110. In some embodiments, the first user input relates to a first perception of the improved hybrid image by the first user 135. As shown by the examples described above, the first user input may indicate whether the first user 135 can perceive the high-pass filtered image B associated with the displayed improved hybrid image.

[0070] In block 1725, one or more electronic processors determine optimized values ​​for media parameters (e.g., minimum / maximum QoE resolution, minimum / maximum QoE resolution range, points on the estimated QoE transfer function, etc.) at least in part based on a first user input. For example, as described above in this specification, one or more electronic processors may determine that the visual ability of user 135 in environment 130 is such that user 135 cannot distinguish between 720p video and 1080p video. Accordingly, one or more electronic processors may set the maximum video resolution of display 230 and the maximum video resolution of any requested media from media server 105 so as not to exceed 720p video resolution (e.g., video resolution values ​​should remain below 720p). In some embodiments, one or more electronic processors may determine that the visual ability of user 135 in environment 130 is such that user 135 can distinguish between 720p video and lower video resolutions. Accordingly, one or more electronic processors may set the desired video resolution of the display 230 and the desired video resolution of any requested media from the media server 105 to 720p video resolution when such video is available (for example, the video resolution should remain at 720p when possible for the user 135's maximum QoE).

[0071] In block 1730, one or more electronic processors provide the first output media to the first playback device 110 on the network 115 according to optimized values ​​of media parameters determined in block 1725. In some embodiments, the first output media is configured to be output to the first playback device 110 for consumption by user 135. For example, the first playback device 110 may request the first output media from the media server 105 according to optimized values ​​of media parameters (e.g., minimum and maximum QoE resolution of user 135 in environment 130, minimum and maximum QoE resolution range of user 135 in environment 130, estimated QoE transfer function, etc.). In some embodiments, the playback device 110 may request the first output media of a specific quality / bitrate, for example, according to optimized values ​​of media parameters determined in block 1725.

[0072] As described in PCT application PCT / US2020 / 044241, filed on 30 July 2020, current International Publication 2021 / 025946, sharing parameters related to playback device characteristics and personalized visual sensitivity coefficients with an upstream device configured to control the transmission of visual media to the playback device can often provide personalized adaptive media delivery based on collected playback-side information without the use of individual sensors. The full details thereof are incorporated herein by reference. Furthermore, the collected playback-side information may indicate personalized quality of experience (QoE) for different users and / or different viewing environments. Thus, improvements in network resource management / media delivery efficiency may exist while maintaining personalized QoE for each user. Continuing the above example, the video resolution of the video output on the first playback device 110 for the first user 135 may be reduced to 720p video resolution instead of 1080p video resolution, and / or maintained, without affecting the QoE of the first user 135.

[0073] As described above in this specification and also in PCT application PCT / US2020 / 044241, adaptive bitrate (ABR) streaming in a visual media delivery chain enables improved network resource management through adaptive selection of bitrate and resolution on a media ladder based on network conditions, playback buffer status, shared network capacity, and other network-affected factors. In addition to ABR streaming, other media delivery methods (including coding methods or source coding methods) may similarly be used to control one or more media parameters of an upstream video encoder / transcoder / translator, such as bitrate, frame rate, resolution, etc. (including other examples described above in this specification).

[0074] Many of the exemplary implementations of Method 1700 provided herein relate to determining an estimated minimum / maximum QoE video resolution or estimated minimum / maximum QoE video resolution range associated with user 135. The following description and examples relate simply to determining the shape of user 135's estimated QoE transfer function, rather than determining user 135's visual ability beyond a particular frequency / video resolution. In some embodiments, the QoE transfer function represents the user's overall QoE, taking into account numerous aspects of the transmission, processing, display, and / or consumption of the visual media (e.g., output media) from the signal path where the visual media is provided to the playback device 110 to the digital representation of the visual media, until user 135 consumes / views the visual media. In other words, the QoE transfer function may represent the net effect of the overall playback-side context information. In some embodiments, the playback-side context information includes the effects of the playback system 110 (e.g., display characteristics), the environment 130 (e.g., ambient lighting conditions and viewing distance), and the human observer 135 (e.g., characteristics of the subject's visual sensitivity). In some embodiments, the QoE transfer function represents a set of functions including the contrast sensitivity function (CSF) of user 135, the modulation transfer function (MTF) of the display 230 used to display the visual media, and / or other functions indicating the quality of the visual media displayed to user 135. In some embodiments, the CSF represents the relationship between the spatial frequency / video resolution of the display 230 and the contrast sensitivity of user 135 in the environment 130. The user 135's CSF is described in more detail in PCT application PCT / US2020 / 044241. In some embodiments, the MTF represents the frequency response of the display 230. The following description refers to the CSF, but it should be understood that the CSF is just one function that may affect the overall QoE transfer function of user 135 in the environment 130. In some embodiments, the user 135's CSF is the primary function affecting user 135's QoE transfer function.

[0075] In some embodiments, several assumptions may be made to estimate the amplitude value (in dB) of the QoE transfer function of user 135 in environment 130. First, it may be assumed that user 135's perception of the improved hybrid image (e.g., which of source image A or B is perceived as dominant by user 135) is determined by the comparison / visibility of the sum of the weighted power spectra (e.g., CSF weighted power spectra) of the low-pass filtered first image A and the high-pass filtered second image B. Second, it may be assumed that masking effects are negligible due to sufficient separation of the two spectra of source images A and B.

[0076] To illustrate the exemplary CSF effect for user 135, Figures 18B, 19B, and 20B show the assumed CSFs 1805, 1905, and 2005 for user 135 at a 3H viewing distance (e.g., a viewing distance of three times the height of display 230). Furthermore, Figures 18B, 19B, and 20B show the first CSF-weighted power spectra 1810, 1910, and 2010 of the first low-pass filtered image A (female) and the second CSF-weighted power spectra 1815, 1915, and 2015 of the second high-pass filtered image (male). In Figures 18B, 19B, and 20B, the above curves are shown along with the contents of Figures 12B, 13B, and 14B. In practice, Figures 18A, 19A, and 20A show the same three exemplary improved hybrid images shown in Figures 12A, 13A, and 14A. The graphs in Figures 18B, 19B, and 20B correspond to the respective hybrid images shown in Figures 18A, 19A, and 20A. In each of the hybrid images shown in Figures 18A, 19A, and 20A, the sum of the power spectra of the two source images A and B is set to be approximately equal. In some embodiments, the logarithms of the exemplary assumed CSFs 1805, 1905, and 2005 are sf maxIt may also be expressed by a simplified version of the truncated logarithmic parabolic form, as shown in Equation 3 below, using =5.28, β=1.30, and δ=0.23, where f represents the spatial frequency in one degree cycle. The above values ​​are exemplary values ​​for representing exemplary CSFs 1805, 1905, and 2005. Furthermore, Equation 3 is just one exemplary formula for representing the parametric form of User 135's CSF to demonstrate how the image spectrum is weighted by the CSF. In some embodiments, Equation 3 may use different values ​​for one or more constants. In some embodiments, the parametric form of User 135's CSF may be expressed by a different formula.

number

[0077] In the upper part of Figures 18B, 19B, and 20B, the difference between the sum of the CSF weighted powers of the high-pass filtered second image B (male) and the CSF weighted power of the low-pass filtered first image A (female) is ΔP CSF (For example, the sum of the areas under each curve 1810 and 1815, 1910 and 1915, and 2010 and 2015). In Figures 18A and 18B, the weighted power of the high-pass filtered second image is greater than the weighted power of the low-pass filtered first image (for example, ΔP CSF(=2.06dB). Therefore, the perception / interpretation of the improved hybrid image shown in Figure 18A may be dominated by the high-pass filtered second image for most human observers of the improved hybrid image. In other words, most human observers (e.g., a user 135 with 20 / 20 visual acuity and no visual disease / impairment) perceive the high-pass filtered second image instead of the low-pass filtered first image. As the size of the improved hybrid image decreases (e.g., as shown in Figures 19A and 20A), the weighted power of the low-pass filtered first image becomes greater than the weighted power of the high-pass filtered second image (e.g., ΔP in Figures 19B and 20B, respectively). CSF (=-7.44dB and -7.91dB). Therefore, the perceptual dominance of the improved hybrid image for most human observers shifts from the high-pass filtered second image in Figure 18A to the low-pass filtered first image in Figures 19A and 20A.

[0078] In some embodiments, the perception / interpretation of the improved hybrid image may be controlled by adjusting the gain (g) of the high-pass filtered second image B as described above herein, provided that perceptually significant portions of the image spectra of both source images A and B are within the user 135's visible range under at least some viewing conditions. The variable g' may be defined as the gain of the high-pass filtered image B, and the image perception / interpretation switches between the low-pass filtered hybrid first image A and the high-pass filtered second image B. Due to different visual abilities and different environments, g' may differ for different users 135 in different environments 130.

[0079] In some embodiments, the first electronic processor 205 measures the variable g + The measured variable g -The g' for the first user 135 in the first environment 130 is determined by calculating the intermediate point gain g' between and . In some embodiments, g - This is the measured gain (g) at which, as the gain (g) of the high-pass filtered second image B increases, the user 135 no longer perceives the low-pass filtered first image A. In some embodiments, g + This is the measured gain (g) at which, as the gain (g) of the high-pass filtered second image B decreases, the user 135 no longer perceives the high-pass filtered second image B. In some embodiments, the second gain of the low-pass filtered first image A may remain constant during the increase and decrease of the gain (g) of the high-pass filtered second image B.

[0080] An experiment was conducted using user 135 at a viewing distance of 3H to measure g' of the improved hybrid images shown in Figures 18A, 19A, and 20A. Figure 21 shows ΔP = P when the gain (g) = g* (for example, the gain g* when user 135's perception of the improved hybrid images in Figures 18A, 19A, and 20A changes from a high-pass filtered second image to a low-pass filtered first image). HP-filtered image -P LP-filtered image This is a graph showing the experimental results (in dB). In some embodiments, P HP-filtered image P is the sum of the power spectra of the second image that has been high-pass filtered in dB units. LP-filtered imageΔP is the sum of the power spectra of the first image after low-pass filtering in dB units. Figure 21 shows the unweighted ΔP curve 2110 and the CSF-weighted ΔP curve 2105 as a function of a second scaling factor S' for different improved hybrid image sizes. Each curve 2105, 2110 is based on three g' points corresponding to the improved hybrid images in Figures 18A, 19A, and 20A, respectively. In the experiment, an assumed CSF is used without considering all factors affecting the actual CSF of user 135, but it is found that the CSF-weighted ΔP curve 2105 is closer to 0 dB and more invariant with respect to image size (e.g., second scaling factor S') than the unweighted ΔP curve 2110.

[0081] From the data shown in Figure 21, it can be assumed that using a predetermined CSF, rather than a personalized CSF adjusted for a specific user 135, is the primary contributor to the deviation of the CSF-weighted ΔP curve 2105 from 0 dB. This assumption may be extended as appropriate to develop a new method for estimating user 135's CSF (or at least partially based on user 135's CSF) from a set of measured midpoint gain values ​​g'. As shown in Figure 21, the experimental results demonstrate the validity of using the CSF-weighted power spectrum rather than the raw power spectrum of the image to represent the midpoint gain g' obtained from an actual human subject / user (for example, because the CSF-weighted ΔP curve 2105 is closer to 0 dB than the unweighted ΔP curve 2110). It should be noted that the CSF used in the calculation of the ΔP value in Figure 21 is not user 135's actual CSF. Rather, the CSF used in the calculation of the ΔP value in Figure 21 is an assumed CSF used for experimental purposes. If the actual CSF of user 135 is used, the CSF-weighted ΔP curve 2105 may be even closer to 0 dB.

[0082] In some embodiments, N improved hybrid images with different second scaling coefficients S' are prepared to measure g' for each improved hybrid image and presented to the user 135. When the gain g = g', the difference between the sum of the power spectra of the high-pass filtered second image B and the low-pass filtered first image A in the improved hybrid image is minimized. Therefore, the optimal CSF in the sense of mean squared error may be estimated by minimizing the cost function J with respect to the parameter set θ using gradient descent defined by Equation 4 below. In some embodiments, the goal of displaying the improved hybrid image and receiving user input on the improved hybrid image is to find a desired parameter set Θ from intermediate points g' measured by the user 135 (e.g., which values ​​of g' provide the user 135 with changes in image perception), given known source images A and B used to generate the improved hybrid image.

number

[0083] In some embodiments, in Equation 4, k = 0, 1, ..., K-1 is a discrete frequency index and n = 0, 1, ..., N-1 is an image sample index. For example, the frequency power spectrum (along the x-axis in Figure 18B, for example) may contain K discrete components, where K covers a frequency range of 0.0 cpp to 0.5 cpp. In some embodiments, N may also be the number of improved hybrid images presented to user 135 to obtain the corresponding N gain switching values ​​g' (e.g., midpoint gain value g').

[0084] In some embodiments, by using Equation 4 in combination with user input received regarding an improved hybrid image displayed on the display 230, the first electronic processor 205 can determine an intermediate point gain value g' for each improved hybrid image for which user input has been received, indicating when the user 135's perception of the improved hybrid image changes from the high-pass filtered second image B to the low-pass filtered first image A as the gain (g) of the high-pass filtered second image B changes. The spatial frequency of the improved hybrid image in cycles / pixel (cpp) is known from the generation / selection of the improved hybrid image, as described above herein (see, for example, Equations 1 and 2). Thus, for each improved hybrid image for which user 135 exhibits an intermediate point gain value g', the first electronic processor 205 may plot the magnitude (in dB) of the intermediate point gain value g' against the spatial frequency to generate a point on user 135's estimated QoE transfer function. In some embodiments, the estimated QoE transfer function may approximate user 135's CSF.

[0085] In some embodiments, the first electronic processor 205 is configured to increase the gain (g) of the high-pass filtered second image B in the displayed improved hybrid image while the improved hybrid image is displayed on the display 230. As described above, the gain of the high-pass filtered second image B may be increased until the first user input indicates that the first interpretation of the low-pass filtered first image A is no longer perceptible to the first user 135. In response to receiving the first user input indicating that the first interpretation of the low-pass filtered first image A is no longer perceptible to the first user 135, the first electronic processor 205 may cease increasing the gain (g) of the high-pass filtered second image in the improved hybrid image and record a first gain value (g-) corresponding to the gain of the high-pass filtered second image B at the time the first user input indicating that the first interpretation is no longer perceptible to the first user 135 was received. In some embodiments, the first electronic processor 205 may optionally reset the gain (g) of the high-pass filtered second image B in the improved hybrid image to the original gain value used when the improved hybrid image was first displayed on the display 230.

[0086] In some embodiments, the first electronic processor is configured to decrease the gain (g) of the high-pass filtered second image B in the improved hybrid image while the improved hybrid image is displayed on the display 230. The gain (g) of the high-pass filtered second image B may decrease until a second user input indicates that a second interpretation of the high-pass filtered second image B is no longer perceptible to the first user 135. Upon receiving a second user input indicating that a second interpretation of the high-pass filtered second image B is no longer perceptible to the first user, the first electronic processor 205 is configured to cease decreasing the gain (g) of the high-pass filtered second image B in the improved hybrid image and to record a second gain value (g+) corresponding to the gain (g) of the high-pass filtered second image B at the time the second user input indicating that a second interpretation is no longer perceptible to the first user 135 was received.

[0087] In some embodiments, the first electronic processor 205 is configured to determine the midpoint gain (g') at which the perception of the improved hybrid image by the first user 135 changes from a first interpretation to a second interpretation. In some embodiments, the midpoint gain (g') is the first gain value (g - ) and the second gain value (g + This is determined by applying a mixture function or a weighting function to ). For example, equation 5 (below) shows that there may be a first weighting (w) for the gain of the first image A and a second weighting (1-w) for the second image B. Equation 5: Weighted Hybrid Image = wA + (1-w)B

[0088] In some embodiments, the midpoint gain (g') is the literal midpoint / arithmetic mean of two source images A and B, such that source images A and B have equal weighting in the hybrid image. In some embodiments, the midpoint gain (g') is not the literal midpoint / arithmetic mean of two source images A and B, but rather weighted so that source images A and B are different in the hybrid image. In some embodiments, the first electronic processor 205 is configured to determine the amplitude value (in dB) of a personalized estimated quality-of-effect (QoE) transfer function for the first user 135. In some embodiments, the amplitude value is used to generate at least one of a low-pass filtered first image A and a high-pass filtered second image B, as described above herein, at a first test frequency (f test ) is associated with the following. For example, the test frequency may include one of the following: a first cutoff frequency of a low-pass filter used to generate a first low-pass filtered image A, and a second cutoff frequency of a high-pass filter used to generate a second high-pass filtered image B.

[0089] In some embodiments, the above operation for determining the amplitude value of the estimated QoE transfer function for a particular test frequency may be repeated by the first electronic processor 205 to determine a plurality of amplitude values ​​and corresponding test frequencies. For example, the first electronic processor 205 may be configured to display a predetermined number of improved hybrid images and generate a predetermined number of points on the user 135's estimated QoE transfer function. In some embodiments, the first electronic processor 205 may be configured to display a plurality of improved hybrid images and generate a plurality of points on the user 135's estimated QoE transfer function until the user 135 completes a training session on the playback device 110.

[0090] In some embodiments, the QoE transfer function is associated with a user 135 in the environment 130. In other words, when requesting visual media from the media server 105 and displaying the visual media on the display 230, quantitative values ​​of the user 135's viewing ability at various spatial frequencies may be recorded and plotted for use by the first electronic processor 205. For example, as described above, and as described in PCT application PCT / US2020 / 044241, the first electronic processor 205 may reduce the quality of the visual media output on the playback device 110 to a level imperceptible to the user, based on the user's QoE transfer function. This results in improvements in network resource management / media delivery efficiency while maintaining personalized QoE for each user.

[0091] One advantage of estimating the QoE transfer function for a specific user 135 within a specific environment 130 is that it is an overall CSF estimate rather than a simple minimum / maximum QoE resolution, as described in the embodiments above. Overall CSF estimation may allow the playback device 110 to overcome the challenge of using an improved hybrid image that is too small to display to user 135 during the execution of the method described herein. For example, when determining the user's minimum / maximum QoE resolution according to the embodiments described herein above, the improved hybrid image may be too small for user 135 to see on the display 230, and the perception / interpretation of the high-pass filtered second image B may be very weak, even with very high gain values ​​due to the small image size. These technical problems can be overcome by the filter's cutoff frequency (f cThis is especially true when the ) is high. However, this technical problem / challenge can be addressed by using parameterized QoE transfer function calculations that enable accurate estimation of user 135's CSF at various spatial frequencies without excessively reducing the size of the improved hybrid image displayed on display 230. In other words, the method used in generating the estimated QoE transfer function enables the generation of data on user 135's visual ability in a very high frequency range that would be difficult to measure using the method for determining the minimum and maximum QoE resolution described above herein. For example, referring to the hybrid images in Figures 18A, 19A, and 20A, testing user 135's minimum and maximum QoE resolution at a high spatial frequency (e.g., a test frequency of 0.333 cpp corresponding to 720p on a 1080p display) may result in the generation of a smaller hybrid image, as shown in Figure 20A, compared to testing user 135's minimum and maximum QoE resolution at a low spatial frequency, as shown in Figures 18A and 19A. However, when estimating the transfer function of Equation 4 (which may be equivalent to fitting the shape of the CSF curve 1805 in Figure 18B using data points generated along the frequency axis (e.g., the x-axis)), it may suffice to use hybrid images ranging in size from those shown in Figure 18A to somewhere between the size of the hybrid image shown in Figure 19A and the size of the hybrid image shown in Figure 20A. Thus, in some embodiments, hybrid images as small as the hybrid image shown in Figure 20A may not need to be displayed to user 135, as the user's assumed CSF 1805 may be calculated without displaying such small hybrid images. Another advantage of estimating user 135's QoE transfer function is that this method can be extended to incorporate all different hybrid images together into the same QoE transfer function estimation procedure.

[0092] In some embodiments, increasing the gain of the high-pass filtered second image B within the improved hybrid image while the improved hybrid image is displayed on the display 230 is performed in response to a third user input controlling the gain (g) of the high-pass filtered second image B within the improved hybrid image while the improved hybrid image is displayed on the display 230. In some embodiments, decreasing the gain of the high-pass filtered second image B within the improved hybrid image while the improved hybrid image is displayed on the display 230 is performed in response to a fourth user input controlling the gain (g) of the high-pass filtered second image B within the improved hybrid image while the improved hybrid image is displayed on the display 230. For example, the display 230 may include a slider bar input that can be operated by user 135 to control the gain (g) of the high-pass filtered second image B.

[0093] In some embodiments, increasing the gain of a high-pass filtered second image B within the improved hybrid image while the improved hybrid image is displayed on the display 230 includes displaying a first plurality of improved hybrid images, each containing a different gain value for the high-pass filtered second image B (see, for example, Figure 15). In some embodiments, a first user input indicating that a first interpretation of a low-pass filtered first image A is no longer perceptible to a first user 135 includes a first selection of a first improved hybrid image from a first plurality of improved hybrid images, where the first interpretation of a low-pass filtered first image A is no longer perceptible to the first user 135. In some embodiments, decreasing the gain of a high-pass filtered second image B within the improved hybrid image while the improved hybrid image is displayed on the display 230 includes displaying a second plurality of improved hybrid images, each containing a different gain value for the high-pass filtered second image B. In some embodiments, a second user input indicating that a second interpretation of a high-pass filtered first image B is no longer perceptible to a first user 135 includes a second selection of a second improved hybrid image from a second plurality of improved hybrid images, wherein the second interpretation of a high-pass filtered first image B is no longer perceptible to the first user 135.

[0094] It should be understood that embodiments are not limited in their application to the details of the configuration and arrangement of components described herein or shown in the accompanying drawings. Embodiments can be implemented or performed in a variety of ways. It should also be understood that the expressions and terms used herein are for illustrative purposes only and should not be considered limiting. The use of “including,” “comprising,” or “having” and variations thereof means to include the items listed thereafter and their equivalents, as well as further items. Unless otherwise specified or limited, the terms “attached,” “connected,” “supported,” and “joined” and variations thereof are used broadly and include both direct and indirect attachment, connection, support, and joining.

[0095] Furthermore, it should be understood that embodiments may include hardware, software, and electronic components or modules that, for the purposes of discussion, are illustrated and described as if the majority of the components were implemented solely in hardware. However, those skilled in the art will recognize, based on reading this detailed description, that in at least one embodiment, the electronic-based aspect may be implemented in software (e.g., stored on a non-temporary computer-readable medium) executable by one or more electronic processors, such as microprocessors and / or application-specific integrated circuits ("ASICs"). Therefore, it should be noted that multiple hardware and software-based devices and multiple different structural components may be used to implement embodiments. For example, the “server” and “computing device” described herein may include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., system buses) for connecting various components.

[0096] Throughout this application, the terms “about” or “approximately” are used to describe the dimensions of various components. In some contexts, the term “about” means that the stated dimension is within 1% of the stated value, within 5% of the stated value, within 10% of the stated value, and so on. When the terms “and / or” are used in this application, it is intended to include any combination of the enumerated components. For example, if a component includes A and / or B, that component may include A only, B only, or A and B.

[0097] Various aspects of the present invention can be recognized from the following listed exemplary embodiments (EEE).

[0098] EEE1. A step of using one or more electronic processors to generate and select a hybrid image associated with a first interpretation corresponding to a first value of a media parameter and a second interpretation corresponding to a second value of the media parameter, wherein the hybrid image includes a first visibility ratio between the first interpretation and the second interpretation. The steps include: improving the hybrid image using one or more electronic processors to create an improved hybrid image that includes a second visibility ratio different from the first visibility ratio; The steps include displaying the improved hybrid image on the display of the first playback device, The steps include: receiving a first user input from a first user using one or more electronic processors, wherein the first user input relates to a first perception of the improved hybrid image by the first user; The steps include determining an optimized value for the media parameter based at least partially on the first user input using one or more electronic processors, The steps include providing a first output media to a first playback device over a network according to the optimized values ​​of the media parameters, wherein the first output media is configured to be output by the first playback device. A method that includes this.

[0099] EEE2. The method according to EEE1, wherein the optimized value of the media parameter includes an approximate minimum resolution for approximate maximum perceived quality (minimum-maximum QoE resolution) that is personalized for the first user based at least in part on the first user input.

[0100] EEE3. The method according to any one of the above EEEs, wherein the optimized value of the media parameter includes an estimated perceived quality (QoE) transfer function personalized for the first user based at least in part on the first user input.

[0101] EEE4. The step of improving the aforementioned hybrid image to create the improved hybrid image is: A step of scaling a first cutoff frequency of a low-pass filter used to filter a first image to a first scaled cutoff frequency by a first coefficient (S), wherein the first cutoff frequency is selected based on the first and second values ​​of the media parameter. The steps include scaling a second cutoff frequency of a high-pass filter used to filter a second image to a second scaled cutoff frequency by the first coefficient, wherein the second cutoff frequency is selected based on the first and second values ​​of the media parameter, The steps include: filtering the first image using the low-pass filter at the first scaled cutoff frequency to generate a low-pass filtered first image; The steps include: filtering the second image using the high-pass filter at the second scaled cutoff frequency to generate a high-pass filtered second image; The steps include: combining the first low-pass filtered image and the second high-pass filtered image to generate a scaled cutoff frequency filtered hybrid image, wherein the low-pass filtered image provides the first interpretation and the second high-pass filtered image provides the second interpretation; A step of generating the improved hybrid image by scaling the size of the scaled cutoff frequency filtered hybrid image, configured to be displayed on the display, to a scaled size by a second coefficient (S'), and A method that includes any one of the above EEEs.

[0102] EEE5. The method according to EEE4, wherein the first coefficient and the second coefficient are equivalent.

[0103] EEE6. The method according to EEE4 or EEE5, wherein the first cutoff frequency and the second cutoff frequency are equivalent.

[0104] EEE7. The step of improving the aforementioned hybrid image to create the improved hybrid image is: The steps include adjusting the gain of the low-pass filtered first image to control the first interpretation of the low-pass filtered first image in the improved hybrid image, A step of adjusting the gain of the high-pass filtered second image to control the second interpretation of the high-pass filtered second image in the improved hybrid image. A method according to any one of EEE4 to 6, further comprising at least one of the above.

[0105] EEE8. The step of displaying a plurality of first improved hybrid images on the display of the first playback device, each having at least one of (i) different gain values ​​and (ii) different sizes of the high-pass filtered second image, The method according to EEE7, wherein the first user input indicates whether the first user perceives the second interpretation of the high-pass filtered second image in one or more of the first plurality of improved hybrid images.

[0106] EEE9. In response to the first user input indicating that the first user perceives the second interpretation of the high-pass filtered second image within one or more of the first plurality of improved hybrid images, the steps of displaying a second plurality of improved hybrid images on the display of the first playback device, each of which is smaller in size than each of the first plurality of improved hybrid images, A step of receiving a second user input from the first user using one or more electronic processors, wherein the second user input indicates whether the first user perceives the second interpretation of the high-pass filtered second image within one or more of the second plurality of improved hybrid images. A step of determining the optimized value of the media parameter based at least in part on the second user input using one or more electronic processors, In response to the first user input indicating that the first user does not perceive the second interpretation of the high-pass filtered second image within one or more of the first plurality of improved hybrid images, the steps of displaying a third plurality of improved hybrid images on the display of the first playback device, each being larger in size than each of the first plurality of improved hybrid images, A step of receiving a third user input from the first user using one or more electronic processors, wherein the third user input indicates whether the first user perceives the second interpretation of the high-pass filtered second image within one or more of the third plurality of improved hybrid images. A step of determining the optimized value of the media parameter based at least in part on the third user input using one or more electronic processors. The method described in EEE8, further including the method described in EEE8.

[0107] EEE10. Each of the first multiple improved hybrid images is displayed simultaneously on the display according to the method described in EEE8 or EEE9.

[0108] EEE11. A step of using one or more electronic processors to increase the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display, the gain of the high-pass filtered second image being increased until the first user input indicates that the first interpretation is no longer perceptible to the first user, Steps include: In response to receiving the first user input, ceasing to increase the gain of the high-pass filtered second image in the improved hybrid image, and recording a first gain value (g-) corresponding to the gain of the high-pass filtered second image when the first user input is received indicating that the first interpretation is no longer perceptible to the first user; A step of using one or more electronic processors to reduce the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display, the gain of the high-pass filtered second image being reduced until a second user input indicates that the second interpretation is no longer perceptible to the first user. Steps of receiving the second user input, ceasing to reduce the gain of the high-pass filtered second image in the improved hybrid image, and recording a second gain value (g+) corresponding to the gain of the high-pass filtered second image when the second user input is received indicating that the second interpretation is no longer perceptible to the first user, A step of using one or more electronic processors to determine the intermediate point gain (g') at which the perception of the improved hybrid image by the first user changes from a first interpretation to a second interpretation, The steps include: determining the amplitude value of a personalized estimated quality of experience (QoE) transfer function for the first user using one or more electronic processors, wherein the amplitude value is associated with a first test frequency used to generate at least one of the low-pass filtered first image and the high-pass filtered second image; and The method described in EEE7, further including the above.

[0109] EEE12. The method according to EEE11, wherein the first test frequency includes one of the first cutoff frequency and the second cutoff frequency.

[0110] EEE13. While the improved hybrid image is displayed on the display, the step of increasing the gain of the high-pass filtered second image in the improved hybrid image is performed in response to a third user input controlling the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display. The method according to EEE11 or EEE12, wherein the step of reducing the gain of the high-pass filtered second image in the improved hybrid image is performed in response to a fourth user input controlling the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display.

[0111] EEE14. While the improved hybrid image is displayed on the display, the step of increasing the gain of the high-pass filtered second image in the improved hybrid image includes the step of displaying a first plurality of improved hybrid images, each having a different gain value for the high-pass filtered second image. The first user input includes a first selection of a first improved hybrid image from among the first multiple improved hybrid images, The step of reducing the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display includes the step of displaying a plurality of second improved hybrid images, each having a different gain value for the high-pass filtered second image. The method according to EEE11 or EEE12, wherein the second user input includes a second selection of a second improved hybrid image from among the second plurality of improved hybrid images.

[0112] EEE14.1. The media parameter is one or more of media bitrate, media frame rate, and media resolution, or includes one or more of these, as described in any one of EEE1 to 14.

[0113] EEE14.2. The method according to any one of EEE1 to 14.1, wherein the hybrid image includes a first image that has been low-pass filtered by a low-pass filter having a predetermined low-frequency cutoff frequency, and a second image that has been high-pass filtered by a high-pass filter having a predetermined high-frequency cutoff frequency.

[0114] EEE14.3. The method according to EEE14.2, wherein the first interpretation corresponds to the first image being visible to the user, and the second interpretation corresponds to the second image being visible to the user.

[0115] EEE 14.4. The method according to EEE 14.2 or 14.3, wherein improving the image includes adjusting one or more of the low-frequency cutoff frequency, the high-frequency cutoff frequency, a combination of the low-frequency cutoff frequency and the high-frequency cutoff frequency, the gain of the first image, the gain of the second image, the size of the first image, and the size of the second image.

[0116] EEE14.5. The optimized value of the media parameter is a third value of the media parameter, as described in any one of EEE1 to 14.4.

[0117] EEE14.6. The method according to any one of EEE1 to 14.5, wherein the step of determining the optimized value of the parameter includes the steps of determining a parameter indicating the visual ability of the first user to view the hybrid image on the display, at least in part on the first user input, and determining the optimized parameter in accordance with the parameter indicating the visual ability of the first user.

[0118] EEE15. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more electronic processors of an electronic computing device including a network interface and the display, wherein the one or more programs include instructions for executing any of the methods of EEE1 to 14.

[0119] EEE16. Electronic computing devices, Network interface and The display and One or more electronic processors, A memory that stores one or more programs configured to be executed by one or more electronic processors, wherein the one or more programs include instructions for executing any of the methods EEE1 to 14, and Electronic devices including

[0120] Various features and advantages are described in the following claims.

Claims

1. A step of using one or more electronic processors to generate and select a hybrid image, wherein the hybrid image comprises a first image low-pass filtered by a low-pass filter and a second image high-pass filtered by a high-pass filter, and the hybrid image is associated with a first interpretation governed by the low-pass filtered first image corresponding to a first value of a media parameter and a second interpretation governed by the high-pass filtered second image corresponding to a second value of the media parameter. The steps include: improving the hybrid image using one or more electronic processors to create an improved hybrid image, wherein the hybrid image is improved such that each of the first and second interpretations is perceptible on the display of a first playback device under certain viewing conditions; The steps include displaying the improved hybrid image on the display of the first playback device, A step of receiving a first user input from a first user using one or more electronic processors, wherein the first user input indicates whether the first user perceives the second interpretation of the high-pass filtered second image within the improved hybrid image. A step of using one or more electronic processors to determine optimized values ​​of the media parameters personalized for the first user, at least in part, based on the first user input. The steps include providing a first output media to a first playback device over a network according to the optimized values ​​of the media parameters, wherein the first output media is configured to be output by the first playback device. Includes, The step of improving the aforementioned hybrid image to create the improved hybrid image is: The steps include scaling the first cutoff frequency of the low-pass filter by a first coefficient (S), wherein the first cutoff frequency is selected based on the first and / or second values ​​of the media parameter, The step of scaling the second cutoff frequency of the high-pass filter by the first coefficient, wherein the second cutoff frequency is selected based on the first and / or second values ​​of the media parameter. The steps include: scaling the size of a scaled cutoff frequency filtered hybrid image, configured to be displayed on the display, to a scaled size by a second coefficient (S') to generate the improved hybrid image; Methods that include...

2. The method according to claim 1, wherein the optimized value of the media parameter includes an approximate minimum resolution for approximate maximum perceived quality (minimum-maximum QoE resolution) that is personalized for the first user based at least in part on the first user input.

3. The method according to claim 1, wherein the optimized value of the media parameter includes a value of an estimated quality-of-expected (QoE) transfer function personalized for the first user based at least in part on the first user input.

4. The step of improving the aforementioned hybrid image to create the improved hybrid image is: The steps include: filtering the first image using the low-pass filter at the scaled first cutoff frequency to generate the low-pass filtered first image; The steps include: filtering the second image using the high-pass filter at the scaled second cutoff frequency to generate the high-pass filtered second image; The steps include: combining the first low-pass filtered image and the second high-pass filtered image to generate the scaled cutoff frequency filtered hybrid image; The method according to claim 1, further comprising:

5. The method according to claim 4, wherein the first coefficient and the second coefficient are equivalent.

6. The method according to claim 4, wherein the first cutoff frequency and the second cutoff frequency are equivalent.

7. The step of improving the aforementioned hybrid image to create the improved hybrid image is: The steps include adjusting the gain of the low-pass filtered first image to control the first interpretation of the low-pass filtered first image in the improved hybrid image, A step of adjusting the gain of the high-pass filtered second image to control the second interpretation of the high-pass filtered second image in the improved hybrid image. The method according to claim 1, further comprising at least one of the following.

8. The step of displaying a plurality of first improved hybrid images on the display of the first playback device, each having at least one of (i) different gain values ​​and (ii) different sizes of the high-pass filtered second image, The method according to claim 7, wherein the first user input indicates whether the first user perceives the second interpretation of the high-pass filtered second image in one or more of the first plurality of improved hybrid images.

9. In response to the first user input indicating that the first user perceives the second interpretation of the high-pass filtered second image within one or more of the first plurality of improved hybrid images, the steps of displaying a second plurality of improved hybrid images on the display of the first playback device, each of which is smaller in size than each of the first plurality of improved hybrid images, A step of receiving a second user input from the first user using one or more electronic processors, wherein the second user input indicates whether the first user perceives the second interpretation of the high-pass filtered second image within one or more of the second plurality of improved hybrid images. A step of determining the optimized value of the media parameter based at least in part on the second user input using one or more electronic processors, In response to the first user input indicating that the first user does not perceive the second interpretation of the high-pass filtered second image within one or more of the first plurality of improved hybrid images, the steps of displaying a third plurality of improved hybrid images on the display of the first playback device, each being larger in size than each of the first plurality of improved hybrid images, A step of receiving a third user input from the first user using one or more electronic processors, wherein the third user input indicates whether the first user perceives the second interpretation of the high-pass filtered second image within one or more of the third plurality of improved hybrid images. A step of determining the optimized value of the media parameter based at least in part on the third user input using one or more electronic processors. The method according to claim 8, further comprising:

10. A step of using one or more electronic processors to increase the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display, the gain of the high-pass filtered second image being increased until the first user input indicates that the first interpretation is no longer perceptible to the first user. In response to receiving the first user input, the gain of the high-pass filtered second image in the improved hybrid image is stopped, and the first gain value (g) corresponds to the gain of the high-pass filtered second image when the first user input is received indicating that the first interpretation is no longer perceptible to the first user. - ) a step to record, A step of using one or more electronic processors to reduce the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display, the gain of the high-pass filtered second image being reduced until a second user input indicates that the second interpretation is no longer perceptible to the first user. In response to receiving the second user input, the reduction of the gain of the high-pass filtered second image in the improved hybrid image is stopped, and a second gain value (g) corresponding to the gain of the high-pass filtered second image when the second user input is received indicating that the second interpretation is no longer perceptible to the first user is set. + ) a step to record, A step of using one or more electronic processors to determine an intermediate gain (g') between the first gain value (g-) and the second gain value (g+) at which the first user's perception of the improved hybrid image changes from the first interpretation to the second interpretation; The steps include: determining the amplitude value of a personalized estimated quality of experience (QoE) transfer function for the first user using one or more electronic processors, wherein the amplitude value is associated with a first test frequency used to generate at least one of the low-pass filtered first image and the high-pass filtered second image; and The method according to claim 7, further comprising:

11. The method according to claim 10, wherein the first test frequency includes one of the first cutoff frequency and the second cutoff frequency.

12. While the improved hybrid image is displayed on the display, the step of increasing the gain of the high-pass filtered second image in the improved hybrid image is performed in response to a third user input controlling the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display. The method according to claim 10, wherein the step of reducing the gain of the high-pass filtered second image in the improved hybrid image is performed in response to a fourth user input controlling the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display.

13. The step of increasing the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display includes the step of displaying a first plurality of improved hybrid images, each having a different gain value for the high-pass filtered second image. The first user input includes a first selection of a first improved hybrid image from among the first multiple improved hybrid images, The step of reducing the gain of the high-pass filtered second image in the improved hybrid image while the improved hybrid image is displayed on the display includes the step of displaying a plurality of second improved hybrid images, each having a different gain value for the high-pass filtered second image. The method according to claim 10, wherein the second user input includes a second selection of a second improved hybrid image from among a second plurality of improved hybrid images.

14. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more electronic processors of an electronic computing device including a network interface and a display, A non-temporary computer-readable storage medium comprising one or more programs including instructions for performing the method according to any one of claims 1 to 13.

15. An electronic computing device, Network interface and The display and One or more electronic processors, A memory that stores one or more programs configured to be executed by one or more electronic processors, wherein the one or more programs include instructions for performing the method described in any one of claims 1 to 13. Electronic devices including

Images