Adaptive encoding based on individual game player's sensitivity to visual artifacts

Abstract

Techniques are described for an encoder or decoder to adaptively change (e.g., 604) code processing based on a particular user's sensitivity to flicker or flash or blocking or other visual artifacts. Alternatively, the video may be pre-processed (e.g., 1000) to suppress artifacts based on the sensitivity of the user prior to encoding, and / or post-processed to suppress artifacts after decoding.

Description

Technical Field

[0001] This application relates to unconventional solutions that must be rooted in computer technology and produce specific technical improvements, and more specifically, to adaptive coding based on individual game players’ sensitivity to visual artifacts. Background Technology

[0002] Playing computer games is typically done over a network, where the game itself can be streamed, and multiple players, geographically distant from each other, may play the same game. Streaming games presents technical challenges related to energy usage and network limitations. For example, stringent energy demands may be imposed on network computing, limiting the network's data-carrying capacity. Furthermore, the massive amounts of data that may be sought to be sent over the network can exceed its capabilities, especially in the so-called "last mile." These limitations lead to several problems, including reduced bandwidth and increased latency. Summary of the Invention

[0003] As this article has been learned, in some cases, and especially for presentation purposes, data transfer load can be reduced by decreasing or eliminating certain correction actions used to encode and / or decode video for some people who may not perceive flicker, flash, blockiness, pixelation, blur, or other visual artifacts as strongly as others.

[0004] Therefore, an apparatus includes at least one processor component configured to receive information relating to the sensitivity of a first user to at least one visual artifact, and information relating to the sensitivity of a second user to at least one visual artifact. The processor component is configured to encode and / or decode, preprocess, and / or postprocess at least one video for presentation on a first display based on the information relating to the first user's sensitivity to visual artifacts. Furthermore, the processor component is configured to encode and / or decode, preprocess, and / or postprocess the video for presentation on a second display based on the information relating to the second user's sensitivity to visual artifacts.

[0005] The video may include at least one computer game.

[0006] The visual artifact can include one or more of flickering, flashing, or blockiness.

[0007] In another aspect, an apparatus includes at least one computer medium that is not a transient signal and also includes instructions executable by at least one processor component to identify a first user’s sensitivity to at least one visual artifact in a video, and to encode and / or decode at least one video for presentation on a first display associated with the first user based on the first user’s sensitivity to the visual artifact.

[0008] In another approach, a method includes receiving an indication of perceptual sensitivity to a video, and preprocessing the video before encoding and / or postprocessing the video after decoding according to the indication. The method includes presenting the video on at least one display.

[0009] The details of this disclosure regarding both its structure and operation can be best understood with reference to the accompanying drawings, in which the same reference numerals refer to the same parts, and in the accompanying drawings: Attached Figure Description

[0010] Figure 1 This is a block diagram of an example system that includes examples consistent with the principles of the present invention; Figure 2 An example encoder-decoder system is shown; Figure 3 An example specific system consistent with the principles of the present invention is shown; Figure 4 Example logic for training an ML model to execute the techniques of this invention is shown in an example flowchart format; Figure 5 Example logic for using an ML model to perform the techniques of this invention is shown in an example flowchart format; Figure 6 and Figure 7 An example encoder-side logic for implementing the techniques of the present invention for a first user and a second user with different sensitivities is shown in an example flowchart format. Figure 8 and Figure 9 An example decoder-side logic for implementing the technique of the present invention for a first user and a second user with different sensitivities is shown in the example flowchart format. Figure 10 and Figure 11 An example encoder-side logic for implementing the preprocessing suppression technique of the present invention is shown in an example flowchart format; and Figure 12 An example decoder-side logic for implementing the post-processing suppression technique of the present invention is shown in an example flowchart format. Detailed Implementation

[0011] This disclosure generally relates to a computer ecosystem that includes various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. Systems described herein may include server components and client components that can be networked to enable the exchange of data between the client components and the server components. Client components may include one or more computing devices, including game consoles (such as Sony PlayStation® or game consoles made by Microsoft, Nintendo, or other manufacturers), extended reality (XR) headsets (such as virtual reality (VR) headsets, augmented reality (AR) headsets), portable televisions (e.g., smart TVs, internet-enabled TVs), portable computers (such as laptops and tablets), and other mobile devices (including smartphones and additional examples discussed below). These client devices may operate in a variety of operating environments. For example, some client computers may use operating systems such as Linux, operating systems from Microsoft, or Unix, or operating systems produced by Apple, Inc., Google, Berkeley Software Distribution, or Berkeley Standard Distribution (BSD) OS (including descendants of BSD). These operating environments can be used to execute one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted on internet servers discussed below. Furthermore, the operating environment according to the principles of the present invention can be used to execute one or more computer game programs.

[0012] A server and / or gateway may be used, which may include one or more processors that execute instructions to configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be instantiated from a game console such as a Sony PlayStation®, a personal computer, etc.

[0013] Information can be exchanged between clients and servers over a network. For this purpose, and for security reasons, servers and / or clients may include firewalls, load balancers, temporary storage and proxies, as well as other network infrastructure for reliability and security. One or more servers can form a device that enables methods for providing secure communities (such as online social networking sites or gaming networks) to network members.

[0014] A processor can be a single-chip or multi-chip processor, which can execute logic using various lines (such as address lines, data lines, and control lines) as well as registers and shift registers. A processor, including a digital signal processor (DSP), can be an implementation of a circuit system. A processor component can include one or more processors.

[0015] Components included in one embodiment can be used in any suitable combination in other embodiments. For example, any of the various components depicted herein and / or in the accompanying drawings can be combined, interchanged, or excluded from other embodiments.

[0016] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes: a system having only A; a system having only B; a system having only C; a system having both A and B; a system having both A and C; a system having both B and C; and / or a system having both A, B and C.

[0017] Now for reference Figure 1 An example system 10 is illustrated, which may include one or more of the example devices mentioned above and further described below according to the principles of the invention. The first device among the example devices included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, such as, but not limited to, a cinema display system (which may be projector-based) or an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling a TV). Alternatively, the AVD 12 may also be a computerized internet-enabled (“smart”) phone, tablet computer, laptop computer, head-mounted device (HMD) and / or head-mounted equipment (such as smart glasses or VR headsets), another wearable computerized device, a computerized internet-enabled music player, a computerized internet-enabled headset, a computerized internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the invention (e.g., to communicate with other CE devices to implement the principles of the invention, to perform the logic described herein, and to perform any other functions and / or operations described herein).

[0018] Therefore, to implement such principles, the AVD 12 can be constructed using some or all of the components shown. For example, the AVD 12 may include one or more touch-enabled displays 14, which may be implemented using a high-definition or ultra-high-definition "4K" or higher resolution flat panel screen. The touch-enabled display 14 may include, for example, a capacitive or resistive touch sensing layer with an electrode grid for touch sensing, consistent with the principles of the present invention.

[0019] AVD 12 may also include: one or more speakers 16 for outputting audio according to the principles of the invention; and at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. Therefore, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to implement the principles of the invention, including controlling other elements of AVD 12 described herein, such as controlling display 14 to present images on the display and receiving input from the display. Furthermore, it should be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above.

[0020] In addition to the foregoing, AVD 12 may also include one or more input and / or output ports 26, such as a High Definition Multimedia Interface (HDMI) port or a Universal Serial Bus (USB) port for physically connecting to another CE device and / or a headphone port for connecting headphones to AVD 12 to present audio from AVD 12 to a user via headphones. For example, input port 26 may be connected via a cable or satellite source 26a to audio / video content, either wired or wirelessly. Therefore, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disk player containing content. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 48.

[0021] AVD 12 may also include one or more computer memory / computer-readable storage media 28 that are not transient signals, such as disk-based storage or solid-state storage. In some cases, the one or more computer memory / computer-readable storage media may be embodied as a stand-alone device within the AVD's housing, or as a personal video recording device (PVR) or video disk player for playing back AV programs, either inside or outside the AVD's housing, or as removable storage media or a server as described below. Furthermore, in some embodiments, AVD 12 may include a location or positioning receiver, such as, but not limited to, a mobile phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or mobile phone base station and provide this information to processor 24 and / or, in conjunction with processor 24, determine the altitude at which AVD 12 is positioned.

[0022] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be a thermal imaging camera, a digital camera (such as a webcam), an IR sensor, an event-based sensor, and / or a camera integrated into AVD 12 and controllable by processor 24 to acquire pictures / images and / or videos according to the principles of the present invention. AVD 12 may also include a Bluetooth® transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0023] Furthermore, the AVD 12 may include one or more auxiliary sensors 38 that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors forming a layer of the touch-enabled display 14 itself, and may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Other sensor examples include pressure sensors, motion sensors (such as accelerometers, gyroscopes, odometers, or magnetic sensors), infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, event-based sensors, and gesture sensors (e.g., for sensing gesture commands). Thus, the sensor 38 may be implemented by one or more motion sensors, such as individual accelerometers, gyroscopes, and magnetometers and / or inertial measurement units (IMUs), which typically include a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions, or by event-based sensors (such as event detection sensors (EDS)). Consistent with this disclosure, the EDS provides an output indicating a change in light intensity sensed by at least one pixel of the light sensing array. For example, if the light sensed by a pixel is decreasing, the EDS output can be -1; if the light sensed by a pixel is increasing, the EDS output can be +1. Outputting a binary signal of 0 can indicate that there is no light intensity change below a certain threshold.

[0024] The AVD 12 may also include an over-the-air (OTA) TV broadcast port 40 that provides input to the processor 24 for receiving OTA TV broadcasts. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, such as a kinetic energy harvester that converts kinetic energy into electrical energy to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field-programmable gate array (FPGA) 46 may also be included. One or more tactile / vibration generators 47 may be provided to generate tactile signals that can be sensed by a person holding or touching the device. Therefore, the haptic generator 47 can use an electric motor to vibrate all or part of the AVD 12, the electric motor being connected via a rotatable shaft to an eccentric and / or unbalanced counterweight, such that the shaft can be rotated under the control of the motor (which can in turn be controlled by a processor such as processor 24) to generate vibrations of various frequencies and / or amplitudes as well as force simulations in various directions.

[0025] It may also include light sources, such as projectors, such as infrared (IR) projectors.

[0026] In addition to AVD 12, System 10 may also include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to send computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or via a server described below, while the second CE device 50 may include components similar to the first CE device 48. In the example shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a head-up transparent or opaque display for presenting AR / MR content or VR content (more generally, extended reality (XR) content), respectively. The HMD may be configured as a glasses-type display or as a larger VR-type display sold by computer game equipment manufacturers.

[0027] In the example shown, only two CE devices are illustrated; it should be understood that fewer or more devices may be used. The devices described herein can implement some or all of the components shown for AVD 12. Any component shown in the following figures may be combined with some or all of the components shown in the case of AVD 12.

[0028] Referring now to the aforementioned at least one server 52, which includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (such as disk-based storage or solid-state storage), and at least one network interface 58, which, under the control of the server processor 54, allows communication with other illustrated devices via network 22 and, in practice, facilitates communication between the server and client devices according to the principles of the invention. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as, for example, a wireless telephone transceiver).

[0029] Therefore, in some implementations, server 52 may be an internet server or an entire server "farm," and in example implementations such as online gaming applications, it may include and perform "cloud" functionality, enabling devices of system 10 to access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown.

[0030] The components shown in the diagram below may include some or all of the components shown in this document. Any user interface (UI) described herein may be combined and / or extended, and UI elements may be mixed and matched between UIs.

[0031] The principles of this invention can be applied to various machine learning models, including deep learning models. Machine learning models consistent with the principles of this invention can use various algorithms trained in ways including: supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms that can be implemented by computer circuit systems include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN called a long short-term memory (LSTM) network. Generative pre-trained Transformers (GPTT) can also be used. Support vector machines (SVMs) and Bayesian networks can also be considered examples of machine learning models. In addition to the network types mentioned above, the model in this paper can also be implemented using a classifier.

[0032] As understood in this paper, performing machine learning can therefore involve accessing training data and then training a model on that training data so that the model can process additional data to make inferences. Thus, an artificial neural network / AI model trained via machine learning can include an input layer, an output layer, and multiple hidden layers located between them, which are configured and weighted to make inferences about the appropriate output.

[0033] Figure 2 A system including a video encoder 200 for encoding / compressing video 202 is shown. A video decoder 204 can receive encoded video and decode / decompress it into output video 206.

[0034] Figure 3 An example system for an end-user gamer is shown, comprising one or more displays 300 (such as touchscreen displays), one or more cameras 302, one or more microphones 304, one or more typing devices 306 (such as keyboards or keypads), and one or more computer game controllers 308 (such as Sony PlayStation controllers). A user interface (UI) (310) may be present, for example, on a display or speaker, or otherwise, to ask the user whether they are sensitive to one or more visual artifacts (such as flash, blockiness, jitter, etc.) in the video. The user may input any indication of such sensitivity via any of the input devices shown, including verbally via a microphone, sign language detected by a camera, manipulating a controller or typing device, or making a selection from a touchscreen. Note that in addition to simply asking the user about a particular artifact, the UI may also show the user examples of artifacts, as indicated at 312.

[0035] Figure 4An alternative technique for using machine learning (ML) to identify a user's sensitivity to video artifacts is shown. At state 400, a training dataset is fed into the ML model to train the model at state 402. The training set can include, for example, gameplay videos without artifacts and gameplay videos with artifacts, as well as ground truth values ​​indicating a player's reaction to artifacts, so that the model learns what types of player behavior indicate sensitivity (or insensitivity) to artifacts.

[0036] This is Figure 5 The following is shown. Starting at state 500, the ML model, trained as described, receives a specific user's gameplay process and identifies artifacts (if any) in the video. Moving to state 502, the model outputs an indication of the user's sensitivity to artifacts, indicated in quantitative terms if needed, such as from "none" to "significantly sensitive".

[0037] Now for reference Figure 6 Beginning at state 602, an indication is made of the sensitivity of a specific first user to one or more visual artifacts in the video. Proceeding to state 604, the video intended for streaming to that user is encoded based on the first user's sensitivity indication. For example, if the user is not sensitive to one or more artifacts, the video can be encoded at a lower bit rate than if the user were sensitive to one or more artifacts. Assume the video is encoded for the first user in a first manner (e.g., at a first bit rate and / or frame rate and / or resolution). At state 604, the encoded video is transmitted over the network to the first user.

[0038] In comparison, Figure 7 The encoding for a second specific user is illustrated. Starting at state 700, an indication of the second user's sensitivity to one or more visual artifacts in the video is identified. Proceeding to state 702, the video intended for streaming to the second user is encoded based on the second user's sensitivity indication. It is assumed that the video is encoded for the second user in a second manner (e.g., at a second bitrate and / or frame rate and / or resolution). At state 704, the encoded video is transmitted to the second user over the network. Therefore, depending on their sensitivity to visual artifacts in the video, the video can be encoded in different ways for different users.

[0039] Alternatively, depending on their sensitivity to visual artifacts in the video, the video can be decoded in different ways for different users. Figure 8 and Figure 9 As shown. In Figure 8Beginning at state 800, an indication is made of the sensitivity of a specific first user to one or more visual artifacts in the video. Proceeding to state 802, the video streamed to the user is decoded based on the indication of the first user's sensitivity. For example, if the user is not sensitive to one or more artifacts, the video can be decoded at a lower bitrate than if the user were sensitive to one or more artifacts. Assume the video is decoded for the first user in a first manner (e.g., at a first bitrate and / or frame rate and / or resolution). At state 804, the decoded video is presented on at least one display associated with the first user.

[0040] In comparison, Figure 9 Decoding for a second specific user is illustrated. Starting at state 900, an indication of the second user's sensitivity to one or more visual artifacts in the video is identified. Proceeding to state 902, the video streamed to the second user is decoded based on the second user's sensitivity indication. It is assumed that the video is decoded for the second user in a second manner (e.g., at a second bitrate and / or frame rate and / or resolution). At state 904, the decoded video is presented on at least one display associated with the second user. Therefore, depending on their sensitivity to visual artifacts in the video, the video can be decoded in different ways for different users.

[0041] Although Figures 6 to 9 The considerations are for streaming video over a network (such as computer games), but these can also be applied to non-streaming environments, such as games played by a local game engine running in a computer game console.

[0042] Note that encoding and decoding techniques can be applied in the presence of multiple artifacts, and are particularly effective for blockiness (blocks in video). Now refer to preprocessing techniques for suppressing artifacts in content such as strong flashes and flickering lights. Figure 10 The techniques described in this paper can also be used to map the original colors in a video to different colors that are more suitable for colorblind users whose color perception characteristics are known. The techniques described in this paper can also be used to change the speed of motion of objects and / or the associated virtual camera "shooting" the video to appropriately alleviate motion sickness in users suffering from motion sickness.

[0043] exist Figure 10 Starting at state 1000, the video intended to be provided to the user via streaming or from a computer console is preprocessed according to the nature of the particular user's sensitivity, in order to suppress artifacts in the content (rather than encoding artifacts), such as any of the relevant artifacts discussed herein. The video is then encoded at state 1002 and transmitted at state 1004 for playback on the system associated with the particular user.

[0044] Suppression at state 1000 in the preprocessing stage can be performed by a human expert or by an ML model. Figure 11 This demonstrates how such a model can be trained.

[0045] Starting at state 1100, the training dataset is sent to the ML model to train the ML model at state 1102. The training set may include videos containing various content-related artifacts, ground truth values ​​indicating what the artifacts are, and preprocessed videos from the original videos in which ground truth artifacts have been suppressed.

[0046] Figure 12 This demonstrates the use of the aforementioned technique in a post-processing step performed at the receiver after decoding a video containing content-related artifacts. Figure 12 Starting at state 1200, video with artifacts is received via streaming or from a computer console. Proceeding to state 1202, the video is decoded by the receiver.

[0047] Moving to state 1204, the video is post-processed based on the nature of the particular user's sensitivity to suppress artifacts in the content (rather than encoding-induced artifacts), such as any of the relevant artifacts discussed herein. Then, at state 1206, the video is displayed on the monitor of the user whose sensitivity was used to suppress artifacts in state 1204.

[0048] The inhibition at preprocessing state 1200 can be performed by human experts or, for example, according to Figure 11 Execute the trained ML model.

[0049] While specific techniques are shown and described in detail herein, it should be understood that the subject matter contained herein is limited only by the claims.

Claims

1. An apparatus comprising: At least one processor component, said at least one processor component being configured to: Receive information relating to the first user's sensitivity to at least one visual artifact; Receive information relating to the second user’s sensitivity to at least one visual artifact; Encode and / or decode and / or preprocess and / or postprocess at least one video for presentation on a first display based on the information relating to the first user’s sensitivity to the visual artifacts; as well as The video is encoded and / or decoded and / or preprocessed and / or postprocessed for presentation on a second display based on the information relating to the second user's sensitivity to the visual artifacts.

2. The device of claim 1, wherein the video includes at least one computer game.

3. The device of claim 1, wherein the processor component is configured to: The video is encoded for presentation on the first display based on information relating to the first user's sensitivity to the visual artifacts; and The video is encoded for presentation on the second display based on information relating to the second user's sensitivity to the visual artifacts.

4. The device of claim 1, wherein the processor component is configured to: Decode the video for presentation on the first display based on information relating to the first user's sensitivity to the visual artifacts; and The video is decoded based on the information relating to the second user's sensitivity to the visual artifacts for presentation on the second display.

5. The device of claim 1, wherein the processor component is configured to: At least one video is preprocessed for presentation on the first display based on information relating to the first user's sensitivity to the visual artifacts; and The video is preprocessed based on the information relating to the second user's sensitivity to the visual artifacts for presentation on the second display.

6. The device of claim 1, wherein the processor component is configured to: The at least one video is post-processed based on information relating to the first user's sensitivity to the visual artifacts for presentation on the first display; and The video is post-processed based on the information relating to the second user's sensitivity to the visual artifacts for presentation on the second display.

7. The device of claim 1, wherein the visual artifact includes flickering or flashing.

8. The device of claim 1, wherein the visual artifacts include block effects.

9. An apparatus comprising: At least one computer medium, which is not a transient signal and includes instructions executable by at least one processor component to: Identify the first user's sensitivity to at least one visual artifact in the video; as well as At least one video is encoded and / or decoded based on the first user's sensitivity to the visual artifacts for presentation on a first display associated with the first user.

10. The device of claim 9, wherein the video includes at least one computer game.

11. The device of claim 9, wherein the instructions are executable to: The video is encoded for presentation on the first display based on the first user's sensitivity to the visual artifacts.

12. The device of claim 9, wherein the instructions are executable to: The video is decoded based on the first user's sensitivity to the visual artifacts for presentation on the first display.

13. The device of claim 9, wherein the visual artifact includes flickering.

14. The device of claim 9, wherein the visual artifact includes a flash.

15. The device of claim 9, wherein the visual artifacts include block effects.

16. A method comprising: Receive instructions on the perceptual sensitivity to video; The video is preprocessed according to the instructions before encoding; or After decoding, the video is post-processed according to the instructions; or The video is subjected to both preprocessing and postprocessing according to the instructions; as well as The video is displayed on at least one monitor.

17. The method of claim 16, wherein the display is associated with at least one input device from which the indication is received.

18. The method of claim 16, comprising: The video is preprocessed according to the instructions before encoding.

19. The method of claim 16, comprising: After decoding, the video is post-processed according to the instructions.

20. The method of claim 16, comprising: The video is preprocessed and postprocessed according to the instructions.

Images