Priority-based calibration of magnification of each computer game object and object portion

Abstract

As understood herein, not all objects (300, 302, 304) in a computer game have the same priority. In view of this knowledge, the present principles apply (402) a series of spatial and temporal resolutions to various objects in the same game according to the priorities of the objects in the game.

Description

Technical Field

[0001] This application relates to an unconventional solution that is technically innovative, necessarily rooted in computer technology and produces specific technical improvements, and more specifically involves prioritizing the scaling of each computer game object and object portion. Background Technology

[0002] Streaming computer games over a network typically requires balancing Quality of Service (QoS) with network issues such as bandwidth and latency. Summary of the Invention

[0003] As understood in this article, not all objects in a computer game have the same priority. Given this understanding, this principle applies a range of spatial and temporal resolutions within the same game based on the priority of objects in the game.

[0004] Therefore, an apparatus includes at least one processor component configured to encode the first object or region in a video at a first frame rate in response to the first object or region having a first priority. The processor component is further configured to encode the second object or region in the video at a second frame rate in response to the second object or region in the video having a second priority.

[0005] The video may include, for example, computer game videos.

[0006] In an example embodiment, the first priority is higher than the second priority, and the first frame rate is faster than the second frame rate.

[0007] If necessary, the processor component can be configured to transmit the first object or region and the second object or region over a network after encoding.

[0008] In a non-limiting embodiment, the processor component may be configured to encode the first object or region at a first resolution in response to the first object or region in the video having the first priority, and to encode the second object or region at a second resolution in response to the second object or region in the video having the second priority.

[0009] In various examples, the processor component may be configured to determine at least the first priority by at least one or more of the following: recognizing the first object or area as a target, recognizing the first object or area as being viewed by a player, recognizing the first object or area as having been viewed by multiple players in the past, recognizing the first object or area as being interacted with by a computer game controller, recognizing the first object or area as a playable object, and recognizing the first object or area as having the first priority as signaled by the game engine. Alternatively, the processor component may be configured to determine at least the second priority by one or more of the following: recognizing the first object or area as a statistical display area, and recognizing the first object or area as including text.

[0010] In another aspect, a device includes at least one computer medium that is not a transient signal and contains instructions executable by at least one processor assembly to: encode the first object or region in a video at a first resolution in response to the first object or region in the video having a first priority; the processor assembly is configured to encode the second object or region in the video at a second resolution in response to the second object or region in the video having a second priority.

[0011] In another aspect, a method includes: encoding a first object in a video at a first spatial and / or temporal resolution, the first object having a higher priority than a second object in the video; and encoding the second object at a second spatial and / or temporal resolution lower than the first spatial and / or temporal resolution.

[0012] The details of this disclosure regarding its structure and operation can be best understood by referring to the accompanying drawings, wherein like reference numerals denote like parts, and in the drawings: Attached Figure Description

[0013] Figure 1 It is a block diagram of an example system that includes examples conforming to this principle;

[0014] Figure 2 An example encoder-decoder system is shown;

[0015] Figure 3 This schematically illustrates a video scene with video objects or frame regions ordered according to their importance.

[0016] Figure 4 An example overall logic for encoding different objects in the same code using different spatial and / or temporal resolutions is shown in the form of an example flowchart;

[0017] Figure 5 and Figure 6 Example logic for determining the priority of objects or regions in a frame in a computer game is shown in the form of an example flowchart;

[0018] Figure 7 Example logic for variable time resolution encoding is shown in the form of an example flowchart;

[0019] Figure 8 Example logic for variable spatial resolution encoding is shown in the form of an example flowchart;

[0020] Figure 9 An example alternative logic for determining the priority of objects or regions in a frame in a computer game is shown in the form of an example flowchart;

[0021] Figure 10 and Figure 11 Example logic for training and using a machine learning (ML) model to determine priorities is shown in the form of an example flowchart;

[0022] Figure 12 An example decoder-side logic for increasing the temporal resolution of an object or region is shown in the form of an example flowchart;

[0023] Figure 13 An example flowchart illustrates sample logic for dynamically changing the encoding resolution based on latency; and

[0024] Figure 14 An example flowchart illustrates the logic for dynamically changing the encoding resolution based on QoS. Detailed Implementation

[0025] This disclosure generally relates to the computer ecosystem, including various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. Systems described herein may include server and client components that can be network-connected, enabling data exchange between the client and server components. Client components may include one or more computing devices, such as game consoles (e.g., Sony PlayStation). ®Alternatively, these client devices may include game consoles manufactured by Microsoft, Nintendo, or other manufacturers; virtual reality (VR) headsets; augmented reality (AR) headsets; extended reality (XR) headsets; portable televisions (e.g., smart TVs, internet-enabled TVs); portable computers such as laptops and tablets; and other mobile devices including smartphones and additional examples discussed below. These client devices can operate with a variety of operating environments. For example, some client computers may use operating systems such as Linux, operating systems from Microsoft, or Unix, or operating systems manufactured by Apple, Inc. or Google, or Berkeley Software Distribution or Berkeley Standard Distribution (BSD) OS (including BSD derivatives). These operating environments can be used to execute one or more browsing programs, such as browsers manufactured by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. Furthermore, operating environments according to this principle can be used to execute one or more computer game programs.

[0026] A server and / or gateway may be used, which may include one or more processors executing instructions that configure the server to receive and send data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be a game console (e.g., Sony PlayStation). ® Instantiation of (such as personal computers, etc.)

[0027] Information can be exchanged between clients and servers over a network. For this purpose, and for security, servers and / or clients may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a means of implementing methods to provide network members with a secure community, such as an online social networking site or a gaming network.

[0028] A processor can be a single-chip or multi-chip processor that executes logic via various lines such as address lines, data lines, and control lines, as well as registers and shift registers. A processor that includes a digital signal processor (DSP) can be an embodiment of a circuit. A processor assembly can include one or more processors.

[0029] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged, or excluded from other embodiments.

[0030] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B, and C") includes systems having only A, only B, only C, A and B together, A and C together, B and C together and / or A, B and C together.

[0031] Now for reference Figure 1 An example system 10 is illustrated, which may include one or more of the example devices mentioned above and further described below in accordance with this principle. A first example device among the example devices included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, such as, but not limited to, a projector-based theater display system, or an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling a TV). The AVD 12 may alternatively be a computerized internet-enabled (“smart”) phone, tablet computer, laptop computer, head-mounted device (HMD) and / or head-mounted receiver (such as smart glasses or VR head-mounted receivers), another wearable computerized device, a computerized internet-enabled music player, a computerized internet-enabled headset, a computerized internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to take this principle (e.g., communicate with other CE devices to take this principle, perform the logic described herein, and perform any other functions and / or operations described herein).

[0032] Therefore, to achieve this principle, the AVD 12 can be constructed from some or all of the components shown. For example, the AVD 12 may include one or more touch-enabled displays 14, which may be implemented using a high-definition or ultra-high-definition "4K" or higher flat panel screen. The touch-enabled displays 14 may include, for example, a capacitive or resistive touch sensing layer having an electrode grid for touch sensing conforming to this principle.

[0033] AVD 12 may also include one or more speakers 16 for outputting audio according to these principles, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to control AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to adopt these principles, including other elements of AVD 12 described herein, such as controlling display 14 to present images thereon and receiving input from it. Furthermore, note that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as described above.

[0034] In addition to the above, the AVD 12 may also include one or more input and / or output ports 26, such as an HDMI port or a USB port, for physical connection to another CE device and / or a headphone port to connect headphones to the AVD 12 so that audio from the AVD 12 can be presented to the user through the headphones. For example, the input port 26 may be connected via a cable or satellite source 26a to the audio / video content, either wired or wirelessly. Thus, the source 26a may be a standalone or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or disc player containing content. When implemented as a game console, the source 26a may include some or all of the components described below with respect to CE device 48.

[0035] AVD 12 may also include one or more computer memory / computer-readable storage media 28, such as non-transient signal disk-based memory or solid-state memory, which in some cases are embedded in the AVD's chassis as a stand-alone device or personal video recording device (PVR) or video disk player located inside or outside the AVD's chassis for playing AV programs, or as removable storage media or servers described below. Furthermore, in some embodiments, AVD 12 may include a location or place receiver, such as, but not limited to, a mobile phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geolocation information from a satellite or mobile phone base station and provide that information to the processor 24 and / or determine the height at which the AVD 12 is placed with the processor 24.

[0036] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, IR sensors, event-based sensors, and / or cameras integrated into AVD 12 and controllable by processor 24 to collect pictures / images and / or videos in accordance with these principles. AVD 12 may also include a Bluetooth® transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0037] Furthermore, the AVD 12 may include one or more auxiliary sensors 38 that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors forming a layer of the touch-enabled display 14 itself, and may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Other sensor examples include pressure sensors, motion sensors (such as accelerometers, gyroscopes, tachometers) or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, event-based sensors, and gesture sensors (e.g., for sensing gesture commands). Thus, the sensor 38 may be implemented by one or more motion sensors (such as individual accelerometers, gyroscopes, and magnetometers and / or typically including combinations of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions) or event-based sensors (such as event detection sensors (EDS)). An EDS conforming to this disclosure provides an output indicating a change in light intensity sensed by at least one pixel of the light-sensing array. For example, if the light sensed by the pixel is decreasing, the output of EDS can be -1; if it is increasing, the output of EDS can be a+1. No change in light intensity below a certain threshold can be indicated by an output binary signal of 0.

[0038] The AVD 12 may also include an over-the-air (OTA) TV broadcast port 40 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, such as a kinetic energy harvester that can convert kinetic energy into electrical energy to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field-programmable gate array (FPGA) 46 may also be included. One or more tactile / vibration generators 47 may be provided to generate tactile signals that can be felt by a person holding or touching the device. Therefore, the haptic generator 47 can use an electric motor to vibrate all or part of the AVD 12, which is connected to an eccentric and / or unbalanced counterweight via a rotatable shaft, such that the shaft can be rotated under the control of the motor (which can in turn be controlled by a processor (e.g., processor 24)) to generate vibrations that simulate forces of various frequencies and / or amplitudes and in various directions.

[0039] It may also include a light source such as a projector (such as an infrared (IR) projector).

[0040] In addition to AVD 12, System 10 may include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to send computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or via a server described below, while the second CE device 50 may include components similar to the first CE device 48. In the example shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a heads-up transparent or opaque display for presenting AR / MR content or VR content (more generally, extended reality (XR) content), respectively. The HMD may be configured as a glasses-type display sold by a computer game device manufacturer or a larger VR-type display.

[0041] In the example shown, only two CE devices are illustrated; it should be understood that fewer or more devices may be used. The devices described herein may implement some or all of the components shown for AVD 12. Any components shown in the following figures may be combined with some or all of the components shown in the case of AVD 12.

[0042] Referring now to at least one server 52 described above, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (such as disk-based or solid-state memory), and at least one network interface 58. The network interface 58, under the control of the server processor 54, allows communication with other illustrated devices via network 22 and, in practice, facilitates communication between server and client devices according to this principle. Note that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as a wireless telephone transceiver.

[0043] Therefore, in some embodiments, server 52 may be an internet server or an entire server "farm" and may include and perform "cloud" functionality, enabling devices of system 10 to access a "cloud" environment via server 52 in an example embodiment for, for example, online gaming applications. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room as other devices shown or nearby.

[0044] The components shown in the following figures may include some or all of the components shown herein. Any user interface (UI) described herein may be combined and / or extended, and UI elements may be mixed and matched between UIs.

[0045] This principle can be applied to various machine learning models, including deep learning models. Machine learning models conforming to this principle can be trained using a variety of algorithms, including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms that can be implemented by computer circuits include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and RNN types known as long short-term memory (LSTM) networks. Generative pre-trained transformers (GPTTs) can also be used. Support vector machines (SVMs) and Bayesian networks can also be considered examples of machine learning models. In addition to the network types described above, the models in this paper can be implemented using classifiers.

[0046] As understood in this paper, performing machine learning can therefore involve accessing a model and then training it on training data so that the model can process further data to make inferences. Thus, an artificial neural network / AI model trained via machine learning can include an input layer, an output layer, and multiple hidden layers in between, configured and weighted to make inferences about the appropriate output.

[0047] Figure 2A system including a video encoder 200 for encoding / compressing video 202 is shown. A video decoder 204 can receive encoded video and decode / decompress it into output video 206.

[0048] Figure 3 The illustration schematically depicts example video objects and / or video regions within a video frame with associated priorities for encoding. For example, an object or region 300 located at the center of action in the video (such as a road in a racing game video) can be assigned the highest priority and thus encoded at the highest spatial resolution (e.g., 4K) and / or the highest temporal resolution (frame rate) (e.g., 60 frames per second (FPS)). On the other hand, an object or region 302 that is important to gameplay but less important than object or region 300 (such as a boss character) can be assigned the second highest priority and thus encoded at the second highest spatial resolution (e.g., 2K) and / or the second highest temporal resolution (e.g., 30 FPS).

[0049] Furthermore, objects or areas 304 that are less important to gameplay (such as a head-up display area showing game statistics) can be assigned the second-highest priority and are therefore encoded at the second-highest spatial resolution (e.g., HD) and / or the second-highest temporal resolution (e.g., 10 FPS). Again, lower-priority objects or areas 306, such as background / skybox areas of computer areas, can be associated with priorities corresponding to 5 FPS and SD, while the lowest-priority objects or areas 308, such as text, can be refreshed only once per second and encoded in SD.

[0050] Figure 4 The above concepts are further illustrated in flowchart form. Starting at state 400, priority is determined for each object and / or region in the frame. This is related to the spatial and / or temporal resolution at state 402, which is used to encode the individual objects / regions at state 404 for transmission and / or storage at state 406.

[0051] Figure 5 and Figure 6The above concepts are further illustrated in the form of a flowchart. Starting at state 500, a series of identifiers are used in the example system with only two priorities (low and high), where in state 502, higher priority is assigned to the target object (500), the object the user is looking at as determined by the gaze detection camera and software (504), the object the user has historically looked at during the video portion of the test as determined by the gaze detection camera and software (506), the object interacting with the computer game controller (600), the player-playable object (604), and the object signaled as high priority by the game engine (606). In state 608, the other objects are assigned lower priority.

[0052] Figure 7 State 700 shows that if an object or region is identified as a high-priority object or region, it can be encoded at a relatively high temporal resolution (e.g., 60 FPS) at state 702. In contrast, lower-priority objects / regions can be encoded at a lower temporal resolution (e.g., 30 FPS) at state 704.

[0053] Figure 8 It is shown that, alternatively, in state 800, if an object or region is identified as a higher priority object or region, it can be encoded at a relatively high spatial resolution (e.g., 4K) in state 802. Conversely, lower priority objects / regions can be encoded at a lower temporal resolution (e.g., 2K) in state 804.

[0054] Figure 9 An example is provided in which if an object is identified as a relatively low-priority object / region, such as a head-up display statistics area (900) or text (904), it can be encoded at a lower-priority resolution in state 902.

[0055] Figure 10 and Figure 11 This demonstrates a machine learning (ML)-based technique consistent with this principle. At state 1000, the object / region type and / or velocity, along with truth-based priorities, are input into the ML model to train the model at state 1002. Once trained, the ML model can be used, such as... Figure 11 As shown. In state 1100, during video playback, objects / regions in the video frames, along with their speeds (if needed), are sent to the ML model. The ML model outputs priorities and / or their equivalent encoding parameters, which are used to encode the objects / regions at state 1102.

[0056] Figure 12An exemplary, non-limiting decoder-side logic is illustrated, which can be used, for example, to restore the frame rate when an object is encoded at a relatively low frame rate. If it is determined that an object or region received from the transmitter over the network was not encoded at a low frame rate, for example, as indicated by signaling from the transmitter indicating the frame rate of the object, the logic ends at state 1202; otherwise, the logic moves to state 1204 to insert additional frames of the object between adjacent images of the object received from the transmitter using a super-frame rate model. Then, in state 1206, the object with the added frames is played back at a higher frame rate than during encoding. Note that this technique can be used only for higher-priority objects / regions to avoid reducing latency.

[0057] Figure 13 and Figure 14 This illustrates how the encoding resolution can be changed dynamically. From Figure 13 Starting at state 1300, network latency is identified. If the latency is low and / or reduced, the spatial and / or temporal resolution for all objects / regions can be increased at state 1302, regardless of priority. Alternatively, only higher-priority objects / regions can have their resolution increased at state 1302. On the other hand, if the latency is high and / or increased, the spatial and / or temporal resolution for all objects / regions can be decreased at state 1304, regardless of priority. Alternatively, only lower-priority objects / regions can have their resolution decreased at state 1304. The above logic can be reversed if needed; that is, high latency can cause the logic flow to state 1302, and low latency can cause the logic flow to state 1304.

[0058] from Figure 14 Starting at state 1400, the network's Quality of Service (QoS) is identified. If the QoS is low and / or degraded, the spatial and / or temporal resolution for all objects / areas can be increased at state 1402, regardless of priority. Alternatively, only higher-priority objects / areas can have one or more of their resolutions increased at state 1402. Conversely, if the QoS is high and / or improved, the spatial and / or temporal resolution for all objects / areas can be decreased at state 1404, regardless of priority. Alternatively, only lower-priority objects / areas can have their resolutions decreased at state 1404. The above logic can be reversed if needed; that is, increasing QoS can cause a logical flow to state 1402, and decreasing QoS can cause a logical flow to state 1404.

[0059] This technique can be achieved by segmenting each frame according to its object / region and sending different parts of the frame at different frame rates. The different parts can overlap each other at the receiver. Object-level coding can be used to improve latency, allowing only a portion of the full frame to be encoded at a high frame rate, while other parts are encoded at a lower frame rate.

[0060] While specific techniques are illustrated and described in detail herein, it should be understood that the subject matter covered by this application is limited only by the claims.

Claims

1. An apparatus comprising: At least one processor component is configured as follows: In response to a first object or region in the video having a first priority, the first object or region is encoded at a first frame rate; and In response to a second object or region in the video having a second priority, the second object or region is encoded at a second frame rate.

2. The device according to claim 1, wherein the video includes computer game videos.

3. The device of claim 1, wherein the first priority is higher than the second priority, and the first frame rate is faster than the second frame rate.

4. The apparatus according to claim 1, wherein, The processor component is configured as follows: This enables the first object or region and the second object or region to be transmitted over a network after encoding.

5. The apparatus according to claim 1, wherein, The processor component is configured as follows: In response to the first object or region in the video having the first priority, the first object or region is encoded at a first resolution; and In response to the second object or region in the video having the second priority, the second object or region is encoded at the second resolution.

6. The apparatus according to claim 1, wherein, The processor component is configured as follows: The first priority is determined at least in part by the following operations: The first object or region is identified as the target.

7. The apparatus according to claim 1, wherein, The processor component is configured as follows: The first priority is determined at least in part by the following operations: The first object or area is being viewed by the player.

8. The apparatus according to claim 1, wherein, The processor component is configured as follows: The first priority is determined at least in part by the following operations: Identify that the first object or region has been viewed by multiple players in history.

9. The apparatus according to claim 1, wherein, The processor component is configured as follows: The first priority is determined at least in part by the following operations: The system identifies that the first object or region is being interacted with by a computer game controller.

10. The apparatus according to claim 1, wherein, The processor component is configured as follows: The first priority is determined at least in part by the following operations: The first object or area is identified as a playable object.

11. The apparatus according to claim 1, wherein, The processor component is configured as follows: The first priority is determined at least in part by the following operations: The game engine signals that the first object or region has the first priority.

12. The apparatus according to claim 1, wherein, The processor component is configured as follows: The second priority is determined at least in part by the following operations: The first object or region is identified as a statistical display area.

13. The apparatus according to claim 1, wherein, The processor component is configured as follows: The second priority is determined at least in part by the following operations: The first object or region is identified as including text.

14. An apparatus comprising: At least one computer medium, said at least one computer medium being not a transient signal and including instructions executable by at least one processor component to: In response to a first object or region in the video having a first priority, the first object or region is encoded at a first resolution; and In response to a second object or region in the video having a second priority, the second object or region is encoded at a second resolution.

15. The device according to claim 14, wherein, The videos include computer game videos.

16. The apparatus according to claim 14, wherein, The first priority is higher than the second priority, and the first resolution is greater than the second resolution.

17. The apparatus according to claim 14, wherein, The instructions are executable to perform the following operations: In response to the first object or region in the video having the first priority, the first object or region is encoded at a first frame rate; and In response to the second object or region in the video having the second priority, the second object or region is encoded at the second frame rate.

18. A method comprising: Encode a first object in the video with a first spatial and / or temporal resolution, wherein the first object has a higher priority than a second object in the video; and The second object is encoded at a second spatial and / or temporal resolution lower than the first spatial and / or temporal resolution.

19. The method of claim 18, further comprising encoding the first object and the second object at corresponding first spatial resolution and second spatial resolution.

20. The method of claim 18, further comprising encoding the first object and the second object at corresponding first and second time resolutions.

Images