Display apparatus, server apparatus and control method thereof

Abstract

A display device is disclosed. The display device includes a communication interface, a display, a memory for storing at least one instruction, and at least one processor. The at least one processor may control the display to: transmit available range information including a minimum value and a maximum value of a variable refresh rate (VRR) to an external server via the communication interface; and displaying the variable frequency image based on the variable frame rate information when a content including the variable frequency image rendered based on the available range information and the network bandwidth information and the variable frame rate information corresponding to the variable frequency image is received from the external server via the communication interface.

Description

Technical Field

[0001] The present disclosure relates to a display device, a server device, and a control method, and more particularly, to a display device that receives streaming content, a server device that streams content, and a control method thereof. Background Art

[0002] Broadcast and Over-the-top (OTT), cloud gaming, mirror servers, etc. are services that transmit a compressed image stream from a server system (e.g., a server system that performs a server role) to a client system for display to a user. The client can receive data and demands from the user and send them to the server, and the server can provide a service that matches the received user data.

[0003] For example, in the case of a cloud gaming service, a gaming service can be provided without being greatly affected by the capabilities of the client system (e.g., the performance of a central processing unit (CPU), a graphics processing unit (GPU), etc.). Therefore, multi-platform services are possible on various terminals (such as, for example but not limited to, a television (TV), a smart phone, etc.) in addition to a personal computer (PC). However, providing such multi-platform services may be limited by factors such as, for example, Internet speed, network quality, server processing speed, transmission / processing latency, etc. Technologies such as enhancement of Internet speed and network quality, enhancement of server processing speed, minimization of latency time, etc. are needed. Summary of the Invention

[0004] Technical Solution

[0005] According to an aspect of the present disclosure, a display device includes: a communication interface; a display; a memory storing at least one instruction; and at least one processor operably connected to the communication interface, the display, and the memory. The at least one processor is configured to execute the at least one instruction to: send available range information including a minimum refresh rate and a maximum refresh rate of a Variable Refresh Rate (VRR) to an external server through the communication interface, receive content including a variable frequency image rendered based on the available range information and network bandwidth information and variable frame rate information corresponding to the variable frequency image from the external server through the communication interface, and control the display to display the variable frequency image based on the variable frame rate information based on the received content.

[0006] The at least one processor may further be configured to execute the at least one instruction to: control the display to be set in a VRR output mode based on an information frame in the received content.

[0007] At least one processor may also be configured to execute at least one instruction to: identify at least one of a minimum refresh rate or a maximum refresh rate that needs to be changed based on at least one of a stream buffer state or a network state; and send adjusted available range information to an external server, the adjusted available range information including at least one of a changed minimum refresh rate or a changed maximum refresh rate.

[0008] The external server may be configured to provide a cloud gaming service, and at least one processor may also be configured to execute at least one instruction to: send a request for game content to the external server; receive a signal for verifying VRR operability from the external server based on the request for game content; and send available range information to the external server based on the signal for verifying VRR operability.

[0009] According to one aspect of the present disclosure, a server device includes: a communication interface; a memory storing at least one instruction; and at least one processor operably connected to the communication interface and the memory. The at least one processor is configured to execute at least one instruction to: receive available range information from a client device through the communication interface, the available range information including a minimum refresh rate and a maximum refresh rate of a variable refresh rate (VRR), render a variable frequency image based on the available range information and network bandwidth information, and send content including the rendered variable frequency image and variable frame rate information corresponding to the variable frequency image to the client device through the communication interface.

[0010] At least one processor may also be configured to execute at least one instruction to: identify a resolution for rendering a variable frequency image based on network bandwidth information; identify a frame rate for rendering a variable frequency image based on the available range information; and render a variable frequency image based on the identified resolution and the identified frame rate.

[0011] At least one processor may also be configured to execute at least one instruction to: render a variable frequency image by varying a frame rate while maintaining a resolution based on network bandwidth information.

[0012] At least one processor may also be configured to execute at least one instruction to: identify a frame rate of a variable frequency image within an available range including the minimum refresh rate and the maximum refresh rate of VRR based on a frequency of a user control command received from the client device.

[0013] The server device may be configured to provide a cloud gaming service, and at least one processor may also be configured to execute at least one instruction to: receive a request for game content from the client device; send a signal for verifying VRR operability to the client device based on the request for game content; and receive available range information from the client device based on the signal for verifying VRR operability.

[0014] According to one aspect of the present disclosure, a method for controlling a display device includes: sending available range information to an external server, the available range information including a minimum refresh rate and a maximum refresh rate of a variable refresh rate (VRR); receiving content from the external server, the content including a variable frequency image rendered based on the available range information and network bandwidth information and variable frame rate information corresponding to the variable frequency image; and outputting the variable frequency image on a display based on the variable frame rate information.

[0015] The method for controlling the display device may further include: setting the display to a VRR output mode based on an information frame in the received content.

[0016] The method for controlling the display device may further include: identifying whether at least one of the minimum refresh rate or the maximum refresh rate needs to be changed based on at least one of a streaming buffer state or a network state; and sending adjusted available range information to the external server based on identifying that at least one of the minimum refresh rate or the maximum refresh rate needs to be changed, the adjusted available range information including at least one of a changed minimum refresh rate and a changed maximum refresh rate.

[0017] The external server may be configured to provide a cloud gaming service, and sending the adjusted available range information to the external server may include: sending a request for game content to the external server; receiving a signal for verifying VRR operability from the external server based on the request for game content; and sending the adjusted available range information to the external server based on the signal for verifying VRR operability.

[0018] According to one aspect of the present disclosure, a method for controlling a server device includes: receiving available range information from a client device, the available range information including a minimum refresh rate and a maximum refresh rate of a variable refresh rate (VRR); rendering a variable frequency image based on the available range information and network bandwidth information; and sending content to the client device, the content including the rendered variable frequency image and variable frame rate information corresponding to the variable frequency image.

[0019] The method for controlling the server device may further include: identifying a resolution for rendering the variable frequency image based on the network bandwidth information; identifying a frame rate for rendering the variable frequency image based on the available range information; and rendering the variable frequency image based on the identified resolution and the identified frame rate.

[0020] The method for controlling the server device may further include: rendering the variable frequency image by varying the frame rate while maintaining the resolution based on the network bandwidth information.

[0021] The method for controlling a server device may further include: receiving a request for game content from a client device; based on the request for game content, sending a signal for verifying VRR operability to the client device; and based on the signal for verifying VRR operability, receiving available range information from the client device.

[0022] The server device may be configured to provide a cloud gaming service, and rendering a variable frequency image may include: receiving a request for game content from a client device, based on the request for game content, sending a signal for verifying VRR operability to the client device; and based on the signal for verifying VRR operability, receiving available range information from the client device.

[0023] According to an aspect of the present disclosure, a non-transitory computer-readable medium stores computer-readable program code or instructions, which can be executed by a processor to execute a method for controlling a display device. The method includes: sending available range information to an external server, the available range information including a minimum refresh rate and a maximum refresh rate of a variable refresh rate (VRR), receiving content from the external server, the content including a variable frequency image rendered based on the available range information and network bandwidth information and variable frame rate information corresponding to the variable frequency image, and outputting the variable frequency image on a display based on the variable frame rate information.

[0024] According to an aspect of the present disclosure, a non-transitory computer-readable medium stores computer-readable program code or instructions, which can be executed by a processor to execute a method for controlling a server device. The method includes: receiving available range information from a client device, the available range information including a minimum refresh rate and a maximum refresh rate of a variable refresh rate (VRR); rendering a variable frequency image based on the available range information and network bandwidth information; and sending content to the client device, the content including the rendered variable frequency image and variable frame rate information corresponding to the variable frequency image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram schematically showing a real-time image streaming system according to one or more embodiments;

[0026] Figure 2a is a block diagram showing a configuration of a display device according to one or more embodiments;

[0027] Figure 2b is a block diagram showing in detail a configuration of a display device according to one or more embodiments;

[0028] Figure 2c is a diagram showing a content processing method of a display device according to one or more embodiments

[0029] Figure 3 is a diagram showing a control method of a display device according to one or more embodiments;

[0030] Figure 4 is a diagram showing a control method of a display device according to one or more embodiments;

[0031] Figure 5 is a diagram showing a configuration of a server device according to one or more embodiments;

[0032] Figure 6 is a diagram showing an operation of a server device according to one or more embodiments;

[0033] Figure 7 is a diagram showing an operation of a server device according to one or more embodiments; and

[0034] Figure 8 is a sequence diagram showing a signal flow between a display device and a server device according to one or more embodiments. Detailed Description

[0035] Terms used in the present disclosure will be briefly described, and the present disclosure will be described in detail.

[0036] The terms used in the present disclosure are general terms widely used in consideration of their functions herein. However, the terms may change according to the intention of those skilled in the relevant art, legal or technical interpretations, emergence of new technologies, etc. In some cases, there may be terms selected arbitrarily, and in such cases, the meaning of the term will be disclosed in more detail in the corresponding description. Therefore, the terms in the present disclosure are only used to describe specific embodiments, and it should be understood that these terms are not intended to limit the scope of the present disclosure.

[0037] In the present disclosure, expressions such as "have", "may have", "include", "may include", etc. are used to specify the existence of corresponding characteristics (e.g., elements such as numerical values, functions, operations, or components), and do not exclude the existence or possibility of additional characteristics.

[0038] In the present disclosure, expressions such as "A or B", "at least one of A and / or B", or "one or more of A and / or B" may include all possible combinations of the items listed together. For example, "A or B", "at least one of A and B", or "at least one of A or B" may refer to all cases including (1) only A, (2) only B, or (3) both A and B.

[0039] Expressions such as "first", "second", "1st", "2nd", etc. used in this document can be used to refer to various elements, regardless of order and / or importance. In addition, it should be noted that these expressions are only used to distinguish one element from another element, rather than limiting the relevant elements.

[0040] When an element (e.g., a first element) is indicated as "operably or communicatively coupled" / "operably or communicatively coupled to" another element (e.g., a second element) or "connected to" another element (e.g., a second element), it can be understood that an element is directly coupled / directly coupled to another element or coupled through other elements (e.g., a third element).

[0041] The expression "configured to..." (or "set to...") used in this disclosure can be interchangeably used with, for example, "suitable for...", "capable of...", "designed to...", "adapted to...", "manufactured to...", or "able to..." based on the situation. The term "configured to..." (or "set to") may not necessarily mean "specifically designed" in terms of hardware.

[0042] In some cases, the expression "a device configured to..." may mean that the device "can perform..." together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, or C" may mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) capable of performing the corresponding operations by executing one or more software programs stored in a memory device.

[0043] Unless otherwise specified, singular expressions include plural expressions. It should be understood that terms such as "formed" or "including" are used herein to specify the existence of a feature, quantity, step, operation, element, component, or a combination thereof, and do not exclude the existence or possibility of adding one or more of other features, quantities, steps, operations, elements, components, or a combination thereof.

[0044] The term "module" or "portion" used in one or more embodiments herein performs at least one function or operation and can be implemented in hardware or software, or in a combination of hardware and software. In addition, multiple "modules" or multiple "portions" other than those that need to be implemented into specific hardware can be integrated into at least one module and implemented in at least one processor.

[0045] Various elements and regions of the drawings have been schematically shown. Therefore, the technical spirit of this disclosure is not limited by the relative sizes and distances shown in the drawings.

[0046] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, in which like reference numerals always denote corresponding features consistently.

[0047] Figure 1 FIG. is a diagram schematically showing a real-time image streaming system according to one or more embodiments of the present disclosure.

[0048] According to Figure 1 the illustrated embodiment, the real-time image streaming system may include a client device 10 and a server device 20.

[0049] In order to perform streaming of high-definition and high-resolution images such as 4K and 8K over a network, image coding techniques that can reduce the network request bandwidth and upscaling / downscaling may be important. For image coding techniques, standard codecs such as H.264 / 265, VP8 / 9, and AV1 are widely used, and OTT providers can provide services by compressing 4K images up to approximately 15 Mbps using H.265. In order to serve each user according to different network environments, it is necessary to compress images into various combinations of image resolutions and transmission rates, and the technique used at this time may be upscaling / downscaling. For example, when attempting to send an 8K image at a level of approximately 15 Mbps, the server device 20 may perform downscaling of the image, encode the downscaled image, and send it to the client device 10. The client device 10 may perform decoding of the image received from the server device 20 and display the decoded image. Additionally, the client device 10 may perform upscaling of the decoded image and display the image when necessary.

[0050] In the case of a real-time image streaming service, the server device 20 may perform real-time downscaling and encoding of the image based on a real-time selected resolution and / or compression bit rate. For example, real-time image streaming services may include cloud gaming services, live streaming services, mirroring services, video communication services, etc.

[0051] In one embodiment, if a cloud gaming service is provided, the server device 20 may perform streaming of game images and audio to the client device 10, as Figure 1 shown. The client device 10 may perform video / audio decoding 13 of the received game images and audio by being streamed received 12 based on the game images through the communication function 11, and output 14 through a display and speakers. The client device 10 may send a game play control signal corresponding to the received user input to the server device 20 based on the user input being received.

[0052] The server device 20 can perform game input processing 22 based on the gameplay control signal received from the client device 10 through the client receiving function 21, and update the game state 23 based on the processed game input. Additionally, the server device 20 can perform graphics rendering 24 of the game images based on the updated game state, and perform video compression 25 (and audio compression) of the rendered images. Furthermore, the server device 20 can send the video-compressed images (and audio-compressed audio) to the client device 10 through the streaming function 26.

[0053] According to an embodiment, for the above cloud game service, content with a high scan rate greater than or equal to 60 Hz may be preferred, but due to the delay of user input in deteriorating situations such as the load in the server device 20, temporary transmission failures in the network, or reduced bandwidth, it may lead to a decline in game performance.

[0054] Therefore, various embodiments for minimizing transmission delay while maintaining quality according to the network state and / or the characteristics of the client device in a low-latency streaming service such as a cloud game service will be described below.

[0055] Figure 2a is a block diagram showing the configuration of a display device according to one or more embodiments.

[0056] Referring to Figure 2a , the display device 100 may include a communication interface 110, a display 120, a memory 130, and at least one processor 140.

[0057] According to an embodiment, the communication interface 110 can support various communication methods. For example, the communication interface 110 can perform communication with external devices, external storage media (e.g., universal serial bus (USB) memory), external servers (e.g., cloud servers), etc. through communication methods such as, for example but not limited to, Bluetooth, AP-based Wi-Fi (e.g., Wi-Fi, wireless LAN network), ZigBee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE 1394, High-Definition Multimedia Interface (HDMI), USB, Mobile High-Definition Link (MHL), Audio Engineering Society / European Broadcasting Union (AES / EBU), optical, coaxial, etc.

[0058] The display 120 may be implemented as a display including self-emissive devices or a display including non-emissive devices and a backlight. The display 120 may be implemented in various forms of displays, such as but not limited to, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a light emitting diode (LED), a micro LED, a mini LED, a plasma display panel (PDP), a quantum dot (QD) display, a quantum dot light emitting diode (QLED), etc. In the display 120, a driving circuit, a backlight unit, etc. may be included, which may be implemented in the form of, for example, an a-si thin film transistor (TFT), a low temperature polycrystalline silicon (LTPS) TFT, an organic TFT (OTFT), etc. In an embodiment, a touch sensor may be disposed on the front surface of the display 120, which detects a touch operation in the form of, for example, but not limited to, a touch film, a touch sheet, a touch pad, etc., and is implemented to detect various types of touch inputs. For example, the display 120 may detect various types of touch inputs, such as but not limited to, for example, a touch input of a user's hand, a touch input of an input device such as a stylus, a touch input of a specific electrostatic material, etc. Here, the input device may be implemented as a pen-shaped input device, which may be referred to by various terms such as but not limited to, for example, an electronic pen, a stylus, an S pen, etc. In an embodiment, the display 120 may be implemented as a flat display, a curved display, a foldable and / or rollable flexible display, etc.

[0059] The memory 130 may store data required for various embodiments. The memory 130 may be implemented in the form of a memory embedded in the display device 100' according to data storage usage, or in the form of a memory that can be attached to or detached from the display device 100. For example, data for driving the display device 100 may be stored in the memory embedded in the display device 100', and data for the extended functions of the display device 100 may be stored in a memory that can be attached to or detached from the display device 100. According to an embodiment, the memory embedded in the display device 100 may be implemented as at least one of a volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)) or a non-volatile memory (e.g., one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard disk drive (HDD), or solid state drive (SSD)). Additionally, the memory that can be attached to or detached from the display device 100 may be implemented in the form of, for example but not limited to, a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (micro-SD), mini secure digital (mini-SD), extreme digital (xD), multimedia card (MMC), etc.), an external memory connectable to a USB port (e.g., a USB memory), etc.

[0060] At least one processor 140 may control the overall operation of the display device 100. Specifically, at least one processor 140 may control the overall operation of the display device 100 by connecting to each configuration of the display device 100. For example, at least one processor 140 may control the overall operation of the display device 100 by being electrically connected to the display 120 and the memory 130. At least one processor 140 may be formed as one or more processors.

[0061] At least one processor 140 may perform the operations of the display device 100 according to various embodiments by executing at least one instruction stored in the memory 130.

[0062] At least one processor 140 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The at least one processor 140 may control one or a random combination of other elements of the display device and perform operations associated with communication or data processing. The at least one processor 140 may execute at least one program or instruction stored in the memory. For example, the at least one processor may execute a method according to one or more embodiments of the present disclosure by executing at least one instruction stored in the memory.

[0063] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be executed by one processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are executed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be executed by a first processor, or the first operation and the second operation may be executed by a first processor (e.g., a general-purpose processor), and the third operation may be executed by a second processor (e.g., a processor dedicated to artificial intelligence).

[0064] The at least one processor 140 may be implemented as a single-core processor including one core or as at least one multi-core processor including a plurality of cores (e.g., a homogeneous multi-core or a heterogeneous multi-core). If the at least one processor 140 is implemented as a multi-core processor, each of the cores included in the multi-core processor may include memories inside the processor, such as cache memories and on-chip memories, and a common cache shared by the plurality of cores may be included in the multi-core processor. In addition, each of the plurality of cores (or a part of the plurality of cores) included in the multi-core processor may independently read and execute program commands for implementing a method according to one or more embodiments, or may read and execute program commands for implementing a method according to one or more embodiments of the present disclosure due to the interconnection of the whole (or a part) of the plurality of cores.

[0065] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be executed by one of the plurality of cores or by the plurality of cores included in the multi-core processor. For example, when a first operation, a second operation, and a third operation are executed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be executed by a first core included in the multi-core processor, or the first operation and the second operation may be executed by a first core included in the multi-core processor, and the third operation may be executed by a second core included in the multi-core processor.

[0066] According to one or more embodiments, a processor may refer to a system-on-chip (SoC) integrating at least one processor and other electronic components, a single-core processor or a multi-core processor, or a core included in a single-core processor or a multi-core processor, and the cores herein may be implemented as a CPU, GPU, APU, MIC, NPU, hardware accelerator, machine learning accelerator, etc., but are not limited to the embodiments of the present disclosure. For ease of description, hereinafter at least one processor 140 may be referred to as processor 140.

[0067] Figure 2b is a block diagram showing in detail the configuration of a display device according to one or more embodiments.

[0068] Referring to Figure 2b , the display device 100' may include a communication interface 110, a display 120, a memory 130, at least one processor 140, a user interface 150, a speaker 160, and a camera 170. The configurations overlapping with those in Figure 2b the configuration shown will be omitted Figure 2a the detailed description of the configuration overlapping with the configuration shown.

[0069] The user interface 150 may be implemented using devices such as buttons, touchpads, mice, and keyboards, or may be implemented as a touch screen capable of performing the above display functions and operation input functions together with them.

[0070] The speaker 160 may be a configuration that not only outputs various audio data but also outputs various notification sounds, voice messages, etc. The processor 140 may control the speaker to output information corresponding to a user interface (UI) screen or various notifications according to various embodiments of the present disclosure in audio form.

[0071] The camera 170 may be turned on and perform capture according to a preset event. The camera 170 may convert the captured image into an electrical signal and generate image data based on the converted signal. For example, an object may be converted into an electrical image signal through a semiconductor charge-coupled device (CCD), and the converted image signal as described above may be signal-processed after being amplified and converted into a digital signal.

[0072] In an embodiment, the display device 100' may include a microphone, a sensor, a tuner, a demodulator, etc.

[0073] The microphone may be a configuration for receiving the input of user voice or other sounds and converting them into audio data. However, the display device 100' according to another embodiment may receive user voice input through the external device through the communication interface 110.

[0074] The sensor may include various types of sensors, such as but not limited to a touch sensor, a proximity sensor, an acceleration sensor, a geomagnetic sensor, a gyro (gryro) sensor, a pressure sensor, a position sensor, an illuminance sensor, etc.

[0075] The tuner may receive a radio frequency (RF) broadcast signal by tuning to a channel selected by a user from the RF broadcast signals received through an antenna or all pre-stored channels.

[0076] The demodulator may receive the digital intermediate frequency (IF) signal converted from the tuner and perform demodulation, channel decoding, etc.

[0077] Figure 2c It is a diagram showing a content processing method of a display device according to one or more embodiments.

[0078] Reference Figure 2c , the content being streamed in real time to the display device 100 may be stored in a streaming buffer 41 (or a receive buffer). Here, the content being streamed may be content encoded from an external server (server device 20, Figure 1 ). Accordingly, the content stored in the streaming buffer 41 may be provided to a decoder 42 and decoded, and the decoded content may be output through a display 120 while being buffered in an output buffer 43. Here, the decoder 42 may be implemented in a type in the form of a DSP.

[0079] Figure 3 It is a diagram showing a control method of a display device according to one or more embodiments.

[0080] Reference Figure 3 , at operation S310, the processor 140 may send variable refresh rate (VRR) available range information to an external server. The VRR available range information may include at least one of a minimum value and a maximum value of the VRR. For example, the VRR available range information may include both the minimum value and the maximum value of the VRR. In an embodiment, the minimum value and the maximum value of the VRR may be values preset according to the panel characteristics of the display 120. However, the above is not limited thereto, and may be values modified / changed according to the network state at the time of transmission. According to an embodiment, if only the minimum value (or the maximum value) is included in the VRR available range information, the external server may identify the VRR available range information of the display device 100 by applying a predefined value with respect to the maximum value (or the minimum value).

[0081] An external server can provide cloud gaming services. According to an example, the processor 140 can send a game service request (i.e., a game content request) to the external server according to a user command, and send available range information to the external server based on a signal received for verifying VRR operability. However, the above is not limited thereto, and the available range information can be sent to the external server together with the game content request, or the available range information can be sent to the external server even if a separate signal for verifying VRR operability is not received after the game content request.

[0082] At operation S320, the processor 140 can receive a variable frequency image rendered based on the available range information and network bandwidth information from the external server, and content including frame rate information corresponding to the variable frequency image. For example, the frame rate information can be included in the metadata, but is not limited thereto.

[0083] In an embodiment, the processor 140 can be recognized based on the variable frequency image as being sent from the external server to control the display 120 to be set in the VRR output mode. For example, the processor 140 can recognize the transmission of the variable frequency image based on the information included in the information frame included in the content received from the external server, and control the display 120 to be set in the VRR output mode. For example, the information frame can include information indicating that the variable frequency image can be sent and that the variable frequency image can be sent through a partial area of the image frame.

[0084] In an embodiment, the display device 100 can be implemented to support multiple output modes, such as, for example, the VRR output mode, the frame rate control (FRC) output mode, and the normal output mode. In this case, the processor 140 can be recognized based on the variable frequency image as being sent from the external server to control the display 120 to be set in the VRR output mode among the multiple output modes.

[0085] At operation S330, the processor 140 can control the display 120 to display the variable frequency image based on the variable frame rate information. Here, the variable frame rate information can be included in the above information frame or in the header of the transmission packet. For example, since the image resolution and frame rate can be changed to at least one frame unit, it may be necessary for the processor 140 to confirm the transmitted image resolution and frame rate. Specifically, the processor 140 can receive the input image in the form of a packet, obtain the resolution information and frame rate included in the header area by de-packeting the packet, and obtain the encoded image included in the payload area. Alternatively, the processor 140 can obtain the image resolution information during the decoding process.

[0086] In an embodiment, the processor 140 may obtain an output image by performing image processing on a received image. The image processing may involve digital image processing of the received image, which includes at least one of image enhancement, image restoration, image transformation, image analysis, image understanding, or image compression. If the input image is a compressed image, the processor 140 may perform image processing after decoding the compressed image.

[0087] Figure 4 FIG. is a diagram illustrating a control method of a display device according to one or more embodiments.

[0088] Referring to Figure 4 , at operation S410, the processor 140 may send available range information including the minimum and maximum values of the VRR to an external server. Operation S410 may be the same as or similar to Figure 3 operation S310 shown.

[0089] At operation S420, the processor 140 may receive content including a variable frequency image rendered based on the available range information and network bandwidth information and variable frame rate information corresponding to the variable frequency image from the external server. Operation S420 may be the same as or similar to Figure 3 operation S320 shown.

[0090] At operation S430, the processor 140 may control the display 120 to display the variable frequency image based on the variable frame rate information. Operation S430 may be the same as or similar to Figure 3 operation S330 shown.

[0091] At operation S440, the processor 140 may identify whether at least one of the minimum or maximum value of the VRR needs to be changed based on at least one of the state of the streaming buffer or the network state.

[0092] For example, the processor 140 may identify whether at least one of the minimum or maximum values of the VRR needs to be changed based on the network state. Here, the network state information may include at least one of the available bandwidth of the network, the packet loss rate, the round-trip time of the packet, the delay gradient of the packet, the received signal strength indicator (RSSI) information, the communication channel information, the link speed information, the channel interference information, or the retry rate information. That is, the network state information may be information associated with various networks affected by network congestion, distance, and transmission rate. The round-trip time (or round-trip delay) of a packet may refer to the time taken for the packet to make a round trip from the network to the receiving end. The delay gradient of a packet may be the difference between the transmission time interval of the packet sent from the sending end and the reception time interval of the packet received from the receiving end. The communication channel information may include information about at least one of the channel and the frequency. The link speed information may be information about the rate of the packet sent from the sending end to the receiving end. The channel interference information may be obtained based on the interference factor of each channel measured using various channel RF parameters such as the channel active time, the channel busy time, and the channel transmission time. The retry rate information may indicate the percentage of transmitted packets that are retried on a second basis.

[0093] For example, the processor 140 may identify whether at least one of the minimum or maximum values of the VRR needs to be changed based on the streaming buffer 41 ( Figure 2c ). For example, the processor 140 may reduce at least one of the minimum or maximum values of the VRR based on the stream stored in the streaming buffer 41 being identified as less than a first threshold. Additionally, the processor 140 may increase at least one of the minimum or maximum values of the VRR based on the stream stored in the streaming buffer being identified as exceeding a second threshold. For example, the first threshold may be a minimum threshold, and the second threshold may be a maximum threshold.

[0094] In an embodiment, the processor 140 may set a first threshold and a second threshold based on content latency characteristic information. The latency characteristic information may be information identified based on the characteristics of the content (or source content). For example, when classifying which type of content requires which latency characteristics, methods such as classification according to the input source such as a game device / set-top box / BDP or using information stored in a content database (DB) or directly analyzing the image may be used. For example, since even within one game content, there may be parts that require low latency and parts that do not require low latency when the user is directly playing and are focused on viewing, the classification may change per chapter or scene. In an embodiment, the external server may identify the latency characteristic information based on the characteristics of the content (or source content). For example, the latency characteristic information may be information that only shows whether there is a low latency request, information in which the degree of the low latency request is quantified, or level information that rates the degree of the low latency request. That is, the latency characteristic information may be various types of information that can show the low latency request of the content.

[0095] For example, the processor 140 may increase the second threshold of the streaming buffer, that is, the maximum threshold (or the allowed maximum value), based on the content latency characteristic information being identified as allowing large latency, and ensure the stability of allowing large latency. Alternatively, the processor 140 may identify the first threshold and / or the second threshold of the streaming buffer 41 by considering not only the content latency characteristic information but also the network state. For example, if the change in the network state is severe, both the first threshold and the second threshold of the streaming buffer 41 may be set relatively large. Additionally, if it is content with low latency characteristics (i.e., game content where the response rate to the user is very important), both the first threshold and the second threshold may be set relatively small.

[0096] At operation S450, the processor 140 may send available range information in which at least one of the minimum value or the maximum value of the VRR is changed based on at least one of the state of the streaming buffer or the network state to the external server when at least one of the minimum value or the maximum value of the VRR is identified as needing to be changed (S440: Yes).

[0097] Then, the processor 140 may receive and display the rendered image based on the changed available range information.

[0098] Figure 5 FIG. is a diagram showing the configuration of a server device according to one or more embodiments.

[0099] Reference Figure 5 , the server device 200 may include a communication interface 210, a memory 220, and at least one processor 230. In an embodiment, the server device 200 may be implemented as Figure 1The server device 20 shown.

[0100] According to an embodiment, the communication interface 210 may support various communication methods. For example, the communication interface 210 may perform communication with an external device (e.g., the display device 100) through communication methods such as, for example but not limited to, Bluetooth, AP-based Wi-Fi (e.g., Wi-Fi, wireless LAN network), ZigBee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE 1394, high-definition multimedia interface (HDMI), etc.

[0101] The memory 220 may store data required for one or more embodiments of the present disclosure. The memory 220 may be the same as or similar to Figure 2a the memory 130 shown, and its detailed description will be omitted.

[0102] At least one processor 230 may control the overall operation of the server device 200. Specifically, at least one processor 230 may control the overall operation of the server device 200 connected to each configuration of the server device 200. For example, at least one processor 230 may control the overall operation of the server device 200 by being electrically connected to the memory 220. At least one processor 230 may be formed of one or more processors. Since the embodiment of at least one processor 230 is the same as or similar to Figure 2a the embodiment of at least one processor 140 shown, its detailed description will be omitted. For ease of description, hereinafter at least one processor 230 may be referred to as processor 230.

[0103] According to an embodiment, the server device 200 may provide a cloud game service. For example, the server device 200 may perform preprocessing such as compression and optimization in an appropriate media form to continuously convert a game-driven screen into a media format such as a moving image scene, and send the media file generated through the preprocessing to a client device (e.g., Figure 2a the display device 100 in). In this case, the media file received from the application side of the client device may be played back, and a cloud game service may be provided to the user. In addition, the server device 200 may update the game-driven screen based on the game play control of the user received from the client device, and send the updated game-driven screen back to the client device.

[0104] According to an embodiment, the server device 200 may send images of various resolutions and various compressed images to the client device. For example, the server device 200 may send at least one image having a resolution greater than or equal to Standard Definition (SD), High Definition (HD), Full High Definition (FHD), or Ultra High Definition (UHD). Additionally, the server device 200 may receive images compressed using, for example but not limited to, Moving Picture Experts Group (MPEG) (e.g., MP2, MP4, MP7, etc.), Joint Photographic Coding Experts Group (JPEG), Advanced Video Coding (AVC), H.264, H.265, High Efficiency Video Codec (HEVC), VC-1, VP8, VP9, AOMedia Video 1 (AV1), etc.

[0105] Figure 6 is a diagram showing the operation of a server device according to one or more embodiments.

[0106] Referring Figure 6 , at operation S610, the processor 230 may receive available range information including the minimum and maximum values of the VRR from the client device (e.g., Figure 2a the display device 100 in).

[0107] In an embodiment, the processor 230 may, based on a game content request received from the client device, send a signal for verifying VRR operability to the client device and receive the available range information in response to the signal for verifying VRR operability from the client device. However, the above is not limited thereto, and the available range information may be received from the client device together with the game content request, or the available range information continuing from the game content request may be received even if a separate signal for verifying VRR operability is not sent to the client device.

[0108] At operation S620, the processor 230 may perform rendering of variable-frequency images based on the available range information received from the client device and the network bandwidth information.

[0109] At operation S623, the processor 230 may send content including the rendered variable-frequency image and variable frame rate information corresponding to the variable-frequency image to the client device. For example, the frame rate information may be included in the metadata, but is not limited thereto.

[0110] Figure 7 is a diagram illustrating operations of a server device according to one or more embodiments.

[0111] Refer to Figure 7 , at operation S710, the processor 230 may receive available range information including the minimum and maximum values of the VRR from the client device.

[0112] In an embodiment, the processor 230 may send a signal for verifying the VRR operability to the client device based on a game content request received from the client device, and receive the available range information as a response to the signal for verifying the VRR operability from the client device. However, the above is not limited thereto, and the available range information may be received from the client device together with the game content request, or the available range information continuing from the game content request may be received even if a separate signal for verifying the VRR operability is not sent to the client device.

[0113] At operation S720, the processor 230 may identify the image resolution based on the network bandwidth information. However, the processor 230 may also identify the image resolution based on network status information different from the network bandwidth information. Here, the network status information may include at least one of a packet loss rate, a round-trip time of a packet, a delay gradient of a packet, received signal strength indicator (RSSI) information, communication channel information, link speed information, channel interference information, or retry rate information. That is, the network status information may be information associated with various networks affected by network congestion, distance, and transmission rate.

[0114] In an embodiment, the network status information may be received from the client device. In this case, the client device may collect the network status information by monitoring the network status, and send the collected network status information to the server device 200 at a preset period or in response to a request from the server device 200. For example, if the network is implemented with the Internet-based TCP / IP communication protocol, TCP / IP may be implemented at the link layer, Internet layer, transport layer, and application layer. In this case, the network status information may be collected by monitoring the network status in real time from the application layer, but is not limited thereto. However, the server device 200 may collect the network status information by directly monitoring the network status as appropriate.

[0115] At operation S730, the processor 230 may identify the frame rate of the image based on the available range information. In an embodiment, the processor 230 may identify the frame rate of the image based on at least one of a minimum value, a maximum value, an average value, or a median value included in the available range information. For example, the frame rate of the image may be identified within the available range based on at least one of the frequency of a user control command received from the client device, network status information, or resolution information. For example, the processor 230 may perform rendering of a variable frequency image by varying the frame rate of the image within a range in which the resolution of the image identified in operation S720 is maintained.

[0116] However, according to another example, the processor 230 may identify the resolution and the frame rate of the image in consideration of both the available range information and the network status information. In an embodiment, the processor 230 may variably identify the resolution and the frame rate of the image using a trained artificial intelligence model. Here, the artificial intelligence model being trained means that the artificial intelligence model set to perform a desired characteristic (or, object) or a predefined operation rule as a basic artificial intelligence model (e.g., an artificial intelligence model including random parameters) is created by being trained using a plurality of training data through a learning algorithm. The learning may be implemented by a separate server and / or system, but is not limited thereto, and may be implemented in the electronic device. Examples of the learning algorithm may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples. Here, the artificial intelligence model may be implemented as, for example but not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network, but is not limited thereto.

[0117] In an embodiment, the trained artificial intelligence model can be trained to output the resolution and frame rate of an image based on input available range information and network state information. For example, the trained artificial intelligence model can be trained to output an optimal combination of resolution and frame rate based on the available range information and network bandwidth information. For example, the resolution information can include resolutions such as standard definition (SD), high definition (HD), full high definition (FHD), quad high definition (QHD), 4K ultra high definition (UHD), 8K UHD, or higher. Therefore, the information output from the trained artificial intelligence model can include one of combinations of different resolutions and different frame rates, such as (FHD, 50Hz), (FHD, 60Hz), (FHD, 120Hz), …, (4K UHD, 50Hz), (4K UHD, 60Hz), (4K UHD, 120Hz), …, (8K UHD, 50Hz), (8K UHD, 60Hz), (8K UHD, 120Hz), etc.

[0118] At operation S740, the processor 230 can perform rendering of the variable frequency image based on the identified resolution and frame rate.

[0119] At operation S750, the processor 230 can send content including the rendered variable frequency image and variable frame rate information corresponding to the variable frequency image to the client device.

[0120] In an embodiment, the processor 230 can generate packets (or frames) based on the encoded image and send the generated packets to the client device. In this case, the resolution information and the frame rate information can be included in the header area of the packet, and the encoded image can be included in the payload area of the packet. In an embodiment, the resolution and the frame rate can be changed to at least one frame unit, and the resolution and the frame rate corresponding to the at least one frame unit can be included in the header of the packet.

[0121] In an embodiment, the processor 230 can perform rendering of the variable frequency image by varying the frame rate of the image while maintaining the resolution of the image based on the network bandwidth information.

[0122] For example, the processor 230 can improve the overall load by reducing the frame rate rendering process based on the available range information of the client device, thereby reducing the latency according to the cloud game control input. For example, content streaming can be provided in a method of maintaining the resolution quality by reducing the frame rate (e.g., 30Hz, 50Hz, 60Hz, etc.) only based on the network bandwidth information.

[0123] In an embodiment, the processor 230 may variably identify the frame rate of a variable frequency image within an available range including the minimum value and the maximum value of the VRR, based on the frequency of a user control command received from a client device.

[0124] For example, the processor 230 may increase the frame rate when the frequency of the user control command is greater than or equal to a threshold frequency, and may decrease the frame rate when the frequency of the user control command is less than the threshold frequency. If the frequency of the user control command is greater than or equal to the threshold frequency, it may be possible to provide a smooth game screen if the frame rate is increased to correspond to the frequency of the user control command. However, if the frequency of the user control command is less than the threshold frequency, the frame rate may be decreased to reduce the total load. Here, the threshold frequency may be a preset value considering display panel characteristics and the like, but is not limited thereto, and may be changed according to the network state and the like. According to the corresponding example, the total load may be improved by varying the frame rate rendering process according to the frequency of the user control command, thereby reducing the latency according to the cloud game control input.

[0125] Figure 8 is a sequence diagram showing a signal flow between a display device and a server device according to one or more embodiments.

[0126] Reference Figure 8 , at operation S810, the display device 100 may request a cloud game service, that is, cloud game content, from the server device 200. For example, a user may connect to the server device 200 that provides the cloud game service through the display device 100.

[0127] At operation S820, the server device 200 may send a VRR function verification request to the display device 100. For example, the server device 200 may request whether the display device 100 may support the VRR function, and may request VRR available range information.

[0128] At operation S830, based on supporting the VRR function, the display device 100 may send the VRR available range information to the server device 200 by responding to the VRR function verification request.

[0129] At operation S840, the server device 200 may perform rendering of an image based on the VRR available range information received from the display device 100. For example, the server device 200 may identify the resolution of the image based on network bandwidth information, and may perform rendering of the image by identifying the frame rate of the image based on the available range information.

[0130] At operation S850, the server device 200 may send the rendered image to the display device 100.

[0131] At operation S860, the display device 100 may be set in the VRR output mode and output a VRR image received from the server device 200. For example, the display device 100 may be implemented to support multiple output modes, such as a variable refresh rate (VRR) output mode, a frame rate control (FRC) output mode, and a normal output mode. In this case, if a VRR image is recognized as being to be sent from the server device 200, the display device 100 may be set in the VRR output mode. For example, the display device 100 may be set in the VRR output mode based on information included in an information frame included in the content received from the server device 200. For example, the information frame may include information indicating that a variable frequency image may be sent and may be sent through a part of the area of the image frame. However, according to another example, before sending the VRR image from the server device 200 to the display device 100, a separate signal for setting the VRR output mode may be sent to the display device 100.

[0132] According to the various exemplary embodiments described above, in a low-latency streaming service such as a cloud game service, the transmission delay can be minimized while maintaining the content quality according to the network state and / or display characteristics. Additionally, since the frame rate or refresh rate between the server and the client is different, stuttering, tearing, etc. that may be generated can be minimized.

[0133] The method according to the various embodiments of the present disclosure described above may be implemented in the form of an application installable in a display device of the related art. Alternatively, the method according to the various embodiments of the present disclosure described above may be performed using a deep learning-based artificial neural network (or deep artificial neural network) (i.e., a learned network model). In an embodiment, at least one of shrinking, decoding, encoding, and magnifying may be performed by a trained neural network model.

[0134] Additionally, the method according to the various embodiments of the present disclosure described above may be implemented only by software updates or hardware updates of a display device of the related art.

[0135] Additionally, the various embodiments of the present disclosure described above may be performed by an embedded server provided in the display device or an external server of the display device.

[0136] The various exemplary embodiments of the present disclosure described above can be implemented using software including instructions stored in a machine-readable storage medium (e.g., a computer). The machine can call the instructions stored in the storage medium, and as a device operable according to the called instructions, it can include a display device according to the above embodiments. Based on the instructions being executed by a processor, the processor can directly or using other elements under the control of the processor execute functions corresponding to the instructions. The instructions can include code generated by a compiler or executed by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. In this context, "non-transitory" only means that the storage medium is tangible and does not include signals, and the term does not distinguish whether the data is stored semi-permanently or temporarily stored in the storage medium.

[0137] The method according to the various embodiments described above can be provided to be included in a computer program product. The computer program product can be traded as a commodity between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed online through an application store (e.g., PLAYSTORE™). In the case of online distribution, at least a part of the computer program product can be at least temporarily stored in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server, or temporarily generated.

[0138] In addition, each element (e.g., a module or a program) according to the various embodiments described above can be formed as a single entity or multiple entities, and some of the above sub-elements can be omitted, or other sub-elements can be further included in the various embodiments. Alternatively or additionally, some elements (e.g., a module or a program) can be integrated into one entity to perform the same or similar functions as performed by the corresponding elements before integration. The operations performed by a module, a program, or another element according to the various embodiments can be executed sequentially, in parallel, repeatedly, or in a heuristic manner, or at least some operations can be executed in a different order, omitted, or different operations can be added.

[0139] Although various exemplary embodiments of the present disclosure have been shown and described, it should be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. Those skilled in the art will understand that various changes in form and detail can be made therein without departing from the true spirit and full scope of the present disclosure (including the appended claims and their equivalents).

Claims

1. A display device, comprising: A communication interface; A display; A memory storing at least one instruction; And At least one processor operably connected to the communication interface, the display, and the memory, wherein the at least one processor is configured to execute the at least one instruction to: Send available range information to an external server through the communication interface, the available range information including a minimum refresh rate and a maximum refresh rate of a variable refresh rate (VRR), Receive content from the external server through the communication interface, the content including a variable frequency image rendered based on the available range information and network bandwidth information and variable frame rate information corresponding to the variable frequency image, and Based on the received content, control the display to display the variable frequency image based on the variable frame rate information.

2. The display device according to claim 1, wherein The at least one processor is further configured to execute the at least one instruction to: Based on an information frame in the received content, control the display to be set in the VRR output mode.

3. The display device according to claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to: Identify that at least one of the minimum refresh rate or the maximum refresh rate needs to be changed based on at least one of a streaming buffer state or a network state; And Send adjusted available range information to the external server, the adjusted available range information including at least one of a changed minimum refresh rate or a changed maximum refresh rate.

4. The display device according to claim 1, wherein, The external server is configured to provide a cloud gaming service, and the at least one processor is further configured to execute the at least one instruction to: Send a request for game content to the external server; Receive a signal for verifying VRR operability from the external server based on the request for game content; And Send available range information to the external server based on the signal for verifying VRR operability.

5. A server device, comprising: A communication interface; A memory storing at least one instruction; And At least one processor operably connected to the communication interface and the memory, Wherein the at least one processor is configured to execute the at least one instruction to: Receive available range information from a client device through the communication interface, the available range information including a minimum refresh rate and a maximum refresh rate of a variable refresh rate (VRR), Render a variable frequency image based on the available range information and network bandwidth information, and Send content to the client device through the communication interface, the content including the rendered variable frequency image and variable frame rate information corresponding to the variable frequency image.

6. The server device according to claim 5, wherein, The at least one processor is further configured to execute the at least one instruction to: Identify the resolution for rendering the variable frequency image based on the network bandwidth information; Identify the frame rate for rendering the variable frequency image based on the available range information; And Render the variable frequency image based on the identified resolution and the identified frame rate.

7. The server device according to claim 6, wherein, The at least one processor is further configured to execute the at least one instruction to: Render the variable frequency image by varying the frame rate while maintaining the resolution based on the network bandwidth information.

8. The server device according to claim 5, wherein, The at least one processor is further configured to execute the at least one instruction to: Identify the frame rate of a variable-frequency image within an available range that includes the minimum and maximum refresh rates of VRR, based on the frequency of user control commands received from a client device.

9. The server device according to claim 5, wherein, The server device is configured to provide a cloud gaming service, and the at least one processor is further configured to execute the at least one instruction to: Receive a request for game content from a client device; Based on the request for game content, send a signal to the client device to verify the operability of VRR; Based on the signal verifying the operability of VRR, receive available range information from the client device.

10. A method for controlling a display device, comprising: Send available range information to an external server, the available range information including the minimum and maximum refresh rates of a variable refresh rate (VRR); Receive content from the external server, the content including a variable-frequency image rendered based on the available range information and network bandwidth information, and variable frame rate information corresponding to the variable-frequency image; And Output the variable-frequency image on the display based on the variable frame rate information.

11. The method according to claim 10, further comprising: Set the display to the VRR output mode based on an information frame in the received content.

12. The method according to claim 10, further comprising: Identify whether at least one of the minimum refresh rate or the maximum refresh rate needs to be changed based on at least one of a streaming buffer state or a network state; And Based on identifying that at least one of the minimum refresh rate or the maximum refresh rate needs to be changed, send adjusted available range information to the external server, the adjusted available range information including at least one of a changed minimum refresh rate and a changed maximum refresh rate.

13. The method according to claim 10, wherein, The external server is configured to provide a cloud gaming service, and wherein sending the adjusted available range information to the external server includes: Send a request for game content to the external server; Based on the request for game content, receive a signal from the external server to verify the operability of VRR; Based on the signal verifying the operability of VRR, send the adjusted available range information to the external server.

14. A method for controlling a server device, the method comprising: Receive available range information from a client device, the available range information including the minimum and maximum refresh rates of a variable refresh rate (VRR); Render a variable-frequency image based on the available range information and network bandwidth information; And Send content to the client device, the content including the rendered variable-frequency image and variable frame rate information corresponding to the variable-frequency image.

15. A non-transitory computer-readable medium for storing computer-readable program code or instructions that can be executed by a processor to perform a method for controlling a server device, the method comprising: Receive available range information from a client device, the available range information including the minimum and maximum refresh rates of a variable refresh rate (VRR); Render a variable-frequency image based on the available range information and network bandwidth information; And Send content to the client device, the content including the rendered variable-frequency image and variable frame rate information corresponding to the variable-frequency image.

Images