Method and device for streaming video with low latency based on re-decoding

Abstract

A method for performing video streaming based on re-decoding in a device according to an embodiment of the present invention may comprise the operations of: when a plurality of frames for video streaming are received from a transmitter, decoding the plurality of frames; storing a decoding state for each of the decoded frames in a state cache; when a lost packet is identified for a K-th (here, K is a natural number) frame among the plurality of frames, requesting the transmitter to retransmit the lost packet; and when the lost packet is retransmitted from the transmitter in response to the request, re-decoding the K-th frame and all frames prior to the retransmitted packet on the basis of the state cache. Various other embodiments may be possible.

Description

Method and device for streaming video with low latency based on re-decoding

[0001] The present invention relates to a method and apparatus for performing video streaming by decoding a frame for video streaming and re-decoding the decoded frame when a lost packet occurs in the decoded frame.

[0002] This application claims priority to Korean Patent Application No. 10-2024-0052190, filed on April 18, 2024, and all contents disclosed in the specification and drawings of the said application are incorporated herein by reference.

[0003] Meanwhile, the present invention was supported by the following national research and development project.

[0004] Assignment ID: 2710006536

[0005] Assignment Number: 0536-20240107

[0006] Ministry of Science and ICT

[0007] Project Management Agency Name: National Research Foundation of Korea

[0008] Research Project Name: Source Technology Development Project

[0009] Research Project Name: High-Precision, High-Reliability Teleinteraction Platform

[0010] Project Performing Institution Name: Seoul National University

[0011] Research Period: July 1, 2024 - December 31, 2024

[0012] Assignment ID: 2710008683

[0013] Assignment Number: 0423-20240074

[0014] Ministry of Science and ICT

[0015] Project Management Agency Name: Information and Communications Technology Planning and Evaluation Institute

[0016] Research Project Name: Information and Communication Broadcasting Research and Development Project

[0017] Research Project Name: Development of Core Technologies for AI Networks Based on In-Network Computing

[0018] Project Performing Institution Name: Seoul National University

[0019] Research Period: January 1, 2024 - December 31, 2024

[0020] In a video streaming system using the standard Transmission Control Protocol (TCP) and TCP-based reliable protocols, if packet loss occurs during video streaming, the receiver may request retransmission of the lost packet and wait for decoding until the requested retransmission is received. Consequently, the receiver may experience high latency because it must wait until the requested retransmission is received.

[0021] On the other hand, video streaming systems can use User Datagram Protocol (UDP) and unreliable UDP-based protocols. This poses a problem: packet loss occurs while decoding packets, causing errors to propagate until the next keyframe is received, degrading decoding quality.

[0022] Accordingly, methods to reduce the amount of errors through error concealment algorithms can be considered. However, due to the frequent occurrence of continuous packet loss in video streaming systems, it is difficult to completely eliminate errors. Furthermore, requesting new keyframes can prevent error propagation, but frequent keyframes have the problem of excessively increasing the data size of video streaming.

[0023] The technical problem to be solved by the present invention is to provide a method for performing re-decoding-based video streaming, in which, when performing decoding on a plurality of frames in performing video streaming, a decoding state is stored in a state cache, and when a retransmission packet due to packet loss is received, a frame including the lost packet is re-decoded based on the stored state cache, and a device therefor.

[0024] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0025] In order to achieve the above technical problem, a method for performing re-decoding-based video streaming in a device according to an embodiment of the present invention may include, when receiving a plurality of frames for video streaming from a transmitter, an operation of decoding the plurality of frames, an operation of storing a decoding state for each of the decoded frames in a state cache, an operation of requesting the transmitter to retransmit the lost packet when a lost packet is confirmed for a K-th frame (where K is a natural number) among the plurality of frames, and an operation of re-decoding all frames received before the reception time of the lost packet for the K-th frame and a K-th frame that combines non-lost packets for the K-th frame and lost packets for the K-th frame based on the state cache when the lost packet for the K-th frame is retransmitted in accordance with the request.

[0026] A method according to one embodiment of the present invention may further include an operation of storing packets corresponding to the plurality of frames in a packet cache and an operation of re-decoding packets corresponding to the K-th frame and all frames in the packet cache.

[0027] A method according to one embodiment of the present invention may further include an operation of initializing the state cache and the packet cache when a key frame is identified among the plurality of frames.

[0028] A method according to one embodiment of the present invention may further include an operation of performing decoding on the plurality of frames in the order in which they were received.

[0029] A method according to one embodiment of the present invention may further include an operation of checking the decoding status according to the performance of the decoding.

[0030] A method according to one embodiment of the present invention may further include an operation of updating the state cache by storing state cache data indicating the decoding state in the state cache when the decoding state is confirmed.

[0031] A method according to one embodiment of the present invention may further include an operation of re-decoding the K-frame and all the frames according to the confirmed decoding order.

[0032] The method according to one embodiment of the present invention may further include an operation of transmitting the decoded and re-decoded frame to an application from which the video streaming is output.

[0033] A device for performing video streaming based on re-decoding according to one embodiment of the present invention may include a communication module for receiving a plurality of frames for video streaming from a transmitter, a decoder for decoding the plurality of frames, a state cache for storing a decoding state for each of the decoded frames, and a processor for controlling the decoder to, when a lost packet is identified for a K-th frame among the plurality of frames, request retransmission of the lost packet to the transmitter, and when the lost packet is retransmitted from the transmitter according to the request, re-decode all frames received before the reception time of the lost packet for the retransmitted K-th frame and an intact K-th frame that combines packets that are not lost for the K-th frame and lost packets for the K-th frame based on the state cache.

[0034] A device according to one embodiment of the present invention further includes a packet cache that stores packets corresponding to the plurality of frames, and the processor can control the decoder to re-decode packets corresponding to the K-th frame and all the frames in the packet cache.

[0035] According to the present invention as described above, in performing video streaming based on re-decoding, a plurality of frames according to video streaming are decoded, and when packet loss occurs in a specific frame, retransmission of the frame in which the packet was lost is requested, and when the requested frame is retransmitted, re-decoding is performed together with the newly arrived retransmission packet for the already decoded frame, thereby obtaining a frame of higher quality compared to the existing video streaming method.

[0036] In addition, the present invention has the effect of preventing errors due to packet loss from being propagated to subsequent frames by performing re-decoding-based video streaming.

[0037] In addition, the present invention has the effect of preventing a delay time for a frame requested for retransmission from occurring in a device receiving the frame by performing decoding on the received frame without waiting until the frame requested for retransmission is received.

[0038] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0039] FIG. 1 is a drawing exemplarily illustrating the overall configuration included in a device for performing re-decoding-based video streaming according to one embodiment of the present invention.

[0040] FIG. 2 is a block diagram illustrating a configuration of a device that performs re-decoding-based video streaming according to one embodiment of the present invention.

[0041] FIG. 3 is a diagram illustrating an operation of performing re-decoding-based video streaming in a device according to one embodiment of the present invention.

[0042] FIG. 4 is a diagram illustrating a configuration of a re-decoding-based video streaming system according to one embodiment of the present invention.

[0043] FIG. 5 is a flowchart illustrating an operation of performing re-decoding-based video streaming in a device according to one embodiment of the present invention.

[0044] FIG. 6 is a graph showing the quality of re-decoding-based video streaming according to one embodiment of the present invention.

[0045] FIG. 7 is a graph showing the degree of accuracy reduction for re-decoding-based video streaming according to one embodiment of the present invention.

[0046] The purpose, technical configuration, and resulting operational effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. The following describes embodiments of the present invention in detail with reference to the attached drawings.

[0047] The embodiments disclosed herein should not be construed or used to limit the scope of the present invention. Those skilled in the art will readily appreciate that the descriptions of the embodiments herein, including the embodiments, have a wide range of applications. Therefore, any embodiments described in the detailed description of the present invention are intended to serve as illustrative examples to better illustrate the present invention and are not intended to limit the scope of the present invention to the embodiments.

[0048] The functional blocks depicted in the drawings and described below are merely examples of possible implementations. Other implementations may utilize other functional blocks without departing from the spirit and scope of the detailed description. Furthermore, while one or more functional blocks of the present invention are depicted as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0049] Additionally, the expression "including certain components" is an "open" expression, simply indicating the presence of those components, and should not be construed as excluding additional components.

[0050] Furthermore, when it is said that a component is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0051] Hereinafter, detailed embodiments of the present invention will be described with reference to the drawings.

[0052] FIG. 1 is a drawing exemplarily illustrating the overall configuration included in a device for performing re-decoding-based video streaming according to one embodiment of the present invention.

[0053] However, this is only a preferred embodiment for achieving the purpose of the present invention, and some components may be added or deleted as needed, and the role performed by one component may be performed by another component as well.

[0054] Referring to FIG. 1, a device (100) for performing re-decoding-based video streaming according to one embodiment of the present invention may include a processor (10), a network interface (20), a memory (30), storage (40), and a data bus (50) connecting them, and it will be understood that the device may further include additional components required to achieve the purpose of the present invention.

[0055] According to one embodiment of the present invention, the processor (10) controls the overall operation of each component. The processor (10) may be any one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), or an artificial intelligence processor of a type widely known in the technical field to which the present invention pertains. In addition, the processor (10) may perform operations for at least one application or program for performing decoding and re-decoding according to one embodiment of the present invention. For this purpose, the processor (10) may include a decoder exhibiting a predetermined structure, which will be described later with reference to FIG. 2.

[0056] According to one embodiment of the present invention, when a plurality of frames are received through the network interface (20), the processor (10) can decode the plurality of frames. For example, the processor (10) can perform decoding on the plurality of frames in the order in which they were received.

[0057] According to one embodiment of the present invention, the processor (10) can confirm the decoding order in the packet according to the performance of the decoding as the decoding status. For example, when the processor (10) confirms the decoding status, it can store state cache data indicating the decoding status.

[0058] According to one embodiment of the present invention, when the processor (10) confirms a lost packet for the Kth frame (where K is a natural number) among the plurality of frames, the processor (10) can control the network interface (20) to request retransmission of the lost packet to a device performing video streaming (e.g., a transmitter (101)).

[0059] According to one embodiment of the present invention, when the processor (10) confirms that packet loss has occurred for the K-th frame among a plurality of frames streaming from the transmitter (101), the processor (10) may transmit a NACK message for the K-th frame to the transmitter (101) to request retransmission of the K-th frame. Thereafter, the processor (10) may receive the K-th frame retransmitted from the transmitter (101) in response to the request.

[0060] According to one embodiment of the present invention, when a lost packet retransmitted from the transmitter (101) is received in accordance with the request, the processor (10) can re-decode a K-frame that combines the non-lost packets for the K-frame and the lost packets for the K-frame, and all frames received before the time of receiving the lost packets for the K-frame.

[0061] According to one embodiment of the present invention, the processor (10) can store packets corresponding to the plurality of frames as packet cache data, and re-decode packets corresponding to the K-th frame and all of the frames from the packet cache data.

[0062] According to one embodiment of the present invention, the processor (10) can re-decode the K-th frame and all frames according to the confirmed decoding order. Thereafter, the processor (10) can control the application to output the decoded and re-decoded frames to output frames according to the video streaming through the application.

[0063] According to one embodiment of the present invention, when the processor (10) confirms a key frame among the plurality of frames, it can initialize the decoder state by deleting the state cache and packet cache data.

[0064] According to one embodiment of the present invention, the network interface (20) supports wired and wireless Internet communication of a device (100) that transmits and receives multiple frames in a video streaming situation of the present invention, and may also support other known communication methods. Accordingly, the network interface (20) may be configured to include a corresponding communication module.

[0065] According to one embodiment of the present invention, the memory (30) may load one or more computer programs (41) from the storage (40) to perform video streaming, store commands and / or information for outputting the streamed video, and perform a method of decoding and re-decoding a plurality of frames according to the video streaming. In FIG. 1, RAM is illustrated as one of the memories (30), but it goes without saying that various storage media may be used as the memory (30).

[0066] According to one embodiment of the present invention, the storage (40) can non-temporarily store one or more computer programs (41) and large-capacity network information (42). The storage (40) can be any one of non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk (HDD), a solid state drive (SSD), a removable disk, or any form of computer-readable recording medium widely known in the art to which the present invention pertains.

[0067] According to one embodiment of the present invention, a computer program (41) is loaded into a memory (30) and is executed by one or more processors (10) to: (a) a first step of decoding a plurality of frames for video streaming when receiving the plurality of frames from a transmitter, (b) a second step of storing a decoding status for each of the decoded frames in a state cache, (c) a third step of requesting the transmitter to retransmit the lost packet when confirming a lost packet for the K-th frame among the plurality of frames, and (d) a fourth step of re-decoding all frames including a previous frame of the retransmitted packet in the K-th frame and subsequent frames of the K-th frame based on the state cache when the lost packet is retransmitted from the transmitter according to the request.

[0068] The operations performed by the computer program (41) briefly mentioned above can be viewed as one function of the computer program (41), and a more detailed description will be provided later in the description of a method for performing re-decoding-based video streaming according to one embodiment of the present invention.

[0069] According to one embodiment of the present invention, the data bus (50) becomes a path for moving commands and / or information between the processor (10), network interface (20), memory (30), and storage (40) described above.

[0070] The device (100) for performing re-decoding-based video streaming according to one embodiment of the present invention, which has been briefly described above, may be in the form of an independent device, for example, an electronic device or a server (including a cloud), and the electronic device may be not only a desktop PC or server device that is fixedly installed and used in one place, but also a portable device that is easy to carry, such as a smart phone, a tablet PC, a notebook PC, a PDA, a PMP, etc. Any electronic device that has a CPU corresponding to the processor (10) installed and has only a network function and an audio (microphone, etc.) and text (input means, etc.) input function may be used.

[0071] Hereinafter, assuming that a device (100) for performing video streaming based on re-decoding according to one embodiment of the present invention is in the form of a “server” among the electronic devices that are independent devices, a detailed configuration of a device (100) for performing video streaming by re-decoding a decoded frame of the present invention through a dedicated application installed on a user terminal (not shown) of a user who wishes to perform video streaming will be described with reference to FIG. 2.

[0072] FIG. 2 is a block diagram illustrating a configuration of a device that performs re-decoding-based video streaming according to one embodiment of the present invention.

[0073] Referring to FIG. 2, the device (100) may include a processor (10), a network interface (20), a decoder (130), a cache memory (140), and a memory (30).

[0074] According to one embodiment of the present invention, the processor (10) can control the operation of retransmission and re-decoding for overall lost packets of the device (100) to be performed.

[0075] According to one embodiment of the present invention, the network interface (20) can receive a plurality of frames for video streaming from the transmitter (101).

[0076] According to one embodiment of the present invention, the decoder (130) can decode the plurality of frames.

[0077] According to one embodiment of the present invention, the cache memory (140) may include a state cache (141) and a packet cache (142).

[0078] According to one embodiment of the present invention, the status cache (141) can store status cache data indicating the decoding status of the decoded frame.

[0079] According to one embodiment of the present invention, the packet cache (142) can store packets for the plurality of frames.

[0080] According to one embodiment of the present invention, the memory (30) may include an application that outputs frames according to video streaming.

[0081] According to one embodiment of the present invention, when a plurality of frames are received through the network interface (20), the processor (10) can control the decoder (130) to decode the plurality of frames. For example, the processor (10) can control the decoder (130) to perform decoding on the plurality of frames in the order in which they were received.

[0082] According to one embodiment of the present invention, the processor (10) can check the decoding status according to the performance of the decoding. For example, when the processor (10) checks the decoding status, it can update the status cache (142) by further storing status cache data indicating the decoding status in the status cache (142).

[0083] According to one embodiment of the present invention, when the processor (10) confirms a lost packet for the Kth frame among the plurality of frames, the processor (10) can control the network interface (20) to request retransmission of the lost packet to the transmitter (101).

[0084] According to one embodiment of the present invention, when the processor (10) confirms that packet loss has occurred for the K-th frame among a plurality of frames streaming from the transmitter (101), the processor (10) may transmit a NACK message for the K-th frame to the transmitter (101) to request retransmission of the K-th frame. Thereafter, the processor (10) may receive the K-th frame retransmitted from the transmitter (101) in response to the request.

[0085] According to one embodiment of the present invention, when a lost packet retransmitted from the transmitter (101) is received in accordance with the request, the processor (10) can control the decoder (130) so that all frames received before the reception time of the lost packet for the K-th frame and the K-th frame that is the combination of the non-lost packet for the K-th frame and the lost packet for the K-th frame are re-decoded based on the status cache (141).

[0086] According to one embodiment of the present invention, the processor (10) can store packets corresponding to the plurality of frames in a packet cache (142), and control the decoder (130) so that packets corresponding to the K-th frame and all frames are re-decoded in the packet cache (142).

[0087] According to one embodiment of the present invention, the processor (10) can re-decode the K-th frame and all frames according to the confirmed decoding order. Thereafter, the processor (10) can control the application to output the decoded and re-decoded frames to output frames according to the video streaming through the application.

[0088] According to one embodiment of the present invention, when the processor (10) confirms a key frame among the plurality of frames, the processor (10) can initialize the state cache (141) and the packet cache (142). For example, the processor (10) can initialize the decoding state of the decoder (130) by deleting data of the state cache (141) and the packet cache (142).

[0089] According to one embodiment of the present invention, the device (100) can perform decoding on the K-frame in which packet loss occurred and transmit the decoding to the application. At this time, if the device (100) performs decoding on the frames following the K-frame, the error due to the packet loss may be propagated to the subsequent frames.

[0090] According to one embodiment of the present invention, if a lost packet for a K-th frame is retransmitted in the device (100), the device (100) can re-perform decoding for all frames received before the point in time at which the lost packet for the retransmitted K-th frame was received, including a complete K-th frame that combines the non-lost packets for the K-th frame and the retransmitted lost packets.

[0091] In a device (100) according to one embodiment of the present invention, since video streaming based on re-decoding is performed, unlike existing video streaming systems, re-decoding is performed together with newly arrived retransmission packets for already decoded frames, thereby preventing errors from being propagated after lost packets are retransmitted.

[0092] Hereinafter, a process for providing a method of performing video streaming by re-decoding a decoded frame in a device (100) according to one embodiment of the present invention will be described with reference to FIGS. 3 and 4.

[0093] FIG. 3 is a diagram illustrating an operation of performing re-decoding-based video streaming in a device according to one embodiment of the present invention.

[0094] According to one embodiment of the present invention, the device (100) can receive a request for video streaming from an application (or app) (151) stored in the memory (30). Thereafter, the device (100) can receive a plurality of frames for video streaming from the transmitter (101).

[0095] Referring to FIG. 3, in operation S110, when the device (100) receives a keyframe for the video streaming, the device (100) can initialize the decoder (130). For example, when decoder state cache data for decoding frame tN-1 before the plurality of frames are received from the state cache (141) is stored in the state cache (141), the device (100) can initialize the decoder (130) based on the state cache data.

[0096] According to one embodiment of the present invention, the device (100) can confirm that packet loss has occurred for frame tN among the plurality of frames. At this time, the device (100) can request retransmission of frame tN by transmitting a NACK message for frame tN to the transmitter (101).

[0097] In operation S120, the device (100) can sequentially decode the earliest received frame tN to the latest received frame t among the plurality of frames after initializing the decoder (130). At this time, the device (100) can transfer packets corresponding to the plurality of frames to the packet cache (142), and sequentially retrieve frames tN to t from the packet cache (142) to perform decoding.

[0098] In operation S130, the device (100) can update the state cache (141) by storing state cache data (e.g., state tN, t) indicating the state of the decoder (130) according to the decoding performed for each frame in the state cache (141).

[0099] In operation S140, the device (100) can transmit the frame t from the frame tN on which decoding was performed to the application requesting video streaming.

[0100] According to one embodiment of the present invention, when a key frame different from the received key frame is received, the device (100) can initialize the state cache (141) and the packet cache (142).

[0101] According to one embodiment of the present invention, the device (100) can perform decoding on a frame tN in which packet loss has occurred and transmit the decoding to an application. At this time, if the device (100) performs decoding on frames subsequent to frame tN, the error due to the packet loss may be propagated to the subsequent frames.

[0102] According to one embodiment of the present invention, the device (100) may receive a retransmitted frame tN and then receive a frame t thereafter. At this time, the device (100) may re-perform decoding for both the frame tN in which packet loss occurred and the frames received before the retransmitted frame tN in the frames after the frame tN.

[0103] After that, frame t will also be decoded, and since none of the frames before frame t contain any lost packets, there will be no error, so no error will be propagated to frame t.

[0104] In a device (100) according to one embodiment of the present invention, by performing re-decoding-based video streaming, re-decoding can be performed together with newly arrived retransmission packets for already decoded frames, as compared to existing video streaming systems, thereby having the effect of preventing errors from being propagated due to lost packets.

[0105] FIG. 4 is a diagram exemplarily illustrating a configuration of a re-decoding-based video streaming system according to one embodiment of the present invention, with a case where K is 1 as an example.

[0106] According to one embodiment of the present invention, a re-decoding-based video streaming system may include a device (or receiver) (100) and a sender (101) that perform video streaming at a rate of 30 FPS (frames per second). At this time, the frame interval may be 33 ms, and the network round-trip time may be 100 ms.

[0107] According to one embodiment of the present invention, when a request for video streaming is received from an application stored in a memory (30), the receiver (100) can perform communication with the transmitter (101) to receive a plurality of frames for video streaming.

[0108] Referring to FIG. 4, when the receiver (100) confirms that packet loss has occurred for the first frame (frame 1) among a plurality of frames streamed from the transmitter (101), the receiver (100) can request retransmission of the first frame by transmitting a NACK message for the first frame to the transmitter (101).

[0109] According to one embodiment of the present invention, when the receiver (100) receives a key frame for the video streaming, the receiver (100) can initialize the decoder (130). For example, when the state cache data for decoding a frame before the plurality of frames are received from the state cache (141) is stored in the state cache (141), the receiver (100) can initialize the decoder (130) by removing the state cache data.

[0110] According to one embodiment of the present invention, the receiver (100) can sequentially decode the first frame received first to the third frame received last among the plurality of frames after initializing the decoder (130). At this time, the receiver (100) can transfer packets corresponding to the plurality of frames to the packet cache (142) and sequentially retrieve the first frame to the third frame from the packet cache (142) to perform decoding.

[0111] According to one embodiment of the present invention, the receiver (100) can update the state cache (141) by storing state cache data indicating the state of the decoder (130) according to the decoding performed for each frame in the state cache (141).

[0112] According to one embodiment of the present invention, the receiver (100) can perform decoding on the first frame and transmit it to an application. In addition, the receiver (100) can also perform decoding on subsequent frames of the first frame and transmit it to the application. Meanwhile, in the receiver (100), an error due to packet loss in the first frame can be propagated to subsequent frames of the first frame (error propagation).

[0113] According to one embodiment of the present invention, the receiver (100) may receive the first frame (retrans frame 1) for which retransmission is requested, more specifically, the lost packets for the first frame, before decoding of the fourth frame (frame 4) is performed. At this time, the receiver (100) may re-perform decoding for the second and third frames received before receiving the fourth frame, including the complete first frame that combines the non-lost packets for the first frame and the lost packets for the retransmitted first frame.

[0114] According to one embodiment of the present invention, the receiver (100) performs re-decoding on the first to third frames (frames 1 to 3) from the time packet loss occurs until the retransmission frame due to packet loss is received, thereby preventing an error due to packet loss from being propagated until the retransmission frame is received.

[0115] FIG. 5 is a flowchart illustrating an operation of performing re-decoding-based video streaming in a device according to one embodiment of the present invention.

[0116] Referring to FIG. 5, in operation S210, the device (100) can receive a plurality of frames for video streaming from a transmitter (101) and decode the received plurality of frames.

[0117] According to one embodiment of the present invention, the device (100) can sequentially decode all frames received thereafter, starting from the K-th frame received first among the plurality of frames. At this time, the device (100) can store packets corresponding to the plurality of frames in the packet cache (142). Thereafter, the device (100) can sequentially retrieve the plurality of frames from the packet cache (142) and perform decoding.

[0118] In operation S220, the device (100) can store the decoding status for the plurality of frames in the status cache (141). For example, the device (100) can check the decoding status according to the decoding performed for each frame.

[0119] According to one embodiment of the present invention, when the device (100) confirms the decoding status, it can store status cache data indicating the decoding status in the status cache (141) and update the status cache (141).

[0120] In operation S230, if the device (100) confirms a lost packet for the Kth frame among the plurality of frames, it can request retransmission of the lost packet.

[0121] According to one embodiment of the present invention, when the device (100) confirms packet loss for the K-th frame among the plurality of frames, it can request retransmission of the K-th frame by transmitting a NACK message for the K-th frame to the transmitter (101). Thereafter, the processor (10) can receive the K-th frame retransmitted from the transmitter (101) in response to the request.

[0122] In operation S240, when the lost packet is retransmitted from the transmitter (101) according to the request, the device (100) can re-decode all frames received before the reception time of the lost packet for the K-th frame and the complete K-th frame that combines the non-lost packet for the K-th frame and the lost packet for the K-th frame based on the status cache (141).

[0123] According to one embodiment of the present invention, the device (100) can re-decode the K-frame and all frames according to the decoding order confirmed through the status cache (141).

[0124] According to one embodiment of the present invention, the device (100) can transmit the decoded and re-decoded frames to the application that outputs the video streaming. Thereafter, when a new keyframe is received, the device (100) can initialize the state of the decoder (130). For example, the device (100) can initialize the state of the decoder (130) by deleting data from the state cache (141) and the packet cache (142).

[0125] According to one embodiment of the present invention, the device (100) can perform decoding on the K-frame in which packet loss occurred and transmit the decoding to the application. At this time, if the device (100) performs decoding on the frames following the K-frame, the error due to the packet loss may be propagated to the subsequent frames.

[0126] According to one embodiment of the present invention, the device (100) may receive a retransmitted K-frame and stream the K-frame. At this time, the device (100) may re-perform decoding for all frames received prior to the reception time of the retransmitted K-frame, including the K-frame in which packet loss occurred.

[0127] In a device (100) according to one embodiment of the present invention, by performing re-decoding-based video streaming, unlike existing video streaming systems, re-decoding is performed on all frames that have already been decoded, thereby preventing errors due to lost packets from being propagated.

[0128] FIG. 6 is a graph showing the quality of video streaming based on re-decoding according to one embodiment of the present invention.

[0129] A video streaming method in a device (100) according to one embodiment of the present invention may include, when receiving a plurality of frames for video streaming from a transmitter (101), an operation of decoding the plurality of frames, an operation of storing a decoding status for each of the decoded frames in a state cache, an operation of requesting the transmitter (101) to retransmit the lost packet when confirming a lost packet for the Kth frame among the plurality of frames, and an operation of re-decoding all frames that are previous frames of the Kth frame and the retransmitted packet based on the state cache when the lost packet is retransmitted from the transmitter (101) according to the request.

[0130] Referring to FIG. 6, a graph (500) represents the PSNR (peak signal to noise ratio), which is an image quality that indicates human perception quality, for the method of the present invention (with re-decoding) that performs video streaming by re-decoding already decoded frames together with retransmitted packets, compared to the existing video streaming method (without re-decoding) that does not perform re-decoding for packet loss. In this case, the network round-trip time can be set to 100 ms.

[0131] According to one embodiment of the present invention, the packet loss rate in video streaming can range from 1% to 15%. With a packet loss rate of 5% (0.05), the PSNR value of the method of the present invention (with re-decoding) can be shown to be improved by 26% compared to the existing method (without re-decoding).

[0132] According to one embodiment of the present invention, the device (100) can perform decoding on the K-frame in which packet loss occurred and transmit the decoding to the application. At this time, if the device (100) performs decoding on the frames following the K-frame, the error due to the packet loss may be propagated to the subsequent frames.

[0133] According to one embodiment of the present invention, if a lost packet for a K-th frame is retransmitted in the device (100), the device (100) can re-perform decoding for all frames received before the point in time at which the lost packet for the retransmitted K-th frame was received, including a complete K-th frame that combines the non-lost packets for the K-th frame and the retransmitted lost packets.

[0134] In a device (100) according to one embodiment of the present invention, video streaming is performed by re-decoding a decoded frame together with a newly arrived retransmission packet, thereby improving the quality of packets compared to existing video streaming systems and reducing the possibility of errors being propagated after a lost packet is retransmitted.

[0135] FIG. 7 is a graph showing the degree of accuracy reduction due to re-decoding-based video streaming according to one embodiment of the present invention.

[0136] A video streaming method in a device (100) according to one embodiment of the present invention may include, when receiving a plurality of frames for video streaming from a transmitter (101), an operation of decoding the plurality of frames, an operation of storing a decoding status for each of the decoded frames in a state cache, an operation of requesting the transmitter (101) to retransmit the lost packet when confirming a lost packet for the Kth frame among the plurality of frames, and an operation of re-decoding all frames that are previous frames of the Kth frame and the retransmitted packet based on the state cache when the lost packet is retransmitted from the transmitter (101) according to the request.

[0137] Referring to FIG. 7, a graph (500) shows the decrease in accuracy (mAP) of an object detection deep learning model (e.g., YOLOv5) for a method of the present invention (with re-decoding) that performs video streaming by re-decoding already decoded frames together with retransmitted packets, compared to a conventional video streaming method (without re-decoding) that does not perform re-decoding for packet loss. In this case, the network round-trip time can be set to 100 ms.

[0138] According to one embodiment of the present invention, the packet loss rate in video streaming can range from 1% to 11%. With a packet loss rate of 5% (0.05), the mAP reduction rate of the present invention's method (with re-decoding) is improved from 29% to 4% compared to the existing method (without re-decoding), demonstrating an increase in the accuracy of the object detection deep learning model.

[0139] In a device (100) according to one embodiment of the present invention, video streaming is performed by re-decoding a decoded frame together with a newly arrived retransmission packet, thereby improving frame quality compared to an existing video streaming system and reducing the possibility of an error being propagated after a lost packet is retransmitted.

[0140] Although embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

[0141] 100: Device, receiver

[0142] 10: Processor

[0143] 20: Network interface

[0144] 30: Memory

[0145] 40: Storage

[0146] 50: Data bus

[0147] 130: Decoder

[0148] 140: Cache memory

[0149] 141: Status Cache

[0150] 142: Packet Cache

[0151] 101: Transmitter

Claims

1. A method for performing video streaming based on re-decoding on a device, An operation of receiving a plurality of frames for video streaming from a transmitter, and decoding the plurality of frames; An operation of storing the decoding status for each of the above decoded frames in a state cache; An operation of requesting retransmission of the lost packet to the transmitter when a lost packet is confirmed for the Kth frame (where K is a natural number) among the above multiple frames; and A method comprising: when the lost packet is retransmitted from the transmitter according to the request, re-decoding the K-frame that combines the non-lost packets for the K-frame and the lost packets for the K-frame based on the state cache, and all frames received before the time of receiving the lost packets for the K-frame.

2. In paragraph 1, An operation of storing packets corresponding to the above multiple frames in a packet cache; and A method further comprising: an operation of re-decoding packets corresponding to the K-th frame and all frames in the packet cache.

3. In paragraph 1, A method further comprising: an operation of initializing the state cache and the packet cache when a key frame is confirmed among the plurality of frames.

4. In paragraph 1, A method further comprising: performing decoding on the plurality of frames in the order in which they were received.

5. In paragraph 4, A method further comprising: an operation of checking the decoding status according to the performance of the decoding.

6. In paragraph 5, A method further comprising: when the decoding status is confirmed, an operation of updating the status cache by storing status cache data indicating the decoding status in the status cache.

7. In paragraph 6, A method further comprising: an operation of re-decoding the K-th frame and all the frames according to the confirmed decoding order.

8. In paragraph 1, A method further comprising: transmitting the decoded and re-decoded frames to an application from which the video streaming is output.

9. In a device that performs video streaming based on re-decoding, A communication module for receiving multiple frames for video streaming from a transmitter; A decoder for decoding the plurality of frames; A state cache that stores the decoding state for each of the decoded frames; and If a lost packet is confirmed for the Kth frame (where K is a natural number) among the above multiple frames, a request is made to the transmitter to retransmit the lost packet, A device comprising: a processor for controlling the decoder to re-decode, based on the state cache, all frames received prior to the reception time of the lost packet for the retransmitted K-frame and a complete K-frame that combines the non-lost packets for the K-frame and the lost packets for the K-frame, when the lost packet is retransmitted from the transmitter in accordance with the request.

10. In paragraph 9, Further comprising a packet cache for storing packets corresponding to the plurality of frames; The above processor, A device that controls the decoder to re-decode packets corresponding to the K-frame and all frames in the packet cache.

Images