Image processing device and method

Abstract

The present disclosure relates to a receiving device of an image processing system. The receiving device is configured to: receive, from a transmitting device, at least one data packet including data in a first frame; carry out decoding of the first frame on the basis of the at least one data packet; identify whether a decoding failure of the first frame occurs; and, on the basis of identifying that a decoding failure of the first frame has occurred, transmit, to the transmitting device, first information related to at least one first block in which the decoding failure has occurred. The first information may include block ID information of the at least one first block in which the decoding failure has occurred, and frame ID information of the first frame to which the at least one first block belongs.

Description

Image processing device and method

[0001] The present disclosure relates to an image processing device and method.

[0002] In order to increase the compression ratio of an image, an image processing device may use a compression method that encodes subsequent frames / pictures by referring to already encoded frames / pictures.

[0003] When using this compression method, errors occurring in the referenced frame are propagated to subsequent frames. For example, a decoding error occurring in the referenced frame can affect the decoding of subsequent frames. This error propagation can lead to a cascading degradation of image quality. Therefore, it is necessary to consider measures to prevent / reduce errors occurring in previous frames from affecting subsequent frames.

[0004] According to an exemplary embodiment of the present disclosure, a receiving device in an image processing system may include a communication unit including a communication circuit; a memory storing a program including at least one command; and at least one processor connected to the communication unit and the memory, including a processing circuit, and configured to execute at least one command of the program stored in the memory. At least one processor is configured to individually and / or collectively: receive, from a transmitting device, at least one data packet including data of a first frame, perform decoding on the first frame based on the at least one data packet, identify whether a decoding failure occurs on the first frame, and control the receiving device to transmit, to the transmitting device, first information related to at least one first block in which the decoding failure occurred based on identifying that the decoding failure occurred on the first frame, wherein the first information may include block identifier (ID) information of the at least one first block in which the decoding failure occurred and frame ID information of the first frame to which the at least one first block belongs.

[0005] In an exemplary embodiment of the present disclosure, a transmitting device in an image processing system may include a communication unit including a communication circuit; a memory storing a program including at least one command; and at least one processor connected to the communication unit and the memory, the processor including a processing circuit, and executing at least one command of the program stored in the memory. The at least one processor may be configured to: transmit, to a receiving device, at least one data packet including data of a first frame; receive, from the receiving device, first information related to at least one block in the first frame in which a decoding failure has occurred, the first information including block ID information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs; identify, based on the first information, the at least one first block in the first frame; process the at least one first block so that it is not referenced by blocks in a second frame that is encoded based on the reception of the first information; and control the transmitting device to perform encoding for the second frame.

[0006] According to an exemplary embodiment of the present disclosure, a method performed by a receiving device in an image processing system includes: receiving, from a transmitting device, at least one data packet including data of a first frame; performing decoding on the first frame based on the at least one data packet; identifying whether a decoding failure has occurred on the first frame; and transmitting, based on identifying that a decoding failure has occurred on the first frame, first information related to at least one first block in which the decoding failure has occurred to the transmitting device, wherein the first information may include block ID information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs.

[0007] In accordance with an exemplary embodiment of the present disclosure, a method performed by a transmitting device in an image processing system may include: transmitting, to a receiving device, at least one data packet including data of a first frame; receiving, from the receiving device, first information related to at least one block in the first frame in which a decoding failure has occurred, the first information including block ID information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs; identifying, based on the first information, the at least one first block in the first frame; processing, based on the first information, the at least one first block so as not to be referenced by blocks in a second frame that is encoded based on the reception of the first information; and performing encoding on the second frame.

[0008] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0009] FIG. 1A is a block diagram illustrating an exemplary image processing system according to one embodiment of the present disclosure.

[0010] FIG. 1b is a block diagram illustrating an exemplary configuration of a transmitting device according to one embodiment of the present disclosure.

[0011] FIG. 1c is a block diagram illustrating an exemplary configuration of a receiving device according to one embodiment of the present disclosure.

[0012] FIG. 2 is a block diagram illustrating an image processing operation of a transmitting device and a receiving device according to one embodiment of the present disclosure.

[0013] FIG. 3 is a block diagram illustrating an exemplary configuration of an image encoding unit of a transmitting device according to one embodiment of the present disclosure.

[0014] FIG. 4 is a schematic block diagram of an image decoding unit of an image receiving device according to one embodiment of the present disclosure.

[0015] FIGS. 5A and 5B are diagrams illustrating exemplary image frame groups according to one embodiment of the present disclosure.

[0016] FIG. 6A is a signal flow diagram illustrating an exemplary image processing procedure of an image processing system according to one embodiment of the present disclosure.

[0017] FIGS. 6b and 6c are diagrams illustrating a case where a decoding failure occurs in an image processing procedure according to one embodiment of the present disclosure.

[0018] FIG. 7 is a signal flow diagram illustrating an exemplary image processing procedure of an image processing system according to one embodiment of the present disclosure.

[0019] FIG. 8 is a flowchart illustrating an exemplary operation of a transmitting device in an image processing procedure according to one embodiment of the present disclosure.

[0020] FIG. 9 is a flowchart illustrating an exemplary operation of a receiving device in an image processing procedure according to one embodiment of the present disclosure.

[0021] FIG. 10 is a signal flow diagram illustrating an exemplary image processing procedure of an image processing system according to one embodiment of the present disclosure.

[0022] FIG. 11A is a flowchart illustrating an exemplary operation of a transmitting device in an image processing procedure according to one embodiment of the present disclosure.

[0023] FIG. 11b is a flowchart illustrating an exemplary operation of a transmitting device in an image processing procedure according to one embodiment of the present disclosure.

[0024] FIG. 12A is a flowchart illustrating an exemplary operation of a receiving device in an image processing procedure according to one embodiment of the present disclosure.

[0025] FIG. 12b is a flowchart illustrating an exemplary operation of a receiving device in an image processing procedure according to one embodiment of the present disclosure.

[0026] FIGS. 13A and 13B are diagrams illustrating an exemplary method for preventing / reducing decoding failure in an image processing procedure according to one embodiment of the present disclosure.

[0027] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the drawings. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0028] At this time, it will be understood that each block of the processing flow diagrams and combinations of the flow diagrams can be performed by computer program instructions.

[0029] Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specific logical function(s). It should also be noted that in some embodiments, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on the corresponding functionality.

[0030] Here, the term '~ unit' used in the present embodiment may refer to, for example, software and / or hardware components such as FPGA (field programmable gate array) or ASIC (application specific integrated circuit). The '~ unit' performs certain roles. However, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to reproduce one or more packet processing devices. Accordingly, as an example, the '~ unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, the components and '~parts' may be implemented to play one or more central processing units (CPUs) within the device or secure multimedia card. In addition, in the embodiment, the '~parts' may include one or more packet processing devices.

[0031] FIG. 1A is a block diagram illustrating an exemplary image processing system according to one embodiment of the present disclosure.

[0032] Referring to FIG. 1A, the image processing system (100) may include a transmitting device (110) and a receiving device (120), each of which includes various circuitry. In the present disclosure, the transmitting device (110) may be referred to as an image transmitting device or an image providing device, and the receiving device (120) may be referred to as an image receiving device or an image reproducing device.

[0033] According to an exemplary embodiment, the transmitting device (110) may include a transmitting circuit and transmit media data (e.g., video and images) to the receiving device (120). For example, the transmitting device (110) may transmit a video signal including media data (e.g., video and images) to the receiving device (120) via a network, and the receiving device (120) may receive and play the video signal from the receiving device (110). The transmitting device (110) may encode the video signal by compressing the video signal by removing redundant information within the video signal to a level that is not visually detectable. The transmitting device (110) may efficiently store, transmit, and manage the video signal by encoding the video signal within a set compression ratio range. The video signal may include, but is not limited to, real-time encoded video including, for example, screen mirroring video, video conferencing video, and game video.

[0034] According to one embodiment, a transmitting device (110) and a receiving device (120) may exchange signaling information (or signaling messages) for delivery and / or consumption of media data (or video signals including media data). As an example, the signaling messages may include FEC-related information.

[0035] According to one embodiment, the transmitting device (110) may include various video source devices such as a TV, a PC, a smartphone, a tablet, a set-top box, a game console, and a server, and the receiving device (120) may include various video playback devices such as a TV, a smartphone, a tablet, and a PC. It will be apparent to those skilled in the art that the transmitting device (110) and the receiving device (120) are not limited to a specific type of device.

[0036] According to one embodiment, the transmitting device (110) and the receiving device (120) can transmit and receive video signals through a network. According to various embodiments, the network connecting the transmitting device (110) and the receiving device (120) can include a short-range communication network such as Wi-Fi (wireless fidelity) and a long-range communication network such as a cellular network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN)), and can communicate based on an IP communication protocol. The cellular network can include GSM (global system for mobile communications), EDGE (enhanced data GSM environment), CDMA (code division multiple access), TDMA (time division multiplexing access), 5G, LTE (long term evolution), and LTE-A (LTE advance). The network can include connections of network elements such as hubs, bridges, routers, switches, and gateways. The network can include one or more connected networks, such as a multi-network environment, including a public network such as the Internet and a private network such as an enterprise private network. Access to the network can be provided via one or more wired or wireless access networks. The network can support an Internet of Things (IoT) network, where information is exchanged and processed between distributed components such as objects.

[0037] FIG. 1b is a block diagram illustrating an exemplary configuration of a transmitting device according to one embodiment of the present disclosure.

[0038] Referring to FIG. 1B, the transmitting device (110) may include a display unit (111) (e.g., including a display), a memory (112), a user interface (113) (e.g., including an interface circuit), an input / output interface (114) (e.g., including an input / output circuit), a control unit (115) (e.g., including a processing circuit), a sensor unit (116) (e.g., including a sensor), a communication unit (117) (e.g., including a communication circuit), and / or a power unit (118) (e.g., including a power supply). In some embodiments, the transmitting device (110) may omit at least one of these components (e.g., the display unit (111)), or may have one or more other components added. In some embodiments, some of these components may be integrated into one component. In the present disclosure, the control unit (115) may be defined as a circuit or an application-specific integrated circuit, or at least one processor.

[0039] In one embodiment, the display unit (111) may include a display and may visually provide information to an external party (e.g., a user) of the transmitting device (110). The display unit (111) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display unit (111) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0040] According to one embodiment, the memory (112) can store various data used by at least one component (e.g., the control unit (115) or the sensor unit (116)) of the transmitting device (110). The data can include, for example, input data or output data for software (e.g., a program) and commands related thereto. The memory (112) can include volatile memory or non-volatile memory.

[0041] According to one embodiment, the user interface (113) may include various circuits and provide an interface for facilitating interaction between a user and the transmitting device (110). The user interface (113) may, for example, provide a collection of methods, circuits, and tools for a user to use in operating the transmitting device (110) and obtaining information. The user interface (113) may include, but is not limited to, a graphical user interface, a text-based user interface, a touch interface, and a voice recognition interface.

[0042] According to one embodiment, the input / output interface (114) may include various circuits and may receive input signals (e.g., commands or data) to be used by components of the transmitting device (110) (e.g., control unit (115)) from an external source (e.g., a user) of the transmitting device (110), or output output signals to the external source. The input / output interface (114) may include, for example, a microphone, a speaker, audio, a mouse, a keyboard, keys (e.g., buttons), or a digital pen (e.g., a stylus pen).

[0043] According to one embodiment, the control unit (115) may include various processing circuits and control the overall operation of the transmitting device (110). The control unit (115) may, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the transmitting device (110) connected to the processor, and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the control unit (115) may store a command or data received from another component (e.g., a sensor unit (116) or a communication unit (117)) in the memory (112), process the command or data stored in the memory (112), and store result data in the memory (112). The control unit (115) according to one embodiment of the present disclosure may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, can encompass various processing circuits that include at least one processor, wherein one or more of the at least one processors can be individually and / or collectively configured to perform the various functions described herein in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform numerous functions, these terms encompass, for example, but are not limited to, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where a single processor can perform all of the recited functions.Additionally, at least one processor may comprise a combination of processors that perform various functions, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0044] According to one embodiment, the control unit (115) may include sub-components (e.g., processors) including, for example, a CPU, a GPU (graphics processing unit), an NPU (neural processing unit), an ISP (image signal processor), a display controller, a memory controller, and / or a CP (communication processor). The above sub-components are merely exemplary. For example, the control unit (115) may further include other sub-components. For example, some sub-components may be omitted from the control unit (115).

[0045] According to one embodiment, the sensor unit (116) may include at least one sensor, and may detect an operating state (e.g., power or temperature) of the transmitting device (110) or an external environmental state (e.g., user state), and generate an electrical signal or data value corresponding to the detected state. The sensor unit (116) may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0046] According to one embodiment, the communication unit (117) may include various communication circuits and may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the transmitting device (110) and an external electronic device (e.g., the receiving device (120), another electronic device, or a server), and the performance of communication through the established communication channel. The communication unit (117) may operate independently from the control unit (115) and may include one or more communication processors (e.g., including a processing circuit) that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication unit (117) may include a communication unit (117) (e.g., a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a LAN communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device via a first network (e.g., a short-range communication network such as Bluetooth, Wi-Fi direct, or IrDA (infrared data association)) or a second network (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0047] In one embodiment, the power supply (118) may include a power supply, supply power to the transmitting device (110), and manage the power supplied to the transmitting device (110).

[0048] FIG. 1c is a block diagram illustrating an exemplary configuration of a receiving device according to one embodiment of the present disclosure.

[0049] Referring to FIG. 1C, the receiving device (120) may include a display unit (121) (e.g., including a display), a memory (122), a user interface (123) (e.g., including an interface circuit), an input / output interface (124) (e.g., including an input / output circuit), a control unit (125) (e.g., including a processing circuit), a sensor unit (126) (e.g., including a sensor), a communication unit (127) (e.g., including a power supply), and / or a power supply unit (128). In some embodiments, the transmitting device (110) may omit at least one of these components (e.g., the sensor unit (126)), or may have one or more other components added. In some embodiments, some of these components may be integrated into one component. In the present disclosure, the control unit (125) may include a circuit or an application-specific integrated circuit, or at least one processor.

[0050] In one embodiment, the display unit (121) may include a display and may visually provide information to an external party (e.g., a user) of the receiving device (120). The display unit (121) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display unit (121) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0051] According to one embodiment, the memory (122) can store various data used by at least one component (e.g., the control unit (125) or the sensor unit (126)) of the receiving device (120). The data can include, for example, input data or output data for software (e.g., a program) and commands related thereto. The memory (122) can include volatile memory or non-volatile memory.

[0052] According to one embodiment, the user interface (123) may include various circuits and provide an interface for facilitating interaction between a user and the receiving device (120). The user interface (123) may, for example, provide a collection of methods, circuits, and tools for a user to use in operating the receiving device (120) and obtaining information. The user interface (123) may include, but is not limited to, a graphical user interface, a text-based user interface, a touch interface, and a voice recognition interface.

[0053] According to one embodiment, the input / output interface (124) may include various circuits and may receive input signals (e.g., commands or data) to be used by components of the receiving device (120) (e.g., control unit (125)) from outside of the receiving device (120) (e.g., a user), or output output signals to the outside. The input / output interface (124) may include, for example, a microphone, a speaker, audio, a mouse, a keyboard, keys (e.g., buttons), or a digital pen (e.g., a stylus pen).

[0054] According to one embodiment, the control unit (125) may include various processing circuits and control the overall operation of the receiving device (120). The control unit (125) may, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the receiving device (120) connected to the processor, and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the control unit (125) may store a command or data received from another component (e.g., a sensor unit (126) or a communication unit (127)) in the memory (122), process the command or data stored in the memory (122), and store result data in the memory (122).

[0055] According to one embodiment, the control unit (125) may include subcomponents (e.g., processors) including, for example, a CPU, a GPU, an NPU, an ISP, a display controller, a memory controller, and / or a CP. The above subcomponents are merely exemplary. For example, the control unit (125) may further include other subcomponents. For example, some subcomponents may be omitted from the control unit (125). The control unit (125) according to one embodiment of the present disclosure may include various processing circuits and / or multiple processors. For example, the term “processor” as used in this disclosure, including the claims, may include various processing circuits including at least one processor, wherein one or more of the at least one processor may be configured to perform various functions described in the present disclosure in an individually and / or collectively distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform multiple functions, these terms encompass, for example, but are not limited to, situations where one processor performs some of the recited functions, other processor(s) perform other parts of the recited functions, and also situations where a single processor can perform all of the recited functions. Additionally, the at least one processor may comprise a combination of processors that perform the various recited / disclosed functions, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform the various functions.

[0056] According to one embodiment, the sensor unit (126) may include at least one sensor, and may detect an operating state (e.g., power or temperature) of the receiving device (120) or an external environmental state (e.g., user state), and generate an electrical signal or data value corresponding to the detected state. The sensor unit (126) may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0057] According to one embodiment, the communication unit (127) may include various communication circuits and may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the receiving device (120) and an external electronic device (e.g., the transmitting device (110), another electronic device, or a server), and the performance of communication through the established communication channel. The communication unit (127) may operate independently from the control unit (125) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication unit (127) may include a communication unit (127) (e.g., a cellular communication module, a short-range communication module, or a GNSS communication module) or a wired communication module (e.g., a LAN communication module or a power line communication module). Any of these communication modules may communicate with an external electronic device via a first network (e.g., a short-range communication network such as Bluetooth, Wi-Fi direct, or IrDA) or a second network (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0058] According to one embodiment, the power supply (128) may include a power supply, supply power to the receiving device (120), and manage the power supplied to the receiving device (120).

[0059] FIG. 2 is a block diagram illustrating exemplary image processing operations of a transmitting device and a receiving device according to one embodiment of the present disclosure.

[0060] Referring to FIG. 2, the transmitting device (110) may include an image input unit (211) (e.g., including various circuits), an image encoding unit (212) (e.g., including an encoder), and an image output unit (213) (e.g., including various circuits). The transmitting device (110) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components.

[0061] According to one embodiment, the operations of the image input unit (211), the image encoding unit (212), and the image output unit (213) may be performed by a transmitting device (e.g., the transmitting device (110) of FIGS. 1A and 1B) and / or a control unit of the transmitting device (e.g., the control unit (115) of FIG. 1B).

[0062] According to one embodiment, the image input unit (211) may include various circuits and may acquire input data. The input data may include, but is not limited to, media data (e.g., image data, video data). The input data may be received from outside the transmission device (110) or may be built into the transmission device (110). The image input unit (211) may receive input data from outside in a wired or wireless manner using a communication unit (e.g., the communication unit (117) of FIG. 1B).

[0063] According to one embodiment, the video encoding unit (212) may include an encoder, and may encode input data using encoding setting information to generate encoded data. For example, the video encoding unit (212) may perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. The encoding setting information may include, but is not limited to, information on an encoding / decoding scheme (e.g., codec information) and information on a bitrate used to encode media data. The encoding / decoding scheme may include, but is not limited to, advanced video coding (AVC) / H.264, high efficiency video coding (HEVC) / H.265, and versatile video coding (VVC) / H.266, for example.

[0064] According to one embodiment, the image encoding unit (212) can perform encoding on a frame (or picture) basis.

[0065] According to one embodiment, the image encoding unit (212) can provide encoded data to the image output unit (213) in the form of a bitstream.

[0066] According to one embodiment, the image output unit (213) may include various circuits and may transmit encoded data to the receiving device (120) using the communication unit (117). For example, the image output unit (213) may transmit the encoded data through at least one packet. As an example, the image output unit (213) may transmit at least one data packet including encoded data to the receiving device (120).

[0067] According to one embodiment, a data packet may include a header and a payload including at least a portion of encoded data.

[0068] According to one embodiment, the header of a data packet may include packet sequence information and / or encoding / decoding related information (e.g., codec information, bit rate information).

[0069] In one embodiment, the packet sequence information may indicate the sequence number of the corresponding data packet. The receiving device (120) can determine the sequence of the corresponding data packet based on the packet sequence number information. Through this, the receiving device (120) can sequentially process (e.g., decode) the data packets for the corresponding frame.

[0070] According to one embodiment, the receiving device (120) can identify, based on packet sequence information, whether a data packet has been acquired (or received) at a time when processing for the data packet is required.

[0071] According to one embodiment, the video output unit (213) (or, the transmitting device (110)) can transmit data packets to the receiving device (120) using a unicast method, a multicast method, or a broadcast method.

[0072] According to one embodiment, the video output unit (213) can transmit at least one data packet containing encoded data based on a transmission rate of the data packet.

[0073] In one embodiment, the transmission rate of a data packet may be, for example, the number of data packets transmitted over a given period of time or the number of bits of data (e.g., encoded data or media data) contained in a data packet transmitted over a given period of time. For example, the transmission rate of a data packet may be the number of data packets transmitted over a given period of time (e.g., 1 / 60 of a second for 60 fps). For example, the transmission rate of a data packet may be the number of bits of source data transmitted per second (e.g., the bit rate (bps: bits per second) of the source data). In one embodiment, the transmission rate of a data packet may be related to the bit rate used to encode the media data. For example, as the bit rate increases, the transmission rate of the data packet may increase (proportional relationship).

[0074] According to one embodiment, the video output unit (213) can transmit encoded data to the receiving device (120) via a digital storage medium. The digital storage medium can include various storage media such as, for example, a universal serial bus (USB), a universal serial bus (SD), a universal serial bus (CD), a universal serial bus (DVD), a Blu-ray, a universal serial bus (HDD), and a universal serial bus (SSD).

[0075] The receiving device (120) may include an image input unit (221) (e.g., including various circuits), an image decoding unit (222) (e.g., including a decoder), and an image output unit (223) (e.g., including various circuits). The image output unit (223) may include a display unit, and the display unit may be configured as a separate device or an external component. The receiving device (120) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components.

[0076] According to one embodiment, the operations of the image input unit (221), the image decoding unit (222), and the image output unit (223) may be performed by a receiving device (e.g., the receiving device (120) of FIGS. 1A and 1C) and / or a control unit of the receiving device (e.g., the control unit (215) of FIG. 1C).

[0077] According to one embodiment, the image input unit (221) may include various circuits and may acquire a received signal (image signal).

[0078] The image input unit (221) can receive an image signal from the transmission device (110) using a communication unit (e.g., the communication unit (127) of FIG. 1C). According to one embodiment, the image input unit (221) can receive encoded data from the transmission device (110) using the communication unit (127). For example, the image input unit (221) can receive at least one packet including encoded data transmitted (or processed) based on a transmission rate of a data packet from the transmission device (110).

[0079] According to one embodiment, the image input unit (221) can obtain encoded data through a digital storage medium.

[0080] According to one embodiment, the image decoding unit (222) may include a decoder and may be configured to perform an inverse operation of the operation of the image encoding unit (212). For example, the image decoding unit (222) may decode encoded data by performing a series of procedures corresponding to the image encoding unit (212), such as inverse quantization, inverse transformation, and prediction.

[0081] According to one embodiment, the image decoding unit (222) can decode forward error correction (FEC) decoded data using decoding configuration information to generate decoded data. The decoding configuration information can include, but is not limited to, information about an encoding / decoding method (e.g., codec information) and information about a bit rate.

[0082] According to one embodiment, the image output unit (223) may include various circuits and may render decoded data. The rendered data may be displayed through the display unit.

[0083] The transmitter (110) and the receiver (120) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. For example, the transmitter (110) may further include an FEC encoder for forward error correction (FEC), and the receiver (120) may further include an FEC decoder. In this case, the transmitter (110) may FEC encode data packets to generate FEC packets for restoring the data packets and transmit the FEC packets together with the related data packets, and the receiver (120) may use the FEC packet(s) to restoring dropped data packet(s). According to one embodiment, FEC encoding in the transmitter (110) may be performed after encoding, and FEC decoding in the receiver (120) may be performed before decoding.

[0084] FIG. 3 is a block diagram illustrating an exemplary image encoding unit of a transmitting device according to one embodiment of the present disclosure.

[0085] Referring to FIG. 3, the image encoding unit (212) may include a prediction unit (310), a transformation unit (320), a quantization unit (330), an entropy encoding unit (340), and an in-loop filtering unit (350), each of which includes various circuits. The image encoding unit (212) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. All or at least some of the plurality of components of the image encoding unit (212) may be implemented as a single hardware component (e.g., an encoder or a processor) according to an embodiment. In addition, the image encoding unit (212) may further include a memory. The memory may be implemented by a digital storage medium, and may include a buffer (e.g., a frame buffer) that buffers encoded pictures / frames during the encoding process.

[0086] According to one embodiment, the video encoding unit (212) can divide the input video signal (or picture / frame) into one or more processing units.

[0087] For example, the processing unit may be a coding unit. The coding unit may be obtained by recursively splitting a coding tree unit (CTU) or a largest coding unit (LCU) according to a quad-tree / binary-tree / ternary-tree (QT / BT / TT) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary tree structure. An encoding procedure according to the present disclosure may be performed based on a final coding unit that is no longer split. The encoding procedure may include procedures such as prediction, transformation, and quantization, which will be described later.

[0088] For example, the processing unit of the encoding procedure may be a prediction unit or a transform unit. The prediction unit and the transform unit may each be split or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit that derives a transform coefficient and / or a unit that derives a residual signal from the transform coefficient. In the present disclosure, a pixel may mean the smallest unit of a picture / frame (or image). In addition, a sample may be used as a term corresponding to a pixel and may represent a pixel or a value of a pixel.

[0089] According to one embodiment, the prediction unit (310) can perform a prediction on the current block (the block to be processed) and generate a predicted block including prediction samples for the current block. The prediction unit (310) can determine whether intra-prediction or inter-prediction is applied in units of the current block or coding unit. The prediction unit (310) can generate various information regarding the prediction of the current block and transmit the information to the entropy encoding unit (340). The information regarding the prediction can be encoded by the entropy encoding unit (340) and output in the form of a bitstream.

[0090] According to one embodiment, the intra prediction unit (311) can predict the current block by referring to samples within the current picture / frame. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the intra prediction mode and / or intra prediction technique.

[0091] According to one embodiment, the inter prediction unit (312) can derive a predicted block for the current block based on a reference block specified by a motion vector on a reference picture / frame. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion prediction information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion prediction information can include a reference picture / frame ID (or index) and a motion vector. The motion prediction information can further include information on the inter prediction direction (L0 direction, L1 direction, etc.).

[0092] According to one embodiment, the inter prediction unit (312) can derive a predicted block for the current block by using at least one piece of motion prediction information as inter prediction information. For example, the inter prediction unit (312) can derive a predicted block for the current block by using at least one piece of first motion prediction information including a first motion vector (MVL0) and second motion prediction information including a second motion vector (MVL1) as inter prediction information for the current block of the video signal.

[0093] The prediction signal generated through the prediction unit (310) can be used to generate a restoration signal or a residual signal. The residual signal (residual block) generated by subtracting the prediction signal (predicted block) from the input image signal (original block) can be transmitted to the conversion unit (320).

[0094] According to one embodiment, the transform unit (320) may include various circuits and may apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a karhunen-loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, GBT may refer to a transform obtained from a graph, for example, when relationship information between pixels is expressed as a graph. CNT may refer to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. The transform process may be applied to a block of pixels having a square equal size, or may be applied to a block of non-square variable size.

[0095] According to one embodiment, the quantization unit (330) may include various circuits, and may quantize transform coefficients and transmit them to the entropy encoding unit (340). The entropy encoding unit (340) may encode a quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantization unit (330) may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and may generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. According to one embodiment, the entropy encoding unit (340) may perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), or context-adaptive binary arithmetic coding (CABAC).

[0096] According to one embodiment, the in-loop filtering unit (350) may include a filter and may restore compression degradation by applying filtering to the resulting image of the encoding process. For example, the in-loop filtering unit (350) may apply various filtering methods to the restored picture / frame to generate a modified restored picture / frame. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The modified restored picture / frame may be stored in the memory and used as a reference picture / frame in the prediction unit (310).

[0097] The above memory can store motion prediction information of a block from which motion prediction information is derived (or encoded) within the current picture / frame and / or motion prediction information of blocks within a picture / frame that have already been restored. The memory can store restoration samples of restored blocks within the current picture / frame, and the stored restoration samples can be transmitted to the intra prediction unit (311).

[0098] FIG. 4 is a block diagram illustrating an exemplary configuration of an image decoding unit of an image receiving device according to one embodiment of the present disclosure.

[0099] Referring to FIG. 4, the image decoding unit (222) may include an entropy decoding unit (410), an inverse quantization unit (420), an inverse transformation unit (430), a prediction unit (440), an addition unit (450), and a filtering unit (460), each of which may include various circuits. The image decoding unit (222) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. All or at least some of the plurality of components of the image decoding unit (222) may be implemented as a single hardware component (e.g., a decoder or a processor) according to an embodiment. In addition, the image decoding unit (222) may further include a memory. The memory may be implemented by a digital storage medium, and may include a buffer (e.g., a frame buffer) for buffering a decoded picture / frame.

[0100] The image decoding unit (222) can restore the image signal by performing a process corresponding to the process performed in the image encoding unit (212) described above with reference to FIG. 3. For example, the image decoding unit (222) can perform decoding using a processing unit applied in the image encoding unit (212). As an example, the decoding processing unit can be, for example, a coding unit. The coding unit can be a coding tree unit or can be obtained by dividing a maximum coding unit.

[0101] According to one embodiment, the entropy decoding unit (410) can parse the bitstream to derive information necessary for image restoration (or picture / frame restoration). For example, the entropy decoding unit (410) can decode information in the bitstream based on a decoding method such as Exponential Golomb, CAVLC, or CABAC, and output information about quantized transform coefficients for residuals necessary for image restoration. Information about prediction among the information decoded by the entropy decoding unit (410) is provided to the prediction unit (440), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (410), for example, quantized transform coefficients and related parameter information, can be input to the inverse quantization unit (420). In addition, information about filtering among the information decoded by the entropy decoding unit (410) can be provided to the filtering unit (460).

[0102] According to one embodiment, the inverse quantization unit (420) may include various circuits and may inverse quantize quantized transform coefficients to output transform coefficients. The inverse quantization unit (420) may rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the image encoding unit (112). The inverse quantization unit (420) may perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0103] According to one embodiment, the inverse transform unit (430) may include various circuits and may inversely transform transform coefficients to obtain a residual signal (residual block).

[0104] According to one embodiment, the prediction unit (440) may include various circuits, perform prediction on a current block, and generate a predicted block including prediction samples for the current block. The prediction unit (440) may determine whether intra prediction or inter prediction is applied to the current block based on information about the prediction output from the entropy decoding unit (410), and may determine a specific intra / inter prediction mode (prediction technique). The intra prediction unit (441) may predict the current block by referring to samples in the current picture / frame. The inter prediction unit (442) may derive a predicted block for the current block based on a reference block specified by a motion vector on a reference picture / frame. The operations of the intra prediction unit (441) and the inter prediction unit (442) correspond to the operations of the intra prediction unit (311) and the inter prediction unit (312) of the image encoding unit (112), respectively.

[0105] According to one embodiment, the addition unit (450) may include various circuits and may generate a restoration signal (restored picture / frame, restoration block) by adding a residual signal obtained from the inverse transformation unit (430) to a prediction signal (predicted block) output from the prediction unit (440).

[0106] According to one embodiment, the filtering unit (460) may include a filter and may apply filtering to a restoration signal to improve image quality. For example, the filtering unit (460) may apply various filtering methods to a restoration picture / frame to generate a modified restoration picture / frame, and may store the modified restoration picture / frame in the memory. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The restoration picture / frame stored in the memory may be used as a reference picture / frame in the prediction unit (440). The memory may store motion prediction information of a block from which motion prediction information is derived (or decoded) in the current picture / frame and / or motion prediction information of blocks in a picture / frame that has already been restored. The memory may store restoration samples of restoration blocks in the current picture / frame, and the stored restoration samples may be transmitted to the intra prediction unit (441).

[0107] FIGS. 5A and 5B are diagrams illustrating exemplary image frame groups according to one embodiment of the present disclosure.

[0108] According to one embodiment, an image frame group may include a plurality of image frames (e.g., encoded frames). According to one embodiment, one image frame group may include an I frame located at the beginning and at least one non-I frame located between the I frame and the next I frame. For example, the image frame group may include an I frame (intra-frame), at least one P frame (predictive frame), and / or at least one B frame (bidirectional frame). As an example, the image frame group may include an I frame located at the beginning of the image frame group, at least one P frame located thereafter, and / or at least one B frame. In the present disclosure, an image frame group may be referred to as a GOP (group of pictures). In the present disclosure, an image frame may be abbreviated as a frame.

[0109] According to one embodiment, an I-frame corresponds to a key (e.g., important, essential) frame of a group of image frames. An I-frame is an independent image frame that does not depend on other image frames and can be decoded independently of other frames. An I-frame may be, for example, a frame that stores original image data as is. An I-frame has a larger capacity than other image frames. In the present disclosure, an I-frame may be referred to as a key frame.

[0110] According to one embodiment, a P frame may be a frame predicted with reference to a previous image frame (e.g., an I frame, a P frame, or a B frame). For example, at least one block (a block to be processed) belonging to a P frame may be predicted with reference to at least one block (a reference block) belonging to a previous frame. A P frame may be a frame that stores a difference from a previous I frame, P frame, or B frame, for example. A P frame has a smaller capacity than an I frame. In the present disclosure, a P frame may be referred to as a forward prediction frame or a unidirectional prediction frame.

[0111] According to one embodiment, a B frame may be a frame predicted with reference to a previous image frame (e.g., an I frame, a P frame, or a B frame) and a subsequent image frame (e.g., an I frame, a P frame, or a B frame). For example, at least one block (a block to be processed) belonging to a B frame may be predicted with reference to at least one block belonging to a previous frame and at least one block belonging to a subsequent frame (a reference block). A B frame may be, for example, a frame located between an I frame and a P frame, storing a difference between a previous frame (e.g., an I frame) and a subsequent frame (e.g., a P frame). A B frame has a smaller capacity than an I frame and a P frame. In the present disclosure, a B frame may be referred to as a bidirectional prediction frame.

[0112] According to one embodiment, an image frame group may have a short structure or a long structure depending on the number of image frames grouped into one image frame group. For example, if the number of image frames grouped into one image frame group is 6 or less, the image frame group may correspond to a short image frame group (short GOP), and if the number of image frames grouped into one image frame group is 12 or more, the image frame group may correspond to a long image frame group (long GOP).

[0113] Fig. 5a illustrates an example of a short group of image frames (short GOP). Referring to Fig. 5a, a first GOP may include, for example, one I frame, one P frame, and four B frames. For example, in the first GOP (first GOP), the first frame is an I frame, the second frame is a B frame referencing a previous I frame and a subsequent P frame, the third frame is a B frame referencing a previous I frame and a subsequent P frame, the fourth frame is a P frame referencing a previous I frame, the fifth frame is a B frame referencing a previous P frame and a subsequent I frame (e.g., an I frame of a second GOP (second GOP)), and the sixth frame is a B frame referencing a previous P frame and a subsequent I frame (e.g., an I frame of the second GOP). The second GOP may have, for example, the same frame structure as the first GOP.

[0114] Fig. 5b illustrates an example of a long group of image frames (long GOP). Referring to Fig. 5b, a GOP (first GOP) may include, for example, one I frame, four or more P frames, and eight or more B frames. For example, in the GOP, the first frame may be an I frame, the second frame may be a B frame referencing a previous I frame and a subsequent P frame, the third frame may be a B frame referencing a previous I frame and a subsequent P frame, and the fourth frame may be a P frame referencing a previous I frame. Similarly, the structure of two B frames and one P frame may be repeated within the GOP. An infinite GOP containing only one I frame for the entire image data may also be an example of a long GOP.

[0115] In one embodiment, when each image frame (e.g., I frame, P frame, and B frame) within a GOP is encoded with the same bit rate, the image quality of an image based on a P frame / B frame encoded with reference to another frame may be higher than that of an image based on an I frame. Accordingly, when each image frame is encoded with the same bit rate in this way, an image based on a long GOP and an infinite GOP may provide an image with a higher image quality because the proportion of B / P frames is higher than that of I frames compared to an image based on a short GOP.

[0116] FIG. 6A is a schematic flowchart of an exemplary image processing procedure of an image processing system according to one embodiment of the present disclosure.

[0117] The image processing procedure of FIG. 6a may be, for example, an image processing procedure for one GOP (e.g., the first GOP of FIG. 5a or FIG. 5b), but is not limited thereto.

[0118] Referring to FIG. 6A, in operation 6010, the transmitting device (110) and the receiving device (120) may establish a wireless connection. In one embodiment, the transmitting device (110) and the receiving device (120) may negotiate parameter(s) for establishing the wireless connection. The transmitting device (110) and the receiving device (120) may communicate with each other through the established wireless connection. In some embodiments, operation 6010 may be omitted.

[0119] In operation 6020, the transmitting device (110) may encode a first frame. In one embodiment, the transmitting device (110) may encode the first frame to generate an I frame. For example, as illustrated in FIGS. 5A and B, the transmitting device (110) may encode the first frame, which is, for example, the first frame of the first GOP, to generate an I frame.

[0120] In operation 6021, the transmitting device (110) may transmit an encoded first frame (e.g., an I frame) to the receiving device (120). According to one embodiment, the transmitting device (110) may transmit the encoded first frame to the receiving device (120) using a plurality of first data packets. As an example, the transmitting device (110) may transmit a first data packet including data of the encoded first frame to the receiving device (120).

[0121] According to one embodiment, each first data packet may include data (or information) of at least one block belonging to an encoded first frame. For example, data of multiple blocks of an I-frame may be included in one first data packet.

[0122] In operation 6022, the receiving device (120) may decode the encoded first frame. According to one embodiment, the receiving device (120) may receive a plurality of first data packets including data of the encoded first frame, and perform decoding on the encoded first frame based on the received first data packets. Through this, the first frame may be obtained (or restored).

[0123] In operation 6030, the transmitting device (110) may encode the Nth frame. Here, N≥2. According to one embodiment, the transmitting device (110) may encode, for example, the Nth frame of the GOP to which the first frame belongs, to generate a P frame (or a B frame). For example, as illustrated in FIGS. 5A and 5B , the transmitting device (110) may encode, for example, the second or third frame of the first GOP, to generate a B frame.

[0124] In operation 6031, the transmitting device (110) may transmit an encoded Nth frame (e.g., a P frame or a B frame) to the receiving device (120). According to one embodiment, the transmitting device (110) may transmit a plurality of Nth data packets including data of the encoded Nth frame to the receiving device (120).

[0125] According to one embodiment, each Nth data packet may include data (or information) of at least one block belonging to an encoded Nth frame. As an example, data of multiple blocks of a P frame or a B frame, each referencing multiple blocks of a previous frame, may be included in one Nth data packet.

[0126] In operation 6032, the receiving device (120) can decode the encoded Nth frame. According to one embodiment, the receiving device (120) can receive a plurality of Nth data packets including data of the encoded Nth frame, and perform decoding on the encoded Nth frame based on the received Nth data packets. Through this, the Nth frame can be obtained (or restored).

[0127] In operation 6040, the transmitting device (110) may encode the M-th frame. Here, M ≥ N. According to one embodiment, the transmitting device (110) may encode, for example, the M-th frame of the GOP to which the first frame belongs, to generate a P frame (or a B frame). For example, as illustrated in FIGS. 5A and 5B , the transmitting device (110) may encode, for example, the fourth frame of the first GOP, to generate a P frame.

[0128] In operation 6041, the transmitting device (110) may transmit an encoded M-th frame (e.g., a P-frame or a B-frame) to the receiving device (120). According to one embodiment, the transmitting device (110) may transmit a plurality of M-th data packets including data of the encoded M-th frame to the receiving device (120).

[0129] According to one embodiment, each M-th data packet may include data (or information) of at least one block belonging to an encoded M-th frame. For example, data of multiple blocks of a P-frame or a B-frame, each referencing multiple blocks of a previous frame, may be included in one M-th data packet.

[0130] In operation 6042, the receiving device (120) can decode the encoded M-th frame. According to one embodiment, the receiving device (120) can receive a plurality of M-th data packets including data of the encoded M-th frame, and perform decoding on the encoded M-th frame based on the received M-th data packets. Through this, the M-th frame can be obtained (or restored).

[0131] According to one embodiment, at operation 6022, 6032, or 6042, the receiving device (120) may initiate decoding of the encoded frame after all of the plurality of data packets containing data of the encoded frame have been received (hereinafter, referred to as a first decoding method). As an example, at operation 6032, the receiving device (120) may initiate decoding of the encoded N-th frame at a point in time when all of the plurality of N-th data packets containing data of the encoded N-th frame have been received.

[0132] In situations where latency is important (e.g., streaming situations in game mode), the receiving device (120) may initiate decoding of the encoded frame after some, but not all, of the data packets have been received (e.g., when the first data packet has been completely received). In one embodiment, at operations 6022, 6032, or 6042, the receiving device (120) may initiate decoding of the encoded frame immediately after receiving a portion of the plurality of data packets including data of the encoded frame (hereinafter, referred to as the second decoding method). As an example, at operation 6032, the receiving device (120) may initiate decoding of the encoded N-th frame at a point in time when a portion of the plurality of N-th data packets including data of the encoded N-th frame has been received.

[0133] For example, when using the second decoding method described above, if an error occurs in a data packet (e.g., a packet loss (or drop) or jitter occurs for the data packet due to network interference, etc.) and the data packet is not acquired by the receiving device at a time when processing for the data packet is required (e.g., at the exact time when decoding processing for the data packet is required), a decoding failure may occur for at least one block associated with the data packet. Hereinafter, a case where a decoding failure occurs in an image processing procedure will be described by way of example with reference to FIGS. 6b and 6c.

[0134] FIGS. 6c and 6c are diagrams illustrating a case where a decoding failure occurs in an image processing procedure according to one embodiment of the present disclosure.

[0135] In the embodiments of FIGS. 6b and 6c, for convenience of explanation, the frames following an I frame in the corresponding GOP are illustrated as P frames, but this is not limited thereto. For example, as illustrated in FIGS. 5a and 5b, at least one B frame may be included as a frame following an I frame.

[0136] According to one embodiment, the transmitting device (110) may encode a series of frames to generate encoded frames. For example, as illustrated in FIGS. 6b and 6c, the transmitting device (110) may encode a first frame (i.e., frame 1) to generate an I frame, encode a second frame (i.e., frame 2) to generate a first P frame referencing the I frame, encode a third frame (i.e., frame 3) to generate a second P frame referencing a previous frame (e.g., the I frame or the first P frame), encode a fourth frame (i.e., frame 4) to generate a third P frame referencing a previous frame (e.g., the I frame or the first P frame or the second P frame), and encode a fifth frame (i.e., frame 5) to generate a fourth P frame referencing a previous frame (e.g., the I frame or the first P frame or the second P frame or the third frame). A description of encoding a frame to generate an encoded frame may refer to the description of 6020, 6030 or 6040 of FIGS. 5A, 5B and 6A. Any duplicate descriptions may not be repeated here.

[0137] According to one embodiment, the transmitting device (110) may transmit the encoded frame to the receiving device (120). For example, as illustrated in FIGS. 6b and 6c, the I frame, the first P frame, the second P frame, the third P frame, and the fourth P frame may be transmitted to the receiving device (120) in the encoded time order. For a description of transmitting the encoded frame, reference may be made to the description of 6021, 6031, or 6041 of FIG. 6a. Any duplicate descriptions may not be repeated herein.

[0138] According to one embodiment, the receiving device (120) may receive an encoded frame and perform decoding on the encoded frame. For example, as illustrated in FIGS. 6b and 6c, decoding may be performed on an I frame, a first P frame, a second P frame, a third P frame, and a fourth P frame. A description of performing decoding on an encoded frame may refer to the description of 6022, 6032, or 6042 of FIG. 6a. Any duplicate descriptions thereof may not be repeated herein.

[0139] According to one embodiment, as described above in FIG. 6A, the receiving device (120) may initiate decoding of an encoded frame after some of the plurality of data packets containing data of the encoded frame have been received (second decoding method). For example, the receiving device (120) may initiate decoding of an I-frame at the time when some of the plurality of data packets containing data of the I-frame have been received (e.g., at the time when the first data packet has been completely received).

[0140] According to one embodiment, when the second decoding method described above is used, if an error occurs in a data packet and the data packet is not delivered to the receiving device at the exact time when processing of the data packet is required, a decoding failure occurs for at least one block associated with the data packet. For example, as illustrated in FIGS. 6b and 6c, an error (e.g., packet drop or jitter) may occur in some of the plurality of packets including data of the second P frame (frame 3) and the data packet(s) may not be delivered to the receiving device at the exact time, which may result in a decoding failure for at least one block (e.g., two blocks) associated with the data packet(s) within the second P frame.

[0141] According to one embodiment, when a decoding failure occurs for at least one block within a frame, screen tearing may occur in a portion associated with at least one block within the frame where the decoding failure occurred. For example, as illustrated in FIGS. 6b and 6c, when a decoding failure occurs for two blocks within a second P frame, screen tearing may occur in a portion of a frame (i.e., frame 3) that is the result of decoding the second P frame (e.g., a portion associated with two blocks of the second P frame where the decoding failure occurred).

[0142] According to one embodiment, at least one block in which a decoding failure occurred may be a block that is directly or indirectly referenced by at least one block in at least one subsequent frame. For example, as illustrated in FIG. 6B, a first block of two blocks in which a decoding failure occurred in a second P frame may be a block that is directly referenced by a first block of a third P frame that is the frame immediately following the second P frame and a first block of a fourth P frame that is the frame immediately following the third P frame, and a second block of two blocks in which a decoding failure occurred may be a block that is directly referenced by a second block of the third P frame. For example, as illustrated in FIG. 6c, among the two blocks in which a decoding failure occurred in the second P frame, the first block and the second block may be blocks directly referenced by the first block and the second block of the third P frame, which is the frame immediately following the second P frame, respectively, and the second block of the two blocks in which a decoding failure occurred may be a block indirectly referenced by the first block of the fourth P frame, which is the frame immediately following the third P frame. In the example of FIG. 6c, the first block of the fourth frame may indirectly reference the second block of the second P frame, for example, through the second block of the third P frame.

[0143] In one embodiment, a block that directly or indirectly references a block in which a decoding failure occurred may also experience a decoding failure. This is because the receiving device uses the block that was not decoded normally (or successfully) to decode the block that directly or indirectly references it.

[0144] For example, as illustrated in FIG. 6B, the first block of the third P frame and the first block of the fourth P frame, which directly reference the first block in the second P frame where a decoding failure occurred, and the second block of the third P frame, which directly references the second block in the second P frame where a decoding failure occurred, also experience decoding failure. In this case, screen tearing may occur in a part of the frame (i.e., frame 4) in which the third P frame is decoded (e.g., a part associated with the first block and the second block of the third P frame in which a decoding failure occurred), and screen tearing may occur in a part of the frame (frame 5) in which the fourth P frame is decoded (e.g., a part associated with the first block of the fourth P frame in which a decoding failure occurred).

[0145] For example, as illustrated in FIG. 6C, the first block and the second block of the third P frame, which directly reference the first block and the second block in the second P frame where the decoding failure occurred, may also experience a decoding failure. In addition, by referencing the second block of the third frame where the decoding failure occurred, the first block of the fourth P frame, which indirectly references the second block in the second P frame where the decoding failure occurred, may also experience a decoding failure. In this case, screen tearing may occur in a part of the frame in which the third P frame is decoded (i.e., frame 4) (e.g., a part associated with the first block and the second block of the third P frame where the decoding failure occurred), and screen tearing may occur in a part of the frame in which the fourth P frame is decoded (i.e., frame 5) (e.g., a part associated with the first block of the fourth P frame where the decoding failure occurred).

[0146] In this way, when the transmitting device refers to a block in which a decoding failure occurred in the receiving device when encoding subsequent frames (e.g., P frames or B frames), the receiving device cannot successfully decode at least one block in the subsequent frame(s) that refers to the block in which the decoding failure occurred. This may result in persistent or serial screen (or image) disruption. For example, as illustrated in FIGS. 6b and 6c, screen disruption occurs not only in the screen of the second P frame that includes blocks in which a decoding failure occurred due to an error in the data packet, but also in the subsequent third P frame and fourth P frame that include at least one block that directly or indirectly refers to blocks in the second P frame.

[0147] Below, various embodiments are described for preventing and / or reducing the occurrence of chain decoding failures by processing the block(s) of a frame in which a decoding failure occurred so that they cannot be referenced in subsequent frames.

[0148] FIG. 7 is a signal flow diagram illustrating an exemplary image processing procedure of an image processing system according to one embodiment of the present disclosure.

[0149] The image processing procedure of FIG. 7 may be, for example, an image processing procedure for one GOP (e.g., the first GOP of FIG. 5a or FIG. 5b), but is not limited thereto.

[0150] Referring to FIG. 7, in operation 7010, the transmitting device (110) and the receiving device (120) may establish a wireless connection. According to one embodiment, the transmitting device (110) and the receiving device (120) may negotiate parameter(s) for establishing the wireless connection. The transmitting device (110) and the receiving device (120) may communicate with each other through the established wireless connection. Depending on the embodiment, operation 7010 may be omitted.

[0151] In operation 7020, the transmitting device (110) may encode the Nth frame. Here, N≥2. According to one embodiment, the transmitting device (110) may generate a P frame (or a B frame) by encoding, for example, the Nth frame of a GOP to which a previous frame referenced by the Nth frame belongs. For example, as illustrated in FIGS. 5A and 5B , the transmitting device (110) may generate a P frame by encoding, for example, the fourth frame of the first GOP. According to one embodiment, the first GOP is a long GOP or an infinite GOP, and each frame belonging to the first GOP may be encoded using the same bit rate.

[0152] In operation 7021, the transmitting device (110) may transmit an encoded Nth frame (e.g., a P frame or a B frame) to the receiving device (120). According to one embodiment, the transmitting device (110) may transmit a plurality of Nth data packets including data of the encoded Nth frame to the receiving device (120).

[0153] In operation 7022, the receiving device (120) can decode the encoded Nth frame. According to one embodiment, the receiving device (120) can receive a plurality of Nth data packets including data of the encoded Nth frame, and can perform decoding on the encoded Nth frame based on the received Nth data packets. As an example, the receiving device (120) can be configured to initiate decoding on the first frame after some of the plurality of data packets including data in the Nth frame transmitted from the transmitting device (110) are received.

[0154] According to one embodiment, each Nth data packet may include data (or information) of at least one block belonging to an encoded Nth frame. As an example, data of multiple blocks of a P frame or a B frame, each referencing multiple blocks of a previous frame, may be included in one Nth data packet.

[0155] In operation 7023, the receiving device (120) can identify whether a decoding failure occurs for the encoded N frames. According to one embodiment, if it is identified that one or more data packets among the plurality of data packets including data in the N frame transmitted from the transmitting device are not obtained at a time when processing for the corresponding data packets is required, the receiving device (120) can identify that a decoding failure occurs for the N frame. For example, if the data packet is dropped or received after a time when processing for the data packet is required due to jitter, the receiving device (120) can identify that the corresponding data packet is not obtained at a time when processing for the corresponding data packet is required.

[0156] If it is identified that a decoding failure has occurred, actions 7024 and 7025 may be performed. If it is identified that a decoding failure has not occurred, actions 7024 and 7025 may not be performed.

[0157] In operation 7024, the receiving device (120) may generate information about block(s) in which a decoding failure occurred. According to one embodiment, the information about block(s) in which a decoding failure occurred may be used by the transmitting device to identify the block(s) in which the decoding failure occurred and the frame(s) to which the corresponding block(s) belong. According to one embodiment, the information about block(s) in which a decoding failure occurred may include block ID information of the block in which the decoding failure occurred and frame ID information of the frame to which the corresponding block(s) belong. According to one embodiment, the block ID information may include information for identifying the block in which the decoding failure occurred (e.g., block number, block ID), and the frame ID information may include information for identifying the frame to which the block in which the decoding failure occurred belongs (e.g., frame number, frame ID). In the present disclosure, a block in which a decoding failure occurred may be referred to as a damaged block, and the information about block(s) in which the decoding failure occurred may be referred to as damaged block-related information.

[0158] In operation 7025, the receiving device (120) can transmit information about the block(s) in which the generated decoding failure occurred (damaged block related information) to the transmitting device (110).

[0159] In operation 7030, the transmitting device (110) can identify whether information about the block(s) in which a decoding failure occurred (information about damaged blocks) has been received from the receiving device (120). If it is identified that information about the block(s) in which a decoding failure occurred has been received, operation 7031 can be performed. If it is identified that information about the block(s) in which a decoding failure occurred has not been received, operation 7031 may not be performed and operation 7032 may be performed directly.

[0160] In operation 7031, the transmitting device (110) may process the block(s) in which the decoding failure occurred so that the block(s) in which the decoding failure occurred are not referenced by subsequent frame(s) based on information about the block(s) in which the decoding failure occurred (information related to damaged blocks). According to one embodiment, the transmitting device may identify the block(s) in which the decoding failure occurred and the frame to which the block(s) belong based on the information about the block(s) in which the decoding failure occurred, and process the block(s) in which the decoding failure occurred within the frame so that the block(s) in which the decoding failure occurred are not referenced in the frame(s) following the frame.

[0161] In operation 7032, the transmitting device (110) may encode the M-th frame. Here, M>N may be the case. According to one embodiment, the transmitting device (110) may encode, for example, the M-th frame of the GOP to which the first frame belongs, to generate a P frame (or a B frame). As an example, as illustrated in FIGS. 5A and 5B , the transmitting device (110) may encode, for example, the sixth frame of the first GOP, to generate a B frame that references a previous P frame and a subsequent I frame (or a subsequent P frame).

[0162] According to one embodiment, the M-frame may be a frame encoded after information about the block(s) in which a decoding failure occurred (damaged block related information) is received.

[0163] According to one embodiment, the transmitting device (110) can encode the M-frame based on information about the block(s) in which a decoding failure occurred (damaged block related information).

[0164] According to one embodiment, the transmitting device (110) can identify the block(s) in which a decoding failure occurred and the Nth frame to which the block(s) belong based on information about the block(s) in which a decoding failure occurred (information related to damaged blocks), and process the block(s) in which a decoding failure occurred within the Nth frame so that they are not referenced by the block(s) within the Mth frame.

[0165] For example, in order to prevent the block(s) (hereinafter, first block(s)) in which a decoding failure occurred within the Nth frame from being referenced by the block(s) in the Mth frame, the transmitting device (110) may mask the first block(s) in which a decoding failure occurred within the Nth frame stored in the frame buffer. The first block(s) thus masked cannot be referenced by blocks in other subsequent frames.

[0166] For example, the transmitting device (110) may identify block(s) (hereinafter, second block(s)) set to reference block(s) in which decoding failure occurred in the N-th frame among the blocks in the M-th frame, and process (or generate) the second block(s) as an intra block, in order to process the first block(s) in which decoding failure occurred in the N-th frame so that they are not referenced by the block(s) in the M-th frame. According to one embodiment, the intra block may be an independent block that does not reference other blocks (or does not depend on other blocks). In the present disclosure, generating (or processing) a block set to reference a block in which decoding failure occurred as an intra block may be referred to as an intra refresh.

[0167] For example, the transmitting device (110) may identify block(s) (hereinafter, third block(s)) referencing the first block(s) in at least one frame encoded before receiving information related to damaged blocks, so that the first block(s) in which a decoding failure occurred in the N-th frame are not referenced by the block(s) in the M-th frame, identify block(s) (hereinafter, second block(s)) set to refer to at least one of the first block(s) or the third block(s) among the blocks in the M-th frame, and process (or generate) the second block(s) as an intra block.

[0168] By processing such that the block(s) in which such decoding failure occurred are not referenced by blocks in subsequent frames, no block in the encoded M-frame can reference the block(s) in which the decoding failure occurred. Accordingly, the M-frame decoded by the receiving device (120) can be free from screen breakage.

[0169] In operation 7033, the transmitting device (110) may transmit an encoded M-th frame (e.g., a P-frame or a B-frame) to the receiving device (120). According to one embodiment, the transmitting device (110) may transmit a plurality of M-th data packets including data of the encoded M-th frame to the receiving device (120).

[0170] According to one embodiment, each M-th data packet may include data (or information) of at least one block belonging to the encoded M-th frame. For example, data of multiple blocks of a P-frame or a B-frame, each referencing multiple blocks of a previous frame (e.g., an N-th frame), may be included in one N-th data packet.

[0171] Meanwhile, in the embodiment of FIG. 7 described above, for the convenience of explanation, the image processing procedures of the transmitting device (110) and the receiving device (120) for N frames are exemplarily described, but the explanation of the embodiment of FIG. 7 is not limited to the image processing procedures for N frames. For example, the same embodiment can be applied to an I frame, which is the first frame of a GOP to which an N frame belongs, and an M frame, which is a frame following the N frame. For example, the explanation of operations 7022 to 7025 and operations 7030 to 7031 of FIG. 7 can be equally applied to the I frame and the M frame.

[0172] FIG. 8 is a flowchart illustrating an exemplary operation of a transmitting device in an image processing procedure according to one embodiment of the present disclosure.

[0173] The image processing procedure of the embodiment of Fig. 8 may be, for example, an example of the image processing procedure of Fig. 7. The description of the image processing procedure of Fig. 7 may be referred to for the description of the image processing procedure of Fig. 8.

[0174] Referring to FIG. 8, in operation 8010, a transmitting device (e.g., the transmitting device (110) of FIGS. 1A, 1B, and 2) may establish a wireless connection with a receiving device (e.g., the receiving device (120) of FIGS. 1A, 1C, and 2). The description of operation 8010 may refer to the description of operation 7010. Therefore, any duplicate description may not be repeated herein. Depending on the embodiment, operation 8010 may be omitted.

[0175] In operation 8020, the transmitting device (110) may encode the Nth frame, where N≥2. Operation 8020 may be, for example, an example of operation 7020 of FIG. 7.

[0176] According to one embodiment, the transmitting device (110) may encode the Nth frame (Nth frame) by referring to at least one of the frames stored in the frame buffer. For example, as illustrated in FIGS. 13a and 13b, the transmitting device (110) may generate a P frame by encoding a third frame, for example, by referring to a first frame or a second frame stored in the frame buffer. The frame encoded in this way may be transmitted to the receiving device (120). For example, as illustrated in FIGS. 13a and 13b, the transmitting device (110) may transmit at least one data packet including data of the generated P frame to the receiving device (120). Each data packet may include, for example, data (or information) of at least one associated block belonging to the P frame.

[0177] In operation 8030, the transmitting device (110) may receive information about block(s) in which a decoding failure occurred (damaged block related information) from the receiving device (120). For example, as illustrated in FIGS. 13A and 13B , the transmitting device (110) may receive information about block(s) in which a decoding failure occurred belonging to frame 3 from the receiving device (120). If it is determined that information about block(s) in which a decoding failure occurred has been received, operation 8040 may be performed. If it is determined that information about block(s) in which a decoding failure occurred has not been received, operation 8020 may be performed again without performing operation 8040. For a description of damaged block related information and reception of the information, reference may be made to operations 7024 and 7025 of FIG. 7 .

[0178] In operation 8040, the transmitting device (110) may process the block(s) in which the decoding failure occurred so that the block(s) in which the decoding failure occurred are not referenced by subsequent frame(s) based on information about the block(s) in which the decoding failure occurred (damaged block related information). For example, as illustrated in FIGS. 13A and 13B , the transmitting device (110) may identify the first block(s) in which the decoding failure occurred within frame 3 based on the damaged block related information received from the receiving device (120), and may mask the first block(s) in frame 3 stored in the frame buffer so that the first block(s) in which the decoding failure occurred are not referenced by the second block(s) in frame 5. For example, as illustrated in FIG. 13a, the transmitting device (110) may identify the first block(s) in which a decoding failure occurred within the third frame based on the damaged block related information received from the receiving device (120), and may generate (or process) the second block(s) set to refer to the first block(s) as an intra block so that the first block(s) in which the decoding failure occurred are not directly referenced by the second block(s) within the fifth frame. For example, as illustrated in FIG. 13A, the transmitting device (110) may identify the first block(s) in which a decoding failure occurred within the third frame based on the damaged block related information received from the receiving device (120), identify the third block(s) referencing the first block(s) within the fourth frame, and generate (or process) the second block(s) that are set to refer to the third block(s) as an intra block so that the first block(s) in which a decoding failure occurred are not indirectly referenced by the second block(s) within the fifth frame. The description of operation 8040 may refer to the description of operation 7031 of FIG. 7. Therefore, the duplicate description may not be repeated here. Thereafter, the transmitting device (110) may perform operation 8020 again.

[0179] In operation 8020, the transmitting device (110) can encode the N+1th frame by referring to at least one of the frames stored in the frame buffer.

[0180] In this way, the transmitting device (110) can encode frames belonging to the corresponding GOP in chronological order by repeatedly performing operations 8020 to 8040 and transmit them to the receiving device (120). At this time, after the damaged block information for the corresponding frame is received, none of the blocks included in the encoded frame will refer to the block(s) in which a decoding failure has occurred, identified based on the damaged block information. Therefore, in the case of the frame encoded after the damaged block information is received, screen breakage does not occur. In other words, the occurrence of serial screen breakage can be prevented and / or reduced.

[0181] FIG. 9 is a flowchart illustrating an exemplary operation of a transmitting device in an image processing procedure according to one embodiment of the present disclosure.

[0182] The image processing procedure of the embodiment of FIG. 9 may be, for example, an example of the image processing procedure of FIG. 7. The description of the image processing procedure of FIG. 7 may be referred to for the description of the image processing procedure of FIG. 9.

[0183] Referring to FIG. 9, in operation 9010, a transmitting device (e.g., the transmitting device (110) of FIGS. 1A, 1B, and 2) may establish a wireless connection with a receiving device (e.g., the receiving device (120) of FIGS. 1A, 1C, and 2). The description of operation 9010 may refer to the description of operation 7010. Therefore, any duplicate description may not be repeated herein. Depending on the embodiment, operation 9010 may be omitted.

[0184] In operation 9020, the receiving device (120) may receive at least one Nth data packet (e.g., a video packet) from the transmitting device (110) that includes data of an encoded Nth frame (e.g., a P frame or a B frame). According to one embodiment, the receiving device (120) may receive from the transmitting device (110) an Nth data packet (e.g., a video packet) that includes data (or information) associated with at least one block of the encoded Nth frame. As an example, as illustrated in FIGS. 13A and 13B , the receiving device (120) may receive from the transmitting device (110) a data packet that includes data associated with at least one block of a third frame (e.g., a P frame).

[0185] In operation 9030, the receiving device (120) may identify whether a decoding failure occurs. According to one embodiment, the receiving device (120) may perform decoding on the received Nth data packet to identify whether a decoding failure occurs for the Nth data packet (or at least one block of an encoded Nth frame included in the Nth data packet). For example, as illustrated in FIGS. 13A and 13B , the receiving device (120) may identify whether a decoding failure occurs for a data packet including data associated with at least one block of a third frame (e.g., a P frame). For example, if the data packet is dropped or jitter occurs and the receiving device (120) does not obtain the packet at a time when processing (e.g., decoding processing) for the data packet is required, the receiving device (120) may identify that a decoding failure has occurred for the data packet. If it is identified that a decoding failure has occurred, operation 9040 may be performed. If it is identified that a decoding failure has not occurred, operation 9020 may be performed again without performing operation 9040.

[0186] In operation 9040, the receiving device (120) may generate information about the block(s) in which a decoding failure occurred (damaged block related information) and transmit the information to the transmitting device (110). For example, as illustrated in FIGS. 13A and 13B , the receiving device (120) may transmit damaged block related information including block ID information of the block in which a decoding failure occurred within frame 3 and frame ID information of frame 3 to the transmitting device (110). The description of operation 9040 may refer to the description of operations 7024 and 7025 of FIG. 7 . Therefore, any duplicate description may not be repeated here. Thereafter, operation 9020 may be performed again.

[0187] At operation 9020, the receiving device (120) may receive the Nth data packet to be received thereafter. Thereafter, the receiving device (120) may perform operations 9030 to 9040 again.

[0188] In this way, the receiving device (120) can transmit information about the block(s) in which decoding failure occurred and the frame including the block(s) (damaged block related information) to the transmitting device (110) by repeatedly performing operations 9020 to 9040. By transmitting this damaged block related information, the transmitting device (110) can process the frame to be encoded after the damaged block related information is received so as not to refer to the block(s) identified by the damaged block related information. The frames processed in this way do not experience decoding failure and thus do not experience image breakage.

[0189] FIG. 10 is a signal flow diagram illustrating an exemplary image processing procedure of an image processing system according to one embodiment of the present disclosure.

[0190] The image processing procedure of Fig. 10 may be, for example, an image processing procedure for one GOP (e.g., the first GOP of Fig. 5a or Fig. 5b), but is not limited thereto. The image processing procedure of Fig. 10 may be an example of the image processing procedure of Fig. 7. Accordingly, the description of the image processing procedure of Fig. 7 may be referred to for the description of the image processing procedure of Fig. 10.

[0191] Referring to FIG. 10, in operation 10010, a transmitting device (110) and a receiving device (120) may establish a wireless connection. According to one embodiment, the transmitting device (110) and the receiving device (120) may negotiate parameter(s) for establishing a wireless connection. The transmitting device (110) and the receiving device (120) may communicate with each other through the established wireless connection. Depending on the embodiment, operation 10010 may be omitted.

[0192] In operation 10020, the transmitting device (110) may encode the Nth frame. Here, N ≥ 2. The description of operation 10020 may refer to the description of operation 7020 of FIG. 7. Therefore, any duplicate description may not be repeated here.

[0193] In operation 10021, the transmitting device (110) may transmit the encoded Nth frame (e.g., a P frame or a B frame) to the receiving device (120). The description of operation 10021 may refer to the description of operation 7021 of FIG. 7. Therefore, any duplicate description may not be repeated here.

[0194] In operation 10022, the receiving device (120) can decode the encoded Nth frame. The description of operation 10022 may refer to the description of operation 7022 of FIG. 7. Therefore, any duplicate description may not be repeated here.

[0195] In operation 10023, the receiving device (120) may identify a damaged block in which a decoding failure has occurred. According to one embodiment, if it is identified that one or more data packets among a plurality of data packets including data in an N-th frame transmitted from a transmitting device are not obtained at a time when processing of the corresponding data packets is required, the receiving device (120) may identify that a decoding failure has occurred for the N-th frame, and identify a block (damaged block) in which the decoding failure has occurred within the N-th frame based on the data packet in which the decoding failure has occurred (e.g., a sequence number of the data packet). For example, if a data packet is dropped or received after a time when processing of the data packet is required due to jitter, the receiving device (120) may identify that the data packet in which the data packet is not obtained at a time when processing of the corresponding data packet is required, and identify a block (damaged block) in which the decoding failure has occurred within the N-th frame based on the data packet in which the decoding failure has occurred (e.g., a sequence number of the data packet). The description of operation 10023 may refer to the description of operation 7023 of Fig. 7. Therefore, any duplicate description may not be repeated here.

[0196] In operation 10024, the receiving device (120) may generate damaged block-related information including block ID information of the damaged block in which a decoding failure occurred and frame ID information of the frame to which the damaged block belongs. The description of operation 10024 may refer to the description of operation 7024 of FIG. 7. Therefore, any duplicate description may not be repeated herein.

[0197] In operation 10025, the receiving device (120) may transmit damaged block related information to the transmitting device (110).

[0198] In operation 10030, the transmitting device (110) may encode the M-th frame, which is encoded after the damaged block-related information is received, based on the damaged block-related information for the N-th frame. The description of operation 10030 may refer to the description of operations 7030, 7031, and 7032 of FIG. 7. Therefore, any duplicate description may not be repeated herein.

[0199] According to one embodiment, the transmitting device (110) may receive the damaged block related information for the N-th frame at a point in time a frame (e.g., 2 frames) after the N-th frame due to a delay time. The transmitting device (110) may not perform encoding processing for the frame using the damage related information for the N-th frame from the N-th frame to the frame prior to receiving the damaged block information for the N-th frame. Therefore, at least one block of the frames prior to receiving the damage information for the N-th frame may refer to the damaged blocks in which the decoding failure of the N-th frame occurred, which causes image corruption.

[0200] According to one embodiment, the delay time may be determined based on, for example, a decoding time of the receiving device, a generation time of the damaged block-related information, and / or a transmission time of the damaged block-related information.

[0201] In operation 10031, the transmitting device (110) may transmit an encoded M-frame (e.g., a P-frame or a B-frame) to the receiving device (120). The description of operation 10031 may refer to the description of operation 7033 of FIG. 7. Any duplicate descriptions may not be repeated herein.

[0202] FIG. 11A is a flowchart illustrating an exemplary operation of a transmitting device in an image processing procedure according to one embodiment of the present disclosure.

[0203] The image processing procedure of the embodiment of FIG. 11a may be, for example, an example of operation 10030 of the image processing procedure of FIG. 10. The description of the image processing procedure of FIG. 11a may be referred to for the description of the image processing procedure of FIG. 10.

[0204] Referring to FIG. 11A, in operation 11010a, the transmitting device (110) may receive, from the receiving device (120), damaged block-related information including block ID information of a first damaged block in which a decoding failure occurred and frame ID information of a first frame to which the first damaged block belongs. For example, as illustrated in FIG. 13A, the transmitting device (110) may receive damaged block-related information for frame 3 from the receiving device (120). The damaged block-related information for frame 3 may include block ID information of the first damaged block in frame 3 and frame ID information of frame 3.

[0205] In operation 11020a, the transmitting device (110) may identify the first damaged block within the first frame based on the damaged block-related information. For example, as illustrated in FIG. 13a, the transmitting device may identify the first damaged block within the third frame based on the damaged block-related information for the third frame.

[0206] In operation 11030a, the transmitting device (110) can identify at least one block referencing the first damaged block in a second frame encoded after the damaged block-related information is received. For example, as illustrated in FIG. 13a, the transmitting device (110) can identify a block referencing the damaged block in frame 3 in frame 5 (or from frame 5) encoded after the damaged block-related information is received.

[0207] In operation 11040a, the transmitting device (110) may generate at least one block referencing the first damaged block within the second frame as an intra block. For example, as illustrated in FIG. 13a, the transmitting device (110) may generate a block directly referencing the damaged block of frame 3 within (or from) frame 5 as an intra block.

[0208] In operation 11050a, the transmitting device (110) may encode a second frame including at least one block generated as an intra block. For example, as illustrated in FIG. 13a, the transmitting device (110) may encode frame 5 including a block generated as an intra block to generate a P frame (or B frame).

[0209] FIG. 11b is a flowchart illustrating an exemplary operation of a transmitting device in an image processing procedure according to one embodiment of the present disclosure.

[0210] The image processing procedure of the embodiment of FIG. 11b may be, for example, an example of operation 10030 of the image processing procedure of FIG. 10. The description of the image processing procedure of FIG. 11b may be referred to for the description of the image processing procedure of FIG. 10.

[0211] Referring to FIG. 11b, in operation 11010b, the transmitting device (110) may receive, from the receiving device (120), damaged block-related information including block ID information of a first damaged block in which a decoding failure occurred and frame ID information of a first frame to which the first damaged block belongs. For example, as illustrated in FIG. 13b, the transmitting device (110) may receive damaged block-related information for frame 3 from the receiving device (120). The damaged block-related information for frame 3 may include block ID information of the first damaged block in frame 3 and frame ID information of frame 3.

[0212] In operation 11020b, the transmitting device (110) can identify the first damaged block within the first frame based on the damaged block-related information. For example, as illustrated in FIG. 13b, the transmitting device can identify the first damaged block within the third frame based on the damaged block-related information for the third frame.

[0213] In operation 11030b, the transmitting device (110) can identify at least one second damaged block that references the first damaged block in at least one frame encoded before the damaged block-related information is received. For example, as illustrated in FIG. 13b, the transmitting device (110) can identify the second damaged block that references the first damaged block in frame 3 in frame 4 encoded before the damaged block-related information is received.

[0214] In operation 11040b, the transmitting device (110) can identify at least one block that references at least one of the first damaged block or at least one second damaged block in a second frame that is encoded after the damaged block-related information is received. For example, as illustrated in FIG. 13b, the transmitting device (110) can identify a block that references the second damaged block in frame 4 in frame 5 that is encoded after the damaged block-related information is received. This block in frame 5 can indirectly reference the first damaged block in frame 3 through the second damaged block in frame 4.

[0215] In operation 11050b, the transmitting device (110) may generate at least one identified block within the second frame as an intra block. For example, as illustrated in FIG. 13b, the transmitting device (110) may generate a block among the blocks within the fifth frame that references the second damaged block of the fourth frame as an intra block.

[0216] In operation 11060b, the transmitting device (110) may encode a second frame including at least one block generated as an intra block. For example, as illustrated in FIG. 13a, the transmitting device (110) may encode frame 5 including a block generated as an intra block to generate a P frame (or a B frame).

[0217] FIG. 12A is a flowchart illustrating an exemplary operation of a receiving device in an image processing procedure according to one embodiment of the present disclosure.

[0218] The image processing procedure of the embodiment of FIG. 12a may be, for example, an example of the image processing procedure of FIG. 10. The description of the image processing procedure of FIG. 12a may be referred to for the description of the image processing procedure of FIG. 10.

[0219] Referring to FIG. 12A, in operation 12010a, the receiving device (120) may receive at least one data packet including data of an encoded first frame from the transmitting device (110). For example, as illustrated in FIGS. 13A and 13B, the receiving device (120) may receive at least one data packet including data of a third frame (e.g., a P frame) from the transmitting device (110). The description of operation 12010a may refer to the description of operation 10021 of FIG. 10. Any duplicate descriptions may not be repeated herein.

[0220] In operation 12020a, the receiving device (120) may perform decoding on at least one data packet. For example, as illustrated in FIGS. 13a and 13b , the receiving device (120) may perform decoding on at least one data packet for frame 3. The description of operation 12020a may refer to the description of operation 10022 of FIG. 10 . Any duplicate descriptions thereof may not be repeated herein.

[0221] In operation 12030a, the receiving device (120) may identify at least one data packet in which a decoding failure has occurred. For example, as illustrated in FIGS. 13a and 13b , the receiving device (120) may identify at least one data packet for frame 3, which includes data (or information) associated with a first damaged block, as a packet in which a decoding failure has occurred. For example, if the receiving device (110) identifies that at least one data packet among a plurality of data packets including data for frame 3 transmitted from the transmitting device (120) is not acquired at a time when processing for the corresponding data packet is required, the receiving device (110) may identify that a decoding failure has occurred for the corresponding data packet. For example, the receiving device (110) may identify that the corresponding data packet is not acquired at a time when processing for the corresponding data packet is required if the corresponding data packet is dropped or received after a time when processing for the corresponding data packet is required due to jitter.

[0222] A data packet that experiences a decoding failure may include data from at least one first damaged block. For example, as illustrated in FIGS. 13A and 13B , a data packet that experiences a decoding failure may include data from at least one first damaged block within frame 3.

[0223] In operation 12040a, the receiving device (120) may identify at least one block associated with a packet in which a decoding failure occurred as a damaged block in which a decoding failure occurred. For example, as illustrated in FIGS. 13a and 13b , the receiving device (120) may identify the first damaged block within frame 3 associated with the packet(s) in which a decoding failure occurred as a damaged block in which a decoding failure occurred.

[0224] FIG. 12b is a flowchart illustrating an exemplary operation of a receiving device in an image processing procedure according to one embodiment of the present disclosure.

[0225] The image processing procedure of the embodiment of FIG. 12b may be an example of a process in which a receiving device, including operations 10023 to 10025 of the image processing procedure of FIG. 10, generates and transmits damaged block-related information. The description of the image processing procedure of FIG. 12b may be referred to for the description of the image processing procedure of FIG. 10.

[0226] In operation 12010b, the receiving device (120) may identify the first damaged block in which a decoding failure occurred. For example, as illustrated in FIGS. 13a and 13b , the receiving device (120) may identify the first damaged block in frame 3 as the damaged block in which a decoding failure occurred. The description of operation 12010b may refer to the description of operation 10023 of FIG. 10 . Therefore, any duplicate description may not be repeated herein.

[0227] In operation 12020b, the receiving device (120) may generate damaged block-related information including block ID information of the first damaged block in which a decoding failure occurred and frame ID information of the first frame to which the first damaged block belongs. For example, as illustrated in FIGS. 13a and 13b, the receiving device (120) may generate damaged block-related information including block ID information of the first damaged block in frame 3 and frame ID information of frame 3. The description of operation 12020b may refer to the description of operation 10024 of FIG. 10. Therefore, any duplicate description may not be repeated herein.

[0228] In operation 12030b, the receiving device (120) may transmit damaged block related information to the transmitting device (110). For example, as illustrated in FIGS. 13a and 13b, the receiving device (120) may transmit damaged block related information including block ID information of the first damaged block in frame 3 and frame ID information of frame 3 to the transmitting device (110).

[0229] FIGS. 13A and 13B are diagrams illustrating an exemplary method for preventing / reducing decoding failures in an image processing procedure according to one embodiment of the present disclosure.

[0230] In the embodiments of FIGS. 13a and 13b, for convenience of explanation, the frames following an I frame are illustrated as P frames, but this is not limited thereto. For example, as illustrated in FIGS. 5a and 5b, at least one B frame may be included as a frame following an I frame.

[0231] The embodiment of FIG. 13a may be an example of a method for preventing and / or reducing decoding failures through intra-refresh for blocks that directly reference a block in which a decoding failure has occurred, identified based on damaged block-related information.

[0232] The embodiment of FIG. 13b may be an example of a method for preventing and / or reducing decoding failures through intra-refresh for blocks that indirectly reference a block in which a decoding failure has occurred, identified based on damaged block-related information.

[0233] According to one embodiment, the transmitting device (110) may encode a series of frames in chronological order to generate encoded frames. For example, as illustrated in FIGS. 13b and 13c, the transmitting device (110) may encode a first frame (frame 1) to generate an I frame, encode a second frame (frame 2) to generate a first P frame referencing the I frame, encode a third frame (frame 3) to generate a second P frame referencing a previous frame (e.g., an I frame or a first P frame), encode a fourth frame (frame 4) to generate a third P frame referencing a previous frame (e.g., an I frame or a first P frame or a second P frame), and encode a fifth frame (frame 5) to generate a fourth P frame referencing a previous frame (e.g., an I frame or a first P frame or a second P frame or a third frame). A description of encoding a frame to generate an encoded frame may refer to the description of 6020, 6030 or 6040 of FIGS. 5 and 6a. Any duplicate descriptions may not be repeated here.

[0234] According to one embodiment, the transmitting device (110) may transmit the encoded frame to the receiving device (120). For example, as illustrated in FIGS. 13A and 13B , the I frame, the first P frame, the second P frame, the third P frame, and the fourth P frame may be transmitted to the receiving device (120) in chronological order. For a description of transmitting the encoded frame, reference may be made to the description of 6021, 6031, or 6041 of FIG. 6 . Any duplicate descriptions thereof may not be repeated herein.

[0235] According to one embodiment, the receiving device (120) may receive an encoded frame and perform decoding on the encoded frame. For example, as illustrated in FIGS. 13A and 13B , decoding may be performed on an I frame, a first P frame, a second P frame, a third P frame, and a fourth P frame. For a description of performing decoding on an encoded frame, reference may be made to the description of 6022, 6032, or 6042 of FIG. 6A . Any duplicate descriptions thereof may not be repeated herein.

[0236] According to one embodiment, as described above in FIG. 6A, the receiving device (120) may initiate decoding of an encoded frame after some of the plurality of data packets containing data of the encoded frame have been received (second decoding method). For example, the receiving device (120) may initiate decoding of an I-frame at the time when some of the plurality of data packets containing data of the I-frame have been received (e.g., at the time when the first data packet has been completely received).

[0237] According to one embodiment, when the second decoding method described above is used, if an error occurs in a data packet and the data packet is not delivered to the receiving device at the exact time when processing of the data packet is required, a decoding failure may occur for at least one block associated with the data packet. For example, as illustrated in FIGS. 13A and 13B , an error (e.g., packet drop or jitter) may occur in some of the plurality of packets including data of the second P frame (frame 3) and the data packet(s) may not be delivered to the receiving device at the exact time, which may result in a decoding failure for at least one block (e.g., two blocks) associated with the data packet(s) within the second P frame.

[0238] According to one embodiment, when a decoding failure occurs for at least one block within a frame, screen tearing may occur in a portion associated with at least one block within the frame where the decoding failure occurred. For example, as illustrated in FIGS. 13a and 13b , when a decoding failure occurs for two blocks within a second P frame, screen tearing may occur in a portion of a frame (i.e., frame 3) that is the result of decoding the second P frame (e.g., a portion associated with the two blocks of the second P frame where the decoding failure occurred). For a description of blocks that directly or indirectly reference blocks where a decoding failure occurred, reference may be made to the descriptions of FIGS. 6b and 6c . Therefore, duplicate descriptions are omitted.

[0239] In one embodiment, a block that directly or indirectly references a block in which a decoding failure occurred may also experience a decoding failure. This is because the receiving device uses the block that was not decoded normally to decode the block that directly or indirectly references it. For a description of the occurrence of a decoding failure in a block that directly or indirectly references a block in which a decoding failure occurred, please refer to the descriptions in FIGS. 6B and 6C . Therefore, any duplicate description will be omitted.

[0240] In this way, when the transmitting device refers to a block in which a decoding failure occurred in the receiving device when encoding subsequent frames (e.g., P frames or B frames), the receiving device cannot successfully decode at least one block in the subsequent frame(s) that refers to the block in which the decoding failure occurred. For example, as illustrated in FIGS. 13a and 13b, screen tearing may occur not only in the screen of the second P frame that includes blocks in which a decoding failure occurred due to an error in the data packet, but also in the subsequent third P frame that includes at least one block that refers to blocks of the second P frame.

[0241] However, in the case of the embodiments of FIGS. 13a and 13b, the transmitting device (110) may receive damaged block-related information for frame 3 from the receiving device (120), for example, before encoding frame 5. In this case, the transmitting device (110) may identify the first damaged block(s) in which a decoding failure occurred in frame 3 based on the damaged block-related information for frame 3. The transmitting device (110) may process the first damaged block(s) in frame 3 so that they are not referenced in blocks of other frames that are encoded after the corresponding damaged block-related information is received. For example, as illustrated in FIGS. 13a and 13b, the transmitting device (110) may generate the corresponding blocks in frame 5 as intra blocks so that the blocks in frame 5 that are encoded after the damaged block-related information for frame 3 is received do not directly or indirectly reference the first damaged block(s). The transmitting device (110) can encode the 5th frame generated as an intra block to generate a P frame (or B frame). In this way, when using the damaged block related information received from the receiving device, unlike the embodiments of FIGS. 6b and 6c that do not use the damaged block related information, no serial image breakage occurs. As illustrated in FIGS. 13a and 13b, the 5th frame encoded after the damaged block related information is received does not experience screen breakage.

[0242] According to one embodiment, in an image processing system, a receiving device may include a communication unit including a communication circuit; a memory storing a program including at least one command; and at least one processor connected to the communication unit and the memory, the processor including a processing circuit, and executing at least one command of the program stored in the memory.

[0243] According to one embodiment, at least one processor is configured to individually and / or collectively: control the receiving device to receive, from a transmitting device, at least one data packet including data in a first frame, perform decoding on the first frame based on the at least one data packet, identify whether a decoding failure occurs on the first frame, and, based on identifying that a decoding failure occurs on the first frame, transmit first information related to at least one first block in which the decoding failure occurred to the transmitting device, wherein the first information may include block ID information of the at least one first block in which the decoding failure occurred and frame ID information of the first frame to which the at least one first block belongs.

[0244] According to one embodiment, at least one processor is configured to individually and / or collectively control the receiving device to identify that the decoding failure for the first frame has occurred based on identifying that at least one data packet among a plurality of data packets including data in the first frame transmitted from the transmitting device is not obtained at a time when processing of the at least one data packet is required, wherein the at least one data packet may include data of the at least one first block.

[0245] In one embodiment, it can be identified that the one or more data packets are dropped or received after a time when processing for the one or more data packets is required due to jitter, and thus are not obtained at a time when processing for the one or more data packets is required.

[0246] According to one embodiment, at least one processor may be configured to individually and / or collectively control the receiving device to initiate decoding of the first frame based on receiving a portion of a plurality of data packets comprising data in the first frame transmitted by the transmitting device.

[0247] According to one embodiment, the first frame may include a frame referenced by at least one frame subsequent to the first frame.

[0248] According to one embodiment, the first frame may include an intra frame (I frame), a predictive frame (P frame), or a bidirectional frame (B frame) of a first group of pictures (GOP), and the at least one frame may be a P frame or a B frame of the first GOP that references the first frame.

[0249] According to one embodiment, the first GOP is a long GOP or an infinite GOP, and the first frame and the at least one frame can be encoded using the same bit rate.

[0250] According to one embodiment, a transmitting device in an image processing system may include a communication unit including a communication circuit; a memory storing a program including at least one command; and at least one processor connected to the communication unit and the memory, the processor including a processing circuit, and executing at least one command of the program stored in the memory.

[0251] According to one embodiment, the at least one processor may be individually and / or collectively configured to: transmit, to a receiving device, at least one data packet including data of a first frame; receive, from the receiving device, first information related to at least one block in the first frame in which a decoding failure has occurred, the first information including block ID information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs; identify, based on the first information, the at least one first block in the first frame; process the at least one first block so that it is not referenced by blocks in a second frame that is encoded based on the reception of the first information; and perform encoding on the second frame.

[0252] According to one embodiment, to process the at least one first block such that it is not referenced by blocks in a second frame that is encoded based on the first information being received, the at least one processor may be individually and / or collectively configured to: identify at least one second block, within a second frame that is encoded after the first information is received, that is set to reference the at least one first block, and process the identified at least one second block as an intra block that includes an independent block that does not depend on other blocks.

[0253] According to one embodiment, to process the at least one first block such that it is not referenced by blocks in a second frame that is encoded based on the first information being received, the at least one processor may be individually and / or collectively configured to: identify at least one third block that references the first block in at least one third frame that is encoded before the first information is received, identify at least one second block that is set to reference at least one of the at least one first block or the at least one third block in a second frame that is encoded after the first information is received, and process the identified at least one second block as an intra block that is an independent block that does not depend on other blocks.

[0254] According to one embodiment, to process the at least one first block so that it is not referenced by blocks in a second frame that are encoded after the first information is received, the at least one processor may be configured to individually and / or collectively mask the at least one first block in the first frame stored in the buffer.

[0255] According to one embodiment, the first frame may include a frame referenced by at least one frame subsequent to the first frame.

[0256] According to one embodiment, the first frame may be an I frame, a P frame, or a B frame of the first GOP, the second frame may be a P frame or a B frame of the first GOP that references the first frame or the third frame, and the third frame may include a P frame or a B frame of the first GOP that references the first frame.

[0257] In one embodiment, the first GOP is a long GOP or an infinite GOP, and the first frame, the second frame, and the third frame can be encoded using the same bit rate.

[0258] According to one embodiment, a method of operating a receiving device in an image processing system includes: receiving, from a transmitting device, at least one data packet including data in a first frame; performing decoding on the first frame based on the at least one data packet; identifying whether a decoding failure has occurred on the first frame; and transmitting, based on identifying that a decoding failure has occurred on the first frame, first information related to at least one first block in which the decoding failure has occurred to the transmitting device, wherein the first information may include block ID information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs.

[0259] According to one embodiment, the operation of identifying whether a decoding failure occurs for the first frame includes: identifying that the decoding failure occurs for the first frame based on identifying that at least one data packet among a plurality of data packets including data in the first frame transmitted from the transmitting device is not acquired at a time when processing of the at least one data packet is required, wherein the at least one data packet may include data of the at least one first block.

[0260] According to one embodiment, the first frame may include an I frame, a P frame (predictive frame), or a B frame of the first GOP, and at least one frame following the first frame may include a P frame or a B frame of the first GOP that refers to the first frame.

[0261] According to one embodiment, a method of operating a receiving device in an image processing system may include: transmitting, to the receiving device, at least one data packet including data of a first frame; receiving, from the receiving device, first information related to at least one block in the first frame in which a decoding failure has occurred, the first information including block ID information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs; identifying, based on the first information, the at least one first block in the first frame; processing, based on the first information, the at least one first block so as not to be referenced by blocks in a second frame that is encoded based on the reception of the first information; and performing encoding on the second frame.

[0262] According to one embodiment, the operation of processing the at least one first block so that it is not referenced by blocks in a second frame that is encoded after the first information is received may include: identifying at least one second block, within the second frame that is encoded after the first information is received, that is set to reference the at least one first block; and processing the identified at least one second block as an intra block that includes an independent block that does not depend on other blocks.

[0263] According to one embodiment, the operation of processing the at least one first block so that it is not referenced by blocks in a second frame that is encoded based on the reception of the first information may include: identifying at least one third block that references the first block in at least one third frame that is encoded before the reception of the first information; identifying at least one second block that is set to reference at least one of the at least one first block or the at least one third block in a second frame that is encoded after the reception of the first information; and processing the identified at least one second block as an intra block that is an independent block that does not depend on other blocks.

[0264] In the various embodiments of the present disclosure described above, components included in the present disclosure are expressed singularly or plurally, depending on the specific embodiment. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Even components expressed in plural may be composed of singular elements, or even components expressed in singular may be composed of plural elements.

[0265] While the present disclosure has been illustrated and described with reference to various exemplary embodiments, it is to be understood that the various exemplary embodiments are illustrative and not limiting. Those skilled in the art will further appreciate that various modifications are possible within the scope of the present disclosure, including the appended claims and their equivalents. It is also to be understood that any embodiment(s) described in the present disclosure may be used in conjunction with other embodiments described in the present disclosure.

Claims

1. In a receiving device in an image processing system, A communication unit including a communication circuit; A memory storing a program containing at least one instruction; and At least one processor connected to the communication unit and the memory, including a processing circuit and configured to execute at least one instruction of a program stored in the memory, wherein the at least one processor: Receive at least one data packet containing data of a first frame from a transmitting device, Based on at least one data packet, decoding is performed for the first frame, Identify whether a decoding failure occurs for the first frame above, Based on identifying that a decoding failure has occurred for the first frame, the receiving device is configured to control the transmitting device to transmit first information related to at least one first block in which the decoding failure has occurred, A receiving device, wherein the first information includes block ID (identifier) ​​information of at least one first block in which the decoding failure occurred and frame ID information of the first frame to which the at least one first block belongs.

2. In paragraph 1, at least one processor individually and / or collectively: is configured to identify that the decoding failure for the first frame has occurred based on identifying that at least one data packet among a plurality of data packets including data in the first frame transmitted from the transmitting device is not obtained at a time when processing for the at least one data packet is required; A receiving device, wherein said one or more data packets include data of said at least one first block.

3. In paragraph 2, A receiving device, wherein it is identified that the one or more data packets are dropped or received after a time when processing of the one or more data packets is required due to jitter, and thus are not acquired at a time when processing of the one or more data packets is required.

4. In paragraph 1, at least one processor, individually and / or collectively: A receiving device configured to initiate decoding of the first frame after some of the plurality of data packets containing data in the first frame transmitted from the transmitting device are received.

5. A receiving device in the first paragraph, wherein the first frame includes a frame referenced by at least one frame following the first frame.

6. A receiving device in the fifth paragraph, wherein the first frame includes an I frame (intra frame), a P frame (predictive frame), or a B frame (bidirectional frame) of a first GOP (group of pictures), and wherein at least one frame includes a P frame or a B frame of the first GOP referencing the first frame.

7. A receiving device in the 6th paragraph, wherein the first GOP is a long GOP or an infinite GOP, and the first frame and the at least one frame are encoded using the same bit rate.

8. In a transmitting device in an image processing system, A communication unit including a communication circuit; A memory storing a program containing at least one instruction; and At least one processor coupled to the communication unit and the memory, including a processing circuit and configured to execute at least one instruction of a program stored in the memory, wherein the at least one processor individually and / or collectively: To a receiving device, transmit at least one data packet containing data of a first frame, Receive first information related to at least one block in which a decoding failure has occurred within the first frame from the receiving device, wherein the first information includes block ID (identifier) ​​information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs. Based on the first information, identifying at least one first block within the first frame, Processing said at least one first block so that it is not referenced by blocks in a second frame that are encoded based on the reception of said first information, A transmitting device, which is set to control the transmitting device to perform encoding for the second frame.

9. In the 8th paragraph, to process the at least one first block so that it is not referenced by blocks in the second frame that are encoded based on the reception of the first information, the at least one processor individually and / or collectively: Identifying at least one second block set to reference at least one first block within a second frame encoded after the first information is received; A transmitter configured to process at least one of the identified second blocks as an intra block comprising an independent block that does not depend on other blocks.

10. In the 8th paragraph, to process the at least one first block so that it is not referenced by blocks in the second frame that are encoded based on the reception of the first information, the at least one processor individually and / or collectively: Identifying at least one third block referencing the first block in at least one third frame encoded before the first information is received; Identifying at least one second block set to refer to at least one of the at least one first block or the at least one third block within a second frame encoded after the first information is received; A transmitter configured to process at least one of the identified second blocks as an intra block comprising an independent block that does not depend on other blocks.

11. In the 8th paragraph, the transmitting device is configured to individually and / or collectively mask the at least one first block within the first frame stored in the buffer so that the at least one first block is not referenced by blocks within the second frame that are encoded based on the reception of the first information.

12. A transmitting device in the 10th paragraph, wherein the first frame is a frame referenced by at least one frame following the first frame.

13. In the 12th paragraph, the first frame includes an I frame, a P frame or a B frame of the first GOP, the second frame includes a P frame or a B frame of the first GOP referencing the first frame or the third frame, and the third frame includes a P frame or a B frame of the first GOP referencing the first frame. A transmitting device, wherein the first GOP is a long GOP or an infinite GOP, and the first frame, the second frame, and the third frame are encoded using the same bit rate.

14. A method for operating a receiving device in an image processing system, An operation of receiving, from a transmitting device, at least one data packet containing data of a first frame; An operation of performing decoding for the first frame based on the at least one data packet; An operation for identifying whether a decoding failure occurs for the first frame; and An operation of transmitting, to the transmitting device, first information related to at least one first block in which the decoding failure occurred, based on identifying that a decoding failure has occurred for the first frame, A method, wherein the first information includes block ID information of the at least one first block in which the decoding failure occurred and frame ID information of the first frame to which the at least one first block belongs.

15. A method for operating a transmitting device in an image processing system, An operation of transmitting, to a receiving device, at least one data packet containing data of a first frame; An operation of receiving, from the receiving device, first information related to at least one block in which a decoding failure has occurred within the first frame, wherein the first information includes block ID information of the at least one first block in which the decoding failure has occurred and frame ID information of the first frame to which the at least one first block belongs; An operation of identifying at least one first block within the first frame based on the first information; An operation for processing at least one first block so that it is not referenced by blocks in a second frame that are encoded based on the reception of the first information; and A method comprising an operation of performing encoding on the second frame.

Images