Method and apparatus for image processing using artificial intelligence technology

The method addresses the performance challenges of standardized codec technologies by integrating pre-processing and post-processing techniques for frame skip, downscaling, and super-resolution, ensuring high-quality video transmission and storage with high compression efficiency.

WO2024237741A1PCT designated stage expired Publication Date: 2024-11-21AIONFLOW CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/006804
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-05-17
Filing Date
2024-05-20
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Current standardized codec technologies, such as H.264/AVC and H.265/HEVC, face challenges in maintaining performance for high-definition video streaming due to increasing data traffic, necessitating a new image processing method for stable transmission, storage, and reproduction of high-quality videos with high compression efficiency at low costs.

Method used

The method employs pre-processing techniques like frame skip, downscaling, and latent vector technology for compression, and post-processing methods like frame interpolation and super-resolution to enhance video quality, ensuring compatibility with standardized encoding/decoding technologies while achieving high compression efficiency.

Benefits of technology

This approach enables stable transmission and storage of high-quality videos with the same or higher quality than the original, supporting high compression efficiency and accurate image processing using artificial intelligence models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024006804_21112024_PF_FP_ABST
    Figure KR2024006804_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure provides an image processing method. The image processing method of the present disclosure may comprise the steps of: acquiring image data including a plurality of image frames; performing pre-processing on the image data; encoding the pre-processed image data in order to generate encoded image data; and transmitting the encoded image data and information related to the pre-processing.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for image processing using artificial intelligence technology

[0001] The present disclosure relates to a method and device for image processing using artificial intelligence technology.

[0002] Standardized codec technologies (e.g., H.264 / AVC (advanced video coding) and H.265 / HEVC (high-efficiency video coding)) are primarily used for video data compression and decompression. While these standardized codec technologies offer high efficiency and compatibility, they are increasingly challenging to achieve as compression technologies in today's rapidly growing data traffic driven by the increasing demand for high-definition streaming services.

[0003] This has led to an increasing demand for new image processing technologies that can reliably transmit high-definition images with high compression efficiency, store them at low cost, and reproduce them in high quality.

[0004] The present disclosure provides a method for stably transmitting and storing an original video (e.g., a high-quality video) with high compression efficiency, and for reproducing the original video with substantially the same quality (e.g., high quality) or higher quality. The present disclosure provides preprocessing and postprocessing methods for providing high-efficiency compression and high-quality video. The preprocessing and postprocessing methods of the present disclosure can have high compatibility with encoding / decoding technologies using standardized codec technologies. The present disclosure provides a preprocessing method using a frame skipping technology and a postprocessing method using a frame interpolation technology and / or a quality enhancement technology. Through this, it is possible to provide a quality level substantially the same as or higher than that of the original video while supporting high compression efficiency.

[0005] The present disclosure provides a preprocessing method using a latent vector technique that expresses the downscaling technology of an image and the resulting loss as a latent vector, and a postprocessing method using a super resolution (SR) technique for resolution enhancement. Through this, it is possible to provide a quality substantially the same as or higher than that of the original image while supporting high compression efficiency. The present disclosure provides a preprocessing method that uses a frame selection technique, a downscaling technique, and a latent vector technique together, and a postprocessing method that uses a frame interpolation technique, a quality enhancement technique, and a super resolution technique together. Through this, it is possible to provide a quality substantially the same as or higher than that of the original image while supporting high compression efficiency.

[0006] The present disclosure provides a method for stably transmitting and storing an original video (e.g., a high-quality video) with high compression efficiency, and reproducing the original video with substantially the same quality (e.g., high-quality) or higher quality. The present disclosure provides a post-processing method using an enhancement technique for restoring a decoded frame using a reference frame of a corresponding frame group (e.g., an I-frame) and a reference frame of a subsequent frame group (e.g., an I-frame). Through this, distortion between frame groups (e.g., a group of pictures (GoP)) can be restored. In the present disclosure, the reference frame may be referred to as a key frame.

[0007] The preprocessing and postprocessing methods of the present disclosure can be implemented using an artificial intelligence model. This can support high-speed and high-accuracy image processing.

[0008] FIG. 1a schematically illustrates an image processing system according to one embodiment of the present disclosure.

[0009] FIG. 1b is a schematic block diagram of an image transmitting device and an image receiving device according to one embodiment of the present disclosure.

[0010] FIG. 1C is a schematic block diagram of an image transmitting device and an image receiving device according to one embodiment of the present disclosure.

[0011] FIG. 2A is a schematic block diagram of an encoding unit according to one embodiment of the present disclosure.

[0012] FIG. 2b is a schematic block diagram of a decryption unit according to one embodiment of the present disclosure.

[0013] FIG. 3 illustrates a frame skipping processing module of a preprocessing unit according to one embodiment of the present disclosure.

[0014] FIG. 4 schematically illustrates the operation of a frame skipping processing module according to one embodiment of the present disclosure.

[0015] FIG. 5 is a flowchart of a frame skipping processing method according to one embodiment of the present disclosure.

[0016] FIG. 6 illustrates a frame skipping application decision operation of a frame skipping processing method according to one embodiment of the present disclosure.

[0017] FIG. 7 illustrates an example of a model for determining frame skipping application according to one embodiment of the present disclosure.

[0018] FIG. 8a illustrates an example of a rate distortion curve and a target rate distortion point on a rate distortion plane according to one embodiment of the present disclosure.

[0019] FIG. 8b illustrates an example of a distance between a rate distortion curve and a target rate distortion point according to one embodiment of the present disclosure.

[0020] FIG. 9 illustrates an example of a method for obtaining a distance between a rate distortion curve and a target rate distortion point, according to one embodiment of the present disclosure.

[0021] FIG. 10 illustrates a frame interpolation processing module and a quality enhancement processing module of a post-processing unit according to one embodiment of the present disclosure.

[0022] FIG. 11 schematically illustrates the operation of a frame interpolation processing module and a quality enhancement processing module according to one embodiment of the present disclosure.

[0023] FIG. 12 is a flowchart of a frame interpolation processing method according to one embodiment of the present disclosure.

[0024] FIG. 13 illustrates an example of a model for frame interpolation processing according to one embodiment of the present disclosure.

[0025] FIG. 14 illustrates an example of a model for quality improvement processing according to one embodiment of the present disclosure.

[0026] FIG. 15 is a flowchart of an image processing method of an image transmission device according to one embodiment of the present disclosure.

[0027] FIG. 16 is a flowchart of an image processing method of an image receiving device according to one embodiment of the present disclosure.

[0028] FIG. 17 illustrates a down sampling processing module and a latency vector generation processing module of a preprocessing unit according to one embodiment of the present disclosure.

[0029] FIG. 18 schematically illustrates the operation of a downsampling processing module and a latency vector generation processing module according to one embodiment of the present disclosure.

[0030] FIG. 19 is a flowchart of a latent vector generation method according to one embodiment of the present disclosure.

[0031] FIG. 20 illustrates an example of a model for latent vector generation processing according to one embodiment of the present disclosure.

[0032] FIG. 21 illustrates a super resolution processing module of a post-processing unit according to one embodiment of the present disclosure.

[0033] FIG. 22a illustrates the operation of a super resolution processing module according to one embodiment of the present disclosure.

[0034] FIG. 22b illustrates an example of a model for super-resolution processing according to one embodiment of the present disclosure.

[0035] FIG. 23 illustrates an example of an image processing procedure according to one embodiment of the present disclosure.

[0036] FIG. 24 is a flowchart of an image processing method of an image transmission device according to one embodiment of the present disclosure.

[0037] FIG. 25 is a flowchart of an image processing method of an image receiving device according to one embodiment of the present disclosure.

[0038] Fig. 26 shows an exemplary configuration of a preprocessing unit according to one embodiment of the present disclosure.

[0039] Fig. 27 shows an exemplary configuration of a post-processing unit according to one embodiment of the present disclosure.

[0040] FIG. 28 illustrates an example of an image processing procedure according to one embodiment of the present disclosure.

[0041] FIG. 29 illustrates an original image and an encoded image according to one embodiment of the present disclosure.

[0042] FIG. 30 illustrates an image showing a first portion of an original image in chronological order and an image showing a second portion of an encoded image corresponding to the first portion of the original image in chronological order, according to one embodiment of the present disclosure.

[0043] FIG. 31 illustrates a GoP enhancement processing module of a post-processing unit according to one embodiment of the present disclosure.

[0044] FIG. 32 illustrates the operation of a GoP enhancement processing module according to one embodiment of the present disclosure.

[0045] FIG. 33 illustrates an example of a procedure for alignment processing of a GoP enhancement processing module according to one embodiment of the present disclosure.

[0046] FIG. 34 is a flowchart of an image processing method of an image receiving device according to one embodiment of the present disclosure.

[0047] FIG. 35 is a flowchart of an alignment processing operation of an image receiving device according to one embodiment of the present disclosure.

[0048] FIG. 36 is a diagram illustrating a format for storing metadata of a codec according to one embodiment of the present disclosure.

[0049] FIGS. 37 and 38 are diagrams illustrating a method of transmitting media data and metadata according to one embodiment of the present disclosure.

[0050] FIGS. 39a and 39b are diagrams illustrating a super resolution procedure according to one embodiment of the present disclosure.

[0051] FIGS. 40A and 40B are diagrams illustrating a frame skipping procedure according to one embodiment of the present disclosure.

[0052] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0053] At this time, it will be understood that each block of the processing flow diagrams and combinations of the flow diagrams can be performed by computer program instructions.

[0054] Additionally, each block may represent a module, segment, or portion of code that contains one or more executable instructions for performing a specific logical function(s). It should also be noted that in some alternative implementation examples, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on their respective functions.

[0055] Here, the term '~ part' used in this embodiment means software or hardware components such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and the '~ part' performs certain roles. However, the '~ part' is not limited to software or hardware. The '~ part' may be configured to be on an addressable storage medium or may be configured to reproduce one or more packet processing devices. Therefore, as an example, the '~ part' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and '~ parts' may be combined into a smaller number of components and '~ parts' or further separated into additional components and '~ parts'. In addition, the components and '~parts' may be implemented to play one or more central processing units (CPUs) within the device or secure multimedia card. In addition, in the embodiment, the '~parts' may include one or more packet processing devices.

[0056] FIG. 1a schematically illustrates an image processing system according to one embodiment of the present disclosure.

[0057] Referring to FIG. 1A, the image processing system (100) may include an image transmitting device (110) and an image receiving device (120). In the present disclosure, the image transmitting device (110) may be referred to as an image providing device, and the image receiving device (120) may be referred to as an image reproducing device.

[0058] According to one embodiment, the image transmitting device (110) transmits an image signal (or image data) including a video and / or an image to the image receiving device (120) via a network (130), and the image receiving device (120) can receive the image signal from the image transmitting device (110).

[0059] According to one embodiment, the video transmission device (110) can encode a video signal to generate a compressed video signal. The video transmission device (110) can efficiently store, transmit, and manage the video signal by encoding the video signal within a preset compression ratio range, for example. The video signal may include, but is not limited to, streaming video, camera video, video video, decompressed video, video conference video, and / or game video.

[0060] According to one embodiment, the video transmission device (110) may include various video source devices such as a TV, a personal computer (PC), a smartphone, a tablet, a set-top box, a game console, and a server. According to one embodiment, the video reception device (120) may include various video playback devices such as a TV, a smartphone, a tablet, and a PC. It will be apparent to those skilled in the art that the video transmission device (110) and the video reception device (120) are not limited to a specific type of device.

[0061] According to one embodiment, the image transmitting device (110) and the image receiving device (120) can transmit and receive image signals through a network (130). The network (130) may include, for example, a local area network such as Wi-Fi, a cellular network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN), and may communicate based on an Internet protocol (IP) communication protocol. The cellular network may include a global system for mobile communications (GSM), an enhanced data GSM environment (EDGE), a code division multiple access (CDMA), a time division multiplexing access (TDMA), a long term evolution (LTE), an LTE advance (LTE-A), a 5G new radio (NR), and a communication network after 5G (e.g., a 6G network or later). The network (130) may include connections of network elements such as hubs, bridges, routers, switches, and gateways. The network (130) may include one or more connected networks, such as a multi-network environment, including a public network such as the Internet and a private network such as a corporate private network. Access to the network (130) may be provided by one or more wired Alternatively, it may be provided via wireless access networks. The network (130) may support an Internet of Things (IoT) network that exchanges and processes information between distributed components such as objects.

[0062] FIG. 1b is a schematic block diagram of an image transmitting device and an image receiving device according to one embodiment of the present disclosure.

[0063] Referring to FIG. 1B, the image transmission device (110) may include an image input unit (111), a preprocessing unit (112), an encoding unit (113), and an image output unit (114). The image transmission device (110) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. For example, the image transmission device (110) may further include a memory, at least one processor, and / or a communication unit.

[0064] According to one embodiment, the memory may store data, such as a program or configuration information, including one or more instructions. The memory may be comprised of, for example, volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. The memory may provide stored data upon request from the processor.

[0065] In one embodiment, the communication unit may provide an interface for communication with other systems or devices. The communication unit may include a network interface card or a wireless transmitter / receiver unit that enables communication over a network (130). The communication unit may perform signal processing for accessing a wireless network. The wireless network may include, for example, at least one of a short-range communication network or a cellular network (e.g., LTE, 5G NR).

[0066] According to one embodiment, at least one processor is electrically connected to the communication unit and the memory, and can execute calculations or data processing related to control and / or communication of at least one other component of the image transmission device (110) using a program stored in the memory. The processor can execute at least one instruction corresponding to the image input unit (111), the preprocessing unit (112), the encoding unit (113), and the image output unit (114). The processor may include, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0067] According to one embodiment, the image input unit (111) can receive an image signal. The image signal may be received from outside the image transmission device (110) or may be generated by the image transmission device (110). The image input unit (111) can receive an image signal from outside in a wired or wireless manner through a communication unit.

[0068] According to one embodiment, the preprocessing unit (112) may preprocess the image signal input by the image input unit (110) prior to encoding processing by the encoding unit (113). For example, the preprocessing unit (112) may perform frame skipping processing on the input image signal. For example, the preprocessing unit (112) may perform downsampling processing on the input image signal. For example, the preprocessing unit (112) may perform latency vector processing that expresses loss due to downsampling processing as a latency vector. For example, the preprocessing unit (112) may perform frame skipping processing, downsampling processing, and latency vector processing.

[0069] According to one embodiment, the preprocessing unit (112) may be implemented by a pre-learned model (e.g., an artificial intelligence (AI) model).

[0070] According to one embodiment, the artificial intelligence model can be generated through machine learning. Such learning can be performed, for example, in the electronic device (e.g., the image transmitting device (110) or the image receiving device (120)) in which the artificial intelligence model is used, or can be performed through a separate electronic device (e.g., a server). The learning algorithm can include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model can include multiple artificial neural network layers. The artificial neural network can be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. An artificial intelligence model may additionally or alternatively include a software structure in addition to a hardware structure.

[0071] Meanwhile, depending on the embodiment, all or part of the operations of the preprocessing unit (112) may be omitted. For example, all or part of the frame skipping processing, downsampling processing, and latency vector processing of the preprocessing unit (112) may be omitted. In the case where the entire operations of the preprocessing unit (112) are omitted, the input image of the image transmission device (110) may be transmitted to the image reception device (120) after only undergoing encoding processing by the encoding unit (113).

[0072] According to one embodiment, the encoding unit (113) can encode an image signal input by the image input unit (110) or an image signal preprocessed by the preprocessing unit (112). The encoding unit (113) can perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. An example of the encoding unit (113) is described below with reference to FIG. 2A.

[0073] According to one embodiment, the encoding unit (113) may encode a video signal using a preset encoding method and encoding parameters associated with the encoding method. The encoding method may be, for example, a method following a standardized codec technology (e.g., H.264 / AVC standard, H.265 / HEVC standard) or an artificial intelligence-based codec technology (AI codec technology), but is not limited thereto. The encoding parameter may include at least a parameter used (or set) to encode (or compress) a video signal according to, for example, a standardized codec technology (e.g., H.264 / AVC standard, H.265 / HEVC standard). At least one parameter may include, for example, a parameter related to a compression ratio (or compression quality) and / or a bit rate (e.g., a quantization parameter (QP)). Generally, a lower QP value may provide higher video quality by applying less quantization, but may require a higher bit rate. In the present disclosure, an encoding method may be referred to as a compression method, and an encoding parameter may be referred to as a compression parameter.

[0074] According to one embodiment, the encoding unit (113) can provide an encoded image signal (or encoded data) to the image output unit (114) in the form of a bitstream.

[0075] According to one embodiment, the preprocessing unit (112) and the encoding unit (113) may be a single integrated configuration. For example, the preprocessing unit (112) may be included in the encoding unit (113). For example, when the encoding unit (113) uses, for example, AI codec technology, the preprocessing unit (112) may be a configuration included in the encoding unit (113). When the preprocessing unit (112) is included in the encoding unit (113), the operation of the preprocessing unit (112) may be performed before the encoding operation (encoding operation) of the encoding unit (113), but is not limited thereto. For example, depending on the embodiment, the operation of the preprocessing unit (112) may be performed together with or after the encoding operation of the encoding unit (113). For example, the operations of the preprocessing unit (112) and the encoding unit (113) may be performed together through at least one AI model.

[0076] According to one embodiment, the image output unit (114) can transmit an encoded image signal to the image receiving device (120) through the communication unit.

[0077] According to one embodiment, the image output unit (114) may transmit information related to preprocessing (e.g., frame skip information) to the image receiving device (120) through the communication unit together with the encoded image signal. The information related to preprocessing (or, frame skip information) may include, but is not limited to, information indicating whether frame skipping has been applied to a corresponding frame (information related to frame skipping) and / or information on the number of frames (e.g., information on a frame rate (e.g., FPS (frame per second))). In the present disclosure, a frame may be referred to as an image frame.

[0078] This related information can be used for image processing in the image receiving device (120). For example, information related to frame skipping can be used for the image receiving device (120) to determine whether to apply frame interpolation to a corresponding frame. For example, information about the frame rate can be used for the image receiving device (120) to determine whether frame skipping has been applied in the image transmitting device (110). For example, if the information about the frame rate is lower than the frame rate for the input image (e.g., 30 FPS), the image receiving device (120) can determine that frame skipping has been applied in the image transmitting device (110). For example, if the information about the frame rate is the same as the frame rate for the input image (e.g., 30 FPS), the image receiving device (120) can determine that frame skipping has not been applied in the image transmitting device (110).

[0079] According to one embodiment, information related to preprocessing may be added to a bitstream including encoded data. For example, information related to preprocessing may be included in an optional area within the bitstream. For example, information related to preprocessing may be included in a description area within the bitstream, but is not limited thereto. The description area may be, for example, an area for describing information and a structure of the bitstream. For example, the description area may include codec information indicating which codec the bitstream was compressed with, media information indicating what type of media data the bitstream includes, encoding setting information including encoding parameters of the codec, and / or timestamp or time information of each frame associated with the bitstream.

[0080] The image receiving device (120) may include an image input unit (121), a decoding unit (122), a post-processing unit (123), and an image output unit (124). The image output unit (124) may include, for example, a display unit, and the display unit may be configured as a separate device or an external component. The image receiving device (120) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. For example, the image receiving device (120) may further include a memory, at least one processor, and a communication unit.

[0081] According to one embodiment, the memory may store data, such as a program or configuration information, including one or more instructions. The memory may be comprised of, for example, volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. The memory may provide stored data upon request from the processor.

[0082] In one embodiment, the communication unit may provide an interface for communication with other systems or devices. The communication unit may include a network interface card or a wireless transmitter / receiver unit that enables communication over a network (130). The communication unit may perform signal processing for accessing a wireless network. The wireless network may include, for example, at least one of a short-range communication network or a cellular network (e.g., LTE, 5G NR).

[0083] According to one embodiment, at least one processor is electrically connected to the communication unit and the memory, and can execute calculations or data processing related to control and / or communication of at least one other component of the image receiving device (120) using a program stored in the memory. The processor can execute at least one command corresponding to the image input unit (121), the decoding unit (122), the post-processing unit (123), and the image output unit (124). The processor may include, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0084] According to one embodiment, the image input unit (121) can receive an image signal. The image input unit (121) can receive the image signal from the image transmission device (110) through the communication unit. The image input unit (121) can receive the image signal from the image transmission device (110) through the communication unit in a wired or wireless manner.

[0085] According to one embodiment, the image input unit (121) can extract (or acquire) an encoded image signal and / or information related to preprocessing from a bitstream received from the image transmission device (110). The image input unit (121) can transmit the acquired encoded image signal and / or information related to preprocessing to the decoding unit (122).

[0086] According to one embodiment, the decoding unit (122) can decode the image signal input by the image input unit (110). For example, the decoding unit (122) can decode the image signal by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding unit (112). An example of the decoding unit (122) is described below with reference to FIG. 2B.

[0087] According to one embodiment, the decoding unit (122) can decode the video signal using a preset decoding method and decoding parameters associated with the decoding method. According to one embodiment, the decoding method may correspond to the encoding method of the encoding unit (113), and the decoding parameters may correspond to encoding parameters associated with the encoding method of the encoding unit (113). For example, the same standard codec technology (e.g., H.264 / AVC, H.265 / HEVC standard) or AI codec technology may be used for encoding and decoding, and a parameter (e.g., a dequantization parameter) corresponding to a parameter (e.g., a QP) used for encoding may be used for decoding. According to one embodiment, the value of a parameter (e.g., a dequantization parameter) for decoding may be set depending on the value of a corresponding parameter (e.g., a QP) for encoding. In the present disclosure, the decryption method may be referred to as a reconstruction method, and the decryption parameters may be referred to as reconstruction parameters.

[0088] According to one embodiment, the post-processing unit (123) can post-process the image signal decrypted by the decryption unit (122).

[0089] For example, the post-processing unit (123) can perform frame interpolation processing on the decoded video signal. Through this, skipped frames can be regenerated. For example, the post-processing unit (123) can perform quality enhancement processing on the decoded video signal. Through this, the quality of degraded video data can be supplemented (or improved). For example, frame interpolation processing can be performed on the decoded video signal, and quality enhancement processing can be performed on the frame-interpolated video signal. Through this, skipped frames can be regenerated, the quality of degraded video data can be improved, and an image having substantially the same quality as the original image can be provided.

[0090] For example, the post-processing unit (123) can perform super-resolution (SR) processing on the decoded image signal. Through this, the resolution of the downscaled image can be improved.

[0091] For example, the post-processing unit (123) may perform GoP (group of picture)-based enhancement (hereinafter, GoP enhancement) processing on the decoded image signal. Through this, a visually smoothed image (temporally smoothed image) may be provided. As an example, GoP enhancement processing may be an example of the quality improvement processing described above.

[0092] For example, the post-processing unit (123) can perform frame interpolation processing, quality enhancement processing, and SR processing on the decoded image signal. Through this, the pre-processed image can be restored for compression efficiency, and the quality of the image can be improved.

[0093] According to one embodiment, the post-processing unit (123) may perform a post-processing operation corresponding to the pre-processing operation performed by the pre-processing unit (112) based on information related to pre-processing. For example, if it is identified that frame skipping processing is performed on an image signal input by the frame pre-processing unit (112) based on information related to frame skipping, the post-processing unit (123) may perform frame interpolation processing on the image signal on which frame skipping processing has been performed. For example, if down-scaling processing and latency vector processing have been performed on an image signal input by the pre-processing unit (112), the post-processing unit (123) may perform SR processing.

[0094] According to one embodiment, the post-processing unit (123) can be implemented using a pre-learned model (e.g., an artificial intelligence model).

[0095] Meanwhile, the video receiving device may not include a post-processing unit that performs the function of the post-processing unit (123) described above. In this case, even if the video receiving device receives information related to preprocessing from the bitstream received from the video transmitting device, it may not be able to perform an operation corresponding to the information. For example, even if the video receiving device obtains information related to frame skipping included in the bitstream, it may not be able to perform a judgment process on whether to apply frame interpolation to the frame and an interpolation process for the frame based on the information. However, the video processing device can confirm that frame skipping has been performed by the video receiving device based on information related to the frame skipping and / or information on the number of frames, and can perform frame interpolation processing at a preset fixed position using a general frame interpolation method.

[0096] According to one embodiment, the post-processing unit (123) and the decryption unit (122) may be a single integrated configuration. For example, the post-processing unit (123) may be included in the decryption unit (122). For example, when the decryption unit (122) uses, for example, AI codec technology, the post-processing unit (123) may be a configuration included in the decryption unit (122). When the post-processing unit (123) is included in the decryption unit (122), the operation of the post-processing unit (123) may be performed after the decryption operation (decoding operation) of the decryption unit (122), but is not limited thereto. For example, depending on the embodiment, the operation of the post-processing unit (123) may be performed together with or after the decryption operation of the decryption unit (122). For example, the operations of the post-processing unit (123) and the decryption unit (122) can be performed together through at least one AI model.

[0097] According to one embodiment, the image output unit (124) can output an image signal decoded by the decoding unit (122) or an image signal post-processed by the post-processing unit (123). For example, the image output unit (124) can render the decoded image signal or the post-processed image signal. The rendered image signal can be displayed, for example, through the display unit.

[0098] FIG. 1c is a schematic block diagram of an image transmitting device and an image receiving device according to one embodiment of the present disclosure.

[0099] Referring to FIG. 1c, the video transmission device (110) may include a video input unit (111a), a preprocessing unit (112a), a codec processing unit (113a), a video output unit (114a), and a latency vector processing unit (115a). The video transmission device (110) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. For example, the video transmission device (110) may further include a memory, at least one processor, and / or a communication unit. A description of the memory, at least one processor, and the communication unit of the video transmission device (110) may refer to the description of FIG. 1b. Therefore, a duplicate description will be omitted.

[0100] According to one embodiment, the image input unit (111a) can perform all or part of the operations performed by the image input unit (111) of FIG. 1b, and further, can perform additional operations.

[0101] According to one embodiment, the preprocessing unit (112a) may perform all or part of the operations performed by the preprocessing unit (112) of FIG. 1B, and further may perform additional operations. The preprocessing unit (112a) may preprocess an image signal input by the image input unit (111a) prior to codec processing by the codec processing unit (113a).

[0102] For example, the preprocessing unit (112a) can perform frame skipping processing on the image signal input by the image input unit (111a). In this case, the preprocessing unit (112a) can transmit frame skipping information to the codec processing unit (113a), the image output unit (114a), and / or the latency vector processing unit (115a).

[0103] For example, the preprocessing unit (112a) can perform downsampling processing on the image signal input by the image input unit (111a). For example, the preprocessing unit (112a) can perform frame skipping processing and downsampling processing together on the image signal input by the image input unit (111a).

[0104] In one embodiment, the preprocessing unit (112a) may be implemented by a pre-trained model (e.g., an artificial intelligence (AI) model). For a description of the AI ​​model, please refer to the description of the AI ​​model in FIG. 1b. Therefore, any duplicate description will be omitted.

[0105] According to one embodiment, the codec processing unit (113a) may perform codec processing on an image signal input by the image input unit (110) or an image signal preprocessed by the preprocessing unit (112). The codec processing of the codec processing unit (113a) may include, for example, encoding processing and encoding / decoding processing. The encoding processing of the codec processing unit (113a) may be the same as the encoding processing of the encoding unit (113) of FIG. 1B, and the decoding processing of the codec processing unit (113a) may be the same as the decoding processing of the decoding unit (122) of FIG. 1B. Therefore, a duplicate description will be omitted.

[0106] For example, the codec processing unit (113a) can perform encoding processing on an image preprocessed by the preprocessing unit (112) (e.g., an image subjected to frame skipping processing, an image subjected to downsampling processing) and transmit the encoded image (e.g., a codec bitstream) to the image output unit (114a).

[0107] For example, the codec processing unit (113a) can perform encoding and decoding on an image preprocessed by the preprocessing unit (112) (e.g., an image subjected to frame skipping processing, an image subjected to downsampling processing) and transmit the encoded and decoded image to the latency vector processing unit (115a). In this case, the latency vector processing unit (115a) can generate a latency vector using the encoded and decoded image. In this way, the latency vector processing unit (115a) can generate a latency vector using the encoded and decoded image, which is an image in which compression degradation has occurred.

[0108] According to one embodiment, the latency vector processing unit (115a) may generate a latency vector by using the encoded and decoded image transmitted from the codec processing unit (113a) and / or the frame skip information transmitted from the preprocessing unit (112a). For example, the latency vector processing unit (115a) may generate a latency vector for a down-sampled image by using the image obtained by encoding and decoding the down-sampled image. For example, the latency vector processing unit (115a) may generate a latency vector for a frame-skipping image by using the image obtained by encoding and decoding the frame-skipping image and the frame skip information.

[0109] According to one embodiment, the latency vector processing unit (115a) can transmit the generated latency vector to the image output unit (114a).

[0110] According to one embodiment, the image output unit (114a) can perform all or part of the operations performed by the image output unit (114) of FIG. 1b, and further perform additional operations.

[0111] For example, the image output unit (114a) can transmit a signal including an encoded image (e.g., a codec bitstream) transmitted from the codec processing unit (113a), a latency vector transmitted from the latency vector processing unit (115a), and / or frame skip information transmitted from the preprocessing unit (112a) to the image receiving device (120) via the communication unit. The final bit stream including the encoded image, the latency vector, and / or the frame skip information can be transmitted in a single container. An example of a method for storing and transmitting the encoded image, the latency vector, and / or the frame skip information is described below with reference to FIGS. 36 and 37.

[0112] According to one embodiment, the image receiving device (120) may include an image input unit (121a), a decoding unit (122a), a post-processing unit (123a), and an image output unit (124a). The image receiving device (120) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. For example, the image receiving device (120) may further include a memory, at least one processor, and a communication unit. For a description of the memory, at least one processor, and the communication unit of the image receiving device (120), reference may be made to the description of FIG. 1B. Therefore, any duplicate description will be omitted.

[0113] According to one embodiment, the image input unit (121a) can perform all or part of the operations performed by the image input unit (121) of FIG. 1b, and further perform additional operations.

[0114] For example, the image input unit (121a) can obtain an encoded image (e.g., a codec bitstream), a latency vector, and / or frame skip information from a signal received from the image transmission device (110). The image input unit (121) can transmit the encoded image to the codec processing unit (122a), and transmit the latency vector and / or frame skip information to the post-processing unit (123a).

[0115] According to one embodiment, the codec processing unit (122a) may perform codec processing (e.g., decoding processing) on ​​an encoded image transmitted from the image input unit (121a). The decoding processing of the codec processing unit (122a) may be identical to the decoding processing of the decoding unit (122) of FIG. 1B . Therefore, a duplicate description will be omitted.

[0116] According to one embodiment, the post-processing unit (123a) may perform all or part of the operations performed by the post-processing unit (123) of FIG. 1b, and may further perform additional operations. The post-processing unit (123a) may post-process an image decoded by the codec processing unit (122a).

[0117] For example, the post-processing unit (123a) may perform frame interpolation processing on the decoded image using the latency vector and / or frame skip information. Through this, skipped frames can be regenerated. For example, the post-processing unit (123a) may perform quality enhancement processing on the decoded image signal. Through this, the quality of degraded image data can be supplemented (or improved). For example, frame interpolation processing may be performed on the decoded image signal, and quality enhancement processing may be performed on the frame-interpolated image signal. Through this, skipped frames can be regenerated, the quality of degraded image data can be improved, and an image having substantially the same quality as the original image can be provided.

[0118] For example, the post-processing unit (123a) can perform super-resolution (SR) processing on the decoded image signal using a latent vector. Through this, the resolution of the downscaled image can be improved.

[0119] For example, the post-processing unit (123a) may perform GoP (group of picture)-based enhancement (hereinafter, GoP enhancement) processing on the decoded image signal. Through this, a visually smoothed image (temporally smoothed image) may be provided. As an example, GoP enhancement processing may be an example of the quality improvement processing described above.

[0120] For example, the post-processing unit (123a) can perform frame interpolation processing, quality enhancement processing, and SR processing on the decoded image signal. Through this, the pre-processed image can be restored for compression efficiency, and the quality of the image can be improved.

[0121] According to one embodiment, the post-processing unit (123a) can be implemented using a pre-learned model (e.g., an artificial intelligence model).

[0122] According to one embodiment, the image output unit (124a) can perform all or part of the operations performed by the image output unit (124) of FIG. 1B, and further perform additional operations. The image output unit (124a) can output an image post-processed by the post-processing unit (123a) through a display.

[0123] FIG. 2A is a schematic block diagram of an encoding unit according to one embodiment of the present disclosure.

[0124] Referring to FIG. 2a, the encoding unit (113) may include an image segmentation unit (210a), a transformation unit (220a), a quantization unit (230a), an entropy encoding unit (240a), an inverse quantization unit (250a), an inverse transformation unit (260a), a filtering unit (270a), a buffer (280a), and / or a prediction unit (290a). The image encoding unit (113) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. All or at least some of the plurality of components constituting the encoding unit (113) may be implemented as a single hardware component (e.g., an encoder or a processor) according to an embodiment.

[0125] According to one embodiment, the image segmentation unit (210a) can segment the input image signal (or picture / frame) input to the encoding unit (113) into one or more processing units (PU: Processing Unit).

[0126] For example, a processing unit may be called a coding unit (CU). A coding unit may be obtained by recursively splitting a coding tree unit (CTU) or a largest coding unit (LCU) according to a QT / BT / TT (Quad-Tree / Binary-Tree / Ternary-Tree) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. An encoding procedure according to the present disclosure may be performed based on a final coding unit that is no longer split. The encoding procedure may include procedures such as prediction, transformation, and quantization, which will be described later.

[0127] As another example, the processing unit may be a prediction unit (PU) or a transform unit (TU). The prediction unit and the transform unit may each be split or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit that derives a transform coefficient and / or a unit that derives a residual signal from the transform coefficient. In the present disclosure, a pixel may mean a minimum unit that constitutes a picture (or image). In addition, a sample may be used as a term corresponding to a pixel and may represent a pixel or a value of a pixel.

[0128] According to one embodiment, the prediction unit (290a) can perform a prediction on the current block (the block to be processed) and generate a predicted block including prediction samples for the current block. The prediction unit (290a) can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. The prediction unit (290a) can generate various information regarding the prediction of the current block and transmit the information to the entropy encoding unit (240a). The information regarding the prediction can be encoded by the entropy encoding unit (240a) and output in the form of a bitstream.

[0129] According to one embodiment, the intra prediction unit (291a) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the intra prediction mode and / or intra prediction technique.

[0130] According to one embodiment, the inter prediction unit (292a) may derive a predicted block for the current block based on a reference block specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion prediction information may be predicted on a block, sub-block, or sample basis based on the correlation of motion information between the surrounding blocks and the current block.

[0131] The prediction signal generated through the prediction unit (290a) can be used to generate a restoration signal or a residual signal. The residual signal (residual block) generated by subtracting the prediction signal (predicted block) from the input image signal (original block) can be transmitted to the conversion unit (220a).

[0132] According to one embodiment, the transform unit (220a) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. The transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0133] According to one embodiment, the quantization unit (230a) can quantize transform coefficients and transmit them to the entropy encoding unit (240a). The entropy encoding unit (240a) can encode a quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit (230a) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. According to one embodiment, the entropy encoding unit (240a) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc.

[0134] The quantized signal output from the quantization unit (230a) can be used to generate a prediction signal. For example, the quantized signal can be restored to a residual signal by applying inverse quantization and inverse transformation through the inverse quantization unit (250a) and inverse transformation unit (260a) within the loop. A restored signal can be generated by adding the restored residual signal to the prediction signal output from the intra prediction unit (291a) or the inter prediction unit (292a).

[0135] The filtering unit (270a) applies filtering to the restored signal and outputs it to a playback device or transmits it to a buffer (280a). The filtering method used in the filtering unit (270a) may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtered signal transmitted to the buffer (280a) can be used as a reference picture in the inter prediction unit (292a). In this way, by using the filtered picture as a reference picture in the inter prediction mode, not only the image quality but also the encoding efficiency can be improved.

[0136] The buffer (280a) can store the filtered picture for use as a reference picture in the inter prediction unit (292a).

[0137] FIG. 2b is a schematic block diagram of a decryption unit according to one embodiment of the present disclosure.

[0138] Referring to FIG. 2b, the decoding unit (122) may include an entropy decoding unit (220b), an inverse quantization unit (220b), an inverse transformation unit (230b), a filtering unit (240b), a buffer (250b), and / or a prediction unit (260b). The decoding unit (122) may include additional components in addition to the illustrated components, or may omit at least one of the illustrated components. All or at least some of the plurality of components constituting the image decoding unit (122) may be implemented as a single hardware component (e.g., a decoder or a processor) according to an embodiment.

[0139] The decoding unit (122) can restore the image signal by performing a process corresponding to the process performed in the encoding unit (113) described above with reference to FIG. 2A. For example, the decoding unit (122) can perform decoding using the processing unit applied in the encoding unit (113). Accordingly, the decoding processing unit can be, for example, a coding unit. The coding unit can be a coding tree unit or can be obtained by dividing a maximum coding unit.

[0140] According to one embodiment, the entropy decoding unit (210b) can parse the bitstream to derive information necessary for image restoration (or picture restoration). For example, the entropy decoding unit (210b) can decode information in the bitstream based on a decoding method such as Exponential Golomb, CAVLC, or CABAC, and output information about quantized transform coefficients for residuals necessary for image restoration. Information about prediction among the information decoded by the entropy decoding unit (210b) is provided to the prediction unit (260b), and the residual value on which entropy decoding is performed by the entropy decoding unit (210b), i.e., quantized transform coefficients and related parameter information, can be input to the inverse quantization unit (220b). In addition, information about filtering among the information decoded by the entropy decoding unit (210b) can be provided to the filtering unit (240b).

[0141] According to one embodiment, the inverse quantization unit (220b) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (220b) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding unit (113). The inverse quantization unit (220b) can perform inverse quantization on the quantized transform coefficients using a quantization parameter (QP) (e.g., quantization step size information) and obtain transform coefficients.

[0142] According to one embodiment, the inverse transform unit (230b) can inversely transform the transform coefficients to obtain a residual signal (residual block).

[0143] According to one embodiment, the prediction unit (260b) can perform prediction on a current block and generate a predicted block including prediction samples for the current block. The prediction unit (260b) can determine whether intra prediction or inter prediction is applied to the current block based on information about the prediction output from the entropy decoding unit (210b), and can determine a specific intra / inter prediction mode (prediction technique). The intra prediction unit (261b) can predict the current block by referring to samples in the current picture. The inter prediction unit (262b) can derive a predicted block for the current block based on a reference block specified by a motion vector on a reference picture. The operations of the intra prediction unit (261b) and the inter prediction unit (262b) correspond to the operations of the intra prediction unit (291a) and the inter prediction unit (292a) of the encoding unit (113), respectively.

[0144] According to one embodiment, a residual signal obtained from the inverse transform unit (230b) can be added to a prediction signal (predicted block) output from the prediction unit (260b) to generate a restoration signal (restored picture, restored block).

[0145] According to one embodiment, the filtering unit (240b) can improve the image quality by applying filtering to the restored signal. For example, the filtering unit (240b) can apply various filtering methods to the restored picture to generate a modified restored picture and store the modified restored picture in the buffer (250b). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The restored picture stored in the buffer (250b) can be used as a reference picture in the prediction unit (260b).

[0146] According to one embodiment, the buffer (250b) can store motion prediction information of a block from which motion prediction information is derived (or decoded) within the current picture and / or motion prediction information of blocks within an already reconstructed picture. The stored motion prediction information can be transmitted to the inter prediction unit (262b) to be used as motion prediction information of a spatial neighboring block or motion prediction information of a temporal neighboring block. The buffer (250b) can store reconstructed samples of reconstructed blocks within the current picture, and the stored reconstructed samples can be transmitted to the intra prediction unit (262b).

[0147] Hereinafter, a preprocessing method using a frame skipping technique and a postprocessing method using a frame interpolation technique and a quality enhancement technique will be described. Meanwhile, a frame skipping algorithm may be used to determine whether to perform such a preprocessing method / postprocessing method. The frame skipping algorithm may be, for example, an algorithm used to determine whether it is advantageous to use a general compression / reconstruction method (e.g., a compression / reconstruction method using a standard codec technique) for a corresponding frame, or whether it is advantageous to use a method (proposed method) using the frame skipping technique, frame interpolation technique, and quality enhancement technique of the present disclosure. Through determination using the frame skipping algorithm, it may be determined whether an image transmitting device performs frame skipping processing before compression and an image receiving device performs frame interpolation and quality enhancement processing after decompression, or whether to process the corresponding frame according to a general compression / reconstruction method according to the proposed method for the corresponding frame. As an example, a method according to the frame skipping algorithm may be implemented by an AI model.

[0148] FIG. 3 illustrates a frame skipping processing module of a preprocessing unit according to an embodiment of the present disclosure. FIG. 4 schematically illustrates the operation of a frame skipping processing module according to an embodiment of the present disclosure. FIG. 5 is a flowchart of a frame skipping processing method according to an embodiment of the present disclosure.

[0149] Referring to FIG. 3, the preprocessing unit (112) may include a frame skipping processing module (310) for frame skipping processing. In the present disclosure, the operation of the frame skipping processing module (310) may be understood as the operation of the image transmission device (110) or at least one processor of the image transmission device (110).

[0150] According to one embodiment, the frame skipping processing module (310) may perform frame skipping processing on an input video signal. For example, the frame skipping processing module (310) may skip one frame from a plurality of identified frames (e.g., three or more frames, but is not limited thereto). For example, as illustrated in FIG. 4, the frame skipping processing module (310) may skip the frame (402), which is the middle frame, from three time-series consecutive frames (401, 402, 403) input to the frame skipping processing module (310). Thereafter, the frame skipping processing module (310) may transmit the remaining frames (e.g., frames (401, 403)) excluding the skipped frame (e.g., frame (402)) to the encoding unit (113). The skipped frame is not encoded by the encoding unit, thereby reducing the compression capacity. That is, through this frame skipping processing, high compression efficiency can be provided.

[0151] According to one embodiment, the frame skipping processing module (310) may perform frame skipping processing on an input video signal using a pre-learned AI model (or a pre-set frame skipping algorithm). For example, the frame skipping processing module (310) may determine whether to apply frame skipping to a corresponding frame using the pre-learned AI model. If it is determined that frame skipping is to be applied to the corresponding frame, the frame skipping processing module (310) may skip the corresponding frame and transmit the remaining frames excluding the skipped frame to the encoding unit (113). If it is determined that frame skipping is not to be applied to the corresponding frame, the frame skipping processing module (310) may transmit frames including the corresponding frame to the encoding unit (113). Through judgment using such an AI model, it is possible to accurately and quickly determine whether frame skipping processing is advantageous or whether compression / reconstruction is advantageous without frame skipping processing.

[0152] According to one embodiment, the frame skipping processing module (310) can selectively determine whether to apply frame skipping to a given frame. Rather than always skipping frames at fixed positions, this selective frame skipping processing based on such judgment can prevent frames whose quality is difficult to restore through, for example, frame interpolation and quality enhancement processing from being skipped.

[0153] Hereinafter, with reference to FIG. 5, exemplary operations of a frame skipping processing method of a frame skipping processing module (310) will be described. Meanwhile, the operations of the frame skipping processing module (310) to be described below may be understood as being controlled or performed by the image transmission device (110) or at least one processor of the image transmission device (110).

[0154] Referring to FIG. 5, in operation 5010, the frame skipping processing module (310) may obtain a frame set (or frame group) including a plurality of frames. For example, as illustrated in FIG. 4, in order to determine whether to apply frame skipping to the first frame (402), the frame skipping processing module (310) may obtain a frame set including the first frame (402), a frame (401) which is a frame preceding the first frame (402), and a frame (403) which is a frame subsequent to the first frame (402). Meanwhile, if the first frame of the frame set (e.g., the frame (401) which is a frame preceding the first frame (402)) is not an I (intra)-frame (e.g., an inter-frame (e.g., a P-frame)), the frame set may further include a frame preceding the frame (e.g., an I-frame) of the frame (401). That is, the frame set may include four frames. The number of frames included in such a frame set may correspond to the number of frames required for encoding processing, for example.

[0155] In operation 5020, the frame skipping processing module (310) may determine whether to apply frame skipping to the first frame (402) included in the frame set using a pre-trained model (or a pre-set frame skipping algorithm). For example, the frame skipping processing module (310) may determine whether to apply frame skipping to the first frame (402) using a pre-trained model (e.g., the model (7010) of FIG. 7). A description of determining whether to apply frame skipping to the corresponding frame will be described below with reference to FIG. 6.

[0156] In operation 5030, if it is determined that frame skipping is applied to the first frame (402), the frame skipping processing module (310) may skip the first frame (402). For example, as illustrated in FIG. 4, the frame skipping processing module (310) may skip the first frame (402) between the frames (401) and (403). The first frame (402) skipped in this manner is not encoded by the encoding unit.

[0157] In operation 5040, the frame skipping processing module (310) may transmit a set of frames in which the first frame (402) is skipped to the encoding unit. For example, the frame skipping processing module (310) may transmit a set of frames including the frame (401) and the frame (403) with the first frame (402) excluded to the encoding unit. Meanwhile, as described above, if the frame (401) is not an I-frame, the frame skipping processing module (310) may transmit a set of frames including the previous frame of the frame (401), the frame (401), and the frame (403) to the encoding unit. The set of frames transmitted in this manner may be encoded by the encoding unit.

[0158] In operation 5050, if it is determined that frame skipping is not applied to the first frame (402), the frame skipping processing module (310) can transfer a frame set including the frame (401), the first frame (402), and the frame (403) to the encoding unit. Meanwhile, as described above, if the frame (401) is not an I-frame, the frame skipping processing module (310) can transfer a frame set including the previous frame of the frame (401), the frame (401), the first frame (402), and the frame (403) to the encoding unit. The frame set transferred in this way can be encoded and processed by the encoding unit.

[0159] Hereinafter, with reference to FIG. 6, the frame skipping application decision operation of the frame skipping processing module (310) is exemplarily described.

[0160] FIG. 6 illustrates a frame skipping application decision operation of a frame skipping processing method according to an embodiment of the present disclosure. FIG. 7 illustrates an example of a model for frame skipping application decision according to an embodiment of the present disclosure. FIG. 8a illustrates an example of a rate distortion curve and a target rate distortion point on a rate distortion plane according to an embodiment of the present disclosure. FIG. 8b illustrates an example of a distance between a rate distortion curve and a target rate distortion point according to an embodiment of the present disclosure.

[0161] The operation of the frame skipping processing module (310) of FIG. 6 can be understood as being controlled or performed by the image transmission device (110) or at least one processor of the image transmission device (110).

[0162] Referring to FIG. 6, in operation 6010, the frame skipping processing module (310) may obtain a plurality of frames (e.g., a plurality of encoded and decoded frames or a plurality of original frames). For example, when determining whether to apply frame skipping to a first frame (e.g., a frame (402) of FIG. 4), the frame skipping processing module (310) may obtain a plurality of encoded and decoded frames (e.g., a plurality of encoded and decoded frames generated by encoding and decoding the frame (401), the first frame (402) and the frame (403) of FIG. 4) or a plurality of original frames (e.g., the frame (401), the first frame (402) and the frame (403) of FIG. 4). In the present disclosure, the encoded and decoded frames may also be referred to as codec-processed frames.

[0163] In operation 6020, the frame skipping processing module (310) may obtain information related to bit rate and distortion (e.g., information about a distance related to bit rate and distortion (distance information) but not limited thereto) using a pre-learned model (or a pre-set frame skipping algorithm) based on a plurality of frames (e.g., a plurality of encoded and decoded frames or a plurality of original frames). For example, as illustrated in FIG. 7, the frame skipping processing module (310) may input input data including a frame set (7001) including a plurality of encoded and decoded frames (e.g., a plurality of encoded and decoded frames generated by encoding and decoding the frame (401), the first frame (402), and the frame (403) of FIG. 4) or a plurality of original frames (e.g., the frame (401), the first frame (402), and the frame (403) of FIG. 4)) into a model (7010) (or a distance predictor), and obtain distance information (7002) as output data. In the present disclosure, the model (7010) may be referred to as a distance prediction model.

[0164] According to one embodiment, the distance information (7002) may include a distance associated with a difference between a first distortion and a second distortion at the same bit rate. The first distortion may be associated with a first set of frames (a first image frame set) to which frame skipping is applied to a first frame (a first image frame), and the second distortion may be associated with a second set of frames (a second image frame set) to which frame skipping is not applied to the first frame (a first image frame). According to one embodiment, the distance may correspond to a value obtained by subtracting the second distortion from the first distortion. Through the distance information obtained using such an AI model, it can be quantitatively determined whether frame skip processing is advantageous or compression / reconstruction without frame skip processing is advantageous. For example, through the distance information (7002), it can be determined whether it is advantageous to use a general compression / reconstruction method (e.g., a compression / reconstruction method using a standard codec technology) or whether it is advantageous to use a method (a proposed method) that uses a frame skipping technique, a frame interpolation technique, and a quality enhancement technique.

[0165] According to one embodiment, the distance prediction model may be an artificial intelligence model comprising multiple artificial neural network layers. The artificial neural network may be, for example, a DNN, a CNN, an RNN, an RBM, a DBN, a BRDNN, a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above.

[0166] According to one embodiment, the distance prediction model may include an input layer, hidden layer(s), and an output layer. According to one embodiment, the distance prediction model may include an input layer, a plurality of convolutional layers, a fully connected layer, and an output layer.

[0167] In one embodiment, the distance prediction model can be trained for each compression method and / or compression parameters associated with the compression method. For example, if the compression method follows the HEVC standard, each distance prediction model can be trained for each compression parameter (e.g., QP). For example, if the compression method uses four QPs, four distance prediction models can be trained individually for each QP.

[0168] According to one embodiment, the frame skipping processing module (310) may select one distance prediction model from among a plurality of pre-learned distance prediction models based on a compression scheme and / or compression parameters associated with the compression scheme. For example, the frame skipping processing module (310) may select a distance prediction model learned based on a target compression scheme (e.g., a scheme following the HEVC standard) and a target compression parameter (e.g., QP) associated with the target compression scheme from among a plurality of distance prediction models. The target compression scheme and the target compression parameters may be compression schemes and compression parameters used by an encoding unit (e.g., an encoding unit (113)) to encode the corresponding frames.

[0169] According to one embodiment, a distance prediction model can be trained based on a pre-collected training data set. An example of a method for collecting a training data set for training a distance prediction model is described below with reference to FIGS. 8a / b. In the present disclosure, the training data set may also be referred to as a training data set.

[0170] According to one embodiment, a rate distortion curve may be associated with rate distortion data (e.g., first rate distortion data (921) of FIG. 9) that includes bit rate values ​​and distortion values ​​obtained based on a plurality of frames to which frame skipping is not applied (e.g., frame (401), frame (402), and frame (403) of FIG. 4). For example, as illustrated in FIG. 8A, a rate distortion curve (8010) may correspond to a curve connecting points formed by pairs of bit rate values ​​and distortion values ​​obtained for each of a plurality of compression parameters (e.g., QP=0 to QP=51) based on a plurality of frames to which frame skipping is not applied (e.g., frame (401), frame (402), and frame (403)).

[0171] According to one embodiment, a target rate distortion point may be associated with rate distortion data (e.g., second rate distortion data (922) of FIG. 9) including bit rate values ​​and distortion values ​​obtained based on a plurality of frames to which frame skipping is applied (e.g., frames (401) and (403) of FIG. 4). For example, as illustrated in FIG. 8A, a target rate distortion point (8001) may be associated with a bit rate value ( ) obtained for a target compression parameter (e.g., QP=n (0<=n<=51)) based on a plurality of frames to which frame skipping is applied (e.g., frames (401) and (403) of FIG. 4). ) and distortion values ​​( ) may correspond to a point consisting of a pair of points.

[0172] According to one embodiment, the information about the distance may indicate the distance between the rate distortion curve and the target rate distortion point. For example, as illustrated in FIG. 8b, the distance (8020) may be the distance between the same bit rate ( ) may be the distance between the first straight line (8011) within the rate distortion curve (8010) and the target rate distortion point (8001). In one embodiment, the distance may be calculated by the following mathematical formula.

[0173] [Mathematical Formula 1]

[0174] distance= - c( ),

[0175] Here, c(rate) is a function for the first straight line (8011).

[0176] This distance can correspond to the difference in distortion when frame skipping is applied and when it is not applied at the same bit rate. When frame skipping is applied, the distortion can be obtained based on three original image frames and two corresponding image frames to be encoded and decoded and one image frame interpolated based on the two image frames, and the bit rate can be obtained based on the two image frames to be encoded and decoded. When frame skipping is not applied, the distortion can be obtained based on three original image frames and three corresponding image frames to be encoded and decoded, and the bit rate can be obtained based on the three image frames to be encoded and decoded. The distortion can be obtained, for example, using a preset method (e.g., the mean square error (MSE) method).

[0177] Meanwhile, depending on the embodiment, the distance may correspond to the vertical distance between the first straight line (8011) and the target rate distortion point (8001), rather than the distance calculated by mathematical expression 1.

[0178] At 6030, the frame skipping processing module (310) may determine whether to apply frame skipping to the first frame (e.g., frame (402) of FIG. 4) based on the information about the acquired distance. For example, if the distance is calculated according to Equation 1 and the distance value is negative, the frame skipping processing module (310) may determine to apply frame skipping to the first frame (e.g., frame (402) of FIG. 4). That is, in this case, the distortion when applying frame skipping at the same bit rate is smaller than the distortion when not applying frame skipping. For example, if the distance is calculated according to Equation 1 and the distance value is positive, the frame skipping processing module (310) may determine not to apply frame skipping to the first frame (e.g., frame (402) of FIG. 4). That is, in this case, the distortion when frame skipping is applied at the same bit rate is greater than the distortion when frame skipping is not applied.

[0179] Hereinafter, with reference to FIG. 9, an algorithm (hereinafter, frame skipping algorithm) for obtaining a distance corresponding to the difference in distortion between cases where frame skipping is applied and cases where it is not applied at the same bit rate is described, and an example of a method for generating a learning data set for learning a distance prediction model for frame skipping processing based on the algorithm is described.

[0180] FIG. 9 illustrates an example of a method for obtaining a distance between a rate distortion curve and a target rate distortion point, according to one embodiment of the present disclosure.

[0181] In the embodiment of FIG. 9, the distance may be, for example, the distance (e.g., distance 8020 in FIG. 8b) between a rate distortion curve (e.g., rate distortion curve 8010 in FIG. 8a) at the same bit rate and a target rate distortion point (e.g., target rate distortion point 8001 in FIG. 8b). This distance may be obtained according to a frame skipping algorithm corresponding to the recursive algorithm described below, but is not limited thereto.

[0182] Referring to FIG. 9, in operation 901, an image transmission device (e.g., image transmission device (110)) may acquire a preset number of frames (e.g., three or more). For example, the image transmission device may acquire three frames sequentially in time.

[0183] In operation 902, the video transmission device may apply frame skipping to one of the acquired frames. For example, the video transmission device may skip the middle frame among the three acquired frames.

[0184] In operation 911, the video transmission device can perform first video processing on the acquired frames without applying frame skipping. For example, the video transmission device can perform encoding processing and decoding processing (general compression / restoration processing) on ​​the three acquired frames without applying frame skipping. For example, when the compression method is HEVC, the compression parameter is QP, and the settable QP value is from 0 to 51, the video transmission device can perform encoding processing and decoding processing on the three acquired frames for each QP, respectively, to obtain data of the first video-processed frames for each QP.

[0185] In operation 921, the video transmission device may obtain first rate distortion data based on data of first image processed frames for each QP. The first rate distortion data may include a bit rate value and a distortion value obtained for each QP. That is, the first rate distortion data is ( , ), … ( , ), … , ( , ) may be included. According to one embodiment, the distortion value for QP may correspond to an MSE value obtained based on the original frames and the first image processed frames according to the QP.

[0186] In operation 912, the video transmission device can perform second video processing on frames to which frame skipping is applied. For example, the video transmission device can perform encoding processing, decoding processing, and post-processing (e.g., frame interpolation and quality enhancement processing) on ​​two frames to which frame skipping is applied. For example, when the compression method is HEVC and the target compression parameter is QP, the video transmission device can perform encoding processing, decoding processing, and post-processing on two frames, respectively, for the target QP, to obtain data of second video-processed frames for the target QP.

[0187] In operation 922, the video transmission device may obtain second rate distortion data based on data of second image processed frames for the target QP. The second rate distortion data may include a bit rate value and a distortion value obtained for the target QP. That is, the second rate distortion data may include ( , ) may be included. According to one embodiment, the distortion value for the target QP may correspond to an MSE value obtained based on the original frames and the second image processed frames according to the target QP.

[0188] In operation 930, the video transmission device can obtain distance data based on the first rate distortion data and the second rate distortion data.

[0189] According to one embodiment, the image transmission device can display a rate distortion curve on a rate distortion plane based on the first rate distortion data. For example, as illustrated in FIG. 8A, the image transmission device can display rate distortion points (e.g., ( , ), … ( , ), … , ( , )) can be displayed on a rate distortion plane, and the points can be connected to display a rate distortion curve (e.g., rate distortion curve (8010) of FIG. 8a).

[0190] According to one embodiment, the image transmission device can display a target rate distortion point on a rate distortion plane based on the second rate distortion data. For example, as illustrated in FIG. 8A, the image transmission device can display a rate distortion point (e.g., ( , )) can be displayed as a target rate distortion point (e.g., target rate distortion point (8001)) on the rate distortion plane.

[0191] According to one embodiment, the video transmission device has a distortion and bit rate for QP (0,51) and a target QP [ , , ] and [ , , ] can be obtained. After that, the video transmission device There are two sections [ , ] and [ , ] to check which one it belongs to. For example, < And, < To satisfy, This section [ , ], the video transmission device again sets three QPs [ to the midpoint of both QPs, 0 and 51. , ,51] is set. The video transmission device performs the above-described process on both sides QP [ , ] can be repeated recursively until the numbers are consecutive. Both QP [ , ] becomes a consecutive number, the video transmission device QP [ , ] About [ , ]and [ , ] and interpolate it to a straight line, and the function for the straight line (e.g., straight line (8011) of Fig. 8b) is obtained. The image transmission device obtains the target point (target rate distortion point) ( , ) can be calculated, for example, using the mathematical expression 1 described above. As described above, the distance may correspond to the difference in distortion at the same bit rate. For example, if the distance is negative, the target point [ , ] shows less distortion at the same bit rate, and conversely, when the distance is positive, the target point [ , ] can show less distortion at the same bit rate.

[0192] Meanwhile, by changing the target QP to all configurable QPs and repeatedly performing the operations described in FIG. 9 for the target QPs, the distances for each of the configurable target QPs can be obtained. In addition, for all available frames, as described above, the operations described in FIG. 9 can be repeatedly performed by grouping them into sets (frame sets) of a preset number (three) of frames.

[0193] However, when using the frame skipping algorithm of FIG. 9 described above, the video transmission device must repeatedly perform the same procedure for multiple QPs to obtain the distance value for the corresponding frame set in the target QP. Therefore, a long encoding time is required. Therefore, in order to obtain the distance, rather than using the frame skipping algorithm of FIG. 9 as is, it may be more useful to train an artificial intelligence model (e.g., the distance prediction model (7010) of FIG. 7) for each QP using the data set obtained using the frame skipping algorithm of FIG. 9, and to use the trained artificial intelligence model for each QP. For example, three frames may be input as input data of a distance prediction model associated with a QP, and the distance value (first distance value), which is the output of the distance prediction model, may be trained such that it is identical to the distance value (second distance value) associated with the QP corresponding to the three frames, which is known in advance through the frame skipping algorithm of FIG. 9. Meanwhile, the learning of the artificial intelligence model may be performed by the video transmission device, but may also be performed by another electronic device (e.g., a server), and then the learned artificial intelligence model (or at least one parameter associated with the artificial intelligence model) may be transmitted to the video transmission device.

[0194] FIG. 10 illustrates a frame interpolation processing module and a quality enhancement processing module of a post-processing unit according to an embodiment of the present disclosure. FIG. 11 schematically illustrates the operations of the frame interpolation processing module and the quality enhancement processing module according to an embodiment of the present disclosure.

[0195] Referring to FIG. 10, the post-processing unit (123) may include a frame interpolation processing module (1010) for frame interpolation processing and a quality enhancement processing module (1020) for quality enhancement processing. In the present disclosure, the operations of the frame interpolation processing module (1010) and the quality enhancement processing module (1020) may be understood as the operations of the image receiving device (120) or at least one processor of the image receiving device (120).

[0196] According to one embodiment, the frame interpolation processing module (1010) may perform frame interpolation processing on a decoded video signal. For example, the frame interpolation processing module (1010) may interpolate one frame using a plurality of decoded frames. For example, as illustrated in FIG. 11, the frame interpolation processing module (1010) may interpolate one frame (1102) between two frames (1101, 1103) input to the frame interpolation processing module (1010). The interpolated frame (1102) may be, for example, a frame corresponding to a time point in the middle of the frames (1101) and (1103) in time series.

[0197] According to one embodiment, the interpolated frame (1102) may correspond to a frame skipped (e.g., frame (402) of FIG. 4) by a frame skipping processing module of a video transmission device (e.g., frame skipping processing module (310) of FIG. 3). Thereafter, the frame interpolation processing module (1010) may transmit frames (1101, 1102, 1103) including the interpolated frame (1102) to the quality enhancement processing module (1020).

[0198] According to one embodiment, the frame interpolation processing module (1010) may determine whether to apply frame interpolation to a corresponding frame set based on information related to frame skipping transmitted from a video transmission device. If it is determined that frame interpolation is to be applied to the corresponding frame set, the frame interpolation processing module (310) may interpolate one frame (a first frame) (e.g., frame (1102) of FIG. 11) between a plurality of frames using a plurality of frames included in the corresponding frame set, and may transmit the frame set in which the first frame is interpolated to the quality enhancement processing module (1020). If it is determined that frame interpolation is not to be applied to the corresponding frame set, the frame interpolation processing module (310) may transmit the frame set in which the first frame is not interpolated to the quality enhancement processing module (1020). Alternatively, if it is determined that frame interpolation is not to be applied to the corresponding frame set, quality enhancement processing by the quality enhancement processing module (1020) may also be omitted.

[0199] According to one embodiment, the frame interpolation processing module (1010) may perform frame interpolation processing on a decoded image signal using a pre-learned AI model (hereinafter, “frame interpolation model”). The description and learning method of the frame interpolation model are described below with reference to FIG. 13.

[0200] According to one embodiment, the quality enhancement processing module (1020) can perform quality enhancement processing on a frame-interpolated image signal or a non-frame-interpolated image signal. For example, the quality enhancement processing module (1020) can perform quality enhancement processing on a frame-interpolated image signal. For example, as illustrated in FIG. 11, the quality enhancement processing module (1020) can perform quality enhancement processing on frame-interpolated frames (1101, 1102, 1103) to generate quality-enhanced frames (1121, 1122, 1123).

[0201] According to one embodiment, the quality enhancement processing module (1020) may perform quality enhancement processing on the corresponding frames using a pre-learned model (hereinafter, "quality enhancement model"). Quality enhancement processing using the quality enhancement model is described below with reference to FIG. 14.

[0202] According to one embodiment, the quality enhancement processing module (1020) may be a GoP enhancement processing module (e.g., the GoP enhancement processing module (1910) of FIG. 19) or may include a GoP enhancement processing module. The GoP enhancement processing module may, for example, perform GoP enhancement processing on a frame-interpolated video signal or a non-frame-interpolated video signal. A description of the GoP enhancement processing module is provided below with reference to FIGS. 17 to 23.

[0203] Hereinafter, with reference to FIG. 12, exemplary operations of the frame interpolation processing method of the frame interpolation processing module (1010) are described.

[0204] Fig. 12 is a flowchart of a frame interpolation processing method according to one embodiment of the present disclosure. Fig. 13 illustrates an example of a model for frame interpolation processing according to one embodiment of the present disclosure.

[0205] The operation of the frame interpolation processing module (1010) described below may be understood as being controlled or performed by the image receiving device (120) or at least one processor of the image receiving device (120).

[0206] Referring to FIG. 12, in operation 12010, the frame interpolation processing module (1010) may obtain a frame set including a plurality of decoded frames. For example, as illustrated in FIG. 11, the frame interpolation processing module (1010) may obtain a frame set including a decoded frame (1101) and a decoded frame (1103).

[0207] In operation 12020, the frame interpolation processing module (1010) may use information related to frame skipping to determine whether to apply frame interpolation to a frame set. For example, the frame interpolation processing module (1010) may use information related to frame skipping to determine whether to apply frame interpolation to a frame set. For example, if the information related to frame skipping indicates that a frame in the corresponding frame set is skipped, the frame interpolation processing module (1010) may determine to apply frame interpolation to the corresponding frame. For example, if the information related to frame skipping indicates that a frame in the corresponding frame set is not skipped, the frame interpolation processing module (1010) may determine not to apply frame interpolation to the corresponding frame.

[0208] In operation 12030, if it is determined to apply frame interpolation to the corresponding frame set, the frame interpolation processing module (1010) can interpolate one frame between multiple frames using multiple frames included in the corresponding frame set. For example, as illustrated in FIG. 11, the frame interpolation processing module (1010) can interpolate (or generate) a first frame (1102) at an intermediate point in time series using frames (1101) and (1103) within the corresponding frame set.

[0209] According to one embodiment, the frame interpolation processing module (1010) may interpolate a first frame using a pre-learned model (frame interpolation model) based on a plurality of frames included in the frame set. For example, as illustrated in FIG. 13, the frame interpolation processing module (1010) may input a frame set (1301) including a plurality of decoded frames (e.g., decoded frames (1101) and decoded frames (1103) of FIG. 11) as input data to the model (1310), and obtain a frame set (1302) in which the first frame is interpolated as output data. In the present disclosure, the model (1310) may be referred to as a frame interpolation model.

[0210] According to one embodiment, the frame interpolation model may be an artificial intelligence model comprising multiple artificial neural network layers. The artificial neural network may be, for example, a DNN, a CNN, an RNN, an RBM, a DBN, a BRDNN, a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above.

[0211] According to one embodiment, a frame interpolation model may include an input layer, a hidden layer(s), and an output layer. According to one embodiment, the frame interpolation model may include an input layer, a plurality of convolutional layers, a fully connected layer, and an output layer.

[0212] According to one embodiment, a frame interpolation model can be trained based on a data set obtained through an algorithm utilizing similarity that takes into account the time (or time axis) of an image. Meanwhile, when a frame interpolation model is trained based on a data set obtained through an algorithm utilizing motion vectors of an image, frame interpolation may be difficult because a damaged motion vector is obtained for an image damaged through compression / restoration (or encoding / decoding). However, this problem can be resolved when a frame interpolation model is trained based on a data set obtained through an algorithm utilizing similarity that takes into account the time axis of an image.

[0213] Meanwhile, the data set for learning the frame interpolation model includes not only data sets that have not experienced degradation due to compression / reconstruction (or encoding / decoding), but also data sets that have experienced degradation due to compression / reconstruction (or encoding / decoding). This allows the frame interpolation model to be learned even for images that have experienced degradation.

[0214] In operation 12040, the frame interpolation processing module (1010) may output a set of interpolated frames of the first frame (1102). For example, the frame interpolation processing module (1010) may transmit the set of interpolated frames of the first frame (1102) to the quality enhancement processing module (1020).

[0215] In operation 12050, if it is determined that frame interpolation is not to be applied to the corresponding frame set, the frame interpolation processing module (1010) can output a frame set in which the first frame (1102) is not interpolated.

[0216] FIG. 14 illustrates an example of a model for quality improvement processing according to one embodiment of the present disclosure.

[0217] The operation of the quality improvement processing module (1020) described below may be understood as being controlled or performed by the image receiving device (120) or at least one processor of the image receiving device (120).

[0218] Referring to FIG. 14, the quality enhancement processing module (1020) may input interpolated image data (1401) (e.g., image data including frames (1101, 1102, 1103) of FIG. 11) as input data to a model (1410), and obtain quality (or image quality) enhanced image data (1402) (e.g., image data including frames (1121, 1122, 1123) of FIG. 11) as output data. In the present disclosure, the model (1310) may be referred to as a quality enhancement model.

[0219] According to one embodiment, the quality improvement model may be an artificial intelligence model comprising multiple artificial neural network layers. The artificial neural network may be, for example, a DNN, a CNN, an RNN, an RBM, a DBN, a BRDNN, a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above.

[0220] In one embodiment, the quality enhancement model may include an input layer, hidden layer(s), and an output layer. In one embodiment, the quality enhancement model may include an input layer, a plurality of convolutional layers, a fully connected layer, and an output layer.

[0221] According to one embodiment, a quality enhancement model can be learned based on a data set obtained through an algorithm for quality enhancement that takes into account the time (or time axis) of an image.

[0222] Meanwhile, since quality enhancement processing is performed after frame interpolation processing as well as compression / reconstruction (or encoding / decoding) processing, the data set for learning the quality enhancement model includes not only data sets that have been degraded due to compression / reconstruction (or encoding / decoding) but also data sets that have been degraded due to frame interpolation. In other words, the quality enhancement model can be learned end-to-end by the entire network that performs frame interpolation and quality enhancement simultaneously.

[0223] FIG. 15 is a flowchart of an image processing method of an image transmission device according to one embodiment of the present disclosure.

[0224] Referring to FIG. 15, an image transmission device (e.g., an image transmission device (110) of FIG. 1a / b / c) can obtain image data including a plurality of image frames (15010).

[0225] The video transmission device can use an artificial intelligence model (or a frame skipping algorithm) to determine whether to apply frame skipping to a first image frame included in a plurality of image frames (15020).

[0226] When it is determined that frame skipping is to be applied to the first image frame, the video transmission device can skip the first image frame and encode the image data with the first image frame skipped to generate compressed image data (15030).

[0227] If it is not determined whether to apply frame skipping to the first image frame, the image transmitting device may encode image data including the first image frame to generate compressed image data (15040).

[0228] The video transmission device can transmit compressed image data and / or information related to frame skipping.

[0229] According to one embodiment, the image transmission device can select an artificial intelligence model from a plurality of artificial intelligence models based on parameters for encoding image data.

[0230] According to one embodiment, an image transmission device may encode image data in which a first image frame is skipped, using parameters for encoding, to encode image data in which a first image frame is skipped.

[0231] According to one embodiment, a parameter for encoding (e.g., a quantization parameter (QP)) may be associated with a compression ratio of the image data.

[0232] In one embodiment, the artificial intelligence model is configured to output a distance associated with a difference between a first distortion and a second distortion at the same bit rate, wherein the first distortion may be associated with a first set of image frames to which frame skipping has been applied for the first image frame, and the second distortion may be associated with a second set of image frames to which frame skipping has not been applied for the first image frame.

[0233] In one embodiment, the first image frame set may include a second image frame that is a previous frame of the first image frame and a third image frame that is a subsequent image frame of the first image frame, and the second image frame set may include the first image frame, the second image frame, and the third image frame.

[0234] In one embodiment, a video transmission device may encode and decode a first image frame, a second image frame which is a frame preceding the first image frame, and a third image frame which is a frame subsequent to the first image frame, to determine whether to apply frame skipping to the first image frame, input the encoded and decoded first image frame, the encoded and decoded second image frame, and the encoded and decoded third image frame as input data to the artificial intelligence model, obtain a distance as output data of the artificial intelligence model, and determine, based on the distance, whether to apply frame skipping to the first image frame.

[0235] In one embodiment, the video transmission device may input a first image frame, a second image frame that is a frame preceding the first image frame, and a third image frame that is a frame following the first image frame as input data to the artificial intelligence model to determine whether to apply frame skipping to the first image frame, obtain a distance as output data of the artificial intelligence model, and determine whether to apply frame skipping to the first image frame based on the distance.

[0236] In one embodiment, the distance may correspond to the first distortion minus the second distortion.

[0237] In one embodiment, to determine whether to apply frame skipping to a first image frame based on a distance, the video transmission device may determine to apply frame skipping to the first image frame if the distance is negative, and may determine not to apply frame skipping to the first image frame if the distance is positive.

[0238] In one embodiment, the artificial intelligence model is learned based on a plurality of learning data sets, and each of the plurality of learning data sets can be acquired based on performing the following operations: acquiring a set of image frames; performing a first image processing operation on the set of image frames for each of settable values ​​of parameters for encoding to acquire first rate distortion data; performing a second image processing operation on a set of image frames to which frame skipping is applied for values ​​of target parameters for encoding to acquire second rate distortion data; and performing an operation of acquiring a distance based on the first rate distortion data and the second rate distortion data.

[0239] According to one embodiment, the first image processing may include encoding processing and decoding processing, and the second image processing may include encoding processing, decoding processing, frame interpolation processing, and quality enhancement processing.

[0240] FIG. 16 is a flowchart of an image processing method of an image receiving device according to one embodiment of the present disclosure.

[0241] Referring to FIG. 16, an image receiving device (e.g., the image receiving device (120) of FIG. 1a / b) can obtain image data including a plurality of decrypted image frames (16010).

[0242] The video receiving device can determine whether to apply frame interpolation to multiple image frames based on information related to frame skipping (16020).

[0243] When it is determined that frame interpolation is to be applied to a plurality of image frames, the image receiving device can obtain image data in which the first image frame is interpolated using a first artificial intelligence model based on the plurality of image frames (16030).

[0244] When it is determined that frame interpolation is not to be applied to multiple image frames, the image receiving device can identify the first image frame as non-interpolated image data (16040).

[0245] According to one embodiment, the image receiving device can obtain enhanced image data using a second artificial intelligence model based on image data in which a first image frame is interpolated.

[0246] According to one embodiment, the information related to frame skipping may be set to either a first value indicating that frame skipping is applied to the first image frame or a second value indicating that frame skipping is not applied to the first image frame.

[0247] In one embodiment, the image receiving device may determine whether to apply frame interpolation, if information related to frame skipping is set to the first value, to apply frame interpolation to a plurality of image frames, and if information related to frame skipping is set to the second value, to not apply frame interpolation to a plurality of image frames.

[0248] According to one embodiment, the plurality of image frames may include a second image frame that is a previous image frame of the first image frame and a third image frame that is a subsequent image frame of the first image frame.

[0249] In order to obtain image data in which a first image frame is interpolated using a first artificial intelligence model according to one embodiment, an image receiving device may input a second image frame and a third image frame as input data to the first artificial intelligence model, and obtain image data in which the first frame is interpolated as output data from the first artificial intelligence model.

[0250] According to one embodiment, the first artificial intelligence model can be learned based on a data set obtained through an algorithm that utilizes similarity considering the time axis of the image.

[0251] Below, a preprocessing method using a technique for reducing the size and / or quality of an original image (e.g., but not limited to, a downscaling technique (or downsampling technique)) and a latency vector technique for compressing and expressing the resulting loss as a latency vector, and a postprocessing method using a technique for improving resolution (e.g., but not limited to, a super-resolution technique) are described.

[0252] In one embodiment, a video transmission device may downsample an original video (e.g., a high-quality video) using a downsampling technique to reduce the size and / or quality of the video. Thereafter, the video transmission device may compress the downsampled video (e.g., a low-quality video), thereby reducing the compression capacity.

[0253] According to one embodiment, a video transmission device can compress information about loss in an original video caused by downsampling (loss information) into a latency vector using latency vector technology. As an embodiment, the latency vector technology can be implemented by an artificial intelligence model (e.g., a deep learning-based artificial intelligence model). The latency vector that compresses the loss information in this way is compressed together with the video and transmitted to an image reception device, and can be used by the image reception device to increase the resolution of the downsampled video. In this way, for example, compared to directly compressing and transmitting a high-quality original video, compressing and transmitting a downsampled low-quality video using a latency vector that provides loss information can be more effective in terms of compression capacity and image quality.

[0254] In one embodiment, an image receiving device may restore an image using super-resolution technology based on a downscaled image and a latency vector providing loss information. In one embodiment, the super-resolution technology may be implemented using an artificial intelligence model (e.g., a deep learning-based artificial intelligence model). The image restored in this manner may have substantially the same quality as the original image, or in some cases, may have higher quality than the original image.

[0255] Hereinafter, with reference to each drawing, a preprocessing method using the above-described downscale technology and latent vector technology and a postprocessing method using super resolution technology for resolution enhancement are exemplarily explained.

[0256] FIG. 17 illustrates a downsampling processing module and a latency vector generation processing module of a preprocessing unit according to an embodiment of the present disclosure. FIG. 18 schematically illustrates the operations of a downsampling processing module and a latency vector generation processing module according to an embodiment of the present disclosure. FIG. 19 is a flowchart of a latency vector generation method according to an embodiment of the present disclosure. FIG. 20 illustrates an example of a model for latency vector generation processing according to an embodiment of the present disclosure.

[0257] Referring to FIG. 17, the preprocessing unit (112) may include a down-sampling processing module (1710) for down-scaling an image (or down-sampling processing) and a latency vector generation processing module (1720) for generating a latency vector that provides loss information of the image. In the present disclosure, the operations of the down-sampling processing module (1710) and the latency vector generation processing module (1720) may be understood to be controlled or performed by the image transmission device (110) or at least one processor of the image transmission device (110). Meanwhile, according to an embodiment, the latency vector generation processing module (1720) may not be included in the preprocessing unit (112) of FIG. 1b or the preprocessing unit (112a) of FIG. 1c, but may be included in the latency vector processing unit (115a) of FIG. 1c.

[0258] According to one embodiment, the down-sampling processing module (1710) may down-sample image data including at least one frame to generate down-sampled image data. For example, as illustrated in FIG. 18, the down-sampling processing module (1710) may down-sample original image data (1810) including at least one original frame to generate down-sampled image data (1820). The down-sampled image data (1820) may include at least one down-sampled frame, and each down-sampled frame may be a frame that is reduced in size, quality, and / or resolution compared to the corresponding original frame. For example, the down-sampled frame may be a frame that is reduced to 1 / 4 the size compared to the original frame.

[0259] According to one embodiment, the downsampling processing module (1710) may extract important information (e.g., information equivalent to 1 / 4 of the original image data) from the original image data using a frame selection technique, and obtain downsampled image data using the extracted important information. For example, the downsampling processing module (1710) may obtain downsampled image data (e.g., image data equivalent to 1 / 4 of the original image data) by removing information other than the extracted important information.

[0260] According to one embodiment, the downsampling processing module (1710) can transmit the downsampled image data to an encoding unit (e.g., the encoding unit (113) of FIG. 1B or the codec processing unit (113a) of FIG. 1C), and the encoding unit can encode the downsampled image data. The encoded image data can be transmitted to an image receiving device (e.g., the image receiving device (120) of FIGS. 1B and 1C). Through such downsampling, the compression capacity can be reduced, and transmission traffic can be saved. Meanwhile, information about the loss caused by the downsampling (loss information) can be transmitted to the image receiving device through a latency vector to be described later, and the image receiving device can improve the quality of the downsampled image by using the latency vector.

[0261] According to one embodiment, the latency vector generation processing module (1720) can generate latency vector data based on original image data and down-sampled image data.

[0262] For example, as illustrated in FIG. 18, the latency vector generation processing module (1720) can generate latency vector data (1830) based on original image data (1810) including at least one original frame and down-sampled image data (1820) including at least one down-sampled frame. According to one embodiment, the latency vector data (1830) can provide information about loss caused by down-sampling (loss information).

[0263] For example, as illustrated in FIG. 39a, the latency vector generation processing module (1720) can generate latency vector data (39113) based on image data (39105) in which compression degradation has occurred, generated through encoding and decoding processing (39103) of a codec processing unit (e.g., codec processing unit (113a) of FIG. 1c) for original image data (39109) including at least one original frame and downsampled image data (39101) including at least one downsampled frame.

[0264] According to one embodiment, the latency vector generation processing module (1720) may generate one latency vector per frame, but is not limited thereto. For example, one latency vector may be generated for multiple frames, or multiple latency vectors may be generated for one frame.

[0265] According to one embodiment, the latency vector generation processing module (1720) may generate latency vector data using a pre-trained AI model. For example, the latency vector generation processing module (1720) may generate latency vector data (1830) using a pre-trained AI model based on original image data (1810) including at least one original frame and down-sampled image data (1820) including at least one down-sampled frame. In the present disclosure, the AI ​​model used to generate the latency vector may be referred to as a latency vector generation model or a latency encoder model.

[0266] According to one embodiment, the latent vector generation model may be an artificial intelligence model comprising multiple artificial neural network layers. The artificial neural network may be, for example, a DNN, a CNN, an RNN, an RBM, a DBN, a BRDNN, a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above.

[0267] According to one embodiment, the latent vector generation model may include an input layer, hidden layer(s), and an output layer. According to one embodiment, the latent vector generation model may include an input layer, a plurality of convolutional layers, a fully connected layer, and an output layer.

[0268] Meanwhile, the latent vector generation model can be trained based on a pre-acquired data set.

[0269] Hereinafter, with reference to FIG. 19, exemplary operations of a latency vector generation method of a latency vector generation processing module (1720) will be described. Meanwhile, the operation of the latency vector generation processing module (1720) to be described below may be understood to be controlled or performed by the image transmission device (110) or at least one processor of the image transmission device (110). In the present disclosure, image data expressed as “down-sampled image data” may be, for example, image data itself generated by the down-sampling processing module (1710) (e.g., image data of FIG. 18 or image data (39101) of FIG. 39a), or image data obtained by encoding and decoding image data generated by the down-sampling processing module (1710) (e.g., image data (39105) of FIG. 39a).

[0270] Referring to FIG. 19, in operation 19010, the latency vector generation processing module (1720) may obtain down-sampled image data (e.g., image data (1820) of FIG. 18 or image data (39105) of FIG. 39a). For example, as illustrated in FIG. 18, the latency vector generation processing module (1720) may obtain down-sampled image data (1820) including a plurality of down-sampled frames (e.g., 3 or 4).

[0271] In operation 19020, the latency vector generation processing module (1720) may perform resolution interpolation on down-sampled image data to obtain resolution-interpolated image data (e.g., image data (1821) of FIG. 20 or image data (39107) of FIG. 39a). For example, as illustrated in FIG. 20, the latency vector generation processing module (1720) may perform resolution interpolation on down-sampled image data (1810) including a plurality of down-sampled frames to generate resolution-interpolated image data (1821) including a plurality of resolution-interpolated frames. The resolution-interpolated frames have the same size as the original frames.

[0272] According to one embodiment, the latent vector generation processing module (1720) may perform resolution interpolation on down-sampled image data using a preset interpolation method (e.g., bicubic interpolation).

[0273] In operation 19030, the latency vector generation processing module (1720) can obtain loss data based on original image data (e.g., image data (1810) of FIG. 20 or image data (39109) of FIG. 39a) and resolution-interpolated image data. For example, as illustrated in FIG. 20, the latency vector generation processing module (1720) can obtain loss data (2001) based on a difference between original image data (1810) including a plurality of original frames and resolution-interpolated image data (1821) including a plurality of resolution-interpolated frames.

[0274] According to one embodiment, the latency vector generation processing module (1720) may obtain loss data for a frame pair by calculating the difference in preset units (e.g., pixel units) for a frame pair consisting of an original frame (e.g., a first frame) and a corresponding resolution-interpolated frame (e.g., a frame obtained by downsampling the first frame and performing resolution interpolation). This calculation of the difference in pixel units may be performed for each frame pair. The overall loss data obtained in this manner may include loss data for each frame pair.

[0275] In operation 19040, the latency vector generation processing module (1720) can generate latency vector data (e.g., latency vector data (2002) of FIG. 20 or latency vector data (39113) of FIG. 39a) based on loss data using a pre-learned model (e.g., model (2010) of FIG. 20 or model (39111) of FIG. 39a). For example, as illustrated in FIG. 20, the latency vector generation processing module (1720) can input loss data (2001) for a plurality of frame pairs into the latency vector generation model (2010) (or a latency encoder including the latency vector generation model (2010)). The latency vector generation processing module (1720) can obtain latency vector data (2002) output from the latency vector generation model (2010). Through this, information about the pixel-level difference (loss information) for each frame pair can be compressed into a latency vector. The latency vector data obtained in this way can include compressed information about the pixel-level difference (loss information) for each frame pair. Since the latency vector data includes compressed information about the loss information rather than the loss information itself, it has a smaller data size compared to the loss information. This can reduce the compression capacity. In addition, since the latency vector data includes loss information, the latency vector data can be used in an image receiving device to restore information lost due to downsampling.

[0276] In operation 19050, the latency vector generation processing module (1720) may transmit the generated latency vector data to the encoding unit. The latency vector data transmitted to the encoding unit may be encoded (or compressed) together with down-sampled image data and transmitted to the image receiving device. As an embodiment, the encoded latency vector data may be added to a bitstream including the encoded image data. For example, a first bitstream generated by encoding the latency vector data (e.g., a first bitstream generated by quantizing the latency vector data and performing entropy coding) may be included in a portion of a second bitstream generated by encoding the image data. For example, the first bitstream generated by encoding the latency vector data may be included in a description area of ​​the second bitstream generated by encoding the image data, but is not limited thereto. The description area may be, for example, an area for describing information and a structure of the corresponding bitstream. Meanwhile, according to an embodiment, operation 19050 may be omitted. For example, when the latency vector generation processing module (1720) is included in the latency vector processing unit (115a) of FIG. 1C, the latency vector generated by the latency vector generation processing module (1720) may be directly transmitted to the image output unit (e.g., the image output unit (114a) of FIG. 1C) without being encoded by the encoding unit (e.g., the codec processing unit (113a) of FIG. 1C).

[0277] FIG. 21 illustrates a super-resolution processing module of a post-processing unit according to an embodiment of the present disclosure. FIG. 22a illustrates the operation of a super-resolution processing module according to an embodiment of the present disclosure. FIG. 22b illustrates an example of a model for super-resolution processing according to an embodiment of the present disclosure.

[0278] Referring to FIG. 21, the post-processing unit (123) may include a super-resolution processing module (2110) for resolution enhancement processing. According to one embodiment, the super-resolution processing module (2110) may perform processing for quality enhancement (e.g., resolution enhancement) of a decoded image signal.

[0279] Referring to FIG. 22a, in operation 22010, the super resolution processing module (2110) can obtain decoded image data and latency vector data (e.g., latency vector data decoded by the decoding unit (122) of FIG. 1b or latency vector data transmitted by the image input unit (121a) of FIG. 1c).

[0280] According to one embodiment, as illustrated in FIG. 23, a decryption unit (e.g., decryption unit (122) of FIG. 1B) may decrypt received image data and latency vector data, and transmit the decrypted image data (e.g., image data (2201) of FIG. 23) and the decrypted latency vector data (e.g., latency vector data (2202) of FIG. 23) to a super-resolution processing module (2110). In this case, the super-resolution processing module (2110) may obtain the decrypted image data and the decrypted latency vector data transmitted from the decryption unit (122).

[0281] According to one embodiment, as illustrated in FIG. 39a, a decoding unit (e.g., codec processing unit (122a) of FIG. 1c) can decode received image data and transmit the decoded image data (e.g., image data (39205) of FIG. 39a) to a super-resolution processing module (e.g., SR model (39207) of FIG. 39a), and an image input unit (e.g., image input unit (121a) of FIG. 1c) can transmit the received latency vector data to the super-resolution processing module (e.g., SR model (39207) of FIG. 39a). In this case, the super-resolution processing module (39207) can obtain the decoded image data transmitted from the codec processing unit (122a) and the latency vector data transmitted from the image input unit (121a).

[0282] In operation 22020, the super resolution processing module (2110) can generate image data with improved resolution using a pre-learned model (hereinafter, a super resolution (SR) model) based on decoded image data and latent vector data (or, decoded latent vector data).

[0283] For example, as illustrated in FIGS. 22b and 23, the super-resolution processing module (2110) can input decoded image data (2201) and decoded latent vector data (2202) as input data to the model (2210), and obtain enhanced image data (2003) as output data.

[0284] For example, as illustrated in FIG. 39a, the super-resolution processing module (39207) can input decoded image data (39205) and (undecoded) latent vector data as input data to the model (39207), and obtain enhanced image data (39209) as output data.

[0285] In one embodiment, a super-resolution model may consist of a vision transformer-based encoder and decoder. In this case, latent vector data may be used between the encoder and decoder to provide additional information (e.g., loss information) for restoring the image. This allows the restored image to have the same size as the original image.

[0286] In one embodiment, the super-resolution model may be an artificial intelligence model comprising multiple artificial neural network layers. The artificial neural network may be, for example, a DNN, a CNN, an RNN, an RBM, a DBN, a BRDNN, a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above.

[0287] According to one embodiment, a super-resolution model may include an input layer, hidden layer(s), and an output layer. According to one embodiment, a super-resolution model may include an input layer, a plurality of convolutional layers, a fully connected layer, and an output layer.

[0288] Meanwhile, since super-resolution processing is performed after latency vector processing and compression / restoration processing, the data set for training the super-resolution model includes not only data sets that have been degraded by latency vector processing, but also data sets that have been degraded by compression / restoration. In other words, the super-resolution model can be trained end-to-end by the entire network that performs latency vector generation and compression / restoration processing together.

[0289] FIG. 23 illustrates an example of an image processing procedure according to one embodiment of the present disclosure.

[0290] The image processing procedure of FIG. 23 may be an example of an image processing procedure that utilizes latent vector technology for pre-compression processing and super-resolution technology for post-restoration processing. For example, the image processing procedure may be an example of an image processing procedure that utilizes a latent vector generation processing module (e.g., the latent vector generation processing module (1710) of FIG. 17) and a super-resolution processing module (e.g., the super-resolution processing module (2110) of FIG. 21).

[0291] Referring to FIG. 23, an image transmission device (e.g., the image transmission device (110) of FIG. 1a / b / c) can obtain original image data (1810) including at least one original frame.

[0292] According to one embodiment, the image transmission device can perform downsampling on original image data (1810) (e.g., original image data including a plurality of frames (e.g., a first frame, a second frame, and a third frame)) to obtain downsampled image data (1820) (e.g., downsampled image data including a plurality of downsampled frames (e.g., a first downsampled frame, a second downsampled frame, and a third downsampled frame)).

[0293] According to one embodiment, the image transmission device can perform resolution interpolation on down-sampled image data (1820) to obtain resolution interpolated image data (1821) (e.g., resolution interpolated image data including a plurality of resolution interpolated frames (e.g., a first resolution interpolated frame, a second resolution interpolated frame, a third resolution interpolated frame)).

[0294] According to one embodiment, the image transmission device can obtain loss data based on original image data (1810) and resolution-interpolated image data (1821). For example, the image transmission device can obtain loss data for a frame pair by calculating a difference in a preset unit (e.g., pixel unit, but not limited thereto) between an original frame in the original image data (1810) and a resolution-interpolated frame corresponding to the original frame in the resolution-interpolated image data (1821) (e.g., a frame pair including a first frame and a first resolution-interpolated frame obtained by downsampling and resolution-interpolating the first frame). In this manner, loss data for the entire frame pair can be obtained.

[0295] According to one embodiment, a video transmission device can input loss data into a latency vector generation model (or a latency encoder including the latency vector generation model) (2010) and obtain latency vector data (1830) output from the latency vector generation model (or the latency encoder) (2010).

[0296] According to one embodiment, an image transmission device may input down-sampled image data (1820) and latency vector data (1830) into an encoding unit (113), and transmit encoded image data and encoded latency vector data output from the encoding unit (113) to an image reception device (e.g., the image reception device (120) of FIG. 2). As an embodiment, the encoded latency vector data may be added to a bitstream including the encoded image data.

[0297] According to one embodiment, an image receiving device can receive encoded image data and encoded latency vector data and transmit them to a decoding unit (122). The image receiving device can obtain decoded image data (2201) and decoded latency vector data (2202) output from the decoding unit (122).

[0298] According to one embodiment, an image receiving device may input decrypted image data (2201) and decrypted latent vector data (2202) into an SR model (2210) and obtain enhanced image data (2003) output from the SR model (2210). The frames included in the enhanced image data (2003) thus obtained may have substantially the same quality as the frames included in the original image data (1810).

[0299] FIG. 24 is a flowchart of an image processing method of an image transmission device according to one embodiment of the present disclosure.

[0300] Referring to FIG. 24, an image transmission device (e.g., an image transmission device (110) of FIG. 2) can down-sample image data (e.g., image data (1810) of FIG. 23 or image data (39109) of FIG. 39a) to obtain down-sampled image data (e.g., image data (1820) of FIG. 23 or image data (39105) of FIG. 39a) (24010).

[0301] The image transmission device can perform resolution interpolation on the down-sampled image data to obtain resolution-interpolated image data (e.g., image data (1821) of FIG. 23 or image data (39107) of FIG. 39a) (24020).

[0302] The image transmission device can obtain loss data (e.g., loss data (2001) of FIG. 23 or loss data (39110) of FIG. 39a) based on the image data and the resolution-interpolated image data (24030).

[0303] The video transmission device can generate latency vector data (e.g., latency vector data (2002) of FIG. 23 or latency vector data (39113) of FIG. 39a) using an artificial intelligence model (e.g., model (2010) of FIG. 23 or model (39111) of FIG. 39a) based on the above loss data (24040).

[0304] The video transmission device can encode the down-sampled image data and / or the latency vector data (24050). For example, as illustrated in FIG. 23, the video transmission device can encode the down-sampled image data (1820) and the latency vector data (2002) using the encoding unit (113). For example, as illustrated in FIG. 39a, the video transmission device can encode the down-sampled image data (39101) using the encoding unit (113a) and not encode the latency vector data (39113).

[0305] The video transmission device can transmit the encoded image data and / or the encoded latency vector data (24050). For example, as illustrated in FIG. 23, the video transmission device can transmit the encoded image data and the encoded latency vector data to the video reception device. For example, as illustrated in FIG. 39a, the video transmission device can transmit the encoded image data (39101) and (unencoded) latency vector data to the video reception device.

[0306] According to one embodiment, the video transmission device can obtain the loss data by calculating a difference in a preset unit for a frame pair consisting of a first frame included in the image data and a second frame corresponding to the first frame included in the resolution-interpolated image data, in order to obtain the loss data.

[0307] According to one embodiment, the preset unit may correspond to a pixel unit.

[0308] According to one embodiment, the image transmission device may include an operation of performing resolution interpolation on the down-sampled image data using a bicubic interpolation method to obtain the resolution interpolated image data.

[0309] According to one embodiment, the video transmission device can generate a bitstream including encoded image data and encoded latency vector data by encoding the down-sampled image data and the latency vector data using a standardized codec technology.

[0310] According to one embodiment, the image transmission device may include an operation of transmitting the bitstream including the encoded image data and the encoded latent vector data.

[0311] FIG. 25 is a flowchart of an image processing method of an image receiving device according to one embodiment of the present disclosure.

[0312] Referring to FIG. 25, an image receiving device (e.g., image receiving device (120) of FIG. 2) can receive encoded image data and latency vector data (e.g., encoded latency vector data of FIG. 23 or (unencoded) latency vector data of FIG. 39a) (25010).

[0313] The image receiving device can decode the encoded image data and / or the encoded latency vector data (25020). For example, as illustrated in FIG. 23, the image receiving device can decode the encoded image data and the encoded latency vector data using a decoding unit (122). For example, as illustrated in FIG. 39a, the image receiving device can decode the encoded image data using a decoding unit (122a) and transmit the latency vector data to the SR model (39207) without decoding it.

[0314] The image receiving device can generate image data with improved resolution (e.g., image data (2003) of FIG. 23 or image data (39209) of FIG. 39a) using an artificial intelligence model (e.g., model (2210) of FIG. 23 or model (39207) of FIG. 39a) based on the decrypted image data and the latency vector data (e.g., decrypted latency vector data (2201) of FIG. 23 or latency vector data (39202) of FIG. 39a) (25030).

[0315] According to one embodiment, the latent vector data is generated based on loss data used for restoration of decoded image data, and the image receiving device can obtain the loss data by calculating a difference of a preset unit for a pair of associated frames.

[0316] According to one embodiment, the preset unit may correspond to a pixel unit.

[0317] According to one embodiment, the artificial intelligence model may be composed of an encoder and decoder based on a vision transformer.

[0318] According to one embodiment, an image receiving device can receive a bitstream including the encoded image data and the encoded latent vector data.

[0319] According to one embodiment, the image receiving device can decode the encoded image data and the encoded latent vector data using a standardized codec technology.

[0320] Meanwhile, the embodiments described above in FIGS. 3 to 16 (the embodiment related to frame omission / frame interpolation / quality enhancement technology (the first embodiment)) and the embodiments described above in FIGS. 17 to 25 (the embodiment related to downsampling / latent vector / SR technology (the second embodiment)) can be combined with each other within a range that does not contradict each other. For example, frame omission, downsampling, and / or latency vector technologies can be combined and used together for preprocessing before compression (encoding), and frame interpolation, quality enhancement, and SR technologies can be combined and used together for postprocessing after restoration (decoding). An example of such a combined embodiment is described below. However, this is merely an example, and the technologies can be combined and used in various orders and methods. Meanwhile, the same content as that described above in FIGS. 3 to 25 can be referenced in FIGS. 26 to 28 below.

[0321] Fig. 26 shows an exemplary configuration of a preprocessing unit according to one embodiment of the present disclosure.

[0322] Referring to FIG. 26, the preprocessing unit (112) (or, the image transmission device (e.g., the image transmission device (100) of FIG. 1) may include a frame skipping processing module (310), a down-sampling processing module (1710), and a latency vector processing module (1720). In one embodiment, the operation of the frame skipping processing module (310) may be performed before the operations of the down-sampling processing module (1710) and the latency vector processing module (1720), but is not limited thereto. For a description of the frame skipping processing module (310), reference may be made to the description of FIGS. 3 to 16 described above. For a description of the down-sampling processing module (1710) and the latency vector processing module (1720), reference may be made to the description of FIGS. 17 to 25 described above. Meanwhile, according to an embodiment, the latency vector processing module (1720) is not included in the preprocessing unit (112) and may be included in the operation of the image transmission device. It may be included in a latency vector processing unit (e.g., the latency vector processing unit (115a) of FIG. 1c).

[0323] Fig. 27 shows an exemplary configuration of a post-processing unit according to one embodiment of the present disclosure.

[0324] Referring to FIG. 27, the post-processing unit (123) may include a frame interpolation processing module (1010), a quality enhancement processing module (1020), and a super-resolution processing module (2110). In one embodiment, the operation of the super-resolution processing module (2110) may be performed after the operations of the frame interpolation processing module (1010) and the quality enhancement processing module (1020), but is not limited thereto. For a description of the frame interpolation processing module (1010) and the quality enhancement processing module (1020), reference may be made to, for example, the description of FIGS. 3 to 16 described above. For a description of the super-resolution processing module (2110), reference may be made to, for example, the description of FIGS. 17 to 25 described above.

[0325] FIG. 28 illustrates an example of an image processing procedure according to one embodiment of the present disclosure.

[0326] In the embodiment of FIG. 28, it is assumed that the operation of the frame skipping processing module (e.g., the frame skipping processing module (310) of FIG. 3) is performed before the operations of the downsampling processing module (e.g., the downsampling processing module (1710) of FIG. 17) and the latency vector processing module (e.g., the latency vector processing module (1720) of FIG. 17), and the operation of the super resolution processing module (e.g., the super resolution processing module (2110) of FIG. 21) is performed after the operations of the frame interpolation processing module (e.g., the frame interpolation processing module (1010) of FIG. 10) and the quality enhancement processing module (e.g., the quality enhancement processing module (1020) of FIG. 10). However, the embodiment is not limited thereto, and the opposite case is also possible.

[0327] Referring to FIG. 28, an image transmission device (e.g., the image transmission device (110) of FIGS. 1A, 1B, and 1C) can obtain original image data including at least one original frame.

[0328] According to one embodiment, a video transmission device may perform a frame selection operation (or frame skipping operation) to obtain frame-skipped image data (2810). For a description of the frame skipping operation, please refer to the descriptions of FIGS. 3 to 9 and FIG. 15.

[0329] According to one embodiment, the image transmission device may perform a downscaling operation (or a downsampling operation) and a latency vector estimation operation (or a latency vector generation operation) to generate downscaled image data and latency vector data (2820). As an example, the downsampling operation may be performed before the latency vector generation operation. For a description of the downsampling operation and the latency vector generation operation, reference may be made to the descriptions of FIGS. 17 to 20 and FIG. 24.

[0330] According to one embodiment, a video transmission device can encode downscaled image data and / or latency vector data using, for example, a standardized codec technology (e.g., HEVC, VVC, H.264), and transmit the encoded image data and latency vector data (or, encoded latency vector data) to a video reception device (e.g., the video reception device (120) of FIGS. 1A, 1B, and 1C), and the video reception device can receive the encoded image data and latency vector data (or, encoded latency vector data), and decode the encoded image data and / or the encoded latency vector data (2830).

[0331] According to one embodiment, the image receiving device may perform an interpolation operation (or frame interpolation operation) and an enhancement operation (or quality enhancement operation) on decoded image data to obtain frame interpolated / quality enhanced image data (2840). As an example, the frame interpolation operation may be performed before the quality enhancement operation, but is not limited thereto. For a description of the frame interpolation operation and the quality enhancement operation, reference may be made to the descriptions of FIGS. 10 to 14 and FIG. 16.

[0332] According to one embodiment, an image receiving device can perform a super-resolution operation on frame interpolation / quality-enhanced image data using latency vector data (or decoded latency vector data) to obtain image data with improved resolution (2850). For a description of the super-resolution operation, refer to the descriptions of FIGS. 21 to 23 and FIG. 25. The final image data thus obtained can provide substantially the same quality as the original image data.

[0333] Below, a post-processing method using GoP enhancement processing is described.

[0334] FIG. 29 illustrates an original image and an encoded and decoded image according to one embodiment of the present disclosure.

[0335] Part (a) of Fig. 29 is an example of an original image. Referring to part (a) of Fig. 29, the original image may include a first part (2910).

[0336] Part (b) of FIG. 29 is an example of a decoded image generated by receiving an encoded image generated by encoding the original image of part (a) of FIG. 29 by an image receiving device (e.g., the image receiving device (120) of FIGS. 1a, 1b, and 1c) and decoding the encoded image. Referring to part (b) of FIG. 29, the encoded and decoded image may include a second part (2920) corresponding to a first part (2910) of the original image. The first part (2910) and the second part (2920) may be parts that include the same area within a frame of the original image and a frame of the encoded and decoded image. In the present disclosure, the encoded and decoded image may also be referred to as an image that has been codec-processed by a codec processing unit (e.g., the codec processing unit (113a) or the codec processing unit (122a) of FIG. 1c).

[0337] According to one embodiment, encoding processing for the original image may be performed by an encoding unit of an image transmitting device (e.g., an encoding unit (113) of an image transmitting device (110) of FIG. 1a / b or a codec processing unit (113a) of an image transmitting device (110) of FIG. 1c), and decoding processing for the encoded image may be performed by a decoding unit of an image receiving device (e.g., a decoding unit (122) of an image receiving device (120) of FIG. 1a / b or a codec processing unit (122a) of an image receiving device (120) of FIG. 1c).

[0338] According to one embodiment, a video transmission device (or encoding unit) may perform encoding processing in units of a frame group including at least one reference frame (e.g., an I frame). The frame group may include, for example, a sequence of consecutive frames including one reference frame. The reference frame may be used to compress (or encode) frames within the frame group. For example, the frame group may correspond to GoP.

[0339] According to one embodiment, a video transmission device (or encoding unit) can compress (or encode) frames within a frame group (or GoP) based on one reference frame within the frame group (or GoP). In this way, video compression (or encoding) can be performed on a frame group (or GoP) basis. In this way, since video compression is performed for a frame group (or GoP) based on a reference frame within the frame group (or GoP), degradation that occurs when compression / reconstruction is performed can occur on a frame group (or GoP) basis.

[0340] Hereinafter, for convenience of explanation, embodiments of the present disclosure will be described assuming that the unit in which encoding processing (or compression processing) is performed is a GoP. However, it should be understood that the description of the embodiments of the present disclosure can also be applied equally to a frame group having the same role / function as the GoP described below.

[0341] According to one embodiment, a GoP may include various types of frames. For example, a GoP may include an intra-coded frame (I frame), a predictive-coded frame (P frame), and / or a bi-predictive-coded frame (B frame).

[0342] In one embodiment, an I-frame may be a frame that is encoded independently without relying on other frames. An I-frame may, for example, contain a complete image at a specific point in a video sequence. In the present disclosure, an I-frame may also be referred to as a reference frame or a key frame.

[0343] In one embodiment, a P frame may be a frame that is predictively encoded based on a previously encoded I frame or P frame. A P frame may be encoded using difference information with respect to an I frame or another P frame.

[0344] According to one embodiment, a B frame may be a frame that is predictively encoded based on previous and subsequent frames. A B frame is located between an I frame and a P frame, and may be encoded by utilizing as much difference information as possible.

[0345] In one embodiment, a GoP includes an I-frame as the first frame within the GoP, and if the GoP includes more than one frame, the GoP may further include at least one B-frame and / or at least one P-frame. For example, if the GoP has a size of 12, the GoP may have a GoP pattern such as IBPPPPBBPBBP.

[0346] In one embodiment, the first frame of a GoP, the I frame, may be compressed using image compression (e.g., intra compression) because it has no reference frames. However, the remaining frames of the GoP may be compressed using video compression (e.g., inter compression) that references other adjacent frames. For example, if a GoP consists of 12 frames, intra compression may be applied to the first frame, and inter compression may be applied to the remaining 11 frames. In this case, the inter compression applied to the 11 frames continues to perform compression by referencing the compressed frame referenced by the first frame. Therefore, distortion due to the compression of the first frame may be transmitted to the following 11 frames. This causes distortion in the following frames within the GoP. Hereinafter, an example of distortion occurring within a GoP will be described with reference to FIG. 30.

[0347] FIG. 30 illustrates an image (first image) showing a first portion of an original image in chronological order and an image (second image) showing a second portion of an encoded and decoded image corresponding to the first portion of the original image in chronological order, according to one embodiment of the present disclosure.

[0348] Part (a) of Fig. 30 shows an image (hereinafter, the first image) in which a first part (e.g., the first part (2910) of the original image shown in part (a) of Fig. 29) is extracted frame by frame from the frames of the original image and displayed stacked from top to bottom in chronological order.

[0349] Part (b) of Fig. 30 shows an image (hereinafter, the second image) in which a second part (e.g., the second part (2920) of the encoded and decoded image shown in part (b) of Fig. 29) is extracted frame by frame from the frames of the encoded and decoded image, and displayed stacked from top to bottom in time order.

[0350] Referring to part (a) of Fig. 30, in the case of the first image associated with the original image, it can be confirmed that the change of the image appears smoothly according to the time sequence.

[0351] In contrast, referring to part (b) of FIG. 30, in the case of the second image associated with the encoded and decoded video, it can be confirmed that the change of the corresponding image is not smooth over time. As illustrated, it can be confirmed that a large change occurs in units of GoP. That is, it can be confirmed that a sharp change occurs for the corresponding image at the boundary of the GoP. This distortion occurs because, as described above, the distortion due to compression of the first I-frame, which is the first frame of the GoP, is transferred to the following 11 frames, and then the distortion due to compression of the second I-frame (different from the first I-frame), which is the first frame of the GoP, is transferred to the following 11 frames. That is, since different distortions occurred in the I-frame, which is the first frame of each GoP, and the remaining frames of each GoP performed compression with reference to the I-frame, the different distortion characteristics of the I-frames are also transferred to the remaining frames. Therefore, different distortion characteristics may exist between GoPs.

[0352] Meanwhile, various methods exist to restore distortion caused by compression. However, these methods are primarily aimed at maintaining consistency within a single frame, or maintaining time-domain consistency between adjacent frames.

[0353] However, as described above, distortion due to compression also occurs between GoPs (or groups of frames). Therefore, a new method for restoring the distortion occurring between GoPs needs to be considered. Below, a post-processing method utilizing GoP enhancement processing to restore the distortion (hereinafter, GoP distortion) occurring between GoPs (e.g., between adjacent GoPs) is described.

[0354] FIG. 31 illustrates a GoP enhancement processing module of a post-processing unit according to an embodiment of the present disclosure. FIG. 32 illustrates the operation of a GoP enhancement processing module according to an embodiment of the present disclosure. FIG. 33 illustrates an example of a procedure for alignment processing of a GoP enhancement processing module according to an embodiment of the present disclosure.

[0355] Referring to FIG. 31, the post-processing unit (123) may include a GoP enhancement processing module (3110) for quality improvement. The frames input to the GoP enhancement processing module (3110) may correspond to, for example, frames decoded by the decoding unit (122) operating before the post-processing unit (123).

[0356] According to one embodiment, the GoP enhancement processing module (3110) can perform processing for quality improvement (e.g., GoP distortion restoration) on a decoded image signal.

[0357] According to one embodiment, the GoP enhancement processing module (3110) may be an example of a quality improvement processing module (1020).

[0358] According to one embodiment, the GoP enhancement processing module (3110) can restore frames in units of a preset number of frames (e.g., 3). For example, the GoP enhancement processing module (3110) can restore the frames by performing processing to restore GoP distortion in units of 3 frames.

[0359] Hereinafter, the operation of the GoP enhancement processing module or the components included in the GoP enhancement processing module may be understood as the operation of an image receiving device (e.g., an image receiving device (120) of FIG. 1a / b) or a post-processing unit of the image receiving device (e.g., a post-processing unit (123) of the image receiving device (120) of FIG. 1a / b).

[0360] Referring to FIG. 32, the GoP enhancement processing module (3110) may include an alignment processing unit (3210) and / or an enhancement network (3220). In the present disclosure, the enhancement network (3220) may be referred to as an enhancement model.

[0361] According to one embodiment, the alignment processing unit (3210) can perform at least one operation for alignment processing on frames (e.g., I frames).

[0362] According to one embodiment, the alignment processing unit (3210) may obtain (or identify) a first frame of a first frame group and a second frame of a second frame group following the first frame group. As an embodiment, the first frame and the second frame may correspond to independently encoded and decoded frames. The first frame and the second frame may be frames used for encoding (or compressing) at least one other frame within the corresponding frame group during encoding. As an example, the first frame and the second frame may be I-frames. As an example, the second frame group may be a frame group immediately following the first frame group. That is, the second frame group may be a frame group adjacent to the first frame group and positioned after the first frame group.

[0363] In one embodiment, the alignment processing unit (3210) may perform processing to align the first frame with a plurality of consecutive frames of the first frame group (e.g., three consecutive frames), thereby generating a plurality of first aligned frames (e.g., three first aligned frames). In the present disclosure, aligning the first frame with the corresponding frame may mean processing the first frame so that the first frame becomes substantially identical to the corresponding frame. In an embodiment, the plurality of frames may correspond to, for example, frames predictively encoded and decoded based on the first frame. As an example, the plurality of frames may be P frames or B frames.

[0364] According to one embodiment, the alignment processing unit (3210) can generate a plurality of first aligned frames by using an optical flow technique and / or a warping technique in the pixel domain. For example, the alignment processing unit (3210) can adjust the positions of pixels within the first frame by using the optical flow technique and the warping technique to align (or align) the first frame with a plurality of consecutive frames of the first frame group.

[0365] In one embodiment, an optical flow technique may be a technique for estimating the movement pattern of pixels in an image. For example, the optical flow technique may be used to estimate the direction and speed of movement of pixels by utilizing changes in the brightness of the pixels.

[0366] According to one embodiment, the alignment processing unit (3210) can identify how individual pixels move in an image by calculating optical flow using an optical flow technique. That is, the alignment processing unit (3210) can obtain motion information of an image by calculating optical flow.

[0367] According to one embodiment, the alignment processing unit (3210) may obtain a flow map including information on how to move (or adjust the position) each pixel of the first frame and / or the second frame using an optical flow technique, and may align the first frame and / or the second frame to a plurality of consecutive frames of the first frame group using the flow map. According to one embodiment, the optical flow technique may be an AI-based optical flow technique.

[0368] In one embodiment, the warping technique corresponds to a coordinate transformation technique based on an image or video. Warping may refer to, for example, translating an image into another coordinate space using a specific transformation function.

[0369] In one embodiment, the optical flow technique and the warping technique may be used together. For example, the alignment processing unit (3210) may use the warping technique to transform a frame by using the motion information of the image acquired by the optical flow technique. For example, the alignment processing unit (3210) may use the optical flow technique to acquire pixel movement information, and then use the warping technique to perform transformation (e.g., transformation through position adjustment of pixels) on the first frame and / or the second frame based on the movement information, thereby generating a first aligned frame and / or a second aligned frame.

[0370] In one embodiment, the alignment processing unit (3210) may perform processing to align the second frame with a plurality of consecutive frames (e.g., three consecutive frames) of the first frame group, thereby generating a plurality of second aligned frames (e.g., three second aligned frames). In the present disclosure, aligning the second frame with the corresponding frame may mean processing the second frame so that the second frame becomes substantially identical to the corresponding frame.

[0371] According to one embodiment, the alignment processing unit (3210) can generate a plurality of second aligned frames using an optical flow technique and / or a warping technique in the pixel domain. For example, the alignment processing unit (3210) can adjust the positions of pixels within the second frame using the optical flow technique and the warping technique to align (or align) the second frame with a plurality of consecutive frames of the first frame group.

[0372] Through this, for a plurality of identical frames (e.g., three frames) to be restored within a first frame group (or, a first GoP), a plurality of first aligned frames (e.g., three first aligned frames) based on the first frame of the first frame group (or, a first GoP) and a plurality of second aligned frames (e.g., three second aligned frames) based on the second frame of the second frame group (or, a second GoP) can be generated, respectively. The plurality of first aligned frames and the plurality of second aligned frames thus generated can be used together to restore the plurality of frames.

[0373] According to one embodiment, the alignment processing unit (3210) may perform processing to align a first frame of a first frame group (or, first GoP) with a plurality of frames within the first frame group (or, first GoP), thereby obtaining first feature data associated with the plurality of first aligned frames. For example, the alignment processing unit (3210) may obtain the first feature data by using an artificial intelligence model based on an attention mechanism in a feature domain (hereinafter, referred to as an attention-based artificial intelligence model).

[0374] In one embodiment, the alignment processing unit (3210) may perform processing to align a second frame of a second frame group (or a second GoP) following a first frame group (or a first GoP) with a plurality of frames within the first frame group (or a first GoP), thereby obtaining second feature data associated with the plurality of second aligned frames. For example, the alignment processing unit (3210) may obtain the second feature data using an attention-based artificial intelligence model in a feature domain. The first feature data and the second feature data thus obtained may be used (e.g., used by an enhancement network (3220)) for restoration of the corresponding frames, instead of the first aligned frame and the second aligned frame.

[0375] According to one embodiment, the attention-based artificial intelligence model may be an attention-based deep learning network.

[0376] In one embodiment, an attention-based AI model may be a learning method that emphasizes or weights important information in a given input. Such an attention-based AI model helps the network focus more on specific parts of the input and learn more important patterns.

[0377] According to one embodiment, the alignment processing unit (3210) may obtain an attention map through an attention-based artificial intelligence model, and may use the attention map to obtain a similarity map indicating which part of a reference frame (e.g., the first frame or the second frame) is most similar to the first part of the corresponding frame. Based on the similarity map as a weight, the alignment processing unit (3210) may take a method of bringing in the part of the reference frame (e.g., the first frame or the second frame) with the highest similarity the most for alignment processing with the corresponding frame.

[0378] According to one embodiment, the enhancement network (3220) can obtain a plurality of reconstructed frames for a plurality of consecutive frames (e.g., frames t-1, t, t+1) of the first frame group (or, first GoP) based on the first frame of the first frame group and the second frame of the second frame group (or, second GoP). In this way, by reconstructing the decoded frames by using not only the first frame (e.g., I frame) used for encoding (or compressing) other frames within the frame group to which the decoded frames to be reconstructed belong, but also the second frame (e.g., I frame) used for encoding (or compressing) other frames within the next frame group, distortion between frame groups (e.g., between GoPs) can be restored. Therefore, a visually smoother image can be provided compared to an image provided using the decoded frames.

[0379] According to one embodiment, the enhancement network (3220) can generate a plurality of reconstructed frames based on a plurality of first aligned frames, a plurality of second aligned frames, and a plurality of frames. For example, the enhancement network (3220) can input a plurality of first aligned frames, a plurality of second aligned frames, and a plurality of frames as input data to a pre-trained artificial intelligence model, and obtain a reconstructed frame corresponding to each of the plurality of frames as output data from the artificial intelligence model.

[0380] According to one embodiment, the enhancement network (3220) can generate a plurality of restored frames based on the first feature data, the second feature data, and the plurality of frames.

[0381] According to one embodiment, the enhancement network (3220) may be implemented as an artificial intelligence model (hereinafter, referred to as an enhancement model) including multiple artificial neural network layers. The artificial neural network may be, for example, one of a DNN, a CNN, an RNN, an RBM, a DBN, a BRDNN, a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above.

[0382] According to one embodiment, the enhancement model may include an input layer, hidden layer(s), and an output layer. According to one embodiment, the enhancement model may include an input layer, a plurality of convolutional layers, a fully connected layer, and an output layer.

[0383] Meanwhile, as illustrated in FIG. 21, the first frame group described above may correspond to the first GoP, and the second frame group may correspond to the second GoP following the first GoP (e.g., the second GoP immediately following the first GoP, but is not limited thereto). The first frame may correspond to the I frame of the first GoP (e.g., the I frame which is the first frame of the first GoP), and the second frame may correspond to the I frame of the second GoP (e.g., the I frame which is the first frame of the second GoP). Each of the plurality of frames may correspond to a P frame or a B frame of the first GoP.

[0384] Hereinafter, with reference to FIG. 33, the operation of the image receiving device (or the alignment processing unit (3210) of the image receiving device) is exemplarily described.

[0385] According to one embodiment, the alignment processing unit (3210) can obtain an I frame within a first GoP (hereinafter, referred to as a first I frame) and an I frame within a second GoP following the first GoP (hereinafter, referred to as a second I frame). For example, the second GoP may be a GoP immediately following the first GoP, but is not limited thereto. According to one embodiment, the video receiving device can store an I frame within each GoP and / or a list of I frames (intra list) in a memory, and use the stored I frames to restore decoded frames.

[0386] According to one embodiment, the alignment processing unit (3210) may obtain a plurality of decrypted frames to be restored. For example, as illustrated in FIG. 33, the alignment processing unit (3210) may obtain three consecutive P frames belonging to the first GoP (e.g., three consecutive P frames (P frames of t-1, t, t+1)).

[0387] According to one embodiment, the decrypted frames may be frames that have been subjected to frame interpolation processing by a frame interpolation processing module (e.g., a frame interpolation processing module (1010) of FIG. 10) after being decrypted by a decryption unit (e.g., a decryption unit (122) of FIG. 1B).

[0388] In one embodiment, the alignment processing unit (3210) may perform processing to align a first I-frame with a plurality of consecutive frames of a first GoP, thereby generating a plurality of first aligned frames. In the present disclosure, aligning an I-frame with a corresponding P-frame may mean processing the I-frame so that the I-frame becomes substantially identical to the corresponding P-frame. For example, the alignment processing unit (3210) may adjust the positions of pixels within the first I-frame using an optical flow technique and / or a warping technique, thereby transforming the first I-frame into three first aligned frames that are aligned with each of the three P-frames.

[0389] In one embodiment, the alignment processing unit (3210) may perform processing to align the second I frame with a plurality of consecutive frames of the first GoP, thereby generating a plurality of second aligned frames. In the present disclosure, aligning the I frame with the corresponding P frame may mean processing the I frame so that the I frame becomes substantially identical to the corresponding P frame. For example, the alignment processing unit (3210) may transform the second I frame into three second aligned frames that are aligned with each of the three P frames by adjusting the positions of pixels within the second frame using an optical flow technique and / or a warping technique.

[0390] The plurality of first aligned frames (e.g., three first aligned frames) and the plurality of second aligned frames (e.g., three second aligned frames) generated in this manner can be transmitted to an enhancement network together with the plurality of decoded frames (e.g., three decoded frames).

[0391] The enhancement network (3220) can input a plurality of first aligned frames, a plurality of second aligned frames, and a plurality of frames as input data to a pre-trained artificial intelligence model, and obtain a reconstructed frame corresponding to each of the plurality of frames as output data from the artificial intelligence model. In this way, by reconstructing the decoded frames using both the I frames within the GoP to which the decoded frames to be reconstructed belong as well as the I frames within the next (or adjacent) GoP, distortion between GoPs can be restored. Therefore, a visually smoother image can be provided compared to an image provided using decoded P frames.

[0392] FIG. 34 is a flowchart of an image processing method of an image receiving device according to one embodiment of the present disclosure.

[0393] In the embodiment of FIG. 34, the image processing method may be performed by a GoP enhancement processing module (e.g., a GoP enhancement processing module (3110) of FIG. 31) within a post-processing unit (e.g., a post-processing unit (123) of FIG. 1B or a post-processing unit (123a) of FIG. 1C) that operates after the operation of a decoding unit (e.g., a decoding unit (122) of FIG. 1B or a codec processing unit (122a) of FIG. 1C). Accordingly, frames input for image processing of FIG. 34 may correspond to frames decoded by the decoding unit.

[0394] Referring to FIG. 34, an image receiving device (e.g., an image receiving device (120) of FIG. 1a / b / c) can obtain a first frame of a first frame group and a second frame of a second frame group following the first frame group (34010).

[0395] The video receiving device can obtain a plurality of restored frames for a plurality of consecutive frames of the first frame group based on the first frame and the second frame (34020).

[0396] According to one embodiment, the first frame and the second frame may correspond to independently encoded and / or decoded frames, and the plurality of frames may correspond to frames encoded and / or decoded based on the first frame (e.g., predictively encoded and / or decoded frames).

[0397] According to one embodiment, the first frame group may correspond to the first GoP, and the second frame group may correspond to the second GoP immediately following the first GoP. According to one embodiment, the first frame may correspond to the I frame, which is the first frame of the first GoP, and the second frame may correspond to the I frame, which is the first frame of the second GoP. According to one embodiment, each of the plurality of frames may correspond to a P frame or a B frame of the first GoP. According to one embodiment, the number of consecutive plurality of frames may be 3.

[0398] FIG. 35 is a flowchart of an alignment processing operation of an image receiving device according to one embodiment of the present disclosure.

[0399] In the embodiment of FIG. 35, the alignment processing may be performed by an alignment processing unit (e.g., an alignment processing unit (3210) of FIG. 32) within a post-processing unit (e.g., a post-processing unit (123) of FIG. 1B or a post-processing unit (123a) of FIG. 1C) that operates after the operation of a decoding unit (e.g., a decoding unit (122) of FIG. 1B or a codec processing unit (122a) of FIG. 1C). Accordingly, frames input for the alignment processing operation of FIG. 35 may correspond to frames decoded by the decoding unit.

[0400] Referring to FIG. 35, an image receiving device (e.g., an image receiving device (120) of FIG. 1a / b / c) may perform processing to align a first frame of a first frame group (or, a first GoP) with a plurality of frames within the first frame group, thereby generating a plurality of first aligned frames (35010).

[0401] The video receiving device can perform processing to align a second frame of a second frame group (or, second GoP) following a first frame group (or, first GoP) with a plurality of frames of the first frame group, thereby generating a plurality of second aligned frames (35020).

[0402] The video receiving device can generate a plurality of restored frames for the plurality of frames based on the plurality of first aligned frames, the plurality of second aligned frames, and the plurality of frames (35030).

[0403] According to one embodiment, the operation of generating a plurality of first aligned frames and the operation of generating a plurality of second aligned frames can be performed using an optical flow technique and a warping technique in the pixel domain.

[0404] In one embodiment, the image receiving device can transform the first frame into a plurality of first aligned frames that are aligned with each of the plurality of frames by adjusting positions of pixels within the first frame using an optical flow technique and the warping technique to generate a plurality of first aligned frames.

[0405] In one embodiment, the image receiving device can transform the second frame into a plurality of second aligned frames that are aligned with each of the plurality of frames by adjusting positions of pixels within the second frame using an optical flow technique and a warping technique to generate a plurality of second aligned frames.

[0406] According to one embodiment, an image receiving device may input a plurality of first aligned frames, a plurality of second aligned frames, and a plurality of frames as input data to a pre-trained artificial intelligence model to generate a plurality of restored frames, and may obtain a restored frame corresponding to each of the plurality of frames as output data from the artificial intelligence model.

[0407] In one embodiment, the image receiving device may perform processing to align a first frame with the plurality of frames to generate the plurality of reconstructed frames, thereby obtaining first feature data associated with the plurality of first aligned frames, perform processing to align a second frame with the plurality of frames to generate second feature data associated with the plurality of second aligned frames, and generate the plurality of reconstructed frames based on the first feature data, the second feature data, and the plurality of frames.

[0408] According to one embodiment, the operation of acquiring first feature data and the operation of acquiring second feature data can be performed using an artificial intelligence model based on an attention mechanism in a feature domain.

[0409] Meanwhile, the embodiments described above in FIGS. 3 to 16 (e.g., the embodiment related to frame omission / frame interpolation / quality enhancement technology (first embodiment)), the embodiments described above in FIGS. 17 to 28 (e.g., the embodiment related to downsampling / latent vector / SR technology (second embodiment)), and / or the embodiments described above in FIGS. 29 to 35 (e.g., the embodiment related to GoP enhancement (third embodiment)) may be combined with each other within a non-contradictory range. For example, frame omission technology, downsampling technology, and / or latency vector technology may be combined and used together for preprocessing before encoding, and frame interpolation, quality enhancement, SR, and / or GoP enhancement technology may be combined and used together for postprocessing after decoding. For example, frame omission, downsampling, and / or latency vector technology may be combined and used together for preprocessing before compression. For example, GoP enhancement processing may be performed after frame interpolation, quality enhancement, and / or SR processing for post-restoration post-processing. However, these are merely examples, and these techniques may be combined and used in various orders and methods.

[0410] According to one embodiment, a method of an image transmission device may include: obtaining image data including a plurality of image frames; performing preprocessing on the image data; encoding the preprocessed image data to generate encoded image data; and transmitting the encoded image data and information related to the preprocessing.

[0411] According to one embodiment, an image transmission device includes a memory; a communication unit; and at least one processor, wherein the at least one processor is configured to perform an operation of obtaining image data including a plurality of image frames; an operation of performing preprocessing on the image data; an operation of encoding the preprocessed image data to generate encoded image data; and an operation of transmitting the encoded image data and information related to the preprocessing.

[0412] According to one embodiment, the operation of performing the preprocessing may include at least one of the operations of the first embodiment described above with reference to FIGS. 3 to 16. According to one embodiment, the operation of performing the preprocessing may include at least one of the operations of the second embodiment described above with reference to FIGS. 17 to 28. According to one embodiment, the operations of the first embodiment (e.g., the operations of the frame skipping processing module (310)) may be performed before the operations of the second embodiment (e.g., the operations of the downsampling processing module (1710) and / or the latency vector generation processing module (1720)), but are not limited thereto.

[0413] According to one embodiment, a method of an image receiving device may include an operation of acquiring image data; an operation of decrypting the image data; and an operation of performing post-processing on the image data based on information related to pre-processing.

[0414] According to one embodiment, an image receiving device includes a memory; a communication unit; and at least one processor, wherein the at least one processor may be configured to perform an operation of acquiring image data; an operation of decrypting the image data; and an operation of performing post-processing on the image data based on information related to pre-processing.

[0415] According to one embodiment, the operation of performing the post-processing may include at least one of the operations of the first embodiment described above in FIGS. 3 to 16. According to one embodiment, the operation of performing the post-processing may include at least one of the operations of the second embodiment described above in FIGS. 17 to 28. According to one embodiment, the operation of performing the post-processing may include at least one of the operations of the first embodiment described above in FIGS. 3 to 16.

[0416] According to one embodiment, the operations of the first embodiment (e.g., operations of the frame interpolation processing module (1010) and / or the quality enhancement processing module (1020)) may be performed before, but are not limited to, the operations of the second embodiment (e.g., operations of the super resolution processing module (2110)). According to one embodiment, the operations of the second embodiment (e.g., operations of the super resolution processing module (2110)) may be performed before, but are not limited to, the operations of the third embodiment (e.g., operations of the GoP enhancement processing module (3110)).

[0417] FIG. 36 is a diagram illustrating a format for storing metadata of a codec according to one embodiment of the present disclosure.

[0418] According to one embodiment, metadata of a codec (e.g., an AI codec) may include frame skip information and / or data of latency vectors.

[0419] Referring to FIG. 36, the metadata of the codec can be stored in the form of an ISO file format (e.g., ISO BMFF (base media file format)). The ISO file format is compatible with, for example, MP4 or MOV file formats and is a standardized format. The ISO file format is composed of box (or atom) units and has a tree-shaped structure. One box can have a size of at least 8 bytes, the first 4 bytes represent the total size of the box, and the following 4 bytes can store ID (type) information to distinguish each box.

[0420] Table 1 below shows an example of a box in the ISO file format.

[0421] box name (FourCC) description ftyp Basic header of ISO file format mdat Media data itself (mainly compressed), such as video, audio, etc. moov Metadata about stored media data (information needed for media playback)

[0422] Table 2 below shows an example of boxes included in the moov box in Table 1.

[0423] box name (FourCC) description mvhd Basic information about the media (creation date, total playing time, etc.) trak Metadata representing a single media (multiple trak boxes can exist) udta Additional information data stored by the user

[0424] In one embodiment, codec metadata may be included in a trak box. For example, if a media file includes codec metadata and has one video and one audio, a trak box for one video, a trak box for one audio, and a trak box for codec metadata may be included in the moov box, for a total of three trak boxes. However, the embodiment is not limited thereto, and the number of trak boxes for video, audio, and metadata may be set in various ways.

[0425] In one embodiment, a trak box may include one or more boxes. For example, a trak box may include an stsd box that stores codec information required to decode the media information of the corresponding trak. Codec metadata may be included in the stsd box. For example, as illustrated in FIG. 36, a trak box may include an mdia box, the mdia box may include a minf box, the minf box may include an stbl box, the stbl box may include an stsd box and / or an stts box, and the stsd box may include an aicm box that includes codec metadata. However, the names and structures of boxes that include codec metadata may vary.

[0426] In one embodiment, the stts box may include timestamp information. For example, if the metadata of a codec and the video codec require synchronization, timestamp information may be included in the stts box within the trak for the codec metadata and the stts box within the trak for the video to ensure time synchronization of the video data and the metadata. In this case, the timestamp information included in the stts box within each trak may be set to the same value to ensure synchronization of the video data and the codec metadata.

[0427] According to one embodiment, the codec metadata (e.g., AI codec metadata) may include, for example, information related to preprocessing (e.g., frame skip information) and / or latency vector data (e.g., latency vector data (39113) of FIGS. 39a and 39b, latency vector data (40113) of FIGS. 40a and 40b).

[0428] In one embodiment, codec metadata (e.g., AI codec metadata) may be stored in a trak box of an ISO file format. The codec metadata may be stored in a separate trak from the trak for video data and the trak for audio data. For example, the codec metadata may be included in the stsd box within the trak for metadata. For example, the codec metadata may be included in the aicm box within the stsd box within the trak for metadata.

[0429] In one embodiment, the number of trak boxes for codec metadata may be equal to the number of trak boxes for video data.

[0430] According to one embodiment, the value of the timestamp information of the stts box within the trak box for the codec metadata may be set to the same value as the value of the timestamp information of the stts box within the trak box for the corresponding video data.

[0431] According to one embodiment, the video receiving device can decode (or obtain) codec metadata included in a trak for codec metadata and play media data (e.g., video data) using the codec metadata.

[0432] FIGS. 37 and 38 are diagrams illustrating a method of transmitting media data and metadata according to one embodiment of the present disclosure.

[0433] Referring to FIG. 37, media data (e.g., video data) and associated signaling messages (e.g., supplemental enhancement information (SEI) messages) for each frame may be transmitted together.

[0434] According to one embodiment, the data of an SEI message may be transmitted after the video data for the corresponding frame (suffix case). For example, as illustrated in part (a) of FIG. 37, the data (or container) for a first frame (e.g., the i-th frame) may include a first unit including video data of the first frame and a NAL unit header (NUH) and a second unit including data and a NUH of an associated SEI message, and the second unit may follow the first unit. The data (or container) for a second frame (e.g., the i+1-th frame) following the first frame (e.g., the i-th frame) may include a first unit including video data of the second frame and a NAL unit header (NUH) and a second unit including data and a NUH of an associated SEI message, and the second unit may follow the first unit.

[0435] According to one embodiment, the data of an SEI message may be transmitted before the video data for the corresponding frame (prefix case). For example, as illustrated in part (b) of FIG. 37, the data (or container) for a first frame (e.g., the i-th frame) may include a first NAL unit including video data of the first frame and a NAL unit header (NUH) and a second NAL unit including data and a NUH of an associated SEI message, and the second NAL unit may precede the first NAL unit. The data (or container) for a second frame (e.g., the i+1-th frame) following the first frame (e.g., the i-th frame) may include a first NAL unit including video data of the second frame and a NAL unit header (NUH) and a second NAL unit including data and a NUH of an associated SEI message, and the second NAL unit may precede the first NAL unit.

[0436] In one embodiment, metadata of a codec (e.g., an AI codec) may be transmitted together with media data for the corresponding frame (e.g., video data encoded by the codec (codec bitstream)). For example, the codec metadata may be transmitted in a region where data of an SEI message is transmitted. The codec metadata may include, for example, frame skip information and / or latency vector data.

[0437] In one embodiment, codec metadata may be transmitted after video data (e.g., codec bitstream) for the corresponding frame (suffix case). For example, as illustrated in part (a) of FIG. 38, data (or a container) for a first frame (e.g., the i-th frame) may include a first NAL unit containing video data of the first frame (e.g., codec bitstream of the first frame) and a NAL unit header (NUH), and a second NAL unit containing associated codec metadata and a NUH, and the second NAL unit may follow the first NAL unit. Data (or a container) for a second frame (e.g., the i+1-th frame) following a first frame (e.g., the i-th frame) includes a first NAL unit containing video data of the second frame (e.g., a codec bitstream of the second frame) and a NAL unit header (NUH), and a second unit containing associated codec metadata and the NUH, and the second NAL unit may follow the first NAL unit.

[0438] In one embodiment, codec metadata may be transmitted before video data (e.g., codec bitstream) for the corresponding frame (prefix case). For example, as illustrated in part (b) of FIG. 37, data (or a container) for a first frame (e.g., the i-th frame) may include a first NAL unit containing video data of the first frame (e.g., codec bitstream of the first frame) and a NAL unit header (NUH), and a second NAL unit containing associated codec metadata and a NUH, and the second NAL unit may precede the first NAL unit. Data (or a container) for a second frame (e.g., the i+1-th frame) following a first frame (e.g., the i-th frame) includes a first NAL unit containing video data of the second frame (e.g., a codec bitstream of the second frame) and a NAL unit header (NUH), and a second NAL unit containing associated codec metadata and the NUH, and the second NAL unit may precede the first NAL unit.

[0439] According to one embodiment, when the codec metadata is transmitted before the codec bitstream (prefix case), the NAL unit type information of the NAL unit for the codec metadata may be set to a first value (e.g., 39), and when the codec metadata is transmitted after the codec bitstream (suffix case), the NAL unit type information of the NAL unit for the codec metadata may be set to a second value (e.g., 40). The NAL unit type information may be included, for example, in the NUH of the NAL unit for the codec metadata.

[0440] Table 3 below illustrates the syntax of an SEI message that includes codec metadata.

[0441]

[0442] Referring to Table 3, an SEI message may include a payloadType field (information), a payloadSize field (information), and a sei_payload field (information).

[0443] According to one embodiment, the codec metadata (e.g., AI codec metadata) may include, for example, information related to preprocessing (e.g., frame skip information) and / or latency vector data (e.g., latency vector data (39113) of FIGS. 39a and 39b, latency vector data (40113) of FIGS. 40a and 40b).

[0444] In one embodiment, codec metadata may be included in the sei_payload field. When codec metadata is included in the sei_payload field, the payloadType field may be set to a value (e.g., 500) indicating that the SEI message (or the sei_payload field) includes codec metadata, and the payloadSize field may be set to a value indicating the size (e.g., byte size) of the codec metadata included in the sei_payload field.

[0445] Table 4 below illustrates a sei_payload field that contains codec metadata (e.g., AI codec metadata), and Table 5 illustrates codec metadata contained in the sei_payload field.

[0446]

[0447]

[0448] Referring to Table 4, the sei_payload field may include a codec metadata field (e.g., an ai_codec_metadata field) that includes codec metadata (e.g., AI codec metadata).

[0449] Referring to Table 5, the ai_codec_metadata field may include an ai_codec_frame_skip_flag field. The ai_codec_frame_skip_flag field may indicate whether video data of the corresponding frame is skipped. If the ai_codec_frame_skip_flag field indicates that the video data of the corresponding frame is skipped (e.g., ai_codec_frame_skip_flag field = 1), the ai_codec_metadata field may include a metadata_block field. The metadata_block field may include information indicating at least one block that is skipped in the corresponding frame.

[0450] According to one embodiment, the video receiving device can obtain the codec metadata included in the SEI message by decoding the SEI message before or after the codec metadata.

[0451] In one embodiment, when codec metadata is included in an SEI message, the payloadType field of the SEI message may be set to a value (e.g., 500) indicating that the SEI message (or the sei_payload field) includes codec metadata.

[0452] According to one embodiment, the video receiving device can determine whether video data of a corresponding frame is skipped using the value of a flag field (e.g., ai_codec_frame_skip_flag) included in codec metadata included in an SEI message, and can identify a skipped block in the corresponding frame using the value of a metadata_block field.

[0453] FIGS. 39A and 39B are diagrams illustrating a super-resolution procedure according to one embodiment of the present disclosure. In FIGS. 39A and 39B, any descriptions that overlap with those described above in FIGS. 1 to 38 are omitted.

[0454] According to one embodiment, the super resolution procedure may include operations (39100) performed by an image transmitting device (e.g., an image transmitting device (110) of FIGS. 1a / b / c) and operations (39200) performed by an image receiving device (e.g., an image receiving device (120) of FIGS. 1a / b / c).

[0455] Referring to FIG. 39A, a video transmission device can obtain down-sampled video data (39101) (e.g., low-quality / low-capacity video). The video transmission device can use a codec processing unit (39103) (e.g., a codec processing unit (113a) of FIG. 1C) to encode the down-sampled video data (39101), obtain encoded video data (e.g., a codec bitstream), and transmit the encoded video data to a bitstream transmission unit (39115) (e.g., a video output unit (114a) of FIG. 1C). The video transmission device can encode and decode down-sampled video data (39101) using a codec processing unit (39103) (e.g., codec processing unit (113a) of FIG. 1C) to obtain encoded and decoded video data (39105) (e.g., compressed decoded video = decoded video). The video transmission device can perform resolution interpolation on the encoded and decoded video data (39105) to obtain resolution-interpolated video data (39107). The video transmission device can calculate the difference between original video data (39109) and resolution-interpolated video data (39107) and obtain loss data (39110) based on the difference. A video transmission device can generate latency vector data (39113) based on loss data (39110) using a pre-learned model (39111) (e.g., latency encoder) and transmit the latency vector data (39113) to a bitstream transmission unit (39115). The video transmission device can transmit encoded video data (e.g., codec bitstream) and latency vector data (39113) to a video reception device using the bitstream transmission unit (39115). The bitstream transmission unit (39115) can transmit a final bitstream including the encoded video data and the latency vector data in a single container.For example, as illustrated in FIG. 38, latent vector data may be included in an SEI message as codec metadata (e.g., AI codec metadata) and transmitted before or after the encoded video data.

[0456] A video receiving device can receive encoded video data (e.g., a codec bitstream) and latency vector data (39113) using a bitstream receiving unit (39201) (e.g., an image input unit (221a) of FIG. 1C). The bitstream receiving unit (39201) can transmit the latency vector data (39113) to an SR model (39207) and transmit the encoded video data (e.g., a codec bitstream) to a codec processing unit (39203) (e.g., a codec processing unit (122a) of FIG. 1C). The video receiving device can decode the encoded video data using the codec processing unit (39203) to obtain decoded video data (39205). The video receiving device can obtain restored video data (39209) (e.g., restored high-resolution video) using an SR model (39207) based on latent vector data (39113) and decoded video data (39205).

[0457] In the embodiment of Fig. 39b, unlike the embodiment of Fig. 39a, GoP enhancement processing can be performed in a super-resolution procedure. In Fig. 39b, any description that overlaps with the description described above in Fig. 39a is omitted.

[0458] Referring to FIG. 39b, GoP enhancement processing can be performed on encoded and decoded video data (39105) in an image transmitting device. Through this, an image receiving device can obtain GoP enhancement-processed video data (39106), and perform resolution interpolation on the GoP enhancement-processed video data (39106) to obtain resolution-interpolated video data (39107).

[0459] In a video receiving device, GoP enhancement processing can be performed on decoded video data (39205). Through this, the video receiving device can obtain GoP enhancement-processed video data (39206) and use the GoP enhancement-processed video data (39206) as input data for an SR model (39207).

[0460] FIGS. 40A and 40B are diagrams illustrating a frame skipping procedure according to one embodiment of the present disclosure. In FIGS. 40A and 40B, any descriptions that overlap with those described above in FIGS. 1 to 38 are omitted.

[0461] According to one embodiment, the frame skipping procedure may include an operation (40100) performed by an image transmitting device (e.g., an image transmitting device (110) of FIGS. 1a / b / c) and an operation (40200) performed by an image receiving device (e.g., an image receiving device (120) of FIGS. 1a / b / c).

[0462] Referring to FIG. 40A, the video transmission device may obtain original video data (40101) (e.g., original video) including a plurality of frames, and transmit the obtained original video data (40101) to a frame skipping processing unit (40103) (e.g., the frame skipping processing module (310) of FIG. 3). The video transmission device may use the frame skipping processing unit (40103) to skip at least one frame among the plurality of frames or at least one block included in the at least one frame. The frame skipping processing unit (40103) may transmit the skipped video data to a latency vector generation model (40111), and transmit the skipped and remaining video data (40105) to a codec processing unit (40107) (e.g., the codec processing unit (113a) of FIG. 1C). The frame skip processing unit (40103) can transmit frame skip information to the bit stream transmission unit (40115) (e.g., the image output unit (114a) of FIG. 1C). The image transmission device can encode image data (40105) using the codec processing unit (40107) to obtain encoded image data (e.g., a codec bit stream) and transmit it to the bit stream transmission unit (40115). The image transmission device can encode and decode image data (40105) using the codec processing unit (40107) to obtain encoded and decoded image data (40109) (e.g., a compressed and decoded image). The image transmission device can obtain latency vector data (40113) using the latency vector generation model (40111) based on the skipped image data and the encoded / decoded image data (40109). Latency vector data (40113) may include information used to restore skipped frames or skipped block(s). The video transmission device may transmit encoded video data (e.g., codec bitstream), latency vector data (40113), and frame skip information to the video reception device using the bitstream transmission unit (40115).The bit stream transmission unit (40115) can transmit a final bit stream including encoded video data (e.g., codec bit stream), latency vector data (40113), and frame skip information in a single container. For example, as illustrated in FIG. 38, latency vector data and frame skip information can be included in an SEI message as codec metadata (e.g., AI codec metadata) and transmitted before or after the encoded video data.

[0463] An image receiving device can receive encoded image data (e.g., a codec bitstream), frame skip information, and latency vector data (40113) using a bitstream receiving unit (40201) (e.g., an image input unit (221a) of FIG. 1C). The bitstream receiving unit (40201) can transfer the latency vector data (40113) and the frame skip information to a frame interpolation model (40207), and transfer the encoded image data (e.g., a codec bitstream) to a codec processing unit (40203) (e.g., a codec processing unit (122a) of FIG. 1C). The image receiving device can decode and process the encoded image data using the codec processing unit (40203) to obtain decoded video data (40205). An image receiving device can obtain restored image data (40209) (e.g., restored high-resolution video) using a frame interpolation model (40207) based on frame skip information, latency vector data (40202), and decoded image data (40205). The frame interpolation model (40207) can be used to restore an original image through frame interpolation.

[0464] In the embodiment of Fig. 40b, unlike the embodiment of Fig. 40a, GoP enhancement processing can be performed in a frame skipping procedure. In Fig. 40b, any description that overlaps with the description described above in Fig. 40a is omitted.

[0465] Referring to FIG. 40b, GoP enhancement processing can be performed on encoded and decoded image data (40105) in an image transmission device. Through this, an image reception device can obtain GoP enhancement-processed image data (40106) and use the GoP enhancement-processed video data (40106) as input data for a latency vector generation model (40111).

[0466] In an image receiving device, GoP enhancement processing can be performed on decoded image data (40205). Through this, the image receiving device can obtain GoP enhancement-processed image data (40206) and use the GoP enhancement-processed image data (40206) as input data for a latency vector generation model (40111).

[0467] In the specific embodiments of the present disclosure described above, components included in the present disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0468] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

Claims

1. In a method performed by a video transmission device, An operation of obtaining image data including multiple image frames; An action for performing preprocessing on the above image data; An operation of encoding the preprocessed image data to generate encoded image data; and A method comprising an operation of transmitting the encoded image data and information related to the preprocessing.

2. In paragraph 1, The actions that perform the above preprocessing are: An operation of determining whether to apply frame skipping to a first image frame included in the plurality of image frames using an artificial intelligence model; and If it is determined that frame skipping is to be applied to the first image frame, an operation of skipping the first image frame is included. The above encoding operation is: The above first image frame includes an operation of encoding skipped image data, A method wherein the information related to the above preprocessing includes information related to the frame skipping.

3. In paragraph 1, The actions that perform the above preprocessing are: Further comprising an operation of selecting the artificial intelligence model from a plurality of artificial intelligence models based on parameters for encoding the image data, The operation of encoding image data in which the first image frame is skipped is: An operation of encoding image data in which the first image frame is skipped using the parameters for the encoding is included, A method wherein the parameters for the above encoding are associated with the compression ratio of the image data.

4. In paragraph 1, The artificial intelligence model is set to output a distance associated with the difference between a first distortion and a second distortion at the same bit rate, wherein the first distortion is associated with a first image frame set to which frame skipping is applied for the first image frame, and the second distortion is associated with a second image frame set to which frame skipping is not applied for the first image frame. A method wherein the first image frame set includes a second image frame that is a previous frame of the first image frame and a third image frame that is a subsequent image frame of the first image frame, and the second image frame set includes the first image frame, the second image frame, and the third image frame.

5. In paragraph 4, The operation for determining whether to apply frame skipping to the first image frame is: An operation of encoding and decoding the first image frame, the second image frame being a frame preceding the first image frame, and the third image frame being a frame following the first image frame; An operation of inputting the encoded and decoded first image frame, the encoded and decoded second image frame, and the encoded and decoded third image frame as input data to the artificial intelligence model, and obtaining the distance as output data of the artificial intelligence model; and An operation for determining whether to apply frame skipping to the first image frame based on the distance, The above distance corresponds to the value obtained by subtracting the second distortion from the first distortion, Based on the above distance, the operation of determining whether to apply frame skipping to the first image frame is: If the above distance is negative, determine to apply frame skipping to the first image frame, A method for determining not to apply frame skipping to the first image frame if the distance is positive.

6. In paragraph 3, The above artificial intelligence model is trained based on multiple learning data sets, Each of the plurality of learning data sets is obtained based on the performance of the following operations: obtaining an image frame set; performing a first image processing operation on the image frame set for each of settable values ​​of the parameter for encoding to obtain first rate distortion data; performing a second image processing operation on the image frame set to which the frame skipping is applied for the value of the target parameter for encoding to obtain second rate distortion data; and performing an operation of obtaining a distance based on the first rate distortion data and the second rate distortion data. A method wherein the first image processing includes encoding processing and decoding processing, and the second image processing includes encoding processing, decoding processing, frame interpolation processing, and quality enhancement processing.

7. In paragraph 1, The actions that perform the above preprocessing are: An operation of downsampling the above image data to obtain downsampled image data; An operation of encoding and decoding the above down-sampled image data; An operation of performing resolution interpolation on the encoded and decoded image data to obtain resolution interpolated image data; An operation of obtaining loss data based on the image data and the resolution-interpolated image data; and Based on the above loss data, an operation of generating latent vector data using an artificial intelligence model is included. The above encoding operation is: Includes an operation of encoding the above downsampled image data together, The above transmitting action is: A method comprising an operation of transmitting the encoded image data and the latent vector data.

8. In paragraph 7, The actions to obtain the above loss data are: An operation of obtaining the loss data by calculating a difference in a preset unit for a frame pair consisting of a first frame included in the image data and a second frame corresponding to the first frame included in the resolution-interpolated image data, The above preset unit corresponds to a pixel unit, the method.

9. In a method performed by an image receiving device, An operation of obtaining received data including encoded image data and information related to preprocessing of the image data; An operation of decrypting the above image data; and A method comprising an operation of performing post-processing on the decrypted image data based on information related to the preprocessing.

10. In paragraph 9, The information related to the above preprocessing includes information related to frame skipping, The actions that perform the above post-processing are: An operation of determining whether to apply frame interpolation to a plurality of image frames included in the image data based on information related to the frame skipping; When it is determined to apply frame interpolation to the plurality of image frames, an operation of obtaining image data in which the first image frame is interpolated using a first artificial intelligence model based on the plurality of image frames; and A method comprising: obtaining enhanced image data using a second artificial intelligence model based on image data interpolated from the first image frame.

11. In paragraph 10, The information related to the frame skipping is set to either a first value indicating that frame skipping is applied to the first image frame or a second value indicating that frame skipping is not applied to the first image frame, The actions that determine whether to apply the above frame interpolation are: When the information related to the above frame skipping is set to the first value, it is determined to apply frame interpolation to the plurality of image frames, If the information related to the above frame skipping is set to the second value, it is determined not to apply frame interpolation to the plurality of image frames, The plurality of image frames include a second image frame which is a previous image frame of the first image frame and a third image frame which is a subsequent image frame of the first image frame, The operation of obtaining image data in which the first image frame is interpolated using the first artificial intelligence model is: An operation of inputting the second image frame and the third image frame as input data into the first artificial intelligence model; An operation of obtaining image data in which the first frame is interpolated as output data from the first artificial intelligence model is included. A method in which the above first artificial intelligence model is learned based on a data set obtained through an algorithm using similarity that takes into account the time axis of the image.

12. In paragraph 9, The above received data includes latent vector data, The above post-processing operation includes an operation of generating image data with improved resolution using an artificial intelligence model based on the latent vector data. The above latency vector data is generated based on loss data used for restoration of decoded image data, and the loss data is obtained by calculating a difference of a preset unit for a pair of associated frames, The above preset unit corresponds to a pixel unit, the method.

13. In paragraph 9, The actions that perform the above post-processing are: An operation of obtaining a first frame of a first frame group included in the image data and a second frame of a second frame group following the first frame group; and An operation of obtaining a plurality of reconstructed frames for a plurality of consecutive frames of the first frame group based on the first frame and the second frame, A method wherein the first frame and the second frame correspond to frames that are independently encoded and decoded, and the plurality of frames correspond to frames that are predictively encoded and decoded based on the first frame.

14. In paragraph 13, The first frame group corresponds to the first GoP (group of pictures), and the second frame group corresponds to the second GoP immediately following the first GoP. The first frame corresponds to an I frame (intra-coded frame), which is the first frame of the first GoP, and the second frame corresponds to an I frame, which is the first frame of the second GoP. A method wherein each of the plurality of frames corresponds to a P frame (predictive-coded frame) or a B frame (bi-predictive-coded frame) of the first GoP.

15. In paragraph 13, The operation of generating the above multiple restored frames is: An operation of performing a process to align the first frame with the plurality of frames, thereby obtaining first feature data associated with the plurality of first aligned frames; An operation of performing a process to align the second frame with the plurality of frames, thereby generating second feature data associated with the plurality of second aligned frames; and A method comprising an operation of generating a plurality of restored frames based on the first feature data, the second feature data, and the plurality of frames.

Citation Information

Patent Citations

  • Content-adaptive application of fixed transfer function to high dynamic range (HDR) and / or wide color gamut (WCG) video data

    KR1020180016383A

  • Signal processing apparatus for weather radar with noise filtering function using external information

    KR1020240057161A

  • Apparatus for compressing image, learning apparatus and method thereof

    KR102245682B1

  • System and method for neviagating moving object

    KR102719267B1

  • Video compression with generative models

    US20200244969A1