Systems and methods for learned image compression with pre-processing

Abstract

A mechanism for processing video data is disclosed. The mechanism determines to apply a pre-processing function to the visual media data as part of an image compression framework. The conversion between the visual media data and the bitstream may then be performed based on the image compression framework and the corresponding pre-processing function.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefits of international patent application PCT / CN2022 / 104313 filed on July 7, 2022. The entire contents of the above patent application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the generation, storage and consumption of digital audio video media information in a file format. Background Art

[0004] Digital video accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demands for digital video usage are likely to continue to grow. Summary of the invention

[0005] A first aspect relates to a method for processing video data, comprising: determining to apply a pre-processing function to visual media data as part of an image compression framework; and performing conversion between the visual media data and a bitstream based on the image compression framework.

[0006] Optionally, in any of the above aspects, another implementation of this aspect provides that the pre-processing function receives a current image of size W0xH0xC0, where W0 is the input width, H0 is the input width and C0 is the number of input channels, and wherein the pre-processing function outputs the current image in size W1xH1xC1, where W1 is the output width, H1 is the output height and C1 is the number of output channels.

[0007] Optionally, in any of the above aspects, another implementation of this aspect provides that W0 is equal to W1, H0 is equal to H1, and C0 is equal to C1.

[0008] Optionally, in any of the above aspects, another implementation of this aspect provides that C0 is not equal to C1.

[0009] Optionally, in any of the above aspects, another implementation of this aspect provides that W0 is not equal to W1 or H0 is not equal to H1.

[0010] Optionally, in any of the above aspects, another implementation of this aspect provides that the preprocessing function includes a deep neural network using parameters determined by deep learning.

[0011] Optionally, in any of the above aspects, another implementation of this aspect provides that the preprocessing function includes a convolution layer with a kernel size of 3x3, 4x5 or 7x7, and wherein the convolution layer uses an expanded convolution with an expansion parameter set to 3, 5 or 7.

[0012] Optionally, in any of the above aspects, another implementation of this aspect provides that the preprocessing function includes a residual block, wherein the residual block includes a convolutional layer, an activation layer, a batch normalization layer, or a combination thereof.

[0013] Optionally, in any of the above aspects, another implementation of this aspect provides that the pre-processing function includes a wobble converter, wherein the number of heads of the wobble converter is two or six, and wherein the depth of the wobble converter is set to two or six.

[0014] Optionally, in any of the above aspects, another implementation of this aspect provides that the pre-processing function includes a dense connection structure.

[0015] Optionally, in any of the above aspects, another implementation of this aspect provides an image compression framework including a compressor, and wherein the preprocessing function is jointly optimized with the compressor.

[0016] Optionally, in any of the above aspects, another implementation of this aspect provides that the compressor is a learning-based variational autoencoder.

[0017] Optionally, in any of the above aspects, another implementation of this aspect provides that the learning-based variational autoencoder is partially trained before training the preprocessing function, and wherein the learning-based variational autoencoder and the preprocessing function are jointly trained.

[0018] Optionally, in any of the above aspects, another implementation of this aspect provides that the compressor is a proxy autoencoder with fixed model parameters.

[0019] Optionally, in any of the above aspects, another implementation of this aspect provides that the pre-processing function includes multiple models for different codec bit rates.

[0020] Optionally, in any of the above aspects, another implementation of this aspect provides that the pre-processing function is configured to adapt to different bit rates based on quantization parameters, codec control parameters or a combination thereof.

[0021] Optionally, in any of the above aspects, another implementation of this aspect provides that the proxy autoencoder includes a prediction function, a transformation function, an inverse transformation function, a quantization function and a rate estimation function.

[0022] Optionally, in any of the above aspects, another implementation of this aspect provides that the proxy autoencoder includes a restoration neural network.

[0023] Optionally, in any of the above aspects, another implementation of this aspect provides that the compressor includes a hybrid codec, which also includes a codec and a learning-based function, and wherein the learning-based function includes a learning-based loop filter, learning-based intra-frame prediction, learning-based inter-frame prediction, learning-based quantization or a combination thereof.

[0024] A second aspect relates to an apparatus for processing video data, comprising: a processor; and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform any of the aforementioned aspects.

[0025] A third aspect relates to a non-transitory computer-readable medium, comprising a computer program product for use with a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs the method described in any one of the preceding aspects.

[0026] A fourth aspect relates to a non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method includes: determining to apply a preprocessing function to visual media data as part of an image compression framework; and generating a bitstream based on the determination.

[0027] A fifth aspect relates to a method for storing a bitstream of a video, comprising: determining to apply a preprocessing function to visual media data as part of an image compression framework; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0028] For the sake of clarity, any of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0029] These and other features will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0031] Figure 1 An example framework for preprocessing-based image compression is shown.

[0032] Figure 2 An example structure of a pre-processing module is shown.

[0033] Figure 3is a block diagram showing an example video processing system.

[0034] Figure 4 is a block diagram of an example video processing device.

[0035] Figure 5 is a flow chart of an example method of video processing.

[0036] Figure 6 is a block diagram illustrating an example video encoding and decoding system.

[0037] Figure 7 is a block diagram illustrating an example encoder.

[0038] Figure 8 is a block diagram illustrating an example decoder.

[0039] Fig. 9 is a schematic diagram of an example encoder. DETAILED DESCRIPTION

[0040] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should not be limited in any way to the illustrative implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims and the full scope of their equivalents.

[0041] The section headings used in this document are for ease of understanding, and do not limit the applicability of the techniques and embodiments disclosed in each section to only that section. In addition, the techniques described herein are also applicable to other video codec protocols and designs.

[0042] 1. Preliminary Discussion

[0043] This document relates to image and / or video processing techniques. Specifically, it is related to the design of algorithms for image preprocessing and compression. These concepts can be applied alone or in various combinations to any image and / or video codec system or part of the encoding and decoding process.

[0044] 2. Further Discussion

[0045] Image compression plays an important role in dealing with the ever-increasing amount of image data. Literature and industry are devoted to improving the compression efficiency. Lossy image compression standards include Joint Photographic Experts Group (JPEG) version 2000 (JPEG2000), Better Portable Graphics (BPG), and Versatile Video Coding (VVC), which are all developed based on a block-by-block hybrid coding framework. This coding scheme includes advanced prediction modules, two-dimensional (2D) transforms, scalar quantization, arithmetic entropy coding and decoding, and loop filters, which effectively eliminate redundant information in images.

[0046] Thanks to the development of deep learning, learning-based strategies can be used in conjunction with image compression. Variational autoencoders (VAEs) can achieve beneficial compression performance compared to standard approaches for encoding and decoding, demonstrating the compression potential of end-to-end optimized encoding and decoding schemes. An end-to-end compression framework may involve convolution operations and generalized divisive normalization (GDN), such that the visual signal is linearly and nonlinearly transformed into latent codes and synthesized back into the reconstructed visual signal. The latent codes are compressed into a bitstream via entropy encoding and decoding. The network can be optimized in a global end-to-end manner to minimize the rate and distortion costs. A range of works can improve the rate-distortion performance of end-to-end image compression. Advanced probability estimation and entropy encoding and decoding schemes such as hyper-prior models, joint autoregressive models, and / or entropy estimation of Gaussian mixture models can be used. Visual transformer networks can be applied to visual tasks such as segmentation and restoration.

[0047] 3. Technical problems solved by the disclosed technical solutions

[0048] Compression inevitably introduces artifacts, especially in low bitrate codec scenarios. A body of work has been devoted to investigating artifact removal strategies as post-processing to achieve better reconstruction quality. Fewer works have been devoted to pre-processing before compression. The advantage of pre-processing is that the modifications are applied on the encoder side while keeping the decoder unchanged. Pre-processing is able to change the input according to the bitrate budget or codec behavior, which is expected to improve the overall rate-distortion performance. Therefore, it may be beneficial to design pre-processing schemes for image and / or video compression.

[0049] 4. List of solutions and implementation examples

[0050] In order to solve the above problems, the methods summarized as follows are disclosed. These embodiments should be regarded as examples to explain the general concept and should not be interpreted narrowly. In addition, these embodiments can be applied alone or in combination in any way. In the following description, the term "image compression" can refer to any variation of the signal processing method that compresses or processes the current input. The input image and / or video includes but is not limited to screen content and natural content. In order to solve the above problems, one or more of the following methods are disclosed.

[0051] Example 1

[0052] In one example, the image compression framework introduces a preprocessing module. The input of the preprocessing module is an image of size W0xH0xC0, where W0 and H0 are the width and height of the image, respectively, and C0 represents the number of channels. The output of the preprocessing module includes features or signals of the current image of size W1xH1xC1.

[0053] Example 2

[0054] In one example, W0, H0, and C0 are equal to W1, H1, and C1, respectively. In one example, C0 and C1 are equal to 3. In one example, C0 and C1 are equal to 1.

[0055] Example 3

[0056] In one example, W0 and H0 are equal to W1 and H1, respectively. C0 is not equal to C1. In one example, C0 is equal to 3, and C1 is greater than or less than C0. In one example, C0 is equal to 1, and C1 is greater than C0.

[0057] Example 4

[0058] In one example, W0 and / or H0 may not be equal to W1 and / or H1. In one example, W1=W0 / s, and H1=H0 / s. C1=s×s×C0. In one example, s is equal to 2. In one example, s is equal to 4. In one example, W1=W0 / s, and H1=H0 / s. C1=C0. In one example, s is equal to 2. In one example, s is equal to 4.

[0059] Example 5

[0060] In one example, the output of the pre-processing module may include features with different spatial dimensions.

[0061] Example 6

[0062] In one example, the preprocessing module may include a deep neural network, and the network parameters are obtained through learning.

[0063] Example 7

[0064] In one example, the convolution layer is included in the preprocessing module. In one example, the kernel size can be 3x3 and / or 5x5 and / or 7x7. In one example, dilated convolution can be used. In one example, multiple dilated convolutions can be applied, where the dilation parameters are set to 3, 5, and 7.

[0065] Example 8

[0066] In one example, the residual block is included in the preprocessing module. In one example, the residual block may include a convolutional layer, an activation layer, and a batch normalization layer. In one example, the batch normalization layer may be excluded from the residual block.

[0067] Example 9

[0068] In one example, a swin-transformer is included in the pre-processing module. In one example, the number of heads of the swin-transformer is 2, and the depth is set to 2. In one example, the number of heads of the swin-transformer is 6, and the depth is set to 6.

[0069] Example 10

[0070] In one example, the densely connected structure is included in the pre-processing module.

[0071] Example 11

[0072] In one example, a mixture of transformer networks, such as wiggling transformers and convolutional layers, are included in the pre-processing module.

[0073] Example 12

[0074] In one example, the network parameters of the preprocessing module can be obtained from another network, such as a visual geometry group network (vgg-net), a residual neural network (res-net), etc.

[0075] Example 13

[0076] In one example, the pre-processing module can be jointly optimized with the compressor.

[0077] Example 14

[0078] In one example, the compressor can be a learning-based variational autoencoder. In one example, the preprocessing module and the variational autoencoder can be jointly trained from scratch. In one example, the variational autoencoder can be trained first. Then, the preprocessing module and the variational autoencoder are jointly trained. In one example, the variational autoencoder can be trained first. Then, the preprocessing module and the variational autoencoder are jointly trained. Finally, the preprocessing module is fixed and the variational autoencoder is fine-tuned.

[0079] In one example, a proxy autoencoder with fixed model parameters is used to assist in the training of the preprocessing network. In one example, multiple preprocessing modules can be provided to accommodate different codec bit rates. In one example, only one preprocessing module is used, which is capable of accommodating different codec bit rates. In one example, a quantization parameter map can be used to guide preprocessing. In one example, codec control parameters can be used to guide preprocessing.

[0080] Example 15

[0081] In one example, the compressor may be a codec such as JPEG, JPEG2000, High Efficiency Video Coding (HEVC), VVC, Audio Video Standard (AVS) Version Three (AVS3), AVS Version Two (AVS2), Advanced Video Coding (AVC) also known as H.264, and Open Media Video Alliance 1 (AV1). In one example, a proxy codec may be used when training a preprocessing module that mimics the behavior of other codecs. The proxy codec may include a prediction module, a transform and inverse transform module, a quantization module, and a rate estimation module. In one example, the transform is differentiable, which may be implemented by discrete cosine transform (DCT), discrete sine transform (DST), and / or wavelet domain transform. In one example, quantization may be implemented with a k-order polynomial approximation. In one example, the rate estimation module may employ the L1 norm, L2 norm, and L0 norm of the coefficients. In one example, the L1 norm, L2 norm, and L0 norm of the coefficients are linearly combined to estimate the codec bits. In one example, a deformation of the L0 norm of the coefficients is used for rate estimation.

[0082] Example 16

[0083] In one example, the restoration neural network can act as a proxy module that imitates the output of other codecs to train the preprocessing module. The proxy module can be attached after the preprocessing module. In addition, a differentiable rate estimation module can be adopted. In one example, the proxy module is trained by minimizing the distance between the reconstruction of the traditional codec and the output of the proxy module. In one example, the distance can be measured by the mean squared error. In one example, the distance can be measured by the mean absolute error. In one example, the distance can be measured in the feature domain, such as the perceptual loss measured according to the visual geometry group (VGG) neural network loss (VGG loss). In one example, the distance can be measured by the structural similarity index (SSIM) loss.

[0084] In one example, the proxy module is fixed and only the preprocessing module is trained by minimizing the distance between the original image and the output image of the proxy module. In one example, the distance can be measured by mean squared error. In one example, the distance can be measured by mean absolute error. In one example, the distance can be measured in a feature domain, such as a perceptual loss (e.g., VGG loss). In one example, the distance can be measured by SSIM loss.

[0085] In one example, the proxy module is fixed and only the preprocessing module uses rate loss and should be trained to minimize the distance between the original image and the output image of the proxy module.

[0086] Example 17

[0087] In one example, the compressor can be a hybrid codec combined with a learning-based module, such as a learning-based loop filter, learning-based intra-frame prediction, learning-based inter-frame prediction, learning-based quantization, etc. In one example, a proxy codec can be used when training the preprocessing module, which imitates the behavior of the hybrid codec. The proxy codec can include a prediction module, a transform and inverse transform module, a quantization module, and a rate estimation module. In one example, the transform should be differentiable, which can be achieved by DCT, DST and / or wavelet domain transform. In one example, quantization can be achieved with k-order polynomial approximation. In one example, the rate estimation module can use the L1 norm, L2 norm, and L0 norm of the coefficients. In one example, the L1 norm, L2 norm, and L0 norm of the coefficients are linearly combined to estimate the codec bits. In one example, the deformation of the L0 norm of the coefficient is used for rate estimation.

[0088] Example 18

[0089] In one example, a proxy codec can be used when training a preprocessing module, which can additionally include a learning-based in-loop filter, a learning-based intra-frame prediction, a learning-based inter-frame prediction, and a learning-based quantization, etc. In one example, a learning-based codec module (such as a learning-based in-loop filter, a learning-based intra-frame prediction, a learning-based inter-frame prediction, and a learning-based quantization) can be jointly optimized with a preprocessing module. In one example, a learning-based codec module (such as a learning-based in-loop filter, a learning-based intra-frame prediction, a learning-based inter-frame prediction, and a learning-based quantization) can be pre-trained first and then jointly trained with a preprocessing module. In one example, a preprocessing module can first be trained with a proxy. Then, a learning-based codec module can be jointly trained with a preprocessing module.

[0090] 5. Example Embodiments

[0091] An example of preprocessing for image compression is shown below. The preprocessing module is integrated with the image compression framework, where the compressor can be a learning-based codec or a traditional codec. The input can be a color picture, frame and / or video with three channels, such as red, green, blue (RGB) and / or luminance and chrominance (YUV), or a single channel picture, frame and / or image.

[0092] A first exemplary embodiment will now be described.

[0093] Figure 1 An example framework for preprocessing-based image compression is shown. Figure 2An example structure of the preprocessing module is shown. In this embodiment, the preprocessing module based on the swing transformer is used in conjunction with the end-to-end image compression framework. Specifically, the input of the preprocessing module is a color image x with three channels (RGB or YUV), and the output of the preprocessing module is x′ with three channels. The reconstruction is represented as The overall workflow is as follows Figure 1 As shown in Figure 1, a variant autoencoder is used for image compression. The framework of the preprocessing module is as follows: Figure 2 shown.

[0094] The loss function can be formulated as follows:

[0095]

[0096] Where λ is the Lagrangian parameter of the weight between the balance rate and distortion. R(y′) represents the rate of compression y′, represents x and The distance between them can be measured by the mean square error. D2(x,x′) represents the distance between x and x′, which can be measured by the mean square error. The preprocessing module and the image compressor can be jointly optimized by minimizing L, thereby improving the rate-distortion performance.

[0097] A second exemplary embodiment will now be described.

[0098] In this embodiment, the pre-processing module based on the swing transformer is used together with the end-to-end image compression frame, such as Figure 1 The preprocessing module based on the swing converter is shown in Figure 2 As shown. Specifically, the input of the preprocessing module is a color image x with three channels (RGB or YUV), and the output of the preprocessing module is x′ with N channels. The loss function can be formulated as follows:

[0099]

[0100] Where λ is the Lagrangian parameter of the weight between the balance rate and distortion. R(y′) represents the rate of compression y′, represents x and The distance between them can be measured by the mean squared error. D3(p(x),p(x′)) represents the perceptual distance between p(x) and p(x′), where p is the deep feature extractor. The preprocessing module and the image compressor can be jointly optimized by minimizing L, thereby improving the rate-distortion performance.

[0101] Figure 34000 is a block diagram illustrating an example video processing system 4000 in which various embodiments disclosed herein may be implemented. Various implementations may include some or all of the components of system 4000. System 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8-bit or 10-bit multi-component pixel values), or may be received in a compressed or encoded format. Input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a passive optical network (PON), and wireless interfaces such as Wi-Fi or a cellular interface.

[0102] System 4000 may include a codec component 4004 that can implement various codecs or encoding methods described in this document. Codec component 4004 can reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 can be stored or transmitted via connected communication, as represented by component 4006. The bitstream (or codec) representation of the stored or transmitted video received at input 4002 can be used by component 4008 to generate pixel values ​​or send to display interface 4010. The process of generating a user-viewable video from a bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that the codec tool or operation is used at the encoder, and the corresponding decoding tool or operation of the inverse codec result will be performed by the decoder.

[0103] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Display Port, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, etc. The embodiments described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0104] Figure 44106 is a block diagram of an example video processing device 4100. Device 4100 can be used to implement one or more methods described herein. Device 4100 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. Device 4100 may include one or more processors 4102, one or more memories 4104, and video processing circuitry 4106. Processor 4102 may be configured to implement one or more methods described in this document. Memory (multiple memories) 4104 may be used to store data and code for implementing the methods and embodiments described herein. Video processing circuitry 4106 may be used to implement some embodiments described in this document in hardware circuits. In some embodiments, video processing circuitry 4106 may be at least partially included in processor 4102, such as a graphics coprocessor.

[0105] Figure 5 4200. The method 4200 includes determining to apply a pre-processing function to visual media data as part of an image compression framework at step 4202. At step 4204, conversion between the visual media data and a bitstream is performed based on the image compression framework. According to an example, the conversion step 4204 may include encoding at an encoder or decoding at a decoder.

[0106] It should be noted that method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In this case, the instructions, when executed by the processor, cause the processor to perform method 4200. In addition, method 4200 may be performed by a non-transitory computer-readable medium, the medium comprising a computer program product for use by a video codec device. The computer program product comprises computer executable instructions stored on a non-transitory computer-readable medium, such that when executed by the processor, the video codec device performs method 4200.

[0107] Figure 6 43 is a block diagram illustrating an example video codec system 4300 that can utilize embodiments of the present disclosure. The video codec system 4300 may include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data, which may be referred to as a video encoding device. The target device 4320 may decode the encoded video data generated by the source device 4310, which may be referred to as a video decoding device.

[0108] Source device 4310 may include video source 4312, video encoder 4314 and input / output (I / O) interface 4316. Video source 4312 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources. Video data may include one or more pictures. Video encoder 4314 encodes video data from video source 4312 to generate a bit stream. The bit stream may include a bit sequence that forms a codec representation of video data. The bit stream may include codec pictures and related data. Codec pictures are codec representations of pictures. Related data may include sequence parameter sets, picture parameter sets and other syntax structures. I / O interface 4316 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be directly transmitted to target device 4320 via network 4330 via I / O interface 4316. Encoded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0109] Target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. The I / O interface 4326 may include a receiver and / or a modem. The I / O interface 4326 may obtain coded video data from source device 4310 or storage medium / server 4340. The video decoder 4324 may decode the coded video data. The display device 4322 may display the decoded video data to a user. The display device 4322 may be integrated with the target device 4320, or may be outside the target device 4320 that may be configured to interface with an external display device.

[0110] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or additional standards.

[0111] Figure 7 is a block diagram showing an example of a video encoder 4400, which may be Figure 6 4300. The video encoder 4314 in the system 4300 shown in . The video encoder 4400 can be configured to perform any or all embodiments of the present disclosure. The video encoder 4400 includes multiple functional components. The embodiments described in the present disclosure can be shared between various components of the video encoder 4400. In some examples, the processor can be configured to perform any or all embodiments described in the present disclosure.

[0112] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402, a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a buffer 4413 and an entropy coding unit 4414, and the prediction unit may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405 and an intra-frame prediction unit 4406.

[0113] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In one example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0114] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405, may be highly integrated but are represented separately in the example of the video encoder 4400 for purposes of explanation.

[0115] The segmentation unit 4401 may segment a picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0116] The mode selection unit 4403 may select one of the coding modes (intra or inter), for example, based on the error result, and provide the resulting intra or inter coded block to the residual generation unit 4407 to generate residual block data and to the reconstruction unit 4412 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 4403 may select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 4403 may also select a resolution of the motion vector for the block (e.g., sub-pixel or integer pixel precision).

[0117] In order to perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information of the current video block by comparing one or more reference frames from the buffer 4413 with the current video block. The motion compensation unit 4405 may determine a predicted video block of the current video block based on the motion information of pictures other than the picture associated with the current video block from the buffer 4413 and decoded samples.

[0118] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0119] In some examples, the motion estimation unit 4404 may perform unidirectional prediction for the current video block, and the motion estimation unit 4404 may search the reference pictures of list 0 or list 1 to find the reference video block of the current video block. The motion estimation unit 4404 may then generate a reference index indicating the reference picture containing the reference video block in list 0 or list 1 and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0120] In other examples, the motion estimation unit 4404 may perform bidirectional prediction for the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference pictures in list 0 and may also search for another reference video block of the current video block in the reference pictures in list 1. Then, the motion estimation unit 4404 may generate a reference index indicating a reference picture containing a reference video block in list 0 and list 1 and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0121] In some examples, motion estimation unit 4404 may output a complete set of motion information for use in a decoding process by a decoder. In some examples, motion estimation unit 4404 may not output a complete set of motion information for the current video. Instead, motion estimation unit 4404 may reference motion information of another video block to signal motion information of the current video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of an adjacent video block.

[0122] In one example, the motion estimation unit 4404 may indicate, in a syntax structure associated with the current video block, a value that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0123] In another example, the motion estimation unit 4404 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0124] As discussed above, the video encoder 4400 can predictively signal motion vectors.Two examples of predictive signaling techniques that can be implemented by the video encoder 4400 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0125] The intra prediction unit 4406 may perform intra prediction on the current video block. When the intra prediction unit 4406 performs intra prediction on the current video block, the intra prediction unit 4406 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0126] The residual generating unit 4407 may generate residual data of the current video block by subtracting the predicted video block of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample point components of the samples in the current video block.

[0127] In other examples, there may be no residual data for the current video block, such as in skip mode, and the residual generation unit 4407 may not perform a subtraction operation.

[0128] The transform processing unit 4408 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0129] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0130] The inverse quantization unit 4410 and the inverse transform unit 4411 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 4412 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in the buffer 4413.

[0131] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.

[0132] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0133] Figure 8 is a block diagram showing an example of a video decoder 4500, which may be Figure 6 4300. The video decoder 4500 may be configured to perform any or all of the embodiments of the present disclosure. In the example shown, the video decoder 4500 includes multiple functional components. The embodiments described in the present disclosure may be shared between various components of the video decoder 4500. In some examples, the processor may be configured to perform any or all of the embodiments described in the present disclosure.

[0134] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 may perform a decoding process that is substantially the opposite of the encoding process described with respect to the video encoder 4400.

[0135] The entropy decoding unit 4501 may retrieve a coded bitstream. The coded bitstream may include entropy-coded video data (e.g., coded video data blocks). The entropy decoding unit 4501 may decode the entropy-coded video data, and based on the entropy-decoded video data, the motion compensation unit 4502 may determine motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. For example, the motion compensation unit 4502 may determine such information by performing AMVP and merge mode.

[0136] The motion compensation unit 4502 may generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter for use with sub-pixel precision may be included in a syntax element.

[0137] The motion compensation unit 4502 may calculate interpolation of sub-integer pixels of a reference block using an interpolation filter used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filter used by the video encoder 4400 according to received syntax information and use the interpolation filter to generate a prediction block.

[0138] The motion compensation unit 4502 can use some syntax information to determine the size of blocks used to encode frames and / or slices of the encoded video sequence, partitioning information describing how each macroblock of the pictures of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame codec block, and other information used to decode the encoded video sequence.

[0139] The intra prediction unit 4503 may form a prediction block from spatially neighboring blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes, i.e., dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0140] The reconstruction unit 4506 may add the residual block to the corresponding prediction block generated by the motion compensation unit 4502 or the intra prediction unit 4503 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra prediction and also produces decoded video for presentation on a display device.

[0141] Fig. 9 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing techniques for VVC. The encoder 4600 includes three in-loop filters, namely a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602, which uses a predefined filter, the SAO 4604 and the ALF 4606 utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, using codec side information that signals the offset and filter coefficients. The ALF 4606 is located at the last processing stage of each picture and can be viewed as a tool that attempts to capture and repair artifacts created by previous stages.

[0142] The encoder 4600 also includes an intra prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive an input video. The intra prediction component 4608 is configured to perform intra prediction, and the ME / MC component 4610 is configured to perform inter prediction using a reference picture obtained from a reference picture buffer 4612. The residual block from the inter prediction or intra prediction is fed to a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are fed to an entropy coding component 4618. The entropy coding component 4618 entropy codes and decodes the prediction result and the quantized transform coefficients and sends them to a video decoder (not shown). The quantized component output from the quantization component 4616 can be fed to an inverse quantization component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 is capable of outputting images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before the images are stored in the reference picture buffer 4612 .

[0143] Next, a list of preferred solutions is provided for some examples.

[0144] The following solutions show examples of the embodiments discussed herein.

[0145] 1. A method for processing video data (e.g., Figure 5 ), comprising: determining (4202) to apply a pre-processing function to visual media data as part of an image compression framework; and performing (4204) conversion between the visual media data and a bitstream based on the image compression framework.

[0146] 2. A method according to solution 1, wherein the pre-processing function receives a current image of size W0xH0xC0, where W0 is the input width, H0 is the input height and C0 is the number of input channels, and wherein the pre-processing function outputs the current image in size W1xH1xC1, where W1 is the output width, H1 is the output height and C1 is the number of output channels.

[0147] 3. A method according to any of solutions 1-2, wherein W0 is equal to W1, H0 is equal to H1, and C0 is equal to C1.

[0148] 4. A method according to any one of solutions 1-3, wherein C0 is not equal to C1.

[0149] 5. A method according to any one of solutions 1-4, wherein W0 is not equal to W1 or H0 is not equal to H1.

[0150] 6. A method according to any one of solutions 1-5, wherein the pre-processing function includes a deep neural network using parameters determined by deep learning.

[0151] 7. A method according to any one of solutions 1-6, wherein the preprocessing function includes a convolution layer with a kernel size of 3x3, 4x5 or 7x7, and wherein the convolution layer uses a dilated convolution with a dilation parameter set to 3, 5 or 7.

[0152] 8. A method according to any one of solutions 1-7, wherein the preprocessing function includes a residual block, wherein the residual block contains a convolutional layer, an activation layer, a batch normalization layer or a combination thereof.

[0153] 9. The method according to any one of solutions 1-8, wherein the pre-processing function includes a wobble converter, and wherein the number of heads of the wobble converter is two or six, and wherein the depth of the wobble converter is set to two or six.

[0154] 10. A method according to any one of solutions 1-9, wherein the preprocessing function includes a densely connected structure.

[0155] 11. A method according to any one of solutions 1-10, wherein the image compression framework includes a compressor, and wherein the pre-processing function is jointly optimized with the compressor.

[0156] 12. A method according to any of solutions 1-11, wherein the compressor is a variational autoencoder based on learning.

[0157] 13. A method according to any one of solutions 1-12, wherein the learning-based variational autoencoder is partially trained before training the preprocessing function, and wherein the learning-based variational autoencoder and the preprocessing function are trained jointly.

[0158] 14. A method according to any of solutions 1-13, wherein the compressor is a proxy autoencoder with fixed model parameters.

[0159] 15. A method according to any one of solutions 1-14, wherein the pre-processing function includes multiple models for different codec bit rates.

[0160] 16. A method according to any one of solutions 1-15, wherein the pre-processing function is configured to adapt to different bit rates based on quantization parameters, codec control parameters or a combination thereof.

[0161] 17. A method according to any one of solutions 1-16, wherein the proxy autoencoder includes a prediction function, a transformation function, an inverse transformation function, a quantization function and a rate estimation function.

[0162] 18. A method according to any one of solutions 1-17, wherein the proxy autoencoder includes a recovery neural network.

[0163] 19. A method according to any one of solutions 1-18, wherein the compressor includes a hybrid codec, the hybrid codec also includes a codec and a learning-based function, and wherein the learning-based function includes a learning-based loop filter, a learning-based intra-frame prediction, a learning-based inter-frame prediction, a learning-based quantization or a combination thereof.

[0164] 20. A device for processing video data, comprising: a processor; and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of solutions 1 to 19.

[0165] 21. A non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs a method according to any one of Solutions 1 to 19.

[0166] 22. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing device, wherein the method comprises: determining to apply a preprocessing function to visual media data as part of an image compression framework; and generating the bitstream based on the determination.

[0167] 23. A method for storing a bitstream of a video, comprising: determining to apply a preprocessing function to visual media data as part of an image compression framework; generating the bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0168] 24. The methods, apparatus, or systems described in this document.

[0169] In the solution described herein, an encoder can comply with the format rules by generating a codec representation according to the format rules. In the solution described herein, a decoder can use the format rules to parse the syntax elements in the codec representation to generate decoded video, knowing the presence or absence of the syntax elements according to the format rules.

[0170] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of the current video block may, for example, correspond to bits located at the same position or distributed at different positions within the bitstream, as defined by the syntax. For example, a macroblock may be encoded based on error residual values ​​of a transform and codec, and may also use bits in the header and other fields in the bitstream. In addition, during conversion, the decoder may parse the bitstream based on this determination with the knowledge of the presence or absence of some fields, as described in the above solution. Similarly, the encoder may determine whether to include certain syntax fields, and generate the codec representation accordingly by including or excluding the syntax fields in the codec representation.

[0171] The disclosed solutions, examples, embodiments, modules and functional operations and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuits or computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or a combination of one or more of them. The disclosed embodiments and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions, which are encoded on a computer-readable medium for execution by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program under consideration, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0172] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program under consideration, or multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located in one location or distributed in multiple locations and interconnected by a communication network.

[0173] The processes and logic flows described in this document can be performed by executing one or more computer programs by one or more programmable processors to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0174] By way of example, processors suitable for executing computer programs include both general and special purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer may also include or be operably coupled to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, receive data from them or transfer data to them, or both. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact disk read-only memory (CD ROM) and digital versatile disk read-only memory (DVD-ROM) disks. The processor and memory may be supplemented by or incorporated in a dedicated logic circuit.

[0175] Although the present disclosure contains many details, these should not be interpreted as limitations on the scope of any subject matter or what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of particular technologies. Certain features described in the present disclosure in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple implementations or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features in the claimed combination may be cut out of the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0176] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired results. In addition, the separation of various system components in the embodiments described in this disclosure should not be understood as requiring such separation in all embodiments.

[0177] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this disclosure.

[0178] A first component is directly coupled to a second component when there are no intermediate components between the first component and the second component other than a line, a trace, or another medium. A first component is indirectly coupled to a second component when there are intermediate components between the first component and the second component other than a line, a trace, or another medium. The term "coupled" and its variations include direct coupling and indirect coupling. Unless otherwise specified, the use of the term "about" is intended to include a range of ±10% of the subsequent number.

[0179] Although several embodiments are provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples should be considered illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0180] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems, and methods described and shown as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as coupled may be directly connected or may be indirectly coupled or communicated through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and modifications may be determined by those skilled in the art and may be made without departing from the spirit and scope of the present disclosure.

Claims

1. A method for processing video data, comprising: Identify preprocessing functions to be applied to visual media data as part of an image compression framework; as well as Conversion between the visual media data and a bitstream is performed based on the image compression framework.

2. The method according to claim 1, wherein: The pre-processing function receives a current image of size W0xH0xC0, where W0 is the input width, H0 is the input height and C0 is the number of input channels, and wherein the pre-processing function outputs the current image in size W1xH1xC1, where W1 is the output width, H1 is the output height and C1 is the number of output channels.

3. The method according to any one of claims 1 to 2, wherein: W0 is equal to W1, H0 is equal to H1, and C0 is equal to C1.

4. The method according to any one of claims 1 to 3, wherein: C0 is not equal to C1.

5. The method according to any one of claims 1 to 4, wherein: W0 is not equal to W1 or H0 is not equal to H1.

6. The method according to any one of claims 1 to 5, wherein: The pre-processing function includes a deep neural network using parameters determined by deep learning.

7. The method according to any one of claims 1 to 6, wherein: The pre-processing function comprises a convolutional layer using a kernel size of 3x3, 5x5 or 7x7, and wherein the convolutional layer uses a dilated convolution with a dilation parameter set to 3, 5 or 7.

8. The method according to any one of claims 1 to 7, wherein: The pre-processing function comprises a residual block, wherein the residual block comprises a convolutional layer, an activation layer, a batch normalization layer, or a combination thereof.

9. The method according to any one of claims 1 to 8, wherein: The pre-processing function includes a wobble converter, wherein the number of heads of the wobble converter is two or six, and wherein the depth of the wobble converter is set to two or six.

10. The method according to any one of claims 1 to 9, wherein: The pre-processing function includes a densely connected structure.

11. The method according to any one of claims 1 to 10, wherein: The image compression framework comprises a compressor, and wherein the pre-processing function is jointly optimized with the compressor.

12. The method according to any one of claims 1 to 11, wherein: The compressor is based on a learned variational autoencoder.

13. The method according to any one of claims 1 to 12, wherein: The learning-based variational autoencoder is partially trained before training the preprocessing function, and wherein the learning-based variational autoencoder and the preprocessing function are jointly trained.

14. The method according to any one of claims 1 to 13, wherein: The compressor is a proxy autoencoder with fixed model parameters.

15. The method according to any one of claims 1 to 14, wherein: The pre-processing function includes multiple models for different codec bit rates.

16. The method according to any one of claims 1 to 15, wherein: The pre-processing function is configured to adapt to different bit rates based on quantization parameters, codec control parameters, or a combination thereof.

17. The method according to any one of claims 1 to 16, wherein: The proxy autoencoder includes a prediction function, a transform function, an inverse transform function, a quantization function, and a rate estimation function.

18. The method according to any one of claims 1 to 17, wherein: The proxy autoencoder includes a restoration neural network.

19. The method according to any one of claims 1 to 18, wherein: The compressor comprises a hybrid codec, which further comprises a codec and a learning-based function, and wherein the learning-based function comprises a learning-based loop filter, learning-based intra-frame prediction, learning-based inter-frame prediction, learning-based quantization, or a combination thereof.

20. An apparatus for processing video data, comprising: processor; as well as A non-volatile memory coupled to the processor and having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1-19.

21. A non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, and the computer executable instructions, when executed by a processor, cause the video codec device to perform the method according to any one of claims 1 to 19.

22. A non-transitory computer-readable recording medium storing a bit stream of a video generated by executing a method by a video processing device, wherein the method comprises: Identify preprocessing functions to be applied to visual media data as part of an image compression framework; as well as The bitstream is generated based on the determination.

23. A method for storing a bit stream of a video, comprising: Identify preprocessing functions to be applied to visual media data as part of an image compression framework; generating the bitstream based on the determination; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images