Video coding method and device based on implicit neural representation, equipment and medium

Abstract

The application discloses a video coding method and device based on implicit neural representation, equipment and medium, relates to the technical field of video processing, and the method comprises the following steps: inputting a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing space-time features; inputting the embedding vector into an implicit neural representation decoder; and decoding the embedding vector to obtain a reconstructed frame corresponding to the current frame by using the implicit neural representation decoder. The application not only uses the frame index as the input of the encoder, but also uses the current frame corresponding to the frame index as the input of the encoder, so that the embedding vector output by the encoder retains more original information of the current frame, thereby improving the quality of the reconstructed frame. Moreover, the implicit neural representation decoder can efficiently compress the data volume of the current frame, so that the reconstructed frame not only maintains high quality but also reduces the data volume, thereby reducing the storage requirement and transmission bandwidth requirement.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to video coding methods, apparatus, devices and media based on implicit neural representations. Background Technology

[0002] With the rapid development of mobile internet technology, the widespread adoption of smart terminal devices, and the continuous upgrading of network infrastructure, video content has experienced explosive growth. Video traffic is not only increasing rapidly in absolute terms, but its proportion of total internet traffic is also rising daily. Against this backdrop, the contradiction between limited network bandwidth and the rapidly increasing volume of video data is becoming increasingly acute. Summary of the Invention

[0003] The main objective of this application is to propose a video coding method, apparatus, device, and medium based on implicit neural representation, so as to efficiently compress video while maintaining high-quality video, thereby reducing storage requirements and transmission bandwidth requirements.

[0004] To achieve the above objectives, one aspect of this application proposes a video coding method based on implicit neural representations, the method comprising the following steps:

[0005] The frame index and the current frame corresponding to the frame index are input into the encoder to obtain an embedding vector containing spatiotemporal features;

[0006] The embedding vector is input into the implicit neural representation decoder;

[0007] The hidden neural representation decoder is used to decode the current frame according to the embedding vector to obtain the reconstructed frame.

[0008] In some embodiments, inputting the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features includes the following steps:

[0009] The temporal features of the frame index are extracted using a network that utilizes the temporal features in the encoder.

[0010] The spatial features of the current frame are extracted using the spatial feature extraction network in the encoder;

[0011] The temporal and spatial features are fused using the spatiotemporal feature fusion network in the encoder to obtain the embedding vector containing the spatiotemporal features.

[0012] In some embodiments, the step of using the implicit neural representation decoder to decode the reconstructed frame corresponding to the current frame based on the embedding vector includes the following steps:

[0013] The embedding vector is decoded using the first decoding block in the implicit neural representation decoder to obtain a first decoded frame; wherein the implicit neural representation decoder includes a plurality of decoding blocks connected in sequence;

[0014] The step of repeatedly executing the step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame is repeated until the last decoding block in the implicit neural representation decoder outputs the corresponding first decoded frame as the first reconstructed frame of the frame index.

[0015] In some embodiments, the method further includes the following steps:

[0016] The time features are respectively input into each of the decoding blocks;

[0017] The process of decoding the embedding vector using the first decoding block in the implicit neural representation decoder to obtain the first decoded frame includes the following steps:

[0018] The second decoded frame is obtained by decoding the embedding vector and the temporal features using the first decoding block in the implicit neural representation decoder;

[0019] The step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame includes the following steps:

[0020] The second decoded frame output by the current decoded block and the time feature are input into the next connected decoded block for decoding to obtain the corresponding second decoded frame.

[0021] In some embodiments, the method further includes the following steps:

[0022] The first decoded frames of different resolutions are output using the output layers of each of the decoded blocks;

[0023] At least one of the first decoded frames of different resolutions is determined as the second reconstructed frame corresponding to the frame index;

[0024] The video is constructed based on each of the second reconstructed frames corresponding to different frame indices.

[0025] In some embodiments, the method further includes the following steps:

[0026] The residual is obtained by subtracting the current frame from the reconstructed frame;

[0027] The residual is input into the encoder for encoding to obtain residual features;

[0028] The residual features are input into the implicit neural representation decoder, and then the decoding output of the implicit neural representation decoder is added to the reconstructed frame to obtain the current reconstructed frame;

[0029] Return to the step of subtracting the current frame from the reconstructed frame to obtain the residual, until the current reconstructed frame reaches the target bitrate.

[0030] In some embodiments, before inputting the embedding vector into the implicit neural representation decoder, the method further includes a step of training the implicit neural representation decoder, the step of training the implicit neural representation decoder including the following steps:

[0031] The mean squared error and structural similarity between the current training frame and the corresponding training reconstructed frame are used as loss functions to train the implicit neural representation decoder.

[0032] The trained implicit neural representation decoder is then quantized, pruned, and weighted entropy encoded.

[0033] To achieve the above objectives, another aspect of this application proposes a video coding apparatus based on implicit neural representations, the apparatus comprising:

[0034] The spatiotemporal coding unit is used to input the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features;

[0035] A spatiotemporal input unit is used to input the embedding vector into the implicit neural representation decoder;

[0036] The spatiotemporal reconstruction unit is used to obtain the reconstructed frame corresponding to the current frame by using the implicit neural representation decoder based on the embedding vector.

[0037] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described video coding method based on implicit neural representation.

[0038] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned video coding method based on implicit neural representations.

[0039] The embodiments of this application include at least the following beneficial effects:

[0040] This application inputs a frame index and the corresponding current frame into an encoder to obtain an embedding vector containing spatiotemporal features; the embedding vector is then input into an implicit neural representation decoder; and the implicit neural representation decoder decodes the current frame based on the embedding vector to obtain the reconstructed frame. This application not only uses the frame index as input to the encoder but also uses the corresponding current frame as input, allowing the encoder's output embedding vector to retain more of the original information of the current frame, thereby improving the quality of the reconstructed frame. Furthermore, the implicit neural representation decoder can efficiently compress the data volume of the current frame, enabling the reconstructed frame to maintain high quality while reducing data volume, thus reducing storage and transmission bandwidth requirements. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating an optional video coding method based on implicit neural representation provided for embodiments of this application;

[0043] Figure 2 A flowchart illustrating the video coding method based on implicit neural representation provided in this application embodiment;

[0044] Figure 3 A flowchart of the implicit neural representation video coding method that integrates spatiotemporal information, provided in an embodiment of this application;

[0045] Figure 4 A flowchart of the enhanced implicit neural representation video coding method for fusing spatiotemporal information provided in the embodiments of this application;

[0046] Figure 5 A flowchart of the multi-resolution output implicit neural representation video coding method provided in the embodiments of this application;

[0047] Figure 6 A flowchart of the multi-bitrate output implicit neural representation video coding method provided in the embodiments of this application;

[0048] Figure 7 A schematic diagram of the structure of a video coding device based on implicit neural representation provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0051] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0052] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0054] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be explained first. The related technologies involved in the embodiments of this application are subject to the following interpretation:

[0055] With the rapid development of mobile internet technology, the widespread adoption of smart terminal devices, and the continuous upgrading of network infrastructure, video content has experienced explosive growth. Video traffic is not only increasing rapidly in absolute terms, but its proportion of total internet traffic is also rising daily. Against this backdrop, the contradiction between limited network bandwidth and the rapidly increasing volume of video data is becoming increasingly acute. How to efficiently compress video while ensuring visual quality, thereby reducing storage and transmission bandwidth requirements, has become a critical issue that urgently needs to be addressed in the field of multimedia technology.

[0056] Traditional video coding methods rely on manually designed algorithms to compress spatial and temporal redundancy in videos. Methods such as AVS, H.264, H.265, and AV1 are widely used. Although these methods have achieved excellent compression performance over the years, their heavy reliance on manually designed coding strategies makes end-to-end optimization difficult, further hindering improvements in compression efficiency.

[0057] In recent years, neural networks and deep learning technologies have been widely applied in the field of computer vision, and research on applying these technologies to video coding tasks has emerged in large numbers. Deep learning-based video coding technologies can be broadly divided into two categories: one is combining with traditional coding methods, replacing some modules in traditional coding methods with neural networks, such as loop filtering and intra-frame prediction; the other is using neural networks for end-to-end video compression. While the first method can be well integrated with existing traditional compression methods, it faces the challenge of optimizing the entire coding system as a whole. The second method better utilizes the powerful fitting ability of neural networks, achieving better compression results than traditional coding methods. The second method can be further subdivided into end-to-end video coding methods that follow the traditional compression framework and video coding methods based on implicit neural representations. The former replaces all modules in traditional coding methods with neural networks, but due to the complex network structure, the decoding speed is slow, making deployment difficult. In contrast, video coding methods based on implicit neural representations use smaller neural networks, embedding video data into the weights of the neural network through training, achieving decoding speeds comparable to or even faster than traditional coding methods, and have broad application prospects. This application pertains to a video coding scheme based on implicit neural representations.

[0058] Reference Figure 1 , Figure 1 A flowchart of an optional video coding method based on implicit neural representation provided in this application.

[0059] exist Figure 1In this scheme, the basic video coding method based on implicit neural representation uses frame indices as input. After positional encoding, a neural network models a function with the current frame index as the independent variable and the output reconstructed frame as the dependent variable. This modeling process is the training process of the neural network. During training, the mean squared error (MSE) between the original frame and the reconstructed frame and the structure similarity index measure (SSIM) can be used as loss functions. After training, the information of the entire video is embedded into the neural network weights in the data embedding region, and the current reconstructed frame can be quickly generated by inputting the frame index. By quantizing, pruning, and weight entropy encoding the neural network in the data embedding region, the amount of data representing the entire video can be further compressed, achieving efficient video coding.

[0060] but Figure 1 The proposed solution also has some problems:

[0061] 1. This scheme only uses the frame index as input, lacks guidance from video spatial information during the neural network modeling process, and the neural network training converges slowly. Furthermore, the detailed information of the frames in the video is easily lost, resulting in the degradation of the overall video quality.

[0062] 2. This scheme can only output reconstructed videos at a specific resolution. For the same content videos at different resolutions, the network structure needs to be redesigned and retrained, which does not make full use of the upsampling and downsampling characteristics of neural networks.

[0063] 3. This solution can only output videos with a specific bitrate and cannot provide videos with different bitrates and the same resolution for different application scenarios.

[0064] To address at least one problem in related technologies, embodiments of this application provide a video coding method, apparatus, device, and medium based on implicit neural representation. The technical solution of this application includes: inputting a frame index and the current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features; inputting the embedding vector into an implicit neural representation decoder; and using the implicit neural representation decoder to decode the current frame based on the embedding vector to obtain a reconstructed frame corresponding to the current frame. This application not only uses the frame index as input to the encoder but also uses the current frame corresponding to the frame index as input to the encoder, so that the embedding vector output by the encoder retains more of the original information of the current frame, thereby improving the quality of the reconstructed frame; moreover, the implicit neural representation decoder can efficiently compress the data volume of the current frame, so that the reconstructed frame maintains high quality while reducing the data volume, thereby reducing storage requirements and transmission bandwidth requirements.

[0065] This application provides a video coding method, apparatus, device, and medium based on implicit neural representations, relating to the field of video processing technology. The video coding method, apparatus, device, and medium based on implicit neural representations provided in this application can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the server can also be a node server in a blockchain network; the software can be an application implementing knowledge extraction methods, but is not limited to the above forms.

[0066] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0067] Reference Figure 2 This application provides a video coding method based on implicit neural representations. This method may include, but is not limited to, steps S200 to S220, as follows:

[0068] S200: Input the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features.

[0069] Furthermore, S200 may include the following steps S201 to S203:

[0070] S201: Use the temporal features in the encoder to organize the network and extract the temporal features of the frame index;

[0071] S202: Extract the spatial features of the current frame using the spatial feature extraction network in the encoder;

[0072] S203: The temporal and spatial features are fused using the spatiotemporal feature fusion network in the encoder to obtain the embedding vector containing the spatiotemporal features.

[0073] S210: Input the embedding vector into the implicit neural representation decoder.

[0074] As an optional implementation, prior to S210, this embodiment may further include a step of training the implicit neural representation decoder in S100, which may include the following steps S101 to S102:

[0075] S101: Use the mean squared error and structural similarity between the current training frame and the corresponding training reconstructed frame as a loss function, and use the loss function to train the implicit neural representation decoder;

[0076] S102: Quantize, prune, and encode the trained implicit neural representation decoder using weight entropy.

[0077] S220: The hidden neural representation decoder is used to decode the reconstructed frame corresponding to the current frame according to the embedding vector.

[0078] Furthermore, S220 may include the following steps S221 to S222:

[0079] S221: The embedding vector is decoded using the first decoding block in the implicit neural representation decoder to obtain a first decoded frame; wherein the implicit neural representation decoder includes a plurality of decoding blocks connected in sequence;

[0080] S222: Repeat the step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame, until the last decoding block in the implicit neural representation decoder outputs the corresponding first decoded frame as the first reconstructed frame of the frame index.

[0081] Furthermore, S220 may also include the following step S223:

[0082] S223: Input the time features into each of the decoding blocks respectively.

[0083] Furthermore, S221 can be more specifically described as follows: using the first decoding block in the implicit neural representation decoder to decode the embedding vector and the temporal features to obtain a second decoded frame.

[0084] S222 can be more specifically described as follows: inputting the second decoded frame output by the current decoded block and the time feature into the next connected decoded block for decoding to obtain the corresponding second decoded frame.

[0085] As another optional implementation, S220 may further include the following steps S224 to S226:

[0086] S224: Output the first decoded frames at different resolutions using the output layers of each of the decoded blocks;

[0087] S225: Determine at least one of the first decoded frames of different resolutions as the second reconstructed frame corresponding to the frame index;

[0088] S226: Construct a video based on each of the second reconstructed frames corresponding to different frame indices.

[0089] The embodiments of this application may further include step S230, which involves reconstructing the current frame at the same resolution to obtain reconstructed frames with different bitrates, and then constructing the video. Specifically, S230 may include the following steps S231 to S234:

[0090] S231: Subtract the current frame from the reconstructed frame to obtain the residual;

[0091] S232: Input the residual into the encoder for encoding to obtain residual features;

[0092] S233: Input the residual features into the implicit neural representation decoder, and then add the decoding output of the implicit neural representation decoder to the reconstructed frame to obtain the current reconstructed frame;

[0093] S234: Return to the step of subtracting the current frame from the reconstructed frame to obtain the residual, until the current reconstructed frame reaches the target bit rate.

[0094] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0095] Reference Figure 3 This embodiment provides an example flowchart of a video coding method based on implicit neural representation.

[0096] This embodiment proposes an implicit neural representation video coding scheme that fuses temporal information extracted from the frame index with spatial information extracted from the current frame, thereby achieving implicit neural representation video coding that integrates spatiotemporal information.

[0097] Specifically, this embodiment may include the following solutions:

[0098] 1. At the encoding end, a spatial feature extraction network is introduced to extract features from the current input frame. This spatial feature extraction network can be any network that helps extract spatial features, such as a convolutional neural network and a visual converter. The frame index t is input into a temporal feature processing network, which can be a multilayer perceptron, etc., and outputs a temporal feature with a similar size to the spatial feature but a different number of channels.

[0099] 2. The obtained temporal and spatial features are fused using a spatiotemporal feature fusion network. This network can be any network that facilitates spatiotemporal feature fusion, such as an attention feature fusion network. The spatiotemporal feature fusion network outputs an embedding vector containing spatiotemporal features. This vector serves as the input to the decoder and, together with the weights of the decoder network, forms the data embedding for storing the video data.

[0100] 3. At the decoding end, the embedding vector will replace the frame index as the input of the decoder. By inputting the embedding vector corresponding to different frame indices into the decoding network, the reconstructed frame corresponding to that frame index can be decoded.

[0101] Reference Figure 4 This embodiment provides a flowchart of an enhanced implicit neural representation video coding method that incorporates spatiotemporal information.

[0102] Specifically, to further improve the fitting ability of the scheme in this embodiment, temporal information can be fed into the decoder network to further improve the video coding efficiency of the implicit neural representation decoder. The enhanced scheme incorporates the temporal information processing network into the data embedding region, enhancing the network's representational ability. However, this scheme requires both the frame index and the embedding vector as inputs during decoding, and the inference complexity increases. In practical applications, the choice can be made based on specific needs.

[0103] The implicit neural representation decoder in the above scheme uses the mean squared error (MSE) and structural similarity index measure (SSIM) between the original current frame and the reconstructed frame as loss functions during training. After training, efficient video encoding is achieved by quantizing, pruning, and weight entropy encoding the neural network in the data embedding region.

[0104] In addition, this embodiment also provides another optional implementation method that enables hierarchical video coding based on implicit neural representations.

[0105] Reference Figure 5 , Figure 5 A flowchart for a video coding method for outputting implicit neural representations at multiple resolutions.

[0106] Specifically, this embodiment can use hierarchical video coding based on implicit neural representations to output videos with different resolutions and different bitrates at the same resolution.

[0107] Video coding schemes based on implicit neural representations embed most of the video data into the weights of a neural network, and the network gradually increases the size of the output features as it reconstructs the video. The amount of video data stored in the network gradually decreases from beginning to end, so high-quality video can be reconstructed even by extracting intermediate variables during the decoding process. Specifically, intermediate variables are extracted during the decoding process, the number of channels in the input and output layers is reduced, and finally, a 3-channel video frame with the same width and height as the intermediate variables is output. During the training of the neural network, the output layers corresponding to video frames of various resolutions can be trained simultaneously, allowing for the generation of output videos of different resolutions with a single encoding. Targeted network structures can also be designed for different resolution output targets. During the video decoding process, if a higher resolution video is no longer needed, the network portion after the output can be omitted, further saving computational costs.

[0108] This embodiment can also output videos with different bitrates at the same resolution, see reference. Figure 6 , Figure 6 A flowchart for a video coding method for outputting implicit neural representations at multiple bit rates.

[0109] Specifically, outputting videos with different bitrates at the same resolution can be achieved using multiple video encoders based on implicit neural representations. Specifically, the encoding obtained by the first encoder is decoded to obtain reconstructed frame 1. The current frame is subtracted from the reconstructed frame to obtain the residual. This residual is then encoded using the second encoder, decoded, and added to reconstructed frame 1 to obtain a higher-quality reconstructed frame 2, and so on. As the reconstruction quality gradually improves and the residual gradually decreases, the complexity of the encoders used will gradually decrease.

[0110] In summary, combining Figure 5 and Figure 6 The method shown can realize a hierarchical video coding scheme with multiple resolutions and multiple bit rates, namely a hierarchical video coding scheme based on implicit neural representations.

[0111] This embodiment may also provide test examples.

[0112] For example, the products and equipment used in the test instance are as follows:

[0113] 1. Hardware environment:

[0114] Linux server, 6GB or more of video memory.

[0115] 2. Software environment:

[0116] Ubuntu 20.04 system;

[0117] Python >= 3.8;

[0118] PyTorch>=1.8.1;

[0119] Torchvision;

[0120] Pytorchvideo.

[0121] The dataset is tested using the UVG dataset, which is commonly used in the field of video encoding and decoding. The neural network is trained using the video to be encoded itself, and the video data is embedded into the weights of the implicit neural representation decoder.

[0122] In terms of evaluation metrics, the most commonly used metrics in the field of video compression are followed, namely the average bit rate per pixel (bpp) of the output video, the peak signal-to-noise ratio (PSNR) of the original video and the compressed video, and the structure similarity index measure (SSIM).

[0123] Reference Figure 7 This application also provides a video coding apparatus based on implicit neural representation, which can implement the above-described video coding method based on implicit neural representation. The apparatus includes:

[0124] The spatiotemporal coding unit is used to input the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features;

[0125] A spatiotemporal input unit is used to input the embedding vector into the implicit neural representation decoder;

[0126] The spatiotemporal reconstruction unit is used to obtain the reconstructed frame corresponding to the current frame by using the implicit neural representation decoder based on the embedding vector.

[0127] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0128] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned video coding method based on implicit neural representation. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0129] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0130] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0131] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0132] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the video coding method based on implicit neural representation of the embodiments of this application.

[0133] The 803 input / output interface is used to implement information input and output.

[0134] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0135] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0136] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0137] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned video coding method based on implicit neural representations.

[0138] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0139] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0141] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0144] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0145] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0147] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0149] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video coding method based on implicit neural representation, characterized in that, The method includes the following steps: The frame index and the current frame corresponding to the frame index are input into the encoder to obtain an embedding vector containing spatiotemporal features; The embedding vector is input into the implicit neural representation decoder; The implicit neural representation decoder is used to decode the current frame based on the embedding vector to obtain the reconstructed frame. The process of using the implicit neural representation decoder to decode the reconstructed frame corresponding to the current frame based on the embedding vector includes the following steps: The first decoded frame is obtained by decoding the embedding vector using the first decoding block in the implicit neural representation decoder; wherein the implicit neural representation decoder includes a plurality of the decoding blocks connected in sequence; The step of repeatedly executing the step of inputting the decoded frame output by the current first decoded block into the next connected decoded block for decoding to obtain the corresponding first decoded frame is repeated until the last decoded block in the implicit neural representation decoder outputs the corresponding first decoded frame as the first reconstructed frame of the frame index. The method further includes the following steps: The time features are input into each of the decoding blocks respectively; The process of decoding the embedding vector using the first decoding block in the implicit neural representation decoder to obtain the first decoded frame includes the following steps: The second decoded frame is obtained by decoding the embedding vector and the temporal features using the first decoding block in the implicit neural representation decoder; The step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame includes the following steps: The second decoded frame output by the current decoded block and the time feature are input into the next connected decoded block for decoding to obtain the corresponding second decoded frame; The step of inputting the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features includes the following steps: The temporal features of the frame index are extracted using a network that utilizes the temporal features in the encoder. The spatial features of the current frame are extracted using the spatial feature extraction network in the encoder; The temporal and spatial features are fused using the spatiotemporal feature fusion network in the encoder to obtain the embedding vector containing the spatiotemporal features.

2. The video coding method based on implicit neural representation according to claim 1, characterized in that, The method further includes the following steps: The first decoded frames of different resolutions are output using the output layers of each of the decoded blocks; At least one of the first decoded frames of different resolutions is determined as the second reconstructed frame corresponding to the frame index; The video is constructed based on each of the second reconstructed frames corresponding to different frame indices.

3. The video coding method based on implicit neural representation according to claim 1, characterized in that, The method further includes the following steps: The residual is obtained by subtracting the current frame from the reconstructed frame; The residual is input into the encoder for encoding to obtain residual features; The residual features are input into the implicit neural representation decoder, and then the decoding output of the implicit neural representation decoder is added to the reconstructed frame to obtain the current reconstructed frame; Return to the step of subtracting the current frame from the reconstructed frame to obtain the residual, until the current reconstructed frame reaches the target bitrate.

4. The video coding method based on implicit neural representation according to any one of claims 1 to 3, characterized in that, Before inputting the embedding vector into the implicit neural representation decoder, the method further includes a step of training the implicit neural representation decoder, the step of training the implicit neural representation decoder including the following steps: The mean squared error and structural similarity between the current training frame and the corresponding training reconstructed frame are used as loss functions to train the implicit neural representation decoder. The trained implicit neural representation decoder is then quantized, pruned, and weighted entropy encoded.

5. A video coding device based on implicit neural representation, characterized in that, The apparatus is used to implement the video coding method based on implicit neural representation as described in claim 1, the apparatus comprising: The spatiotemporal coding unit is used to input the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features; A spatiotemporal input unit is used to input the embedding vector into the implicit neural representation decoder; The spatiotemporal reconstruction unit is used to obtain the reconstructed frame corresponding to the current frame by using the implicit neural representation decoder based on the embedding vector.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the video coding method based on implicit neural representation as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video coding method based on implicit neural representation as described in any one of claims 1 to 4.

Images