Video coding method and device based on implicit neural representation, equipment and medium

Abstract

The invention discloses a video coding method and device based on implicit neural representation, equipment and a medium, and relates to the technical field of video processing, and the method comprises the steps: inputting a frame index and a current frame corresponding to the frame index into a coder, and obtaining an embedded vector containing spatial-temporal features; inputting the embedded vector into an implicit neural representation decoder; and decoding by using an implicit neural representation decoder according to the embedded vector to obtain a reconstructed frame corresponding to the current frame. According to the invention, the frame index is used as the input of the encoder, and the current frame corresponding to the frame index is also used as the input of the encoder, so that the embedded vector output by the encoder retains more original information of the current frame, and the quality of the reconstructed frame is improved; and the implicit neural representation decoder can efficiently compress the data volume of the current frame, so that the reconstructed frame keeps high quality, the data volume is reduced, and the storage requirement and the transmission bandwidth requirement are reduced.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video encoding method, apparatus, device and medium based on implicit neural representation. Background Art

[0002] With the vigorous development of mobile Internet technology, the widespread popularity of smart terminal devices and the continuous upgrading of network infrastructure, video content has shown an explosive growth trend. Not only has the absolute value of video traffic increased rapidly, but its proportion in the total Internet traffic has also been increasing. In this context, the contradiction between limited network bandwidth and the rapidly growing amount of video data has become increasingly acute. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a video encoding method, device, equipment and medium based on implicit neural representation, so as to efficiently compress the video while maintaining high-quality video, thereby reducing storage requirements and transmission bandwidth requirements.

[0004] To achieve the above object, an embodiment of the present application provides a video coding method based on implicit neural representation, the method comprising the following steps:

[0005] Inputting a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features;

[0006] Inputting the embedding vector into an implicit neural representation decoder;

[0007] The implicit neural representation decoder is used to decode the embedded vector to obtain a reconstructed frame corresponding to the current frame.

[0008] In some embodiments, the step of inputting a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features comprises the following steps:

[0009] Extracting the temporal features of the frame index using a temporal feature arrangement network in the encoder;

[0010] Extracting spatial features of the current frame using a spatial feature extraction network in the encoder;

[0011] The temporal features and the spatial features are fused using a spatiotemporal feature fusion network in the encoder to obtain the embedding vector containing spatiotemporal features.

[0012] In some embodiments, the using the implicit neural representation decoder to decode according to the embedded vector to obtain a reconstructed frame corresponding to the current frame comprises the following steps:

[0013] Decoding the embedding vector using the first decoding block in the implicit neural representation decoder to obtain a first decoded frame; wherein the implicit neural representation decoder includes a plurality of the decoding blocks connected in sequence;

[0014] Repeat the step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame, until the last decoding block in the implicit neural representation decoder outputs the corresponding first decoded frame as the first reconstructed frame of the frame index.

[0015] In some embodiments, the method further comprises the following steps:

[0016] Inputting the time features into each of the decoding blocks respectively;

[0017] The step of decoding the embedded vector using the first decoding block in the implicit neural representation decoder to obtain a first decoded frame comprises the following steps:

[0018] Decoding the embedding vector and the temporal feature using the first decoding block in the implicit neural representation decoder to obtain a second decoded frame;

[0019] The step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame comprises the following steps:

[0020] The second decoded frame and the time feature output by the current decoding block are input to the next connected decoding block for decoding to obtain the corresponding second decoded frame.

[0021] In some embodiments, the method further comprises the following steps:

[0022] Outputting the first decoded frames of different resolutions using output layers of the respective decoding blocks;

[0023] Determine at least one of the first decoded frames of different resolutions as a second reconstructed frame corresponding to the frame index;

[0024] The video is constructed according to each of the second reconstructed frames corresponding to different frame indexes.

[0025] In some embodiments, the method further comprises the following steps:

[0026] Subtracting the current frame from the reconstructed frame to obtain a residual;

[0027] Inputting the residual into the encoder for encoding to obtain residual features;

[0028] Inputting the residual feature into the implicit neural representation decoder, and then adding the decoded output of the implicit neural representation decoder to the reconstructed frame to obtain the current reconstructed frame;

[0029] Return to the step of subtracting the current frame from the reconstructed frame to obtain a residual until the current reconstructed frame reaches a target bit rate.

[0030] In some embodiments, before inputting the embedding vector into the implicit neural representation decoder, the method further comprises the step of training the implicit neural representation decoder, wherein the step of training the implicit neural representation decoder comprises the following steps:

[0031] Using the mean square error and structural similarity between the training current frame and the corresponding training reconstructed frame as a loss function, and using the loss function to train the implicit neural representation decoder;

[0032] The trained implicit neural representation decoder is quantized, pruned and weighted entropy encoded.

[0033] To achieve the above object, another aspect of the embodiment of the present application provides a video encoding device based on implicit neural representation, the device comprising:

[0034] A spatiotemporal encoding unit, configured to input a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features;

[0035] a spatiotemporal input unit for inputting the embedding vector into an implicit neural representation decoder;

[0036] The spatiotemporal reconstruction unit is used to use the implicit neural representation decoder to decode according to the embedded vector to obtain a reconstructed frame corresponding to the current frame.

[0037] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned video encoding method based on implicit neural representation when executing the computer program.

[0038] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned video encoding method based on implicit neural representation is implemented.

[0039] The embodiments of the present application include at least the following beneficial effects:

[0040] The present application can input the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features; input the embedding vector into the implicit neural representation decoder; and use the implicit neural representation decoder to decode the embedding vector to obtain the reconstructed frame corresponding to the current frame. The present application not only uses the frame index as the input of the encoder, but also uses the current frame corresponding to the frame index as the input of the encoder, so that the embedding vector output by the encoder retains more original information of the current frame, thereby improving the quality of the reconstructed frame; and the implicit neural representation decoder can efficiently compress the data volume of the current frame, so that the reconstructed frame maintains high quality while reducing the data volume, thereby reducing storage requirements and transmission bandwidth requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A flowchart of an optional video encoding method based on implicit neural representation provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a process flow of a video encoding method based on implicit neural representation provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of an implicit neural representation video coding method integrating spatiotemporal information provided in an embodiment of the present application;

[0045] Figure 4 A flowchart of an enhanced implicit neural representation video coding method integrating spatiotemporal information provided in an embodiment of the present application;

[0046] Figure 5 A flowchart of a multi-resolution output implicit neural representation video encoding method provided in an embodiment of the present application;

[0047] Figure 6 A flowchart of a multi-bitrate output implicit neural representation video encoding method provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of a video encoding device based on implicit neural representation provided in an embodiment of the present application;

[0049] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] Before describing the embodiments of the present application in detail, some related technologies involved in the embodiments of the present application are first described. The related technologies involved in the embodiments of the present application are applicable to the following explanations:

[0055] With the vigorous development of mobile Internet technology, the widespread popularity of smart terminal devices and the continuous upgrading of network infrastructure, video content has shown an explosive growth trend. Not only has the absolute value of video traffic increased rapidly, but its proportion in the total Internet traffic has also increased. In this context, the contradiction between limited network bandwidth and the rapidly growing amount of video data has become increasingly acute. How to efficiently compress video while ensuring the visual quality of the video, thereby reducing storage requirements and transmission bandwidth requirements, has become a key issue that needs to be urgently solved in the current field of multimedia technology.

[0056] Traditional video coding methods rely on manually designed algorithms to compress spatial and temporal redundancy in videos. Traditional coding methods such as AVS, H.264, H.265, and AV1 are widely used. Although traditional video coding methods have achieved excellent compression performance after years of development, these methods are highly dependent on manually designed coding strategies and are difficult to optimize end-to-end, making it difficult to further improve compression efficiency.

[0057] In recent years, neural networks and deep learning technologies have been widely used in the field of computer vision, and research on applying the above technologies to video coding tasks has also emerged in an endless stream. Video coding technologies based on deep learning can be roughly divided into two categories. One is to combine with traditional coding methods and replace some modules in traditional coding methods with neural networks, such as loop filtering, intra-frame prediction, etc. The other is to use neural networks for end-to-end video compression. Although the first method can be well combined with the current traditional compression method, it faces the problem of difficulty in optimizing the entire coding system as a whole. The second method makes better use of the powerful fitting ability of neural networks and can achieve better compression effects than traditional coding methods. The second method can be further divided into an end-to-end video coding method that follows the traditional compression framework and a video coding method based on implicit neural representation. The former uses a neural network to replace all modules in the traditional coding method, but due to the complex network structure and slow decoding speed, it is difficult to deploy and apply. The video coding method based on implicit neural representation uses a smaller neural network, and embeds the video data into the weights of the neural network through training, which can achieve a decoding speed comparable to or even faster than the traditional coding method, and has broad application prospects. This application belongs to a video coding scheme based on implicit neural representation.

[0058] Reference Figure 1 , Figure 1 A flowchart of an optional video encoding method based on implicit neural representation provided for this application.

[0059] exist Figure 1In the scheme, the basic implicit neural representation-based video coding method uses the frame index as input. After position encoding, the neural network models the function with the current frame index as the independent variable and the output reconstructed frame as the dependent variable. The modeling process is the training process of the neural network. During the training process, the mean squared error (MSE) and structure similarity index measure (SSIM) between the original frame and the reconstructed frame can be used as loss functions. After the training is completed, the information of the entire video is embedded in the neural network weights in the data embedding area, and the current reconstructed frame can be quickly generated by inputting the frame index. By quantizing, pruning and weight entropy coding the neural network in the data embedding area, the amount of data representing the entire video can be further compressed to achieve efficient video coding.

[0060] but Figure 1 There are also some problems with the solution:

[0061] 1. This scheme only uses frame index as input, lacks the guidance of video spatial information in the neural network modeling process, and the neural network training converges slowly. In addition, the detailed information of the frames in the video is easily lost, resulting in the degradation of the overall video quality.

[0062] 2. This solution can only output reconstructed videos of a specific resolution. For videos with the same content at different resolutions, the network structure needs to be redesigned and retrained, which does not fully utilize the up- and down-sampling characteristics of the neural network.

[0063] 3. This solution can only output videos with a specific bit rate and cannot provide videos with the same resolution but with different bit rates for different application scenarios.

[0064] In order to solve at least one problem of the related art, the embodiments of the present application provide a video encoding method, apparatus, device and medium based on implicit neural representation. The technical solution of the present application includes: inputting the frame index and the current frame corresponding to the frame index into the encoder to obtain an embedding vector containing spatiotemporal features; inputting the embedding vector into the implicit neural representation decoder; using the implicit neural representation decoder to decode according to the embedding vector to obtain a reconstructed frame corresponding to the current frame. The present application not only uses the frame index as the input of the encoder, but also uses the current frame corresponding to the frame index as the input of the encoder, so that the embedding vector output by the encoder retains more original information of the current frame, thereby improving the quality of the reconstructed frame; and the implicit neural representation decoder can efficiently compress the data volume of the current frame, so that the reconstructed frame maintains high quality while reducing the data volume, thereby reducing storage requirements and transmission bandwidth requirements.

[0065] The embodiments of the present application provide a video encoding method, device, equipment and medium based on implicit neural representation, and relate to the field of video processing technology. The video encoding method, device, equipment and medium based on implicit neural representation provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements the knowledge extraction method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] Reference Figure 2 , the embodiment of the present application provides a video encoding method based on implicit neural representation, the method may include but is not limited to including S200 to S220, specifically as follows:

[0068] S200: Inputting a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features.

[0069] Further, S200 may include the following steps S201 to S203:

[0070] S201: Extracting the time feature of the frame index using a time feature arrangement network in the encoder;

[0071] S202: Extracting spatial features of the current frame using a spatial feature extraction network in the encoder;

[0072] S203: Utilize the spatiotemporal feature fusion network in the encoder to fuse the temporal feature and the spatial feature to obtain the embedding vector containing the spatiotemporal feature.

[0073] S210: Input the embedding vector into the implicit neural representation decoder.

[0074] As an optional implementation, before S210, the embodiment of the present application may further include a step of S100 training the implicit neural representation decoder, and S100 may include the following steps S101-S102:

[0075] S101: taking the mean square error and structural similarity between the training current frame and the corresponding training reconstructed frame as a loss function, and using the loss function to train the implicit neural representation decoder;

[0076] S102: quantizing, pruning and weight entropy encoding the trained implicit neural representation decoder.

[0077] S220: Utilize the implicit neural representation decoder to decode according to the embedded vector to obtain a reconstructed frame corresponding to the current frame.

[0078] Further, S220 may include the following steps S221-S222:

[0079] S221: Decoding the embedding vector using the first decoding block in the implicit neural representation decoder to obtain a first decoded frame; wherein the implicit neural representation decoder includes a plurality of decoding blocks connected in sequence;

[0080] S222: Repeat the step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame, until the last decoding block in the implicit neural representation decoder outputs the corresponding first decoded frame as the first reconstructed frame of the frame index.

[0081] Furthermore, S220 may further include the following steps S223:

[0082] S223: Input the time features into each of the decoding blocks respectively.

[0083] Furthermore, S221 may be more specifically: using the first decoding block in the implicit neural representation decoder to decode the embedding vector and the temporal feature to obtain a second decoded frame.

[0084] S222 may be more specifically: inputting the second decoded frame and the time feature output by the current decoding block into the next connected decoding block for decoding to obtain the corresponding second decoded frame.

[0085] As another optional implementation, S220 may further include the following steps S224 to S226:

[0086] S224: Outputting the first decoded frames of different resolutions using output layers of the respective decoding blocks;

[0087] S225: Determine at least one of the first decoded frames of different resolutions as a second reconstructed frame corresponding to the frame index;

[0088] S226: Construct a video according to each of the second reconstructed frames corresponding to different frame indexes.

[0089] The embodiment of the present application may further include step S230 of reconstructing the current frame of the same resolution to obtain reconstructed frames of different bit rates to construct a video. Specifically, S230 may include the following steps S231 to S234:

[0090] S231: Subtract the current frame from the reconstructed frame to obtain a residual;

[0091] S232: Input the residual to the encoder for encoding to obtain residual features;

[0092] S233: inputting the residual feature into the implicit neural representation decoder, and then adding the decoded output of the implicit neural representation decoder to the reconstructed frame to obtain the current reconstructed frame;

[0093] S234: Return to the step of subtracting the current frame from the reconstructed frame to obtain a residual, until the current reconstructed frame reaches the target bit rate.

[0094] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0095] Reference Figure 3 , this embodiment provides an example flowchart of a video encoding method based on implicit neural representation.

[0096] This embodiment proposes an implicit neural representation video coding scheme that fuses the time information extracted from the frame index and the spatial information extracted from the current frame, thereby realizing implicit neural representation video coding that fuses temporal and spatial information.

[0097] Specifically, this embodiment may include the following solutions:

[0098] 1. At the encoding end, a spatial feature extraction network is introduced to extract the features of the current frame from the input current frame. The spatial feature extraction network can be any network that helps to extract spatial features, such as convolutional neural networks and visual transformers. The frame index t is input into the temporal feature arrangement network, which can be a multi-layer perceptron, etc., and outputs a temporal feature with a similar size to the spatial feature but a different number of channels.

[0099] 2. The obtained temporal and spatial features are fused through a spatiotemporal feature fusion network, which can be any network that facilitates spatiotemporal feature fusion, such as an attention feature fusion network. The spatiotemporal feature fusion network will output an embedding vector containing spatiotemporal features, which will be used as the input of the decoder and together with the weights of the decoder network as the data embedding for storing video data.

[0100] 3. At the decoding end, the embedded vector will replace the frame index as the input of the decoder. By inputting the embedded vector corresponding to different frame indexes into the decoding network, the reconstructed frame corresponding to the frame index can be decoded.

[0101] Reference Figure 4 , this embodiment provides a flowchart of an enhanced implicit neural representation video coding method that integrates spatiotemporal information.

[0102] Specifically, in order to further improve the fitting ability of the scheme of this embodiment, the time information can also be sent to the decoder network to further improve the video coding efficiency of the implicit neural representation decoder. The enhanced scheme incorporates the time information arrangement network into the data embedding area, which enhances the representation ability of the network. However, this scheme requires both the frame index and the embedding vector as inputs during decoding, and the reasoning complexity increases. In actual application, it can be selected according to specific needs.

[0103] The implicit neural representation decoder in the above scheme uses the mean squared error (MSE) and structure similarity index measure (SSIM) between the original current frame and the reconstructed frame as the loss function during the training process. After the training, the neural network in the data embedding area is quantized, pruned, and weighted entropy encoded to achieve efficient video encoding.

[0104] In addition, this embodiment also provides another optional implementation method, which can realize layered video coding based on implicit neural representation.

[0105] Reference Figure 5 , Figure 5 Flowchart of the implicit neural representation video encoding method for multi-resolution output.

[0106] Specifically, this embodiment can use implicit neural representation-based layered video coding to output videos of different resolutions and different bit rates at the same resolution.

[0107] The video coding scheme based on implicit neural representation embeds most of the video data into the weights of the neural network, and the process of reconstructing the video by the network gradually expands the size of the output features from the embedding vector. The amount of video data stored in the network gradually decreases from the front to the back, so taking out the intermediate variables during the decoding process can also reconstruct high-quality videos. Specifically, the intermediate variables in the decoding process are taken out, the number of channels in the input and output layers is reduced, and finally a 3-channel video frame with the same width and height as the intermediate variables is output. During the training of the neural network, the output layers corresponding to the video frames of each resolution can be trained together, that is, output videos of different resolutions can be obtained through one encoding, and targeted network structures can also be designed for different resolution output targets. During the video decoding process, if a video with a higher resolution is no longer needed, the network part after the output can no longer be calculated, which can further save computing costs.

[0108] This embodiment can also output videos with different bit rates at the same resolution. Figure 6 , Figure 6 Flowchart of the implicit neural representation video coding method for multi-bitrate output.

[0109] Specifically, outputting videos with different bit rates at the same resolution can be achieved through multiple implicit neural representation-based video encoders. Specifically, the code obtained by the first encoder is decoded to obtain reconstructed frame 1, the current frame is subtracted from the reconstructed frame to obtain the residual, and the residual is encoded by the second encoder, and after decoding, it is added to the reconstructed frame 1 to obtain a higher quality reconstructed frame 2, and so on. As the reconstruction quality will gradually improve and the residual will gradually decrease, the complexity of the encoder used will gradually decrease.

[0110] In summary, combined Figure 5 and Figure 6 The method shown can realize a multi-resolution and multi-bitrate layered video coding scheme, that is, a layered video coding scheme based on implicit neural representation.

[0111] This embodiment can also provide a test example.

[0112] For example, the products and equipment of the test example are as follows:

[0113] 1. Hardware environment:

[0114] Linux server, 6G or above video memory.

[0115] 2. Software environment:

[0116] Ubuntu 20.04 system;

[0117] Python>=3.8;

[0118] PyTorch>=1.8.1;

[0119] Torchvision;

[0120] Pytorch video.

[0121] The dataset is tested using the UVG dataset commonly used in the field of video encoding and decoding. The neural network is trained using the video to be encoded itself, and the video data is embedded into the weights of the implicit neural representation decoder.

[0122] In terms of evaluation indicators, the most commonly used indicators in the field of video compression are followed, namely the average bit rate per pixel (bits per pixel, bpp) of the output video, the peak signal-to-noise ratio (Peak Signal-to-Noise Ratio, PSNR) of the original video and the compressed video, and the Structure Similarity Index Measure, SSIM.

[0123] Reference Figure 7 The embodiment of the present application further provides a video encoding device based on implicit neural representation, which can implement the above-mentioned video encoding method based on implicit neural representation, and the device includes:

[0124] A spatiotemporal encoding unit, configured to input a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features;

[0125] a spatiotemporal input unit for inputting the embedding vector into an implicit neural representation decoder;

[0126] The spatiotemporal reconstruction unit is used to use the implicit neural representation decoder to decode according to the embedded vector to obtain a reconstructed frame corresponding to the current frame.

[0127] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0128] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned video encoding method based on implicit neural representation when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0129] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0130] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0131] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0132] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the video encoding method based on implicit neural representation in the embodiment of this application;

[0133] Input / output interface 803, used to implement information input and output;

[0134] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0135] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0136] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0137] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned video encoding method based on implicit neural representation.

[0138] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0139] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0141] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0142] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0144] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0145] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0147] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0150] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video coding method based on implicit neural representation, characterized in that: The method comprises the following steps: Inputting a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features; Inputting the embedding vector into an implicit neural representation decoder; The implicit neural representation decoder is used to decode the embedded vector to obtain a reconstructed frame corresponding to the current frame.

2. The video coding method based on implicit neural representation according to claim 1, characterized in that: The step of inputting a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features comprises the following steps: Extracting the temporal features of the frame index using a temporal feature arrangement network in the encoder; Extracting spatial features of the current frame using a spatial feature extraction network in the encoder; The temporal features and the spatial features are fused using a spatiotemporal feature fusion network in the encoder to obtain the embedding vector containing spatiotemporal features.

3. The video coding method based on implicit neural representation according to claim 1, characterized in that: The method of using the implicit neural representation decoder to decode according to the embedded vector to obtain a reconstructed frame corresponding to the current frame includes the following steps: Decoding the embedding vector using the first decoding block in the implicit neural representation decoder to obtain a first decoded frame; wherein the implicit neural representation decoder includes a plurality of the decoding blocks connected in sequence; Repeat the step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame, until the last decoding block in the implicit neural representation decoder outputs the corresponding first decoded frame as the first reconstructed frame of the frame index.

4. The video coding method based on implicit neural representation according to claim 3, characterized in that: The method further comprises the following steps: Inputting the time features into each of the decoding blocks respectively; The step of decoding the embedded vector using the first decoding block in the implicit neural representation decoder to obtain a first decoded frame comprises the following steps: Decoding the embedding vector and the temporal feature using the first decoding block in the implicit neural representation decoder to obtain a second decoded frame; The step of inputting the decoded frame output by the current first decoding block into the next connected decoding block for decoding to obtain the corresponding first decoded frame comprises the following steps: The second decoded frame and the time feature output by the current decoding block are input to the next connected decoding block for decoding to obtain the corresponding second decoded frame.

5. The video coding method based on implicit neural representation according to claim 3, characterized in that: The method further comprises the following steps: Outputting the first decoded frames of different resolutions using output layers of the respective decoding blocks; Determine at least one of the first decoded frames of different resolutions as a second reconstructed frame corresponding to the frame index; The video is constructed according to each of the second reconstructed frames corresponding to different frame indexes.

6. The video coding method based on implicit neural representation according to claim 1, characterized in that: The method further comprises the following steps: Subtracting the current frame from the reconstructed frame to obtain a residual; Inputting the residual into the encoder for encoding to obtain residual features; Inputting the residual feature into the implicit neural representation decoder, and then adding the decoded output of the implicit neural representation decoder to the reconstructed frame to obtain the current reconstructed frame; Return to the step of subtracting the current frame from the reconstructed frame to obtain a residual until the current reconstructed frame reaches a target bit rate.

7. The video coding method based on implicit neural representation according to any one of claims 1 to 6, characterized in that: Before inputting the embedding vector into the implicit neural representation decoder, the method further comprises the step of training the implicit neural representation decoder, wherein the step of training the implicit neural representation decoder comprises the following steps: Using the mean square error and structural similarity between the training current frame and the corresponding training reconstructed frame as a loss function, and using the loss function to train the implicit neural representation decoder; The trained implicit neural representation decoder is quantized, pruned and weighted entropy encoded.

8. A video coding device based on implicit neural representation, characterized in that: The device comprises: A spatiotemporal encoding unit, configured to input a frame index and a current frame corresponding to the frame index into an encoder to obtain an embedding vector containing spatiotemporal features; a spatiotemporal input unit for inputting the embedding vector into an implicit neural representation decoder; The spatiotemporal reconstruction unit is used to use the implicit neural representation decoder to decode according to the embedded vector to obtain a reconstructed frame corresponding to the current frame.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the implicit neural representation-based video encoding method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video encoding method based on implicit neural representation as claimed in any one of claims 1 to 7 is implemented.

Images