Video coding method, video decoding method, video coding device, video decoding device and communication equipment

Abstract

The invention relates to a video coding method, a video decoding method, a video coding device, a video decoding device and communication equipment. According to the method, a plurality of video features corresponding to a video frame are fused and encoded to obtain an encoding feature, and then the encoding feature is sent to a receiving device. According to the method, coding is carried out after different video features are fused, the capability of coding multiple and / or multi-scale video features is achieved, the coding efficiency of the video data is improved to a certain extent, and therefore when a large number of video features need to be coded and then transmitted, the coding efficiency of the video data is improved. And the video data transmission rate can be improved to a certain extent. Besides, due to the fact that fusion processing is carried out on different video features, semantic redundant features existing in the multiple video features can be removed, the coding effect can be improved to a certain degree, and then the restoration degree of the features obtained after decoding and restoration are carried out based on the coding features in the later period can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video encoding method, a video decoding method, an apparatus and a communication device. Background Art

[0002] With the popularization of deep learning, more and more video images are oriented to machine vision tasks. Therefore, encoding and decoding algorithms for machine vision tasks have become a new trend in academia, industry, and standards.

[0003] However, the traditional codec algorithm directly encodes the video data. When the codec algorithm is applied to the machine vision task network, it is necessary to encode the features output by the feature extraction network. In this scenario, the traditional codec algorithm has the problem of low coding efficiency. Summary of the invention

[0004] The embodiments of the present application provide a video encoding method, a video decoding method, an apparatus and a communication device, which can encode and decode multiple quantities and / or multi-scale video features corresponding to video frames, thereby improving the encoding and decoding efficiency of video data and further improving the transmission rate of video data.

[0005] In a first aspect, a video encoding method is provided, the method being applied to a sending device, the method comprising:

[0006] Fusing and encoding multiple video features corresponding to the video frame to obtain encoded coding features;

[0007] The encoded feature is sent to a receiving end.

[0008] In a second aspect, a video decoding method is provided, the method being applied to a receiving device, the method comprising:

[0009] Receive a coding feature sent by a sending device; the coding feature is obtained by the sending device fusing and encoding multiple video features corresponding to a video frame;

[0010] The encoded features are decoded and restored to obtain a plurality of first restored video features corresponding to the video frame.

[0011] According to a third aspect, a video encoding device is provided, the device comprising:

[0012] A fusion coding module is used to fuse and encode multiple video features corresponding to the video frame to obtain a coding feature;

[0013] A sending module is used to send the coding feature to a receiving device.

[0014] According to a fourth aspect, a video encoding device is provided, the device comprising:

[0015] A receiving module, used to receive a coding feature sent by a sending device; the coding feature is obtained by the sending device fusing and encoding multiple video features corresponding to a video frame;

[0016] The decoding and restoration module is used to decode and restore the encoded features to obtain multiple first restored video features corresponding to the video frame.

[0017] A fifth aspect is a communication device, the communication device comprising a transmitter and a processor;

[0018] The processor is used to fuse and encode multiple video features corresponding to the video frame to obtain a coded feature;

[0019] The transmitter is used to send the coding feature to a receiving device.

[0020] A sixth aspect, a communication device, the communication device comprising a receiver and a processor;

[0021] The receiver is used to receive the coding features sent by the sending device; the coding features are obtained by the sending device fusing and encoding multiple video features corresponding to the video frame;

[0022] The processor is used to decode and restore the encoded features to obtain multiple first restored video features corresponding to the video frame.

[0023] In a seventh aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0024] In an eighth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the second aspect are implemented.

[0025] In a ninth aspect, a computer program product comprises a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0026] In a tenth aspect, a computer program product comprises a computer program, which, when executed by a processor, implements the steps of the method described in the second aspect.

[0027] The above-mentioned video encoding method, video decoding method, device and storage medium obtain encoding features by fusing and encoding multiple video features corresponding to the video frame, and then send the encoding features to the receiving device. This method has the ability to encode video features of multiple numbers and / or multiple scales by fusing different video features for encoding, which improves the encoding efficiency of video data to a certain extent, and further, when a large number of video features need to be encoded and transmitted, the rate of video data transmission can be increased to a certain extent. In addition, since different video features are fused, semantically redundant features existing in multiple video features can be removed, which can improve the encoding effect to a certain extent, and further, the restoration degree of features decoded and restored based on the encoding features can be improved in the later stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of an information transmission system of a video encoding method in an embodiment;

[0029] Figure 2 is a schematic flow chart of a video encoding method in one embodiment;

[0030] Figure 3 is a schematic flow chart of a video encoding method in another embodiment;

[0031] Figure 3A is a schematic diagram of the structure of a first encoding and decoding network in an embodiment;

[0032] Figure 4 is a schematic flow chart of a video encoding method in another embodiment;

[0033] Figure 4A is a schematic diagram of the structure of a first encoding subnetwork in an embodiment;

[0034] Figure 5 is a schematic flow chart of a video encoding method in another embodiment;

[0035] Figure 5A is a schematic diagram of the structure of a first convolution module in one embodiment;

[0036] Figure 5B is a schematic diagram of the structure of a first encoding and decoding network in an embodiment;

[0037] Figure 6 is a schematic flow chart of a video encoding method in another embodiment;

[0038] Fig. 6A A schematic diagram of the structure of a decoding subnetwork in another embodiment;

[0039] Figure 7 is a schematic flow chart of a video encoding method in another embodiment;

[0040] Fig. 7A is a schematic structural diagram of a first deconvolution module in one embodiment;

[0041] Figure 7B is a schematic diagram of the structure of a second encoding and decoding network in another embodiment;

[0042] Figure 8 is a schematic flow chart of a video encoding method in another embodiment;

[0043] Fig. 8A A schematic diagram of the structure of a prediction network in one embodiment;

[0044] Fig. 9 is a schematic flow chart of a video encoding method in another embodiment;

[0045] Fig.9A A schematic diagram of the structure of a motion codec network in one embodiment;

[0046] Fig.10 is a schematic flow chart of a video encoding method in another embodiment;

[0047] Fig. 10A is a schematic diagram of the structure of a motion coding network in one embodiment;

[0048] Fig.11 is a schematic flow chart of a video encoding method in another embodiment;

[0049] Fig.11A is a schematic diagram of the structure of a motion decoding network in one embodiment;

[0050] Fig.12 is a schematic flow chart of a video encoding method in another embodiment;

[0051] Fig. 12A is a schematic structural diagram of a second deconvolution module in another embodiment;

[0052] Fig.13 is a schematic flow chart of a video encoding method in another embodiment;

[0053] Fig.13A A schematic diagram of the structure of an inter-frame prediction network in another embodiment;

[0054] Fig. 13B is a schematic diagram of the structure of another inter-frame prediction network in another embodiment;

[0055] Fig.14 is a schematic flow chart of a video encoding method in another embodiment;

[0056] Fig.15is a schematic flow chart of a video encoding method in another embodiment;

[0057] Fig.15A A schematic diagram of the structure of a residual coding and decoding network in one embodiment;

[0058] Fig. 15B is a schematic diagram of the structure of a residual coding network in one embodiment;

[0059] Fig.16 is a schematic flow chart of a video encoding method in another embodiment;

[0060] Fig.16A A schematic diagram of the structure of a residual coding and decoding network in another embodiment;

[0061] Fig. 16B is a schematic diagram of the structure of a second encoding and decoding network in another embodiment;

[0062] Fig.17 is a schematic flow chart of a video encoding method in another embodiment;

[0063] Fig.17A A schematic diagram of the structure of a residual decoding network in another embodiment;

[0064] Fig.18 is a schematic flow chart of a video encoding method in another embodiment;

[0065] Fig.18A A schematic diagram of the structure of a residual decoding network in another embodiment;

[0066] Fig.19 is a schematic flow chart of a video encoding method in another embodiment;

[0067] Fig.19A A schematic diagram of the structure of a coding and decoding network in another embodiment;

[0068] Fig. 20 is a schematic flow chart of a video decoding method in one embodiment;

[0069] Fig.21 is a flow chart of a video decoding method in another embodiment;

[0070] Fig. 22 is a flow chart of a video decoding method in another embodiment;

[0071] Fig.22A is a schematic diagram of the structure of a second decoding network in another embodiment;

[0072] Fig.23 is a flow chart of a video decoding method in another embodiment;

[0073] Fig.24 is a flow chart of a video decoding method in another embodiment;

[0074] Fig.24A is a schematic diagram of the structure of another second decoding network in another embodiment;

[0075] Fig.25 is a flow chart of a video decoding method in another embodiment;

[0076] Fig.26 is a flow chart of a video decoding method in another embodiment;

[0077] Fig. 27 is a flow chart of a training method in one embodiment;

[0078] Fig.28 is a schematic structural diagram of a video encoding device in another embodiment;

[0079] Fig.29 FIG. 4 is a schematic diagram of the structure of a video decoding device in another embodiment. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0081] Figure 1 A schematic diagram of an information transmission system of a video encoding method provided in an embodiment of the present application. Figure 1 As shown, in the information transmission system, there are a sending device 101, a receiving device 102 and a task processing device 103. Among them, the sending device 101 and the receiving device 102 transmit data through the network. Specifically, after the sending device 101 performs feature extraction, feature fusion and feature encoding on the video data to be transmitted, the encoded video code stream data is transmitted to the receiving device 102 through the network. After receiving the encoded video code stream data, the receiving device 102 decodes and restores the features of the encoded video code stream data to obtain the restored feature data corresponding to the above video data, and then can perform corresponding machine intuitive task processing based on the restored feature data to obtain the processing result.

[0082] Among them, the sending device 101 access network device 300 can be a base station (Base Transceiver Station, BTS) in Global System of Mobile communication (GSM) or Code Division Multiple Access (CDMA), or a base station (NodeB, NB) in Wideband Code Division Multiple Access (WCDMA), or an evolved base station (Evolutional Node B, eNB or eNodeB) in LTE, or a relay station or access point, or a base station in a 5G network, etc., which is not limited here.

[0083] The first user terminal 100 and the receiving device 102 may be terminals, which may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Optionally, the sending device 101 and the receiving device 102 may also be servers, which may be implemented as independent servers or server clusters consisting of multiple servers. Optionally, the sending device 101 and the receiving device 102 may also be wireless terminals, which may be devices that provide voice and / or other business data connectivity to users, or handheld devices with wireless connection functions, or other processing devices connected to wireless modems. The wireless terminal may be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device. A wireless terminal may also be referred to as a system, a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, a remote terminal, an access terminal, a user terminal, a user agent, or a user device or user equipment, without limitation herein.

[0084] In traditional technology, when video coding and decoding technology is aimed at human vision tasks, mean square error, peak signal-to-noise ratio, and structural similarity are usually used as evaluation indicators to measure fidelity to optimize the coding and decoding algorithm. In more than 20 years of development, the traditional video coding and decoding framework algorithm has become mature and has been widely used in various fields (live broadcast, short video, video surveillance, etc.). With the popularization of deep learning, more and more video images are oriented to machine vision tasks rather than human vision tasks. However, although there is a certain correlation between the fidelity of image videos and the task performance of machine vision, they are not completely consistent. Therefore, the traditional coding and decoding algorithm optimized with fidelity as the evaluation indicator is not the optimal solution in the application scenario of machine vision. Therefore, in academia, industry, and standards, coding and decoding algorithms for machine vision and corresponding optimization methods are becoming a new trend.

[0085] In the existing transmission network of machine vision tasks, the feature extraction network in the transmission network will extract features from the input video data, and then the task network connected to it will perform task processing based on the extracted features. The feature extraction network usually includes a feature extraction intermediate layer of the feature pyramid network (FPN) structure, which will output multiple features and / or multi-scale features. If it is necessary to add codec technology to the above transmission network to improve the video data transmission rate and security, the codec is required to encode and decode the multiple features and / or multi-scale features output by the feature extraction intermediate layer of the FPN structure, but the research and industry circles do not have a suitable codec for multi-scale video features. That is, the existing video codec algorithm is not suitable for the existing transmission network of machine vision tasks. Even if a conventional codec is used to encode and decode the multiple features and / or multi-scale features output by the feature extraction intermediate layer, in the encoding and decoding process, the encoder is only used to process the video features one by one. Based on this, there is a problem of low coding efficiency in traditional video coding and decoding technology.

[0086] Based on the above-mentioned traditional technology, the embodiment of the present application provides a video encoding method, which can encode and decode video features of multiple quantities and / or multiple scales, and can improve the efficiency of feature encoding to a certain extent, thereby increasing the rate of video data transmission.

[0087] It should be noted that the beneficial effects brought about by the embodiments of the present application or the technical problems solved are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0088] Before introducing the specific embodiments of the present invention, the professional terms involved in the present invention are explained first:

[0089] Compression: Reducing the size of data using some technical method.

[0090] Decompression: Restoring data to its original form and size through some technical method after reducing its size.

[0091] Coding: It is a specific technical method of compression.

[0092] Decoding: is a specific method of decompression.

[0093] For the sake of convenience, this article will no longer strictly distinguish the subtle differences between compression / encoding and decompression / decoding.

[0094] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0095] In one embodiment, Figure 2 As shown, a video encoding method is provided, which is applied to Figure 1 The sending device in the example is used as an example to illustrate, including the following steps:

[0096] S201, fusing and encoding multiple video features corresponding to a video frame to obtain a coded feature.

[0097] The video frame is a video frame to be transmitted. The multiple video features may be multiple video features of the same scale, or multiple video features of different scales, or video features of different channels, or video features of the same channel.

[0098] In the embodiment of the present application, the sending device may first obtain the original video to be transmitted, and process the original video in segments or frames to obtain each video frame to be transmitted, and then process it frame by frame. Specifically, a feature extraction network or a feature extraction algorithm may be used to extract features from the video frame of the current frame to obtain multiple video features corresponding to the video frame; and a feature fusion network or a feature fusion algorithm may be used to fuse multiple video features to obtain a video feature, and then a coding network, or an encoder, or a coding algorithm may be used to encode the fused video features to obtain a coding feature. It should be noted that the above fusion processing may be a scale fusion of features of different scales or features of the same scale, or a channel fusion of features of different channels or features of the same channel. In addition, since there may be a large number of semantically redundant features in the multiple video features extracted by the features, the above fusion processing essentially fuses multiple video features to achieve the effect of removing semantically redundant features; optionally, in the above fusion processing process, the semantically redundant features in the multiple video features may also be removed first, and then the multiple video features after the feature removal processing are fused to obtain a fused video feature. The above-mentioned encoding processing may be code stream encoding processing, that is, performing code stream data conversion on the fused video features to obtain encoding features of the code stream data format; optionally, the above-mentioned encoding processing may also be message encoding processing, that is, performing message data conversion on the fused video features to obtain encoding features of the message data format; optionally, the above-mentioned encoding processing may also be encoding processing of other data conversion types, that is, performing data conversion on other data format types to obtain encoding features of other data format types. The data conversion of other data format types may be determined according to the data transmission method between the sending device and the receiving device, which is not limited here.

[0099] S202, sending the coding feature to a receiving device.

[0100] In the embodiment of the present application, when the sending device obtains the coding feature based on the above steps, the coding feature can be transmitted to the receiving device through the network, or transmitted to the receiving device through a wired transmission medium. When the receiving device receives the coding feature, it can first use a decoding network, a decoder, or a decoding algorithm to decode the coding feature, and then use a feature restoration network or a feature restoration algorithm to perform feature restoration processing on the decoded feature to obtain multiple restored video features corresponding to the video frame. It should be noted that the above decoding process can correspond to the coding processing method of the sending device. For example, if the coding processing method of the sending device is code stream coding processing, the decoding process here is code stream decoding processing; if the above coding process is message coding processing, the decoding process here is message decoding processing; if the above coding process is coding processing of other data types, the decoding process here is decoding processing of other data types. The above feature restoration processing can be a restoration process of different scale features or the same scale features, or it can be a restoration process of different channel features or the same channel features.

[0101] In the above-mentioned video encoding method, the sending device obtains the encoding features by fusing and encoding the multiple video features corresponding to the video frame, and then sends the encoding features to the receiving device. This method has the ability to encode multiple numbers and / or multiple scales of video features by fusing different video features for encoding, thereby improving the encoding efficiency of video data to a certain extent, and further, when a large number of video features need to be encoded and transmitted, the rate of video data transmission can be improved to a certain extent. In addition, since different video features are fused, semantically redundant features existing in multiple video features can be removed, which can improve the encoding effect to a certain extent, and further, the restoration degree of the features decoded and restored based on the encoding features can be improved.

[0102] In one embodiment, a method for fusing and encoding multiple video features is provided, such as Figure 3 As shown, the above S201 "merging and encoding multiple video features corresponding to the video frame to obtain encoded features" may include:

[0103] S301, determining the type of the video frame.

[0104] The types of video frames include key frames and non-key frames.

[0105] In an embodiment of the present application, when the sending device obtains the original video to be transmitted, the original video can be first segmented or framed, that is, the type of each video frame in the original video can be determined first. For example, a video frame can be selected as a key frame every K frames in the original video, and the rest can be used as non-key frames. K can be determined based on actual coding accuracy, coding accuracy, coding efficiency and other indicators. For example, if higher coding accuracy or coding accuracy is required, the number of K can be appropriately increased to increase the number of non-key frames, thereby improving coding accuracy or coding accuracy; if higher coding efficiency is required, the number of K can be appropriately reduced to increase the number of key frames, thereby improving coding efficiency.

[0106] S302, inputting multiple video features into a codec network corresponding to the type of the video frame for fusion and encoding to obtain encoded features.

[0107] Among them, the codec network can be a machine learning network or a neural network, which can be obtained in advance through training, so that the codec network has the ability to fuse and encode multiple input video features through training. Different types of video frames correspond to different codec networks. For example, key frames correspond to one codec network, and non-key frames correspond to another codec network. The codec networks corresponding to key frames and non-key frames have different structures but the same functions, that is, the codec network corresponding to key frames is used to fuse and encode key frames to obtain corresponding encoding features, and the codec network corresponding to non-key frames is used to fuse and encode non-key frames to obtain corresponding encoding features. The correspondence between the type of video frame and the codec network can be predetermined and stored in the sending device as a mapping relationship, so that the sending device can determine the corresponding codec network according to different types of video frames.

[0108] In an embodiment of the present application, when the sending device determines the type of each video frame to be transmitted, it can further select a codec network corresponding to the type of each video frame, and use the selected codec network to successively fuse and encode multiple video features corresponding to each video frame to obtain coding features. For example, the sending device can use a codec network to fuse and encode multiple video features corresponding to key frames to obtain coding features corresponding to key frames; the sending device can use another codec network to fuse and encode multiple video features corresponding to non-key frames to obtain coding features corresponding to non-key frames. It should be noted that when the sending device obtains the coding feature, the coding feature can be type-identified according to the type of the current video frame, so that when the receiving device receives the coding feature, it can determine the type of video frame corresponding to the coding feature according to the type identification of the coding feature.

[0109] The embodiment of the present application selects the corresponding codec network for fusion and encoding according to the type of different video frames, thereby achieving matching processing between the type of video frame and the codec network, which can improve the encoding efficiency and encoding accuracy to a certain extent.

[0110] The following embodiments will specifically describe the method of fusing and encoding different types of video frames, that is, provide two implementation methods of the above S302.

[0111] The first one is that when the sending device executes the above S302 "inputting multiple video features into the codec network corresponding to the type of video frame for fusion and encoding to obtain coding features", the specific execution steps are: when the type of the video frame is a key frame, inputting multiple video features into the first codec network for fusion and encoding to obtain coding features.

[0112] The first codec network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the first codec network may have the ability to fuse and encode multiple video features corresponding to key frames through training.

[0113] In an embodiment of the present application, when the sending device determines that the type of video frame to be transmitted is a key frame, it can further select a codec network corresponding to the key frame, that is, a first codec network, and use the selected first codec network to successively perform fusion processing and encoding processing on multiple video features corresponding to the key frame to obtain encoding features.

[0114] Furthermore, a specific structure of the first encoding and decoding network is provided, such as Figure 3A As shown, the first codec network includes a first coding sub-network and a first encoder, and the output end of the first coding sub-network is connected to the input end of the first encoder. Under the first codec network of this structure, the above method for fusing and encoding multiple video features is as follows: Figure 4 As shown, including:

[0115] S401, input multiple video features into a first encoding sub-network for fusion processing to obtain a first fusion feature.

[0116] The first encoding subnetwork may be a machine learning network or a neural network, which may be obtained in advance through training, so that the first encoding subnetwork has the ability to fuse multiple input video features through training. Optionally, the first encoding subnetwork may be a side keyframe feature encoder (corresponding English expression: Side KeyFrame Feature Encoder).

[0117] In an embodiment of the present application, when the sending device obtains multiple video features corresponding to the key frame and the first coding subnetwork, the multiple video features can be input into the first coding subnetwork to fuse features of different scales to obtain a fused first fused feature; optionally, the sending device can also input multiple video features into the first coding subnetwork to fuse features of different channels to obtain a fused first fused feature; optionally, the sending device can also input multiple video features into the first coding subnetwork to fuse redundant semantic features to obtain a fused first fused feature.

[0118] S402: Input the first fusion feature into a first encoder for encoding to obtain an encoded feature.

[0119] The first encoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the first encoder has the ability to encode the input video features through training. Optionally, the first encoder may be a hyper-prior feature encoder (corresponding English expression: Hyper Encoder) in a hyper-prior feature codec (corresponding English expression: Hyper Coder).

[0120] In the embodiment of the present application, when the sending device obtains the first fusion feature, the first fusion feature can be further input into the first encoder for code stream encoding processing to obtain the encoded coding feature; optionally, the sending device can also input the first fusion feature into the first encoder for message encoding processing to obtain the encoded coding feature; optionally, the sending device can also input the first fusion feature into the first encoder for encoding processing of other data format types to obtain the encoded coding feature. In the embodiment of the present application, a first encoding subnetwork for fusion features is set to realize fusion processing of multiple video features, thereby facilitating the subsequent encoding of the fused features and thereby improving the encoding efficiency.

[0121] Furthermore, a specific structure of the first encoding subnetwork is provided, such as Figure 4A As shown, the first coding sub-network includes multiple first convolution modules and a first splicing module, and the output end of each first convolution module is connected to the input end of the first splicing module. Under the first coding sub-network of this structure, the above S401 "inputs multiple video features into the first coding sub-network for fusion processing to obtain a first fusion feature" is as shown in Figure 5 As shown, including:

[0122] S501, input each video feature into the corresponding first convolution module for resizing, and obtain a plurality of first adjusted features after resizing.

[0123] The sizes of the first adjustment features are consistent. The first convolution module may be composed of a plurality of convolution layers, and the first convolution module is used to perform convolution processing on the input video features, thereby achieving dimensionality reduction processing or size adjustment processing on the input features. The first convolution module may be obtained in advance through training.

[0124] In an embodiment of the present application, when the sending device inputs multiple video features into the first coding subnetwork, each video feature can be input into the corresponding first convolution module in the first coding subnetwork for resizing, and the size of each video feature can be specifically adjusted to a uniform size, and the uniform size can be determined in advance according to the actual feature extraction requirements. For example, the first coding subnetwork includes three first convolution modules (#1 first convolution module, #2 first convolution module, #3 first convolution module), the #1 video feature is input into the #1 first convolution module, the #2 video feature is input into the #2 first convolution module, and the #3 video feature is input into the #3 first convolution module. The #1 first convolution module resizes the #1 video feature to obtain the adjusted #1 video feature; the #2 first convolution module resizes the #2 video feature to obtain the adjusted #2 video feature; the #3 first convolution module resizes the #3 video feature to obtain the adjusted #3 video feature.

[0125] Optionally, the size adjustment may specifically be to adjust the width and height of the video frame. For example, a key frame is passed through a feature extraction network to extract N features of different scales: (C 1 , H 1 , W 1 ), (C 2 , H 2 , W 2 ), ...., (C N , H N , W N ), where C i Indicates the number of channels, H i Indicates the height of the i-th channel, W i Represents the width of the i-th channel. If the N features of different scales are resized to a uniform height and width, they are resized to a uniform size (C i , H, W), and finally we can get N first adjusted features of the same scale: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W).

[0126] S502: Input multiple first adjustment features into a first stitching module for stitching to obtain a first fusion feature.

[0127] The first concatenation module is used to concatenate multiple input features, and specifically can concatenate multiple features by channel concatenation, size concatenation, or semantic feature concatenation. The first concatenation module can be obtained in advance through training. The first concatenation module can be expressed in English as Feature Concatenation.

[0128] In the embodiment of the present application, the sending device may input multiple first adjustment features into the first splicing module for splicing, so as to splice the multiple first adjustment features into one feature, that is, to obtain a first fusion feature. Optionally, when channel splicing is performed on multiple first adjustment features, accumulation and channel processing are performed on the same or different channels to obtain the first fusion feature. For example, the N first adjustment features obtained in the above example: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W) are concatenated, if the number of channels of the above N first adjustment features is the same, a feature with a size of (C*N, H, W) can be obtained after concatenation; if the number of channels of the above N first adjustment features is different, a feature with a size of (C 1 +C 2 ....C N , H, W) characteristics.

[0129] Optionally, a specific structure of a first convolution module is provided, such as Figure 5A As shown, the first convolution module includes multiple first linear convolution layers, multiple first linear convolution layers are cascaded, and each first linear convolution layer performs convolution processing on the input features. Each first linear convolution layer is used to perform convolution and linear activation processing on the features output by the previous stage. Among them, each first linear convolution layer can be represented by Conv+LeakyReLU.

[0130] Under the first convolution module of this structure, the above-mentioned S501 method for resizing includes: inputting each video feature into the first first linear convolution layer cascaded in the corresponding first convolution module, the first linear convolution layer performs convolution and linear activation processing on the input video feature, outputs the processed video feature, and inputs the processed video feature into the second first linear convolution layer for convolution and linear activation processing, outputs the video feature after secondary processing, and so on, layer by layer convolution and linear activation processing, until the last first linear convolution layer outputs the last processed feature, that is, the first adjusted feature is obtained. The first convolution module can adjust the size of the input feature through layer by layer convolution and linear activation processing, and can also perform dimensionality reduction processing on the input feature, so as to obtain the resized feature.

[0131] Optionally, another specific structure of the first encoding and decoding network is also provided, such as Figure 5B As shown, in Figure 3A Based on the first codec network shown in FIG. , the first codec network further includes a first decoder and a decoding sub-network, and the output end of the first decoder is connected to the input end of the decoding sub-network. Under the first codec network of this structure, as shown in FIG. Figure 6 As shown, Figure 2 The method described in the embodiment may also include the following steps:

[0132] S601, inputting the encoded feature into a first decoder for decoding to obtain a decoded first decoded feature.

[0133] The first decoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the first decoder has the ability to decode the input video features through training. Optionally, the first decoder may be an edge key frame feature encoder (corresponding English expression: Side Key Frame Feature Decoder) in a hyper-prior video feature decoder (corresponding English expression: Hyper Coder).

[0134] In the embodiment of the present application, when the sending device is based on the above Figure 2-Figure 5 After the method described in the embodiment obtains the coding feature, the coding feature can be further input into the first decoder for code stream decoding processing to obtain the first decoding feature after decoding; optionally, the sending device can also input the coding feature into the first decoder for message decoding processing to obtain the first decoding feature after decoding; optionally, the sending device can also input the coding feature into the first decoder for decoding processing of other data format types to obtain the first decoding feature after decoding. In the embodiment of the present application, a decoding subnetwork for restoring features is set up to realize the restoration processing of the fused features, so as to facilitate the subsequent other processing of the restored video frame using the current frame, or assist in the encoding processing of the next video frame, etc., thereby improving the encoding efficiency.

[0135] S602: Input the first decoded feature into the decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame.

[0136] The decoding subnetwork may be a machine learning network or a neural network, which may be obtained in advance through training, so that the decoding subnetwork has the ability to restore the input first decoding feature into multiple restored video features through training. Optionally, the decoding subnetwork may be a side key frame feature decoder (Side Key Frame Feature Decoder). It should be noted that the above restoration process may be the same as Figure 2-Figure 5The fusion processing described in the embodiment corresponds to the above, that is, when the fusion processing is a fusion processing of different sizes, the corresponding restoration processing is to restore the features to features of multiple sizes; when the fusion processing is a fusion processing of different channels, the corresponding restoration processing is to restore the features to features of multiple different channels; when the fusion processing is a fusion processing of features with different redundant semantics, the corresponding restoration processing is to restore the features to features with multiple different redundant semantics.

[0137] In the embodiment of the present application, when the sending device obtains the first decoding feature, the first decoding feature can be input into the decoding subnetwork to perform restoration processing of features of multiple different sizes, and multiple first restored video features can be obtained after restoration; optionally, the sending device can also input the first decoding feature into the decoding subnetwork to perform restoration processing of features of multiple different channels, and multiple first restored video features can be obtained after restoration; optionally, the sending device can also input the first decoding feature into the decoding subnetwork to perform restoration processing of features of multiple different redundant semantics, and multiple first restored video features can be obtained after restoration. In the embodiment of the present application, a decoding subnetwork for restoring features is provided, and restoration processing of fused features is realized, so as to facilitate other processing of the restored video frame using the current frame in the later stage, or assist in the encoding processing of the next video frame, etc., thereby improving the encoding efficiency.

[0138] Furthermore, a specific structure of the above decoding sub-network is provided, such as Fig. 6A As shown, the decoding subnetwork includes a first splitting module and a plurality of first deconvolution modules, and the output end of the first splitting module is respectively connected to the input end of each first deconvolution module. Under the decoding subnetwork of this structure, the above S602 "inputs the first decoding feature into the decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame" is as shown in FIG. Figure 7 As shown, including:

[0139] S701, input the first decoded feature into a first splitting module for feature splitting to obtain a plurality of first initial restored video features after splitting.

[0140] The first splitting module is used to split the input features, and specifically can split a first decoding feature into multiple features of the same size. The first splitting module can be obtained in advance through training. The first splitting module can be expressed in English as Feature Distribution.

[0141] In the embodiment of the present application, the sending device may input the first decoded feature into the first splitting module for feature splitting, so as to split the first decoded feature into multiple features of the same size, that is, to obtain multiple first initial restored video features. Figure 2-Figure 5When the fusion processing described in the embodiment is to perform fusion processing of different sizes on multiple video features, when the first decoding feature is split, it can be split into multiple first decoding features of the same size in different channels, or it can be split into multiple first decoding features of the same size in the same channel. For example, the first decoding feature: (C*N, H, W) is split into multiple first initial restored video features of different sizes in the same channel: (C, H, W), (C, H, W), ...., (C, H, W), or the first decoding feature (C 1 +C 2 ....C N , H, W) are split into multiple first initial restored video features of different sizes with different channels: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W).

[0142] S702: Input each first initial restored video feature into a corresponding first deconvolution module for restoration processing to obtain a plurality of first restored video features.

[0143] The first deconvolution module may be composed of a plurality of deconvolution layers, and the first deconvolution module is used to perform deconvolution processing on the input first initial restored video feature, thereby realizing the size adjustment processing on the first initial restored video feature. The first deconvolution module may be obtained in advance through training.

[0144] In an embodiment of the present application, when the sending device inputs multiple first initial restored video features into the decoding subnetwork, each first initial restored video feature can be input into the corresponding first deconvolution module in the decoding subnetwork for restoration processing. Specifically, size adjustment can be performed, that is, the size of the unified first initial restored video features is adjusted to different sizes, and the size of each adjusted feature is consistent with the size of multiple video features corresponding to the video frame before the previous fusion. For example, the decoding subnetwork includes three first deconvolution modules (#1 first deconvolution module, #2 first deconvolution module, #3 first deconvolution module), the #1 first initial restored video feature is input into the #1 first deconvolution module, the #2 first initial restored video feature is input into the #2 first deconvolution module, and the #3 first initial restored video feature is input into the #3 first deconvolution module. The #1 first deconvolution module restores the #1 first initial restored video feature to obtain the restored #1 first restored video feature; the #2 first deconvolution module restores the #2 first initial restored video feature to obtain the restored #2 first restored video feature; the #3 first deconvolution module restores the #3 first initial restored video feature to obtain the restored #3 first restored video feature.

[0145] Optionally, the above-mentioned resizing for restoration may specifically be to adjust the width and height of the first initial restored video feature. For example, if the plurality of first initial restored video features are: (C, H, W), (C, H, W), ..., (C, H, W), then after adjusting the width and height of the plurality of first initial restored video features accordingly, a plurality of first restored video features with different widths and heights may be obtained: (C 1 , H 1 , W 1 ), (C 2 , H 2 , W 2 ), ...., (C N , H N , W N ); or if the first initial restored video features are: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W), then after adjusting the widths and heights of the multiple first initial restored video features, multiple first restored video features with different widths and heights can be obtained: (C 1 , H 1 , W 1 ), (C 2 , H 2 , W 2 ), ...., (C N , H N , W N ).

[0146] Optionally, a specific structure of a first deconvolution module is provided, such as Fig. 7A As shown, the first deconvolution module includes multiple first linear deconvolution layers, multiple first linear deconvolution layers are cascaded, and each first linear deconvolution layer performs deconvolution processing on the input features. Each first linear deconvolution layer is used to perform deconvolution and linear activation processing on the features output by the previous stage. Among them, each first linear deconvolution layer can be represented by Subpel Conv+LeakyReLU.

[0147] Under the first deconvolution module of this structure, the above-mentioned S702 method for restoration processing includes: inputting each first initial restored video feature into the first first linear deconvolution layer cascaded in the corresponding first deconvolution module, the first linear deconvolution layer performs deconvolution and linear activation processing on the input first initial restored video feature, outputs the processed feature, and inputs the processed feature into the second first linear deconvolution layer for deconvolution and linear activation processing, outputs the second processed feature, and so on, layer by layer deconvolution and linear activation processing, until the last first linear deconvolution layer outputs the last processed feature, that is, the first restored video feature is obtained. The first deconvolution module can adjust the size of the input feature through layer by layer deconvolution and linear activation processing, thereby restoring the feature.

[0148] The above embodiment describes a method in which a sending device fuses and encodes key frames. The following embodiment describes another method in which a sending device fuses and encodes non-key frames.

[0149] The second type is that when the sending device executes the above S302 "inputting multiple video features into the codec network corresponding to the type of video frame for fusion and encoding to obtain coding features", the specific execution steps are: when the type of the input video frame is a non-key frame, inputting multiple video features into the second codec network for fusion and encoding to obtain coding features.

[0150] The second codec network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the second codec network may have the ability to fuse and encode multiple video features corresponding to non-key frames through training.

[0151] In an embodiment of the present application, when the sending device determines that the type of video frame to be transmitted is a non-key frame, it can further select a codec network corresponding to the non-key frame, that is, a second codec network, and use the selected second codec network to successively perform fusion processing and encoding processing on multiple video features corresponding to the non-key frame to obtain encoding features.

[0152] Furthermore, a specific structure of the second encoding and decoding network is provided, such as Figure 7B As shown, the second codec network includes a prediction network and a residual codec network, and the output end of the prediction network is connected to the input end of the residual codec network. Under the second codec network of this structure, the above method for fusing and encoding multiple video features is as follows: Figure 8 As shown, including:

[0153] S801, inputting a plurality of second restored video features and a plurality of video features corresponding to a previous video frame into a prediction network for prediction to obtain a plurality of prediction features.

[0154] The prediction network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the prediction network has the ability to predict multiple input video features through training.

[0155] In the embodiment of the present application, when the sending device obtains multiple video features corresponding to the non-key frame, multiple second restored video frames corresponding to the previous video frame, and the prediction network, the multiple second restored video features corresponding to the previous video frame and the multiple video features can be simultaneously input into the prediction network to predict the features of the current frame, and multiple predicted features can be obtained. When performing the prediction, the sending device can predict the multiple video features with reference to the multiple second restored video features to obtain multiple predicted features.

[0156] S802, inputting the multiple prediction features and the multiple video features into the residual codec network for fusion and encoding to obtain encoding features.

[0157] The residual codec network may be a machine learning network or a neural network, which may be obtained by training in advance, so that the residual codec network has the ability to fuse and encode multiple input video features through training. Optionally, the residual codec network may be a residual coder (Residual Coder).

[0158] In an embodiment of the present application, when the sending device obtains multiple video features, multiple prediction features, and a residual codec network corresponding to non-key frames, the multiple video features and multiple prediction features can be simultaneously input into the residual codec network for fusion and encoding to obtain coding features.

[0159] Furthermore, a specific structure of the above prediction network is provided, such as Fig. 8A As shown, the prediction network includes a motion codec network and an inter-frame prediction network. The output end of the motion codec network is connected to the input end of the inter-frame prediction network. Under the prediction network of this structure, the above S801 "inputs a plurality of second restored video features and a plurality of video features corresponding to the previous video frame into the prediction network for prediction to obtain a plurality of prediction features", such as Fig. 9 As shown, including:

[0160] S901, inputting a plurality of second restored video features and a plurality of video features into a motion codec network for fusion and encoding and decoding to obtain a plurality of codec features.

[0161] The motion codec network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the motion codec network has the ability to fuse and encode multiple input video features through training. The multiple codec features are in a one-to-one correspondence with the multiple video features, that is, each video feature corresponds to its own codec feature.

[0162] In an embodiment of the present application, when the sending device obtains multiple video features corresponding to the current non-key frame, multiple second restored video features corresponding to the previous frame, and a motion codec network, the multiple video features and the multiple second restored video features can be simultaneously input into the motion codec network for encoding and decoding to obtain multiple codec features.

[0163] S902, inputting a plurality of encoding and decoding features and a plurality of second restored video features into an inter-frame prediction network for prediction to obtain a plurality of prediction features.

[0164] The inter-frame prediction network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the inter-frame prediction network has the ability to perform integrated prediction on multiple input codec features through training. The multiple codec features and the multiple second restored video features are in a one-to-one correspondence, that is, each codec feature corresponds to its own second restored video feature. The inter-frame prediction network may be represented by the English word Inter Prediction.

[0165] In an embodiment of the present application, when the sending device obtains multiple codec features and multiple second restored video features corresponding to the current non-key frame, as well as an inter-frame prediction network, the multiple codec features and multiple second restored video features can be simultaneously input into the inter-frame prediction network for feature prediction to obtain multiple prediction features.

[0166] The embodiment of the present application predicts the features corresponding to the current video frame by referring to multiple second restored video features corresponding to the previous video frame, that is, the encoding and decoding result of the previous video frame through an inter-frame prediction network, which can improve the restoration degree of multiple video features corresponding to the restored video frame by the final encoding and decoding network to a certain extent.

[0167] Furthermore, a specific structure of the motion codec network is provided, such as Fig.9A As shown, the motion codec network includes a motion coding network, a codec and a motion decoding network. The output end of the motion coding network is connected to the input end of the codec, and the output end of the codec is connected to the input end of the motion decoding network. Under the motion codec network of this structure, the above-mentioned method for fusing and encoding and decoding multiple video features, that is, the above-mentioned S901 "inputting multiple second restored video features and multiple video features into the motion codec network for fusing and encoding and decoding to obtain multiple codec features" is as shown in Fig.10 As shown, including:

[0168] S1001, input multiple second restored video features and multiple video features into a motion coding network for fusion processing to obtain second fusion features.

[0169] The motion coding network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the motion coding network has the ability to fuse multiple input video features through training.

[0170] In the embodiment of the present application, when the sending device obtains multiple video features corresponding to the current non-key frame, multiple second restored video features corresponding to the previous frame, and a motion coding network, the multiple video features and the multiple second restored video features can be simultaneously input into the motion coding network for fusion and encoding to obtain the fused second fusion features. The motion coding network can be a side motion feature encoder (Side Motion Encoder).

[0171] In an embodiment of the present application, when the sending device obtains multiple video features corresponding to non-key frames, multiple second restored video features of a previous video frame, and a motion coding network, the multiple video features and the multiple second restored video features can be simultaneously input into the motion coding network for different scale feature fusion to obtain a fused second fused feature; optionally, the sending device can also simultaneously input multiple video features and multiple second restored video features into the motion coding network for different channel feature fusion to obtain a fused second fused feature; optionally, the sending device can also simultaneously input multiple video features and multiple second restored video features into the motion coding network for redundant semantic feature fusion to obtain a fused second fused feature.

[0172] S1002: Input the second fused feature into a codec for encoding and decoding to obtain an initial codec feature.

[0173] The codec may be a machine learning network or a neural network, which may be obtained by training in advance, so that the codec has the ability to encode and decode input video features through training. Optionally, the codec may be a hyper-motion video feature encoder (corresponding English expression: Hyper Motion Coder).

[0174] In the embodiment of the present application, when the sending device obtains the second fusion feature, the second fusion feature can be further input into the codec for code stream encoding and decoding processing to obtain the initial codec feature after encoding and decoding; optionally, the sending device can also input the second fusion feature into the codec for message encoding and decoding processing to obtain the initial codec feature after encoding and decoding; optionally, the sending device can also input the second fusion feature into the codec for encoding and decoding processing of other data format types to obtain the initial codec feature after encoding and decoding. In the embodiment of the present application, a motion coding network for fusion features is provided, and the motion coding network can fuse multiple video features of the current frame with reference to the second restored video feature corresponding to the previous video frame, which can improve the accuracy of the fusion feature to a certain extent, thereby improving the accuracy of the subsequent encoding.

[0175] Optionally, when the sending device obtains the initial codec feature output by the codec, the initial codec feature can also be sent to the receiving device, so that after the receiving device receives the initial codec feature, the initial codec feature can be used as a reference feature to assist the receiving device in restoring the coding feature to improve the accuracy of the restored feature. The timing of sending the above initial codec feature can be sending the coding features corresponding to multiple video features together.

[0176] S1003, inputting the initial codec features into a motion decoding network for feature restoration to obtain a plurality of codec features.

[0177] Among them, the motion decoding network can be a machine learning network or a neural network, which can be obtained in advance through training, so that the motion decoding network has the ability to restore the input initial codec features into multiple codec features through training. Optionally, the motion decoding network can be a side motion feature decoder (Side Motion Decoder). It should be noted that the above-mentioned restoration process can correspond to the fusion process in S1001, that is, when the fusion process is a fusion process of different sizes, the corresponding restoration process is to restore the features to features of multiple different sizes; when the fusion process is a fusion process of different channels, the corresponding restoration process is to restore the features to features of multiple different channels; when the fusion process is a fusion process of features with different redundant semantics, the corresponding restoration process is to restore the features to features with multiple different redundant semantics.

[0178] In an embodiment of the present application, when the sending device obtains the initial codec feature, the initial codec feature can be input into the motion decoding network to perform restoration processing of features of multiple different sizes to obtain multiple restored codec features; optionally, the sending device can also input the initial codec feature into the motion decoding network to perform restoration processing of features of multiple different channels to obtain multiple restored codec features; optionally, the sending device can also input the initial codec feature into the motion decoding network to perform restoration processing of features of multiple different redundant semantics to obtain multiple restored codec features.

[0179] Furthermore, a specific structure of the motion coding network is provided, such as Fig. 10A As shown, the motion coding network includes multiple splicing layers, multiple second convolution modules and second splicing modules, each splicing layer is connected to the input end of the corresponding second convolution module, and the output end of each second convolution module is connected to the second splicing module. Under the motion coding network of this structure, the above S1001 "inputs multiple second restored video features and multiple video features into the motion coding network for fusion processing to obtain second fusion features" is as shown in Fig.11 As shown, including:

[0180] S1101, pairing a plurality of second restored video features with a plurality of video features, and inputting each pair of paired features into a corresponding splicing layer for splicing, to obtain spliced ​​video features output by each splicing layer.

[0181] The concatenation layer can be expressed in English as Concatenation.

[0182] In the embodiment of the present application, when the sending device obtains multiple second restored video features and multiple video features, the multiple second restored video features and the multiple video features can be further paired. Since the multiple second restored video features and the multiple video features are in a one-to-one correspondence, the corresponding second restored video features and video features are paired. When the pairing is completed, each pair of paired features can be input into the corresponding splicing layer for splicing, that is, the video features and the second restored video features contained in each pair of features are spliced ​​into one feature, namely, the spliced ​​video feature.

[0183] S1102: Input each spliced ​​video feature into a corresponding second convolution module for resizing to obtain a plurality of resized second adjusted features.

[0184] The sizes of the second adjustment features are consistent. The second convolution module can be composed of multiple convolution layers, and the second convolution module is used to perform convolution processing on the input video features, thereby achieving dimensionality reduction or size adjustment processing on the input features. The second convolution module can be obtained in advance through training.

[0185] In an embodiment of the present application, when the sending device inputs multiple video features and multiple restored video features into a motion coding network, the multiple video features and the multiple restored video features can be first paired and then spliced, and then each spliced ​​video feature is input into the corresponding second convolution module in the motion coding network for size adjustment. Specifically, the size of each spliced ​​video feature is adjusted to a uniform size, and the uniform size can be determined in advance based on actual feature extraction requirements. For example, the motion coding network includes three second convolution modules (#1 second convolution module, #2 second convolution module, #3 second convolution module), the #1 spliced ​​video feature is input into the #1 second convolution module, the #2 spliced ​​video feature is input into the #2 second convolution module, and the #3 spliced ​​video feature is input into the #3 second convolution module. The #1 second convolution module resizes the #1 spliced ​​video feature to obtain the adjusted #1 second adjusted feature; the #2 second convolution module resizes the #2 spliced ​​video feature to obtain the adjusted #2 second adjusted feature; the #3 second convolution module resizes the #3 spliced ​​video feature to obtain the adjusted #3 second adjusted feature.

[0186] S1103, input the plurality of second adjustment features into a second stitching module for stitching to obtain a second fusion feature.

[0187] The second concatenation module is used to concatenate multiple input features, and specifically can concatenate multiple features by channel concatenation, size concatenation, or semantic feature concatenation. The second concatenation module can be obtained in advance through training. The second concatenation module can be represented by Feature Concatenation in English.

[0188] In the embodiment of the present application, the sending device may input multiple second adjustment features into the second splicing module for splicing, so as to splice the multiple second adjustment features into one feature, that is, obtain the second fusion feature. Optionally, when channel splicing is performed on multiple second adjustment features, accumulation and channel processing are performed on the same or different channels to obtain the second fusion feature. For example, N second adjustment features are processed: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W) are concatenated, if the number of channels of the above N first adjustment features is the same, a feature with a size of (C*N, H, W) can be obtained after concatenation; if the number of channels of the above N first adjustment features is different, a feature with a size of (C 1 +C 2 ....C N , H, W) characteristics.

[0189] Optionally, a specific structure of a second convolution module is provided, such as Fig.11A As shown, the second convolution module includes multiple second linear convolution layers, multiple second linear convolution layers are cascaded, and each second linear convolution layer performs convolution processing on the input features. Each second linear convolution layer is used to perform convolution and linear activation processing on the features output by the previous stage. Among them, each second linear convolution layer can be represented by Conv+LeakyReLU.

[0190] Under the second convolution module of this structure, the above-mentioned S1102 method for resizing includes: inputting each spliced ​​video feature into the first second linear convolution layer cascaded in the corresponding second convolution module, the second linear convolution layer performs convolution and linear activation processing on the input spliced ​​video feature, outputs the processed video feature, and inputs the processed video feature into the second second linear convolution layer for convolution and linear activation processing, outputs the video feature after secondary processing, and so on, layer by layer convolution and linear activation processing, until the last second linear convolution layer outputs the last processed feature, that is, the second adjusted feature is obtained. The second convolution module can adjust the size of the input feature through layer by layer convolution and linear activation processing, and can also achieve dimensionality reduction processing on the input feature, so as to obtain the resized feature.

[0191] In one embodiment, a specific structure of the motion decoding network is also provided, such as Fig.11A As shown, the motion decoding network includes a second splitting module and a plurality of second deconvolution modules, the output end of the second splitting module is respectively connected to the input end of each second deconvolution module. In the motion decoding network of this structure, the above Fig.10 S1003 in “inputting the initial codec features into the motion decoding network for feature restoration to obtain multiple codec features”, such as Fig.12 As shown, including:

[0192] S1201, input the initial encoding and decoding features into a second splitting module for feature splitting to obtain a plurality of second initial restored video features after splitting.

[0193] The second splitting module is used to split the input features, and specifically can split an initial encoding and decoding feature into multiple features of the same size. The second splitting module can be obtained in advance through training. The second splitting module can be expressed in English as Feature Distribution.

[0194] In an embodiment of the present application, the sending device can input the initial codec feature into the second splitting module for feature splitting, so as to split an initial codec feature into multiple features of the same size, that is, to obtain multiple second initial restored video features. If the fusion processing in the aforementioned motion coding network is to perform fusion processing of multiple video features of different sizes, when the initial codec feature is split, it can be split into multiple second initial restored video features of the same size in different channels, or it can be split into multiple second initial restored video features of the same size in the same channel. For example, the initial codec feature: (C*N, H, W) is split into multiple second initial restored video features of different sizes in the same channel: (C, H, W), (C, H, W), ...., (C, H, W), or the initial codec feature (C 1 +C 2 ....C N , H, W) are split into multiple second initial restored video features of different sizes with different channels: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W).

[0195] S1202: Input each second initial restored video feature into a corresponding second deconvolution module for restoration processing to obtain a plurality of encoding and decoding features.

[0196] The second deconvolution module may be composed of a plurality of deconvolution layers, and the second deconvolution module is used to perform deconvolution processing on the input second initial restored video features, thereby realizing the size adjustment processing on the second initial restored video features. The second deconvolution module may be obtained in advance through training.

[0197] In an embodiment of the present application, when the sending device inputs multiple second initial restored video features into the motion decoding network, each second initial restored video feature can be input into the corresponding second deconvolution module in the motion decoding network for restoration processing. Specifically, the size can be adjusted, that is, the size of the unified second initial restored video features is adjusted to different sizes, and the size of each feature after adjustment is consistent with the size of multiple video features corresponding to the video frame before the previous fusion. For example, the motion decoding network includes three second deconvolution modules (#1 second deconvolution module, #2 second deconvolution module, #3 second deconvolution module), the #1 second initial restored video feature is input into the #1 second deconvolution module, the #2 second initial restored video feature is input into the #2 second deconvolution module, the #3 second initial restored video feature is input into the #3 second deconvolution module, the #1 second deconvolution module restores the #1 second initial restored video feature to obtain the restored #1 second restored video feature; the #2 second deconvolution module restores the #2 second initial restored video feature to obtain the restored #2 second restored video feature; the #3 second deconvolution module restores the #3 second initial restored video feature to obtain the restored #3 second restored video feature.

[0198] Optionally, the above-mentioned resizing for restoration may specifically be to adjust the width and height of the second initial restored video feature. For example, if the plurality of second initial restored video features are: (C, H, W), (C, H, W), ..., (C, H, W), then after adjusting the width and height of the plurality of second initial restored video features accordingly, a plurality of second restored video features with different widths and heights may be obtained: (C 1 , H 1 , W 1 ), (C 2 , H 2 , W 2 ), ...., (C N , H N , W N ); or if the features of the plurality of second initial restored videos are: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W), then after adjusting the widths and heights of the multiple second initial restored video features, multiple second restored video features with different widths and heights can be obtained: (C 1 , H 1 , W 1 ), (C 2 , H 2 , W 2 ), ...., (C N , H N , WN ).

[0199] Optionally, a specific structure of a second deconvolution module is provided, such as Fig. 12A As shown, the second deconvolution module includes multiple second linear deconvolution layers, multiple second linear deconvolution layers are cascaded, and each second linear deconvolution layer performs deconvolution processing on the input features. Each second linear deconvolution layer is used to perform deconvolution and linear activation processing on the features output by the previous stage. Among them, each second linear deconvolution layer can be represented by Subpel Conv+LeakyReLU.

[0200] Under the second deconvolution module of this structure, the above-mentioned S1202 method for restoration processing includes: inputting each second initial restored video feature into the first second linear deconvolution layer cascaded in the corresponding second deconvolution module, the second linear deconvolution layer performs deconvolution and linear activation processing on the input second initial restored video feature, outputs the processed feature, and inputs the processed feature into the second second linear deconvolution layer for deconvolution and linear activation processing, outputs the second processed feature, and so on, layer by layer deconvolution and linear activation processing, until the last second linear deconvolution layer outputs the last processed feature, that is, the second restored video feature is obtained. The second deconvolution module can adjust the size of the input feature through layer by layer deconvolution and linear activation processing, thereby restoring the feature.

[0201] In one embodiment, a codec is further provided, the codec comprising a second encoder and a second decoder. Accordingly, Fig.10 S1002 "inputting the second fused feature into the codec for encoding and decoding to obtain the initial codec feature" includes the steps of: inputting the second fused feature into the second encoder for encoding to obtain the fused coding feature; inputting the fused coding feature into the second decoder for decoding to obtain the initial codec feature.

[0202] The second encoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the second encoder has the ability to encode the input video features through training. Optionally, the second encoder may be a hyper priori motion feature encoder (corresponding English expression: Hyper Incoder) in a hyper priori motion feature codec (corresponding English expression: Hyper Coder).

[0203] The second decoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the second decoder has the ability to decode the input video features through training. Optionally, the second decoder may be a hyper-prior motion feature decoder (corresponding English expression: Hyper Decoder) in a hyper-prior motion feature codec (corresponding English expression: Hyper Coder).

[0204] In an embodiment of the present application, after the sending device obtains the second fusion feature based on the aforementioned steps, the second fusion feature can be further input into the second encoder for bitstream encoding processing to obtain the encoded fusion coding feature; then, the sending device can input the encoded fusion coding feature into the second decoder for bitstream decoding processing to obtain the decoded initial coding and decoding feature.

[0205] Optionally, the sending device may also input the second fusion feature into the second encoder for message encoding processing to obtain the encoded fusion coding feature; then, the sending device may input the encoded fusion coding feature into the second decoder for message decoding processing to obtain the decoded initial coding and decoding feature.

[0206] Optionally, the sending device may also input the second fused feature into a second encoder for encoding processing of other data format types to obtain encoded fused coding features. Then, the sending device may input the encoded fused coding features into a second decoder for decoding processing of other data format types to obtain decoded initial coding and decoding features.

[0207] In one embodiment, a specific structure of the inter-frame prediction network in the above prediction network is also provided, such as Fig.13A As shown, the inter-frame prediction network includes a Gaussian convolution generator and a convolution mapping module, the output end of the Gaussian convolution generator is connected to the input end of the convolution mapping module, and the input end of the convolution mapping module is connected to the output end of the motion codec network. Under the inter-frame prediction network of this structure, the above S902 "inputs multiple codec features and multiple second restored video features into the inter-frame prediction network for prediction to obtain multiple prediction features" is as shown in Fig.13 As shown, including:

[0208] S1301, input multiple second restored video features into a Gaussian convolution generator to generate Gaussian features corresponding to each second restored video feature.

[0209] The Gaussian convolution generator may be a machine learning network or a neural network, which may be obtained in advance through training, so that the Gaussian convolution generator has the ability to perform Gaussian convolution on multiple input video features through training. The Gaussian convolution generator may be represented by the English Gaussian Volume Generation.

[0210] In an embodiment of the present application, when the sending device obtains multiple second restored video features corresponding to the previous video frame and a Gaussian convolution generator, the multiple second restored video features can be input into the Gaussian convolution generator to generate Gaussian features corresponding to each second restored video feature.

[0211] S1302, inputting multiple Gaussian features and multiple encoding and decoding features into a convolutional mapping module simultaneously for feature prediction to obtain multiple predicted features.

[0212] Among them, the convolution mapping module can be a machine learning network or a neural network, which can be obtained in advance through training, so that the volume mapping module has the ability to predict multiple input video features through training. The convolution mapping module can be represented by the English word Volume Warping. The above-mentioned convolution mapping module is also used to split each codec feature into optical flow information and feature information, and refer to each Gaussian feature, and predict the current frame according to the optical flow information and feature information to obtain multiple predicted features.

[0213] In an embodiment of the present application, when the sending device obtains multiple Gaussian features, multiple codec features, and a convolution mapping module, the multiple Gaussian features and multiple codec features can be simultaneously input into the convolution mapping module to generate multiple prediction features.

[0214] Optionally, when the sending device obtains multiple prediction features output by the inter-frame prediction network, the multiple prediction features can also be sent to the receiving device, so that after the receiving device receives the multiple prediction features, the multiple prediction features can be used as reference features to assist the receiving device in restoring the coding features to improve the accuracy of the restored features. The timing of sending the above multiple prediction features can be sending them together with the coding features corresponding to the multiple video features.

[0215] Optionally, another inter-frame prediction network is provided, such as Fig. 13B As shown, in Fig.13A Frame shown

[0216] Based on the inter-frame prediction network, the inter-frame prediction network also includes a feature segmentation module, the input end of the feature segmentation module is connected to the output end of the motion codec network; the output end of the feature segmentation module is connected to the input end of the convolution mapping module. Under the inter-frame prediction network of this structure, the above-mentioned method for predicting multiple codec features, that is, the above-mentioned S1202 "multiple Gaussian features and the multiple codec features are simultaneously input into the convolution mapping module for feature prediction to obtain the multiple prediction features" is as shown in Fig.14 As shown, including:

[0217] S1401, input multiple codec features into a feature segmentation module to generate optical flow features and scale features corresponding to each codec feature.

[0218] The feature segmentation module may be a machine learning network or a neural network, which may be obtained in advance through training, so that the feature segmentation module has the ability to split each input video feature into an optical flow feature and a scale feature through training. The feature segmentation module may be represented by Distribution in English.

[0219] In an embodiment of the present application, when the sending device obtains multiple codec features and a feature segmentation module, the multiple codec features can be input into the feature segmentation module to perform feature segmentation of each codec feature to generate optical flow features and proportion features corresponding to each codec feature.

[0220] S1402: Input the optical flow features and scale features corresponding to each encoding and decoding feature into a convolutional mapping module to obtain a plurality of prediction features.

[0221] The convolutional mapping module may be a machine learning network or a neural network, which may be obtained by training in advance, so that the convolutional mapping module has the ability to predict input features through training. The convolutional mapping module may be represented by Volume Warping.

[0222] In an embodiment of the present application, when the sending device obtains the optical flow features and proportion features corresponding to multiple codec features, as well as the convolution mapping module, the optical flow features and proportion features corresponding to all codec features can be input into the convolution mapping module for feature prediction to generate predicted features corresponding to each codec feature.

[0223] In one embodiment, a specific structure of a residual codec network is provided, such as Fig.15A As shown,

[0224] The residual codec network includes a residual coding network and a third encoder, the output end of the residual coding network is connected to the input end of the third encoder, and under the residual coding network of this structure, the above Figure 8In the embodiment, S802 "inputting multiple prediction features and multiple video features into a residual codec network for fusion and encoding to obtain coding features" is as follows Fig.15 As shown, including:

[0225] S1501, input multiple prediction features and multiple video features into the residual coding network for fusion processing to obtain a third fusion feature.

[0226] The residual coding network may be a machine learning network or a neural network, which may be obtained by training in advance, so that the residual coding network has the ability to fuse multiple input video features through training. Optionally, the residual coding network may be a side residual feature encoder (corresponding English expression: Side Residual Encoder).

[0227] In an embodiment of the present application, when the sending device obtains multiple video features, multiple prediction features, and a residual coding network corresponding to non-key frames, the multiple video features and the multiple prediction features can be simultaneously input into the residual coding network to perform feature fusion of different scales to obtain a fused third fused feature; optionally, the sending device can also simultaneously input multiple video features and multiple prediction features into the residual coding network to perform feature fusion of different channels to obtain a fused third fused feature; optionally, the sending device can also simultaneously input multiple video features and multiple prediction features into the residual coding network to perform redundant semantic feature fusion to obtain a fused third fused feature.

[0228] S1502: Input the third fusion feature into a third encoder for encoding to obtain an encoded feature.

[0229] The third encoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the third encoder has the ability to encode the input video features through training. Optionally, the third encoder may be a hyper priori residual feature encoder (corresponding English expression: Hyper Residual Encoder) in a hyper priori residual feature codec (corresponding English expression: Hyper Residual Coder).

[0230] In the embodiment of the present application, when the sending device obtains the third fusion feature, the third fusion feature can be further input into the third encoder for code stream encoding processing to obtain the encoded coding feature; optionally, the sending device can also input the third fusion feature into the third encoder for message encoding processing to obtain the encoded coding feature; optionally, the sending device can also input the third fusion feature into the third encoder for encoding processing of other data format types to obtain the encoded coding feature. In the embodiment of the present application, a residual coding network for fusion feature is set up to realize fusion processing of multiple video features, thereby facilitating the subsequent encoding of the fused features and improving the coding efficiency.

[0231] Furthermore, a specific structure of the above residual coding network is provided, such as Fig. 15B As shown, the residual coding network includes multiple third convolution modules and a third splicing module, and the output end of each third convolution module is connected to the input end of the third splicing module. Under the residual coding network of this structure, the above S1501 "inputs multiple prediction features and multiple video features into the residual coding network for fusion processing to obtain a third fusion feature" is as shown in Fig.16 As shown, including:

[0232] S1601, after pairing a plurality of predicted frequency features and a plurality of video features, a plurality of pairs of features are obtained, and a residual operation is performed on each pair of features to obtain a plurality of residual features.

[0233] In the embodiment of the present application, when the sending device obtains multiple predicted frequency features and multiple video features, the multiple predicted frequency features and multiple video features can be further paired. Since the multiple predicted frequency features and multiple video features are in a one-to-one correspondence, the predicted frequency features and video features in the corresponding relationship are paired. When the pairing is completed, the residual operation can be performed on each pair of features after the pairing. Optionally, the residual operation can be specifically a difference operation, so as to obtain the residual features corresponding to each pair of features. For example, multiple prediction frequency features include three prediction features: #1 prediction feature, #2 prediction feature, and #3 prediction feature; multiple video features include three video features: #1 video feature, #2 video feature, and #3 video feature; after pairing the three prediction features and the three video features, three pairs of features can be obtained: (#1 prediction feature, #1 video feature), (#2 prediction feature, #2 video feature), and (#3 prediction feature, #3 video feature), and after performing residual operations on the three pairs of features accordingly, three residual features can be obtained: #1 residual feature = #1 video feature - #1 prediction feature, #2 residual feature = #2 video feature - #2 prediction feature, and #3 residual feature = #3 video feature - #3 prediction feature.

[0234] S1602: Input each residual feature into the corresponding third convolution module for resizing to obtain a plurality of third adjusted features after resizing.

[0235] The sizes of the third adjustment features are consistent. The third convolution module can be composed of multiple convolution layers, and the third convolution module is used to perform convolution processing on the input residual features, thereby achieving dimensionality reduction or size adjustment processing on the residual features. The third convolution module can be obtained in advance through training.

[0236] In an embodiment of the present application, when the sending device inputs multiple video features and multiple prediction features into the residual coding network, the multiple video features and the multiple prediction features can be first paired and then residual operations can be performed, and then each residual feature can be input into the corresponding third convolution module in the residual coding network for size adjustment, and the size of each residual feature can be specifically adjusted to a uniform size, and the uniform size can be determined in advance according to the actual feature extraction requirements. For example, the residual coding network includes three third convolution modules (#1 third convolution module, #2 third convolution module, #3 third convolution module), the #1 residual feature is input into the #1 third convolution module, the #2 residual feature is input into the #2 third convolution module, and the #3 residual feature is input into the #3 third convolution module. The #1 third convolution module resizes the #1 residual feature to obtain the adjusted #1 third adjustment feature; the #2 third convolution module resizes the #2 residual feature to obtain the adjusted #2 third adjustment feature; the #3 third convolution module resizes the #3 residual feature to obtain the adjusted #3 third adjustment feature.

[0237] S1603, input the multiple third adjustment features into the third stitching module for stitching to obtain the third fusion feature.

[0238] The third concatenation module is used to concatenate multiple input features, and specifically, channel concatenation, size concatenation, or semantic feature concatenation can be performed on multiple features. The third concatenation module can be obtained in advance through training. The third concatenation module can be represented by Feature Concatenation in English.

[0239] In the embodiment of the present application, the sending device may input multiple third adjustment features into the third splicing module for splicing, so as to splice the multiple third adjustment features into one feature, that is, to obtain a third fusion feature. Optionally, when channel splicing is performed on multiple third adjustment features, accumulation and channel processing are performed on the same or different channels to obtain the third fusion feature. For example, N third adjustment features are processed: (C 1 , H, W), (C 2 , H, W), ...., (C N, H, W) are concatenated, if the number of channels of the above N third adjustment features is the same, a feature with a size of (C*N, H, W) can be obtained after concatenation; if the number of channels of the above N third adjustment features is different, a feature with a size of (C 1 +C 2 ....C N , H, W) characteristics.

[0240] Optionally, a specific structure of a third convolution module is provided, such as Fig.16A As shown, the third convolution module includes multiple third linear convolution layers, multiple third linear convolution layers are cascaded, and each third linear convolution layer performs convolution processing on the input features. Each third linear convolution layer is used to perform convolution and linear activation processing on the features output by the previous stage. Among them, each third linear convolution layer can be represented by Conv+LeakyReLU.

[0241] Under the third convolution module of this structure, the above-mentioned S1602 method for resizing includes: inputting each residual feature into the first third linear convolution layer cascaded in the corresponding third convolution module, the third linear convolution layer performs convolution and linear activation processing on the input residual feature, outputs the processed video feature, and inputs the processed video feature into the second third linear convolution layer for convolution and linear activation processing, outputs the video feature after secondary processing, and so on, layer by layer convolution and linear activation processing, until the last third linear convolution layer outputs the last processed feature, that is, the third adjusted feature is obtained. The third convolution module can adjust the size of the input feature through layer by layer convolution and linear activation processing, and can also achieve dimensionality reduction processing on the input feature, so as to obtain the resized feature.

[0242] In one embodiment, another specific structure of the residual codec network is also provided, such as Fig.16A As shown, in Fig.15A Based on the residual codec network shown in FIG. , the residual codec network further includes a third decoder and a residual decoding network, such as Fig. 16B As shown, the corresponding Figure 7B The second codec network shown also includes an adder, the output end of the third decoder is connected to the input end of the residual decoding network, and the output end of the residual decoding network is connected to the input end of the adder. In the residual codec network of this structure, Fig.17 As shown, Figure 8 The method described in the embodiment may also include the following steps:

[0243] S1701, input the encoded feature into a third decoder for decoding to obtain a decoded third decoded feature.

[0244] The third decoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the third decoder has the ability to decode the input video features through training. Optionally, the third decoder may be a hyper priori residual feature decoder (corresponding English expression: Hyper Residual Decoder) in a hyper priori video feature decoder (corresponding English expression: Hyper Coder).

[0245] In an embodiment of the present application, after the sending device obtains the coding feature based on the aforementioned method, the coding feature can be further input into a third decoder for code stream decoding processing to obtain a first decoding feature after decoding; optionally, the sending device can also input the coding feature into a third decoder for message decoding processing to obtain a first decoding feature after decoding; optionally, the sending device can also input the coding feature into a third decoder for decoding processing of other data format types to obtain a first decoding feature after decoding.

[0246] S1702, input the third decoding feature and the reference feature into the residual decoding network for restoration processing to obtain multiple intermediate restored video features.

[0247] The reference feature is the initial codec feature output by the codec in the prediction network, that is, the reference feature can be the aforementioned Fig.9A The residual decoding network can be a machine learning network or a neural network, which can be obtained by training in advance, so that the residual decoding network has the ability to restore the input third decoding feature into multiple restored video features through training. Optionally, the residual decoding network can be a side residual feature decoder (Side Residual Decoder). It should be noted that the above restoration process can be Figure 2-Figure 8 The fusion processing described in the embodiment corresponds to the above, that is, when the fusion processing is a fusion processing of different sizes, the corresponding restoration processing is to restore the features to features of multiple sizes; when the fusion processing is a fusion processing of different channels, the corresponding restoration processing is to restore the features to features of multiple different channels; when the fusion processing is a fusion processing of features with different redundant semantics, the corresponding restoration processing is to restore the features to features with multiple different redundant semantics.

[0248] In an embodiment of the present application, when the sending device obtains the third decoding feature, the third decoding feature and the reference feature can be input into the residual decoding network to perform restoration processing of features of multiple different sizes, so as to obtain multiple intermediate restored video features after restoration; optionally, the sending device can also input the third decoding feature and the reference feature into the residual decoding network to perform restoration processing of features of multiple different channels, so as to obtain multiple intermediate restored video features after restoration; optionally, the sending device can also input the third decoding feature and the reference feature into the residual decoding network to perform restoration processing of features of multiple different redundant semantics, so as to obtain multiple intermediate restored video features after restoration.

[0249] S1703, input the multiple prediction features and the multiple intermediate restored video features into an adder for accumulation processing to obtain multiple first restored video features corresponding to the video frame.

[0250] When the sending device obtains multiple prediction features and multiple intermediate restored video features based on the above steps, the multiple prediction features and multiple intermediate restored video features can be simultaneously input into the adder to perform addition operations on each pair of features. Since the multiple prediction features and the multiple intermediate restored video features are one-to-one corresponding, when the adder performs the addition operation, the one-to-one corresponding prediction features and intermediate restored video features are added to obtain multiple summed features, i.e., multiple first restored video features. The embodiment of the present application realizes feature restoration assisted by reference features, which can improve the accuracy and efficiency of feature restoration to a certain extent.

[0251] In one embodiment, a specific structure of the residual decoding network is also provided, such as Fig.17A As shown, the residual decoding network includes a fourth splicing module, a third splitting module and a plurality of third deconvolution modules, the output end of the fourth splicing module is connected to the input end of the third splitting module, and the output end of the third splitting module is respectively connected to the input end of each third deconvolution module. In the residual decoding network of this structure, the above Fig.17 S1702 in “inputting the third decoding feature and the reference feature into the residual decoding network for restoration processing to obtain multiple intermediate restored video features”, such as Fig.18 As shown, including:

[0252] S1801, input the third decoding feature and the reference feature into a fourth splicing module for splicing to obtain a splicing feature.

[0253] The fourth concatenation module can be expressed in English as Concatenation.

[0254] In the embodiment of the present application, when the sending device obtains the third decoding feature and the reference feature, the third decoding feature and the reference feature can be simultaneously input into the fourth splicing module for feature splicing, and the splicing is performed into one feature, that is, a splicing feature is obtained.

[0255] S1802: Input the spliced ​​features into a third splitting module for feature splitting to obtain a plurality of third initial restored video features after splitting.

[0256] The third splitting module is used to split the input features, and specifically can split a third decoding feature into multiple features of the same size. The third splitting module can be obtained in advance through training. The third splitting module can be expressed in English as Feature Distribution.

[0257] In an embodiment of the present application, the sending device can input the splicing feature into the third splitting module for feature splitting, so as to split a splicing feature into multiple features of the same size, that is, to obtain multiple third initial restored video features. If the fusion processing in the aforementioned residual coding network is to perform fusion processing of multiple video features of different sizes, when the splicing feature is split, it can be split into multiple third initial restored video features of the same size in different channels, or it can be split into multiple third initial restored video features of the same size in the same channel. For example, the splicing feature: (C*N, H, W) is split into multiple third initial restored video features of different sizes in the same channel: (C, H, W), (C, H, W), ...., (C, H, W), or the splicing feature (C 1 +C 2 ....C N , H, W) are split into multiple third initial restored video features of different sizes with different channels: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W).

[0258] S1803: Input each third initial restored video feature into a corresponding third deconvolution module for restoration processing to obtain a plurality of intermediate restored video features.

[0259] The third deconvolution module may be composed of a plurality of deconvolution layers, and the third deconvolution module is used to perform deconvolution processing on the input third initial restored video feature, thereby realizing the size adjustment processing on the third initial restored video feature. The third deconvolution module may be obtained in advance through training.

[0260] In an embodiment of the present application, the sending device can input each third initial restored video feature into the third deconvolution module for restoration processing, and specifically can adjust the size, that is, adjust the size of each unified third initial restored video feature to a different size, and the size of each feature after adjustment is consistent with the size of multiple video features corresponding to the video frame before the previous fusion. For example, the residual decoding network includes three third deconvolution modules (#1 third deconvolution module, #2 third deconvolution module, #3 third deconvolution module), the #1 third initial restored video feature is input into the #1 third deconvolution module, the #2 third initial restored video feature is input into the #2 third deconvolution module, the #3 third initial restored video feature is input into the #3 third deconvolution module, the #1 third deconvolution module performs restoration processing on the #1 third initial restored video feature to obtain the restored #1 third restored video feature; the #2 third deconvolution module performs restoration processing on the #2 third initial restored video feature to obtain the restored #2 third restored video feature; the #3 third deconvolution module performs restoration processing on the #3 second initial restored video feature to obtain the restored #3 third restored video feature.

[0261] Optionally, the above-mentioned resizing for restoration may specifically be to adjust the width and height of the third initial restored video feature. For example, if the plurality of third initial restored video features are: (C, H, W), (C, H, W), ..., (C, H, W), then after adjusting the width and height of the plurality of third initial restored video features accordingly, a plurality of third restored video features with different widths and heights may be obtained: (C 1 , H 1 , W 1 ), (C 2 , H 2 , W 2 ), ...., (C N , H N , W N ); or if the features of the plurality of third initially restored videos are: (C 1 , H, W), (C 2 , H, W), ...., (C N , H, W), then after adjusting the widths and heights of the plurality of third initial restored video features, a plurality of third restored video features with different widths and heights can be obtained: (C 1 , H 1 , W 1 ), (C 2 , H 2 , W 2 ), ...., (C N , H N , W N ).

[0262] Optionally, a specific structure of a third deconvolution module is provided, such as Fig.18A As shown, the third deconvolution module includes multiple third linear deconvolution layers, multiple third linear deconvolution layers are cascaded, and each third linear deconvolution layer performs deconvolution processing on the input features. Each third linear deconvolution layer is used to perform deconvolution and linear activation processing on the features output by the previous stage. Among them, each third linear deconvolution layer can be represented by the Subpel Conv+LeakyReLU identifier.

[0263] Under the third deconvolution module of this structure, the above-mentioned S1803 method for restoration processing includes: inputting each third initial restored video feature into the first third linear deconvolution layer cascaded in the corresponding third deconvolution module, the third linear deconvolution layer performs deconvolution and linear activation processing on the input third initial restored video feature, outputs the processed feature, and inputs the processed feature into the second third linear deconvolution layer for deconvolution and linear activation processing, outputs the second processed feature, and so on, layer by layer deconvolution and linear activation processing, until the last third linear deconvolution layer outputs the last processed feature, that is, the intermediate restored video feature is obtained. The third deconvolution module can adjust the size of the input feature through layer by layer deconvolution and linear activation processing, thereby restoring the feature.

[0264] In one embodiment, a feature extraction network can also be used to extract features from the video frame to be transmitted, thereby obtaining multiple video features corresponding to the video frame. Figure 2 The video encoding method described in the embodiment is as follows: Fig.19 The above also includes:

[0265] S203, inputting the video frame into a feature extraction network for feature extraction to obtain multiple video features.

[0266] The feature extraction network includes a feature extraction layer of a pyramid structure. The feature extraction network can be a machine learning network or a neural network, which can be obtained by training in advance, so that the feature extraction network has the ability to extract multiple quantities and / or multi-scale features from the input video frames through training. The feature extraction network can adapt to the English Feature Extraction representation.

[0267] In the embodiment of the present application, the sending device may first obtain the original video to be transmitted, and process the original video in segments or frames to obtain each video frame to be transmitted, and then process it frame by frame. Specifically, a pre-trained feature extraction network may be used to extract multiple quantity and / or multi-scale features from the video frame of the current frame to obtain multiple video features corresponding to the video frame.

[0268] The foregoing Figure 2-Figure 19 The video encoding method described in the embodiment is a video encoding method standing on the sending device side. Correspondingly, after receiving the encoding features sent by the sending device, the receiving device can decode and restore the encoding features. The following embodiment will specifically illustrate the method for the receiving device to decode and restore the encoding features.

[0269] Based on the video encoding methods described in all the above embodiments, the present application also provides a video encoding and decoding network, such as Fig.19A As shown, the video codec network includes a first codec network and a second codec network, the first codec network is used to fuse and encode multiple video features corresponding to key frames, and the second codec network is used to fuse and encode multiple video features corresponding to non-key frames. The specific structures of the first codec network and the second codec network, and the corresponding fusion and encoding and decoding methods are consistent with the first codec network and the second codec network involved in the aforementioned embodiment, and the corresponding fusion and encoding and decoding methods. For details, please refer to the contents of the aforementioned embodiment, which will not be repeated here.

[0270] In one embodiment, Fig. 20 As shown, a video decoding method is provided, which is applied to Figure 1 The receiving device in the example is used to illustrate, including the following steps:

[0271] S2001, receiving the coding feature sent by the sending device.

[0272] The coding feature is obtained by fusing and encoding multiple video features corresponding to the video frame by the sending device.

[0273] In the embodiment of the present application, after the sending device fuses and encodes multiple video features corresponding to the video frame to be transmitted to obtain the encoded features, when the encoded features are transmitted to the receiving device through the network, the receiving device can receive the encoded features. Figure 2 The methods described in the embodiments are basically the same. Please refer to the above description for details, which will not be repeated here.

[0274] S2002, decoding and restoring the encoded features to obtain a plurality of first restored video features corresponding to the video frames.

[0275] In an embodiment of the present application, when receiving the coded features, the receiving device may first use a decoding network, a decoder, or a decoding algorithm to decode the coded features, and then use a feature restoration network or a feature restoration algorithm to perform feature restoration processing on the decoded features to obtain multiple first restored video features corresponding to the video frame. It should be noted that the above decoding processing may correspond to the coding processing method of the sending device. For example, if the coding processing method of the sending device is code stream coding processing, the decoding processing here is code stream decoding processing; if the above coding processing is message coding processing, the decoding processing here is message decoding processing; if the above coding processing is coding processing of other data types, the decoding processing here is decoding processing of other data types. The above feature restoration processing may be a restoration processing of features of different scales or features of the same scale, or a restoration processing of features of different channels or features of the same channel.

[0276] In the above-mentioned video decoding method, the receiving device receives the coding features sent by the sending device, and decodes and restores the coding features to obtain multiple first restored video features corresponding to the video frame, wherein the coding features are obtained after the sending device fuses and encodes the multiple video features corresponding to the video frame. Since the sending device encodes after fusing different video features, it has the ability to encode video features of multiple numbers and / or multiple scales, which improves the encoding efficiency of video data to a certain extent, and then when a large number of video features need to be encoded and transmitted, the rate of video data transmission can be improved to a certain extent. In addition, since different video features are fused, semantically redundant features existing in multiple video features can be removed, the encoding effect can be improved to a certain extent, and then the restoration degree of the features decoded and restored based on the coding features in the later stage can be improved. Moreover, the receiving device restores the fused coding features, which can assist in the encoding and decoding of multiple video features, so that the method can be applied to any machine vision network that performs task processing based on multiple features.

[0277] In one embodiment, a method for decoding and restoring the coded features is provided, such as Fig.21 As shown, the above S2002 "decoding and restoring the encoded features to obtain a plurality of first restored video features corresponding to the video frame" may include:

[0278] S2101, determining the type of the video frame corresponding to the coding feature.

[0279] The types of video frames include key frames and non-key frames.

[0280] In an embodiment of the present application, when the receiving device receives a coding feature, it can first determine the type of video frame corresponding to the coding feature. When specifically determining the type of the video frame, it can first identify the type identifier of the coding feature, and determine the type of video frame corresponding to the coding feature based on the type identifier.

[0281] S2102, inputting the encoded features into a decoding network corresponding to the type of the video frame for decoding and restoration, to obtain a plurality of first restored video features.

[0282] Among them, the decoding network can be a machine learning network or a neural network, which can be obtained in advance through training, so that the decoding network has the ability to decode and restore the input features through training. Different types of video frames correspond to different decoding networks. For example, key frames correspond to one decoding network, and non-key frames correspond to another decoding network. The structures of the decoding networks corresponding to key frames and non-key frames are different, but the functions are the same, that is, the decoding network corresponding to the key frames is used to decode and restore the key frames to obtain the restored first restored video features, and the decoding network corresponding to the non-key frames is used to decode and restore the non-key frames to obtain the restored first restored video features. The correspondence between the type of video frame and the decoding network can be predetermined and stored in the pick-up device as a mapping relationship, so that the pick-up device can determine the corresponding decoding network according to the type of video frame.

[0283] In an embodiment of the present application, when the receiving device determines the type of video frame corresponding to the coding feature, it can further select a decoding network corresponding to the type of video frame, and use the selected decoding network to decode and restore the coding feature in sequence to obtain multiple first restored video features. For example, the receiving device can use a decoding network to decode and restore the coding feature corresponding to the key frame to obtain multiple first restored video features corresponding to the key frame; the sending device can use another decoding network to decode and restore the coding feature corresponding to the non-key frame to obtain multiple first restored video features corresponding to the non-key frame.

[0284] The embodiment of the present application selects a corresponding decoding network for decoding and restoration according to different video frame types, thereby matching the video frame type with the decoding network, which can improve the decoding efficiency and decoding accuracy to a certain extent.

[0285] The following embodiments will specifically describe methods for decoding and restoring different types of video frames, that is, provide two implementations of the above S2102.

[0286] First, when the receiving device executes the above S2102 "inputting the encoded features into a decoding network corresponding to the type of the video frame for decoding and restoration to obtain multiple first restored video features", the specific execution steps are: when the type of the video frame is a key frame, inputting the encoded features into a first decoding network for decoding and restoration to obtain multiple first restored video features.

[0287] The first decoding network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the first decoding network may have the ability to decode and restore the coding features corresponding to the key frames through training.

[0288] In an embodiment of the present application, when the receiving device determines that the type of video frame to be transmitted is a key frame, it can further select a decoding network corresponding to the key frame, that is, a first decoding network, and use the selected first decoding network to successively decode and restore the encoding features corresponding to the key frame to obtain multiple first restored video features.

[0289] Furthermore, a specific structure of the first decoding network is provided, wherein the first decoding network includes a first decoder and a decoding subnetwork. Under the first decoding network of the structure, the method for decoding and restoring the coding feature is as follows: Fig. 22 As shown, including:

[0290] S2201, input the encoding feature into a first decoder for decoding to obtain a decoded first decoding feature.

[0291] The first decoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the first decoder has the ability to decode the input video features through training. Optionally, the first decoder may be an edge key frame feature encoder (corresponding English expression: Side Key Frame Feature Decoder) in a hyper-prior video feature decoder (corresponding English expression: Hyper Coder).

[0292] In an embodiment of the present application, after the receiving device receives the coding feature, the coding feature can be further input into the first decoder for code stream decoding processing to obtain a first decoding feature after decoding; optionally, the receiving device can also input the coding feature into the first decoder for message decoding processing to obtain a first decoding feature after decoding; optionally, the receiving device can also input the coding feature into the first decoder for decoding processing of other data format types to obtain a first decoding feature after decoding.

[0293] S2202: Input the first decoded feature into the decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame.

[0294] The decoding subnetwork may be a machine learning network or a neural network, which may be obtained in advance through training, so that the decoding subnetwork has the ability to restore the input first decoding feature into multiple first restored video features through training. Optionally, the decoding subnetwork may be a side key frame feature decoder (Side Key Frame Feature Decoder). It should be noted that the above restoration process may be the same as Figure 2-Figure 5 The fusion processing described in the embodiment corresponds to the above, that is, when the fusion processing is a fusion processing of different sizes, the corresponding restoration processing is to restore the features to features of multiple sizes; when the fusion processing is a fusion processing of different channels, the corresponding restoration processing is to restore the features to features of multiple different channels; when the fusion processing is a fusion processing of features with different redundant semantics, the corresponding restoration processing is to restore the features to features with multiple different redundant semantics.

[0295] In an embodiment of the present application, when the receiving device obtains the first decoding feature, the first decoding feature can be input into the decoding subnetwork to perform restoration processing of features of multiple different sizes to obtain multiple first restored video features after restoration; optionally, the receiving device can also input the first decoding feature into the decoding subnetwork to perform restoration processing of features of multiple different channels to obtain multiple first restored video features after restoration; optionally, the receiving device can also input the first decoding feature into the decoding subnetwork to perform restoration processing of features of multiple different redundant semantics to obtain multiple first restored video features after restoration.

[0296] Furthermore, a specific structure of the above decoding subnetwork is provided, and the specific structure of the decoding subnetwork is similar to the above Fig. 6A The structure of the decoding subnetwork shown in the embodiment is basically the same, please refer to the above 6A, the decoding subnetwork includes a first splitting module and multiple first deconvolution modules, the output end of the first splitting module is respectively connected to the input end of each first deconvolution module, under the decoding subnetwork of this structure, the above S2202 "inputs the first decoding feature into the decoding subnetwork for restoration processing to obtain multiple first restored video features corresponding to the video frame", such as Fig.23 As shown, including:

[0297] S2301, inputting the first decoded feature into a first splitting module for feature splitting to obtain a plurality of first initial restored video features after splitting;

[0298] S2302: Input each first initial restored video feature into a corresponding first deconvolution module for restoration processing to obtain a plurality of first restored video features.

[0299] The method described in the embodiments of the present application is different from the above Figure 7 The methods described in the embodiments are basically the same. Please refer to the above description for details, which will not be repeated here.

[0300] Optionally, a specific structure of a first deconvolution module is provided, and the specific structure of the first deconvolution module is the same as that of the aforementioned Fig. 7A The structure of the first deconvolution module shown in the embodiment is basically the same, see the aforementioned Fig. 7A As shown, the first deconvolution module includes multiple first linear deconvolution layers, multiple first linear deconvolution layers are cascaded, and each first linear deconvolution layer performs deconvolution processing on the input features. Each first linear deconvolution layer is used to perform deconvolution and linear activation processing on the features output by the previous stage. Among them, each first linear deconvolution layer can be represented by Subpel Conv+LeakyReLU.

[0301] The method described in the embodiments of the present application is different from the above Fig. 7A The methods described in the embodiments are basically the same. Please refer to the above description for details, which will not be repeated here.

[0302] The above embodiment describes a method in which a receiving device decodes and restores a key frame. The following embodiment describes another method in which a receiving device decodes and restores a non-key frame.

[0303] The second type is that when the receiving device executes the above S2102 "inputting the encoded features into a decoding network corresponding to the type of video frame for decoding and restoration to obtain multiple first restored video features", the specific execution steps are: when the type of the video frame is a non-key frame, inputting the encoded features into a second decoding network for decoding and restoration to obtain multiple first restored video features.

[0304] The second decoding network may be a machine learning network or a neural network, which may be obtained in advance through training, so that the second decoding network may have the ability to decode and restore the coding features corresponding to non-key frames through training.

[0305] In an embodiment of the present application, when the receiving device determines that the type of video frame to be transmitted is a non-key frame, it can further select a decoding network corresponding to the non-key frame, that is, a second decoding network, and use the selected second decoding network to successively decode and restore the encoding features corresponding to the non-key frame to obtain multiple first restored video features.

[0306] Furthermore, a specific structure of the second decoding network is provided, such as Fig.22AAs shown, the second decoding network includes a fourth decoder and a receiving decoding subnetwork. Under the second decoding network of this structure, the above method for decoding and restoring the coding feature is as follows: Fig.23 As shown, including:

[0307] S2301, input the encoding feature to a fourth decoder for decoding to obtain a decoded first decoding feature.

[0308] The fourth decoder may be a machine learning network or a neural network, which may be obtained in advance through training, so that the fourth decoder has the ability to decode the input video features through training. Optionally, the fourth decoder may be an edge key frame feature encoder (corresponding English expression: Side Key Frame Feature Decoder) in a hyper-prior video feature decoder (corresponding English expression: Hyper Coder).

[0309] In an embodiment of the present application, after the receiving device receives the coding feature, the coding feature can be further input into the fourth decoder for code stream decoding processing to obtain the first decoding feature after decoding; optionally, the receiving device can also input the coding feature into the fourth decoder for message decoding processing to obtain the first decoding feature after decoding; optionally, the receiving device can also input the coding feature into the fourth decoder for decoding processing of other data format types to obtain the first decoding feature after decoding.

[0310] S2302: Input the first decoded feature into a receiving decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame.

[0311] The receiving decoding subnetwork may be a machine learning network or a neural network, which may be obtained in advance through training, so that the receiving decoding subnetwork has the ability to restore the input first decoding feature into multiple first restored video features through training. It should be noted that the above restoration process may be the same as Figure 2-Figure 5 The fusion processing described in the embodiment corresponds to the above, that is, when the fusion processing is a fusion processing of different sizes, the corresponding restoration processing is to restore the features to features of multiple sizes; when the fusion processing is a fusion processing of different channels, the corresponding restoration processing is to restore the features to features of multiple different channels; when the fusion processing is a fusion processing of features with different redundant semantics, the corresponding restoration processing is to restore the features to features with multiple different redundant semantics.

[0312] In an embodiment of the present application, when the receiving device obtains the first decoding feature, the first decoding feature can be input into the receiving decoding subnetwork to perform restoration processing of features of multiple different sizes to obtain multiple first restored video features after restoration; optionally, the receiving device can also input the first decoding feature into the receiving decoding subnetwork to perform restoration processing of features of multiple different channels to obtain multiple first restored video features after restoration; optionally, the receiving device can also input the first decoding feature into the receiving decoding subnetwork to perform restoration processing of features of multiple different redundant semantics to obtain multiple first restored video features after restoration.

[0313] Furthermore, a specific structure of the above-mentioned receiving and decoding sub-network is provided, and the specific structure of the receiving and decoding sub-network is similar to the above-mentioned Fig. 6A The structure of the decoding subnetwork shown in the embodiment is basically the same, please refer to the above 6A, the receiving decoding subnetwork includes a fourth splitting module and multiple fourth deconvolution modules, the output end of the fourth splitting module is respectively connected to the input end of each fourth deconvolution module, under the receiving decoding subnetwork of this structure, the above S2302 "inputs the first decoding feature into the receiving decoding subnetwork for restoration processing to obtain multiple first restored video features corresponding to the video frame", such as Fig.24 As shown, including:

[0314] S2401, input the first decoded feature into a fourth splitting module for feature splitting to obtain a plurality of first initial restored video features after splitting.

[0315] S2402: Input each first initial restored video feature into a corresponding fourth deconvolution module for restoration processing to obtain a plurality of first restored video features.

[0316] The fourth splitting module in the above receiving and decoding sub-network is the same as the above Figure 7 The function of the first splitting module in the decoding subnetwork described in the embodiment is similar, and the fourth deconvolution module in the receiving decoding subnetwork is similar to the aforementioned Figure 7 The function of the fourth deconvolution module in the decoding subnetwork described in the embodiment is similar, and the specific structure of the receiving decoding subnetwork and the method described in the embodiment of the present application are not repeated here. Figure 7 The methods described in the embodiments are basically the same. Please refer to the above description for details, which will not be repeated here.

[0317] In one embodiment, the second decoding network can also refer to the initial encoding and decoding features and the prediction features to restore the encoding features, so a specific structure of the second decoding network is also provided. Fig.22A The second decoding network shown is based on Fig.24AAs shown, the second decoding network also includes: a fifth splicing module and an adder; under the structure of the second decoding network, the above S2302 "inputs the first decoding feature into the receiving decoding subnetwork for restoration processing to obtain multiple first restored video features corresponding to the video frame", such as Fig.25 As shown, including:

[0318] S2501, input the reference feature and the first decoding feature into the fifth splicing module for splicing to obtain a splicing feature.

[0319] The reference feature is the initial codec feature output by the codec in the prediction network, that is, the reference feature can be the aforementioned Fig.9A The initial codec feature output by the codec is obtained by the sending device and then sent to the receiving device. The fifth concatenation module can be represented by the English word Concatenation.

[0320] In the embodiment of the present application, when the receiving device obtains the first decoding feature and the reference feature, the first decoding feature and the reference feature can be simultaneously input into the fifth splicing module for feature splicing, and the splicing is performed into one feature, that is, a splicing feature is obtained.

[0321] S2502, input the splicing features into the receiving decoding sub-network for restoration processing to obtain multiple intermediate restored video features.

[0322] In an embodiment of the present application, when the receiving device obtains the splicing feature, the splicing feature can be input into the receiving decoding subnetwork to perform restoration processing of multiple features of different sizes to obtain multiple first restored video features after restoration; optionally, the receiving device can also input the splicing feature into the receiving decoding subnetwork to perform restoration processing of features of multiple different channels to obtain multiple first restored video features after restoration; optionally, the receiving device can also input the splicing feature into the receiving decoding subnetwork to perform restoration processing of features of multiple different redundant semantics to obtain multiple first restored video features after restoration.

[0323] S2503, input the multiple intermediate restored video features and the multiple predicted features corresponding to the video frames into an adder for accumulation processing to obtain multiple first restored video features.

[0324] Among them, the multiple prediction features corresponding to the video frame can be the aforementioned Fig.13A Or the multiple prediction features output by the inter-frame prediction network shown in 13B are obtained by the sending device and then sent to the receiving device.

[0325] When the receiving device obtains multiple prediction features and multiple intermediate restored video features based on the above steps, the multiple prediction features and multiple intermediate restored video features can be simultaneously input into the adder to perform addition operations on each pair of features. Since the multiple prediction features and the multiple intermediate restored video features are one-to-one corresponding, when the adder performs the addition operation, the one-to-one corresponding prediction features and the intermediate restored video features are added, thereby obtaining multiple summed features, i.e., multiple first restored video features. The embodiment of the present application realizes feature restoration assisted by reference features, which can improve the accuracy and efficiency of feature restoration to a certain extent.

[0326] In one embodiment, when the receiving device is based on the aforementioned Figure 20-Figure 25 When the method described in any embodiment obtains multiple first restored video features, corresponding machine task processing can be further performed. Fig. 20 The method described in the embodiment, such as Fig.26 As shown, including:

[0327] S2003, inputting the plurality of first restored video features into a preset task processing network for task processing to obtain a processing result.

[0328] The preset task processing network may be a machine learning network or a neural network, for example, different types of recognition networks, segmentation networks, analysis networks, etc., which may be obtained in advance through training, so that the preset task processing network has the corresponding task processing function through training. The feature extraction network may be adapted to the English Machine Vision Tasks representation.

[0329] In the embodiment of the present application, after the receiving device can obtain multiple first restored video features, the multiple first restored video features can be input into the trained preset task processing network for task processing, thereby obtaining a processing result. Since the multiple first restored video features are restored features of multiple quantities and / or multiple scales of features corresponding to the original video frame, performing task processing based on the multiple quantities and / or multiple scales of first restored video features can improve the accuracy of task processing to a certain extent.

[0330] In one embodiment, a training network is also provided, the training network comprising a feature extraction network and an initial codec network, and the training method of the codec network in the above embodiment is obtained. Fig. 27 As shown, the training method includes:

[0331] S2701, obtain a sample video.

[0332] S2702, dividing the sample video into small segments of K frames each, and determining the type of the sample video frame of the first frame of each sample video frame segment as a key frame, and the other sample video frames as non-key frames.

[0333] S2703, input the sample video frames into the feature extraction network segment by segment to extract features of each sample video frame in each segment of the sample video frames, and obtain multiple sample video features of each sample video frame in each segment of the sample video frames.

[0334] S2704, inputting a plurality of sample video features corresponding to each sample video frame into an initial encoding and decoding network for restoration and encoding and decoding, and obtaining a plurality of restored sample video features corresponding to each sample video frame.

[0335] S2705: Determine a first loss corresponding to each sample video frame according to a plurality of sample video features corresponding to each sample video frame and a plurality of restored sample video features corresponding to each sample video frame.

[0336] Among them, the loss function corresponding to the first loss can be determined by the maximum entropy estimation loss function and the mean square error loss function. The maximum entropy estimation loss function can be expressed in English as Entropy Estimation Loss, or expressed as L entropy The mean square error loss function can be expressed in English as MSE, or as L MSE express.

[0337] Optionally, the loss function corresponding to the first loss can be determined by the following equation (1):

[0338] L T =L MSE +L entropy (1);

[0339]

[0340]

[0341]

[0342] Among them, L T represents the loss function corresponding to the first loss; L entropy represents the maximum entropy estimation loss function; L MSE represents the mean square error loss function; Represents the training video features corresponding to each sample video frame in each sample video frame; σ j represents the weight coefficient corresponding to the jth training video; p c represents the cth sample video feature corresponding to the sample video frame; q crepresents the cth restored sample video feature corresponding to the sample video frame; N represents the number of sample video features or restored sample video features; K represents the number of sample video frames in each sample video frame. The training method described in the embodiment of the present application can flexibly control the restoration accuracy of each video frame by controlling the weight system in the loss mean square error loss function during the training process.

[0343] S2706: Train the initial codec network according to the first loss to obtain a codec network.

[0344] When the sending device obtains the first loss, the parameters of the initial codec network can be adjusted accordingly to train the initial codec network until the first loss converges or meets the preset training conditions, thereby obtaining the trained codec network.

[0345] The video encoding method and the video decoding method described in the embodiments of the present application have higher encoding efficiency and can encode and decode multiple number and multi-scale video features. To a certain extent, they retain the generalization ability of multiple number and multi-scale video features, support multi-task processing, and retain task accuracy.

[0346] It should be understood that although Figure 2-27 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-27 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0347] In one embodiment, Fig.28 As shown, a video encoding device is provided, comprising:

[0348] A fusion coding module 11 is used to fuse and encode multiple video features corresponding to the video frame to obtain a coding feature;

[0349] The sending module 12 is used to send the coding feature to a receiving device.

[0350] In one embodiment, Fig.29 As shown, a video encoding device is provided, comprising:

[0351] The receiving module 21 is used to receive the coding features sent by the sending device; the coding features are obtained by the sending device fusing and encoding multiple video features corresponding to the video frame;

[0352] The decoding and restoration module 22 is used to decode and restore the encoded features to obtain a plurality of first restored video features corresponding to the video frame.

[0353] For the specific definition of the video encoding device, please refer to the definition of the video encoding method above, which will not be repeated here. For the specific definition of the video decoding device, please refer to the definition of the video decoding method above, which will not be repeated here. The various modules in the above-mentioned video encoding device and the above-mentioned video decoding device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0354] In one embodiment, a communication device is provided, including: a transmitter and a processor; the processor is used to fuse and encode multiple video features corresponding to a video frame to obtain a coded feature; the transmitter is used to send the coded feature to a receiving device.

[0355] In one embodiment, a communication device is provided, characterized in that the communication device includes a receiver and a processor; the receiver is used to receive coding features sent by a sending device; the coding features are obtained after the sending device fuses and encodes multiple video features corresponding to a video frame; the processor is used to decode and restore the coding features to obtain multiple first restored video features corresponding to the video frame.

[0356] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0357] Fusion and encoding of multiple video features corresponding to the video frame to obtain coded features;

[0358] The coded feature is transmitted to a receiving device.

[0359] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0360] Receive a coding feature sent by a sending device; the coding feature is obtained by the sending device fusing and encoding multiple video features corresponding to a video frame;

[0361] The encoded features are decoded and restored to obtain a plurality of first restored video features corresponding to the video frame.

[0362] The present application also provides a computer program product including instructions, which, when executed on a computer, causes the computer to perform the following steps:

[0363] Fusion and encoding of multiple video features corresponding to the video frame to obtain coded features;

[0364] The coded feature is transmitted to a receiving device.

[0365] The present application also provides a computer program product including instructions, which, when executed on a computer, causes the computer to perform the following steps:

[0366] Receive a coding feature sent by a sending device; the coding feature is obtained by the sending device fusing and encoding multiple video features corresponding to a video frame;

[0367] The encoded features are decoded and restored to obtain a plurality of first restored video features corresponding to the video frame.

[0368] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0369] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0370] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video encoding method, characterized in that: The method is applied to a sending device, and the method includes: Fusion and encoding of multiple video features corresponding to the video frame to obtain coded features; The coded feature is transmitted to a receiving device.

2. The method according to claim 1, characterized in that The fusing and encoding of the multiple video features corresponding to the video frame to obtain the encoded features includes: Determining the type of the video frame; The multiple video features are input into a codec network corresponding to the type of the video frame for fusion and encoding to obtain the encoded features.

3. The method according to claim 2, characterized in that The step of inputting the plurality of video features into a codec network corresponding to the type of the video frame for fusion and encoding to obtain the encoded features includes: In the case where the type of the video frame is a key frame, the multiple video features are input into a first codec network for fusion and encoding to obtain the encoded features.

4. The method according to claim 3, characterized in that The first codec network includes a first coding subnetwork and a first encoder; the inputting the plurality of video features into the first codec network for fusion and encoding to obtain the coding features includes: Inputting the multiple video features into the first encoding subnetwork for fusion processing to obtain a first fusion feature; The first fused feature is input into the first encoder for encoding to obtain the encoded feature.

5. The method according to claim 4, characterized in that The first encoding sub-network includes a plurality of first convolution modules and a first splicing module, and the output end of each of the first convolution modules is connected to the input end of the first splicing module; The step of inputting the plurality of video features into the first encoding subnetwork for fusion processing to obtain a first fusion feature includes: Inputting each of the video features into a corresponding first convolution module for resizing, to obtain a plurality of first resized features after resizing; the sizes of the first resized features are consistent; The multiple first adjustment features are input into the first splicing module for splicing to obtain the first fusion feature.

6. The method according to claim 5, characterized in that The first convolution module includes a plurality of first linear convolution layers, and the plurality of first linear convolution layers are cascade-connected.

7. The method according to any one of claims 4 to 6, characterized in that: The first encoding and decoding network further includes a first decoder and a decoding sub-network, and the method further includes: Inputting the encoded feature into the first decoder for decoding to obtain a decoded first decoded feature; The first decoding feature is input into the decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame.

8. The method according to claim 7, characterized in that The decoding sub-network comprises: a first splitting module and a plurality of first deconvolution modules; the output end of the first splitting module is respectively connected to the input end of each of the first deconvolution modules; The step of inputting the first decoding feature into the decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame includes: Inputting the first decoded feature into the first splitting module for feature splitting to obtain a plurality of first initial restored video features after splitting; Each of the first initial restored video features is input into a corresponding first deconvolution module for restoration processing to obtain the plurality of first restored video features.

9. The method according to claim 8, characterized in that The first deconvolution module includes a plurality of first linear deconvolution layers, and the plurality of first linear deconvolution layers are cascade-connected.

10. The method according to claim 2, characterized in that The step of inputting the plurality of video features into a codec network corresponding to the type of the video frame for fusion and encoding to obtain the encoded features includes: In the case where the type of the video frame is a non-key frame, the multiple video features are input into a second codec network for fusion and encoding to obtain the encoded features.

11. The method according to claim 10, characterized in that The second codec network includes a prediction network and a residual codec network; The step of inputting the plurality of video features into a second codec network for fusion and encoding to obtain the encoded features includes: Inputting a plurality of second restored video features corresponding to the previous video frame and the plurality of video features into the prediction network for prediction to obtain a plurality of prediction features; The multiple prediction features and the multiple video features are input into the residual codec network for fusion and encoding to obtain the encoding features.

12. The method according to claim 11, characterized in that The prediction network includes a motion coding network and an inter-frame prediction network, and the plurality of second restored video features corresponding to the previous video frame and the plurality of video features are input into the prediction network for prediction to obtain a plurality of prediction features, including: Inputting the plurality of second restored video features and the plurality of video features into the motion codec network for fusion and encoding and decoding to obtain a plurality of codec features; The multiple encoding and decoding features and the multiple second restored video features are input into the inter-frame prediction network for prediction to obtain the multiple prediction features.

13. The method according to claim 12, characterized in that The motion coding and decoding network includes a motion coding network, a codec and a motion decoding network; The step of inputting the plurality of second restored video features and the plurality of video features into the motion codec network for fusion and encoding and decoding to obtain a plurality of codec features includes: Inputting the plurality of second restored video features and the plurality of video features into the motion coding network for fusion processing to obtain second fusion features; Inputting the second fused features into the codec for encoding and decoding to obtain initial codec features; The initial codec features are input into the motion decoding network for feature restoration to obtain the multiple codec features.

14. The method according to claim 13, characterized in that The motion coding network comprises: a plurality of splicing layers, a plurality of second convolution modules and a second splicing module; each of the splicing layers is connected to an input end of a corresponding second convolution module, and an output end of each of the second convolution modules is connected to the second splicing module; The step of inputting the plurality of second restored video features and the plurality of video features into the motion coding network for fusion processing to obtain second fusion features includes: Pairing the plurality of second restored video features with the plurality of video features, and inputting each pair of paired features into a corresponding splicing layer for splicing, to obtain spliced ​​video features output by each splicing layer; Inputting each of the spliced ​​video features into a corresponding second convolution module for resizing, to obtain a plurality of resized second adjusted features; the sizes of each of the second adjusted features are consistent; The multiple second adjustment features are input into the second stitching module for stitching to obtain the second fusion feature.

15. The method according to claim 14, characterized in that The second convolution module includes multiple second linear convolution layers, and the multiple second linear convolution layers are cascaded.

16. The method according to claim 13, characterized in that The motion decoding network includes: a second splitting module and a plurality of second deconvolution modules; the output end of the second splitting module is respectively connected to the input end of each of the second deconvolution modules; The inputting the initial codec features into the motion decoding network for feature restoration to obtain the multiple codec features includes: Inputting the initial codec feature into the second splitting module for feature splitting to obtain a plurality of second initial restored video features after splitting; Each of the second initially restored video features is input into a corresponding second deconvolution module for restoration processing to obtain the multiple codec features.

17. The method according to claim 16, characterized in that The second deconvolution module includes a plurality of second linear deconvolution layers, and the plurality of second linear deconvolution layers are cascade-connected.

18. The method according to claim 13, characterized in that The codec includes: a second encoder and a second decoder; The step of inputting the second fused feature into the codec for encoding and decoding to obtain an initial codec feature includes: Inputting the second fused feature into the second encoder for encoding to obtain a fused encoded feature; The fused coding feature is input into the second decoder for decoding to obtain the initial coding and decoding feature.

19. The method according to claim 12, characterized in that The inter-frame prediction network includes a Gaussian convolution generator and a convolution mapping module; the output end of the Gaussian convolution generator is connected to the input end of the convolution mapping module; The step of inputting the plurality of encoding and decoding features and the plurality of second restored video features into the inter-frame prediction network for prediction to obtain the plurality of prediction features comprises: Inputting the plurality of second restored video features into the Gaussian convolution generator to generate Gaussian features corresponding to each of the second restored video features; The multiple Gaussian features and the multiple encoding and decoding features are simultaneously input into the convolution mapping module for feature prediction to obtain the multiple predicted features.

20. The method according to claim 19, characterized in that The inter-frame prediction network also includes a feature segmentation module; The input end of the feature segmentation module is connected to the output end of the motion codec network; the output end of the feature segmentation module is connected to the input end of the convolution mapping module; The step of simultaneously inputting the plurality of Gaussian features and the plurality of encoding and decoding features into the convolutional mapping module for feature prediction to obtain the plurality of predicted features comprises: Input the plurality of codec features into the feature segmentation module to generate optical flow features and scale features corresponding to each of the codec features; The optical flow features and the scale features corresponding to each of the encoding and decoding features are input into the convolution mapping module to obtain the multiple prediction features.

21. The method according to any one of claims 13 to 19, characterized in that: The method further comprises: sending the initial codec features output by the codec in the motion codec network to a receiving device; The prediction features output by the inter-frame prediction network are sent to the receiving device.

22. The method according to claim 11, characterized in that The residual codec network includes a residual coding network and a third encoder; The step of inputting the plurality of prediction features and the plurality of video features into the residual codec network for fusion and encoding to obtain the encoding features includes: Inputting the plurality of prediction features and the plurality of video features into the residual coding network for fusion processing to obtain a third fusion feature; The third fused feature is input into the third encoder for encoding to obtain the encoded feature.

23. The method according to claim 22, characterized in that The residual coding network includes: a plurality of third convolution modules and a third splicing module; the output end of each of the third convolution modules is connected to the third splicing module; The step of inputting the plurality of prediction features and the plurality of video features into the residual coding network for fusion processing to obtain a third fusion feature comprises: After pairing the multiple prediction features and the multiple video features, multiple pairs of features are obtained, and residual operations are performed on each pair of features to obtain multiple residual features; Inputting each of the residual features into a corresponding third convolution module for resizing, to obtain a plurality of third adjusted features after resizing; the sizes of the third adjusted features are consistent; The multiple third adjustment features are input into the third splicing module for splicing to obtain the third fusion feature.

24. The method according to claim 23, characterized in that The third convolution module includes multiple third linear convolution layers, and the multiple third linear convolution layers are cascaded.

25. The method according to claim 22, characterized in that The residual codec network further includes a third decoder and a residual decoding network, the second codec network further includes an adder, and the method further includes: Inputting the encoded feature into the third decoder for decoding to obtain a decoded third decoded feature; Inputting the third decoding feature and the reference feature into the residual decoding network for restoration processing to obtain a plurality of intermediate restored video features; the reference feature is the initial codec feature output by the codec in the prediction network; The multiple predicted features and the multiple intermediate restored video features are input into the adder for accumulation processing to obtain multiple first restored video features corresponding to the video frame.

26. The method according to claim 25, characterized in that The residual decoding network includes: a fourth splicing module, a third splitting module and a plurality of third deconvolution modules; the output end of the fourth splicing module is connected to the input end of the third splitting module, and the output end of the third splitting module is respectively connected to the input end of each of the third deconvolution modules; The third decoding feature and the reference feature are input into the residual decoding network for restoration processing to obtain a plurality of intermediate restored video features, including: Inputting the third decoding feature and the reference feature into the fourth splicing module for splicing to obtain a splicing feature; Inputting the splicing features into the third splitting module for feature splitting to obtain a plurality of third initial restored video features after splitting; Each of the third initial restored video features is input into a corresponding third deconvolution module for restoration processing to obtain the plurality of intermediate restored video features.

27. The method according to claim 26, characterized in that The third deconvolution module includes multiple third linear deconvolution layers, and the multiple third linear deconvolution layers are cascaded.

28. The method according to claim 1, characterized in that The method further comprises: The video frame is input into a feature extraction network for feature extraction to obtain the multiple video features; the feature extraction network includes a feature extraction layer with a pyramid structure.

29. A video decoding method, characterized in that: The method is applied to a receiving device, and the method comprises: Receive a coding feature sent by a sending device; the coding feature is obtained by the sending device fusing and encoding multiple video features corresponding to a video frame; The encoded features are decoded and restored to obtain a plurality of first restored video features corresponding to the video frame.

30. The method according to claim 29, characterized in that The decoding and restoring of the encoded features to obtain a plurality of first restored video features corresponding to the video frame includes: Determine the type of the video frame corresponding to the encoding feature; The encoded features are input into a decoding network corresponding to the type of the video frame for decoding and restoration to obtain the plurality of first restored video features.

31. The method according to claim 30, characterized in that The step of inputting the encoded features into a decoding network corresponding to the type of the video frame for decoding and restoration to obtain the plurality of first restored video features includes: In the case where the type of the video frame is a key frame, the encoded features are input into a first decoding network for decoding and restoration to obtain the plurality of first restored video features.

32. The method according to claim 31, characterized in that The first decoding network includes a first decoder and a decoding sub-network, and the encoding features are input into the first decoding network for decoding and restoration to obtain the plurality of first restored video features, including: Inputting the encoded feature into the first decoder for decoding to obtain a decoded first decoded feature; The first decoding feature is input into the decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame.

33. The method according to claim 32, characterized in that The decoding sub-network comprises: a first splitting module and a plurality of first deconvolution modules; the output end of the first splitting module is respectively connected to the input end of each of the first deconvolution modules; The step of inputting the first decoding feature into the decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame includes: Inputting the first decoded feature into the first splitting module for feature splitting to obtain a plurality of first initial restored video features after splitting; Each of the first initial restored video features is input into a corresponding first deconvolution module for restoration processing to obtain the plurality of first restored video features.

34. The method according to claim 33, characterized in that The first deconvolution module includes a plurality of first linear deconvolution layers, and the plurality of first linear deconvolution layers are cascade-connected.

35. The method according to claim 30, characterized in that The step of inputting the encoded features into a decoding network corresponding to the type of the video frame for decoding and restoration to obtain the plurality of first restored video features includes: In the case where the type of the video frame is a non-key frame, the encoded features are input into a second decoding network for decoding and restoration to obtain the plurality of first restored video features.

36. The method according to claim 35, characterized in that The second decoding network includes a fourth decoder and a receiving decoding subnetwork; the encoding features are input into the first decoding network for decoding and restoration to obtain the plurality of first restored video features, including: Inputting the encoded feature into the fourth decoder for decoding to obtain a decoded first decoded feature; The first decoding feature is input into the receiving decoding subnetwork for restoration processing to obtain multiple first restored video features corresponding to the video frame.

37. The method according to claim 36, characterized in that The receiving decoding sub-network comprises: a fourth splitting module and a plurality of fourth deconvolution modules; the output end of the fourth splitting module is respectively connected to the input end of each of the fourth deconvolution modules; The step of inputting the first decoding feature into the receiving decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame includes: Inputting the first decoded feature into the fourth splitting module for feature splitting to obtain a plurality of first initial restored video features after splitting; Each of the first initial restored video features is input into a corresponding fourth deconvolution module for restoration processing to obtain the plurality of first restored video features.

38. The method according to claim 36, characterized in that The second decoding network further includes: a fifth splicing module and an adder; The step of inputting the first decoding feature into the receiving decoding subnetwork for restoration processing to obtain a plurality of first restored video features corresponding to the video frame includes: Inputting the reference feature and the first decoding feature into the fifth splicing module for splicing to obtain a splicing feature; the reference feature is the initial codec feature output by the codec in the prediction network; Inputting the splicing features into the receiving decoding subnetwork for restoration processing to obtain multiple intermediate restored video features; The multiple intermediate restored video features and the multiple predicted features corresponding to the video frames are input into the adder for accumulation processing to obtain the multiple first restored video features.

39. The method according to any one of claims 29 to 38, characterized in that The method further comprises: The multiple first restored video features are input into a preset task processing network for task processing to obtain a processing result.

40. A video encoding device, characterized in that: The device comprises: A fusion coding module is used to fuse and encode multiple video features corresponding to the video frame to obtain a coding feature; A sending module is used to send the coding feature to a receiving device.

41. A video decoding device, characterized in that: The device comprises: A receiving module, used to receive a coding feature sent by a sending device; the coding feature is obtained by the sending device fusing and encoding multiple video features corresponding to a video frame; The decoding and restoration module is used to decode and restore the encoded features to obtain multiple first restored video features corresponding to the video frame.

42. A communication device, characterized in that: The communication device includes a transmitter and a processor; The processor is used to fuse and encode multiple video features corresponding to the video frame to obtain a coded feature; The transmitter is used to send the coding feature to a receiving device.

43. A communication device, characterized in that: The communication device includes a receiver and a processor; The receiver is used to receive the coding features sent by the sending device; the coding features are obtained by the sending device fusing and encoding multiple video features corresponding to the video frame; The processor is used to decode and restore the encoded features to obtain multiple first restored video features corresponding to the video frame.

44. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 39 are implemented.

45. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 39 are implemented.

Images