Machine vision task-oriented video coding method and decoding method, and related device

Abstract

The present disclosure relates to the technical field of machine vision, and provides a machine vision task-oriented video coding method and decoding method, and a related device. The decoding method comprises: decoding a video bitstream to be decoded to obtain decoded video sequence and time domain sampling information; and on the basis of the time domain sampling information, performing time domain recovery processing on the decoded video sequence. The present disclosure can improve the transmission efficiency of video data while ensuring vision tasks.

Description

Video encoding method, decoding method and related equipment for machine vision tasks

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure claims priority to Chinese patent applications with application number 202311014435.7 filed on August 11, 2023, entitled “Video encoding method, decoding method and related equipment for machine vision tasks” and with application number 202410773493.6 filed on June 14, 2024, entitled “Video encoding method, decoding method and related equipment for machine vision tasks”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of machine vision technology, and in particular to a video encoding method, a decoding method, and related devices for machine vision tasks. Background Art

[0004] With the deep integration of informatization and industrialization, the intelligence level of machine equipment is rapidly increasing. Intelligent and efficient processing of machine vision data has long been a hot topic. With the widespread application of machine learning in areas such as the Internet of Vehicles, video surveillance, and smart cities, massive amounts of video data need to be transmitted. This surge in video transmission data volume has directly resulted in previous video encoding solutions designed for human vision failing to meet the latency and scale requirements of machine vision tasks. Consequently, there is an urgent need for a video encoding solution specifically tailored to machine vision tasks.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field.

[0006] Summary of the Invention

[0007] The present disclosure provides a video encoding method, decoding method and related equipment for machine vision tasks, which at least to some extent overcome the technical problem that the video encoding and decoding solutions provided in the related art are difficult to apply to machine vision tasks.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0009] According to one aspect of the present disclosure, a video encoding method for machine vision tasks is provided, comprising: obtaining time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of a first video sequence by a video encoding end; performing time domain sampling on the first video sequence based on the time domain sampling information; encoding the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is configured to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence based on the decoded time domain sampling information, and generate a video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0010] According to another aspect of the present disclosure, a video decoding method for machine vision tasks is also provided, including: decoding a code stream from a video encoding end to obtain a second video sequence and time domain sampling information, wherein the code stream is an encoding result after the video encoding end encodes the time domain sampling result and time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine the time domain sampling multiple used by the video encoding end to time domain sample the first video sequence; determining the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence based on the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame.

[0011] In some embodiments, the time domain sampling information includes: time domain sampling multiple index information, wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; wherein, based on the decoded time domain sampling information, the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence is determined, including: parsing the decoded time domain sampling information to obtain the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence; and determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence.

[0012] In some embodiments, the time domain sampling information further includes: time domain sampling position information, where the time domain sampling position information is used to determine a position at which the video encoder performs time domain sampling on the first video sequence.

[0013] In some embodiments, the first video sequence includes: multiple sub-video sequences; the multiple sub-video sequences use different time domain sampling multiples for time domain sampling; wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple for time domain sampling of each sub-video sequence in the first video sequence; the time domain sampling position information is used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

[0014] In some embodiments, the time domain sampling information further includes: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0015] In some embodiments, based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, a video frame to be inserted between the first video frame and the second video frame is generated, including: if the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is multiple, repeating the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: based on the first video frame and the second video frame, generating a first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located in the middle position between the first video frame and the second video frame; based on the first video frame and the first intermediate video frame, Generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located in the middle position between the first video frame and the first intermediate video frame; based on the first intermediate video frame and the second video frame, generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located in the middle position between the first intermediate video frame and the second video frame.

[0016] In some embodiments, the method further includes: inputting any two video frames into a pre-built video frame infill network model, and outputting a video frame located in the middle position between the two video frames, wherein the video frame infill network model is used to generate a video frame located in the middle position between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0017] In some embodiments, the code stream is an encoding result obtained by encoding the first video sequence and the time domain sampling information by the video encoding end using a hybrid encoding method, where the hybrid encoding method is an encoding method combining inter-frame encoding and intra-frame encoding.

[0018] In some embodiments, the hybrid coding method is a coding method that combines traditional inter-frame coding and end-to-end intra-frame coding; wherein, before generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: obtaining the first video frame obtained by decoding the code stream; filtering the first video frame; and replacing the first video frame in the second video sequence with the filtered video frame.

[0019] In some embodiments, after generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: inserting the video frame to be inserted between the first video frame and the second video frame into a corresponding position between the first video frame and the second video frame in the second video sequence.

[0020] According to another aspect of the present disclosure, a video encoding device is further provided, comprising: a time domain sampling information acquisition module, configured to acquire time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple at which a video encoding end performs time domain sampling on a first video sequence; a time domain sampling module, configured to perform time domain sampling on the first video sequence based on the time domain sampling information; and an encoding module, configured to encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is configured to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, and generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0021] According to another aspect of the present disclosure, a video decoding device is provided, comprising: a decoding module configured to decode a bitstream from a video encoder to obtain a second video sequence and time-domain sampling information, wherein the bitstream is an encoding result obtained by the video encoder encoding the time-domain sampling result and the time-domain sampling information of a first video sequence, and the time-domain sampling information is used at least to determine a time-domain sampling multiple used by the video encoder to perform time-domain sampling on the first video sequence; a video frame interpolation information determination module configured to determine, based on the decoded time-domain sampling information, the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; and a video frame interpolation module configured to generate a video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0022] According to another aspect of the present disclosure, a video processing system for machine vision tasks is also provided, including: a video encoding device and a video decoding device; wherein the video encoding device is used to obtain time domain sampling information, perform time domain sampling on a first video sequence based on the time domain sampling information, and encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding device; the video decoding device is used to receive the code stream from the video encoding device, obtain a second video sequence and time domain sampling information, and determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence based on the decoded time domain sampling information, and then generate a video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0023] According to another aspect of the present disclosure, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned video encoding method for machine vision tasks, or any one of the above-mentioned video encoding methods for machine vision tasks, by executing the executable instructions.

[0024] According to another aspect of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned video encoding method for machine vision tasks, or any one of the above-mentioned video encoding methods for machine vision tasks.

[0025] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the above-mentioned video encoding method for machine vision tasks, or any one of the above-mentioned video encoding methods for machine vision tasks.

[0026] In the embodiments of the present disclosure, a video encoding method, a decoding method, and related devices for machine vision tasks are provided. A video encoding end performs time-domain sampling on a first video sequence based on acquired time-domain sampling information, and then transmits a code stream obtained by encoding the time-domain sampling result and the time-domain sampling information to a video decoding end, so that the video decoding end decodes the received code stream to obtain a decoded second video sequence and the time-domain sampling information. The number of video frames to be inserted between a first video frame and a second video frame in the second video sequence is determined based on the decoded time-domain sampling information, and the video frame to be inserted between the first video frame and the second video frame is generated based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0027] The solution provided by the embodiments of the present disclosure can improve the transmission efficiency of video data by sampling the video data in the time domain while ensuring the visual task.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0030] FIG1 shows a schematic diagram of a video processing system for machine vision tasks according to an embodiment of the present disclosure;

[0031] FIG2 shows a flow chart of a video encoding method for machine vision tasks according to an embodiment of the present disclosure;

[0032] FIG3 shows a flow chart of a video decoding method for machine vision tasks according to an embodiment of the present disclosure;

[0033] FIG4 shows a schematic diagram of temporal downsampling for machine vision tasks according to an embodiment of the present disclosure;

[0034] FIG5 shows a schematic diagram of temporal upsampling for machine vision tasks according to an embodiment of the present disclosure;

[0035] FIG6 shows a schematic structural diagram of a video frame interpolation network model according to an embodiment of the present disclosure;

[0036] FIG7 shows a flow chart of another video decoding method for machine vision tasks according to an embodiment of the present disclosure;

[0037] FIG8 shows a flow chart of a method for generating an interpolation frame according to an embodiment of the present disclosure;

[0038] FIG9 shows a schematic diagram of a video encoding device according to an embodiment of the present disclosure;

[0039] FIG10 shows a schematic diagram of a video decoding device according to an embodiment of the present disclosure;

[0040] FIG11 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure;

[0041] FIG12 shows a schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0043] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0044] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0045] FIG1 shows a schematic diagram of a video processing system for machine vision tasks according to an embodiment of the present disclosure. As shown in FIG1 , the system includes: a video encoding device 10 and a video decoding device 20 .

[0046] The video encoding device 10 is configured to obtain time domain sampling information, perform time domain sampling on the first video sequence according to the time domain sampling information, and encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding device 20;

[0047] The video decoding device 20 is used to receive the code stream from the video encoding device 10, obtain the second video sequence and time domain sampling information, and determine the video frame information (for example, the number of video frames) to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, and then generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0048] It should be noted that the video encoding device 10 can be any device that encodes image / video data, and the video decoding device 20 can be any device that decodes image / video data. In some embodiments, the video encoding device 10 and the video decoding device 20 can be deployed on a single device; in other embodiments, the video encoding device 10 and the video decoding device 20 can be deployed on different devices.

[0049] In one embodiment, the above-mentioned time domain sampling information may include: time domain sampling multiple index information, wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; then the above-mentioned video decoding device 20 is further used to: parse the decoded time domain sampling information to obtain the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence; and determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence.

[0050] Furthermore, in some embodiments, the time domain sampling information may further include: time domain sampling position information, where the time domain sampling position information is used to determine a position where the video encoder performs time domain sampling on the first video sequence.

[0051] In one embodiment, when the first video sequence includes multiple sub-video sequences, and the multiple sub-video sequences use different time domain sampling multiples for time domain sampling; the above-mentioned time domain sampling multiple index information is also used to determine the time domain sampling multiples for time domain sampling of each sub-video sequence in the first video sequence; the above-mentioned time domain sampling position information is also used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

[0052] Furthermore, in some embodiments, the time domain sampling information may further include: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0053] In some embodiments, the video decoding device 20 is further configured to: if the number of video frames to be inserted between the first video frame and the second video frame is one, directly generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is multiple, repeatedly perform the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: generate a first intermediate video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame based on the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame based on the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

[0054] In some embodiments, the above-mentioned video decoding device 20 is also used to: input any two video frames into a pre-built video frame complementation network model, and output a video frame located in the middle position between the two video frames, wherein the video frame complementation network model is used to generate a video frame located in the middle position between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0055] In one embodiment, the code stream in the embodiment of the present disclosure may be an encoding result after the video encoding end encodes the first video sequence and the time domain sampling information using a hybrid encoding method, where the hybrid encoding method is an encoding method that combines inter-frame coding and intra-frame coding.

[0056] In a specific embodiment, the hybrid coding method in the embodiment of the present disclosure can be a coding method that combines traditional inter-frame coding and end-to-end intra-frame coding; in this embodiment, the above-mentioned video decoding device 20 is also used to: obtain the first video frame obtained by decoding the code stream; filter the first video frame; and use the filtered video frame to replace the first video frame in the second video sequence.

[0057] In one embodiment, the video decoding apparatus 20 is further configured to filter the first video frame using the following filter:

[0058] Among them, filter represents a filter.

[0059] In a specific implementation, a 3-channel matrix of the first decoded frame is obtained, and the first decoded frame is filtered using the above filter.

[0060] In some embodiments, the video decoding apparatus 20 is further configured to: insert the video frame to be inserted between the first video frame and the second video frame into a corresponding position between the first video frame and the second video frame in the second video sequence.

[0061] Those skilled in the art will appreciate that the number of video encoding devices and video decoding devices in FIG1 is merely illustrative, and any number of video encoding devices and video decoding devices may be provided based on actual needs, which is not limited in the present disclosure.

[0062] Under the above-mentioned system architecture, the embodiments of the present disclosure provide a video encoding method and a decoding method for machine vision tasks. In principle, the video encoding method and the decoding method for machine vision tasks provided in the embodiments of the present disclosure can be executed by any electronic device with computing and processing capabilities. In some embodiments, the video encoding method for machine vision tasks provided in the embodiments of the present disclosure can be executed by the video encoding device of the above-mentioned system architecture; the video decoding method for machine vision tasks provided in the embodiments of the present disclosure can be executed by the video decoding device of the above-mentioned system architecture; in other embodiments, the video encoding method and the decoding method for machine vision tasks provided in the embodiments of the present disclosure can be implemented by the video encoding device and the video decoding device in the above-mentioned system architecture through interaction.

[0063] FIG2 shows a flow chart of a video encoding method for machine vision tasks according to an embodiment of the present disclosure. As shown in FIG2 , the video encoding method may include the following steps:

[0064] S202: Acquire time domain sampling information, where the time domain sampling information is at least used to determine a time domain sampling multiple used by a video encoder to perform time domain sampling on a first video sequence.

[0065] It should be noted that the above-mentioned time domain sampling information refers to the relevant configuration information for the video encoding end to perform time domain sampling on the original video sequence (that is, the first video sequence), which can be used but not limited to determining the time domain sampling multiple of the video encoding end to perform time domain sampling on the first video sequence.

[0066] In one embodiment, the above-mentioned time domain sampling information may include but is not limited to the following information: a time domain sampling flag, time domain sampling position information and time domain sampling multiple index information, wherein the time domain sampling multiple index information (which can be a time domain sampling multiple or information having a certain mapping relationship with the time domain sampling multiple) is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence; the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0067] It should be noted that, in specific implementation, the time domain sampling flag, time domain sampling position information and time domain sampling multiple index information in the above-mentioned time domain sampling information may be included in full or in part, and may also include other information about time domain sampling in addition to the above three types of information, as long as the information can be used by the decoding end to determine how to insert frames, and the present disclosure does not limit this.

[0068] S204: Perform time-domain sampling on the first video sequence according to the time-domain sampling information.

[0069] In the embodiment of the present disclosure, the video encoding end performs time domain sampling on the original video sequence (i.e., the first video sequence) according to the time domain sampling information to obtain a video sequence after time domain sampling. Since the number of frames of the video sequence after time domain sampling is smaller than the number of frames of the original video sequence (i.e., the first video sequence), time domain sampling is performed on the original video sequence, and only the video sequence after time domain sampling is transmitted, which can improve the transmission efficiency of video data.

[0070] Assume that the original video sequence is: video frame 0, video frame 1, video frame 2, video frame 3, video frame 4, video frame 5, video frame 6, video frame 7 and video frame 8; when the time domain sampling multiple is 2, one video frame is extracted from every two video frames in the original video sequence, and the video sequence after time domain sampling includes: video frame 0, video frame 2, video frame 4, video frame 6 and video frame 8; when the time domain sampling multiple is 4, one video frame is extracted from every four video frames in the original video sequence, and the video sequence after time domain sampling includes: video frame 0, video frame 4 and video frame 8.

[0071] S206 , encoding the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to a video decoding end.

[0072] It should be noted that in the video encoding and decoding solution for machine vision tasks, for a video sequence that has undergone time domain sampling, the video decoding end needs to perform frame insertion (also known as frame supplementation) processing on the decoded video sequence after decoding the video sequence. Therefore, when transmitting the code stream, the video encoding end needs to encode the video sequence after time domain sampling (i.e., the time domain sampling result of the first video sequence) and the time domain sampling information, so that after the video decoding end decodes the received code stream to obtain the second video sequence and the time domain sampling information, it determines the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, and then generates the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame. The first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0073] In some embodiments, the time domain sampling information may include time domain sampling multiple index information, so that the video decoding end can determine the video frame information (e.g., the number of video frames) to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling multiple index information. The time domain sampling multiple index information can be information that has a certain mapping relationship with the time domain sampling multiple. The video decoding end can determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence, the video frame sequence number, and other information based on the decoded time domain sampling multiple index information.

[0074] In some embodiments, if the number of video frames to be inserted between the first video frame and the second video frame is one, the video decoding end may directly generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame.

[0075] In other embodiments, if there are multiple video frames to be inserted between the first video frame and the second video frame, the video decoding end may repeatedly perform the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: based on the first video frame and the second video frame, generate a first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located in the middle position between the first video frame and the second video frame; based on the first video frame and the first intermediate video frame, generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located in the middle position between the first video frame and the first intermediate video frame; based on the first intermediate video frame and the second video frame, generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located in the middle position between the first intermediate video frame and the second video frame.

[0076] In one embodiment, any two video frames can be input into a pre-built video frame infill network model, and a video frame located in the middle between the two video frames can be output, wherein the video frame infill network model is used to generate a video frame located in the middle between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0077] For example, the first video frame and the second video frame are input into the video frame complementation network model, and the first intermediate video frame (located in the middle position between the first video frame and the second video frame) is output; the first video frame and the first intermediate video frame are input into the video frame complementation network model, and the second intermediate video frame (located in the middle position between the first video frame and the first intermediate video frame) is output; the first intermediate video frame and the second video frame are input into the video frame complementation network model, and the third intermediate video frame (located in the middle position between the first intermediate video frame and the second video frame) is output.

[0078] Furthermore, in some embodiments, the time domain sampling information may further include: time domain sampling position information, where the time domain sampling position information is used to determine a position where the video encoder performs time domain sampling on the first video sequence.

[0079] In one embodiment, when the first video sequence includes multiple sub-video sequences, and the multiple sub-video sequences use different time domain sampling multiples for time domain sampling; the above-mentioned time domain sampling multiple index information is also used to determine the time domain sampling multiples for time domain sampling of each sub-video sequence in the first video sequence; the above-mentioned time domain sampling position information is also used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

[0080] Furthermore, in some embodiments, the time domain sampling information may further include: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0081] In some embodiments, before generating the video frame to be inserted, the video decoding end is further used to: obtain the first video frame obtained by decoding the code stream; filter the first video frame; and use the filtered video frame to replace the first video frame in the second video sequence.

[0082] In some embodiments, the video decoding end may filter the first video frame using, but not limited to, the filter shown in the above formula (1).

[0083] In some embodiments, the video decoding end is further configured to insert the video frame to be inserted between the first video frame and the second video frame into a corresponding position between the first video frame and the second video frame in the second video sequence.

[0084] FIG3 shows a flow chart of a video decoding method for machine vision tasks according to an embodiment of the present disclosure. As shown in FIG3 , the video decoding method may include the following steps:

[0085] S302: Decode a bitstream from the video encoder to obtain a second video sequence and time-domain sampling information, wherein the bitstream is an encoding result of the video encoder encoding the time-domain sampling result and the time-domain sampling information of the first video sequence, and the time-domain sampling information is used to determine at least a time-domain sampling multiple used by the video encoder to perform time-domain sampling on the first video sequence.

[0086] S304 : Determine the number of video frames to be inserted between a first video frame and a second video frame in a second video sequence according to the decoded time-domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0087] It should be noted that, in the embodiment of the present disclosure, both the first video sequence and the second video sequence are sequences comprising one or more video frames. The first video sequence refers to the original video sequence before encoding, and the second video sequence is the video sequence after decoding.

[0088] In some embodiments, the above S304 can be implemented by the following steps: if the number of video frames to be inserted between the first video frame and the second video frame is one, then directly generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is multiple, then repeat the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: based on the first video frame and the second video frame, generate a first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; based on the first video frame and the first intermediate video frame, generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; based on the first intermediate video frame and the second video frame, generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

[0089] In one embodiment, when determining a video frame located in the middle position between two video frames, any two video frames can be input into a pre-built video frame infill network model, and a video frame located in the middle position between the two video frames can be output, wherein the video frame infill network model is used to generate a video frame located in the middle position between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0090] S306 : Generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0091] In some embodiments, the above-mentioned time domain sampling information includes: time domain sampling multiple index information, wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; the above-mentioned S304 can be specifically implemented through the following steps: parsing the decoded time domain sampling information to obtain the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence; according to the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence, determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence.

[0092] Furthermore, in some embodiments, the time domain sampling information may further include: time domain sampling position information, where the time domain sampling position information is used to determine a position where the video encoder performs time domain sampling on the first video sequence.

[0093] In one embodiment, when a first video sequence includes multiple sub-video sequences, and the multiple sub-video sequences use different temporal sampling multiples for temporal sampling, the temporal sampling multiple index information is further used to determine the temporal sampling multiples used for temporal sampling of each sub-video sequence in the first video sequence; and the temporal sampling position information is further used to determine the position of temporal sampling of each sub-video sequence in the first video sequence. Optionally, the multiple sub-video sequences of the first video sequence that use different temporal sampling multiples for temporal sampling can be continuous or discontinuous, and can be part of the video sequence in the first video sequence, or the entire video sequence.

[0094] Furthermore, in some embodiments, the time domain sampling information may further include: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0095] Based on the above embodiments, in one embodiment, after decoding the code stream from the video encoding end to obtain the second video sequence and time domain sampling information, the video decoding method for machine vision tasks provided in the embodiments of the present disclosure may also include the following steps: determining whether the video encoding end has performed time domain sampling on the first video sequence based on the decoded time domain sampling information; if the video encoding end has not performed time domain sampling on the first video sequence, the video decoding end does not need to perform subsequent frame insertion operations; conversely, if the video encoding end has performed time domain sampling on the first video sequence, determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, and then generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame.

[0096] In some embodiments, when the code stream is an encoding result obtained by encoding a first video sequence and time domain sampling information using a hybrid encoding method at a video encoding end, before generating a video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame (also known as time domain upsampling), the video decoding method for machine vision tasks provided in the embodiments of the present disclosure may further include the following steps: obtaining a first video frame obtained by decoding the code stream; filtering the first video frame; and replacing the first video frame in the second video sequence with the filtered video frame.

[0097] In some embodiments, the video decoding method for machine vision tasks provided in the embodiments of the present disclosure filters the first video frame using the filter shown in the above formula (1).

[0098] In some embodiments, after generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame, the video encoding method for machine vision tasks provided in the embodiments of the present disclosure may also include the following steps: inserting the video frame to be inserted between the first video frame and the second video frame into the corresponding position between the first video frame and the second video frame in the second video sequence.

[0099] Taking the time domain sampling multiples of 2 and 4 as an example, the video encoding method and video decoding method for machine vision tasks provided in the embodiments of the present disclosure are described in detail with reference to FIG4 to FIG6 .

[0100] FIG4 shows a schematic diagram of temporal downsampling for machine vision; FIG5 shows a schematic diagram of temporal upsampling for machine vision; and FIG5 shows a video interpolation network model (neural network model) for video interpolation processing.

[0101] In specific implementation, the video encoding end encodes the time domain sampling flag, time domain sampling multiple index information, time domain sampling position and other time domain sampling information and the video image information after time domain sampling and sends them to the video decoding end; after the video decoding end decodes the received code stream, it obtains the time domain sampling flag, time domain sampling multiple index information, time domain sampling position and other time domain sampling information and the video image information after time domain sampling, and restores the time domain sampling multiple according to the time domain sampling multiple index information.

[0102] After decoding and obtaining the second video sequence, the video decoding end may perform the following processing on two consecutive frames:

[0103] When the time domain sampling multiple is 2, the discarded frame The frames I' before and after decoding nT and I′ nT+2 Insertion:

[0104] When the time domain sampling multiple is 4, the discarded frames are inserted in a cascade manner, where the discarded frames The frames I' before and after decoding nT and I′ nT+4 Insertion:

[0105] Next, the discarded frames and The decoded frame I' iT and I′ iT+4 Separate and interpolated frames Fill in the frame.

[0106] Wherein, n represents a natural number, and its value is n=1, 2, 3, ...; F represents a filter.

[0107] As shown in Figure 6, the video frame interpolation network model includes: FMI-Updater (optical flow, mask and image updater) and IFBlock (frame interpolation module). The video frame interpolation network model directly predicts the intermediate frame from the input frame. For 2x temporal upsampling (temporal sampling multiple is 2), the frame interpolation By input frame (I' nT ,I' nT+2 ) prediction. For 4 times time domain upsampling (time domain sampling multiple is 4), From the forward frame I′ nTand backward frame I′ nT+2 Prediction, frame interpolation By input frame (I' nT ,I' nT+4 )predict, From the forward frame I′ nT+2 Backward frame I′ nT+4 Prediction. The entire network has a stacked hourglass structure, consisting of three consecutive FMI-Updaters and a final image merging module. FMI-Updater is used to learn the residual optical flow and mask from the input frame and update the optical flow and mask by their respective residuals. In addition to the updated optical flow and mask (F, M) i , the module output consists of the input frame (I' nT ,I' nT+k ) i-1 Updated Optical Flow and The updated frame obtained by distortion (I' nT ,I' nT+k ) i .

[0108] In some embodiments, the time domain sampling multiple index [0, 1] corresponds to the time domain sampling multiple [2, 4] respectively, that is, when the time domain sampling multiple is 2, the time domain sampling multiple index is 0, and when the time domain sampling multiple is 4, the time domain sampling multiple index is 1.

[0109] Two specific examples are listed below to illustrate the embodiments of the present disclosure:

[0110] Example 1: Taking the SFU-HW Blowing Bubble (416x240, 50fps, 97 frames) as an example, the encoder extracts one frame every four frames from the original video and sends the extracted video frame, the temporal sampling flag, and the temporal sampling multiple index 2 to the hybrid encoder VCMRS. The temporal sampling flag and the temporal sampling multiple index can use the NNPFC SEI syntax element.

[0111] Wherein, nnpfc_purpose=0x08; nnpfc_interpolated_pics=3; nnpfc_id is the specified serial number of the interpolation model (such as 256), nnpfc_num_input_pics_minus1=1 (numInputPics=nnpfc_num_input_pics_minus1+1); the newly added SEI syntax elements temporal_resampling_flag=1, temporal_resampling_ratio_idx=2 can also be used.

[0112] temporal_resampling_flag indicates the time domain sampling flag;

[0113] temporal_resampling_ratio_idx represents the time domain sampling multiple index.

[0114] The decoder parses the received bitstream to obtain the decoded video frame, the temporal sampling flag, and the temporal sampling multiple index. The temporal sampling multiple is then mapped. The hybrid encoder (VCMRS) uses end-to-end encoding for key frames. The reconstructed key frames from the end-to-end encoding and decoding are fed into the VTM encoding to replace the intra-frame encoding. VTM is then used for inter-frame encoding. The video frame V1 reconstructed by the VTM can optionally be adapted for machine vision tasks to obtain the reconstructed video frame V2.

[0115] The first frame in the video frame decoded by the hybrid encoder VCMRS is a key frame. End-to-end encoding is used. The reconstructed image, which is jointly trained with deep learning-based end-to-end image coding and machine vision tasks, is prone to checkerboard artifacts, which affects temporal upsampling of the video. Therefore, filtering is applied to the first frame in the video frame decoded by the hybrid encoder.

[0116] img = cv2.imread(srcimg);

[0117] kernel=np.array([[0.0625,0.125,0.0625],[0.125,0.25,0.125],[0.0625,0.125,0.0625]]);

[0118] desimg=cv2.filter2D(img,-1,kernel);

[0119] The filtered image is overlaid on the original image, and the decoded video frames are interpolated in pairs. nT ,I nT+4 ) are interleaved in sequence (I nT+2 ,I nT+1 ,I nT+3 ).

[0120] Example 2: Using TVD 01_1 (1920x1080, 500 frames at 50fps) as an example, with 4:1 downsampling and temporal upsampling, frames 498-499-500 can be discarded and decoded without reconstruction. Alternatively, a copy-and-repeat operation can be used, where three identical images are copied from the reconstructed 497 frame to form a complete 500-frame image.

[0121] FIG7 shows a flow chart of another video decoding method for machine vision tasks according to an embodiment of the present disclosure. As shown in FIG7 , the video decoding method for machine vision tasks provided in an embodiment of the present disclosure may include the following steps:

[0122] S702, decoding the video stream to be decoded to obtain a decoded video sequence and time domain sampling information;

[0123] S704 , performing time domain recovery processing on the decoded video sequence according to the time domain sampling information.

[0124] In some embodiments, the time domain sampling information includes: a time domain recovery flag and time domain resampling ratio information, wherein the time domain recovery flag is used to indicate whether time domain recovery processing is enabled; the time domain resampling ratio information is used to determine the time domain resampling ratio during time domain recovery processing.

[0125] In the embodiment of the present disclosure, the time domain sampling information can be sequence-level information or frame-level information, that is, the time domain recovery flag and the time domain resampling ratio information can be sequence-level information or frame-level information.

[0126] In one embodiment, the time domain resampling ratio information may be directly the time domain resampling ratio; in another embodiment, the time domain resampling ratio information may be a time domain resampling ratio index, and then the time domain resampling ratio during the time domain recovery processing is determined according to the time domain resampling ratio index.

[0127] In some embodiments, time domain recovery processing is performed on the decoded video sequence based on the time domain sampling information, which can be achieved through the following steps: determining whether to perform time domain recovery processing on the decoded video sequence based on the time domain recovery flag contained in the time domain sampling information; when it is determined that time domain recovery processing is to be performed on the video stream to be decoded, determining the time domain resampling ratio based on the time domain resampling ratio information contained in the time domain sampling information; and performing time domain recovery processing on the decoded video sequence based on the time domain resampling ratio.

[0128] In some embodiments, time domain recovery processing is performed on the decoded video sequence according to the time domain resampling ratio, which can be achieved by the following steps: determining the number of video frames to be inserted between the first video frame and the second video frame according to the time domain resampling ratio, wherein the first video frame and the second video frame are any two consecutive video frames in the decoded video sequence; generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame.

[0129] In some embodiments, generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame can be achieved by the following steps: if the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is multiple, repeating the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: according to the first video frame and the second video frame, Generate a first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located in the middle position between the first video frame and the second video frame; based on the first video frame and the first intermediate video frame, generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located in the middle position between the first video frame and the first intermediate video frame; based on the first intermediate video frame and the second video frame, generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located in the middle position between the first intermediate video frame and the second video frame.

[0130] In some embodiments, when there are multiple second intermediate video frames and multiple third intermediate video frames to be inserted between the first video frame and the second video frame, in a specific implementation, the steps of generating the second intermediate video frame and the third intermediate video frame are executed in any one of the following order:

[0131] ① First, execute the step of generating the second intermediate video frame until all the second intermediate video frames are generated, and then execute the step of generating the third intermediate video frame until all the third intermediate video frames are generated;

[0132] ② respectively executing the steps of generating a second intermediate video frame and generating a third intermediate video frame;

[0133] ③ First execute the step of generating the third intermediate video frame until all the third intermediate video frames are generated, and then execute the step of generating the second intermediate video frame until all the second intermediate video frames are generated.

[0134] In some embodiments, the video decoding method for machine vision tasks provided in the embodiments of the present disclosure may also include the following steps: generating an interpolated frame between two video frames based on a pre-built video frame interpolation network model, wherein the video frame interpolation network model is used to generate an interpolated frame between two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0135] In some embodiments, the video frame interpolation network model is an intermediate flow estimation network IFNet model.

[0136] FIG8 shows a flow chart of a method for generating an interpolated frame in an embodiment of the present disclosure. As shown in FIG8 , the video decoding method for machine vision tasks provided in an embodiment of the present disclosure can also generate an interpolated frame through the following steps:

[0137] S802, converting the first video frame in YUV format into a third video frame in RGB format;

[0138] S804, converting the second video frame in YUV format into a fourth video frame in RGB format;

[0139] S806 , generating an interpolated frame in RGB format according to the third video frame and the fourth video frame based on a pre-built video frame interpolation network model;

[0140] S808, converting the interpolation frame in RGB format into an interpolation frame in YUV format;

[0141] S810: Insert an interpolated frame in a YUV format between the first video frame and the second video frame.

[0142] In one embodiment, a YUV format video frame is converted into an RGB format video frame by the following steps: resizing the YUV format video frame to be converted to obtain a YUV format image that meets a preset size; extracting the Y component, U component, and V component of each pixel from the YUV format image; performing a bitwise right shift operation on the Y component, U component, and V component of each pixel; calculating the R component, G component, and B component of each pixel based on the Y component, U component, and V component after the bitwise right shift operation, wherein, if there are remaining pixels in the resized image, the R component, G component, and B component of the last column of pixels are used as the R component, G component, and B component of the remaining column pixels; the R component, G component, and B component of the last row of pixels are used as the R component, G component, and B component of the remaining row pixels; and the calculated R component, G component, and B component of each pixel are normalized.

[0143] In the above embodiment, the R component, G component, and B component of each pixel after normalization are as follows:

[0144] R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / 255;

[0145] G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / 255;

[0146] B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / 255;

[0147] Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

[0148] In another embodiment, a YUV format video frame is converted into an RGB format video frame by the following steps: the YUV format video frame to be converted is resized to obtain a YUV format image that meets a preset size (such as the image width and height are both adjusted to a multiple of 32); the Y component, U component, and V component of each pixel are extracted from the YUV format image; the R component, G component, and B component of each pixel are calculated based on the Y component, U component, and V component of each pixel, wherein, if there are remaining pixels in the image after resizing (such as the image width and height are both adjusted to a multiple of 32), the R component, G component, and B component of the last column of pixels are used as the R component, G component, and B component of the remaining column pixels; the R component, G component, and B component of the last row of pixels are used as the R component, G component, and B component of the remaining row pixels; and the calculated R component, G component, and B component of each pixel are normalized.

[0149] In the above embodiment, the R component, G component, and B component of each pixel after normalization are as follows:

[0150] R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / scale;

[0151] G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / scale;

[0152] B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / scale;

[0153] Where scale'=2^(BitDepth);

[0154] Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

[0155] In one embodiment, a video frame in RGB format is converted into a video frame in YUV format through the following steps: extracting the R component, G component, and B component of each pixel; scaling the R component, G component, and B component of each pixel so that the values ​​of the scaled R component, G component, and B component are limited to a preset range; calculating the Y component, U component, and V component of each pixel based on the scaled R component, G component, and B component; and performing a bitwise left shift operation on the calculated Y component, U component, and V component of each pixel.

[0156] In the above embodiment, the Y component, U component, and V component of each 2x2 pixel block after the bitwise left shift operation are as follows:

[0157] pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale);

[0158] pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j+1]+0.114*p[2,i+1,j])*scale);

[0159] pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1,2j]+0.114*p[2,i+1,j])*scale);

[0160] pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587*p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale);

[0161] pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2,i,j])*scale);

[0162] pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2, i,j])*scale);

[0163] Where scale = 2^(BitDepth - 8);

[0164] Where BitDepth is the bit width of all luminance and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component of pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB components of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB components of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB components of pixel block (i,j) in channel k=2.

[0165] In another embodiment, a video frame in RGB format is converted into a video frame in YUV format by the following steps: extracting the R component, G component, and B component of each pixel; calculating the Y component, U component, and V component of each pixel based on the R component, G component, and B component of each pixel; and performing a bitwise left shift operation on the calculated Y component, U component, and V component of each pixel.

[0166] In the above embodiment, the Y component, U component, and V component of each pixel after the bitwise left shift operation are as follows:

[0167] pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale);

[0168] pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j+1]+0.114*p[2,i+1,j])*scale);

[0169] pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1,2j]+0.114*p[2,i+1,j])*scale);

[0170] pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587*p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale);

[0171] pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2,i,j])*scale);

[0172] pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2,i,j])*scale);

[0173] Where scale'=2^(BitDepth);

[0174] Where BitDepth is the bit width of all luminance and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component of pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB components of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB components of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB components of pixel block (i,j) in channel k=2.

[0175] Table 1

[0176] In one embodiment, the time domain sampling information in the embodiment of the present disclosure includes the content shown in Table 1, which at least includes a time domain restoration flag (temporal_restoration_flag) and a time domain resampling ratio index (temporal_resampling_ratio_idx), wherein the time domain restoration flag is used to indicate whether the time domain restoration (time domain resampling) process is enabled, 1 indicates that the time domain restoration process is enabled, and 0 indicates that the time domain restoration process is disabled; the time domain resampling ratio index is used to determine the time domain resampling ratio. The relationship between the time domain resampling ratio and the time domain resampling ratio index is as follows:

[0177] TemporalResamplingRatio=2temporal_resampling_ratio_idx;

[0178] Among them, TemporalResamplingRatio represents the time domain resampling ratio;

[0179] temporal_resampling_ratio_idx represents the temporal resampling ratio index.

[0180] When temporal_restoration_flag=1, the following temporal restoration process is performed. The input and output of this process are as follows:

[0181] Input: pictListIn, an input picture list containing pairs of adjacent frames, each pair of adjacent frames is in YUV format (such as YUV420P10LE format);

[0182] Output: Output picture list pictListOut after inserting the recovered frame.

[0183] The implementation process is:

[0184] 1) Format conversion: convert a pair of adjacent frames in the input picture list pictListIn from YUV format (such as YUV420P10LE) to RGB format (such as RGB24 format).

[0185] 2) Generate temporal restored frames (interpolated frames): Using the converted adjacent frames and the number of interpolated frames between adjacent frames (i.e., the number of video frames to be inserted between two video frames, which is taken as TemporalResamplingRatio-1), the following temporal interpolation process is used to generate restored frames, ultimately obtaining (TemporalResamplingRatio-1) RGB format (e.g., RGB24 format) restored frames.

[0186] 3) Format conversion: converting the recovered frame from RGB format (such as RGB24 format) back to YUV format (such as YUV420P10LE format).

[0187] 4) Insert the restored frame between corresponding adjacent frames in the input picture list pictListIn to obtain the output picture list pictListOut.

[0188] The following describes in detail the various sub-processes involved in the time domain recovery process in the embodiment of the present disclosure:

[0189] 1) Format conversion: This part includes two sub-parts: converting the YUV format to RGB format and converting the RGB format to YUV format.

[0190] ① Convert YUV420P10LE format to RGB24 format:

[0191] Input image pictIn: YUV420P10LE format, where the sizes of the luminance component (Y component) and the two chrominance components (U component and V component) are: pictInY[hIn, wIn], pictInU[hIn / 2, wIn / 2], pictInV[hIn / 2, wIn / 2], respectively. pictInY represents the luminance component of the input image, pictInU represents the U component of the input image, and pictInV represents the V component of the input image. hIn represents the image height of the input image, and wIn represents the image width of the input image.

[0192] Output image pictOut: RGB24 format, size is [3, hOut, wOut], where hOut represents the image height of the output image and wOut represents the image width of the output image.

[0193] The steps to convert from YUV420P10LE format to RGB24 format are as follows:

[0194] I. Adjust image size: Round up the output image height hOut and width wOut to multiples of 32 using the following formulas: hOut = Ceil(hIn / 32)*32; wOut = Ceil(wIn / 32)*32. Ceil(x) represents x rounded up to the nearest integer.

[0195] II. For each pixel (i, j) in the input image, where i ranges from 0 to hIn-1 (inclusive) and j ranges from 0 to wIn-1 (inclusive), perform a bitwise right shift of pictInY[i,j], pictInU[i / 2,j / 2], and pictInV[i / 2,j / 2] by (BitDepth-8) bits to obtain the corresponding Y, U, and V components, respectively. BitDepth is the bit width of all decoded luma and chroma components, as specified by the VCM syntax.

[0196] III. Using the YUV component to RGB component conversion formula, calculate the R component, G component, and B component based on the Y component, U component, and V component that perform the bitwise right shift operation above:

[0197] R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / 255;

[0198] G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / 255;

[0199] B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / 255;

[0200] Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j); in the above formula, the calculation of the R component, G component and B component divided by 255 is the normalization processing of the converted R component, G component and B component.

[0201] IV. Store the calculated R component, G component, and B component into the output picture pictOut.

[0202] V. Fill remaining pixels: If there are still remaining pixels after the image size is adjusted, the RGB value of the last pixel of the original image is copied to the remaining pixels.

[0203] VI. Output the image pictOut in RGB24 format.

[0204] The implementation code example is as follows:

[0205] ②Convert RGB24 format to YUV420P10LE format:

[0206] Input image pictIn: RGB24 format, size [3, hIn, wIn], where hIn represents the image height of the input image, and wIn represents the image width of the input image;

[0207] Output image pictOut: YUV420P10LE format, where the sizes of the luminance component (Y component) and the two chrominance components (U component and V component) are: pictOutY[hOut, wOut], pictOutU[hOut / 2, wOut / 2], pictOutV[hOut / 2, wOut / 2], respectively; pictOutY represents the luminance component of the output image, pictOutU represents the U component of the output image, pictOutV represents the V component of the output image, hIn represents the image height of the input image, and wIn represents the image width of the input image.

[0208] The steps to convert from RGB24 format to YUV420P10LE format are as follows:

[0209] I. RGB component scaling: Limit the RGB component values ​​to the range of [0, 255]: Scale the RGB components of each pixel (i, j) in the input image so that their values ​​are in the range of [0, 255]. For each pixel (i, j) and each channel k (0, 1, 2), the RGB component scaling is performed using the following formula:

[0210] p[k,i,j]=clip3(0,255,(pictIn[k,i,j]*255));

[0211] Among them, clip3(x,min,max) means limiting the value of the input data x to between the minimum value min and the maximum value max.

[0212] II. Using the RGB to YUV conversion formula, calculate the Y, U, and V components based on the scaled R, G, and B components:

[0213] Y=0.299*R+0.587*G+0.114*B;

[0214] U=-0.169*R-0.331*G+0.499*B;

[0215] V=0.399*R-0.418*G-0.0813*B;

[0216] III. Perform a bitwise left shift operation on the Y component, U component, and V component: multiply the Y component, U component, and V component by a scaling factor scale, scale = 2^(BitDepth-8), where BitDepth is the bit width of all luminance and chrominance components, indicated by the VCM syntax element.

[0217] The Y component, U component, and V component of each pixel are calculated as follows:

[0218] pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale);

[0219] pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j+1]+0.114*p[2,i+1,j])*scale);

[0220] pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1,2j]+0.114*p[2,i+1,j])*scale);

[0221] pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587*p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale);

[0222] pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2,i,j])*scale);

[0223] pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2,i,j])*scale).

[0224] Among them, BitDepth is the bit width of all luminance and chrominance components; pictOutY[i,j] represents the Y component of pixel (i,j); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel (i,j); p[0,i,j] represents the RGB components of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB components of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB components of pixel (i,j) in channel k=2.

[0225] Output: Output image pictOut in YUV420P10LE format.

[0226] The implementation code example is as follows:

[0227] 2) Time domain interpolation process:

[0228] The input is: ① the first reference image img0 and the second reference image img1, a pair of images in RGB24 format; ②n, the number of images that need to be interpolated between img0 and img1.

[0229] The output is: pListOut.

[0230] The time interpolation process includes the following steps:

[0231] I. Initialize pListOut to empty.

[0232] II. Update pListOut through the time domain interpolation sub-process.

[0233] Among them, the time domain interpolation sub-process:

[0234] Input: ① first reference image, aImg; ② second reference image, bImg; ③ number of images to be interpolated, m;

[0235] Output: Updated pListOut.

[0236] The time interpolation sub-process includes the following steps:

[0237] Through the picture interpolation process described below, an interpolated picture midImg between the first and second reference pictures is generated.

[0238] If pListOut is empty, add midImg to pListOut;

[0239] Otherwise, if the first reference picture is one of the pictures in pListOut, insert midImg into pListOut at a position after the first reference picture;

[0240] Otherwise, insert midImg into pListOut at a position before the second reference picture.

[0241] If m>1, update m to m / 2.

[0242] Furthermore, the temporal interpolation sub-process is iteratively executed with the following data as input: ① aImg as the first reference picture; ② midImg as the second reference picture; ③ m as the number of pictures to be interpolated.

[0243] Furthermore, the temporal interpolation sub-process is iteratively executed with the following data as input: ① midImg as the first reference picture; ② bImg as the second reference picture; ③ m as the number of pictures to be interpolated.

[0244] 3) Image interpolation process:

[0245] Input: ①fwdImg and bckImg, a pair of RGB24 format images with shape [3,h,w]; ②weight additional identifier.

[0246] Output: an interpolated image outImg in RGB24 format with a shape of [3, h, w].

[0247] The image interpolation process includes the following steps:

[0248] ① Initialize the following data:

[0249] scaleList = [4, 2, 1];

[0250] Set all elements of flow[4,h,w] to 0;

[0251] Set all elements of mask[1,h,w] to 0;

[0252] p0=fwdImg;

[0253] p1 = bckImg;

[0254] ②For i=0,…,2, apply the following operations:

[0255] Execute the IFBlock procedure with the following data as input:

[0256] ①Concatenate((p0,p1,mask),0);

[0257] ② Optical flow information flow;

[0258] ③scaleList[i];

[0259] ④The weight of the additional identifier is IFNet.IFBlock;

[0260] The outputs of the IFBlock process are: f0 and m0.

[0261] Furthermore, the IFBlock process is executed with the following data as input:

[0262] ①Concatenate((p1,p0,(–mask)),0);

[0263] ②Concatenate((flow[2:4,:,:],flow[:2,:,:]),0);

[0264] ③scaleList[i];

[0265] ④The weight of the additional identifier is IFNet.IFBlock;

[0266] The outputs of the IFBlock process are: f1 and m1.

[0267] flow=flow+(f0+Concatenate((f1[2:4],f1[:2]),0)) / 2;

[0268] mask=mask+(m0-m1) / 2;

[0269] p0=Warp(fwdImg,flow[:2,:,:]), the specific process is as follows.

[0270] p1=Warp(bckImg,flow[2:4,:,:]). The specific process is as follows.

[0271] maskOut=ApproxSigmoid(mask);

[0272] outImg=p0*maskOut+p1*(1-maskOut).

[0273] 4) IFBlock (interpolation frame block) process:

[0274] 4.1) Overview:

[0275] The input is: ① a tensor x[7,h,w]; ② a tensor flow[4,h,w]; ③ a variable scaleVal; ④ a weight additional identifier.

[0276] The output is: a tensor flowOut[4,h,w] and a tensor maskOut[1,h,w].

[0277] The IFBlock process includes the following steps:

[0278] I. Assign x1 to the nearest exact interpolation NearestInterpolate(x, scaleVal -1 ,7,h,w) the results obtained, the NearestInterpolate function is described as follows.

[0279] II. Assign x2 to the nearest exact interpolation NearestInterpolate(flow, scaleVal -1 ,4,h,w) the results obtained.

[0280] III. Update all elements of x2 to be multiplied by scaleVal -1 .

[0281] IV. Concatenate x1 and x2 along the channel dimension and assign the result to x3.

[0282] V. Assign x4 to the result of the following IncreConv function:

[0283] IncreConv(x3,IFBlock.conv0.0.0.conv,IFBlock.conv0.0.0.bias,IFBlock.conv0.0.1.

[0284] act,IFBlock.conv0.1.0.conv,IFBlock.conv0.1.0.bias,IFBlock.conv0.1.1.act,11,90).

[0285] The IncreConv function is described below.

[0286] VI. Set x[0] to x4. For i ranging from 1 to 4 (inclusive), perform the following steps:

[0287] Assign x[i] to the calculation result of the following StackConv function:

[0288] StackConv(x[i-1],IFBlock.convblock.0.0.conv,IFBlock.convblock.0.0.bias,IFBlock.convblock.0.1.act,IFBlock.convblock.1.0.conv,IFBlock.convblock.1.0.bias,IFBlock k.convblock.1.1.act,90). The StackConv function is described below.

[0289] Set x[i] to x[i] plus x[i-1].

[0290] VII. Assign flow to the calculation result of the following ShrinkTransConv function:

[0291] ShrinkTransConv(x[4],IFBlock.conv1.0.transConv,IFBlock.conv1.0.transBias,IFBloc k.conv1.1.act,IFBlock.conv1.2.transConv,IFBlock.conv1.2.transBias,90,4). The ShrinkTransConv function is described below.

[0292] VIII. Assign Mask to the result of the following ShrinkTrans function:

[0293] ShrinkTransConv(x[4],IFBlock.conv2.0.transConv,IFBlock.conv2.0.transBias,IFBloc k.conv2.1.act,IFBlock.conv2.2.transConv,IFBlock.conv2.2.transBias,90,1).

[0294] IX. Output tensor:

[0295] flow_out=NearestInterpolate(flow,scaleVal,4,h,w)*scaleVal;

[0296] X. Output tensor mask_out = NearestInterpolate(mask, scaleVal, 1, h, w).

[0297] 4.2) Nearest interpolation function:

[0298] The nearest interpolation function is expressed as NearestInterpolate(x, scale, cIn, hIn, wIn).

[0299] The input to this function is:

[0300] ①x: A tensor of shape [cIn, hIn, wIn].

[0301] ②scale: scaling factor of spatial dimensions.

[0302] ③cIn,hIn,wIn: the size of the input tensor x.

[0303] The output of this function is the tensor x_out[cIn,hOut,wOut], which is derived as follows:

[0304] x_out[c,i,j]=x[c,Round(i*scale),Round(j*scale)];

[0305] Among them, the range of c is 0 to cIn-1; the range of i is 0 to hOut-1; the range of j is 0 to wOut-1; hOut=Round(hIn*scale); wOut=Round(wIn*scale).

[0306] Nearest interpolation is an image scaling technique that enlarges an image by copying pixel values ​​to the nearest location. In this function, each pixel (c,i,j) in the input tensor x is copied to the position (c,Round(i*scale),Round(j*scale)) in the output tensor x_out, where scale is the scaling factor that controls the size of the output image.

[0307] 4.3) Quantized convolution process with activation:

[0308] The quantized convolution function with activation is represented as ConvAct(x,weightConv,bias,weightAct,kVer,kHor,cIn,cOut,s).

[0309] The input to this function is:

[0310] ①x: a tensor of shape [cIn, hIn, wIn];

[0311] ②weightConv: A tensor of shape [cOut, cIn, kVer, kHor], representing the convolution kernel weight;

[0312] ③bias: A tensor of shape [cOut], representing the convolution kernel bias;

[0313] ④weightAct: A tensor of shape [cOut], representing the activation function weight;

[0314] ⑤kVer,kHor: vertical and horizontal sizes of the convolution kernel;

[0315] ⑥cIn: the number of input channels;

[0316] ⑦cOut: the number of output channels;

[0317] ⑧s: step length;

[0318] The output of this process is the tensor x_out[cOut,hIn / s,wIn / s], which is derived as follows:

[0319] The output of this process is derived as follows:

[0320] x0=QConv(x,weightConv,bias,kVer,kHor,cIn,cOut,s);

[0321] x_out=LeakyReLU(x0,weightAct).

[0322] The quantized convolution with activation function combines quantized convolution with activation functions. First, the quantized convolution multiplies the input tensor x by the convolution kernel weightConv and adds the bias bias to produce the intermediate result x0. Then, the parameterized rectified linear unit multiplies x0 by the activation function weight weightAct to produce the final output tensor x_out.

[0323] 4.4) Incremental convolution process:

[0324] The input to this process is:

[0325] ① A tensor x[cIn,hIn,wIn];

[0326] ②A tensor weight0Conv[cOut / 2,cIn,kVer,kHor];

[0327] ③A tensor weight0Bias[cOut / 2];

[0328] ④ A tensor weight0Act[cOut / 2];

[0329] ⑤A tensor weight1Conv[cOut,cOut / 2,kVer,kHor];

[0330] ⑥A tensor weight1Bias[cOut / 2];

[0331] ⑦A tensor weight1Act[cOut];

[0332] ⑧Input channel size cIn;

[0333] ⑨Output channel size cOut.

[0334] The output of this process is the tensor x_out[cOut,hIn / 4,wIn / 4].

[0335] The incremental convolution process is expressed as:

[0336] IncreConv(x,weight0Conv,weight0Bias,weight0Act,weight1Conv,weight1Bias,weight1Act,cIn,cOut). The output of this process is derived as follows:

[0337] x0=ConvAct(x,weight0Conv,weight0Bias,weigth0Act,3,3,cIn,cOut / 2,2)

[0338] x_out=ConvAct(x0,weight1Conv,weight1Bias,weight1Act,3,3,cOut / 2,cOut.

[0339] 4.5) Stacked convolution process with activation:

[0340] The input to this process is:

[0341] ① A tensor x[c,h,w];

[0342] ②A tensor weight0Conv[c,c,kVer,kHor];

[0343] ③A tensor weight0Bias[c];

[0344] ④A tensor weight0Act[c];

[0345] ⑤A tensor weight1Conv[c,c,kVer,kHor];

[0346] ⑥A tensor weight1Bias[c];

[0347] ⑦A tensor weight1Act[c];

[0348] ⑧Input and output channel size c.

[0349] The output of this process is: tensor x_out[c,h,w].

[0350] The stacked convolution process with activation is represented as StackConv(x,weight0Conv,weight0Bias,weight0Act,weight1Conv,weight1Bias,weight1Act,c). The output of this process is derived as follows:

[0351] x0=ConvAct(x,weight0Conv,weight0Bias,weight0Act,3,3,c,c,1);

[0352] x_out=ConvAct(x0,weight1Conv,weight1Bias,weight1Act,3,3,c,c,1).

[0353] 4.6) Contraction transposed convolution process:

[0354] The input to this process is:

[0355] ① A tensor x[c,hIn,wIn];

[0356] ②A tensor weightTransConv0[cIn / 2,cIn,kVer,kHor];

[0357] ③A tensor weightTransBias0[cIn / 2]

[0358] ④A tensor weightAct[cIn / 2];

[0359] ⑤A tensor weightTransConv1[cOut,cIn / 2,kVer,kHor];

[0360] ⑥A tensor weightTransBias1[cOut]

[0361] ⑦ Input channel size cIn;

[0362] ⑧Output channel size cOut.

[0363] The output of this process is the tensor x_out[cOut,hIn*4,wIn*4].

[0364] The shrinkage transposed convolution process is expressed as:

[0365] ShrinkTransConv(x,weightTransConv0,weightTransBias0,weightAct,weightTransConv1,weightTransBias1,cIn,cOut).

[0366] The output of this process is derived as follows:

[0367] x0=QTransposedConv(x,weightTransConv0,weightTransBias0,4,4,cIn,cIn / 2,2);

[0368] x1=LeakyReLU(x0,weightAct);

[0369] x_out=QTransposedConv(x1,weightTransConv1,weightTransBias1,4,4,cIn / 2, Cout,5).

[0370] 4.7) Pixel mapping process:

[0371] 4.7.1) Overview:

[0372] The input to this process is:

[0373] ①x: a tensor of shape [c,h,w];

[0374] ②flow: A tensor of shape [2,h,w], flow[h,w,2].

[0375] The output of this process is the tensor xOut[c,h,w], which is derived as follows:

[0376] For a tensor tenHorizontal of shape [1,h,w], tenHorizontal[0,m,:] = LinearSpace(-1.0,1.0,w), where m ranges from 0 to h-1.

[0377] For a tensor tenVertical of shape [1,h,w], tenVertical[0,:,n]=LinearSpace(-1.0,1.0,h), where n ranges from 0 to w-1.

[0378] Concatenate tenHorizontal and tenVertical along the channel dimension and assign the result to tenGrid.

[0379] For a tensor tenFlow of shape [2,h,w], tenFlow[0,m,n] = (flow[0,m,n]*2) / (w-1), where m ranges from 0 to h-1 and n ranges from 0 to w-1. tenFlow[1,m,n] = (flow[1:2,:,:]*2) / (h-1), where m ranges from 0 to h-1 and n ranges from 0 to w-1.

[0380] tenGF[l,m,n]=tenGrid[l,m,n]+tenFlow[l,m,n], where l ranges from 0 to 1, m ranges from 0 to h-1, and n ranges from 0 to w-1.

[0381] For a tensor x5 of shape [h,w,2], x5[:,:,0]=tenGF[0,:,:]x5[:,:,1]=tenGR[1,:,:];

[0382] Use the mapping procedure defined in 4.7.2) with inputs x and x5 and assign the output of the procedure to xOut.

[0383] 4.7.2) Linear space function:

[0384] The input of this function is: ①tart: starting value; ②end: ending value; ③steps: number of steps.

[0385] The output of this function is: a one-dimensional tensor x_out with a shape of [steps]. Its derivation is as follows:

[0386] x_out[i] = start + ((end – start) * i) ÷ (steps – 1), where i ranges from 0 to steps - 1.

[0387] The linear space function generates a one-dimensional tensor containing steps values ​​uniformly distributed between start and end.

[0388] 4.7.3) Mapping:

[0389] The input of this process is: ①x: a tensor with shape [c,h,w]; ②flow: a tensor with shape [h,w,2].

[0390] The output of this process is a tensor x_out of shape [c,h,w], which is derived as follows:

[0391] For i, the range is 0 to h-1, and for j, the range is 0 to wOut-1.

[0392] hor=((flow[i,j,0]+1) / 2)*(h-1);

[0393] ver=((flow[i,j,1]+1) / 2)*(w-1);

[0394] x_out[k,i,j]=x[k,hor,ver], where 0<=k<=c-1.

[0395] Below, we take the input img0 and img 1 and generate the intermediate img0.5 as an example:

[0396] img0: dimension is (3, h, w);

[0397] img1: dimension is (3, h, w);

[0398] flow[:4,h,w]: generated optical flow, flow[:2,h,w] represents the optical flow information of 0->0.5, and flow[2:4,h,w] represents the optical flow information of 1->0.5;

[0399] f1[:2]: is the 0->0.5 residual optical flow output by the module, which is accumulated on flow[:2,h,w] to obtain the optical flow output by the block;

[0400] f1[2:4]: is the 1->0.5 residual optical flow output by the module, which is accumulated on flow[2:4,h,w] to obtain the optical flow output by the block;

[0401] warped_img0[:3,:,:] warps the image img0 based on the flow[:2,h,w] optical flow to obtain an approximate img0.5;

[0402] warped_img1[:3,:,:]: warps the image img1 based on the flow[2:4,h,w] optical flow to obtain an approximate img0.5;

[0403] mask: mask, dimension is (1, h, w), value range is 0-1, warped_img0 and warped_img1 are combined to generate the final img0.5, the formula is as follows:

[0404] img0.5=mask*warped_img0+warped_img1*(1-mask).

[0405] Based on the same inventive concept, the present disclosure also provides a video encoding device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0406] FIG9 shows a schematic diagram of a video encoding device according to an embodiment of the present disclosure. As shown in FIG9 , the device includes: a time domain sampling information acquisition module 901 , a time domain sampling module 902 , and an encoding module 903 .

[0407] Among them, the time domain sampling information acquisition module 901 is used to obtain time domain sampling information, wherein the time domain sampling information is at least used to determine the time domain sampling multiple of the time domain sampling performed by the video encoding end on the first video sequence; the time domain sampling module 902 is used to perform time domain sampling on the first video sequence according to the time domain sampling information; the encoding module 903 is used to encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding end, wherein the video decoding end is used to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, and generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, where the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0408] It should be noted that the time-domain sampling information acquisition module 901, the time-domain sampling module 902, and the encoding module 903 correspond to S202 to S206 in the method embodiment. The examples and application scenarios implemented by these modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system, such as a set of computer-executable instructions.

[0409] Based on the same inventive concept, the present disclosure also provides a video decoding device for machine vision tasks, as described in the following embodiments. Since the principles of the device embodiment are similar to those of the above-mentioned method embodiment, the implementation of the device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0410] FIG10 shows a schematic diagram of a video decoding device for machine vision tasks in an embodiment of the present disclosure. As shown in FIG10 , the device includes: a decoding module 1001 , a video frame interpolation information determination module 1002 , and a video frame interpolation module 1003 .

[0411] Among them, the decoding module 1001 is used to decode the code stream from the video encoding end to obtain a second video sequence and time domain sampling information, wherein the code stream is the encoding result after the video encoding end encodes the time domain sampling result and time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine the time domain sampling multiple used by the video encoding end to time domain sample the first video sequence; the video frame interpolation information determination module 1002 is used to determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; the video frame interpolation module 1003 is used to generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0412] It should be noted that the decoding module 1001, the video frame interpolation information determination module 1002, and the video frame interpolation module 1003 described above correspond to S302 to S306 in the method embodiment. The examples and application scenarios implemented by these modules and corresponding steps are the same, but are not limited to the contents disclosed in the method embodiment described above. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system, such as a set of computer-executable instructions.

[0413] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0414] The electronic device 1100 according to this embodiment of the present disclosure is described below with reference to Figure 11. The electronic device 1100 shown in Figure 11 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0415] As shown in Figure 11, electronic device 1100 is implemented as a general-purpose computing device. Components of electronic device 1100 may include, but are not limited to, the aforementioned at least one processing unit 1110, the aforementioned at least one storage unit 1120, and a bus 1130 connecting various system components (including storage unit 1120 and processing unit 1110).

[0416] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps described in the above “Exemplary Method” section of this specification according to various exemplary embodiments of the present disclosure.

[0417] In some embodiments, when the electronic device 1100 is a video encoding device, the processing unit 1110 may execute the following steps of the method embodiment: obtaining time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of a first video sequence by a video encoding end; performing time domain sampling on the first video sequence according to the time domain sampling information; encoding the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is configured to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0418] In other embodiments, when the electronic device 1100 is a video decoding device, the processing unit 1110 may perform the following steps of the method embodiment: decoding a bitstream from a video encoder to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result obtained by encoding the time domain sampling result and the time domain sampling information of the first video sequence by the video encoder, and the time domain sampling information is at least used to determine a time domain sampling multiple used by the video encoder to perform time domain sampling on the first video sequence; determining, based on the decoded time domain sampling information, the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; and generating, based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, a video frame to be inserted between the first video frame and the second video frame.

[0419] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 11201 and / or a cache memory unit 11202 , and may further include a read-only memory unit (ROM) 11203 .

[0420] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0421] The bus 1130 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0422] Electronic device 1100 may also communicate with one or more external devices 1140 (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication may occur via input / output (I / O) interface 1150. Furthermore, electronic device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 1100, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0423] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0424] In particular, according to an embodiment of the present disclosure, the process described in the above reference flowchart can be implemented as a computer program product, which includes: a computer program, which, when executed by a processor, implements any one of the above-mentioned video encoding methods for machine vision tasks; or any one of the above-mentioned video decoding methods for machine vision tasks.

[0425] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is further provided, which may be a readable signal medium or a readable storage medium. FIG12 shows a schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure. As shown in FIG12 , a program product capable of implementing the above-mentioned method of the present disclosure is stored on the computer-readable storage medium 1200. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0426] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0427] In the present disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0428] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0429] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0430] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0431] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0432] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0433] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A video encoding method for machine vision tasks, comprising: Acquire time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of the first video sequence by the video encoding end; Performing time-domain sampling on the first video sequence according to the time-domain sampling information; The time domain sampling result and the time domain sampling information of the first video sequence are encoded to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is used to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

2. A video decoding method for machine vision tasks, comprising: Decoding a bitstream from a video encoder to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result after the video encoder encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of the first video sequence by the video encoder; Determine, according to the decoded time domain sampling information, the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; The video frame to be inserted between the first video frame and the second video frame is generated according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

3. The video decoding method for machine vision tasks according to claim 2, wherein: The time domain sampling information includes: time domain sampling multiple index information, and the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; The method of determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information includes: Parsing the decoded time domain sampling information to obtain a time domain sampling multiple for time domain sampling of the first video sequence by the video encoding end; The number of video frames to be inserted between the first video frame and the second video frame in the second video sequence is determined according to a time domain sampling multiple of time domain sampling performed by the video encoding end on the first video sequence.

4. The video decoding method for machine vision tasks according to claim 3, wherein: The time domain sampling information further includes: time domain sampling position information, and the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence.

5. The video decoding method for machine vision tasks according to claim 4, wherein: The first video sequence includes: a plurality of sub-video sequences; the plurality of sub-video sequences are sampled in the time domain using different time domain sampling multiples; The time domain sampling multiple index information is used to determine the time domain sampling multiples for time domain sampling of each sub-video sequence in the first video sequence; the time domain sampling position information is used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

6. The video decoding method for machine vision tasks according to any one of claims 3 to 5, wherein: The time domain sampling information further includes: a time domain sampling flag, and the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

7. The video decoding method for machine vision tasks according to claim 2, wherein: Generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, comprising: If the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame; If there are multiple video frames to be inserted between the first video frame and the second video frame, the following steps are repeatedly performed until all video frames to be inserted between the first video frame and the second video frame are generated: Generate a first intermediate video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; Generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame according to the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; A third intermediate video frame to be inserted between the first intermediate video frame and the second video frame is generated according to the first intermediate video frame and the second intermediate video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

8. The video decoding method for machine vision tasks according to claim 7, wherein: The method further comprises: Input any two video frames into a pre-built video frame infill network model, and output a video frame located in the middle between the two video frames, wherein the video frame infill network model is used to generate a video frame located in the middle between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

9. The video decoding method for machine vision tasks according to claim 2, wherein: The bit stream is an encoding result after the video encoding end encodes the first video sequence and the time domain sampling information in a hybrid encoding method, and the hybrid encoding method is an encoding method combining inter-frame encoding and intra-frame encoding.

10. The video decoding method for machine vision tasks according to claim 9, wherein: The inter-frame coding is traditional inter-frame coding, and the intra-frame coding is end-to-end intra-frame coding.

11. The video decoding method for machine vision tasks according to claim 9 or 10, wherein: Before generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: Obtaining a first video frame obtained by decoding the bitstream; Filter the first video frame; The first video frame in the second video sequence is replaced by the filtered video frame.

12. The video decoding method for machine vision tasks according to claim 2, wherein: After generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: The video frame to be inserted between the first video frame and the second video frame is inserted into a corresponding position between the first video frame and the second video frame in the second video sequence.

13. A video encoding device, comprising: A time domain sampling information acquisition module, configured to acquire time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of a first video sequence by a video encoding end; A time domain sampling module, configured to perform time domain sampling on the first video sequence according to the time domain sampling information; The encoding module is configured to encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding end, wherein the video decoding end is used to decode the received code stream to obtain the second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

14. A video decoding device, comprising: a decoding module, configured to decode a bitstream from a video encoding end to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result after the video encoding end encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine a time domain sampling multiple for the video encoding end to perform time domain sampling on the first video sequence; a video interpolation frame information determination module, configured to determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; a video frame interpolation module configured to generate the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame; The video frame to be inserted between two video frames.

15. A video processing system for machine vision tasks, comprising: Video encoding device and video decoding device; The video encoding device is used to obtain time domain sampling information, perform time domain sampling on the first video sequence according to the time domain sampling information, and encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding device; The video decoding device is used to receive a code stream from the video encoding device, obtain a second video sequence and time domain sampling information, and determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, and then generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

16. A video decoding method for machine vision tasks, comprising: Decode the video bitstream to be decoded to obtain the decoded video sequence and time domain sampling information; According to the time domain sampling information, a time domain recovery process is performed on the decoded video sequence.

17. The video decoding method for machine vision tasks according to claim 16, wherein: The time domain sampling information includes: a time domain recovery flag and time domain resampling ratio information, wherein the time domain recovery flag is used to indicate whether the time domain recovery process is enabled; and the time domain resampling ratio information is used to determine the time domain resampling ratio during the time domain recovery process.

18. The video decoding method for machine vision tasks according to claim 17, wherein: The time domain resampling ratio information is a time domain resampling ratio or a time domain resampling ratio index.

19. The video decoding method for machine vision tasks according to claim 17, wherein: According to the time domain sampling information, performing time domain recovery processing on the decoded video sequence includes: Determining whether to perform time domain recovery processing on the decoded video sequence according to the time domain recovery flag bit included in the time domain sampling information; When it is determined to perform time domain recovery processing on the to-be-decoded video code stream, determining a time domain resampling ratio according to the time domain resampling ratio information included in the time domain sampling information; According to the time domain resampling ratio, a time domain restoration process is performed on the decoded video sequence.

20. The video decoding method for machine vision tasks according to claim 19, wherein: According to the time domain resampling ratio, performing time domain recovery processing on the decoded video sequence includes: Determining the number of video frames to be inserted between a first video frame and a second video frame according to the time domain resampling ratio, wherein the first video frame and the second video frame are any two consecutive video frames in the decoded video sequence; The video frame to be inserted between the first video frame and the second video frame is generated according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame.

21. The video decoding method for machine vision tasks according to claim 20, wherein: Generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame, comprises: If the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame; If there are multiple video frames to be inserted between the first video frame and the second video frame, the following steps are repeatedly performed until all video frames to be inserted between the first video frame and the second video frame are generated: Generate a first intermediate video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; Generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame according to the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; A third intermediate video frame to be inserted between the first intermediate video frame and the second video frame is generated according to the first intermediate video frame and the second intermediate video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

22. The video decoding method for machine vision tasks according to claim 21, wherein: When there are multiple second intermediate video frames and multiple third intermediate video frames to be inserted between the first video frame and the second video frame, the execution order of the steps of generating the second intermediate video frame and the third intermediate video frame includes any one of the following: Firstly, the step of generating the second intermediate video frame is performed until all the second intermediate video frames are generated, and then the step of generating the third intermediate video frame is performed until all the third intermediate video frames are generated; Respectively executing the step of generating a second intermediate video frame and the step of generating a third intermediate video frame; The step of generating the third intermediate video frame is first performed until all the third intermediate video frames are generated, and then the step of generating the second intermediate video frame is performed until all the second intermediate video frames are generated.

23. The video decoding method for machine vision tasks according to claim 21, wherein: The number of video frames to be inserted between the first video frame and the second video frame is n, where n is a natural number; the number of video frames to be inserted between the first video frame and the first intermediate video frame is floor(n / 2); the number of video frames to be inserted between the first intermediate video frame and the second video frame is floor(n / 2); wherein floor is a floor function.

24. The video decoding method for machine vision tasks according to claim 21, wherein: The method further comprises: Based on a pre-built video frame interpolation network model, an interpolation frame between two video frames is generated, wherein the video frame interpolation network model is used to generate an interpolation frame between the two video frames by learning optical flow and / or mask residual information of the two video frames.

25. The video decoding method for machine vision tasks according to claim 24, wherein: The video frame interpolation network model is an intermediate flow estimation network IFNet model.

26. The video decoding method for machine vision tasks according to claim 21, wherein: Based on the pre-built video frame interpolation network model, an interpolation frame between two video frames is generated, including: Converting a first video frame in a YUV format into a third video frame in an RGB format; Converting the second video frame in a YUV format into a fourth video frame in an RGB format; Based on a pre-built video frame interpolation network model, generating an interpolation frame in RGB format according to the third video frame and the fourth video frame; Converting the interpolated frame in the RGB format into an interpolated frame in the YUV format; The interpolated frame in the YUV format is inserted between the first video frame and the second video frame.

27. The video decoding method for machine vision tasks according to claim 26, wherein: Convert the YUV video frame to RGB video frame by following the steps below: The YUV format video frame to be converted is resized to obtain a YUV format image that meets a preset size; Extracting a Y component, a U component, and a V component of each pixel from the YUV format image; Perform a bitwise right shift operation on the Y component, U component, and V component of each pixel; Calculate the R component, G component and B component of each pixel according to the Y component, U component and V component after the bitwise right shift operation, wherein if there are remaining pixels in the resized image, the R component, G component and B component of the last column of pixels are used as the R component, G component and B component of the remaining column pixels; the R component, G component and B component of the last row of pixels are used as the R component, G component and B component of the remaining row pixels; The calculated R component, G component, and B component of each pixel are normalized.

28. The video decoding method for machine vision tasks according to claim 27, wherein: The R, G, and B components of each pixel after normalization are as follows: R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / 255; G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / 255; B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / 255; Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

29. The video decoding method for machine vision tasks according to claim 26, wherein: Convert the YUV video frame to RGB video frame by following the steps below: The YUV format video frame to be converted is resized to obtain a YUV format image that meets a preset size; Extracting a Y component, a U component, and a V component of each pixel from the YUV format image; According to the Y component, U component and V component of each pixel, the R component, G component and B components, wherein if there are remaining pixels in the resized image, the R components, G components, and B components of the last column of pixels are used as the R components, G components, and B components of the remaining column pixels; the R components, G components, and B components of the last row of pixels are used as the R components, G components, and B components of the remaining row pixels; The calculated R component, G component, and B component of each pixel are normalized.

30. The video decoding method for machine vision tasks according to claim 29, wherein: The R, G, and B components of each pixel after normalization are as follows: R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / scale; G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / scale; B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / scale; Where scale' = 2^(BitDepth); Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

31. The video decoding method for machine vision tasks according to claim 26, wherein: Convert the RGB video frame to the YUV video frame by following the steps below: Extract the R component, G component and B component of each pixel; Scaling the R component, G component, and B component of each pixel so that the values ​​of the R component, G component, and B component after the scaling process are limited to a preset range; Calculate the Y component, U component and V component of each pixel according to the R component, G component and B component after scaling processing; Perform a bitwise left shift operation on the calculated Y component, U component, and V component of each pixel.

32. The video decoding method for machine vision tasks according to claim 31, wherein: ,2j])*scale); pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j +1]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1, 2j]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587* p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale); pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2, i,j])*scale); pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2, i,j])*scale); Where scale = 2^(BitDepth-8); Where BitDepth is the bit width of all brightness and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component in pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB component of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB component of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB component of pixel block (i,j) in channel k=2.

33. The video decoding method for machine vision tasks according to claim 26, wherein: Convert the RGB video frame to the YUV video frame by following the steps below: Extract the R component, G component and B component of each pixel; Calculate the Y component, U component and V component of each pixel according to the R component, G component and B component of each pixel; Perform a bitwise left shift operation on the calculated Y component, U component, and V component of each pixel.

34. The video decoding method for machine vision tasks according to claim 33, wherein: ,2j])*scale); pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j +1]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1, 2j]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587* p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale); pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2, i,j])*scale); pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2, i,j])*scale); Where scale' = 2^(BitDepth); Where BitDepth is the bit width of all brightness and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component in pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB component of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB component of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB component of pixel block (i,j) in channel k=2.

35. The video decoding method for machine vision tasks according to any one of claims 16 to 34, wherein the time domain sampling information is sequence-level or frame-level information.

36. An electronic device comprising: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the video encoding method for machine vision tasks described in claim 1, or the video decoding method for machine vision tasks described in any one of claims 2 to 12, or the video decoding method for machine vision tasks described in any one of claims 16 to 35 by executing the executable instructions.

37. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the video encoding method for machine vision tasks described in claim 1, or the video decoding method for machine vision tasks described in any one of claims 2 to 12, or the video decoding method for machine vision tasks described in any one of claims 16 to 35.

38. A computer program product, comprising a computer program, which, when executed by a processor, implements the video encoding method for machine vision tasks described in claim 1, or the video decoding method for machine vision tasks described in any one of claims 2 to 12, or the video decoding method for machine vision tasks described in any one of claims 16 to 35.

Images