Video coding method and decoding method for machine vision task and related equipment

Abstract

The invention provides a machine vision task-oriented video coding method, a machine vision task-oriented video decoding method and related equipment, and relates to the technical field of machine vision. The decoding method comprises the following steps: decoding a video code stream to be decoded to obtain a decoded video sequence and time domain sampling information; and performing time domain recovery processing on the decoded video sequence according to the time domain sampling information. According to the invention, the transmission efficiency of the video data can be improved under the condition of ensuring the visual task.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application numbered 202311014435.7 filed on August 11, 2023, entitled “Video encoding method, decoding method and related devices for machine vision tasks”, the entire contents of which are incorporated herein by reference Background Art

[0004] With the deep integration of informatization and industrialization, the information intelligence of machine equipment is rapidly improving. Intelligent and efficient processing of machine vision data has always been a concern. With the widespread application of machine learning in the fields of Internet of Vehicles, video surveillance, smart cities, etc., massive amounts of video data need to be transmitted. The increase in the amount of video transmission data directly leads to the previous video encoding scheme for human vision, which is difficult to meet the requirements of machine vision tasks in terms of delay and scale. Therefore, there is an urgent need to provide a video encoding solution for machine vision tasks.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a video encoding method, a decoding method and related devices for machine vision tasks, which at least to some extent overcome the technical problem that the video encoding and decoding solutions provided in the related art are difficult to apply to machine vision tasks.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a video encoding method for machine vision tasks is provided, including: obtaining time domain sampling information, wherein the time domain sampling information is at least used to determine the time domain sampling multiple of a first video sequence performed by a video encoding end; performing time domain sampling on the first video sequence according to the time domain sampling information; encoding the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is used to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, and generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0009] According to another aspect of the present disclosure, a video decoding method for machine vision tasks is also provided, including: decoding a bitstream from a video encoding end to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result after the video encoding end encodes the time domain sampling result and time domain sampling information of a first video sequence, and the time domain sampling information is at least used to determine the time domain sampling multiple of the video encoding end for time domain sampling of the first video sequence; determining the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame.

[0010] In some embodiments, the time domain sampling information includes: time domain sampling multiple index information, wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; wherein, based on the decoded time domain sampling information, determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence, including: parsing the decoded time domain sampling information to obtain the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence; determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence.

[0011] In some embodiments, the time domain sampling information further includes: time domain sampling position information, and the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence.

[0012] In some embodiments, the first video sequence includes: multiple sub-video sequences; the multiple sub-video sequences use different time domain sampling multiples for time domain sampling; wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple for time domain sampling of each sub-video sequence in the first video sequence; the time domain sampling position information is used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

[0013] In some embodiments, the time domain sampling information further includes: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0014] In some embodiments, based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, a video frame to be inserted between the first video frame and the second video frame is generated, including: if the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is more than one, repeating the following steps until all the video frames to be inserted between the first video frame and the second video frame are generated: generating the video frames to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame A first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located in the middle position between the first video frame and the second video frame; based on the first video frame and the first intermediate video frame, a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame is generated, wherein the second intermediate video frame is a video frame located in the middle position between the first video frame and the first intermediate video frame; based on the first intermediate video frame and the second video frame, a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame is generated, wherein the third intermediate video frame is a video frame located in the middle position between the first intermediate video frame and the second video frame.

[0015] In some embodiments, the method further includes: inputting any two video frames into a pre-built video frame infill network model, and outputting a video frame located in the middle between the two video frames, wherein the video frame infill network model is used to generate a video frame located in the middle between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0016] In some embodiments, the bitstream is a coding result after the video encoding end encodes the first video sequence and the time domain sampling information in a hybrid coding method, and the hybrid coding method is a coding method combining inter-frame coding and intra-frame coding.

[0017] In some embodiments, the hybrid coding method is a coding method that combines traditional inter-frame coding with end-to-end intra-frame coding; wherein, before generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: obtaining a first video frame obtained by decoding the code stream; filtering the first video frame; and replacing the first video frame in the second video sequence with the filtered video frame.

[0018] In some embodiments, after generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: inserting the video frame to be inserted between the first video frame and the second video frame into a corresponding position between the first video frame and the second video frame in the second video sequence.

[0019] According to another aspect of the present disclosure, a video encoding device is also provided, including: a time domain sampling information acquisition module, used to acquire time domain sampling information, wherein the time domain sampling information is at least used to determine the time domain sampling multiple of the video encoding end for time domain sampling of a first video sequence; a time domain sampling module, used to perform time domain sampling on the first video sequence according to the time domain sampling information; an encoding module, used to encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is used to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0020] According to another aspect of the present disclosure, a video decoding device is also provided, including: a decoding module, used to decode a code stream from a video encoding end to obtain a second video sequence and time domain sampling information, wherein the code stream is an encoding result after the video encoding end encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine the time domain sampling multiple of the time domain sampling of the first video sequence by the video encoding end; a video interpolation frame information determination module, used to determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; a video interpolation frame module, used to generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame.

[0021] According to another aspect of the present disclosure, a video processing system for machine vision tasks is also provided, including: a video encoding device and a video decoding device; wherein the video encoding device is used to obtain time domain sampling information, perform time domain sampling on a first video sequence according to the time domain sampling information, and encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding device; the video decoding device is used to receive the code stream from the video encoding device, obtain a second video sequence and time domain sampling information, and determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, and then generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0022] According to another aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned video encoding method for machine vision tasks, or any one of the above-mentioned video encoding methods for machine vision tasks, by executing the executable instructions.

[0023] According to another aspect of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned video encoding method for machine vision tasks, or any one of the above-mentioned video encoding methods for machine vision tasks.

[0024] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the above-mentioned video encoding method for machine vision tasks, or any one of the above-mentioned video encoding methods for machine vision tasks.

[0025] In the video encoding method, decoding method and related devices for machine vision tasks provided in the embodiments of the present disclosure, a video encoding end performs time-domain sampling on a first video sequence according to the acquired time-domain sampling information, and then transmits a code stream after encoding the time-domain sampling result and the time-domain sampling information to a video decoding end, so that the video decoding end decodes the received code stream to obtain a decoded second video sequence and the time-domain sampling information, determines the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time-domain sampling information, and generates a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0026] The solution provided by the embodiment of the present disclosure can improve the transmission efficiency of video data by sampling the video data in the time domain while ensuring the visual task.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0029] Figure 1 A schematic diagram of a video processing system for machine vision tasks in an embodiment of the present disclosure is shown;

[0030] Figure 2 A flow chart of a video encoding method for machine vision tasks in an embodiment of the present disclosure is shown;

[0031] Figure 3 A flow chart of a video decoding method for machine vision tasks in an embodiment of the present disclosure is shown;

[0032] Figure 4 A schematic diagram of time downsampling for machine vision tasks in an embodiment of the present disclosure is shown;

[0033] Figure 5 A schematic diagram of temporal upsampling for machine vision tasks in an embodiment of the present disclosure is shown;

[0034] Figure 6 A schematic diagram showing the structure of a video frame interpolation network model in an embodiment of the present disclosure is shown;

[0035] Figure 7 A flow chart of another video decoding method for machine vision tasks in an embodiment of the present disclosure is shown;

[0036] Figure 8 A flow chart of a method for generating an interpolation frame in an embodiment of the present disclosure is shown;

[0037] Fig. 9 A schematic diagram of a video encoding device in an embodiment of the present disclosure is shown;

[0038] Fig.10 A schematic diagram of a video decoding device in an embodiment of the present disclosure is shown;

[0039] Fig.11 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0040] Fig.12 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0042] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0043] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0044] Figure 1 FIG. 2 shows a schematic diagram of a video processing system for machine vision tasks in an embodiment of the present disclosure. Figure 1 As shown, the system includes: a video encoding device 10 and a video decoding device 20.

[0045] The video encoding device 10 is used to obtain time domain sampling information, perform time domain sampling on the first video sequence according to the time domain sampling information, and encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding device 20;

[0046] The video decoding device 20 is used to receive the code stream from the video encoding device 10, obtain the second video sequence and time domain sampling information, and determine the video frame information (for example, the number of video frames) to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, and then generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0047] It should be noted that the video encoding device 10 may be any device for encoding image / video data; the video decoding device 20 may be any device for decoding image / video data. In some embodiments, the video encoding device 10 and the video decoding device 20 may be deployed on one device; in other embodiments, the video encoding device 10 and the video decoding device 20 may be deployed on different devices.

[0048] In one embodiment, the above-mentioned time domain sampling information may include: time domain sampling multiple index information, wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; then the above-mentioned video decoding device 20 is also used to: parse the decoded time domain sampling information to obtain the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence; determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence.

[0049] Furthermore, in some embodiments, the above-mentioned time domain sampling information may also include: time domain sampling position information, and the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence.

[0050] In one embodiment, when the first video sequence includes multiple sub-video sequences, and the multiple sub-video sequences use different time domain sampling multiples for time domain sampling; the above-mentioned time domain sampling multiple index information is also used to determine the time domain sampling multiples for time domain sampling of each sub-video sequence in the first video sequence; the above-mentioned time domain sampling position information is also used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

[0051] Furthermore, in some embodiments, the time domain sampling information may further include: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0052] In some embodiments, the video decoding device 20 is further used for: if the number of video frames to be inserted between the first video frame and the second video frame is one, then directly generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is multiple, then repeating the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: generating a first intermediate video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; generating a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame based on the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; generating a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame based on the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

[0053] In some embodiments, the above-mentioned video decoding device 20 is also used to: input any two video frames into a pre-constructed video frame infill network model, and output a video frame located in the middle position between the two video frames, wherein the video frame infill network model is used to generate a video frame located in the middle position between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0054] In one embodiment, the bitstream in the embodiment of the present disclosure may be an encoding result after the video encoding end encodes the first video sequence and the time domain sampling information in a hybrid encoding method, where the hybrid encoding method is an encoding method that combines inter-frame encoding and intra-frame encoding.

[0055] In a specific embodiment, the hybrid coding method in the disclosed embodiment can be a coding method that combines traditional inter-frame coding and end-to-end intra-frame coding; in this embodiment, the above-mentioned video decoding device 20 is also used to: obtain the first video frame obtained by decoding the code stream; filter the first video frame; and use the filtered video frame to replace the first video frame in the second video sequence.

[0056] In one embodiment, the video decoding device 20 is further configured to filter the first video frame using the following filter:

[0057]

[0058] Among them, filter represents a filter.

[0059] In a specific implementation, a 3-channel matrix of a first decoded frame is obtained, and the first decoded frame is filtered using the above filter.

[0060] In some embodiments, the video decoding device 20 is further configured to: insert the video frame to be inserted between the first video frame and the second video frame into a corresponding position between the first video frame and the second video frame in the second video sequence.

[0061] Those skilled in the art will know that Figure 1 The number of video encoding devices and video decoding devices in the embodiment is only for illustration, and any number of video encoding devices and video decoding devices may be provided according to actual needs. The embodiments of the present disclosure do not limit this.

[0062] Under the above system architecture, a video encoding method and a decoding method for machine vision tasks are provided in the embodiments of the present disclosure. In principle, the video encoding method and the decoding method for machine vision tasks provided in the embodiments of the present disclosure can be executed by any electronic device with computing and processing capabilities. In some embodiments, the video encoding method for machine vision tasks provided in the embodiments of the present disclosure can be executed by the video encoding device of the above system architecture; the video decoding method for machine vision tasks provided in the embodiments of the present disclosure can be executed by the video decoding device of the above system architecture; in other embodiments, the video encoding method and the decoding method for machine vision tasks provided in the embodiments of the present disclosure can be implemented by the video encoding device and the video decoding device in the above system architecture in an interactive manner.

[0063] Figure 2 A flow chart of a video encoding method for machine vision tasks in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the video encoding method may include the following steps:

[0064] S202: Acquire time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling performed by a video encoding end on a first video sequence.

[0065] It should be noted that the above-mentioned time domain sampling information refers to the relevant configuration information for the video encoding end to perform time domain sampling on the original video sequence (that is, the first video sequence), which can be used for but not limited to determining the time domain sampling multiple for the video encoding end to perform time domain sampling on the first video sequence.

[0066] In one embodiment, the time domain sampling information may include but is not limited to the following information: a time domain sampling flag, time domain sampling position information and time domain sampling multiple index information, wherein the time domain sampling multiple index information (which may be a time domain sampling multiple or information having a certain mapping relationship with the time domain sampling multiple) is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence; and the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0067] It should be noted that, in a specific implementation, the time domain sampling flag, time domain sampling position information and time domain sampling multiple index information in the above-mentioned time domain sampling information may be included in full or in part, and may also include other information about time domain sampling in addition to the above three types of information, as long as the information can be used by the decoding end to determine how to insert frames, and the present disclosure does not limit this.

[0068] S204: Perform time domain sampling on the first video sequence according to the time domain sampling information.

[0069] In the disclosed embodiment, the video encoding end performs time domain sampling on the original video sequence (i.e., the first video sequence) according to the time domain sampling information to obtain a video sequence after time domain sampling. Since the number of frames of the video sequence after time domain sampling is smaller than the number of frames of the original video sequence (i.e., the first video sequence), time domain sampling is performed on the original video sequence and only the video sequence after time domain sampling is transmitted, thereby improving the transmission efficiency of video data.

[0070] Assume that the original video sequence is: video frame 0, video frame 1, video frame 2, video frame 3, video frame 4, video frame 5, video frame 6, video frame 7 and video frame 8; when the time domain sampling multiple is 2, one video frame is extracted from every 2 video frames in the original video sequence, and the video sequence after time domain sampling includes: video frame 0, video frame 2, video frame 4, video frame 6 and video frame 8; when the time domain sampling multiple is 4, one video frame is extracted from every 4 video frames in the original video sequence, and the video sequence after time domain sampling includes: video frame 0, video frame 4 and video frame 8.

[0071] S206, encoding the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding end.

[0072] It should be noted that in the video encoding and decoding scheme for machine vision tasks, for the video sequence after time domain sampling, the video decoding end needs to perform frame insertion (also called frame supplementation) processing on the decoded video sequence after decoding the video sequence. Therefore, when transmitting the code stream, the video encoding end needs to encode the video sequence after time domain sampling (that is, the time domain sampling result of the first video sequence) and the time domain sampling information, so that after the video decoding end decodes the received code stream to obtain the second video sequence and the time domain sampling information, the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence is determined according to the decoded time domain sampling information, and then the video frame to be inserted between the first video frame and the second video frame is generated according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame. The first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0073] In some embodiments, the above-mentioned time domain sampling information may include: time domain sampling multiple index information, so that the video decoding end determines the video frame information (for example, the number of video frames) to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling multiple index information. The time domain sampling multiple index information may be information having a certain mapping relationship with the time domain sampling multiple. The video decoding end may determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence, the video frame sequence number and other information according to the decoded time domain sampling multiple index information.

[0074] In some embodiments, if the number of video frames to be inserted between the first video frame and the second video frame is one, the video decoding end may directly generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame.

[0075] In other embodiments, if there are multiple video frames to be inserted between the first video frame and the second video frame, the video decoding end may repeatedly perform the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: based on the first video frame and the second video frame, generate a first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; based on the first video frame and the first intermediate video frame, generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; based on the first intermediate video frame and the second video frame, generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

[0076] In one embodiment, any two video frames can be input into a pre-built video frame infill network model, and a video frame located in the middle between the two video frames can be output, wherein the video frame infill network model is used to generate a video frame located in the middle between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0077] For example, the first video frame and the second video frame are input into the video frame incomplete network model, and the first intermediate video frame (located in the middle position between the first video frame and the second video frame) is output; the first video frame and the first intermediate video frame are input into the video frame incomplete network model, and the second intermediate video frame (located in the middle position between the first video frame and the first intermediate video frame) is output; the first intermediate video frame and the second video frame are input into the video frame incomplete network model, and the third intermediate video frame (located in the middle position between the first intermediate video frame and the second video frame) is output.

[0078] Furthermore, in some embodiments, the above-mentioned time domain sampling information may also include: time domain sampling position information, and the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence.

[0079] In one embodiment, when the first video sequence includes multiple sub-video sequences, and the multiple sub-video sequences use different time domain sampling multiples for time domain sampling; the above-mentioned time domain sampling multiple index information is also used to determine the time domain sampling multiples for time domain sampling of each sub-video sequence in the first video sequence; the above-mentioned time domain sampling position information is also used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

[0080] Furthermore, in some embodiments, the time domain sampling information may further include: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0081] In some embodiments, before generating the video frame to be inserted, the video decoding end is also used to: obtain the first video frame obtained by decoding the bit stream; filter the first video frame; and use the filtered video frame to replace the first video frame in the second video sequence.

[0082] In some embodiments, the video decoding end may filter the first video frame using, but not limited to, the filter shown in the above formula (1).

[0083] In some embodiments, the video decoding end is further used to insert the video frame to be inserted between the first video frame and the second video frame into a corresponding position between the first video frame and the second video frame in the second video sequence.

[0084] Figure 3 A flow chart of a video decoding method for machine vision tasks in an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the video decoding method may include the following steps:

[0085] S302, decoding a bitstream from a video encoder to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result after the video encoder encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of the first video sequence by the video encoder;

[0086] S304, determining the number of video frames to be inserted between a first video frame and a second video frame in a second video sequence according to the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0087] It should be noted that, in the embodiment of the present disclosure, both the first video sequence and the second video sequence are sequences containing one or more video frames. The first video sequence refers to the original video sequence before encoding, and the second video sequence is the decoded video sequence.

[0088] In some embodiments, the above S304 can be implemented by the following steps: if the number of video frames to be inserted between the first video frame and the second video frame is one, then directly generate the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is multiple, then repeat the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: based on the first video frame and the second video frame, generate a first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; based on the first video frame and the first intermediate video frame, generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; based on the first intermediate video frame and the second video frame, generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

[0089] In one embodiment, when determining a video frame located in the middle between two video frames, any two video frames can be input into a pre-built video frame infill network model, and a video frame located in the middle between the two video frames is output, wherein the video frame infill network model is used to generate a video frame located in the middle between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0090] S306: Generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0091] In some embodiments, the above-mentioned time domain sampling information includes: time domain sampling multiple index information, wherein the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; the above-mentioned S304 can be specifically implemented through the following steps: parsing the decoded time domain sampling information to obtain the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence; determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the time domain sampling multiple used by the video encoding end to perform time domain sampling on the first video sequence.

[0092] Furthermore, in some embodiments, the above-mentioned time domain sampling information may also include: time domain sampling position information, and the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence.

[0093] In one embodiment, when the first video sequence includes multiple sub-video sequences, and the multiple sub-video sequences use different time domain sampling multiples for time domain sampling; the above time domain sampling multiple index information is also used to determine the time domain sampling multiples for time domain sampling of each sub-video sequence in the first video sequence; the above time domain sampling position information is also used to determine the position for time domain sampling of each sub-video sequence in the first video sequence. Optionally, the multiple sub-video sequences of the first video sequence using different time domain sampling multiples for time domain sampling can be continuous or discontinuous, and can be part of the video sequence in the first video sequence or all of the video sequence.

[0094] Furthermore, in some embodiments, the time domain sampling information may further include: a time domain sampling flag, wherein the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

[0095] Based on the above embodiments, in one embodiment, after decoding the bitstream from the video encoding end to obtain the second video sequence and time domain sampling information, the video decoding method for machine vision tasks provided in the embodiments of the present disclosure may also include the following steps: determining whether the video encoding end has performed time domain sampling on the first video sequence based on the decoded time domain sampling information; if the video encoding end has not performed time domain sampling on the first video sequence, the video decoding end does not need to perform subsequent frame insertion operations; conversely, if the video encoding end has performed time domain sampling on the first video sequence, determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence based on the decoded time domain sampling information, and then generating the video frame to be inserted between the first video frame and the second video frame based on the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0096] In some embodiments, when the bitstream is an encoding result after the video encoding end encodes the first video sequence and time domain sampling information in a hybrid encoding manner, before generating the video frame to be inserted between the first video frame and the second video frame (also known as time domain upsampling) based on the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame, the video decoding method for machine vision tasks provided in the embodiments of the present disclosure may also include the following steps: obtaining the first video frame obtained by decoding the bitstream; filtering the first video frame; and replacing the first video frame in the second video sequence with the filtered video frame.

[0097] In some embodiments, the video decoding method for machine vision tasks provided in the embodiments of the present disclosure filters the first video frame using the filter shown in the above formula (1).

[0098] In some embodiments, after generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the video encoding method for machine vision tasks provided in the embodiments of the present disclosure may also include the following steps: inserting the video frame to be inserted between the first video frame and the second video frame into the corresponding position between the first video frame and the second video frame in the second video sequence.

[0099] Taking the time domain sampling multiples of 2 and 4 as an example, the following combination Figures 4 to 6 The video encoding method and video decoding method for machine vision tasks provided in the embodiments of the present disclosure are described in detail.

[0100] Figure 4Shown is a schematic diagram of temporal downsampling for machine vision; Figure 5 The figure shows a schematic diagram of temporal upsampling for machine vision. Figure 5 Shown is a video frame interpolation network model (neural network model) used for video frame interpolation processing.

[0101] In a specific implementation, the video encoding end encodes the time domain sampling information such as the time domain sampling flag, the time domain sampling multiple index information, the time domain sampling position, and the video image information after the time domain sampling, and sends them to the video decoding end; after the video decoding end decodes the received bit stream, it obtains the time domain sampling flag, the time domain sampling multiple index information, the time domain sampling position, and the video image information after the time domain sampling, and restores the time domain sampling multiple according to the time domain sampling multiple index information.

[0102] After decoding and obtaining the second video sequence, the video decoding end may perform the following processing on two consecutive frames:

[0103] When the time domain sampling multiple is 2, the discarded frames The frame I' before and after decoding nT and I' nT+2 Insertion:

[0104]

[0105] When the time domain sampling multiple is 4, the discarded frames are inserted in a cascaded manner, where the discarded frames The frame I' before and after decoding nT and I' nT+4 Insertion:

[0106]

[0107] Next, the discarded frames and The decoded frame I' iT and I′ iT+4 Separate and interpolated frames Fill frame.

[0108]

[0109] Wherein, n represents a natural number, and its value is n=1, 2, 3, ...; F represents a filter.

[0110] like Figure 6 As shown in Figure 1, the video frame interpolation network model includes: FMI-Updater (optical flow, mask and image updater) and IFBlock (frame interpolation module). The video frame interpolation network model directly predicts the intermediate frame from the input frame. For 2x temporal upsampling (temporal sampling multiple is 2), frame interpolation By input frame (I'nT ,I' nT+2 ) prediction. For 4 times time domain upsampling (time domain sampling multiple is 4), From the forward frame I' nT and backward frame I' nT+2 Prediction, frame filling By input frame (I' nT ,I' nT+4 )predict, From the forward frame I' nT+2 Backward frame I' nT+4 Prediction. The entire network has a stacked hourglass structure, consisting of three consecutive FMI-Updaters and a final image merging module. FMI-Updater is used to learn residual optical flow and mask from the input frame and update the optical flow and mask by their respective residuals. In addition to the updated optical flow and mask (F, M) i The module output consists of the input frame (I n ' T ,I' nT+k ) i-1 Updated Optical Flow and The updated frame obtained by distortion (I' nT ,I n ' T+k ) i .

[0111] In some embodiments, the time domain sampling multiple index [0, 1] corresponds to the time domain sampling multiple [2, 4] respectively, that is, when the time domain sampling multiple is 2, the time domain sampling multiple index is 0, and when the time domain sampling multiple is 4, the time domain sampling multiple index is 1.

[0112] Two specific examples are listed below to illustrate the embodiments of the present disclosure:

[0113] Example 1: Take SFU-HW Blowing Bubble (416x240 50fps 97 frames) as an example. The encoder extracts one frame every 4 frames of the original video, and sends the extracted video frame, time domain sampling flag, and time domain sampling multiple index 2 to the hybrid encoder VCMRS. Among them, the time domain sampling flag and the time domain sampling multiple index can use the NNPFC SEI syntax element.

[0114] Among them, nnpfc_purpose=0x08; nnpfc_interpolated_pics=3; nnpfc_id is the specified serial number of the interpolation model (such as 256), nnpfc_num_input_pics_minus1=1 (numInputPics=nnpfc_num_input_pics_minus1+1); the newly added SEI syntax elements temporal_resampling_flag=1, temporal_resampling_ratio_idx=2 can also be used.

[0115] temporal_resampling_flag indicates the time domain sampling flag;

[0116] temporal_resampling_ratio_idx represents the temporal domain sampling multiple index.

[0117] The decoding end parses the received bitstream to obtain the decoded video frame, time domain sampling flag and time domain sampling multiple index, and obtains the time domain sampling multiple through mapping. The hybrid encoder VCMRS uses end-to-end encoding for key frames, and sends the reconstructed key frames of end-to-end encoding and decoding to VTM encoding to replace intra-frame encoding, and then uses VTM for inter-frame encoding. For the video frame V1 reconstructed by VTM, you can choose to perform adaptation operations for machine vision tasks to obtain the reconstructed video frame V2.

[0118] The first frame in the video frame decoded by the hybrid encoder VCMRS is a key frame. End-to-end encoding is adopted. The reconstructed image based on end-to-end image coding based on deep learning and joint training of machine vision tasks is prone to checkerboard phenomenon, which affects the temporal upsampling of the video. Therefore, filtering operation is used for the first frame in the video frame decoded by the hybrid encoder.

[0119] img = cv2.imread(srcimg);

[0120] kernel=np.array([[0.0625,0.125,0.0625],[0.125,0.25,0.125],[0.0625,0.125,0.0625]]);

[0121] desimg=cv2.filter2D(img,-1,kernel);

[0122] The filtered image is overlaid on the original image, and the decoded video frames are interpolated in pairs. nT ,I nT+4 ) are interlaced in sequence (I nT+2,I nT+1 ,I nT+3 ).

[0123] Example 2: Taking TVD 01_1 (1920x1080 50fps 500 frames) as an example, downsampling and temporal upsampling are performed at a ratio of 4:1. For the three frames 498-499-500, the discard operation can be used, and the decoding is not reconstructed; or the copy and repeat operation can be used to copy 3 identical pictures of the reconstructed 497 frames to form a complete 500 frames.

[0124] Figure 7 A flow chart of another video decoding method for machine vision tasks in an embodiment of the present disclosure is shown. Figure 7 As shown, the video decoding method for machine vision tasks provided in the embodiment of the present disclosure may include the following steps:

[0125] S702, decoding the video code stream to be decoded to obtain a decoded video sequence and time domain sampling information;

[0126] S704, performing time domain recovery processing on the decoded video sequence according to the time domain sampling information.

[0127] In some embodiments, the time domain sampling information includes: a time domain recovery flag and time domain resampling ratio information, wherein the time domain recovery flag is used to indicate whether time domain recovery processing is enabled; and the time domain resampling ratio information is used to determine the time domain resampling ratio during time domain recovery processing.

[0128] In one embodiment, the time domain resampling ratio information may be directly the time domain resampling ratio; in another embodiment, the time domain resampling ratio information may be a time domain resampling ratio index, and then the time domain resampling ratio during the time domain recovery processing is determined according to the time domain resampling ratio index.

[0129] In some embodiments, time domain recovery processing is performed on the decoded video sequence according to the time domain sampling information, which can be achieved through the following steps: determining whether to perform time domain recovery processing on the decoded video sequence according to the time domain recovery flag contained in the time domain sampling information; when it is determined to perform time domain recovery processing on the video stream to be decoded, determining the time domain resampling ratio according to the time domain resampling ratio information contained in the time domain sampling information; and performing time domain recovery processing on the decoded video sequence according to the time domain resampling ratio.

[0130] In some embodiments, time domain recovery processing is performed on the decoded video sequence according to the time domain resampling ratio, which can be achieved through the following steps: determining the number of video frames to be inserted between the first video frame and the second video frame according to the time domain resampling ratio, wherein the first video frame and the second video frame are any two consecutive video frames in the decoded video sequence; generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame.

[0131] In some embodiments, generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame can be achieved by the following steps: if the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame; if the number of video frames to be inserted between the first video frame and the second video frame is more than one, repeatedly performing the following steps until all video frames to be inserted between the first video frame and the second video frame are generated: according to the first video frame and the second video frame, Generate a first intermediate video frame to be inserted between the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located in the middle between the first video frame and the second video frame; generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame based on the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located in the middle between the first video frame and the first intermediate video frame; generate a third intermediate video frame to be inserted between the first intermediate video frame and the second video frame based on the first intermediate video frame and the second video frame, wherein the third intermediate video frame is a video frame located in the middle between the first intermediate video frame and the second video frame.

[0132] In some embodiments, when there are multiple second intermediate video frames and multiple third intermediate video frames to be inserted between the first video frame and the second video frame, in a specific implementation, the execution order of the steps of generating the second intermediate video frame and the third intermediate video frame includes any one of the following:

[0133] ① Firstly, the step of generating the second intermediate video frame is performed until all the second intermediate video frames are generated, and then the step of generating the third intermediate video frame is performed until all the third intermediate video frames are generated;

[0134] ② respectively executing the step of generating a second intermediate video frame and the step of generating a third intermediate video frame;

[0135] ③ First execute the step of generating the third intermediate video frame until all the third intermediate video frames are generated, and then execute the step of generating the second intermediate video frame until all the second intermediate video frames are generated.

[0136] In some embodiments, the video decoding method for machine vision tasks provided in the embodiments of the present disclosure may also include the following steps: generating an interpolated frame between two video frames based on a pre-built video frame infill network model, wherein the video frame infill network model is used to generate an interpolated frame between two video frames by learning the optical flow and / or mask residual information of the two video frames.

[0137] In some embodiments, the video frame interpolation network model is an intermediate flow estimation network IFNet model.

[0138] Figure 8 A flow chart of a method for generating an interpolation frame in an embodiment of the present disclosure is shown as follows: Figure 8 As shown, the video decoding method for machine vision tasks provided in the embodiment of the present disclosure can also generate interpolation frames through the following steps:

[0139] S802, converting the first video frame in YUV format into a third video frame in RGB format;

[0140] S804, converting the second video frame in the YUV format into a fourth video frame in the RGB format;

[0141] S806, generating an interpolation frame in RGB format according to the third video frame and the fourth video frame based on a pre-built video frame interpolation network model;

[0142] S808, converting the interpolation frame in RGB format into an interpolation frame in YUV format;

[0143] S810: insert an interpolated frame in a YUV format between the first video frame and the second video frame.

[0144] In one embodiment, a video frame in a YUV format is converted into a video frame in an RGB format by the following steps: the YUV format video frame to be converted is resized to obtain a YUV format image that meets a preset size; the Y component, U component and V component of each pixel are extracted from the YUV format image; a bitwise right shift operation is performed on the Y component, U component and V component of each pixel; the R component, G component and B component of each pixel are calculated based on the Y component, U component and V component after the bitwise right shift operation, wherein if there are remaining pixels in the resized image, the R component, G component and B component of the last column of pixels are used as the R component, G component and B component of the remaining column of pixels; the R component, G component and B component of the last row of pixels are used as the R component, G component and B component of the remaining row of pixels; and the calculated R component, G component and B component of each pixel are normalized.

[0145] In the above embodiment, the R component, G component and B component of each pixel after normalization are as follows:

[0146] R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / 255;

[0147] G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / 255;

[0148] B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / 255;

[0149] Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

[0150] In another embodiment, a video frame in a YUV format is converted into a video frame in an RGB format by the following steps: the YUV format video frame to be converted is resized to obtain a YUV format image that satisfies a preset size (such as the image width and height are both adjusted to a multiple of 32); the Y component, U component and V component of each pixel are extracted from the YUV format image; the R component, G component and B component of each pixel are calculated based on the Y component, U component and V component of each pixel, wherein if there are remaining pixels in the image after the resize (such as the image width and height are both adjusted to a multiple of 32), the R component, G component and B component of the last column of pixels are used as the R component, G component and B component of the remaining column of pixels; the R component, G component and B component of the last row of pixels are used as the R component, G component and B component of the remaining row of pixels; and the calculated R component, G component and B component of each pixel are normalized.

[0151] In the above embodiment, the R component, G component and B component of each pixel after normalization are as follows:

[0152] R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / scale;

[0153] G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / scale;

[0154] B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / scale;

[0155] Where, scale'=2^(BitDepth);

[0156] Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

[0157] In one embodiment, a video frame in RGB format is converted into a video frame in YUV format by the following steps: extracting the R component, G component and B component of each pixel; scaling the R component, G component and B component of each pixel so that the values ​​of the scaled R component, G component and B component are limited to a preset range; calculating the Y component, U component and V component of each pixel based on the scaled R component, G component and B component; and performing a bitwise left shift operation on the calculated Y component, U component and V component of each pixel.

[0158] In the above embodiment, the Y component, U component and V component of each 2x2 pixel block after the bitwise left shift operation are as follows:

[0159] pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale);

[0160] pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j+1]+0.114*p[2,i+1,j])*scale);

[0161] pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1,2j]+0.114*p[2,i+1,j])*scale);

[0162] pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587*p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale);

[0163] pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2,i,j])*scale);

[0164] pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2,i,j])*scale);

[0165] Where scale = 2^(BitDepth-8);

[0166] Where BitDepth is the bit width of all brightness and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component in pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB component of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB component of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB component of pixel block (i,j) in channel k=2.

[0167] In another embodiment, a video frame in RGB format is converted into a video frame in YUV format by the following steps: extracting the R component, G component and B component of each pixel; calculating the Y component, U component and V component of each pixel based on the R component, G component and B component of each pixel; and performing a bitwise left shift operation on the calculated Y component, U component and V component of each pixel.

[0168] In the above embodiment, the Y component, U component and V component of each pixel after the bitwise left shift operation are as follows:

[0169] pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale);

[0170] pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j+1]+0.114*p[2,i+1,j])*scale);

[0171] pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1,2j]+0.114*p[2,i+1,j])*scale);

[0172] pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587*p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale);

[0173] pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2,i,j])*scale);

[0174] pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2,i,j])*scale);

[0175] Where scale' = 2^(BitDepth);

[0176] Where BitDepth is the bit width of all brightness and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component in pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB component of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB component of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB component of pixel block (i,j) in channel k=2.

[0177] Table 1

[0178]

[0179]

[0180] In one embodiment, the time domain sampling information in the embodiment of the present disclosure includes the content as shown in Table 1, which at least includes a time domain restoration flag (temporal_restoration_flag) and a time domain resampling ratio index (temporal_resampling_ratio_idx), wherein the time domain restoration flag is used to indicate whether the time domain restoration (time domain resampling) process is enabled, 1 indicates enabling the time domain restoration process, and 0 indicates disabling the time domain restoration process; the time domain resampling ratio index is used to determine the time domain resampling ratio. The relationship between the time domain resampling ratio and the time domain resampling ratio index is as follows:

[0181] TemporalResamplingRatio=2 temporal_resampling_ratio_idx ;

[0182] Among them, TemporalResamplingRatio represents the time domain resampling ratio;

[0183] temporal_resampling_ratio_idx represents the temporal resampling ratio index.

[0184] When temporal_restoration_flag = 1, the following temporal restoration process is performed, and the input and output of the process are as follows:

[0185] Input: pictListIn, an input picture list containing pairs of adjacent frames, each pair of adjacent frames is in YUV format (such as YUV420P10LE format);

[0186] Output: Output picture list pictListOut after inserting the recovered frame.

[0187] The implementation process is:

[0188] 1) Format conversion: convert a pair of adjacent frames in the input picture list pictListIn from a YUV format (such as YUV420P10LE) to an RGB format (such as RGB24 format).

[0189] 2) Generate temporal restored frames (interpolated frames): Use the adjacent frames after the conversion format and the number of interpolated frames between adjacent frames (that is, the number of video frames to be inserted between two video frames, which is TemporalResamplingRatio-1), and use the following temporal interpolation process to generate restored frames, and finally obtain (TemporalResamplingRatio-1) RGB format (such as RGB24 format) restored frames.

[0190] 3) Format conversion: converting the recovered frame from RGB format (such as RGB24 format) back to YUV format (such as YUV420P10LE format).

[0191] 4) Insert the restored frame between corresponding adjacent frames in the input picture list pictListIn to obtain the output picture list pictListOut.

[0192] The following describes in detail the various sub-processes involved in the time domain recovery process in the embodiment of the present disclosure:

[0193] 1) Format conversion: This part includes two sub-parts: converting the YUV format to the RGB format and converting the RGB format to the YUV format.

[0194] ① Convert YUV420P10LE format to RGB24 format:

[0195] Input picture pictIn: YUV420P10LE format, where the sizes of the luminance component (Y component) and the two chrominance components (U component and V component) are: pictInY[hIn, wIn], pictInU[hIn / 2, wIn / 2], pictInV[hIn / 2, wIn / 2]; where pictInY represents the luminance component of the input picture, pictInU represents the U component of the input picture, pictInV represents the V component of the input picture, hIn represents the image height of the input picture, and wIn represents the image width of the input picture;

[0196] Output image pictOut: RGB24 format, size [3, hOut, wOut], where hOut represents the image height of the output image and wOut represents the image width of the output image.

[0197] The steps to convert from YUV420P10LE format to RGB24 format are as follows:

[0198] I. Adjust the image size: The image height hOut and width wOut of the output image are rounded up to multiples of 32 using the following formulas: hOut = Ceil (hIn / 32) * 32; wOut = Ceil (wIn / 32) * 32. Where Ceil (x) means rounding x up to the nearest integer.

[0199] II. For each pixel (i, j) in the input image, where i is 0 to hIn-1 (inclusive) and j is 0 to wIn-1 (inclusive), perform a bitwise right shift operation on pictInY[i,j], pictInU[i / 2,j / 2], and pictInV[i / 2,j / 2] by (BitDepth-8) bits to obtain the corresponding Y, U, and V components. BitDepth is the bit width of all decoded luminance and chrominance components, specified by the VCM syntax.

[0200] III. Using the YUV component to RGB component conversion formula, the R component, G component and B component are calculated based on the Y component, U component and V component that perform the bitwise right shift operation as above:

[0201] R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / 255;

[0202] G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / 255;

[0203] B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / 255;

[0204] Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j); in the above formula, the calculation of the R component, G component and B component divided by 255 is the normalization processing of the converted R component, G component and B component.

[0205] IV. Store the calculated R component, G component, and B component into the output picture pictOut.

[0206] V. Fill remaining pixels: If there are still remaining pixels after the image size is adjusted, copy the RGB value of the last pixel of the original image to the remaining pixels.

[0207] VI. Output the image pictOut in RGB24 format.

[0208] The implementation code example is as follows:

[0209]

[0210] ”

[0211] ②Convert RGB24 format to YUV420P10LE format:

[0212] Input image pictIn: RGB24 format, size [3, hIn, wIn], where hIn represents the image height of the input image, and wIn represents the image width of the input image;

[0213] Output picture pictOut: YUV420P10LE format, where the sizes of the luminance component (Y component) and the two chrominance components (U component and V component) are: pictOutY[hOut, wOut], pictOutU[hOut / 2, wOut / 2], pictOutV[hOut / 2, wOut / 2], respectively; where pictOutY represents the luminance component of the output picture, pictOutU represents the U component of the output picture, pictOutV represents the V component of the output picture, hIn represents the image height of the input picture, and wIn represents the image width of the input picture.

[0214] The steps to convert from RGB24 format to YUV420P10LE format are as follows:

[0215] I. RGB component scaling: Limit the RGB component values ​​to the range of [0, 255]: Scale the RGB component of each pixel (i, j) in the input image so that its value range is between [0, 255]. For each pixel (i, j) and each channel k (0, 1, 2), the RGB component is scaled using the following formula:

[0216] p[k,i,j]=clip3(0,255,(pictIn[k,i,j]*255));

[0217] Among them, clip3(x,min,max) means limiting the value of the input data x to between the minimum value min and the maximum value max.

[0218] II. Using the conversion formula from RGB components to YUV components, calculate the Y component, U component, and V component based on the scaled R component, G component, and B component:

[0219] Y=0.299*R+0.587*G+0.114*B;

[0220] U=-0.169*R-0.331*G+0.499*B;

[0221] V=0.399*R-0.418*G-0.0813*B;

[0222] III. Perform a bitwise left shift operation on the Y component, U component and V component: multiply the Y component, U component and V component by a scaling factor scale, scale = 2^(BitDepth-8), where BitDepth is the bit width of all luminance and chrominance components, indicated by the VCM syntax element.

[0223] The Y component, U component, and V component of each pixel are calculated as follows:

[0224] pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale);

[0225] pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587

[0226]

[0227] Among them, BitDepth is the bit width of all brightness and chrominance components; pictOutY[i,j] represents the Y component of pixel (i,j); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel (i,j); p[0,i,j] represents the RGB component of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB component of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB component of pixel (i,j) in channel k=2.

[0228] Output: Output picture pictOut in YUV420P10LE format.

[0229] The implementation code example is as follows:

[0230]

[0231]

[0232] 2) Time domain interpolation process:

[0233] The input is: ① the first reference image img0 and the second reference image img1, a pair of images in RGB24 format; ②n, the number of images that need to be interpolated between img0 and img1.

[0234] The output is: pListOut.

[0235] The time interpolation process includes the following steps:

[0236] I. Initialize pListOut to empty.

[0237] II. Update pListOut through the time domain interpolation subprocess.

[0238] Among them, the time domain interpolation subprocess:

[0239] Input: ① first reference image, aImg; ② second reference image, bImg; ③ number of images to be interpolated, m;

[0240] Output: Updated pListOut.

[0241] The time interpolation subprocess includes the following steps:

[0242] Through the picture interpolation process described below, an interpolation picture midImg between the first and second reference pictures is generated.

[0243] If pListOut is empty, add midImg to pListOut;

[0244] Otherwise, if the first reference picture is one of the pictures in pListOut, insert midImg into pListOut at a position after the first reference picture;

[0245] Otherwise, insert midImg into pListOut at a position before the second reference picture.

[0246] If m>1, update m to m / 2.

[0247] Furthermore, the following data is used as input to iteratively execute the time interpolation sub-process: ① aImg as the first reference picture; ② midImg as the second reference picture; ③ m as the number of pictures to be interpolated.

[0248] Furthermore, the following data is used as input to iteratively execute the time interpolation sub-process: ① midImg as the first reference picture; ② bImg as the second reference picture; ③ m as the number of pictures to be interpolated.

[0249] 3) Image interpolation process:

[0250] Input: ①fwdImg and bckImg, a pair of RGB24 format images with shape [3,h,w]; ②weight additional identifier.

[0251] Output: An interpolated image outImg in RGB24 format with a shape of [3, h, w].

[0252] The image interpolation process includes the following steps:

[0253] ①Initialize the following data:

[0254] scaleList = [4,2,1];

[0255] Set all elements of flow[4,h,w] to 0;

[0256] Set all elements of mask[1,h,w] to 0;

[0257] p0 = fwdImg;

[0258] p1 = bckImg;

[0259] ②For i=0,…,2, apply the following operations:

[0260] Execute the IFBlock procedure with the following data as input:

[0261] ①Concatenate((p0,p1,mask),0);

[0262] ②Optical flow information flow;

[0263] ③scaleList[i];

[0264] ④The weight of the additional identifier is IFNet.IFBlock;

[0265] The outputs of the IFBlock process are: f0 and m0.

[0266] Furthermore, the IFBlock process is executed with the following data as input:

[0267] ①Concatenate((p1,p0,(–mask)),0);

[0268] ②Concatenate((flow[2:4,:,:],flow[:2,:,:]),0);

[0269] ③scaleList[i];

[0270] ④The weight of the additional identifier is IFNet.IFBlock;

[0271] The outputs of the IFBlock process are: f1 and m1.

[0272] flow=flow+(f0+Concatenate((f1[2:4],f1[:2]),0)) / 2;

[0273] mask = mask + (m0 - m1) / 2;

[0274] p0=Warp(fwdImg,flow[:2,:,:]), the specific process is as follows.

[0275] p1=Warp(bckImg,flow[2:4,:,:],the specific process is as follows.

[0276] maskOut=ApproxSigmoid(mask);

[0277] outImg=p0*maskOut+p1*(1-maskOut).

[0278] 4) IFBlock (interpolation frame block) process:

[0279] 4.1) Overview:

[0280] The input is: ① a tensor x[7,h,w]; ② a tensor flow[4,h,w]; ③ a variable scaleVal; ④ a weight additional identifier.

[0281] The output is: a tensor flowOut[4,h,w] and a tensor maskOut[1,h,w].

[0282] The IFBlock process includes the following steps:

[0283] I. Assign x1 to the nearest exact interpolation NearestInterpolate(x, scaleVal -1 ,7,h,w) the results obtained, the NearestInterpolate function is described as follows.

[0284] II. Assign x2 to the nearest exact interpolation NearestInterpolate(flow, scaleVal -1 ,4,h,w) the results obtained.

[0285] III. Update all elements of x2 to be multiplied by scaleVal -1 .

[0286] IV. Concatenate x1 and x2 along the channel dimension and assign the result to x3.

[0287] V. Assign x4 to the result of the following IncreConv function:

[0288] IncreConv(x3,IFBlock.conv0.0.0.conv,IFBlock.conv0.0.0.bias,IFBloc

[0289] k.conv0.0.1.act,IFBlock.conv0.1.0.conv,IFBlock.conv0.1.0.bias,IFBlo

[0290] ck.conv0.1.1.act,11,90). The IncreConv function is described below.

[0291] VI. Set x[0] to x4. For i ranging from 1 to 4 (inclusive), perform the following steps:

[0292] Assign x[i] to the calculation result of the following StackConv function:

[0293] StackConv(x[i-1],IFBlock.convblock.0.0.conv,IFBlock.convblock.0.0.bias,IFBlock.convblock.0.1.act,IFBlock.convblock.1.0.conv,IFBlock.convblock.1.0.bias,IFBlock.convblock.1.1.act,90). The StackConv function is described below.

[0294] Set x[i] to x[i] plus x[i-1].

[0295] VII. Assign flow to the calculation result of the following ShrinkTransConv function:

[0296] ShrinkTransConv(x[4],IFBlock.conv1.0.transConv,IFBlock.conv1.0.transBias,IFBlock.conv1.1.act,IFBlock.conv1.2.transConv,IFBlock.conv1.2.transBias,90,4). The ShrinkTransConv function is described as follows.

[0297] VIII. Assign Mask to the calculation result of the following ShrinkTrans function:

[0298] ShrinkTransConv(x[4],IFBlock.conv2.0.transConv,IFBlock.conv2.0.transBias,IFBlock.conv2.1.act,IFBlock.conv2.2.transConv,IFBlock.conv2.2.transBias,90,1).

[0299] IX. Output tensor:

[0300] flow_out=NearestInterpolate(flow,scaleVal,4,h,w)*scaleVal;

[0301] X. Output tensor mask_out = NearestInterpolate(mask, scaleVal, 1, h, w).

[0302] 4.2) Nearest interpolation function:

[0303] The nearest interpolation function is expressed as NearestInterpolate(x,scale,cIn,hIn,wIn).

[0304] The input to this function is:

[0305] ①x: A tensor of shape [cIn, hIn, wIn].

[0306] ②scale: scaling factor of spatial dimensions.

[0307] ③cIn,hIn,wIn: input tensor x size value.

[0308] The output of this function is: tensor x_out[cIn,hOut,wOut], which is derived as follows:

[0309] x_out[c,i,j]=x[c,Round(i*scale),Round(j*scale)];

[0310] Among them, the range of c is 0 to cIn-1; the range of i is 0 to hOut-1; the range of j is 0 to wOut-1; hOut=Round(hIn*scale); wOut=Round(wIn*scale).

[0311] Nearest interpolation is an image scaling technique that enlarges an image by copying pixel values ​​to the nearest location. In this function, each pixel (c,i,j) in the input tensor x is copied to the position (c,Round(i*scale),Round(j*scale)) in the output tensor x_out, where scale is the scaling factor that controls the size of the output image.

[0312] 4.3) Quantized convolution process with activation:

[0313] The quantized convolution function with activation is denoted as ConvAct(x,weightConv,bias,weightAct,kVer,kHor,cIn,cOut,s).

[0314] The input to this function is:

[0315] ①x: a tensor of shape [cIn, hIn, wIn];

[0316] ②weightConv: A tensor of shape [cOut, cIn, kVer, kHor], representing the convolution kernel weight;

[0317] ③bias: A tensor of shape [cOut], representing the convolution kernel bias;

[0318] ④weightAct: A tensor of shape [cOut], representing the activation function weight;

[0319] ⑤kVer, kHor: vertical and horizontal sizes of the convolution kernel;

[0320] ⑥cIn: the number of input channels;

[0321] ⑦cOut: the number of output channels;

[0322] ⑧s: step length;

[0323] The output of this process is: tensor x_out[cOut,hIn / s,wIn / s], which is derived as follows:

[0324] The output of this process is derived as follows:

[0325] x0=QConv(x,weightConv,bias,kVer,kHor,cIn,cOut,s);

[0326] x_out=LeakyReLU(x0,weightAct).

[0327] The quantized convolution function with activation combines quantized convolution and activation function together. First, the quantized convolution multiplies the input tensor x with the convolution kernel weightConv and adds the bias bias to get the intermediate result x0. Then, the parameterized rectified linear unit multiplies x0 with the activation function weight weightAct to get the final output tensor x_out.

[0328] 4.4) Incremental convolution process:

[0329] The input to this process is:

[0330] ① A tensor x[cIn,hIn,wIn];

[0331] ②A tensor weight0Conv[cOut / 2,cIn,kVer,kHor];

[0332] ③A tensor weight0Bias[cOut / 2];

[0333] ④A tensor weight0Act[cOut / 2];

[0334] ⑤A tensor weight1Conv[cOut,cOut / 2,kVer,kHor];

[0335] ⑥A tensor weight1Bias[cOut / 2];

[0336] ⑦A tensor weight1Act[cOut];

[0337] ⑧Input channel size cIn;

[0338] ⑨Output channel size cOut.

[0339] The output of this process is the tensor x_out[cOut,hIn / 4,wIn / 4].

[0340] The incremental convolution process is expressed as:

[0341] IncreConv(x,weight0Conv,weight0Bias,weight0Act,weight1Conv,weight1Bias,weight1Act,cIn,cOut). The output of this process is derived as follows:

[0342] x0=ConvAct(x,weight0Conv,weight0Bias,weigth0Act,3,3,cIn,cOut / 2,2)

[0343] x_out=ConvAct(x0,weight1Conv,weight1Bias,weight1Act,3,3,cOut / 2,cOut.

[0344] 4.5) Stacked convolution process with activation:

[0345] The input to this process is:

[0346] ① A tensor x[c,h,w];

[0347] ②A tensor weight0Conv[c,c,kVer,kHor];

[0348] ③A tensor weight0Bias[c];

[0349] ④A tensor weight0Act[c];

[0350] ⑤A tensor weight1Conv[c,c,kVer,kHor];

[0351] ⑥A tensor weight1Bias[c];

[0352] ⑦A tensor weight1Act[c];

[0353] ⑧Input and output channel size c.

[0354] The output of this process is: tensor x_out[c,h,w].

[0355] The stacked convolution process with activation is represented as StackConv(x,weight0Conv,weight0Bias,weight0Act,weight1Conv,weight1Bias,weight1Act,c). The output of this process is derived as follows:

[0356] x0=ConvAct(x,weight0Conv,weight0Bias,weight0Act,3,3,c,c,1);

[0357] x_out=ConvAct(x0,weight1Conv,weight1Bias,weight1Act,3,3,c,c,1).

[0358] 4.6) Contraction transposed convolution process:

[0359] The input to this process is:

[0360] ① A tensor x[c,hIn,wIn];

[0361] ②A tensor weightTransConv0[cIn / 2,cIn,kVer,kHor];

[0362] ③A tensor weightTransBias0[cIn / 2]

[0363] ④A tensor weightAct[cIn / 2];

[0364] ⑤A tensor weightTransConv1[cOut,cIn / 2,kVer,kHor];

[0365] ⑥A tensor weightTransBias1[cOut]

[0366] ⑦ Input channel size cIn;

[0367] ⑧Output channel size cOut.

[0368] The output of this process is the tensor x_out[cOut,hIn*4,wIn*4].

[0369] The shrinking transposed convolution process is expressed as:

[0370] ShrinkTransConv(x,weightTransConv0,weightTransBias0,weightAct,weightTransConv1,weightTransBias1,cIn,cOut).

[0371] The output of this process is derived as follows:

[0372] x0=QTransposedConv(x,weightTransConv0,weightTransBias0,4,4,cIn,cIn / 2,2);

[0373] x1=LeakyReLU(x0,weightAct);

[0374] x_out=QTransposedConv(x1,weightTransConv1,weightTransBias1,4,4,cIn / 2,Cout,5).

[0375] 4.7)Pixel mapping process:

[0376] 4.7.1) Overview:

[0377] The input to this process is:

[0378] ①x: a tensor of shape [c,h,w];

[0379] ②flow: A tensor of shape [2,h,w], flow[h,w,2].

[0380] The output of this process is the tensor xOut[c,h,w], which is derived as follows:

[0381] For a tensor tenHorizontal of shape [1,h,w], tenHorizontal[0,m,:] = LinearSpace(-1.0,1.0,w), where m ranges from 0 to h-1.

[0382] For a tensor tenVertical of shape [1,h,w], tenVertical[0,:,n]=LinearSpace(-1.0,1.0,h), where n ranges from 0 to w-1.

[0383] Concatenate tenHorizontal and tenVertical across the channel dimension and assign the result to tenGrid.

[0384] For a tensor tenFlow of shape [2,h,w], tenFlow[0,m,n]=(flow[0,m,n]*2) / (w-1), where m ranges from 0 to h-1 and n ranges from 0 to w-1. tenFlow[1,m,n]=(flow[1:2,:,:]*2) / (h-1), where m ranges from 0 to h-1 and n ranges from 0 to w-1.

[0385] tenGF[l,m,n]=tenGrid[l,m,n]+tenFlow[l,m,n], where l ranges from 0 to 1, m ranges from 0 to h-1, and n ranges from 0 to w-1.

[0386] For a tensor x5 of shape [h,w,2], x5[:,:,0]=tenGF[0,:,:]x5[:,:,1]=tenGR[1,:,:];

[0387] Use the mapping procedure defined in 4.7.2) with inputs x and x5 and assign the output of the procedure to xOut.

[0388] 4.7.2) Linear space functions:

[0389] The input of this function is: ①tart: starting value; ②end: ending value; ③steps: number of steps.

[0390] The output of this function is: a one-dimensional tensor x_out with a shape of [steps]. Its derivation is as follows:

[0391] x_out[i]=start+((end–start)*i)÷(steps–1), where i ranges from 0 to steps-1.

[0392] The linear space function generates a 1D tensor containing steps values ​​evenly distributed between start and end.

[0393] 4.7.3) Mapping:

[0394] The input of this process is: ①x: a tensor with shape [c,h,w]; ②flow: a tensor with shape [h,w,2].

[0395] The output of this process is a tensor x_out of shape [c,h,w], which is derived as follows:

[0396] For i, the range is 0 to h-1, and for j, the range is 0 to wOut-1.

[0397] hor=((flow[i,j,0]+1) / 2)*(h-1);

[0398] ver=((flow[i,j,1]+1) / 2)*(w-1);

[0399] x_out[k,i,j]=x[k,hor,ver], where 0<=k<=c-1.

[0400] Below, we take the input of img0 and img 1 to generate the intermediate img0.5 as an example:

[0401] img0: dimension is (3, h, w);

[0402] img1: dimension is (3, h, w);

[0403] flow[:4,h,w]: generated optical flow, flow[:2,h,w] represents the optical flow information of 0->0.5, and flow[2:4,h,w] represents the optical flow information of 1->0.5;

[0404] f1[:2]: is the 0->0.5 residual optical flow output by the module, which is accumulated to flow[:2,h,w] to obtain the optical flow output by the block;

[0405] f1[2:4]: is the 1->0.5 residual optical flow output by the module, which is accumulated to flow[2:4,h,w] to obtain the optical flow output by the block;

[0406] warped_img0[:3,:,:] warps the image img0 based on the optical flow of flow[:2,h,w] to obtain the approximate img0.5;

[0407] warped_img1[:3,:,:]: warps the image img1 based on the flow[2:4,h,w] optical flow to get the approximate img0.5;

[0408] mask: mask, dimension is (1, h, w), value range is 0-1, warped_img0 and warped_img1 are combined to generate the final img0.5, the formula is as follows:

[0409] img0.5=mask*warped_img0+warped_img1*(1-mask).

[0410] Based on the same inventive concept, the present disclosure also provides a video encoding device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0411] Fig. 9 A schematic diagram of a video encoding device in an embodiment of the present disclosure is shown. Fig. 9 As shown, the device includes: a time domain sampling information acquisition module 901, a time domain sampling module 902 and a coding module 903.

[0412] Among them, the time domain sampling information acquisition module 901 is used to obtain the time domain sampling information, wherein the time domain sampling information is at least used to determine the time domain sampling multiple of the video encoding end for time domain sampling of the first video sequence; the time domain sampling module 902 is used to perform time domain sampling on the first video sequence according to the time domain sampling information; the encoding module 903 is used to encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding end, wherein the video decoding end is used to decode the received code stream to obtain the second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

[0413] It should be noted that the above-mentioned time domain sampling information acquisition module 901, time domain sampling module 902 and encoding module 903 correspond to S202 to S206 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0414] Based on the same inventive concept, the present disclosure also provides a video decoding device for machine vision tasks, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0415] Fig.10 A schematic diagram of a video decoding device for machine vision tasks in an embodiment of the present disclosure is shown. Fig.10 As shown, the device includes: a decoding module 1001, a video frame interpolation information determination module 1002 and a video frame interpolation module 1003.

[0416] The decoding module 1001 is used to decode the bitstream from the video encoding end to obtain the second video sequence and time domain sampling information, wherein the bitstream is the encoding result after the video encoding end encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine the time domain sampling multiple of the video encoding end for time domain sampling of the first video sequence; the video interpolation frame information determination module 1002 is used to determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; the video interpolation frame module 1003 is used to generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence, and the number of video frames to be inserted between the first video frame and the second video frame.

[0417] It should be noted that the above decoding module 1001, the video interpolation frame information determination module 1002 and the video interpolation frame module 1003 correspond to S302 to S306 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0418] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0419] Refer to the following Fig.11 1100 according to this embodiment of the present disclosure is described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0420] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110).

[0421] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0422] In some embodiments, when the electronic device 1100 is a video encoding device, the processing unit 1110 may execute the following steps of the method embodiment: obtaining time domain sampling information, wherein the time domain sampling information is at least used to determine the time domain sampling multiple of the video encoding end for time domain sampling of the first video sequence; performing time domain sampling on the first video sequence according to the time domain sampling information; encoding the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding end, wherein the video decoding end is used to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0423] In other embodiments, when the electronic device 1100 is a video decoding device, the processing unit 1110 may perform the following steps of the method embodiment: decoding a bitstream from a video encoding end to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result after the video encoding end encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of the first video sequence by the video encoding end; determining the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

[0424] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202 , and may further include a read-only storage unit (ROM) 11203 .

[0425] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0426] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0427] The electronic device 1100 may also communicate with one or more external devices 1140 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0428] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0429] In particular, according to an embodiment of the present disclosure, the process described in the above reference flowchart can be implemented as a computer program product, which includes: a computer program, which, when executed by a processor, implements any one of the above-mentioned video encoding methods for machine vision tasks; or any one of the above-mentioned video decoding methods for machine vision tasks.

[0430] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is further provided, which may be a readable signal medium or a readable storage medium. Fig.12 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. Fig.12As shown, a program product capable of implementing the above method of the present disclosure is stored on the computer-readable storage medium 1200. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of this specification.

[0431] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0432] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0433] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0434] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0435] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0436] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0437] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0438] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A video encoding method for machine vision tasks, characterized in that: include: Acquire time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of the first video sequence by the video encoding end; Performing time-domain sampling on the first video sequence according to the time-domain sampling information; The time domain sampling result and the time domain sampling information of the first video sequence are encoded to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is used to decode the received code stream to obtain a second video sequence and the time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

2. A video decoding method for machine vision tasks, characterized in that: include: Decoding a bitstream from a video encoder to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result after the video encoder encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of the first video sequence by the video encoder; Determine, according to the decoded time domain sampling information, the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; The video frame to be inserted between the first video frame and the second video frame is generated according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

3. The video decoding method for machine vision tasks according to claim 2, characterized in that: The time domain sampling information includes: time domain sampling multiple index information, and the time domain sampling multiple index information is used to determine the time domain sampling multiple used by the video encoding end when performing time domain sampling on the first video sequence; The method of determining the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information includes: Parsing the decoded time domain sampling information to obtain a time domain sampling multiple for time domain sampling of the first video sequence by the video encoding end; The number of video frames to be inserted between the first video frame and the second video frame in the second video sequence is determined according to a time domain sampling multiple of time domain sampling performed by the video encoding end on the first video sequence.

4. The video decoding method for machine vision tasks according to claim 3, characterized in that: The time domain sampling information further includes: time domain sampling position information, and the time domain sampling position information is used to determine the position where the video encoding end performs time domain sampling on the first video sequence.

5. The video decoding method for machine vision tasks according to claim 4, characterized in that: The first video sequence includes: a plurality of sub-video sequences; the plurality of sub-video sequences are sampled in the time domain using different time domain sampling multiples; The time domain sampling multiple index information is used to determine the time domain sampling multiples for time domain sampling of each sub-video sequence in the first video sequence; the time domain sampling position information is used to determine the position for time domain sampling of each sub-video sequence in the first video sequence.

6. The video decoding method for machine vision tasks according to any one of claims 3 to 5, characterized in that: The time domain sampling information further includes: a time domain sampling flag, and the time domain sampling flag is used to indicate whether the video encoding end has performed time domain sampling on the first video sequence.

7. The video decoding method for machine vision tasks according to claim 2, characterized in that: Generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, comprising: If the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame; If there are multiple video frames to be inserted between the first video frame and the second video frame, the following steps are repeatedly performed until all video frames to be inserted between the first video frame and the second video frame are generated: Generate a first intermediate video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; Generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame according to the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; A third intermediate video frame to be inserted between the first intermediate video frame and the second video frame is generated according to the first intermediate video frame and the second intermediate video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

8. The video decoding method for machine vision tasks according to claim 7, characterized in that: The method further comprises: Input any two video frames into a pre-built video frame infill network model, and output a video frame located in the middle between the two video frames, wherein the video frame infill network model is used to generate a video frame located in the middle between the two video frames by learning the optical flow and / or mask residual information of the two video frames.

9. The video decoding method for machine vision tasks according to claim 2, characterized in that: The bit stream is an encoding result after the video encoding end encodes the first video sequence and the time domain sampling information in a hybrid encoding method, and the hybrid encoding method is an encoding method combining inter-frame encoding and intra-frame encoding.

10. The video decoding method for machine vision tasks according to claim 9, characterized in that: The inter-frame coding is traditional inter-frame coding, and the intra-frame coding is end-to-end intra-frame coding.

11. The video decoding method for machine vision tasks according to claim 9 or 10, characterized in that: Before generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: Obtaining a first video frame obtained by decoding the bitstream; Filter the first video frame; The first video frame in the second video sequence is replaced by the filtered video frame.

12. The video decoding method for machine vision tasks according to claim 2, characterized in that: After generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, the method further includes: The video frame to be inserted between the first video frame and the second video frame is inserted into a corresponding position between the first video frame and the second video frame in the second video sequence.

13. A video encoding device, characterized in that: include: A time domain sampling information acquisition module, used to acquire time domain sampling information, wherein the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of a first video sequence by a video encoding end; A time domain sampling module, configured to perform time domain sampling on the first video sequence according to the time domain sampling information; An encoding module is used to encode the time domain sampling result and time domain sampling information of the first video sequence to obtain a code stream to be transmitted to a video decoding end, wherein the video decoding end is used to decode the received code stream to obtain a second video sequence and time domain sampling information, determine the number of video frames to be inserted between the first video frame and the second video frame in the second video sequence according to the decoded time domain sampling information, and generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

14. A video decoding device, characterized in that: include: a decoding module, configured to decode a bitstream from a video encoding end to obtain a second video sequence and time domain sampling information, wherein the bitstream is an encoding result after the video encoding end encodes the time domain sampling result and the time domain sampling information of the first video sequence, and the time domain sampling information is at least used to determine a time domain sampling multiple for time domain sampling of the first video sequence by the video encoding end; a video interpolation frame information determination module, configured to determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence; The video frame interpolation module is used to generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame.

15. A video processing system for machine vision tasks, characterized in that: include: Video encoding device and video decoding device; The video encoding device is used to obtain time domain sampling information, perform time domain sampling on the first video sequence according to the time domain sampling information, and encode the time domain sampling result and the time domain sampling information of the first video sequence to obtain a code stream to be transmitted to the video decoding device; The video decoding device is used to receive a code stream from the video encoding device, obtain a second video sequence and time domain sampling information, and determine the number of video frames to be inserted between a first video frame and a second video frame in the second video sequence according to the decoded time domain sampling information, and then generate a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame in the second video sequence and the number of video frames to be inserted between the first video frame and the second video frame, wherein the first video frame and the second video frame are any two consecutive video frames in the second video sequence.

16. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the video encoding method for machine vision tasks according to claim 1 or the video encoding method for machine vision tasks according to any one of claims 2 to 11 by executing the executable instructions.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video encoding method for machine vision tasks according to claim 1 or the video encoding method for machine vision tasks according to any one of claims 2 to 11 is implemented.

18. A video decoding method for machine vision tasks, characterized in that: include: Decode the video bitstream to be decoded to obtain the decoded video sequence and time domain sampling information; According to the time domain sampling information, a time domain recovery process is performed on the decoded video sequence.

19. The video decoding method for machine vision tasks according to claim 18, characterized in that: The time domain sampling information includes: a time domain recovery flag and time domain resampling ratio information, wherein the time domain recovery flag is used to indicate whether the time domain recovery process is enabled; and the time domain resampling ratio information is used to determine the time domain resampling ratio during the time domain recovery process.

20. The video decoding method for machine vision tasks according to claim 19, characterized in that: The time domain resampling ratio information is a time domain resampling ratio or a time domain resampling ratio index.

21. The video decoding method for machine vision tasks according to claim 19, characterized in that: According to the time domain sampling information, performing time domain recovery processing on the decoded video sequence includes: Determining whether to perform time domain recovery processing on the decoded video sequence according to the time domain recovery flag bit included in the time domain sampling information; When it is determined to perform time domain recovery processing on the to-be-decoded video code stream, determining a time domain resampling ratio according to the time domain resampling ratio information included in the time domain sampling information; According to the time domain resampling ratio, a time domain restoration process is performed on the decoded video sequence.

22. The video decoding method for machine vision tasks according to claim 21, characterized in that: According to the time domain resampling ratio, performing time domain recovery processing on the decoded video sequence includes: Determining the number of video frames to be inserted between a first video frame and a second video frame according to the time domain resampling ratio, wherein the first video frame and the second video frame are any two consecutive video frames in the decoded video sequence; The video frame to be inserted between the first video frame and the second video frame is generated according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame.

23. The video decoding method for machine vision tasks according to claim 22, characterized in that: Generating a video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, and the number of video frames to be inserted between the first video frame and the second video frame, comprises: If the number of video frames to be inserted between the first video frame and the second video frame is one, directly generating the video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame; If there are multiple video frames to be inserted between the first video frame and the second video frame, the following steps are repeatedly performed until all video frames to be inserted between the first video frame and the second video frame are generated: Generate a first intermediate video frame to be inserted between the first video frame and the second video frame according to the first video frame and the second video frame, wherein the first intermediate video frame is a video frame located at an intermediate position between the first video frame and the second video frame; Generate a second intermediate video frame to be inserted between the first video frame and the first intermediate video frame according to the first video frame and the first intermediate video frame, wherein the second intermediate video frame is a video frame located at an intermediate position between the first video frame and the first intermediate video frame; A third intermediate video frame to be inserted between the first intermediate video frame and the second video frame is generated according to the first intermediate video frame and the second intermediate video frame, wherein the third intermediate video frame is a video frame located at an intermediate position between the first intermediate video frame and the second video frame.

24. The video decoding method for machine vision tasks according to claim 23, characterized in that: When there are multiple second intermediate video frames and multiple third intermediate video frames to be inserted between the first video frame and the second video frame, the execution order of the steps of generating the second intermediate video frame and the third intermediate video frame includes any one of the following: Firstly, the step of generating the second intermediate video frame is performed until all the second intermediate video frames are generated, and then the step of generating the third intermediate video frame is performed until all the third intermediate video frames are generated; Respectively executing the step of generating a second intermediate video frame and the step of generating a third intermediate video frame; The step of generating the third intermediate video frame is first performed until all the third intermediate video frames are generated, and then the step of generating the second intermediate video frame is performed until all the second intermediate video frames are generated.

25. The video decoding method for machine vision tasks according to claim 23, characterized in that: The number of video frames to be inserted between the first video frame and the second video frame is n, where n is a natural number; the number of video frames to be inserted between the first video frame and the first intermediate video frame is floor(n / 2); the number of video frames to be inserted between the first intermediate video frame and the second video frame is floor(n / 2); wherein floor is a floor function.

26. The video decoding method for machine vision tasks according to claim 23, characterized in that: The method further comprises: Based on a pre-built video frame interpolation network model, an interpolation frame between two video frames is generated, wherein the video frame interpolation network model is used to generate an interpolation frame between the two video frames by learning optical flow and / or mask residual information of the two video frames.

27. The video decoding method for machine vision tasks according to claim 26, characterized in that: The video frame interpolation network model is an intermediate flow estimation network IFNet model.

28. The video decoding method for machine vision tasks according to claim 23, characterized in that: Based on the pre-built video frame interpolation network model, an interpolation frame between two video frames is generated, including: Converting a first video frame in a YUV format into a third video frame in an RGB format; Converting the second video frame in a YUV format into a fourth video frame in an RGB format; Based on a pre-built video frame interpolation network model, generating an interpolation frame in RGB format according to the third video frame and the fourth video frame; Converting the interpolated frame in the RGB format into an interpolated frame in the YUV format; The interpolated frame in the YUV format is inserted between the first video frame and the second video frame.

29. The video decoding method for machine vision tasks according to claim 28, characterized in that: Convert the YUV video frame to RGB video frame by following the steps below: The YUV format video frame to be converted is resized to obtain a YUV format image that meets a preset size; Extracting a Y component, a U component, and a V component of each pixel from the YUV format image; Perform a bitwise right shift operation on the Y component, U component, and V component of each pixel; Calculate the R component, G component and B component of each pixel according to the Y component, U component and V component after the bitwise right shift operation, wherein if there are remaining pixels in the resized image, the R component, G component and B component of the last column of pixels are used as the R component, G component and B component of the remaining column pixels; the R component, G component and B component of the last row of pixels are used as the R component, G component and B component of the remaining row pixels; The calculated R component, G component, and B component of each pixel are normalized.

30. The video decoding method for machine vision tasks according to claim 29, characterized in that: The R component, G component, and B component of each pixel after normalization are as follows: R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / 255; G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / 255; B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / 255; Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

31. The video decoding method for machine vision tasks according to claim 28, characterized in that: Convert the YUV video frame to RGB video frame by following the steps below: The YUV format video frame to be converted is resized to obtain a YUV format image that meets a preset size; Extracting a Y component, a U component, and a V component of each pixel from the YUV format image; Calculate the R component, G component and B component of each pixel according to the Y component, U component and V component of each pixel, wherein if there are remaining pixels in the resized image, the R component, G component and B component of the last column of pixels are used as the R component, G component and B component of the remaining column of pixels; the R component, G component and B component of the last row of pixels are used as the R component, G component and B component of the remaining row of pixels; The calculated R component, G component, and B component of each pixel are normalized.

32. The video decoding method for machine vision tasks according to claim 31, characterized in that: The R component, G component, and B component of each pixel after normalization are as follows: R[i,j]=(Y[i,j]+1.402*(V[i,j]-128)) / scale; G[i,j]=(Y[i,j]-0.344*(U[i,j]-128)-0.714*(V[i,j]-128)) / scale; B[i,j]=(Y[i,j]+1.772*(U[i,j]-128)) / scale; Where, scale'=2^(BitDepth); Among them, R[i,j] represents the R component of pixel (i,j); G[i,j] represents the G component of pixel (i,j); B[i,j] represents the B component of pixel (i,j); Y[i,j] represents the Y component of pixel (i,j); U[i,j] represents the U component of pixel (i,j); V[i,j] represents the V component of pixel (i,j).

33. The video decoding method for machine vision tasks according to claim 28, characterized in that: Convert the RGB video frame to the YUV video frame by following the steps below: Extract the R component, G component and B component of each pixel; Scaling the R component, G component, and B component of each pixel so that the values ​​of the R component, G component, and B component after the scaling process are limited to a preset range; Calculate the Y component, U component and V component of each pixel according to the R component, G component and B component after scaling processing; Perform a bitwise left shift operation on the calculated Y component, U component, and V component of each pixel.

34. The video decoding method for machine vision tasks according to claim 33, characterized in that: The Y component, U component, and V component of each pixel after the bitwise left shift operation are as follows: pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale); pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j+1]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1,2j]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587*p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale); pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2,i,j])*scale); pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2,i,j])*scale); Where scale = 2^(BitDepth-8); Where BitDepth is the bit width of all brightness and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component in pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB component of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB component of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB component of pixel block (i,j) in channel k=2.

35. The video decoding method for machine vision tasks according to claim 28, characterized in that: Convert the RGB video frame to the YUV video frame by following the steps below: Extract the R component, G component and B component of each pixel; Calculate the Y component, U component and V component of each pixel according to the R component, G component and B component of each pixel; Perform a bitwise left shift operation on the calculated Y component, U component, and V component of each pixel.

36. The video decoding method for machine vision tasks according to claim 35, characterized in that: The Y component, U component, and V component of each pixel after the bitwise left shift operation are as follows: pictOutY[2i,2j]=clip3(0,BitDepth,(0.299*p[0,2i,2j]+0.587*p[1,2i,2j]+0.114*p[2,i,j])*scale); pictOutY[2i,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i,2j+1]+0.587*p[1,2i,2j+1]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j]+0.587*p[1,2i+1,2j]+0.114*p[2,i+1,j])*scale); pictOutY[2i+1,2j+1]=clip3(0,BitDepth,(0.299*p[0,2i+1,2j+1]+0.587*p[1,2i+1,2j+1]+0.114*p[2,i+1,j+1])*scale); pictOutU[i,j]=clip3(0,BitDepth,(-0.169*p[0,i,j]-0.331*p[1,i,j]+0.499*p[2,i,j])*scale); pictOutV[i,j]=clip3(0,BitDepth,(0.399*p[0,i,j]-0.418*p[1,i,j]-0.0813*p[2,i,j])*scale); Where, scale'=2^(BitDepth); Where BitDepth is the bit width of all brightness and chrominance components; pictOutY[2i,2j] represents the Y component of pixel block (2i,2j); pictOutY[2i,2j+1] represents the Y component of pixel block (2i,2j+1); pictOutY[2i+1,2j] represents the Y component in pixel (2i+1,2j); pictOutY[2i+1,2j+1] represents the Y component of pixel (2i+1,2j+1); pictOutU[i,j] represents the U component of pixel (i,j); pictOutV[i,j] represents the V component of pixel block (i,j); p[0,i,j] represents the RGB component of pixel (i,j) in channel k=0; p[1,i,j] represents the RGB component of pixel (i,j) in channel k=1; p[2,i,j] represents the RGB component of pixel block (i,j) in channel k=2.

Images