Temporal sampling processing method and device of video sequence, computer device, readable storage medium and program product

Abstract

The application relates to a time domain sampling processing method, device, equipment, medium and program product of a video sequence, and relates to the technical field of artificial intelligence and video coding. The method can improve the flexibility of sampling rate selection of the video sequence. The method comprises the following steps: extracting reference video frame images from a video sequence, performing target detection on a current video frame image and the reference video frame images to generate a first foreground mask image and a second foreground mask image; predicting a sampling rate through a sampling rate prediction model according to motion feature information, and obtaining a video sequence segment according to the current video frame image and the sampling rate; determining a new current video frame image according to the video sequence segment, and jumping to the step of extracting reference video frame images corresponding to a preset number of subsequent frames of the current video frame image from the video sequence to be processed until a video sampling stop condition is met, so as to determine each video sequence segment and the sampling rate; and obtaining time domain sampling configuration information of the video sequence according to the video sequence segments and the sampling rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and video coding, in particular to a time domain sampling processing method and device of a video sequence, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Video coding can reduce the storage and transmission cost by compressing the redundant information in the video and reducing the data volume while maintaining the video quality as much as possible. Time domain processing can effectively reduce the storage and transmission cost of video data by removing the inter-frame redundant information in video coding.

[0003] In the pre-processing of the video coding end, the redundant frames can be discarded by using time domain downsampling to reduce the number of video frames. In the post-processing of the video decoding end, the original frame number can be restored by using time domain upsampling. The time domain sampling rate has an important influence on the balance between the code rate and the quality of video coding.

[0004] In the traditional pre-coding processing, a fixed sampling rate is usually used for sampling processing of the video sequence, which has the problem of insufficient flexibility and is difficult to be applied to, for example, coding tasks for machine vision. SUMMARY

[0005] Therefore, it is necessary to provide a time domain sampling processing method and device of a video sequence, a computer device, a computer readable storage medium and a computer program product to solve the above technical problems.

[0006] In a first aspect, the present application provides a time domain sampling processing method of a video sequence, comprising:

[0007] extracting reference video frame images corresponding to a preset number of frames after a current video frame image from a video sequence to be processed, performing target detection on the current video frame image and the reference video frame images to generate a first foreground mask image and a second foreground mask image for a moving target;

[0008] predicting a sampling rate by a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image, and obtaining a video sequence segment in the video sequence according to the current video frame image and the sampling rate;

[0009] determining a new current video frame image according to the video sequence segment, and returning to execute the step of extracting reference video frame images corresponding to a preset number of frames after a current video frame image from a video sequence to be processed until a video sampling stop condition is met, and determining each group of video sequence segments and the corresponding sampling rate of the video sequence;

[0010] According to the video sequence segments of each group and the corresponding sampling rates, time domain sampling configuration information of the video sequence is obtained.

[0011] In one of the embodiments, before the sampling rate is predicted according to the motion feature information of the first foreground mask image and the second foreground mask image by the sampling rate prediction model, the method further comprises:

[0012] The peak signal-to-noise ratio, the frame difference, and the target total area of the first foreground mask image and the second foreground mask image are obtained, and the motion feature information is obtained according to the peak signal-to-noise ratio, the frame difference, and the target total area.

[0013] In one of the embodiments, the sampling rate is predicted according to the motion feature information of the first foreground mask image and the second foreground mask image by the sampling rate prediction model, comprising:

[0014] The motion feature information is input into the sampling rate prediction model for prediction to obtain a probability distribution of multiple candidate sampling rates, and the candidate sampling rate with the maximum probability is identified as the sampling rate according to the probability distribution.

[0015] In one of the embodiments, the motion feature information is input into the sampling rate prediction model for prediction to obtain a probability distribution of multiple candidate sampling rates, comprising:

[0016] The motion feature information is input into the sampling rate prediction model, the motion feature information is linearly transformed by a classifier in the sampling rate prediction model to obtain scores corresponding to multiple candidate sampling rates, and the scores are converted into probability distributions corresponding to the candidate sampling rates by a conversion function in the classifier.

[0017] In one of the embodiments, the method further comprises:

[0018] The sampling rate data set is split into a training set and a test set, the initial classifier is trained by using the training set, and the model parameters of the initial classifier are updated until the accuracy of the classifier after parameter updating on the test set meets a threshold condition, and the sampling rate prediction model containing the classifier is obtained.

[0019] In one of the embodiments, before the sampling rate data set is split into a training set and a test set, the method further comprises:

[0020] Obtaining a video sequence sample set containing multiple resolutions, extracting a plurality of continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set; inputting each video sequence segment sample in the video sequence segment sample set into a machine vision video coding system at a plurality of sampling rates, and counting video coding index evaluation values and target detection index evaluation values corresponding to each sampling rate; for each video sequence segment sample, determining a target sampling rate from the plurality of sampling rates according to the video coding index evaluation values and the target detection index evaluation values; and obtaining the sampling rate data set according to motion feature information of each video sequence segment sample in the video sequence segment sample set and the target sampling rate of each video sequence segment sample.

[0021] In one of the embodiments, the obtaining of the time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates comprises:

[0022] forming a sampling rate pair by the video sequence segment and the corresponding sampling rate, arranging the sampling rate pair to form a sampling rate array according to the sequence of the video sequence segments, and obtaining the time domain sampling configuration information according to the sampling rate array.

[0023] After the obtaining of the time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates, the method further comprises:

[0024] transmitting a sampling video sequence obtained by performing time domain downsampling on the video sequence according to the time domain sampling configuration information into an encoder to obtain a bitstream output by the encoder, appending the time domain sampling configuration information as header information to the bitstream to form a bitstream file, and transmitting the bitstream file to a decoding end, wherein the bitstream file is used to extract the time domain sampling configuration information and the sampling video sequence by the decoding end, and to recover the video sequence according to the time domain sampling configuration information and the sampling video sequence.

[0025] In a second aspect, the application further provides a time domain sampling processing device of a video sequence, comprising:

[0026] a target detection module configured to extract reference video frame images corresponding to a preset number of frames after a current video frame image from a video sequence to be processed, and perform target detection on the current video frame image and the reference video frame images to generate first and second foreground mask images for a moving target;

[0027] a sampling rate prediction module configured to predict a sampling rate according to motion feature information of the first foreground mask image and the second foreground mask image by using a sampling rate prediction model, and obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate;

[0028] a loop execution module configured to determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting reference video frame images corresponding to a preset number of subsequent frames of the current video frame image from the video sequence to be processed until a video sampling stop condition is met, and determine each group of the video sequence segments and the corresponding sampling rates of the video sequence;

[0029] a sampling configuration module configured to obtain time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates.

[0030] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0031] extracting reference video frame images corresponding to a preset number of subsequent frames of a current video frame image from a video sequence to be processed, performing target detection on the current video frame image and the reference video frame images to generate a first foreground mask image and a second foreground mask image for a moving target, predicting a sampling rate according to motion feature information of the first foreground mask image and the second foreground mask image by using a sampling rate prediction model, and obtaining a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate; determining a new current video frame image according to the video sequence segment, and returning to execute the step of extracting reference video frame images corresponding to a preset number of subsequent frames of the current video frame image from the video sequence to be processed until a video sampling stop condition is met, and determining each group of the video sequence segments and the corresponding sampling rates of the video sequence; and obtaining time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates.

[0032] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0033] extract a reference video frame image corresponding to a preset number of frames behind a current video frame image from a video sequence to be processed, perform target detection on the current video frame image and the reference video frame image to generate a first foreground mask image and a second foreground mask image for a moving target, predict a sampling rate according to motion feature information of the first foreground mask image and the second foreground mask image through a sampling rate prediction model, obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate, determine a new current video frame image according to the video sequence segment, and return to perform the step of extracting a reference video frame image corresponding to a preset number of frames behind a current video frame image from a video sequence to be processed until a video sampling stop condition is met, to determine each group of video sequence segments and the corresponding sampling rate of the video sequence; and obtain time domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rate.

[0034] In a fifth aspect, the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the following steps:

[0035] extract a reference video frame image corresponding to a preset number of frames behind a current video frame image from a video sequence to be processed, perform target detection on the current video frame image and the reference video frame image to generate a first foreground mask image and a second foreground mask image for a moving target, predict a sampling rate according to motion feature information of the first foreground mask image and the second foreground mask image through a sampling rate prediction model, obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate, determine a new current video frame image according to the video sequence segment, and return to perform the step of extracting a reference video frame image corresponding to a preset number of frames behind a current video frame image from a video sequence to be processed until a video sampling stop condition is met, to determine each group of video sequence segments and the corresponding sampling rate of the video sequence; and obtain time domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rate.

[0036] The time domain sampling processing method, device, computer device, computer readable storage medium and computer program product of the video sequence can reflect the object motion condition between the current video frame image and the corresponding reference video frame image by analyzing the motion feature of the moving target in the video segment in the video sequence, and can adaptively predict the sampling rate corresponding to each video segment through the sampling rate prediction model according to the motion feature information, so that the sampling rate of each video segment in the video sequence can be flexibly determined, the flexibility of the sampling rate selection of the video sequence is improved, and the video sequence can be flexibly applied to, for example, a machine vision-oriented coding task. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained on the basis of these drawings without creative labor.

[0038] Figure 1 An application environment diagram of the time domain sampling processing method of the video sequence in one embodiment;

[0039] Figure 2 A flowchart of the time domain sampling processing method of the video sequence in one embodiment;

[0040] Figure 3 A flowchart of the time domain sampling processing method of the video sequence in another embodiment;

[0041] Figure 4 A flowchart of the step of constructing the sampling rate dataset in one embodiment;

[0042] Figure 5 A flowchart of the time domain sampling processing method of the video sequence in one specific embodiment;

[0043] Figure 6 A structural block diagram of the time domain sampling processing device of the video sequence in one embodiment;

[0044] Figure 7 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0046] Related term explanations of the present application:

[0047] Time domain sampling rate: refers to sampling of video frames in a time domain adaptive sampling module of video encoding according to a specific sampling rate. For example, a sampling rate of 4 indicates that a window of 4 frames and a step of 4 are moved on the video sequence, and the 1st frame in each window is sampled, and all the sampled frames are spliced to generate a time domain down-sampled video sequence. It should be noted that a sampling rate of 0 indicates that no time domain down-sampling is performed.

[0048] Machine vision video coding: refers to the video coding mode designed for machine vision tasks (such as target detection, object tracking, etc.), and the decoded video is mainly used for analysis and processing of machine vision tasks.

[0049] In video coding, the time domain sampling rate has an important influence on the rate and quality balance of video coding. For video sequences with complex motion, it is difficult to perform frame interpolation at the decoding end, so a lower sampling rate should be used to ensure the frame interpolation effect; while for video sequences with simple motion, it is relatively easy to perform frame interpolation at the decoding end, so a higher sampling rate can be used to improve the compression performance.

[0050] In traditional pre-coding processing, a fixed sampling rate is usually used for sampling processing of video sequences, for example:

[0051] 1. Fixed time domain sampling rate: a fixed time domain sampling rate (such as a sampling rate of 4) is always used, regardless of the type of video sequence or the specific needs of the coding task, the same sampling rate is fixedly used, such as the sampling rate used in machine-oriented video coding VCM.

[0052] 2. Decoding task-oriented sampling rate selection: in human eye vision-oriented video coding tasks, a lower sampling rate is usually selected to ensure high fidelity of reconstructed video frames; while in machine vision-oriented coding tasks, a higher sampling rate is usually selected to achieve low delay and high compression ratio in complex scenes.

[0053] 3. Dynamic time domain sampling rate selection: the perceptual quality difference between the down-sampled video frames in time domain and the original full frame rate video frames is evaluated by frame rate related quality metric FRQM, when the FRQM exceeds the set threshold, a specific sampling rate is selected for down-sampling; otherwise, no sampling operation is performed.

[0054] Therefore, there is a problem of insufficient flexibility in traditional pre-coding processing. The first two sampling rate selection methods use a fixed sampling rate throughout the video sequence, but a single video sequence often contains segments with different motion patterns. Using the same sampling rate in segments with different motion complexities is obviously unreasonable, which can lead to reduced compression performance or damaged reconstructed video quality. The third method can dynamically select whether to sample according to the video content, but it can only select a single sampling rate and cannot achieve flexible adjustment of multiple sampling rates, so the flexibility is still insufficient. Moreover, the traditional pre-coding processing also lacks adaptability to machine vision video coding. Among them, in machine vision-oriented coding tasks, the coding needs of foreground objects are more critical, and the above-mentioned methods do not consider the influence of foreground objects on coding performance, and lack adaptability to machine vision tasks.

[0055] To this end, the video sequence time domain sampling processing method provided in the application can flexibly determine the respective sampling rates of each video segment in the video sequence, improve the flexibility of sampling rate selection of the video sequence, and make it applicable to, for example, machine vision-oriented coding tasks (such as target detection, object tracking, and the like).

[0056] The video sequence time domain sampling processing method provided in the embodiments of the application can be applied to, for example, the application environment as shown in Figure 1 . In the application environment, a terminal communicates with a server through a network. A data storage system can store data required to be processed by the server. The data storage system can be integrated on the server, or placed on a cloud or other network server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, and tablet computers. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0057] In one embodiment, as shown in Figure 2 , a video sequence time domain sampling processing method is provided. The method is applied to, for example, a terminal in Figure 1 . The method includes the following steps:

[0058] In step S201, a reference video frame image corresponding to a preset number of frames after a current video frame image is extracted from a video sequence to be processed. The current video frame image and the reference video frame image are subjected to target detection to generate a first foreground mask image and a second foreground mask image for a moving target.

[0059] The target detection is a task in the time domain sampling process, and aims to identify all objects of interest in the current video frame image and the reference video frame image and accurately locate them.

[0060] Specifically, as shown in Figure 3 , the terminal extracts a reference video frame image corresponding to a preset number of frames after a current video frame image from a video sequence to be processed by using a double-pointer method. The current video frame image and the reference video frame image are subjected to target detection. A Faster-RCNN model (Region-based Convolutional Neural Network, a deep learning model for target detection) is used to obtain a target detection frame of each video frame image. A first foreground mask image and a second foreground mask image for a moving target are generated based on the detected moving target. In this embodiment, the foreground region is retained and the background is covered to achieve background removal. The purpose of this is because in machine vision video coding, the coding focuses on the moving target, and the decoded video is also used for machine vision task analysis.

[0061] Step S202, predicting a sampling rate according to the motion feature information of the first foreground mask image and the second foreground mask image through a sampling rate prediction model, and obtaining a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate.

[0062] The motion feature information can include a peak signal-to-noise ratio, a frame difference, and a target total area, and the like.

[0063] To achieve the goal of dynamically adjusting the time-domain sampling rate according to the video content, the application uses a double-pointer method to process the video frame images in a video sequence. For a segment with a complex motion mode, a low sampling rate is used; for a segment with a simple motion mode, a high sampling rate is used, so as to more reasonably allocate the code rate while ensuring the video reconstruction quality.

[0064] As shown in Figure 3 , the specific process is as follows: define two pointers and , which correspond to the 0th frame and the 4th frame of the video respectively. Read the images and corresponding to the pointers and , calculate the optimal sampling rate by using a time-domain adaptive sampling method through a sampling rate prediction model according to the peak signal-to-noise ratio, the frame difference, and the target total area of the images and . Then, adjust the position of the pointer according to the sampling rate , move the pointer by -4 positions (a positive value indicates backward movement, and a negative value indicates forward movement), record the sampling rate of the video sequence segment, and obtain a corresponding video sequence segment in the video sequence according to the pointer and the sampling rate (the sequence number of the first frame is ).

[0065] Step S203, determining a new current video frame image according to the video sequence segment, and returning to execute the step of extracting the reference video frame images of the subsequent preset number of frames corresponding to the current video frame image from the video sequence to be processed until a video sampling stop condition is met, to determine each group of video sequence segments and the corresponding sampling rates of the video sequence.

[0066] The video sampling stop condition can be set by a person skilled in the art according to actual needs, for example: when the pointer exceeds the length of the video sequence, stop video sampling.

[0067] The "returning to execute" refers to repeatedly executing the "determining a new current video frame image according to the video sequence segment" in the step S201, the step S202 and the step S203.

[0068] Specifically, after entering the next iteration, the terminal determines a new current video frame image according to the video sequence segment, that is, assigns the position of the pointer to the pointer , and redefines the pointer as +4, and repeats the above operation until the pointer goes beyond the length of the video sequence, and finally determines each group of video sequence segments and the corresponding sampling rate of the video sequence.

[0069] In the step S204, the time domain sampling configuration information of the video sequence is obtained according to each group of video sequence segments and the corresponding sampling rate.

[0070] Specifically, the terminal records the sampling rate in the sampling rate array, and the data structure is a sampling rate pair corresponding to each sampling position, which indicates that the sampling rate of the video sequence segment (the sequence number of the first frame is ) is , and the time domain sampling configuration information of the video sequence is obtained by combining a plurality of sampling rate pairs.

[0071] In the above-mentioned time domain sampling processing method of the video sequence, the motion characteristics of the moving target in the video segment in the video sequence are analyzed to reflect the object motion between the current video frame image and the corresponding reference video frame image, and the sampling rate corresponding to each video segment is adaptively predicted by the sampling rate prediction model according to the motion characteristic information, so that the sampling rate of each video segment in the video sequence can be flexibly determined, the flexibility of the sampling rate selection of the video sequence is improved, and the video sequence can be flexibly applied to, for example, a machine vision-oriented coding task.

[0072] In one embodiment, before the sampling rate is predicted by the sampling rate prediction model according to the motion characteristic information of the first foreground mask image and the second foreground mask image, the method of the present application further includes the following steps:

[0073] Obtaining the peak signal-to-noise ratio, the frame difference and the target total area between the first foreground mask image and the second foreground mask image; and obtaining the motion characteristic information according to the peak signal-to-noise ratio, the frame difference and the target total area.

[0074] In the step S202, the sampling rate is predicted by the sampling rate prediction model according to the motion characteristic information of the first foreground mask image and the second foreground mask image, and specifically includes the following steps:

[0075] The motion feature information is input into the sampling rate prediction model, the motion feature information is linearly transformed by a classifier in the sampling rate prediction model, and scores corresponding to a plurality of candidate sampling rates are obtained; the scores are converted into probability distributions corresponding to the candidate sampling rates by using a conversion function in the classifier; and the candidate sampling rate with the maximum probability is identified as the sampling rate according to the probability distributions.

[0076] The candidate sampling rate can be [0, 2, 4, 8], the classifier in the sampling rate prediction model can be a trained Softmax regression model, and the conversion function in the classifier can be a Softmax function.

[0077] Specifically, to select a suitable sampling rate, the peak signal-to-noise ratio of two frames of images with masks, the frame difference and the target total area of two frames of images need to be calculated, which are used to evaluate the motion features and target size of the objects. The peak signal-to-noise ratio is mainly used to measure the similarity between two frames of images. The greater the peak signal-to-noise ratio, the smaller the difference between the two frames, indicating that the motion or content change is smaller, and a higher sampling rate can be considered. On the contrary, the smaller the peak signal-to-noise ratio, the greater the difference between the frames, indicating that there is strong motion or complex content change, and a lower sampling rate may be needed to retain more frames. The calculation formula of the peak signal-to-noise ratio (PSNR) is as follows:

[0078] (1)

[0079] Frame difference is the absolute difference of pixel values between two frames of images, which is used to describe the motion information of the images. The greater the frame difference, the more significant the motion or object displacement between the two frames of images. The frame difference can help detect the significance and degree of inter-frame motion, which is very important for determining the temporal sampling rate. When the frame difference is large, appropriately reducing the sampling rate helps to retain this motion information. The calculation formula of the frame difference is as follows:

[0080] (2)

[0081] Target total area reflects the size of the foreground target, and the encoding of the foreground target is usually prioritized over the background in machine vision video encoding. The larger the foreground target, the higher the encoding and storage cost. In order to achieve efficient encoding performance, when the foreground target is large, a higher sampling rate may be selected to discard more frames to reduce the storage and transmission burden.

[0082] It should be noted that the present application focuses on video coding for machine vision tasks (such as target detection, object tracking, etc.), that is, coding methods designed for machine vision applications. In this method, the coding process focuses more on the identification and preservation of foreground objects. Therefore, the time domain sampling rate selection in the present application mainly focuses on factors such as the motion complexity and target size of foreground objects in video frames.

[0083] After calculating the above three indicators, the feature vector is obtained, which has the following specific form:

[0084] (3)

[0085] The three indicators have a clear correlation with the sampling rate. PSNR and present a positive correlation with the sampling rate, and the sampling rate presents a negative correlation. In the case of a small number of features and a clear linear relationship between the features and the labels, we use a Softmax regression model as the classifier, inputting the feature vector . The model maps the input to the probability distribution of multiple categories through linear transformation and the Softmax function. In this embodiment, different categories correspond to different sampling rates [0, 2, 4, 8]. The linear transformation process in the Softmax regression is as follows:

[0086] (4)

[0087] where is the weight vector of category k, is the bias term of category k, represents the score for category k. Next, each category score is input into the softmax function to obtain the probability distribution of each category, and the expression of the softmax function is:

[0088] (5)

[0089] where represents the probability that the feature vector belongs to category k. Finally, the model selects the category with the highest probability as the sampling rate prediction result. In this embodiment, through the Softmax regression model, the input three features are linearly transformed and input into the Softmax function to obtain the probability distribution corresponding to the sampling rate [0, 2, 4, 8], and the sampling rate with the highest probability is selected as the final prediction result.

[0090] It should be noted that the Softmax regression model is trained on the time domain sampling rate dataset, and a cross-entropy loss function is used in the model training process. By minimizing the loss, the model can continuously adjust the weights and bias so that the predicted probability distribution is as close as possible to the actual class distribution. Finally, the model achieves a prediction accuracy of more than 90% on the test set, and the output model. In addition, the model needs to be frozen when used in the time domain adaptive sampling method.

[0091] In this embodiment, the motion characteristics and target size of the object are evaluated by analyzing the peak signal-to-noise ratio, frame difference, and target total area of the first and second foreground mask images, thereby obtaining motion characteristic information. Then, the motion characteristic information is linearly transformed and score converted by the sampling rate prediction model, effectively obtaining the probability distribution corresponding to each candidate sampling rate. Meanwhile, the balance between code rate and reconstructed video quality is considered, and the candidate sampling rate with the highest probability is adaptively selected as the optimal sampling rate, thereby realizing intelligent adjustment of the sampling rate, significantly optimizing the code rate allocation, and effectively improving the flexibility of sampling rate selection and video encoding performance.

[0092] In one of the embodiments, the method of the present application further comprises the following steps:

[0093] The sampling rate dataset is divided into a training set and a test set. The initial classifier is trained using the training set, and the model parameters of the initial classifier are updated until the accuracy of the classifier after parameter update on the test set meets the threshold condition, and a sampling rate prediction model containing the classifier is obtained.

[0094] The threshold condition can be set by relevant technical personnel according to actual needs, for example: the accuracy of the classifier on the test set is more than 90%.

[0095] Specifically, the terminal prepares a time domain sampling rate dataset and divides it into a training set and a test set according to an 8:2 ratio. A Softmax regression model is constructed as an initial classifier. The initial classifier is trained using the training set, and the model parameters of the classifier are updated using a gradient descent optimization algorithm to improve classification accuracy. After each round of training, the model performance of the classifier is evaluated using the test set. If the accuracy of the classifier on the test set does not reach 90%, the model parameters of the classifier are adjusted and the above steps are repeated until the accuracy of the classifier exceeds 90%. When the accuracy of the classifier on the test set reaches more than 90%, the model parameters of the classifier are frozen, a sampling rate prediction model containing the classifier is obtained, and it is applied to the time domain adaptive sampling method for inference prediction.

[0096] In this embodiment, the initial classifier is trained by using the training set, and the trained classifier is evaluated by using the test set to fit the best model parameters of the classifier, thereby effectively improving the accuracy of the sampling rate prediction model.

[0097] In one embodiment, as shown in Figure 4 Before splitting the sampling rate dataset into the training set and the test set, the method of the present application further includes the following steps:

[0098] Step S401, obtaining a video sequence sample set containing multiple resolutions, extracting a plurality of continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set.

[0099] Step S402, inputting each video sequence segment sample in the video sequence segment sample set to a machine vision video coding system at multiple sampling rates, and counting the video coding index evaluation value and the target detection index evaluation value corresponding to each sampling rate.

[0100] Step S403, for each video sequence segment sample, determining a target sampling rate from the multiple sampling rates according to the video coding index evaluation value and the target detection index evaluation value.

[0101] Step S404, obtaining a sampling rate dataset according to the motion feature information of each video sequence segment sample in the video sequence segment sample set and the target sampling rate of each video sequence segment sample.

[0102] The video coding index can be BD-Rate (Bjøntegaard Delta Rate, an index used to evaluate the change between different compression ratios and video quality of a video coding algorithm), and the target detection index can be BD-mAP (Bjøntegaard Delta mean Average Precision, an index used to evaluate the performance difference between two target detection models at the same or different detection accuracies).

[0103] Specifically, the time domain sampling rate dataset can be constructed based on the BVI-DVC dataset. The BVI-DVC dataset contains video sequences of multiple resolutions, from high definition to lower resolutions, and the four common resolutions are A (3840x2176), B (1920x1088), C (960x544) and D (480x272). The BVI-DVC dataset contains a total of 800 video sequences, 200 video sequences for each resolution, and covers a wide range of dynamic scenes, such as indoor and outdoor scenes, moving objects, and static backgrounds, which can truly reflect the video characteristics in actual applications. In this embodiment, 8 consecutive frames of images are randomly extracted from each video sequence in the BVI-DVC dataset to form 800 video segments. Each video segment is input into the machine vision video coding system at four different sampling rates [0, 2, 4, 8], and the target detection is used as the downstream task to calculate the values of BD-Rate and BD-mAP, and the sampling rate with the lowest BD-Rate and the highest BD-mAP is selected as the label value of the video segment. The BD-Rate and the BD-mAP are negatively correlated, and the selection of this sampling rate can ensure that the compression rate of the video segment is optimal and the target detection performance is the highest. The motion feature information of each video segment is calculated according to the peak signal-to-noise ratio of the first frame and the fifth frame image, the frame difference and the total target area . After the above dataset construction steps, a time domain sampling rate dataset containing 800 sample features and corresponding label values is obtained, which is used for training the sampling rate prediction model.

[0104] In this embodiment, a sampling rate dataset containing 800 video segments is constructed, with the motion characteristics of the foreground target and the target size as the feature vector, and the best time domain sampling rate as the label value, providing basic data for training the sampling rate prediction model.

[0105] In one of the embodiments, in the step S204, the time domain sampling configuration information of the video sequence is obtained according to each group of video sequence segments and the corresponding sampling rates, which includes the following steps:

[0106] The video sequence segment and the corresponding sampling rate form a sampling rate pair; the sampling rate pairs are arranged in the order of the video sequence segments to form a sampling rate array; and the time domain sampling configuration information is obtained according to the sampling rate array.

[0107] Specifically, the terminal records the sampling rate of each sampling position in the sampling rate array, and the data structure is a sampling rate pair corresponding to each sampling position, which indicates that the sampling rate of the video sequence segment (the sequence number of the first frame is ) is The multiple groups of sampling rate pairs are arranged in the order of the video sequence segments to form a sampling rate array, and finally, time domain sampling configuration information is obtained according to the sampling rate array.

[0108] In the embodiment, the video sequence segments and the corresponding sampling rates are formed into sampling rate pairs, and the multiple groups of sampling rate pairs are arranged in the order of the video sequence segments to form a sampling rate array, so that the time domain sampling configuration information of the video sequence can be accurately generated on the basis of the sampling rate array.

[0109] In one of the embodiments, after the time domain sampling configuration information of the video sequence is obtained according to the groups of video sequence segments and the corresponding sampling rates, the method of the present application further includes the following steps:

[0110] The sampling video sequence obtained by performing time domain downsampling on the video sequence according to the time domain sampling configuration information is transmitted into an encoder to be encoded to obtain a bitstream output by the encoder; the time domain sampling configuration information is attached to the bitstream as header information to form a bitstream file; the bitstream file is transmitted to a decoding end; and the bitstream file is used to extract the time domain sampling configuration information and to decode the sampling video sequence at the decoding end, so as to recover the video sequence according to the time domain sampling configuration information and the sampling video sequence.

[0111] Specifically, the terminal performs time domain downsampling on the video sequence according to the time domain sampling configuration information to obtain a sampling video sequence, and the sampling video sequence enters other pre-processing modules (such as a spatial domain module, a ROI module, etc.) for further processing; after the pre-processing is completed, the video sequence enters an encoder to be encoded to generate a bitstream, and the time domain sampling configuration information is attached to the bitstream as header information to form a bitstream file, which is transmitted to a decoding end; the decoding end receives the bitstream file, reads the header information to extract the time domain sampling configuration information, and decodes the bitstream file by using a decoder to generate a video sequence; and the video sequence is then time domain upsampled according to the time domain sampling configuration information to recover the original video frame length.

[0112] In the embodiment, the sampling video sequence obtained by performing time domain downsampling is encoded by the encoder to obtain a bitstream, and the time domain sampling configuration information is fused to generate a bitstream file, which can be used for decoding at the decoding end to recover the original sequence.

[0113] As an application example, a machine vision video encoding system consists of an encoding end and a decoding end. At the encoding end, the input video sequence sequentially enters a temporal adaptive sampling module and other preprocessing modules. The temporal adaptive sampling module, based on sampling information obtained from video content analysis, guides the temporal downsampling module to adaptively downsample the video sequence and writes the sampling information as header information into the bitstream. At the decoding end, sampled frames and temporal sampling configuration information are decoded from the bitstream. The sampled frames enter the post-processing module for restoration, while the temporal sampling configuration information guides the temporal upsampling module to recover the original video sequence length. Figure 5 As shown, a temporal sampling processing method for a video sequence is provided in a specific embodiment, which specifically includes the following steps:

[0114] First, the steps for constructing the time-domain sampling rate dataset are as follows:

[0115] Step 1: Download and prepare the dataset:

[0116] Download and prepare 800 video sequences from the BVI-DVC dataset;

[0117] Step 2: Extract video clips:

[0118] Eight consecutive frames are randomly selected from each video sequence to form 800 video clips;

[0119] Step 3: Input into the coding system and collect statistics:

[0120] Each video segment was input into the machine vision video coding system at four different temporal sampling rates [0, 2, 4, 8], and object detection was used as the downstream task to calculate its BD-Rate and BD-mAP values.

[0121] Step 4: Calculate the label value:

[0122] For each video segment, the sampling rate with the lowest BD-Rate and the highest BD-mAP is selected as the label value. ;

[0123] Step 5: Calculate the feature vector:

[0124] Calculate the feature vector of each video segment Based on the images in frames 1 and 5, the PSNR and frame difference are calculated. and the total area of ​​the target in the two frames ;

[0125] Step 6: Construct the dataset:

[0126] Ultimately, each video segment contains a feature vector. and a tag value , and a total of 800 video clips constitute the time domain sampling rate dataset.

[0127] Secondly, the training steps of the classification model are as follows:

[0128] Step 1, prepare the dataset:

[0129] Prepare the time domain sampling rate dataset and split it into training set and test set according to 8:2;

[0130] Step 2, define the model:

[0131] Build a Softmax regression model;

[0132] Step 3, train the model:

[0133] Use the training set to train the model, use gradient descent optimization algorithm to update the model parameters, and improve the classification accuracy;

[0134] Step 4, evaluate the model:

[0135] After each round of training, use the test set to evaluate the model performance;

[0136] Step 5, adjust and optimize:

[0137] If the accuracy of the model on the test set does not reach 90%, adjust the model parameters and repeat the above steps 3 and 4 until the accuracy of the model exceeds 90%;

[0138] Step 6, output the model:

[0139] When the model reaches an accuracy of 90% or more on the test set, freeze the model parameters and apply them to the time domain adaptive sampling method for inference prediction.

[0140] Then, the processing process of the time domain adaptive sampling module is as follows:

[0141] Step 1, input the video sequence:

[0142] Input a video sequence;

[0143] Step 2, initialize parameters:

[0144] Define the pointer And , respectively initialized to 0 and 4; Initialize a sampling rate array to record the sampling rate corresponding to different frame positions;

[0145] Step 3, get the video frame:

[0146] Get the th frame and the th frame image, respectively denoted as and ;

[0147] Step 4: Calculate the optimal sampling rate:

[0148] right and Perform object detection and generate a masked image that only displays foreground objects; calculate PSNR and frame difference. and target total area Three feature values ​​are input into the classification model to predict the optimal sampling rate. ;

[0149] Step 5: Record the sampling rate:

[0150] sampling rate Recorded in a sampling rate array, the data structure is a sampling rate pair corresponding to each sampling position [ ];

[0151] Step 6: Update the pointer position:

[0152] Update pointer position: Move pointer According to step size -4 is used for offset, and the updated position is assigned to ;Will pointer updated to +4;

[0153] Step 7: Repeat the process until finished.

[0154] Repeat steps 3-6 above until the pointer... Exceeds the video sequence length;

[0155] Step 8, Output Results:

[0156] The output sampling rate array is the output result of the time-domain adaptive sampling module.

[0157] In summary, the application steps of the temporal adaptive sampling module in a machine vision video coding system are as follows:

[0158] Step 1: Input video sequence:

[0159] Input a video sequence to be processed;

[0160] Step 2: Generate time-domain sampling configuration information:

[0161] The video sequence enters the temporal adaptive sampling module, which determines the current video frame and the reference video frame in the video sequence. Based on the current video frame and the reference video frame (e.g., the first...),... Frame and the The motion feature information (object motion feature and target size) between the four frames (the window length is 4) is used to estimate the compression and reconstruction performance of the four video frame images, the optimal sampling rate is output through the sampling rate prediction model, and the time domain sampling configuration information corresponding to the video sequence is generated according to the sampling rate;

[0162] Step 3: Time domain downsampling:

[0163] The video sequence is subjected to time domain downsampling according to the time domain sampling configuration information;

[0164] Step 4: Pre-processing stage:

[0165] The video sequence after time domain downsampling enters other pre-processing modules (such as spatial domain modules, ROI modules, etc.) for further processing;

[0166] Step 5: Generating encoding code stream:

[0167] After the pre-processing is completed, the video sequence enters the encoder to generate the code stream, and the time domain sampling configuration information is attached as the header information to the code stream for transmission;

[0168] Step 6: Decoding and extracting sampling configuration information:

[0169] The decoding end receives the code stream file, reads the header information to extract the time domain sampling configuration information, and decodes the code stream file through the decoder to generate the video sequence;

[0170] Step 7: Post-processing stage:

[0171] The decoded video sequence enters the post-processing module (including ROI module, spatial domain module, etc.) for processing;

[0172] Step 8: Time domain upsampling:

[0173] The processed video sequence enters the time domain upsampling module, and the video sequence is subjected to upsampling according to the time domain sampling configuration information to restore the original video frame length;

[0174] Step 9: Output video sequence:

[0175] The processed video sequence is output.

[0176] The beneficial effects brought by the above embodiment are as follows:

[0177] 1) The motion feature of the moving target in the video sequence is analyzed to reflect the object motion between the current video frame image and the reference video frame image, and then the difficulty of frame interpolation is reflected, and the balance between the code rate and the reconstruction video quality is considered, and the optimal sampling rate is adaptively selected, so that the code rate allocation is more reasonable, and the compression efficiency of video encoding and the reconstruction quality of the decoded video can be effectively improved.

[0178] 2) It can be applied to machine vision video coding tasks to dynamically adjust the sampling rate in the implementation of the machine vision video coding process, analyze the motion characteristics and target size of the foreground target in the video frame, effectively exclude background interference, and adaptively select the optimal sampling rate based on the video content, making it more suitable for machine vision video coding systems.

[0179] 3) A time domain sampling rate dataset is constructed, containing 800 video segments, with the motion characteristics and target size of the foreground target as the feature vector, and the optimal time domain sampling rate as the label value, providing basic data for training the time domain sampling rate classification model.

[0180] 4) The optimal sampling rate of each video sequence segment in the video sequence is determined by using the double pointer method to process the video sequence.

[0181] It should be understood that although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0182] Based on the same inventive concept, the embodiments of the present application also provide a video sequence time domain sampling processing device for implementing the above-mentioned video sequence time domain sampling processing method. The problem-solving implementation scheme provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more video sequence time domain sampling processing device embodiments provided below can refer to the limitations of the video sequence time domain sampling processing method in the above text, which will not be repeated here.

[0183] In an exemplary embodiment, as shown in Figure 6 a video sequence time domain sampling processing device is provided, comprising:

[0184] The target detection module 601 is configured to extract reference video frame images corresponding to a preset number of frames after the current video frame image from the video sequence to be processed, perform target detection on the current video frame image and the reference video frame image, and generate a first foreground mask image and a second foreground mask image for the moving target.

[0185] The sampling rate prediction module 602 is configured to predict a sampling rate according to the motion feature information of the first foreground mask image and the second foreground mask image by using a sampling rate prediction model, and obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate.

[0186] The loop execution module 603 is configured to determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting the reference video frame image corresponding to the subsequent preset number of frames from the current video frame image in the to-be-processed video sequence until a video sampling stop condition is met, to determine each group of video sequence segments and the corresponding sampling rates of the video sequence.

[0187] The sampling configuration module 604 is configured to obtain time-domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rates.

[0188] In an embodiment, the time-domain sampling processing apparatus of the video sequence further comprises a feature extraction module configured to obtain a peak signal-to-noise ratio, a frame difference, and a target total area of the first foreground mask image and the second foreground mask image between the first foreground mask image and the second foreground mask image; and obtain the motion feature information according to the peak signal-to-noise ratio, the frame difference, and the target total area.

[0189] In an embodiment, the sampling rate prediction module 602 is further configured to input the motion feature information into the sampling rate prediction model to obtain a probability distribution of a plurality of candidate sampling rates, and identify a candidate sampling rate with the maximum probability as the sampling rate according to the probability distribution.

[0190] In an embodiment, the sampling rate prediction module 602 is further configured to input the motion feature information into the sampling rate prediction model, perform linear transformation on the motion feature information by using a classifier in the sampling rate prediction model, and obtain scores corresponding to the plurality of candidate sampling rates; and convert the scores into a probability distribution corresponding to each candidate sampling rate by using a conversion function in the classifier.

[0191] In an embodiment, the time-domain sampling processing apparatus of the video sequence further comprises a model training module configured to split a sampling rate data set into a training set and a test set; train an initial classifier by using the training set and update model parameters of the initial classifier until an accuracy of the classifier after the parameter update meets a threshold condition on the test set, to obtain the sampling rate prediction model comprising the classifier.

[0192] In an embodiment, the time domain sampling processing apparatus of the video sequence further comprises a dataset construction module configured to obtain a video sequence sample set containing multiple resolutions, extract a plurality of continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set, input each video sequence segment sample in the video sequence segment sample set into the machine vision video coding system at multiple sampling rates, and count video coding index evaluation values and target detection index evaluation values corresponding to the multiple sampling rates; for each video sequence segment sample, determine a target sampling rate from the multiple sampling rates according to the video coding index evaluation values and the target detection index evaluation values; and obtain a sampling rate dataset according to the motion feature information of each video sequence segment sample in the video sequence segment sample set and the target sampling rate of each video sequence segment sample.

[0193] In an embodiment, the sampling configuration module 604 is further configured to form a sampling rate pair by combining a video sequence segment and a corresponding sampling rate, arrange the sampling rate pairs in a sequence according to the sequence of the video sequence segments to form a sampling rate array, and obtain the time domain sampling configuration information according to the sampling rate array.

[0194] In an embodiment, the time domain sampling processing apparatus of the video sequence further comprises a sampling processing module configured to input a sampled video sequence obtained by performing time domain downsampling on the video sequence according to the time domain sampling configuration information into an encoder to obtain a bitstream output by the encoder, attach the time domain sampling configuration information as header information to the bitstream to form a bitstream file, and transmit the bitstream file to a decoding end, wherein the bitstream file is used to extract the time domain sampling configuration information and decode the sampled video sequence at the decoding end, so as to recover the video sequence according to the time domain sampling configuration information and the sampled video sequence.

[0195] The above-mentioned modules in the time domain sampling processing apparatus of the video sequence can be realized by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in the computer device in a software form, so as to be called and executed by a processor to perform the operations corresponding to the above-mentioned modules.

[0196] In an exemplary embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 7The computer device shown in the figure includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (NFC) or other technologies. The computer program is executed by the processor to realize a time domain sampling processing method of a video sequence. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0197] Those skilled in the art can understand that, Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0198] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps in each of the above method embodiments.

[0199] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0200] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0201] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0202] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0203] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0204] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method of temporal sampling of a video sequence, characterized in that, The method comprises: extracting reference video frame images corresponding to a preset number of frames behind a current video frame image from a video sequence to be processed, performing target detection on the current video frame image and the reference video frame images to generate a first foreground mask image and a second foreground mask image for a moving target; obtaining a peak signal-to-noise ratio, a frame difference and a target total area of the first foreground mask image and the second foreground mask image between the first foreground mask image and the second foreground mask image, obtaining motion feature information of the first foreground mask image and the second foreground mask image according to the peak signal-to-noise ratio, the frame difference and the target total area, predicting a sampling rate by a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image, and obtaining a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate; determining a new current video frame image according to the video sequence segment, and returning to execute the step of extracting reference video frame images corresponding to a preset number of frames behind a current video frame image from a video sequence to be processed until a video sampling stop condition is met, to determine each group of video sequence segments of the video sequence and the corresponding sampling rate; obtaining time domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rate.

2. The method of claim 1, wherein, The method further comprises: inputting the motion feature information into the sampling rate prediction model for prediction to obtain a probability distribution of a plurality of candidate sampling rates; identifying a candidate sampling rate with the largest probability as the sampling rate according to the probability distribution.

3. The method of claim 2, wherein, The method further comprises: inputting the motion feature information into the sampling rate prediction model, performing linear transformation on the motion feature information by a classifier in the sampling rate prediction model to obtain scores corresponding to the plurality of candidate sampling rates, and converting the scores into probability distributions corresponding to the plurality of candidate sampling rates by a conversion function in the classifier. The method further comprises:

4. The method of claim 3, wherein, splitting a sampling rate data set into a training set and a test set; training an initial classifier using the training set and updating model parameters of the initial classifier until the accuracy of the classifier after parameter updating on the test set meets a threshold condition to obtain the sampling rate prediction model containing the classifier. Before splitting the sampling rate data set into the training set and the test set, the method further comprises:

5. The method of claim 4, wherein, obtaining a video sequence sample set containing a plurality of resolutions, extracting a plurality of continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set; inputting each video sequence segment sample in the video sequence segment sample set into a machine vision video coding system at a plurality of sampling rates, and statistically obtaining video coding index evaluation values and target detection index evaluation values corresponding to various sampling rates. The method further comprises: splitting a sampling rate data set into a training set and a test set; training an initial classifier using the training set and updating model parameters of the initial classifier until the accuracy of the classifier after parameter updating on the test set meets a threshold condition to obtain the sampling rate prediction model containing the classifier. Before splitting the sampling rate data set into the training set and the test set, the method further comprises: obtaining a video sequence sample set containing a plurality of resolutions, extracting a plurality of continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set; inputting each video sequence segment sample in the video sequence segment sample set into a machine vision video coding system at a plurality of sampling rates, and statistically obtaining video coding index evaluation values and target detection index evaluation values corresponding to various sampling rates. For each of the video sequence segment samples, a target sampling rate is determined from the video coding index evaluation value and the target detection index evaluation value among the plurality of sampling rates; According to the motion feature information of each of the video sequence segment samples in the video sequence segment sample set and the target sampling rate of each of the video sequence segment samples, the sampling rate data set is obtained.

6. The method of claim 1, wherein, The time domain sampling configuration information of the video sequence is obtained according to each group of the video sequence segments and the corresponding sampling rates, and the time domain sampling configuration information includes: The video sequence segment and the corresponding sampling rate form a sampling rate pair; The sampling rate pairs are arranged in the order of the video sequence segments to form a sampling rate array; The time domain sampling configuration information is obtained according to the sampling rate array.

7. The method according to any one of claims 1 to 6, characterized in that, After the time domain sampling configuration information of the video sequence is obtained according to each group of the video sequence segments and the corresponding sampling rates, the method further includes: The sampled video sequence obtained by performing time domain downsampling on the video sequence according to the time domain sampling configuration information is transmitted to an encoder for encoding to obtain a bitstream output by the encoder; The time domain sampling configuration information is attached to the bitstream as header information to form a bitstream file; The bitstream file is transmitted to a decoding end; the bitstream file is used to extract the time domain sampling configuration information and decode the sampled video sequence at the decoding end, so as to recover the video sequence according to the time domain sampling configuration information and the sampled video sequence.

8. An apparatus for temporal sampling of a video sequence, characterized in that The device includes: A target detection module configured to extract reference video frame images corresponding to a preset number of frames after a current video frame image from a video sequence to be processed, perform target detection on the current video frame image and the reference video frame images, and generate first and second foreground mask images for a moving target; A sampling rate prediction module configured to predict a sampling rate by a sampling rate prediction model according to motion feature information of the first and second foreground mask images, and obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate; the motion feature information is obtained by: obtaining a peak signal-to-noise ratio, a frame difference, and a target total area of the first and second foreground mask images between the first and second foreground mask images; and obtaining the motion feature information according to the peak signal-to-noise ratio, the frame difference, and the target total area; A loop execution module configured to determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting reference video frame images corresponding to a preset number of frames after a current video frame image from a video sequence to be processed until a video sampling stop condition is met, so as to determine each group of the video sequence segments and the corresponding sampling rates of the video sequence; A sampling configuration module configured to obtain time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program. The processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which when executed by a processor, implements the steps of the method of any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by a processor, implements the steps of the method of any one of claims 1 to 7.

Images