Time domain sampling processing method and device of video sequence, computer equipment, readable storage medium and program product

Abstract

The invention relates to a time domain sampling processing method and device of a video sequence, equipment, a medium and a program product, and relates to the technical field of artificial intelligence and video coding. The method can improve the flexibility of sampling rate selection of the video sequence. The method comprises the following steps: extracting a reference video frame image from a video sequence, performing target detection on a current video frame image and the reference video frame image, and generating a first foreground mask image and a second foreground mask image; predicting through a sampling rate prediction model according to the motion feature information to obtain a sampling rate, and obtaining a video sequence segment according to the current video frame image and the sampling rate; determining a new current video frame image according to the video sequence segments, skipping to the step of extracting reference video frame images of a rear preset frame number corresponding to the current video frame image from the to-be-processed video sequence until a video sampling stop condition is met, and determining each video sequence segment and a sampling rate; and obtaining time domain sampling configuration information of the video sequence according to each video sequence segment and the sampling rate.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and video coding technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for time-domain sampling processing of a video sequence. Background Art

[0002] Video coding can reduce the amount of data by compressing the redundant information in the video while maintaining the video quality as much as possible, thereby reducing storage and transmission costs. Time domain processing can effectively reduce the storage and transmission costs of video data by removing redundant information between frames in video coding.

[0003] In the pre-processing of the video encoding end, time domain downsampling can be used to discard redundant frames to reduce the number of video frames. In the post-processing of the video decoding end, time domain upsampling can be used to restore the original number of frames. Among them, the time domain sampling rate has an important impact on the bit rate and quality balance of video encoding.

[0004] In traditional pre-coding processing, a fixed sampling rate is usually used to sample the video sequence, which has the problem of insufficient flexibility and is difficult to apply to coding tasks such as machine vision. Summary of the invention

[0005] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium and computer program product for time domain sampling processing of a video sequence in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for time domain sampling processing of a video sequence, comprising:

[0007] Extracting a reference video frame image corresponding to a current video frame image and a preset number of frames later from a video sequence to be processed, performing target detection on the current video frame image and the reference video frame image, and generating a first foreground mask image and a second foreground mask image for a moving target;

[0008] According to the motion feature information of the first foreground mask image and the second foreground mask image, a sampling rate is predicted by a sampling rate prediction model, and according to the current video frame image and the sampling rate, a corresponding video sequence segment in the video sequence is obtained;

[0009] Determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting a reference video frame image corresponding to the current video frame image and a preset number of frames later from the video sequence to be processed until a video sampling stop condition is met, and determine each group of the video sequence segments of the video sequence and the corresponding sampling rate;

[0010] According to each group of the video sequence segments and the corresponding sampling rates, time domain sampling configuration information of the video sequence is obtained.

[0011] In one of the embodiments, before predicting the sampling rate using a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image, the method further includes:

[0012] Acquire a peak signal-to-noise ratio, a frame difference, and a total target area of ​​the first foreground mask image and the second foreground mask image between the first foreground mask image and the second foreground mask image; and obtain the motion feature information according to the peak signal-to-noise ratio, the frame difference, and the total target area.

[0013] In one of the embodiments, the step of predicting the sampling rate using a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image comprises:

[0014] The motion feature information is input into the sampling rate prediction model for prediction to obtain a probability distribution of multiple candidate sampling rates; based on the probability distribution, the candidate sampling rate with the highest probability is identified as the sampling rate.

[0015] In one embodiment, the motion feature information is input into the sampling rate prediction model for prediction to obtain a probability distribution of multiple candidate sampling rates, including:

[0016] The motion feature information is input into the sampling rate prediction model, and the classifier in the sampling rate prediction model performs a linear transformation on the motion feature information to obtain scores corresponding to multiple candidate sampling rates; and the conversion function in the classifier is used to convert the scores into probability distributions corresponding to each candidate sampling rate.

[0017] In one embodiment, the method further comprises:

[0018] The sampling rate data set is split into a training set and a test set; the initial classifier is trained using the training set and the model parameters of the initial classifier are updated until the accuracy of the classifier after the parameter update on the test set meets the threshold condition, thereby obtaining the sampling rate prediction model containing the classifier.

[0019] In one embodiment, before splitting the sampling rate data set into a training set and a test set, the method further includes:

[0020] A video sequence sample set containing multiple resolutions is obtained, and multiple continuous video frame image samples are extracted from each video sequence sample in the video sequence sample set to form a video sequence segment sample set; each video sequence segment sample in the video sequence segment sample set is input into a machine vision video coding system at multiple sampling rates, and video coding index evaluation values ​​and target detection index evaluation values ​​corresponding to various sampling rates are counted; for each video sequence segment sample, a target sampling rate is determined from the multiple sampling rates according to the video coding index evaluation value and the target detection index evaluation value; the sampling rate data set is obtained according to the motion feature information of each video sequence segment sample in the video sequence segment sample set and the target sampling rate of each video sequence segment sample.

[0021] In one embodiment, obtaining the time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates includes:

[0022] The video sequence segments and the corresponding sampling rates form a sampling rate pair; the sampling rate pairs are arranged in the order of the video sequence segments to form a sampling rate array; and the time domain sampling configuration information is obtained according to the sampling rate array.

[0023] After obtaining the time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates, the method further includes:

[0024] The sampled video sequence obtained by time-domain downsampling the video sequence according to the time-domain sampling configuration information is input into an encoder for encoding to obtain a bitstream output by the encoder; the time-domain sampling configuration information is attached to the bitstream as header information to form a bitstream file; the bitstream file is transmitted to a decoding end; the bitstream file is used by the decoding end to extract the time-domain sampling configuration information and decode to obtain the sampled video sequence, so as to restore the video sequence according to the time-domain sampling configuration information and the sampled video sequence.

[0025] In a second aspect, the present application further provides a time domain sampling processing device for a video sequence, comprising:

[0026] The target detection module is used to extract the reference video frame image corresponding to the current video frame image and the reference video frame image from the video sequence to be processed, perform target detection on the current video frame image and the reference video frame image, and generate a first foreground mask image and a second foreground mask image for the moving target;

[0027] A sampling rate prediction module, configured to predict a sampling rate through a sampling rate prediction model according to motion feature information of the first foreground mask image and the second foreground mask image, and obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate;

[0028] a loop execution module, configured to determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting a reference video frame image corresponding to the current video frame image and a preset number of frames later from the video sequence to be processed until a video sampling stop condition is met, and determine each group of the video sequence segments of the video sequence and the corresponding sampling rate;

[0029] The sampling configuration module is used to obtain the time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] A reference video frame image corresponding to a preset number of frames behind the current video frame image is extracted from the video sequence to be processed, target detection is performed on the current video frame image and the reference video frame image, and a first foreground mask image and a second foreground mask image for a moving target are generated; a sampling rate is predicted by a sampling rate prediction model according to motion feature information of the first foreground mask image and the second foreground mask image, and a corresponding video sequence segment in the video sequence is obtained according to the current video frame image and the sampling rate; a new current video frame image is determined according to the video sequence segment, and the step of extracting a reference video frame image corresponding to a preset number of frames behind the current video frame image from the video sequence to be processed is returned to execute until a video sampling stop condition is met, and each group of the video sequence segments of the video sequence and the corresponding sampling rate are determined; and time domain sampling configuration information of the video sequence is obtained according to each group of the video sequence segments and the corresponding sampling rate.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] A reference video frame image corresponding to a preset number of frames behind the current video frame image is extracted from the video sequence to be processed, target detection is performed on the current video frame image and the reference video frame image, and a first foreground mask image and a second foreground mask image for a moving target are generated; a sampling rate is predicted by a sampling rate prediction model according to motion feature information of the first foreground mask image and the second foreground mask image, and a corresponding video sequence segment in the video sequence is obtained according to the current video frame image and the sampling rate; a new current video frame image is determined according to the video sequence segment, and the step of extracting a reference video frame image corresponding to a preset number of frames behind the current video frame image from the video sequence to be processed is returned to execute until a video sampling stop condition is met, and each group of the video sequence segments of the video sequence and the corresponding sampling rate are determined; and time domain sampling configuration information of the video sequence is obtained according to each group of the video sequence segments and the corresponding sampling rate.

[0034] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0035] A reference video frame image corresponding to a preset number of frames behind the current video frame image is extracted from the video sequence to be processed, target detection is performed on the current video frame image and the reference video frame image, and a first foreground mask image and a second foreground mask image for a moving target are generated; a sampling rate is predicted by a sampling rate prediction model according to motion feature information of the first foreground mask image and the second foreground mask image, and a corresponding video sequence segment in the video sequence is obtained according to the current video frame image and the sampling rate; a new current video frame image is determined according to the video sequence segment, and the step of extracting a reference video frame image corresponding to a preset number of frames behind the current video frame image from the video sequence to be processed is returned to execute until a video sampling stop condition is met, and each group of the video sequence segments of the video sequence and the corresponding sampling rate are determined; and time domain sampling configuration information of the video sequence is obtained according to each group of the video sequence segments and the corresponding sampling rate.

[0036] The time domain sampling processing method, apparatus, computer equipment, computer-readable storage medium and computer program product of the above-mentioned video sequence can reflect the object motion between the current video frame image and the corresponding reference video frame image by analyzing the motion characteristics of the moving targets in the video clips in the video sequence, and obtain the sampling rate corresponding to each video clip by adaptively predicting the motion feature information through the sampling rate prediction model, thereby flexibly determining the sampling rate of each video clip in the video sequence, improving the flexibility of the sampling rate selection of the video sequence, so that it can be flexibly applied to, for example, encoding tasks for machine vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A diagram of an application environment of a method for processing time-domain sampling of a video sequence in one embodiment;

[0039] Figure 2 Schematic diagram of a flow chart of a method for time-domain sampling processing of a video sequence in one embodiment;

[0040] Figure 3 is a schematic flow chart of a method for processing time-domain sampling of a video sequence in another embodiment;

[0041] Figure 4 A schematic diagram of a flow chart of steps for constructing a sampling rate data set in one embodiment;

[0042] Figure 5 It is a flowchart of a method for time-domain sampling processing of a video sequence in a specific embodiment;

[0043] Figure 6 is a structural block diagram of a device for processing time-domain sampling of a video sequence in one embodiment;

[0044] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] Glossary of terms related to this application:

[0047] Temporal sampling rate: refers to the sampling of video frames at a specific sampling rate in the temporal adaptive sampling module of video coding. For example, a sampling rate of 4 means moving on the video sequence with a window of 4 frames and a step size of 4, sampling the first frame in each window, and splicing all sampled frames to generate a temporal downsampled video sequence. It should be noted that a sampling rate of 0 means no temporal downsampling.

[0048] Machine vision video coding: refers to the video coding method designed for machine vision tasks (such as target detection, object tracking, etc.). The decoded video is mainly used for analysis and processing of machine vision tasks.

[0049] In video coding, the temporal sampling rate has an important impact on the balance between the bit rate and quality of video coding. For video sequences with complex motion, since it is difficult to interpolate frames at the decoding end, a lower sampling rate should be used to ensure the interpolation effect; while for video sequences with simple motion, since it is relatively easy to interpolate frames at the decoding end, a higher sampling rate can be used to improve compression performance.

[0050] In traditional pre-coding processing, a fixed sampling rate is usually used to sample the video sequence, for example:

[0051] 1. Fixed temporal sampling rate: A fixed temporal sampling rate (such as a sampling rate of 4) is always used, regardless of the type of video sequence or the specific requirements of the encoding task. For example, the sampling rate method used in machine-oriented video coding VCM is used.

[0052] 2. Sampling rate selection based on decoding task orientation: In video encoding tasks dominated by human vision, a lower sampling rate is usually selected to ensure high-fidelity effects of reconstructed video frames; while in encoding tasks dominated by machine vision, a higher sampling rate is usually selected to achieve low latency and high compression ratio in complex scenarios.

[0053] 3. Dynamic temporal sampling rate selection: The perceptual quality difference between the temporal downsampled video frames and the original full-frame rate video frames is evaluated through the frame rate-related quality metric FRQM. When FRQM exceeds the set threshold, a specific sampling rate is selected for downsampling; otherwise, no sampling operation is performed.

[0054] Therefore, there is a problem of insufficient flexibility in traditional pre-coding processing. Among them, the first two sampling rate selection methods use a fixed sampling rate in the entire video sequence, but a single video sequence often contains segments with diverse motion patterns. It is obviously unreasonable to use the same sampling rate in segments with different motion complexity, which will lead to reduced compression performance or damaged reconstructed video quality. Although the third method can dynamically choose whether to sample according to the video content, it can only select a single sampling rate and cannot achieve flexible adjustment of multiple sampling rates, so the flexibility is still insufficient. Moreover, traditional pre-coding processing lacks adaptability to machine vision video coding. Among them, in machine vision-oriented coding tasks, the coding requirements of foreground targets are more critical. The above methods do not consider the impact of foreground targets on coding performance and lack adaptability to machine vision tasks.

[0055] In this regard, the time domain sampling processing method of the video sequence of the present application can flexibly determine the sampling rate of each video segment in the video sequence, thereby improving the flexibility of selecting the sampling rate of the video sequence, so that it can be flexibly applied to, for example, coding tasks for machine vision (such as target detection, object tracking, etc.).

[0056] The time domain sampling processing method of the video sequence provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server, or placed on the cloud or other network servers. The terminal can be, but is not limited to, various personal computers, laptops, smart phones and tablets. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0057] In one embodiment, Figure 2 As shown, a method for time domain sampling processing of a video sequence is provided, and the method is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0058] Step S201, extracting a reference video frame image corresponding to a current video frame image and a preset number of frames later from a video sequence to be processed, performing target detection on the current video frame image and the reference video frame image, and generating a first foreground mask image and a second foreground mask image for a moving target.

[0059] Among them, target detection is a task in the time domain sampling process, which aims to identify all objects of interest in the current video frame image and the reference video frame image and accurately locate them.

[0060] Specifically, Figure 3 As shown, the terminal uses a double pointer method to extract the reference video frame image corresponding to the current video frame image from the video sequence to be processed, and performs target detection on the current video frame image and the reference video frame image. The Faster-RCNN model (Region-based Convolutional Neural Network, a deep learning model for target detection) is used to obtain the target detection frame of each video frame image, and a first foreground mask image and a second foreground mask image for the moving target are generated based on the detected moving target. In this embodiment, the foreground area is retained and the background is covered to achieve background removal. The purpose of this is because the emphasis in machine vision video coding is on the coding of moving targets, and the decoded video is also used for machine vision task analysis.

[0061] Step S202: predicting a sampling rate through a sampling rate prediction model according to motion feature information of the first foreground mask image and the second foreground mask image, and obtaining a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate.

[0062] The motion feature information may include relevant indicators such as peak signal-to-noise ratio, frame difference, and total target area.

[0063] In order to achieve the goal of dynamically adjusting the temporal sampling rate with the video content, the present application uses a dual pointer method to process video frame images in a video sequence. For segments with complex motion patterns, a low sampling rate is used; for segments with simple motion patterns, a high sampling rate is used, thereby ensuring the quality of video reconstruction while allocating the bit rate more reasonably.

[0064] like Figure 3 As shown, the specific process is as follows: define two pointers and , corresponding to the 0th and 4th frames of the video respectively. Read pointer and Corresponding image and , according to the image and The peak signal-to-noise ratio, frame difference, total target area and other related indicators are used to calculate the optimal sampling rate through the sampling rate prediction model using the time domain adaptive sampling method. Then, according to the sampling rate Adjust the pointer Position, move the pointer move -4 positions (positive value means moving backward, negative value means moving forward), record the sampling rate of the video sequence segment, according to the pointer and sampling rate to obtain the corresponding video sequence segment in the video sequence (the first frame is numbered ).

[0065] Step S203, determining a new current video frame image according to the video sequence segment, and returning to execute the step of extracting a reference video frame image corresponding to the current video frame image and a preset number of frames later from the video sequence to be processed, until the video sampling stop condition is met, and determining each group of video sequence segments of the video sequence and the corresponding sampling rate.

[0066] The video sampling stop condition can be set by relevant technicians according to actual needs, for example: When the video sequence length is exceeded, video sampling stops.

[0067] The above-mentioned “return execution” refers to looping execution of step S201, step S202 and “determining a new current video frame image according to the video sequence segment” in step S203.

[0068] Specifically, after entering the next iteration, the terminal determines a new current video frame image according to the video sequence segment, that is, the pointer Assign the position to the pointer , and reposition the pointer Defined as +4, repeat the above steps until the pointer The length of the video sequence is exceeded, and each group of video sequence segments of the video sequence and the corresponding sampling rates are finally determined.

[0069] Step S204: obtaining time domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rates.

[0070] Specifically, the terminal sets the sampling rate Recorded in the sampling rate array, the data structure is the sampling rate pair corresponding to each sampling position [ ], the sampling rate pair represents the video sequence segment (the first frame is ) has a sampling rate of , multiple groups of sampling rate pairs are combined to obtain the time domain sampling configuration information of the video sequence.

[0071] In the time domain sampling processing method of the above-mentioned video sequence, the motion characteristics of the moving targets in the video clips in the video sequence can be analyzed to reflect the object motion between the current video frame image and the corresponding reference video frame image. The sampling rate prediction model can adaptively predict the sampling rate corresponding to each video clip according to the motion feature information. In this way, the sampling rate of each video clip in the video sequence can be flexibly determined, thereby improving the flexibility of the sampling rate selection of the video sequence, so that it can be flexibly applied to, for example, encoding tasks for machine vision.

[0072] In one embodiment, before predicting the sampling rate through a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image, the method of the present application further includes the following steps:

[0073] The peak signal-to-noise ratio, frame difference and total target area of ​​the first foreground mask image and the second foreground mask image are obtained; and motion feature information is obtained according to the peak signal-to-noise ratio, frame difference and total target area.

[0074] In the above step S202, the sampling rate is predicted by a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image, which specifically includes the following steps:

[0075] The motion feature information is input into the sampling rate prediction model, and the classifier in the sampling rate prediction model performs a linear transformation on the motion feature information to obtain scores corresponding to multiple candidate sampling rates; the conversion function in the classifier is used to convert the scores into probability distributions corresponding to each candidate sampling rate; based on the probability distribution, the candidate sampling rate with the largest probability is identified as the sampling rate.

[0076] The candidate sampling rates may be [0, 2, 4, 8], the classifier in the sampling rate prediction model may be a trained Softmax regression model, and the conversion function in the classifier may be a Softmax function.

[0077] Specifically, to select a suitable sampling rate, it is necessary to calculate the peak signal-to-noise ratio and frame difference of two frames of masked images. and the total target area of ​​the two frames , these indicators are used to evaluate the motion characteristics and target size of objects. The peak signal-to-noise ratio is mainly used to measure the similarity between two frames of images. The larger the peak signal-to-noise ratio, the smaller the difference between the two frames, indicating that the motion or content changes are smaller, and a higher sampling rate can be considered. Conversely, the smaller the peak signal-to-noise ratio, the greater the difference between frames, indicating that there is strong motion or complex content changes, and a lower sampling rate may be required to retain more frames. The calculation formula for PSNR (PeakSignal-to-Noise Ratio) is:

[0078] (1)

[0079] Frame Difference It is the absolute difference of pixel values ​​between two frames, which is used to describe the motion information of the image. The larger the frame difference, the more significant the motion or object displacement between the two frames. Frame difference can help detect the significance and degree of inter-frame motion, which is very important for determining the temporal sampling rate. When the frame difference is large, appropriately reducing the sampling rate can help retain the motion information. The calculation formula of frame difference is as follows:

[0080] (2)

[0081] Target total area It reflects the size of the foreground object. In machine vision video encoding, the encoding of foreground objects usually takes precedence over the background. The larger the foreground object, the higher the encoding and storage cost. In order to achieve efficient encoding performance, when the foreground object is large, it may be necessary to select a higher sampling rate to reduce the storage and transmission burden by discarding more frames.

[0082] It should be noted that this application focuses on video coding for machine vision tasks (such as target detection, object tracking, etc.), that is, a coding method designed to adapt to machine vision applications. In this method, the coding process focuses more on the recognition and retention of foreground targets. Therefore, the selection of the time domain sampling rate in this application mainly focuses on factors such as the motion complexity and target size of the foreground target in the video frame.

[0083] After completing the calculation of the above three indicators, the eigenvector , the specific form is as follows:

[0084] (3)

[0085] All three indicators have a significant correlation with the sampling rate, PSNR and It is positively correlated with the sampling rate. It is negatively correlated with the sampling rate. When the number of features is small and there is an obvious linear relationship between the features and the labels, we use the Softmax regression model as the classifier and input the feature vector , the model maps the input to the probability distribution of multiple categories through linear transformation and Softmax function. In this embodiment, different categories correspond to different sampling rates [0, 2, 4, 8]. The linear transformation process in Softmax regression is as follows:

[0086] (4)

[0087] in, is the weight vector for category k, is the bias term for class k, is represented as the score of category k. Next, the score of each category is Input into the softmax function to get the probability distribution of each category. The expression of the softmax function is:

[0088] (5)

[0089] in, Represents the feature vector The probability of belonging to category k. Finally, the model selects the category with the highest probability as the sampling rate prediction result. In this embodiment, through the Softmax regression model, the three input features are linearly transformed and the Softmax function is used to obtain the probability distribution corresponding to the sampling rate [0, 2, 4, 8], and the sampling rate with the highest probability is selected as the final prediction result.

[0090] It should be noted that the Softmax regression model is trained on the time domain sampling rate dataset. The cross entropy loss function is used during the model training process. By minimizing the loss, the model can continuously adjust the weights. and bias , so that the predicted probability distribution is as close as possible to the actual category distribution. Finally, the prediction accuracy of the model on the test set is as high as more than 90%, and the model is output. In addition, the model parameters need to be frozen when used in the time domain adaptive sampling method.

[0091] In this embodiment, the motion characteristics and target size of the object are evaluated by analyzing indicators such as the peak signal-to-noise ratio, frame difference, and total target area of ​​the first foreground mask image and the second foreground mask image, thereby obtaining motion feature information. The motion feature information is then linearly transformed and scored through a sampling rate prediction model to effectively obtain the probability distribution corresponding to each candidate sampling rate. At the same time, considering the balance between bit rate and reconstructed video quality, the candidate sampling rate with the highest probability is adaptively selected as the optimal sampling rate, thereby realizing intelligent adjustment of the sampling rate, significantly optimizing bit rate allocation, and effectively improving the flexibility of sampling rate selection and video encoding performance.

[0092] In one embodiment, the method of the present application further includes the following steps:

[0093] The sampling rate data set is split into a training set and a test set; the initial classifier is trained using the training set and the model parameters of the initial classifier are updated until the accuracy of the classifier after the parameter update on the test set meets the threshold condition, thereby obtaining a sampling rate prediction model containing the classifier.

[0094] Among them, the threshold condition can be set by relevant technical personnel according to actual needs, for example: the accuracy of the classifier on the test set exceeds 90%.

[0095] Specifically, the terminal prepares a time domain sampling rate data set and splits it into a training set and a test set at a ratio of 8:2; constructs a Softmax regression model as the initial classifier; uses the training set to train the initial classifier, and uses the gradient descent optimization algorithm to update the model parameters of the classifier to improve the classification accuracy; after each round of training, the test set is used to evaluate the model performance of the classifier. If the accuracy of the classifier on the test set does not reach 90%, adjust the model parameters of the classifier and repeat the above until the accuracy of the classifier exceeds 90%. When the classifier reaches an accuracy of more than 90% on the test set, freeze the model parameters of the classifier, obtain the sampling rate prediction model containing the classifier, and apply it to the time domain adaptive sampling method for inference prediction.

[0096] In this embodiment, the initial classifier is trained using the training set, and the performance of the trained classifier is evaluated using the test set to fit the optimal model parameters of the classifier, thereby effectively improving the accuracy of the sampling rate prediction model.

[0097] In one embodiment, if Figure 4 As shown, before splitting the sampling rate data set into a training set and a test set, the method of the present application further includes the following steps:

[0098] Step S401, obtaining a video sequence sample set containing multiple resolutions, extracting a plurality of continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set.

[0099] Step S402: input each video sequence segment sample in the video sequence segment sample set into a machine vision video coding system at a plurality of sampling rates, and count the video coding index evaluation values ​​and target detection index evaluation values ​​corresponding to the various sampling rates.

[0100] Step S403: for each video sequence segment sample, a target sampling rate is determined from a plurality of sampling rates according to the video encoding index evaluation value and the target detection index evaluation value.

[0101] Step S404: obtaining a sampling rate data set according to the motion feature information of each video sequence segment sample in the video sequence segment sample set and the target sampling rate of each video sequence segment sample.

[0102] Among them, the video encoding indicator can be BD-Rate (Bjøntegaard Delta Rate, an indicator used to evaluate the changes in video encoding algorithms between different compression ratios and video quality), and the target detection indicator can be BD-mAP (Bjøntegaard Delta mean Average Precision, an indicator used to evaluate the performance difference between two target detection models at the same or different detection accuracies).

[0103] Specifically, the time domain sampling rate dataset can be constructed based on the BVI-DVC dataset. The BVI-DVC dataset contains video sequences of various resolutions, from high definition to lower resolution. The four common resolutions are A (3840x2176), B (1920x1088), C (960x544) and D (480x272). The BVI-DVC dataset contains a total of 800 video sequences, 200 video sequences at each resolution, covering a variety of dynamic scenes, such as indoor and outdoor scenes, moving objects, static backgrounds, etc., which can truly reflect the video characteristics in practical applications. In this embodiment, 8 consecutive frames of images are randomly extracted from each video sequence in the BVI-DVC dataset to form 800 video clips. Each video clip is input into the machine vision video coding system at 4 different sampling rates [0, 2, 4, 8], and target detection is used as a downstream task. The values ​​of BD-Rate and BD-mAP are counted, and the sampling rate with the lowest BD-Rate and the highest BD-mAP is selected as the label value of the video clip. BD-Rate and BD-mAP are negatively correlated. Selecting this sampling rate can ensure the best compression rate of the video clip and the highest target detection performance. The motion feature information of each video clip is obtained based on the peak signal-to-noise ratio of the first frame and the fifth frame, the frame difference and the total target area After the above data set construction steps, a time domain sampling rate data set containing 800 sample features and corresponding label values ​​is obtained, and is used to train the sampling rate prediction model.

[0104] In this embodiment, a sampling rate dataset is constructed, which includes 800 video clips, takes the motion characteristics and target size of the foreground target as the feature vector, and uses the optimal time domain sampling rate as the label value, providing basic data for training the sampling rate prediction model.

[0105] In one embodiment, in the above step S204, obtaining the time domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rates specifically includes the following steps:

[0106] The video sequence segments and the corresponding sampling rates form sampling rate pairs; the sampling rate pairs are arranged in the order of the video sequence segments to form a sampling rate array; and time domain sampling configuration information is obtained according to the sampling rate array.

[0107] Specifically, the terminal sets the sampling rate Recorded in the sampling rate array, the data structure is the sampling rate pair corresponding to each sampling position [ ], the sampling rate pair represents the video sequence segment (the first frame is ) has a sampling rate of , and then arrange multiple groups of sampling rate pairs in the order of the video sequence segments to form a sampling rate array, and finally obtain the time domain sampling configuration information according to the sampling rate array.

[0108] In this embodiment, sampling rate pairs are formed by video sequence segments and corresponding sampling rates, and multiple groups of sampling rate pairs are arranged in the order of the video sequence segments to form a sampling rate array, thereby facilitating accurate generation of time domain sampling configuration information of the video sequence based on the sampling rate array.

[0109] In one embodiment, after obtaining the time domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rates, the method of the present application further includes the following steps:

[0110] The sampled video sequence obtained after time-domain downsampling of the video sequence according to the time-domain sampling configuration information is input into the encoder for encoding to obtain the code stream output by the encoder; the time-domain sampling configuration information is attached to the code stream as header information to form a code stream file; the code stream file is transmitted to the decoding end; the code stream file is used by the decoding end to extract the time-domain sampling configuration information and decode to obtain the sampled video sequence, so as to restore the video sequence according to the time-domain sampling configuration information and the sampled video sequence.

[0111] Specifically, the terminal will perform time-domain downsampling on the video sequence according to the time-domain sampling configuration information to obtain a sampled video sequence, and the sampled video sequence will enter other pre-processing modules (such as a spatial domain module, an ROI module, etc.) for further processing; after the pre-processing is completed, the video sequence enters the encoder for encoding to generate a bitstream, and at the same time, the time-domain sampling configuration information is attached to the bitstream as header information to form a bitstream file, and the bitstream file is transmitted to the decoding end; the decoding end receives the bitstream file, reads the header information to extract the time-domain sampling configuration information, and decodes the bitstream file through the decoder to generate a video sequence; then the video sequence is upsampled in the time domain according to the time-domain sampling configuration information to restore the original video frame length;

[0112] In this embodiment, the encoder encodes the sampled video sequence obtained after time domain downsampling to obtain a bitstream, and integrates the time domain sampling configuration information to generate a bitstream file, which can be used for decoding at the decoding end to restore the original sequence.

[0113] As an application embodiment, the machine vision video encoding system is divided into an encoding end and a decoding end. At the encoding end, the input video sequence enters the time domain adaptive sampling module and other pre-processing modules in sequence. The time domain adaptive sampling module guides the time domain down-sampling module to adaptively down-sample the video sequence based on the sampling information obtained by video content analysis, and writes the sampling information into the bitstream as header information. At the decoding end, the sampling frame and the time domain sampling configuration information are decoded from the bitstream. The sampling frame enters the post-processing module for restoration, and the time domain sampling configuration information is used to guide the time domain up-sampling module to restore the original video sequence length. Figure 5 As shown, a time domain sampling processing method for a video sequence in a specific embodiment is provided, which specifically includes the following steps:

[0114] First, the steps for constructing the time domain sampling rate dataset are as follows:

[0115] Step 1. Download and prepare the dataset:

[0116] Download and prepare 800 video sequences from the BVI-DVC dataset;

[0117] Step 2: Extract video clips:

[0118] Randomly extract 8 consecutive frames from each video sequence to form 800 video clips;

[0119] Step 3: Input the coding system and calculate the indicators:

[0120] Each video clip is input into the machine vision video coding system at 4 different temporal sampling rates [0, 2, 4, 8], and object detection is used as the downstream task to calculate the BD-Rate and BD-mAP values;

[0121] Step 4: Calculate the label value:

[0122] For each video clip, select the sampling rate with the lowest BD-Rate and the highest BD-mAP as the label value ;

[0123] Step 5. Calculate the eigenvector:

[0124] Calculate the feature vector for each video segment : Based on the 1st and 5th frame images, calculate the PSNR and frame difference and the total target area in the two frames ;

[0125] Step 6: Build the dataset:

[0126] Finally, each video segment contains a feature vector and a tag value , a total of 800 video clips constitute the time domain sampling rate dataset.

[0127] Secondly, the training steps of the classification model are as follows:

[0128] Step 1. Prepare the dataset:

[0129] Prepare a time domain sampling rate dataset and split it into a training set and a test set in a ratio of 8:2;

[0130] Step 2. Define the model:

[0131] Build a Softmax regression model;

[0132] Step 3: Training the model:

[0133] Use the training set to train the model, and use the gradient descent optimization algorithm to update the model parameters to improve classification accuracy;

[0134] Step 4. Evaluate the model:

[0135] After each round of training, the test set is used to evaluate the model performance;

[0136] Step 5: Adjustment and optimization:

[0137] If the model's accuracy on the test set does not reach 90%, adjust the model parameters and repeat steps 3 and 4 above until the model's accuracy exceeds 90%;

[0138] Step 6. Output model:

[0139] When the model reaches an accuracy of more than 90% on the test set, the model parameters are frozen and applied to the time domain adaptive sampling method for inference and prediction.

[0140] Then, the processing of the time domain adaptive sampling module is as follows:

[0141] Step 1. Input video sequence:

[0142] Input a video sequence;

[0143] Step 2. Initialize parameters:

[0144] Defining pointers and , initialized to 0 and 4 respectively; initialize a sampling rate array to record the sampling rates corresponding to different frame positions;

[0145] Step 3: Get the video frame:

[0146] Get the Frame and frame images, respectively. and ;

[0147] Step 4. Calculate the optimal sampling rate:

[0148] right and Perform target detection and generate a mask image showing only the foreground target; calculate PSNR and frame difference and target total area Three eigenvalues, input the eigenvalues ​​into the classification model to predict the optimal sampling rate ;

[0149] Step 5. Record the sampling rate:

[0150] Set the sampling rate Recorded in the sampling rate array, the data structure is the sampling rate pair corresponding to each sampling position [ ];

[0151] Step 6: Update the pointer position:

[0152] Update pointer position: According to the step length -4 is used for offset, and the updated position is assigned to ;Will The pointer is updated to +4;

[0153] Step 7: Repeat until finished:

[0154] Repeat steps 3-6 until the pointer Exceeding the video sequence length;

[0155] Step 8. Output results:

[0156] The output sampling rate array is the output result of the time domain adaptive sampling module.

[0157] In summary, the application steps of the time domain adaptive sampling module in the machine vision video coding system are as follows:

[0158] Step 1. Input video sequence:

[0159] Input a video sequence to be processed;

[0160] Step 2: Generate time domain sampling configuration information:

[0161] The video sequence enters the time domain adaptive sampling module, which determines the current video frame image and the reference video frame image in the video sequence, and based on the current video frame image and the reference video frame image (for example, the first Frame and The motion feature information (object motion features and target size) between the four frames is used to estimate the compression and reconstruction performance of the four video frames, and the optimal sampling rate is output through the sampling rate prediction model. The time domain sampling configuration information corresponding to the video sequence is generated according to the sampling rate.

[0162] Step 3: Downsampling in time domain:

[0163] Performing temporal downsampling on the video sequence according to the temporal sampling configuration information;

[0164] Step 4: Pre-processing stage:

[0165] The video sequence after time domain downsampling enters other pre-processing modules (such as spatial domain module, ROI module, etc.) for further processing;

[0166] Step 5: Generate the encoded bitstream:

[0167] After the pre-processing is completed, the video sequence enters the encoder to generate a bitstream, and the time domain sampling configuration information is attached to the bitstream as header information for transmission;

[0168] Step 6: Decode and extract sampling configuration information:

[0169] The decoding end receives the bitstream file, reads the header information to extract the time domain sampling configuration information, and decodes the bitstream file through the decoder to generate a video sequence;

[0170] Step 7: Post-processing stage:

[0171] The decoded video sequence enters the post-processing module (including ROI module, spatial domain module, etc.) for processing;

[0172] Step 8: Perform time domain upsampling:

[0173] The processed video sequence enters the time domain upsampling module, which upsamples the video sequence according to the time domain sampling configuration information to restore the original video frame length;

[0174] Step 9: Output video sequence:

[0175] Output the processed video sequence.

[0176] The beneficial effects brought by the above embodiments are as follows:

[0177] 1) By analyzing the motion characteristics of moving targets in video sequences, the object motion between the current video frame image and the reference video frame image is reflected, and then the difficulty of frame supplementation is reflected. At the same time, the balance between bit rate and reconstructed video quality is considered, and the optimal sampling rate is adaptively selected to make the bit rate allocation more reasonable, which can effectively improve the compression efficiency of video encoding and the reconstruction quality of decoded video.

[0178] 2) It can be applied to machine vision video coding tasks to dynamically adjust the sampling rate during the machine vision video coding process, analyze the motion characteristics and target size of the foreground target in the video frame, effectively eliminate background interference, and adaptively select the optimal sampling rate based on the video content, making it more suitable for machine vision video coding systems.

[0179] 3) A time domain sampling rate dataset was constructed, which contains 800 video clips, takes the motion characteristics and target size of foreground targets as feature vectors, and uses the optimal time domain sampling rate as the label value, providing basic data for training the time domain sampling rate classification model.

[0180] 4) The optimal sampling rate of each video sequence segment in the video sequence is determined by processing the video sequence using a double pointer method.

[0181] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0182] Based on the same inventive concept, the embodiment of the present application also provides a device for processing time-domain sampling of video sequences for implementing the method for processing time-domain sampling of video sequences involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the device for processing time-domain sampling of one or more video sequences provided below can refer to the limitations of the method for processing time-domain sampling of video sequences above, and will not be repeated here.

[0183] In an exemplary embodiment, Figure 6 As shown, a time domain sampling processing device for a video sequence is provided, comprising:

[0184] The target detection module 601 is used to extract the reference video frame image corresponding to the current video frame image and the reference video frame image from the video sequence to be processed, perform target detection on the current video frame image and the reference video frame image, and generate a first foreground mask image and a second foreground mask image for the moving target;

[0185] A sampling rate prediction module 602 is used to predict the sampling rate through a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image, and obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate;

[0186] A loop execution module 603 is used to determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting a reference video frame image corresponding to the current video frame image and a preset number of frames later from the video sequence to be processed until a video sampling stop condition is met, and determine each group of video sequence segments of the video sequence and the corresponding sampling rate;

[0187] The sampling configuration module 604 is used to obtain the time domain sampling configuration information of the video sequence according to each group of video sequence segments and the corresponding sampling rates.

[0188] In one embodiment, the time domain sampling processing device of the video sequence also includes a feature extraction module, which is used to obtain the peak signal-to-noise ratio, frame difference and total target area between the first foreground mask image and the second foreground mask image; and obtain motion feature information based on the peak signal-to-noise ratio, frame difference and total target area.

[0189] In one embodiment, the sampling rate prediction module 602 is further used to input motion feature information into a sampling rate prediction model for prediction to obtain a probability distribution of multiple candidate sampling rates; based on the probability distribution, identify the candidate sampling rate with the highest probability as the sampling rate.

[0190] In one embodiment, the sampling rate prediction module 602 is also used to input motion feature information into the sampling rate prediction model, and the classifier in the sampling rate prediction model performs a linear transformation on the motion feature information to obtain scores corresponding to multiple candidate sampling rates; and uses the conversion function in the classifier to convert the scores into probability distributions corresponding to each candidate sampling rate.

[0191] In one embodiment, the time domain sampling processing device of the video sequence also includes a model training module, which is used to split the sampling rate data set into a training set and a test set; use the training set to train the initial classifier and update the model parameters of the initial classifier until the accuracy of the classifier after the parameter update on the test set meets the threshold condition, thereby obtaining a sampling rate prediction model containing the classifier.

[0192] In one embodiment, the time domain sampling processing device of a video sequence also includes a data set construction module, which is used to obtain a video sequence sample set containing multiple resolutions, extract multiple continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set; input each video sequence segment sample in the video sequence segment sample set into a machine vision video coding system at multiple sampling rates, and count the video coding index evaluation values ​​and target detection index evaluation values ​​corresponding to various sampling rates; for each video sequence segment sample, determine the target sampling rate from multiple sampling rates according to the video coding index evaluation value and the target detection index evaluation value; and obtain a sampling rate data set according to the motion feature information of each video sequence segment sample in the video sequence segment sample set and the target sampling rate of each video sequence segment sample.

[0193] In one embodiment, the sampling configuration module 604 is further used to form a sampling rate pair with the video sequence segments and the corresponding sampling rates; arrange the sampling rate pairs in the order of the video sequence segments to form a sampling rate array; and obtain time domain sampling configuration information according to the sampling rate array.

[0194] In one embodiment, the time domain sampling processing device of the video sequence also includes a sampling processing module, which is used to pass a sampled video sequence obtained by time domain down-sampling the video sequence according to the time domain sampling configuration information into an encoder for encoding to obtain a code stream output by the encoder; the time domain sampling configuration information is attached to the code stream as header information to form a code stream file; the code stream file is transmitted to the decoding end; the code stream file is used by the decoding end to extract the time domain sampling configuration information and decode to obtain the sampled video sequence, so as to restore the video sequence according to the time domain sampling configuration information and the sampled video sequence.

[0195] Each module in the above-mentioned video sequence time domain sampling processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0196] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a time domain sampling processing method for a video sequence is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0197] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0198] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0199] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0200] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0201] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0202] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0203] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0204] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for time domain sampling processing of a video sequence, characterized in that: The method comprises: Extracting a reference video frame image corresponding to a current video frame image and a preset number of frames later from a video sequence to be processed, performing target detection on the current video frame image and the reference video frame image, and generating a first foreground mask image and a second foreground mask image for a moving target; According to the motion feature information of the first foreground mask image and the second foreground mask image, a sampling rate is predicted by a sampling rate prediction model, and according to the current video frame image and the sampling rate, a corresponding video sequence segment in the video sequence is obtained; Determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting a reference video frame image corresponding to the current video frame image and a preset number of frames later from the video sequence to be processed until a video sampling stop condition is met, and determine each group of the video sequence segments of the video sequence and the corresponding sampling rate; According to each group of the video sequence segments and the corresponding sampling rates, time domain sampling configuration information of the video sequence is obtained.

2. The method according to claim 1, characterized in that Before predicting the sampling rate by a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image, the method further includes: Acquire a peak signal-to-noise ratio, a frame difference, and a total target area of ​​the first foreground mask image and the second foreground mask image between the first foreground mask image and the second foreground mask image; The motion feature information is obtained according to the peak signal-to-noise ratio, the frame difference and the total target area.

3. The method according to claim 1, characterized in that The step of predicting the sampling rate by a sampling rate prediction model according to the motion feature information of the first foreground mask image and the second foreground mask image includes: Inputting the motion feature information into the sampling rate prediction model for prediction to obtain probability distribution of multiple candidate sampling rates; According to the probability distribution, a candidate sampling rate with the highest probability is identified as the sampling rate.

4. The method according to claim 3, characterized in that The motion feature information is input into the sampling rate prediction model for prediction to obtain probability distribution of multiple candidate sampling rates, including: Inputting the motion feature information into the sampling rate prediction model, and using a classifier in the sampling rate prediction model to perform a linear transformation on the motion feature information to obtain scores corresponding to a plurality of candidate sampling rates; The score is converted into a probability distribution corresponding to each of the candidate sampling rates using a conversion function in the classifier.

5. The method according to claim 4, characterized in that The method further comprises: Split the sampling rate dataset into training and testing sets; The training set is used to train the initial classifier and update the model parameters of the initial classifier until the accuracy of the classifier after parameter update on the test set meets the threshold condition, thereby obtaining the sampling rate prediction model including the classifier.

6. The method according to claim 5, characterized in that Before splitting the sampling rate data set into a training set and a test set, the method further includes: Acquire a video sequence sample set containing multiple resolutions, extract a plurality of continuous video frame image samples from each video sequence sample in the video sequence sample set to form a video sequence segment sample set; Inputting each video sequence segment sample in the video sequence segment sample set into a machine vision video coding system at a plurality of sampling rates, and counting video coding index evaluation values ​​and target detection index evaluation values ​​corresponding to the various sampling rates; For each of the video sequence segment samples, determining a target sampling rate from the multiple sampling rates according to the video encoding index evaluation value and the target detection index evaluation value; The sampling rate data set is obtained according to the motion feature information of each of the video sequence segment samples in the video sequence segment sample set and the target sampling rate of each of the video sequence segment samples.

7. The method according to claim 1, characterized in that The obtaining, according to each group of the video sequence segments and the corresponding sampling rates, time domain sampling configuration information of the video sequence includes: Forming a sampling rate pair with the video sequence segment and the corresponding sampling rate; Arranging the sampling rate pairs according to the sequence of the video sequence segments to form a sampling rate array; The time domain sampling configuration information is obtained according to the sampling rate array.

8. The method according to any one of claims 1 to 7, characterized in that: After obtaining the time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates, the method further includes: The sampled video sequence obtained by time-domain downsampling the video sequence according to the time-domain sampling configuration information is input into the encoder for encoding to obtain a bit stream output by the encoder; Adding the time domain sampling configuration information as header information to the bitstream to form a bitstream file; The bitstream file is transmitted to a decoding end; the bitstream file is used by the decoding end to extract the time domain sampling configuration information and decode to obtain the sampled video sequence, so as to restore the video sequence according to the time domain sampling configuration information and the sampled video sequence.

9. A time domain sampling processing device for a video sequence, characterized in that: The device comprises: The target detection module is used to extract the reference video frame image corresponding to the current video frame image and the reference video frame image from the video sequence to be processed, perform target detection on the current video frame image and the reference video frame image, and generate a first foreground mask image and a second foreground mask image for the moving target; A sampling rate prediction module, configured to predict a sampling rate through a sampling rate prediction model according to motion feature information of the first foreground mask image and the second foreground mask image, and obtain a corresponding video sequence segment in the video sequence according to the current video frame image and the sampling rate; a loop execution module, configured to determine a new current video frame image according to the video sequence segment, and return to execute the step of extracting a reference video frame image corresponding to the current video frame image and a preset number of frames later from the video sequence to be processed until a video sampling stop condition is met, and determine each group of the video sequence segments of the video sequence and the corresponding sampling rate; The sampling configuration module is used to obtain the time domain sampling configuration information of the video sequence according to each group of the video sequence segments and the corresponding sampling rates.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images