A sampling method and device of video data, electronic equipment and storage medium

Abstract

The application discloses a video data sampling method and device, electronic equipment and storage medium, the method comprises the following steps: obtaining a video to be sampled; generating a current sliding window according to a preset window length, and sampling data of the video to be sampled according to the current sliding window; when a sampling condition is met, determining the data type of the data frame corresponding to the current sliding window, sliding the window according to the data type of each data frame to obtain a next sliding window, and sampling data, wherein when there is a header class in each data type, the step length of the window sliding is a first current sliding step length, when there is no header class in each data type, the step length of the window sliding is a second current sliding step length, the first current sliding step length is greater than the second current sliding step length; taking the next sliding window as a new current sliding window and returning to the step of determining the data type of the data frame when the sampling condition is met. The problems of long tail and uneven data distribution in the video are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for sampling video data. Background Technology

[0002] The fundamental video tasks that deep learning algorithms focus on mainly include recognition, localization, and segmentation. In different tasks and video datasets, a short video clip can be used as a training sample, labeled; alternatively, a long, unsegmented video can be used, with each long video containing multiple small sample units. During training, a long input segment is matched to each small sample unit, with overlap between inputs from different samples. Real-world datasets often suffer from long-tailed class distribution or uneven data distribution. The mainstream solution is resampling, which involves using upsampling and downsampling strategies to achieve a more balanced distribution of resampled samples. When data distribution is uneven, the model consistently encounters head-class samples, leading to an increase in head-class weights under the supervision of the loss function, thus losing the ability to learn tail-class samples. Resampling can directly eliminate the dominance of head-class samples. However, improper resampling methods can easily lead to overfitting.

[0003] In deep learning, image data processing for upsampling algorithms primarily involves three approaches: direct copying, image data augmentation, and generative data augmentation. Direct copying, as the name suggests, involves completely copying and expanding samples from a subset of the tail classes. This method is essentially identical to introducing class weights into the loss function; the same input data is calculated twice. This approach doesn't provide additional information gain to the model, limiting its generalization ability. Image data augmentation involves using image processing techniques such as adding noise, flipping, and cropping to diversify the information presented in the dataset. However, when the base model's capabilities are insufficient, it cannot generalize well to the augmented data domain, essentially introducing noise.

[0004] Downsampling algorithms are limited to filtering from an existing sample pool. However, they do not take into account the streaming media sampling process itself when processing small sample units as input in long videos. The distribution of sample segments selected by existing methods in the original video is not taken into account, which may lead to some data redundancy. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for sampling video data to solve the problem of uneven distribution of various types of data in video.

[0006] According to one aspect of the present invention, a method for sampling video data is provided, the method comprising:

[0007] Obtain the video to be sampled;

[0008] A current sliding window is generated based on a preset window length, and data is sampled from the video to be sampled based on the current sliding window.

[0009] When the sampling conditions are met, the data type of the data frame corresponding to the current sliding window is determined, and the window is slid according to the data type of each data frame to obtain the next sliding window and perform data sampling. When a header class exists in each of the data types, the window sliding step size is the first current sliding step size. When a header class does not exist in each of the data types, the window sliding step size is the second current sliding step size. The first current sliding step size is greater than the second current sliding step size.

[0010] The next sliding window is used as the new current sliding window, and the process of determining the data type of the execution data frame is returned when the sampling condition is met.

[0011] According to another aspect of the present invention, a video data sampling apparatus is provided, the apparatus comprising:

[0012] The video acquisition module is used to acquire the video to be sampled.

[0013] The sampling module is used to generate a current sliding window based on a preset window length, and to sample data from the video to be sampled based on the current sliding window.

[0014] The window sliding module is used to determine the data type of the data frame corresponding to the current sliding window when the sampling conditions are met, slide the window accordingly based on the data type of each data frame to obtain the next sliding window, and perform data sampling. When a header class exists in each of the data types, the window sliding step size is the first current sliding step size. When a header class does not exist in each of the data types, the window sliding step size is the second current sliding step size. The first current sliding step size is greater than the second current sliding step size.

[0015] The return module is used to take the next sliding window as the new current sliding window and return to the step of determining the data type of the execution data frame when the sampling condition is met.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video data sampling method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video data sampling method according to any embodiment of the present invention.

[0021] The technical solution of this invention involves: acquiring a video to be sampled; generating a current sliding window based on a preset window length; sampling data from the video to be sampled based on the current sliding window; when sampling conditions are met, determining the data type of the data frame corresponding to the current sliding window; sliding the window accordingly based on the data type of each data frame to obtain the next sliding window; and sampling data from the next sliding window. Specifically, when a header class exists in each of the data types, the sliding step size is a first current sliding step size; when no header class exists in each of the data types, the sliding step size is a second current sliding step size, where the first current sliding step size is greater than the second current sliding step size. The next sliding window is then used as the new current sliding window, and the process returns to the step of determining the data type of the data frame when the sampling conditions are met. This method addresses the issues of long tails and uneven data distribution in video processing. It employs a sliding window approach for data sampling. After sampling, the sliding window method is determined based on the data type of each frame within the current window. A larger step size is used when a header-like data type is present, while a smaller step size is used when no header-like data type is present. Different step sizes are used for window sliding and sampling based on different data types to reduce the impact of long tails. Since data sampling is performed using a sliding window, the window sliding can be controlled by different data types, thereby controlling the data acquisition frequency. By controlling the length and step size of the sliding window, data upsampling or downsampling is achieved, resulting in a more even distribution of sampled data. Furthermore, the sampling process does not introduce noise or data redundancy, improving data processing efficiency and accuracy while mitigating the impact of uneven data distribution.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a video data sampling method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a video data sampling method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is an example diagram illustrating a data sampling implementation according to Embodiment 2 of the present invention;

[0027] Figure 4 This is a sampling result comparison display diagram provided according to Embodiment 2 of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of a video data sampling device according to Embodiment 3 of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the video data sampling method provided in Embodiment 4 of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "target," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a video data sampling method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where data distribution in a video is uneven and data is sampled. The method can be executed by a video data sampling device, which can be implemented in hardware and / or software and can be configured in a computer device. As shown in Figure 1, the method includes:

[0034] S101. Obtain the video to be sampled.

[0035] In this embodiment, the video to be sampled can be specifically understood as a video with sampling requirements. The video to be sampled can be captured by an image acquisition device, such as a camera. The video to be sampled can be a video taken in any environment. For example, a camera installed in a parking lot can collect parking information to provide technical support for automated parking space allocation; or, a camera installed indoors can collect user actions so that smart home devices can provide corresponding services to the user based on the user's actions, and so on.

[0036] The video data sampling method of this application can sample video data of any type. Taking video data containing action as an example, the long-tail distribution phenomenon in video data can lead to poor accuracy of the trained model when predicting action, making it unable to accurately predict actions. Here, long-tail distribution refers to a distribution in deep learning scenarios where a minority class constitutes the vast majority of the samples, while a class with very few samples has a large number of classes. Therefore, when a distribution curve of sample number versus class is plotted, a long tail is drawn. Due to the large number of classes and the lack of sufficient samples for training in the minority classes, the model's classification accuracy is poor, or it tends to classify the head classes.

[0037] S102. Generate the current sliding window according to the preset window length, and sample the video to be sampled according to the current sliding window.

[0038] In this embodiment, the preset window length can be understood as the pre-defined length of the sliding window, which is the basic unit processed by the algorithm each time; the current sliding window can be understood as the sliding window currently being sampled. Because the sliding window needs to slide, its position may be different at different data sampling times, and each data sampling time has a corresponding current sliding window.

[0039] Specifically, a window length is preset, which is the input length of each sample unit required by the model prediction task, and is preset according to the actual engineering application requirements. A current sliding window of the corresponding length is generated according to the preset window length. The start position of the current sliding window is aligned with the first data frame of the video to be sampled. Data frames are extracted from the video to be sampled according to the length of the current sliding window, i.e., the preset window length, and data sampling is performed on the extracted data frames.

[0040] S103. When the sampling condition is met, determine the data type of the data frame corresponding to the current sliding window, slide the window accordingly based on the data type of each data frame to obtain the next sliding window, and sample the data. When a header class exists in each data type, the window sliding step size is the first current sliding step size. When no header class exists in each data type, the window sliding step size is the second current sliding step size. The first current sliding step size is greater than the second current sliding step size.

[0041] In this embodiment, the sampling condition can be specifically understood as the condition for determining whether to sample the video to be sampled. If the sampling condition is not met, the sampling of the video to be sampled ends. The data type can be specifically understood as the sample type of the data frame, such as background, header, and tail. Among them, background represents data frames that are background segments without actual meaning; header represents data frames that are segments with a large number of samples; and tail represents data frames that are segments with a small number of samples. The first current sliding step can be specifically understood as the step size of the sliding window when header data frames exist; the second current sliding step can be specifically understood as the step size of the sliding window when header data frames do not exist. The sizes of the first and second current sliding windows are set according to requirements, and the first current sliding step is larger than the second current sliding step.

[0042] Specifically, the data frames corresponding to the current sliding window in the video to be sampled are determined. The number of data frames is related to the length of the current sliding window. The data type of each data frame is determined. Usually, the data types of each data frame in the video to be sampled are pre-labeled, or the corresponding data types can be determined by analyzing the data frames using algorithms or models. After determining the data type of each data frame, the window slides according to the corresponding rules based on the data type. When the data type includes headers, there are data frames with a large number of samples in the current sliding window. In this case, to reduce the sampling of high-frequency data frames, a larger first current sliding step size can be used to control the sliding window to slide forward to obtain the next sliding window. When the data type does not include headers, there are data frames with a small number of samples in the current sliding window. In this case, to increase the sampling of low-frequency data frames, a smaller second current sliding step size can be used to control the sliding window to slide forward to obtain the next sliding window. After obtaining the next sliding window, data is sampled from the data frames corresponding to the next sliding window in the video to be sampled. The sampling method can adopt the data sampling method in S102. When sliding the window, the window length can also be adjusted according to the data type of each data frame.

[0043] S104. Set the next sliding window as the new current sliding window and return to execute S103.

[0044] The next sliding window is used as the new current sliding window. The sampling conditions are re-evaluated. If the sampling conditions are met, the window sliding operation is repeated.

[0045] As an optional embodiment of this example, the sampling conditions are further optimized. The sampling conditions include: the end of the current sliding window does not exceed the last data frame in the video to be sampled.

[0046] The process involves determining whether the end of the current sliding window extends beyond the last data frame in the video to be sampled, i.e., whether any data frames in the video have not yet been sampled. If the current sliding window includes the last data frame in the video, all data frames in the video have been sampled, and the sampling process ends. If the end of the current sliding window does not extend beyond the last data frame in the video, there are data frames in the video that have not yet been sampled, satisfying the sampling condition. The process then repeats to determine the data type of the data frames corresponding to the current sliding window, and continues sliding the window to ensure that all video frames are processed.

[0047] In this embodiment, it is determined whether the end of the current sliding window exceeds the last data frame. If not, the sliding continues to the right to obtain the next sliding window. If the end of the next sliding window exceeds the last data frame, and the number of data frames corresponding to the next sliding window is less than the preset window length, sampling of this sliding window can be abandoned, or sampling can proceed normally. If the number of data frames obtained by sampling according to the same rules is less than the number of data frames obtained by sampling other current sliding windows (the end of the current sliding window does not exceed the last data frame), the sampling rules can be changed to ensure that the number of data frames collected is consistent. Therefore, when sampling data from a sliding window, the number of data frames within the sliding window can be determined first.

[0048] This invention provides a video data sampling method, which involves: acquiring a video to be sampled; generating a current sliding window based on a preset window length; sampling data from the video to be sampled based on the current sliding window; when sampling conditions are met, determining the data type of the data frame corresponding to the current sliding window; sliding the window according to the data type of each data frame to obtain the next sliding window and sampling data thereon; wherein, when a header class exists in each of the data types, the sliding step size is a first current sliding step size; when no header class exists in each of the data types, the sliding step size is a second current sliding step size, and the first current sliding step size is greater than the second current sliding step size; using the next sliding window as the new current sliding window and returning to the step of determining the data type of the data frame when the sampling conditions are met. This method addresses the issues of long tails and uneven data distribution in video processing. It employs a sliding window approach for data sampling. After sampling, the sliding window method is determined based on the data type of each frame within the current sliding window. A large first sliding step is used when a header class exists, while a small second sliding step is used when no header class exists. Different step sizes are used for window sliding and sampling for different data types, reducing the impact of long tails. Since data sampling is based on a sliding window, the window sliding can be controlled by different data types, thereby controlling the data acquisition frequency. By controlling the length and step size of the sliding window, data upsampling or downsampling is achieved, resulting in a more even distribution of sampled data. Furthermore, the sampling process does not introduce noise or data redundancy, improving data processing efficiency and accuracy while mitigating the impact of uneven data distribution.

[0049] Example 2

[0050] Figure 2This is a flowchart of a video data sampling method provided in Embodiment 2 of the present invention. This embodiment refines the above embodiment by specifying that the step of sampling data from the video to be sampled based on the current sliding window is further refined as follows: determining the data frame corresponding to the current sliding window in the video to be sampled; and sampling data from each data frame according to a preset sampling interval. Figure 2 As shown, the method includes:

[0051] S201. Obtain the video to be sampled.

[0052] S202. Generate the current sliding window based on the preset window length.

[0053] S203. Determine the data frame corresponding to the current sliding window in the video to be sampled.

[0054] Specifically, during the first sampling, the starting point of the current sliding window is typically aligned with the first data frame of the video to be sampled. If the video to be sampled does not start from the first data frame, then the starting point of the current sliding window is aligned with the data frame at which sampling begins. The ending point of the current sliding window is determined based on a preset window length and aligned with the data frames in the video to be sampled. This determines all data frames between the two data frames corresponding to the starting and ending points, i.e., the data frames corresponding to the current sliding window in the video to be sampled. For example, if the preset window length is 600 frames and the video to be sampled includes 10,000 data frames, taking the video to be sampled as starting from the first data frame as an example, the data frames corresponding to the current sliding window in the video to be sampled are the data frames from frames 1 to 600.

[0055] S204. Sample data for each data frame according to the preset sampling interval.

[0056] In this embodiment, the preset sampling interval can be understood as a pre-set sampling interval. It can be sampling in seconds, for example, sampling once every 3 seconds, or sampling in frames, for example, sampling once every 10 frames.

[0057] Specifically, a preset sampling interval is set in advance according to the actual engineering requirements. Data frames are sampled according to the preset sampling interval. The sampling process can be uniform sampling or non-uniform sampling. If non-uniform sampling is used, there can be multiple preset sampling intervals, which are sampled sequentially according to the preset sampling intervals; or the preset sampling interval can be used as the starting interval, and the sampling interval can be adjusted based on this. For example, the sampling interval can be adjusted by increasing it sequentially according to a certain step size. For example, if the preset sampling interval is 2 and the step size is 1, the sampling intervals will be 2, 3, 4, 5... sequentially.

[0058] S205. When the sampling conditions are met, determine the data type of the data frame corresponding to the current sliding window.

[0059] As an optional embodiment of this example, this optional embodiment further optimizes the data types to include background class, header class and tail class.

[0060] S206. Determine whether a header class exists in the data type of each data frame. If yes, execute S207; otherwise, execute S209.

[0061] Determine whether the data types of each data frame corresponding to the current sliding window contain a header class. Data frames with a header class can be in any position—the first frame, a middle frame, or the last frame. If the data types of each data frame contain a header class, then the current sliding window contains data frames with a large number of samples, and steps S207-S208 are executed to slide the window. If the data types do not contain a header class, then the current sliding window contains data frames with a small number of samples, and steps S209-S210 are executed to slide the window.

[0062] S207. Determine the end frame corresponding to the current sliding window.

[0063] The end position of the current sliding window is determined by identifying the corresponding data frame in the video to be sampled, and this data frame is used as the end frame. The purpose of determining the end frame is to determine the starting position of the next sliding window.

[0064] As an optional embodiment of this example, this optional embodiment further determines the end frame corresponding to the current sliding window, including:

[0065] A. Determine the end position of the current sliding window.

[0066] In this embodiment, the end position of the window can be specifically understood as the position of the tail of the current sliding window in the video to be sampled, for example, the 600th frame.

[0067] B. Determine the target data frame in the video to be sampled where the window ends.

[0068] In this embodiment, the target data frame can be specifically understood as the data frame corresponding to the end position of the window, and the target data frame is a data frame in the data frames to be sampled. The target data frame corresponding to the end of the current sliding window in the video to be sampled is determined based on the end position of the window. For example, if the end position of the window is frame 600, then frame 600 in the video to be sampled is the target data frame.

[0069] C. Determine the target data frame as the end frame.

[0070] The target data frame is the end frame corresponding to the current sliding window.

[0071] S208. Take the next frame after the end frame as the starting frame of the next sliding window, slide to the right according to the preset sliding step size to obtain the next sliding window. The sum of the preset sliding step size and the length of the current sliding window is equal to the first current sliding step size.

[0072] In this embodiment, the preset sliding step size can be specifically understood as the step size of the sliding window when it slides in the presence of header-type data frames. The preset sliding step size can be pre-set and stored, and retrieved from the corresponding storage space when the window slides. The preset sliding step size can be 0, 600 frames, etc.

[0073] It should be noted that, in this application, data frames are arranged from left to right in chronological order, with data frames having smaller timestamps on the left and data frames having larger timestamps on the right. Therefore, the sliding to the right in this application embodiment refers to the movement of data frames backward in chronological order.

[0074] Specifically, the frame following the end of the current sliding window is used as the starting point of the next sliding window. The current sliding window is then slid to the right to obtain the next sliding window. When the preset sliding step size is 0, the starting frame of the next sliding window is adjacent to the ending frame of the current sliding window. In this case, the next sliding window is obtained by sliding the current sliding window by the preset window length. When the preset sliding step size is greater than 0, some data frames between the next and current sliding windows are not collected. For example, if the current sliding window is located at frames 1-600 and the preset sliding step size is 10, frame 601 is used as the starting frame of the next sliding window. Sliding 10 frames to the right, the next sliding window will be located at frames 611-1210. In this case, frames 601-610 are not in any sliding window, thus achieving coarse sampling of header-type data frames. Since the data frame contains a header class, the frame following the end frame of the current sliding window is used as the starting frame of the next sliding window. Therefore, the first current sliding step is equal to the sum of the preset sliding step and the length of the current sliding window, meaning that no data frame in the current sliding window is sampled again.

[0075] In this embodiment, the implementation of sliding the window according to the data type of each data frame to obtain the next sliding window can also be as follows: if the data type of each data frame contains a header class, slide to the right according to the first current sliding step size to obtain the next sliding window. When using this method to slide the window, if it is ensured that no data frame in the current sliding window is sampled again, the first current sliding step size must be greater than or equal to the length of the current sliding window.

[0076] S209. Determine the second current sliding step size.

[0077] In this embodiment, the second current sliding step size can be specifically understood as the step size when the sliding window slides without header-type data frames. The second current sliding step size can also be preset and stored, and retrieved from the corresponding storage space when the window slides. The second current sliding step size can be 100 frames, 200 frames, etc., and is usually smaller than the length of the current sliding window. The second current sliding step size and the first current sliding step size can be dynamically adjusted.

[0078] S210. Slide the current sliding window to the right according to the second current sliding step to obtain the next sliding window.

[0079] Slide the current sliding window to the right by the second current sliding step to obtain the next sliding window. For example, if the current sliding window is located at frames 1-600, the second current sliding step is 200, and the next sliding window is located at frames 201-800.

[0080] S211, Perform data sampling.

[0081] Data sampling is performed based on the next sliding window, using the same sampling method as described above.

[0082] S212, Set the next sliding window as the new current sliding window, and execute S205.

[0083] As an optional embodiment of this example, this optional embodiment is further optimized to include training the neural network model based on each sampled data frame.

[0084] In this embodiment, the neural network model can be any type of deep learning model, which can be used to predict actions, predict categories, etc. The sampled data frames can form a sequence and be used as training samples, or used individually as training samples for training the neural network model. This effectively reduces the long-tail effect when data is uneven, improving model accuracy with fewer training samples, thereby improving both the quality and accuracy of training.

[0085] For example, Figure 3This diagram illustrates an implementation example of data sampling according to an embodiment of the present invention. The data frames in the video 31 to be sampled include three data types: background, head, and tail. Different colors are used to distinguish between background and non-background types (head and tail). Head and tail types are labeled (e.g., for action prediction). Data frames with a dashed vertical line at the tail indicate that the data type is head. Taking the first frame of the video 31 to be sampled as the starting position of the current sliding window 32A, with a preset window length of 600 frames, a first current sliding step of 0 frames, and a second current sliding step of 200 frames, and with the first and second current sliding steps remaining unchanged, the data types of the data frames in the current sliding window 32A do not include the head type. Therefore, after data sampling is completed, the window slides 200 frames to the right according to the second current sliding step to obtain the next sliding window 32B and perform data sampling. The next sliding window 32B is used as the new current sliding window 32B. The end of the current sliding window 32B does not exceed the last data frame in the video to be sampled. The data type of each data frame in the current sliding window 32B is determined to exclude headers. Slide 200 frames to the right to obtain the next sliding window 32C, and perform data sampling. The next sliding window 32C is used as the new current sliding window 32C. The end of the current sliding window 32C does not exceed the last data frame in the video to be sampled. The data type of each data frame in the current sliding window 32C is determined to include headers. The frame after the end frame of the current sliding window 32C is used as the starting frame of the next sliding window 32D. Slide 0 frames to the right according to the first current sliding step size to obtain the next sliding window 32D. Sliding the current sliding window 32C to the right for 600 frames also yields the current sliding window 32D. Slide to the right sequentially in the above manner to obtain the second sliding window 32E… If the current sliding window includes the last data frame of the video to be sampled, and the end position of the current sliding window exceeds the frame number of the last data frame, the current sliding window can be discarded directly without data sampling, or data sampling can be performed.

[0086] For example, Figure 4 A comparison chart of sampling results is provided. For example... Figure 4As shown, the horizontal axis represents the i-th category of the sample, i = 0, 1, 2...3500; the vertical axis represents the number of samples. Curve 1 shows the distribution of the number of samples in each category of the training set before sampling. It can be clearly seen that it is a typical long-tail distribution, with more than three thousand categories. The number of samples in the head category is over a thousand, but the number of samples in more than two thousand tail categories is only in single digits. Curve 3 shows the sampling result of the video data provided in this application. The intersection with the x-axis is further forward, thus maintaining the general distribution of the original training set, but the long-tail trend is reduced. Curve 2 represents the existing ordinary threshold random sampling result. The tail curve completely overlaps with Curve 1, and the head category still accounts for a large proportion.

[0087] This invention provides a video data sampling method that solves the problems of long tails and uneven data distribution in videos. It employs a sliding window to sample video data. After sampling, the method of window sliding is determined based on the data type of each data frame corresponding to the current sliding window. Different step sizes are used for window sliding and sampling for different data types. A large step size is used when a header-type data frame exists, and a small step size is used when no header-type data frame exists, reducing the impact of long tails. The sliding method is determined based on the data type of each data frame, and the corresponding window sliding is performed. A large step size is used for header-type data frames to achieve downsampling, while a small step size is used for tail-type and background-type data frames. The sliding step size can be adjusted to achieve flexible sampling. Downsampling can also be achieved with a large window, and similarly, upsampling can be achieved with a small window. By controlling the length of the sliding window and the sliding step size, data upsampling or downsampling is achieved, resulting in a balanced distribution of sampled data. Furthermore, the sampling process does not introduce noise or data redundancy, improving data processing efficiency and accuracy, and reducing the impact of uneven data distribution.

[0088] Example 3

[0089] Figure 5 This is a schematic diagram of a video data sampling device provided in Embodiment 3 of the present invention. Figure 5 As shown, the device includes: a video acquisition module 41, a sampling module 42, a window sliding module 43, and a return module 44.

[0090] Video acquisition module 41 is used to acquire the video to be sampled;

[0091] Sampling module 42 is used to generate a current sliding window according to a preset window length, and to sample data from the video to be sampled according to the current sliding window;

[0092] The window sliding module 43 is used to determine the data type of the data frame corresponding to the current sliding window, slide the window accordingly based on the data type of each data frame to obtain the next sliding window, and perform data sampling. When a header class exists in each of the data types, the window sliding step size is the first current sliding step size. When a header class does not exist in each of the data types, the window sliding step size is the second current sliding step size. The first current sliding step size is greater than the second current sliding step size.

[0093] Return module 44 is used to take the next sliding window as the new current sliding window and return the step of determining the data type of the execution data frame when the sampling condition is met.

[0094] This invention provides a video data sampling device that solves the problems of long tails and uneven data distribution in videos. It employs a sliding window to sample video data. After sampling, the sliding window method is determined based on the data type of each data frame corresponding to the current sliding window. Different step sizes are used for window sliding and sampling depending on the data type of the data frame. A larger step size is used when a header class exists, and a smaller step size is used when no header class exists, reducing the impact of long tails. Since data sampling is based on a sliding window, the window sliding can be controlled by different data types, thereby controlling the data acquisition frequency. By controlling the length of the sliding window and the sliding step size, data upsampling or downsampling is achieved, resulting in a balanced distribution of sampled data. Furthermore, the sampling process does not introduce noise or cause data redundancy, improving data processing efficiency and accuracy, and reducing the impact of uneven data distribution.

[0095] Optionally, the sampling module 42 includes:

[0096] A data frame determination unit is used to determine the data frame corresponding to the current sliding window in the video to be sampled.

[0097] The data sampling unit is used to sample data from each of the data frames according to a preset sampling interval.

[0098] Optionally, the sampling conditions include: the end of the current sliding window does not exceed the last data frame in the video to be sampled.

[0099] Optionally, the window sliding module 43 includes:

[0100] The end frame determination unit is used to determine the end frame corresponding to the current sliding window if a header class exists in the data type of each of the data frames.

[0101] The first sliding unit is used to take the next frame of the ending frame as the starting frame of the next sliding window, slide to the right according to a preset sliding step size to obtain the next sliding window, wherein the sum of the preset sliding step size and the length of the current sliding window is equal to the first current sliding step size.

[0102] Optionally, the end frame determination unit is specifically used to determine the end position of the current sliding window; determine the target data frame corresponding to the end position of the window in the video to be sampled; and determine the target data frame as the end frame.

[0103] Optionally, the window sliding module 43 includes:

[0104] The step size determination unit is used to determine a second current sliding step size if there is no header class in the data type of each of the data frames;

[0105] The second sliding unit is used to slide the current sliding window to the right according to the second current sliding step size to obtain the next sliding window.

[0106] Optionally, the device may also include:

[0107] The training module is used to train the neural network model based on each sampled data frame.

[0108] The video data sampling device provided in the embodiments of the present invention can execute the video data sampling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0109] Example 4

[0110] Figure 6 A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0111] like Figure 6As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded into the RAM 63 from storage unit 68. The RAM 63 may also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0112] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0113] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as video data sampling methods.

[0114] In some embodiments, the video data sampling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the video data sampling method described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to perform the video data sampling method by any other suitable means (e.g., by means of firmware).

[0115] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0120] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0121] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for sampling video data, characterized in that, include: Obtain the video to be sampled; the data frames of the video to be sampled have pre-labeled data types, including header and tail classes; A current sliding window is generated based on a preset window length, and data is sampled from the video to be sampled based on the current sliding window. When the sampling conditions are met, the data type of the data frame corresponding to the current sliding window is determined. The window is then slid according to the data type of each data frame to obtain the next sliding window, and data sampling is performed. When a header class exists in each of the data types, the end frame corresponding to the current sliding window is determined, and the next frame of the end frame is taken as the starting frame of the next sliding window. The window is slid to the right according to a preset sliding step size to obtain the next sliding window. The sum of the preset sliding step size and the length of the current sliding window is the first current sliding step size, so that no data frame in the current sliding window is sampled again. When no header class exists in any of the data types, determine the second current sliding step size, and slide the current sliding window to the right according to the second current sliding step size to obtain the next sliding window; The next sliding window is used as the new current sliding window, and the process of determining the data type of the execution data frame is returned when the sampling condition is met.

2. The method according to claim 1, characterized in that, The step of sampling data from the video to be sampled based on the current sliding window includes: Determine the data frame corresponding to the current sliding window in the video to be sampled; Data is sampled from each data frame according to a preset sampling interval.

3. The method according to claim 1, characterized in that, The sampling conditions include: the end of the current sliding window does not exceed the last data frame in the video to be sampled.

4. The method according to claim 1, characterized in that, Determining the end frame corresponding to the current sliding window includes: Determine the end position of the current sliding window; Determine the target data frame in the video to be sampled where the window ends. The target data frame is determined as the end frame.

5. The method according to any one of claims 1-4, characterized in that, Also includes: The neural network model is trained based on each sampled data frame.

6. A video data sampling device, characterized in that, include: The video acquisition module is used to acquire the video to be sampled; the data frames of the video to be sampled have pre-labeled data types, including header class and tail class; The sampling module is used to generate a current sliding window based on a preset window length, and to sample data from the video to be sampled based on the current sliding window. The window sliding module is used to determine the data type of the data frame corresponding to the current sliding window when the sampling conditions are met, slide the window accordingly based on the data type of each data frame to obtain the next sliding window, and perform data sampling. Specifically, if a header class exists in any of the data types, the module determines the end frame corresponding to the current sliding window, takes the next frame of the end frame as the start frame of the next sliding window, and slides to the right according to a preset sliding step size to obtain the next sliding window. The sum of the preset sliding step size and the length of the current sliding window is the first current sliding step size, ensuring that no data frame in the current sliding window is sampled again. If no header class exists in any of the data types, the module determines a second current sliding step size, and slides the current sliding window to the right according to the second current sliding step size to obtain the next sliding window. The return module is used to take the next sliding window as the new current sliding window and return to the step of determining the data type of the execution data frame when the sampling condition is met.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video data sampling method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video data sampling method according to any one of claims 1-5.

Images