Model training method and device based on positive and negative examples, equipment and medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING YOUZHUJU NETWORK TECH CO LTD
- Filing Date
- 2022-11-17
- Publication Date
- 2026-06-09
Smart Images
Figure CN115758146B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, and medium based on positive and negative examples. Background Technology
[0002] With the development of internet technology, live video streaming has gradually entered people's lives. Live video streaming refers to the ability for people to watch live audio and video broadcasts from a remote location via the internet, such as sporting events, conferences, lectures, surgeries, and so on. The core idea of live video streaming is to utilize existing networks to achieve real-time transmission of audio and video signals and enable smooth viewing at a remote location.
[0003] Online video has become a tool for social interaction, learning, and work. However, the content of live videos is diverse, and some may contain inappropriate or false information. Therefore, the application of content understanding in live videos has emerged, and further, video classification and clustering can be performed based on the content of live videos. In existing technologies, deep learning can be used for video content analysis and classification. Specifically, by segmenting videos into fixed-length video slices, a video classification model is obtained by training a model based on these video slices.
[0004] However, existing video classification models do not perform well in classifying live videos. Summary of the Invention
[0005] This application provides a model training method, apparatus, device, and medium based on positive and negative examples, which improves the accuracy of model classification.
[0006] In a first aspect, embodiments of this application provide a model training method based on positive and negative examples, the method comprising:
[0007] During each round of model training, B samples are randomly selected from the full sample set to form a batch training set, where the samples in the full sample set are live slices.
[0008] Extract features from each training sample in the batch training set, wherein each training sample includes at least two features;
[0009] For each training sample in the batch training set, positive and negative examples are constructed, wherein, for any training sample, the positive examples of the first feature of the training sample include: the second feature of the training sample, and the negative examples of the training sample include: the second features of other training samples in the batch training set excluding the training sample.
[0010] The classification model is trained using positive and negative examples from each training sample in the batch training set, and a loss value is generated based on the training results. The loss value is used to characterize the difference between the similarity of positive examples and the similarity of negative examples.
[0011] In some embodiments, the features of the training samples include image features, audio features, and speech recognition features;
[0012] The first feature is any one of the image features, audio features, and speech recognition features, and the second feature is any one of the image features, audio features, and speech recognition features other than the first feature.
[0013] In some embodiments, when the features of the training samples include audio features, extracting the features of each training sample in the batch training set includes:
[0014] The starting position and / or truncation step size of the ASR text of the training samples are determined using a random method.
[0015] Based on the determined starting position and truncation step size, the ASR text of the training sample is truncated;
[0016] Using a speech extraction model, features are extracted from the truncated ASR text to obtain the speech recognition features of the training samples.
[0017] In some embodiments, the number of samples for each label in the full sample set meets a preset ratio, wherein when the number of samples for each label meets the preset ratio, the label distribution of the samples in the full sample set is balanced.
[0018] In some embodiments, the method further includes:
[0019] A preset number of live video segments are randomly selected from the live video, and tags are added to the live video segments;
[0020] Count the number of samples for each label and calculate the proportion of samples for each label;
[0021] If the ratio of the number of samples for each label does not meet the preset ratio, adjust the number of samples for each label so that the ratio of the number of samples for each label meets the preset ratio.
[0022] In some embodiments, the positive examples of the training samples further include: the features of m1 time-related slices of the training samples, wherein the time-related slices belong to the same live broadcast room as the training samples, and the time of the time-related slices is within a first target duration, wherein the first target duration includes a first preset duration before the time of the training samples, and / or a second preset duration after the time of the training samples.
[0023] In some embodiments, constructing positive and negative examples for each training sample in the batch training set includes:
[0024] For each training sample in the batch training set, obtain a slice of the target live streaming room to which the training sample belongs from the full sample set;
[0025] Based on the time of the training samples, a first candidate slice set is determined from the slices of the target live broadcast room that belong to the target duration;
[0026] Select m1 slices from the first candidate slice set as positive examples of the training samples;
[0027] Extract the features of the m1 slices.
[0028] In some embodiments, the first preset duration is the number of first slices, and the second preset duration is the number of second slices.
[0029] In some embodiments, the negative examples of the training sample further include: the features of n1 time-unrelated slices of the training sample, wherein the time-unrelated slices belong to the same live broadcast room as the training sample, and the time of the time-unrelated slices does not fall within the second target duration, wherein the second target duration includes: a third preset duration before the time of the training sample, and / or a fourth preset duration after the time of the training sample.
[0030] In some embodiments, constructing positive and negative examples for each training sample in the batch training set includes:
[0031] For each training sample in the batch training set, obtain a slice of the target live streaming room to which the training sample belongs from the full sample set;
[0032] Based on the time of the training samples, segments that do not belong to the second target duration are determined from the segments of the target live broadcast room to form a second candidate segment set;
[0033] Select n1 slices from the second candidate slice set as negative examples of the training samples;
[0034] Extract the features from the n1 slices.
[0035] In some embodiments, the third preset duration is the third number of slices, and the fourth preset duration is the fourth number of slices.
[0036] In some embodiments, the image features of the training samples are extracted using a convolutional neural network (CNN), the audio features of the training samples are extracted using a CNN-14, and the speech recognition features of the training samples are extracted using a BERT model.
[0037] On the other hand, embodiments of this application provide a model training apparatus based on positive and negative examples, comprising:
[0038] The extraction module is used to randomly select B samples from the full sample set to form a batch training set during each round of model training. The samples in the full sample set are live slices.
[0039] The feature extraction module is used to extract features from each training sample in the batch training set, wherein the training sample includes at least two features;
[0040] A positive and negative example construction module is used to construct positive and negative examples for each training sample in the batch training set. For any training sample, the positive examples of the first feature of the training sample include the second feature of the training sample, and the negative examples of the training sample include the second features of other training samples in the batch training set besides the training sample.
[0041] The training module is used to train a classification model using positive and negative examples of each training sample in the batch training set, and to generate a loss value based on the training results. The loss value is used to characterize the difference between the similarity of positive examples and the similarity of negative examples of the samples.
[0042] On the other hand, embodiments of this application provide an electronic device, the electronic device including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method as described in any of the above.
[0043] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in any of the foregoing descriptions.
[0044] On the other hand, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the above-mentioned embodiments.
[0045] The model training method, apparatus, device, and medium based on positive and negative examples provided in this application, in each round of model training, randomly selects B live-stream slice samples from the full sample set to form a batch training set. Features of each training sample in the batch training set are extracted, and each training sample includes at least two features. Positive and negative examples are constructed for each training sample. For any given training sample, the positive examples of the first feature of the training sample include the second feature of the training sample, and the negative examples of the training sample include the second features of other training samples. The positive and negative examples of each training sample in the batch training set are used to train the classification model. This method constructs positive and negative examples across modalities based on different features of the training samples when constructing positive and negative examples, making the features of the same sample as similar as possible across modalities and the features of different samples as dissimilar as possible, thereby improving the accuracy of model classification. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a flowchart of the model training method based on positive and negative examples provided in Embodiment 1 of this application;
[0048] Figure 2 A flowchart illustrating the speech recognition features of the training samples provided in Embodiment 2 of this application;
[0049] Figure 3 A flowchart for label equalization of the full sample set provided in Embodiment 3 of this application;
[0050] Figure 4 The method for constructing positive and negative examples of each training sample provided in Embodiment 4 of this application;
[0051] Figure 5 This is a schematic diagram of the structure of the model training device based on positive and negative examples provided in Embodiment 5 of this application;
[0052] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of this application. Detailed Implementation
[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0054] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0055] This application provides a model training method based on positive and negative examples for training live stream segments.
[0056] To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts involved in all embodiments of this application will be appropriately explained as follows:
[0057] Live streaming can be understood as creating information live on-site as events unfold and then disseminating it online. With the rapid popularization of 4G / 5G mobile internet and smartphones, the supply of content based on live streaming is becoming increasingly rich, and its application across various industries is becoming increasingly widespread. Especially in e-commerce, online education, gaming, tourism, real estate, and automotive sectors, live streaming has become an efficient means of information dissemination and business development.
[0058] Live video slicing refers to the segmentation of a live video stream into shorter segments. Live video streams are typically quite long, such as those created by users streaming on a live streaming application (app). These live videos can often last 30 minutes, an hour, or even several hours. Slicing technology divides these longer live videos into shorter segments, such as 20-second or 30-second segments.
[0059] Live videos often contain inappropriate or false information, leading to the development of content understanding applications. Further applications can be used for video classification and clustering based on the content. Understanding live video content can be based on the video's images (pictures), audio, or Automatic Speech Recognition (ASR) text. ASR is a technology that converts human speech into text; therefore, ASR text can be understood as what the host or viewers say in the live video.
[0060] Based on live content understanding, live stream segments can be classified and clustered. Currently, deep neural network models are commonly used for understanding and classifying live stream content. However, most current models are trained on short videos or traditional videos (such as TV series, movies, or recordings), and are not suitable for live videos. Live videos have their own unique characteristics. For example, short videos, due to their short duration, usually lack interconnected segments, while live videos are longer, and each live stream segment contains interconnected segments. Furthermore, traditional videos have fast scene transitions, while live videos have slow scene transitions, with the host often speaking alone in front of the camera for most of the time.
[0061] In deep neural network models, a common training method is to simultaneously provide the model with positive examples and negative examples. Positive and negative examples are relative. In classification problems, the definition of positive and negative examples requires a given class as a reference; samples belonging to that class are positive, and those not belonging to that class are negative. After constructing positive and negative examples, a loss function needs to be constructed to increase the discriminative power of the examples, thereby learning information from the data. The process of constructing negative examples relative to positive examples based on a certain strategy is called negative sampling.
[0062] Negative sampling, as the name suggests, involves selecting a subset of negative samples from a set of negative samples for model training. The reason for not using all negative samples is twofold: firstly, to reduce the training complexity of the model, and secondly, to ensure the training effect. Even with sufficient computational resources to optimize all negative examples each time, using a specific strategy to sample and select negative examples can achieve the same or even better results.
[0063] Generally speaking, the number of positive examples available for model training is very limited compared to the number of randomly constructed negative examples. Even with data augmentation of positive examples, the number of positive examples and candidate negative examples is often not on the same order of magnitude. Although the candidate set of negative examples is very large, negative examples that can bring information gain are the key to training, so high-quality negative sampling is crucial.
[0064] Analysis shows that a very small number of negative samples (i.e., 5% strong negatives) directly determine the model's performance, while a large number of simple negative samples have little impact on the model's performance. The desired sampling is strong negatives (hard negatives), which can improve the model's training effect. However, if the number of strong negatives exceeds a certain threshold, future positive examples may be sampled; these examples are usually called false negatives.
[0065] The model training method based on positive and negative examples provided in this application will be described in detail below with reference to the accompanying drawings.
[0066] Figure 1 This is a flowchart of a model training method based on positive and negative examples provided in Embodiment 1 of this application. This method is applied in electronic devices, such as mobile terminals, personal computers (PCs), laptops, dedicated devices, servers, etc. Figure 1 As shown, the model training method based on positive and negative examples in this embodiment includes the following steps.
[0067] S101. During each round of model training, B samples are randomly selected from the full sample set to form a batch training set. The samples in the full sample set are live slices.
[0068] The full sample set includes a massive number of live stream segments, each of which is called a sample. The full sample set also includes data for each sample. For example, the sample data includes: the segment identifier, the segment time, information about the live stream room to which the segment belongs, and the segment tag.
[0069] The slice identifier is used to uniquely identify a sample in the full sample set, and the slice time includes the start and end times of the slice.
[0070] Live video clips are typically part of live video streams. Each live video stream corresponds one-to-one with a live streaming room, and the information of the live streaming room to which the clip belongs can be the identifier of the live streaming room to which the clip belongs.
[0071] The tags for slices are used to distinguish the categories of slices. For example, the tags for slices can be singing, dancing, outdoor, e-commerce, games, chat, etc.
[0072] In this embodiment, the Mini-Batch Gradient Descent (MBGD) optimization algorithm is used for model training. MBGD is a compromise between batch gradient descent and stochastic gradient descent. Its idea is to update the model parameters using a batch of samples in each iteration. During training, the full sample set is divided into batches. In each round of training, B (i.e., batch_size) samples are randomly selected from the full sample set to form a batch training set. This batch training set is called a batch, and the batch_size is B. The value of B is usually much smaller than the size of the full sample set, which is the number of samples in the full sample set. For example, if the size of the full sample set is 1 million, the value of B is 500.
[0073] The main advantages of using MBGD are as follows: compared to stochastic gradient descent, optimizing model parameters on a batch each time through matrix operations is not much slower than optimizing single data; using a batch each time can greatly reduce the number of iterations required for model convergence, and at the same time, it can make the converged result closer to the effect of gradient descent.
[0074] S102. Extract the features of each training sample in the batch training set. Each training sample includes at least two features.
[0075] For example, the features of the training samples include image features, audio features, and speech recognition features, with optional difficulty. Other features, such as bullet screen features, can also be included. Neural networks can be used to extract features from the training samples.
[0076] Optionally, convolutional neural networks (CNNs) can be used to extract image features from training samples, CNN-14 can be used to extract audio features from training samples, and the BERT model can be used to extract speech recognition features. In the process of extracting speech recognition features, the speech data needs to be converted into text data first, and then the BERT model can be used to extract them.
[0077] S103. Construct positive and negative examples for each training sample in the batch training set. For any training sample, the positive examples of the first feature of the training sample include the second feature of the training sample, and the negative examples of the training sample include the second features of other training samples in the batch training set besides the training sample.
[0078] This embodiment defines a batch-based positive and negative example construction matching task. Batch-based positive and negative example construction refers to negative sampling from a batch to find negative samples of the training samples within the batch. This matching task is a multimodal matching task, where each feature of the training samples represents a modality. Multimodal matching means matching any two different modalities. Taking the training sample features as including image features, audio features, and speech recognition features as an example, the multimodal matching task includes: matching image features with audio features, or matching image features with speech recognition features, or matching audio features with speech recognition features. The negative examples constructed based on batches can be understood as pseudo-negative examples.
[0079] The optimization objective of multimodal matching is to make all modalities within the same slice as similar as possible, and to make all modalities across different slices as dissimilar as possible. Multimodal matching enables a better understanding of the content of live stream slices, improving the metrics of downstream tasks such as live stream slice classification and clustering.
[0080] Taking a batch size of B and training samples featuring image features, audio features, and speech recognition features as an example, for training sample i in the batch, the image feature image_emb_i of training sample i, and the corresponding other modal features asr_emb_i (speech recognition features of training sample i) and audio_emb_i (audio features of training sample i) are all cross-modal positive examples of image_emb_i. The asr_emb_j (speech recognition features of training sample j) and audio_emb_j (audio features of training sample j) of other slice samples j (j = 0 to B-1 and j != i) are all negative classes of image_emb_i.
[0081] The batch-based positive and negative example construction method described above effectively avoids the trouble of global negative example enumeration. Global negative example enumeration can be understood as negative sampling from the full sample set, while batch-based positive and negative example construction performs negative sampling from a batch. The number of samples in a batch is much smaller than the number of samples in the full sample set, thereby reducing the computational complexity of the classification loss of positive and negative examples.
[0082] In addition, the training effect based on batch positive and negative examples can achieve a training effect that is close to that of global negative example enumeration. This is because in each round of training, samples are randomly selected from the full sample set to form a batch training set. By randomly scattering the samples, the negative examples of the samples are sufficiently diverse. As the training iterates, it can approximately achieve the training effect of global negative example enumeration.
[0083] It should be noted that model training can define one or more matching tasks. For example, two matching tasks can be defined: one is cross-modal matching of image features and audio features, and the other is cross-modal matching of image features and speech recognition features.
[0084] S104. Use the positive and negative examples of each training sample in the batch training set to train the classification model, and generate loss values based on the training results.
[0085] After constructing positive and negative examples from each training sample, the training sets are input into the classification model in batches for training, and loss values are generated based on the training results. The loss value is obtained based on the loss function, which represents the difference between the result predicted by the classification model and the actual sample.
[0086] A loss function is a function that maps the values of a random event or its related random variables to non-negative real numbers to represent the "difference" or "loss" of that random event. In machine learning, it is used for parameter estimation of models. The loss function is often associated with the optimization problem as a learning criterion; that is, the model is solved and evaluated by minimizing the loss function.
[0087] This loss value is used to characterize the difference between the similarity of positive and negative examples in a sample. This loss value can also be understood as contrastive loss, and the training of the model can also be understood as contrastive learning. Contrastive learning aims to learn a model by automatically constructing similar and dissimilar instances, so that similar instances are closer together, while dissimilar instances are farther apart.
[0088] For example, in a batch of size B, for each matching task, each training sample in the batch has one positive example and B-1 negative examples. The first feature and the second feature of training sample i constitute a positive example pair, and the first feature of training sample i and the second feature of training sample j (j = 0 to B-1 and j != i) constitute a negative example pair. When calculating the classification loss, the similarity between the two features in the positive example pair and the similarity between the two features in the negative example pair are calculated. Then, based on the difference between the similarity between the two features in the positive example pair (i.e., the similarity of the positive examples of the samples) and the similarity between the two features in the negative example pair (i.e., the similarity of the negative examples of the samples), the contrastive loss is calculated. The optimization objective of the model is to maximize the similarity between the two features in the positive example pair and minimize the similarity between the two features in the negative example pair.
[0089] In this embodiment, during each round of model training, B live-stream slices are randomly selected from the full sample set to form a batch training set. Features are extracted from each training sample in the batch training set, and each training sample includes at least two features. Positive and negative examples are constructed for each training sample. For any training sample, the positive examples of the first feature of the training sample include the second feature of the training sample, and the negative examples of the training sample include the second features of other training samples. The classification model is trained using the positive and negative examples of each training sample in the batch training set. When constructing positive and negative examples for training samples, this method performs cross-modal construction based on different features of the training samples, making the features of the same sample as similar as possible across modalities and the features of different samples as dissimilar as possible, thereby improving the accuracy of model classification.
[0090] Based on Embodiment 1, Embodiment 2 of this application performs data augmentation on the ASR text of the training samples using a random offset method. Data augmentation, also known as data expansion, aims to obtain diverse data without substantially increasing the amount of data, allowing limited data to generate value equivalent to more data. Figure 2 This is a flowchart of the speech recognition features of the training samples provided in Embodiment 2 of this application. This embodiment is used to illustrate step S102 in Embodiment 1, as follows: Figure 2 As shown, the method provided in this embodiment includes the following steps:
[0091] S201. For each training sample in the batch training set, the starting position and / or step size of the training sample are determined randomly.
[0092] The number of words in the ASR text of different live stream segments varies, with some having dozens of words and others having hundreds of words. Through statistical analysis of a large number of samples, the average number of words in the ASR text of live stream segments is 70-80 words. However, existing speech extraction models usually only use the first 32 words of the ASR text for speech recognition feature extraction, and discard the rest of the text. Using only the first 32 words of the ASR text for speech feature extraction is too simplistic and affects the accuracy of the model.
[0093] Therefore, in this embodiment, two parameters are defined: the starting position of the ASR text and the truncation step size. The starting position of the ASR text is used to determine the starting position of the truncated text. The starting position of the ASR text can be the order of the starting characters. For example, if the starting position of the ASR text is 6, then when truncating the ASR text of the training sample, the truncation starts from the 6th character of the ASR text of the training sample. If the starting position of the ASR text is 15, then when truncating the ASR text of the training sample, the truncation starts from the 15th character of the ASR text of the training sample.
[0094] The truncation step is used to determine the length of the truncation text. The truncation step can be the number of characters to be truncated. For example, if the truncation step is 25, then 25 characters will be truncated from the beginning of the ASR text.
[0095] In one implementation, the starting position of the ASR text for each training sample is fixed, but the truncation step size for each training sample is random. In another implementation, the starting position of the ASR text for each training sample is random, but the truncation step size for each training sample is fixed. In yet another implementation, both the starting position and the truncation step size of the ASR text for each training sample are random.
[0096] In this embodiment, before extracting the speech recognition features of each training sample, the starting position and truncation step of the ASR text of the training sample are determined in a random manner, so that the starting position and truncation length of the ASR text of each training sample may be different, thereby achieving the purpose of data diversification and data augmentation.
[0097] S202. Based on the determined starting position and truncation step size, truncate the ASR text of the training samples.
[0098] For example, if the starting position of the ASR text is 8 and the truncation step is 45, then 45 characters are truncated starting from the 8th character of the ASR text in the training sample.
[0099] S203. Using a speech extraction model, feature extraction is performed on the extracted ASR text to obtain the speech recognition features of the training samples.
[0100] For example, the speech extraction model is the BERT model, which is used to extract features from the truncated ASR text.
[0101] In this embodiment, for each training sample in the batch training set, the starting position and / or step size of the training sample are determined randomly. Based on the determined starting position and truncation step size, the ASR text of the training sample is truncated. A speech extraction model is then used to extract features from the truncated ASR text to obtain the speech recognition features of the training sample. This method uses a random method to determine the starting position and truncation step size of the ASR text of the training samples, ensuring that the starting position and truncation length of the ASR text for each training sample may be different, achieving data diversification and data augmentation. Data augmentation can improve the performance of the model.
[0102] The long-tail problem is prevalent in deep learning. Essentially, it stems from class imbalance in the data, where a small number of classes constitute the majority of samples, while the majority classes have only a small number of samples. Directly using long-tail data to train a classification model often leads to overfitting to the head data, causing the tail classes to be ignored during prediction. Therefore, class balance in the training data is crucial for model performance.
[0103] The long-tail problem also exists in live stream clips. Simply randomly selecting a batch of live stream clips results in most clips being footage of the anchor chatting in front of the camera. This is not conducive to understanding the multimodal content of the clips, as it fails to learn useful gain information. Therefore, in Embodiment 3 of this application, the entire sample set is balanced so that the number of samples for each label in the entire sample set meets a preset ratio. When the number of samples for each label meets the preset ratio, the label distribution of the samples in the entire sample set is balanced.
[0104] Figure 3 The flowchart for label equalization of the full sample set provided in Embodiment 3 of this application is as follows: Figure 3 As shown, the method provided in this embodiment includes the following steps.
[0105] S301. Obtain a preset number of live video segments from the live video using a random method, and add tags to the live video segments.
[0106] For example, tags for live stream segments include singing, dancing, outdoor activities, e-commerce, gaming, and chatting.
[0107] S302. Count the number of samples for each label and calculate the proportion of samples for each label.
[0108] S303. When the ratio of the number of samples for each label does not meet the preset ratio, adjust the number of samples for each label so that the ratio of the number of samples for each label meets the preset ratio.
[0109] The proportion of samples for each label is compared with a preset ratio. If the proportion of samples for each label does not meet the preset ratio, it means that the labels of the samples in the full sample set are unbalanced and the number of samples for each label needs to be adjusted. For example, if the proportion of samples for a certain label is significantly too large, some samples corresponding to that label are deleted, some live broadcast segments are reacquired, and labels are added to the reacquired live broadcast segments.
[0110] Typically, a batch of completely random live stream segments are randomly selected. Most of the live stream segments are labeled as chat. In this case, the chat label is used as the header label. However, the number of samples under the header label is too large, so it is necessary to reduce the number of samples corresponding to the chat label so that samples with some uncommon labels can be paid attention to during model training.
[0111] It is understandable that when the proportion of samples for each label does not meet the preset proportion, it may be necessary to make multiple adjustments to ensure that the proportion of samples for each label meets the preset proportion.
[0112] The method in this embodiment adjusts the number of samples for each label in the full sample set so that the ratio of the number of samples for each label meets the preset ratio, and the label distribution of the samples in the full sample set is balanced, thereby reducing the impact of the long tail effect on model training and improving the performance of the model.
[0113] Besides the simple construction of positive and negative examples based on batches, live stream segments have a very important characteristic: temporal sequence. Most live stream segments have several consecutive adjacent segments preceding and following them, and these adjacent segments are very valuable for comparative learning of live stream segments. Therefore, this application's embodiments attempt to utilize the temporal sequence between segments within a live stream to construct positive and negative samples for the current segment within the same live stream, thereby improving model performance.
[0114] In one implementation, the positive examples of the training samples further include: the features of m1 time-related slices of the training samples, wherein the time-related slices belong to the same live broadcast room as the training samples, and the time of the time-related slices belongs to a first target duration, wherein the first target duration includes a first preset duration before the time of the training samples, and / or a second preset duration after the time of the training samples.
[0115] The first and second preset durations can be either a time length or a number of slices. For example, if the first preset duration is the first number of slices M1 and the second preset duration is the second number of slices M2, then when the first and second preset durations are the number of slices, the time-series associated slices are: those belonging to the same live stream as the training sample and whose time is within the M1 slices preceding the training sample, and / or those whose time is within the M2 slices following the training sample. Optionally, M1 and M2 can have the same value.
[0116] In another implementation, the negative examples of the training samples also include: the features of n1 time-unrelated slices of the training samples, which belong to the same live broadcast room as the training samples, and whose time does not fall within the second target duration, which includes: a third preset duration before the time of the training samples, and / or a fourth preset duration after the time of the training samples.
[0117] The third and fourth preset durations can be a time length or a number of slices. For example, if the third preset duration is the number of slices N1 and the fourth preset duration is the number of slices N2, then when the third and fourth preset durations are the number of slices, the associated slice for this timing step is: belonging to the same live stream as the training sample, and the time cannot be within the N1 slices before the training sample, and / or, the time cannot be within the N2 slices after the training sample; or described as: belonging to the same live stream as the training sample, and the time is outside the M1 slices before the training sample, and / or, the time is outside the M2 slices after the training sample. Optionally, N1 and N2 can have the same value.
[0118] In this embodiment, the positive examples of the training samples can be augmented only based on the temporal nature of the slices, or the negative examples of the training samples can be augmented only based on the temporal nature of the slices, or both the positive and negative examples of the training samples can be augmented based on the temporal nature of the slices.
[0119] Figure 4 This embodiment describes the method for constructing positive and negative examples of each training sample provided in Embodiment 4 of this application. It uses the example of expanding both positive and negative examples of the training samples based on the temporal sequence of slices for illustration. This embodiment is used to provide a detailed explanation of step S103 in Embodiment 1. Figure 4 As shown, the method provided in this embodiment includes the following steps.
[0120] S401. For each training sample in the batch training set, obtain the slice of the target live room to which the training sample belongs from the full sample set.
[0121] The data for each training sample includes the information of the live streaming room to which the training sample belongs and the time of the training sample. Based on the identifier of the target live streaming room to which the training sample belongs, the slice of the same live streaming room is determined from the full sample set.
[0122] S402. Based on the time of the training samples, determine the first candidate slice set consisting of slices belonging to the target duration from the slices of the target live broadcast room, and select m1 slices from the first candidate slice set as positive examples of the training samples.
[0123] The first target duration includes: a first preset duration prior to the training sample time, and / or, a second preset duration after the training sample time. Correspondingly, the first candidate slice set includes: slices within the first preset duration prior to the training sample time, and / or, slices within the second preset duration after the training sample time.
[0124] The value of m1 can be equal to or less than the number of slices in the first candidate slice set, meaning that all or part of the slices in the first candidate slice set can be selected as positive examples of time-related training samples. When selecting a portion of the slices as positive examples of time-related training samples, a preset number of slices can be selected. Optionally, a preset proportion of slices can also be selected, for example, 1 / 3 or 1 / 2 of the slices in the first candidate slice set can be selected as positive examples of training samples.
[0125] S403. Based on the time of the training samples, determine the slices that do not belong to the second target duration from the slices of the target live room to form a second candidate slice set, and select n1 slices from the second candidate slice set as negative examples of the training samples.
[0126] The second target duration includes: a third preset duration prior to the training sample time, and / or, a third preset duration after the training sample time. Correspondingly, the second candidate slice set includes: slices outside the third preset duration prior to the training sample time, and / or, slices outside the fourth preset duration after the training sample time.
[0127] The value of n1 can be equal to or less than the number of slices in the second candidate slice set, meaning that all or part of the slices in the second candidate slice set can be selected as time-related negative examples of the training samples. When selecting a portion of the slices as time-related negative examples of the training samples, a preset number of slices can be selected, and optionally, a preset proportion of slices can also be selected.
[0128] For example, if the first preset duration, second preset duration, third preset duration, and fourth preset duration are all equal, and the number of slices is M, then the first candidate slice set includes: M slices before the time of the training sample, and / or M slices after the time of the training sample. The second candidate slice set includes: slices other than the M slices before the time of the training sample, and / or slices other than the M slices after the time of the training sample.
[0129] Optionally, slices within the same live stream can be numbered sequentially. When determining which slices are temporally related and which are not, the slice numbering is used. For example, assuming the training sample's sequence in the target live stream is 20 and M is 10, the slice numbers in the first candidate slice set must be greater than or equal to 10 and less than or equal to 30, i.e., slice numbers 10-19 and 21-30. Correspondingly, the slice numbers in the second candidate slice set must be less than 10 or greater than 30, i.e., slice numbers 0-9 and 31-40.
[0130] S404. Extract features from m1 slices and n1 slices.
[0131] Feature extraction is performed on the positive and negative examples constructed based on time series. The feature extraction of positive and negative examples in this step is the same as that of the training samples mentioned above, and will not be repeated here.
[0132] During model training, simple positive and negative examples constructed using batch and extended positive and negative examples based on time sequence constructed in this embodiment are used for training. The extended positive and negative examples based on time sequence belong to the same live broadcast room as the training slice. Compared with the simple positive and negative examples constructed using batch, the extended positive and negative examples are difficult positive and negative examples, which greatly improve the training of the model.
[0133] To facilitate better implementation of the model training method based on positive and negative examples in the embodiments of this application, the embodiments of this application also provide a model training device based on positive and negative examples. Figure 5 This is a schematic diagram of the model training device based on positive and negative examples provided in Embodiment 5 of this application, as shown below. Figure 5 As shown, the model training device 100 constructed based on positive and negative examples may include:
[0134] Extraction module 11 is used to randomly select B samples from the full sample set to form a batch training set during each round of model training. The samples in the full sample set are live slices.
[0135] Feature extraction module 12 is used to extract features of each training sample in the batch training set, wherein the training sample includes at least two features;
[0136] The positive and negative example construction module 13 is used to construct positive and negative examples for each training sample in the batch training set. For any training sample, the positive examples of the first feature of the training sample include the second feature of the training sample, and the negative examples of the training sample include the second features of other training samples in the batch training set besides the training sample.
[0137] Training module 14 is used to train a classification model using positive and negative examples of each training sample in the batch training set, and to generate a loss value based on the training results. The loss value is used to characterize the difference between the similarity of positive examples and the similarity of negative examples of the samples.
[0138] In some embodiments, the features of the training samples include image features, audio features, and speech recognition features; the first feature is any one of the image features, audio features, and speech recognition features, and the second feature is any one of the image features, audio features, and speech recognition features other than the first feature.
[0139] In some embodiments, when the features of the training samples include audio features, the feature extraction module 12 is specifically used for:
[0140] The starting position and / or truncation step size of the ASR text of the training samples are determined using a random method.
[0141] Based on the determined starting position and truncation step size, the ASR text of the training sample is truncated;
[0142] Using a speech extraction model, features are extracted from the truncated ASR text to obtain the speech recognition features of the training samples.
[0143] In some embodiments, the number of samples for each label in the full sample set meets a preset ratio, wherein when the number of samples for each label meets the preset ratio, the label distribution of the samples in the full sample set is balanced.
[0144] In some embodiments, an equalization module is further included, for:
[0145] A preset number of live video segments are randomly selected from the live video, and tags are added to the live video segments;
[0146] Count the number of samples for each label and calculate the proportion of samples for each label;
[0147] If the ratio of the number of samples for each label does not meet the preset ratio, adjust the number of samples for each label so that the ratio of the number of samples for each label meets the preset ratio.
[0148] In some embodiments, the positive examples of the training samples further include: the features of m1 time-related slices of the training samples, wherein the time-related slices belong to the same live broadcast room as the training samples, and the time of the time-related slices is within a first target duration, wherein the first target duration includes a first preset duration before the time of the training samples, and / or a second preset duration after the time of the training samples.
[0149] In some embodiments, the positive and negative example construction module 13 is specifically used for:
[0150] For each training sample in the batch training set, obtain a slice of the target live streaming room to which the training sample belongs from the full sample set;
[0151] Based on the time of the training samples, a first candidate slice set is determined from the slices of the target live broadcast room that belong to the target duration;
[0152] Select m1 slices from the first candidate slice set as positive examples of the training samples;
[0153] Extract the features of the m1 slices.
[0154] In some embodiments, the first preset duration is the number of first slices, and the second preset duration is the number of second slices.
[0155] In some embodiments, the negative examples of the training sample further include: the features of n1 time-unrelated slices of the training sample, wherein the time-unrelated slices belong to the same live broadcast room as the training sample, and the time of the time-unrelated slices does not fall within the second target duration, wherein the second target duration includes: a third preset duration before the time of the training sample, and / or a fourth preset duration after the time of the training sample.
[0156] In some embodiments, the positive and negative example construction module 13 is specifically used for:
[0157] For each training sample in the batch training set, obtain a slice of the target live streaming room to which the training sample belongs from the full sample set;
[0158] Based on the time of the training samples, segments that do not belong to the second target duration are determined from the segments of the target live broadcast room to form a second candidate segment set;
[0159] Select n1 slices from the second candidate slice set as negative examples of the training samples;
[0160] Extract the features from the n1 slices.
[0161] In some embodiments, the third preset duration is the third number of slices, and the fourth preset duration is the fourth number of slices.
[0162] In some embodiments, the image features of the training samples are extracted using a convolutional neural network (CNN), the audio features of the training samples are extracted using a CNN-14, and the speech recognition features of the training samples are extracted using a BERT model.
[0163] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.
[0164] The apparatus 100 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly manifested as execution by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.
[0165] This application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of this application, as shown below. Figure 6 As shown, the electronic device 200 may include:
[0166] The system includes a memory 21 and a processor 22. The memory 21 stores computer programs and transfers the program code to the processor 22. In other words, the processor 22 can retrieve and run the computer programs from the memory 21 to implement the methods described in the embodiments of this application.
[0167] For example, the processor 22 can be used to execute the above-described method embodiments according to instructions in the computer program.
[0168] In some embodiments of this application, the processor 22 may include, but is not limited to:
[0169] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0170] In some embodiments of this application, the memory 21 includes, but is not limited to:
[0171] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).
[0172] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in an electronic device.
[0173] like Figure 6 As shown, the electronic device may further include a transceiver 23, which can be connected to the processor 22 or the memory 21.
[0174] The processor 22 can control the transceiver 23 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 23 may include a transmitter and a receiver. The transceiver 23 may further include antennas, and the number of antennas may be one or more.
[0175] Understandable, although Figure 6 As not shown in the diagram, the electronic device 200 may also include a camera module, a Wi-Fi module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be described in detail here.
[0176] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.
[0177] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.
[0178] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the corresponding flow in the user position control method in the virtual scene of this application embodiment. For simplicity, further details are omitted here.
[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0180] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0181] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A model training method based on positive and negative examples, characterized in that, The method includes: During each round of model training, B samples are randomly selected from the full sample set to form a batch training set, where the samples in the full sample set are live slices. Extract features from each training sample in the batch training set, wherein each training sample includes at least two features; For each training sample in the batch training set, positive and negative examples are constructed, wherein, for any training sample, the positive examples of the first feature of the training sample include: the second feature of the training sample, and the negative examples of the training sample include: the second features of other training samples in the batch training set excluding the training sample. The classification model is trained using positive and negative examples from each training sample in the batch training set, and a loss value is generated based on the training results. The loss value is used to characterize the difference between the similarity of positive examples and the similarity of negative examples. The features of the training samples include image features, audio features, and speech recognition features; The first feature is any one of the image features, audio features, and speech recognition features, and the second feature is any one of the image features, audio features, and speech recognition features other than the first feature.
2. The method according to claim 1, characterized in that, When the features of the training samples include audio features, the extraction of features from each training sample in the batch training set includes: The starting position and / or truncation step size of the ASR text of the training samples are determined using a random method. Based on the determined starting position and truncation step size, the ASR text of the training sample is truncated; Using a speech extraction model, features are extracted from the truncated ASR text to obtain the speech recognition features of the training samples.
3. The method according to claim 1, characterized in that, The number of samples for each label in the full sample set meets a preset ratio, wherein when the number of samples for each label meets the preset ratio, the label distribution of the samples in the full sample set is balanced.
4. The method according to claim 3, characterized in that, The method further includes: A preset number of live video segments are randomly selected from the live video, and tags are added to the live video segments; Count the number of samples for each label and calculate the proportion of samples for each label; If the ratio of the number of samples for each label does not meet the preset ratio, adjust the number of samples for each label so that the ratio of the number of samples for each label meets the preset ratio.
5. The method according to claim 1, characterized in that, The positive examples of the training samples also include: the features of m1 time-related slices of the training samples, wherein the time-related slices belong to the same live broadcast room as the training samples, and the time of the time-related slices is within a first target duration, wherein the first target duration includes a first preset duration before the time of the training samples, and / or a second preset duration after the time of the training samples.
6. The method according to claim 5, characterized in that, The construction of positive and negative examples for each training sample in the batch training set includes: For each training sample in the batch training set, obtain a slice of the target live streaming room to which the training sample belongs from the full sample set; Based on the time of the training samples, a first candidate slice set is determined from the slices of the target live broadcast room that belong to the target duration; Select m1 slices from the first candidate slice set as positive examples of the training samples; Extract the features of the m1 slices.
7. The method according to claim 5 or 6, characterized in that, The first preset duration is the number of first slices, and the second preset duration is the number of second slices.
8. The method according to any one of claims 1-6, characterized in that, The negative examples of the training samples also include: the features of n1 time-independent slices of the training samples, wherein the time-independent slices belong to the same live broadcast room as the training samples, and the time of the time-independent slices does not fall within the second target duration, wherein the second target duration includes: a third preset duration before the time of the training samples, and / or a fourth preset duration after the time of the training samples.
9. The method according to claim 8, characterized in that, The construction of positive and negative examples for each training sample in the batch training set includes: For each training sample in the batch training set, obtain a slice of the target live streaming room to which the training sample belongs from the full sample set; Based on the time of the training samples, segments that do not belong to the second target duration are determined from the segments of the target live broadcast room to form a second candidate segment set; Select n1 slices from the second candidate slice set as negative examples of the training samples; Extract the features from the n1 slices.
10. The method according to claim 9, characterized in that, The third preset duration is the third number of slices, and the fourth preset duration is the fourth number of slices.
11. The method according to claim 1, characterized in that, The image features of the training samples are extracted using a convolutional neural network (CNN), the audio features of the training samples are extracted using a CNN-14, and the speech recognition features of the training samples are extracted using a BERT model.
12. A model training device based on positive and negative examples, characterized in that, include: The extraction module is used to randomly select B samples from the full sample set to form a batch training set during each round of model training. The samples in the full sample set are live slices. The feature extraction module is used to extract features from each training sample in the batch training set, wherein the training sample includes at least two features; A positive and negative example construction module is used to construct positive and negative examples for each training sample in the batch training set. For any training sample, the positive examples of the first feature of the training sample include the second feature of the training sample, and the negative examples of the training sample include the second features of other training samples in the batch training set besides the training sample. The training module is used to train the classification model using positive and negative examples of each training sample in the batch training set, and to generate a loss value based on the training results. The loss value is used to characterize the difference between the similarity of positive examples and the similarity of negative examples of the samples. The features of the training samples include image features, audio features, and speech recognition features; The first feature is any one of the image features, audio features, and speech recognition features, and the second feature is any one of the image features, audio features, and speech recognition features other than the first feature.
13. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 11.
15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 11.