Uncertain positive and unmarked learning method suitable for video content identification

Abstract

The invention relates to the technical field of video content recognition, in particular to an uncertain positive and unlabeled learning method suitable for video content recognition, which comprises the following steps of: extracting video frames from a video monitoring system, converting the video frames into standard image data, and dividing the standard image data into a labeled positive sample set and an unlabeled sample set; preheating training is carried out on an initial model based on an improved non-negative positive unlabeled (nnPU) loss function, positive sample and unlabeled sample contribution is balanced, smooth soft labels are generated by adopting an exponential moving average (EMA) strategy, historical prediction and current model output are dynamically fused, and the prediction accuracy is improved. The method comprises the following steps: calculating a sample uncertainty score, generating an uncertainty weight based on a radial basis function kernel, performing segmentation optimization on an unlabeled sample according to a soft label threshold value, and distinguishing high / low confidence samples; the method effectively improves the accuracy and stability of video content recognition, balances the calculation efficiency and model performance, and has wide scene adaptability and industrial practicability.

Description

Technical Field

[0001] The present invention relates to the technical field of video content recognition, and in particular to an uncertain positive and negative labeling learning method suitable for video content recognition. Background Art

[0002] In today's digital age, video surveillance systems have been widely used in various fields, such as open kitchen monitoring, elevator safety monitoring, and abnormal behavior detection in public places, aiming to ensure safety, standardize operations, and provide effective management basis. With the massive growth of video data, how to efficiently and accurately identify video content has become a key issue.

[0003] In the field of video content recognition, the positive unlabeled learning (PUlearning) method has gradually become a research hotspot because it can use a large amount of unlabeled data to assist a small amount of labeled positive samples for model training. Traditional positive unlabeled learning methods, such as the two-step method, usually first screen or classify unlabeled samples, and then combine them with positive samples for model training. However, this method often has difficulty in accurately distinguishing positive and negative samples in unlabeled samples, which can easily lead to the model learning wrong information and affect recognition performance.

[0004] Generative adversarial network methods are also used in PU learning, which enhances the model's ability to process unlabeled samples through adversarial training of the generator and the discriminator. However, generative adversarial networks have problems such as unstable training and difficulty in convergence, and they require high computing resources, which limits their practical applications.

[0005] In addition, in the task of video content recognition, due to the temporal and dynamic characteristics of video data, existing methods have shortcomings in dealing with these characteristics. On the one hand, the fluctuation problem of a single prediction will affect the model's accurate judgment of the video content; on the other hand, it is difficult for the model to effectively capture the dynamic temporal information between video frames, resulting in limited recognition of complex behaviors and states in the video.

[0006] At the same time, for different video surveillance scenarios, such as standard operation recognition in open kitchen monitoring, normal state judgment in elevator safety monitoring, and abnormal behavior detection in public places, existing methods lack sufficient adaptability and flexibility to meet diverse practical needs. Moreover, after the model is deployed, in the face of the continuous generation of new data, there is a lack of effective online update mechanism, and the new labeled data cannot be used in time to optimize the model, resulting in a gradual decline in model performance over time.

[0007] In summary, the existing video content recognition methods have many shortcomings in processing unlabeled samples, capturing timing information, adapting to different scenarios and model updating. Therefore, an uncertain positive unlabeled learning method suitable for video content recognition is proposed to address the above problems. Summary of the invention

[0008] The purpose of the present invention is to provide an uncertain positive and negative labeling learning method suitable for video content recognition to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] An uncertain unlabeled learning method for video content recognition includes the following steps:

[0011] Step S1, data preprocessing: extracting video frames from the video surveillance system, converting them into standard image data, and dividing them into a labeled positive sample set and an unlabeled sample set;

[0012] Step S2, model preheating: The initial model is trained based on the improved non-negative positive unlabeled nnPU loss function. The loss function prevents the model from being excessively negatively biased to unlabeled data by balancing the contribution of positive samples and unlabeled samples.

[0013] Step S3, soft label generation: using the exponential moving average EMA strategy, dynamically integrating historical prediction information with the current model output, and generating smooth soft labels, the formula is: ,in, For the Samples at iteration Soft label, is the weight parameter, For the Samples at iteration Soft label, For the The model prediction results of the iteration;

[0014] Step S4: Adaptive uncertainty learning:

[0015] Uncertainty weight generation: By comparing the differences in soft labels at different training stages, the sample uncertainty score is calculated, and the uncertainty weight is generated based on the radial basis function kernel;

[0016] Segment optimization: Unlabeled samples are divided into soft positive samples, soft negative samples, and uncertain samples according to the soft label threshold, and weighted loss functions are designed for optimization respectively;

[0017] Step S5, iterative training: repeat steps S3 and S4, update soft labels and optimize model parameters until the model converges.

[0018] As a preferred solution, the improved nnPU loss function in step S2 is defined as: , used to balance the gradient contribution of positive samples and unlabeled samples, where is the improved non-negative positive unlabeled nnPU loss function value, is the prior probability of the positive sample, is the number of positive samples, is the number of unlabeled samples, is the cross entropy loss function, For the model Positive samples The prediction results, For the model Unlabeled samples prediction results.

[0019] As a preferred solution, the calculation formula of the uncertainty score in step S4 is: ,in, For sample The uncertainty score, For the Samples at iteration Soft label, For the Samples at iteration Soft label, and , .

[0020] As a preferred solution, the uncertainty weight is generated by the radial basis function kernel, and the formula is: ,in, For sample The uncertainty weight of is the bandwidth parameter, ranging from 0.05 to 0.3, which is used to adjust the sensitivity of the uncertainty weight.

[0021] As a preferred solution, the threshold value for segment optimization in step S4 is set as:

[0022] Soft positive samples: , corresponding to high confidence positive samples;

[0023] Soft negative samples: , corresponding to high confidence negative samples;

[0024] Uncertain samples: .7, corresponding to low confidence samples, requires conservative optimization, where Representative samples Soft label.

[0025] As a preferred solution, the total loss function of piecewise optimization is:

[0026] ,in, is the total loss function value of the segmented optimization, is the loss function value of the soft positive sample, is the loss function value of the soft negative sample, Hyperparameter, ranging from 0.3 to 0.7, is used to adjust the optimization intensity of uncertain samples. is the loss function value of uncertain samples; , used to strengthen soft positive sample learning, where is the number of samples in the soft positive sample set, For the model Soft positive samples The prediction results, For the Soft positive samples The uncertainty weight of , used to strengthen soft negative sample learning, where is the number of samples in the soft negative sample set; , used to limit the negative gradient of uncertain samples, where is the number of samples in the uncertain sample set, is the prior probability of the positive sample, is the number of positive samples, For the model Positive samples The prediction results, is the cross entropy loss function.

[0027] As a preferred solution, the model uses ResNet-50 as the backbone network, replaces the end with a binary classification fully connected layer, and extracts spatial features through a global average pooling layer.

[0028] As a preferred solution, the learning rate in the iterative training phase adopts an adaptive decay strategy. The initial learning rate is 0.001, and it decays to 0.1 times the original value every 100 epochs of training. The total number of training rounds is no less than 300.

[0029] It can be seen from the technical solution provided by the present invention that the uncertain positive and negative labeling learning method suitable for video content recognition provided by the present invention has the following beneficial effects:

[0030] Improve model prediction stability: Generate soft labels through the exponential moving average (EMA) strategy, dynamically integrate historical prediction information with current model output, effectively solve the problem of single prediction fluctuations, make the model's judgment of sample categories more stable and reliable, and provide a solid foundation for subsequent adaptive uncertainty learning;

[0031] Enhanced unlabeled sample processing capability: Based on the soft label difference, the prediction reliability is quantified. By calculating the uncertainty score and generating the uncertainty weight, combined with the segmented optimization strategy, high / low confidence samples can be distinguished. This can more reasonably utilize the information in the unlabeled samples, reduce noise interference, and improve the performance of the model when processing unlabeled samples, thereby achieving efficient learning even with limited labeled data;

[0032] Balance computing efficiency and model performance: ResNet-50 is used as the backbone network, combined with an adaptive learning rate strategy, to ensure that the model has a strong feature extraction capability while reasonably controlling computing resource consumption, so that the model can accurately identify various types of video content in video content recognition tasks and run efficiently, meeting the dual requirements of computing efficiency and model performance in actual application scenarios;

[0033] Wide range of scenario adaptability: It is suitable for a variety of video surveillance scenarios, including open kitchen monitoring, elevator safety monitoring, abnormal behavior detection in public places, etc.; it supports temporal context feature extraction of video frames, and has an online update mechanism after model deployment. It can dynamically integrate new annotated data with historical model parameters through incremental learning, and can adapt well to dynamically changing video surveillance environments, continuously optimize recognition effects, and ensure efficient completion of monitoring tasks in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the steps of an uncertain positive and negative labeling learning method suitable for video content recognition according to the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0037] like Figure 1As shown, an embodiment of the present invention provides an uncertain positive and negative labeling learning method suitable for video content recognition, comprising the following steps:

[0038] Step S1, data preprocessing: extracting video frames from the video surveillance system, converting them into standard image data, and dividing them into a labeled positive sample set and an unlabeled sample set;

[0039] Step S2, model preheating: The initial model is trained based on the improved non-negative positive unlabeled nnPU loss function. The loss function prevents the model from being excessively negatively biased to unlabeled data by balancing the contribution of positive samples and unlabeled samples.

[0040] Step S3, soft label generation: using the exponential moving average EMA strategy, dynamically integrating historical prediction information with the current model output, and generating smooth soft labels, the formula is: ,in, For the Samples at iteration Soft label, is the weight parameter, For the Samples at iteration Soft label, For the The model prediction results of the iteration;

[0041] Step S4: Adaptive uncertainty learning:

[0042] Uncertainty weight generation: By comparing the differences in soft labels at different training stages, the sample uncertainty score is calculated, and the uncertainty weight is generated based on the radial basis function kernel;

[0043] Segment optimization: Unlabeled samples are divided into soft positive samples, soft negative samples, and uncertain samples according to the soft label threshold, and weighted loss functions are designed for optimization respectively;

[0044] Step S5, iterative training: repeat steps S3 and S4, update soft labels and optimize model parameters until the model converges.

[0045] In this embodiment, step S1, data preprocessing, is the basic step of the entire uncertain positive and unlabeled learning method suitable for video content recognition, which provides high-quality, standardized data for subsequent model training and learning; this step mainly includes extracting video frames from the video surveillance system, converting them into standard image data, and dividing the labeled positive sample set and the unlabeled sample set. These key operations are specifically embodied as follows:

[0046] 1. Extract video frames from video surveillance system:

[0047] Source of video surveillance system: This method can be applied to a variety of video surveillance scenarios, such as open kitchen monitoring, elevator safety monitoring, or abnormal behavior detection in public places. The video surveillance systems in these scenarios will continuously record video data, which is the source of our raw data.

[0048] Extraction method: Use appropriate video processing technology to extract video frames from the video stream at certain time intervals; for example, a certain number of frames can be extracted per second to ensure the integrity and continuity of the data without making the data volume too large; the extracted video frames are the basic units for subsequent processing, they contain rich visual information and can reflect the scenes and behaviors recorded in the video;

[0049] 2. Convert the extracted video frames into standard image data:

[0050] Size normalization: Video frames captured by different video surveillance systems may have different sizes. In order to facilitate subsequent model processing, all video frames need to be adjusted to the same size. A common practice is to adjust them to a fixed resolution. For example, in common image recognition tasks, images may be adjusted to 224×224 pixels or other sizes suitable for model input.

[0051] Color space conversion: Ensure that all video frames use the same color space, such as RGB (red, green, blue) color space; this ensures that the model is consistent when processing data and avoids inaccurate feature extraction caused by color space differences;

[0052] Unified data format: Convert video frames into standard image data formats, such as JPEG, PNG, etc. These formats are commonly used in the field of image recognition and are convenient for subsequent storage, processing, and model input;

[0053] 3. Divide the labeled positive sample set and the unlabeled sample set:

[0054] Positive sample set annotation: The positive sample set contains annotated data of standard operations, normal status or safe behaviors. The annotators manually annotate some extracted video frames according to the specific application scenarios and task requirements to make it clear that these samples belong to the positive category. For example, in the open kitchen monitoring scenario, the positive sample may be a video frame of a chef wearing a mask and hat in a standard manner while cooking. In the elevator safety monitoring scenario, the positive sample may be a video frame of an elevator operating normally and passengers riding in a civilized manner.

[0055] Unlabeled sample set: In addition to the video frames marked as positive samples, the remaining video frames constitute the unlabeled sample set; these samples do not have clear category labels, and the model needs to infer their categories through uncertain positive unlabeled learning methods in the subsequent learning process; the unlabeled sample set usually contains a large amount of data, which can provide the model with richer information and help the model better generalize to different scenarios;

[0056] Through the above data preprocessing steps, the original video surveillance data can be converted into standard image data suitable for model training, and the positive sample set and unlabeled sample set can be reasonably divided, laying a solid foundation for subsequent model preheating, soft label generation, adaptive uncertainty learning and iterative training steps.

[0057] In this embodiment, step S2, model preheating, is a very critical step in the method of the present invention. Its main purpose is to train the initial model based on the improved non-negative positive unlabeled (nnPU) loss function to balance the contribution of positive samples and unlabeled samples and prevent the model from generating excessive negative bias on unlabeled data. The following describes in detail the definition of the loss function, the working principle and the training process, which are specifically embodied as follows:

[0058] 1. Improved non-negative positive unlabeled (nnPU) loss function definition:

[0059] The improved nnPU loss function is defined as follows: , used to balance the gradient contribution of positive samples and unlabeled samples, where The improved non-negative positive unlabeled nnPU loss function value is used to measure the difference between the model prediction result and the actual situation. It is the goal that needs to be minimized during the model training process. is the prior probability of the positive sample, which reflects the proportion of positive samples in the entire sample set and is an important pre-set parameter. is the number of positive samples, that is, the number of samples in the labeled positive sample set obtained in step S1. is the number of unlabeled samples, that is, the number of samples in the unlabeled sample set in step S1, is the cross entropy loss function, which is a commonly used loss function used to measure the difference in probability distribution between the model prediction results and the true labels. For the model Positive samples The prediction results, For the model Unlabeled samples The prediction results;

[0060] 2. The working principle of loss function:

[0061] Balance the contribution of positive samples and unlabeled samples: The design of this loss function takes into account the different characteristics of positive samples and unlabeled samples; the first half of the formula Focus on the loss of positive samples, through the prior probability of positive samples The loss of positive samples is weighted, emphasizing the importance of positive samples in training; the second half The loss of unlabeled samples is processed by subtracting part of the loss of positive samples to avoid excessive negative bias of the model on unlabeled samples, thereby achieving a balance between the contribution of positive samples and unlabeled samples.

[0062] Prevent excessive negative bias: In traditional positive unlabeled learning, the model may tend to judge all unlabeled samples as negative samples, resulting in excessive negative bias. The improved nnPU loss function uses the above design. When the negative loss of unlabeled samples is too large, it compares and adjusts the negative loss with the positive sample, making the model more cautious when processing unlabeled samples, thus avoiding the problem of excessive negative bias.

[0063] 3. Model training process based on loss function:

[0064] Select the initial model: The model in this invention uses ResNet-50 as the backbone network, replaces the end with a binary classification fully connected layer, and extracts spatial features through a global average pooling layer; this network structure has strong feature extraction and classification capabilities, and is suitable for video content recognition tasks;

[0065] Training process: During the training process, the positive sample set and the unlabeled sample set divided in step S1 are input into the initial model, and the model predicts each sample to obtain the prediction result. and ; Then calculate the loss value according to the improved nnPU loss function ; Then, use optimization algorithms (such as stochastic gradient descent, etc.) to update the model parameters according to the loss value, so that the loss value gradually decreases; through multiple iterative training, the model continuously learns the characteristics of positive samples and unlabeled samples, gradually improves its classification performance, and lays a good foundation for subsequent soft label generation and adaptive uncertainty learning.

[0066] In summary, step S2 trains the initial model through the improved nnPU loss function, which effectively balances the contribution of positive samples and unlabeled samples, prevents the model from being excessively negatively biased towards unlabeled data, and provides an important guarantee for the successful implementation of the entire uncertain positive and unlabeled learning method.

[0067] In this embodiment, step S3 uses an exponential moving average (EMA) strategy to generate smooth soft labels. The core of this strategy is to dynamically integrate historical prediction information with the current model output to solve the problem of single prediction fluctuations and provide more stable and reliable label information for subsequent adaptive uncertainty learning. The following describes in detail the strategy principle, formula parameters, specific generation process, and the significance of parameter settings:

[0068] 1. Exponential Moving Average (EMA) Strategy Principle:

[0069] During the model training process, a single prediction result may be affected by various factors, such as data noise, local fluctuations of the model, etc., which may lead to unstable prediction results. The EMA strategy introduces historical prediction information to smooth the prediction results of the current model, so that the soft label can more accurately reflect the true category tendency of the sample. It is based on the idea that the historical prediction results contain certain effective information, and weighted fusion of it with the current prediction results can reduce the randomness of a single prediction and improve the reliability of the label.

[0070] 2.Soft label generation formula and parameter meaning:

[0071] The soft label generation formula is: ,in, For the Samples at iteration The soft label is a comprehensive value that combines historical information and current prediction results, reflecting the At the iteration, the model Judgment of category belonging; is a weight parameter with a value range of 0.05 to 0.2. This parameter controls the prediction results of the current model. and historical soft tags The relative importance when generating new soft labels, The larger the value, the higher the weight of the current prediction result, and the faster the model responds to new information; conversely, The smaller the value, the higher the weight of historical information and the smoother the change of soft labels; For the Samples at iteration The soft label represents the model's evaluation of the sample in the previous iteration. judgment information; For the The model prediction result of the iteration, that is, the model At the iteration, the sample The predicted value of the output;

[0072] 3. Specific process of soft label generation:

[0073] Initial soft label setting: Initial soft label It is set to 0.5. This is because at the beginning of training, the model has not yet made a clear judgment on the classification of the sample. Setting the initial soft label to 0.5 means that the probability of the sample belonging to the positive class and the negative class is equal, which avoids the random fluctuation of the initial prediction from having too much impact on subsequent training.

[0074] Iteratively update soft labels: In each iteration of training, the model updates the samples Make a prediction and get ; Then according to the above formula, combined with the soft label of the previous iteration and weight parameters , calculate the soft label of this iteration ; As the number of iterations increases, the soft labels will be continuously updated, gradually integrating more historical information and current prediction information, becoming more stable and accurate;

[0075] 4. The significance of parameter settings:

[0076] Weight Parameters The value range of The range of 0.05 to 0.2 is a balanced choice; if If the value is too small, the update speed of the soft label will be very slow, and the model may not be able to adapt to the changes in the data in time. If the value is too large, the soft label will be too dependent on the current prediction result and easily affected by the fluctuation of a single prediction, thus losing the smoothing effect.

[0077] Initial soft label Set to 0.5: This setting allows the model to maintain neutral classification judgments for all samples in the early stages of training, avoiding the accumulation of deviations caused by initial random predictions, helping the model to train more stably and gradually learn the true features and category information of the samples;

[0078] In summary, step S3 generates smooth soft labels through the EMA strategy, which effectively solves the problem of single prediction fluctuations, provides a solid foundation for subsequent adaptive uncertainty learning based on soft labels, and helps to improve the performance and stability of the model in video content recognition tasks.

[0079] In this embodiment, step S4: adaptive uncertainty learning is one of the core links of the present invention, which mainly includes two sub-steps: uncertainty weight generation and segment optimization, which aims to quantify the prediction reliability based on the soft label difference, distinguish high / low confidence samples, reduce noise interference, and improve the performance of the model when processing unlabeled samples; the following is a detailed description:

[0080] 1. Uncertainty weight generation:

[0081] (1) Uncertainty score calculation:

[0082] The sample uncertainty score is calculated by comparing the soft label differences at different training stages. The calculation formula is: ;

[0083] in, For sample The uncertainty score reflects the sample The degree of change of soft labels in different training stages. The greater the degree of change, the more unstable the model's prediction of the sample is and the higher the uncertainty is.

[0084] For the Samples at iteration Soft label, For the Samples at iteration Soft label, and ; This setting is to ensure that the soft labels separated by a certain number of iterations are compared, to avoid the influence of short-term fluctuations, and to more accurately reflect the long-term prediction stability of the model for samples;

[0085] is a very small positive number, , used to avoid the situation where the denominator is zero and ensure the validity of the calculation;

[0086] (2) Generation of uncertainty weights:

[0087] According to the calculated uncertainty score, the uncertainty weight is generated based on the radial basis function kernel, and the formula is: ,in, For sample The uncertainty weight of the sample ranges from 0 to 1. The larger the weight, the lower the uncertainty of the sample and the more reliable the model's prediction of the sample. The smaller the weight, the higher the uncertainty of the sample.

[0088] is the bandwidth parameter, ranging from 0.05 to 0.3, which is used to adjust the sensitivity of the uncertainty weight. The larger the value, the less sensitive the uncertainty weight is to changes in the uncertainty score, and the weight changes relatively smoothly; The smaller the value, the more sensitive the weight is to changes in the uncertainty score, and the larger the change in the weight;

[0089] 2. Segment optimization:

[0090] (1) Sample division threshold:

[0091] Unlabeled samples are divided into soft positive samples, soft negative samples and uncertain samples according to the soft label threshold. The specific threshold is set as follows:

[0092] Soft positive samples: , corresponding to high-confidence positive samples, that is, the model has a high degree of confidence that these samples belong to the positive class;

[0093] Soft negative samples: , corresponding to high-confidence negative samples, indicating that the model has a high degree of confidence that these samples belong to the negative class;

[0094] Uncertain samples: .7, corresponding to low confidence samples, the model is not sure about the classification of these samples, and conservative optimization is required, where: Representative samples Soft label;

[0095] 2) Loss function of piecewise optimization:

[0096] The total loss function of piecewise optimization is: ;

[0097] in, The total loss function value optimized for each segment is the goal to be minimized during model training;

[0098] is the loss function value of the soft positive sample, and the calculation formula is , used to strengthen soft positive sample learning, where is the number of samples in the soft positive sample set, For the model Soft positive samples The prediction results, For the Soft positive samples The uncertainty weight of

[0099] is the loss function value of the soft negative sample, and the calculation formula is , used to strengthen soft negative sample learning, where is the number of samples in the soft negative sample set;

[0100] is the loss function value of the uncertain sample, and the calculation formula is , used to limit the negative gradient of uncertain samples, where is the number of samples in the uncertain sample set, is the prior probability of the positive sample, is the number of positive samples, For the model Positive samples The prediction results, is the cross entropy loss function; this loss function is used to limit the negative gradient of uncertain samples, avoid the model from over-learning uncertain samples, and play a role in conservative optimization;

[0101] Hyperparameter, with a value range of 0.3 to 0.7, is used to adjust the optimization intensity of uncertain samples. The larger the value, the greater the optimization effort for uncertain samples; otherwise, the smaller the value;

[0102] Through the above-mentioned process of uncertainty weight generation and segmented optimization, the method of the present invention can better handle the uncertainty in unlabeled samples, reduce noise interference, and improve the performance and stability of the model in video content recognition tasks.

[0103] In this embodiment, the model uses ResNet-50 as the backbone network, the end is replaced by a binary classification fully connected layer, and the spatial features are extracted through a global average pooling layer;

[0104] Specifically, the present invention adopts a specific network architecture to realize the video content recognition task, specifically using ResNet-50 as the backbone network, replacing the end with a binary classification fully connected layer, and extracting spatial features through a global average pooling layer; the various parts of the architecture are described in detail below:

[0105] 1.ResNet-50 backbone network:

[0106] (1) Introduction to ResNet-50:

[0107] ResNet (Residual Network) is a deep convolutional neural network proposed by He Kaiming and others from Microsoft Research in 2015. ResNet-50 is a specific model in the ResNet series, which contains 50 layers of neural networks, mainly composed of convolutional layers, batch normalization layers, activation function layers, and pooling layers. Its core innovation is the introduction of residual blocks, which solve the gradient vanishing and gradient exploding problems that occur in the training process of deep neural networks through skip connections, allowing the network to train deeper layers and learn more complex and abstract features.

[0108] (2) Role in the present invention:

[0109] In the video content recognition task of the present invention, ResNet-50 is used as the backbone network to extract features from the input video frame images; it can automatically learn various visual features in the image, from the bottom-level features such as edges and textures to the high-level features such as object shapes and semantic information; these features will serve as the basis for subsequent classification tasks and provide rich information for the model to judge the video content;

[0110] 2. Global average pooling layer:

[0111] (1) The principle of global average pooling layer:

[0112] Global Average Pooling (GAP) is a pooling operation that performs average pooling on each channel of the feature map and compresses the two-dimensional feature map of each channel into a scalar value; specifically, for a size of The feature map of and are the height and width of the feature map, respectively. is the number of channels), global average pooling will be performed on each channel elements to find the average value, and finally get a length of A vector of

[0113] (2) Role in the present invention:

[0114] Dimensionality reduction and feature aggregation: The feature maps extracted by ResNet-50 usually have high dimensions. The global average pooling layer can reduce the dimensionality of these high-dimensional feature maps, reduce the number of model parameters, and reduce the computational complexity. At the same time, it can aggregate the spatial information in the feature map and integrate the feature information of each channel to obtain a more representative feature vector.

[0115] Improve the generalization ability of the model: The global average pooling layer has no learnable parameters, which avoids the overfitting problem that may be caused by the fully connected layer. It makes the model more robust to the spatial transformation of the input image by averaging the entire feature map, thereby improving the generalization ability of the model.

[0116] 3. Two-class fully connected layer:

[0117] (1) The principle of the fully connected layer:

[0118] A fully connected layer is a common layer in a neural network. It connects all input neurons to all output neurons. Each connection has a corresponding weight. The function of a fully connected layer is to linearly transform and combine input features, mapping low-dimensional features to high-dimensional space, thereby achieving tasks such as classification or regression.

[0119] (2) Role in the present invention:

[0120] In the present invention, the feature vectors output by the ResNet-50 backbone network and the global average pooling layer are input into a binary classification fully connected layer; since the video content recognition task of the present invention is a binary classification problem (for example, distinguishing whether the behavior in the video is standard or not, whether the state is normal or abnormal, etc.), the output dimension of the binary classification fully connected layer is 2, corresponding to two categories respectively; this layer will calculate the score of each category according to the input feature vector, through the learned weights and biases, and then convert the score into a probability value through an activation function such as softmax, and finally the model determines the category to which the input video frame belongs according to the probability value;

[0121] In summary, the present invention adopts ResNet-50 as the backbone network for feature extraction, uses the global average pooling layer for feature dimensionality reduction and aggregation, and finally realizes the binary classification task of video content through the binary classification fully connected layer. This network architecture combines the powerful feature learning ability of the deep convolutional network and the classification ability of the fully connected layer. At the same time, the global average pooling layer improves the efficiency and generalization ability of the model, which is suitable for the uncertain positive and unlabeled learning method of video content recognition.

[0122] As a preferred solution, the learning rate in the iterative training phase adopts an adaptive decay strategy, with an initial learning rate of 0.001, which decays to 0.1 times the original value every 100 epochs of training, and a total training round of no less than 300 times;

[0123] Specifically, in the iterative training stage of the present invention, the setting of the learning rate plays a key role in the training effect of the model; an adaptive decay strategy is adopted here, which is specifically set to an initial learning rate of 0.001, and decays to 0.1 times the original value every 100 epochs of training, and the total number of training rounds is not less than 300 times. The key points of this strategy will be elaborated in detail below;

[0124] 1. Importance of learning rate:

[0125] The learning rate determines the step size of the model parameter update during the training process. If the learning rate is set too large, the model may skip the optimal solution in the parameter space, resulting in failure to converge, and the loss function value fluctuates or even increases. If the learning rate is set too small, the model training speed will be very slow, and it will take a lot of time and computing resources to achieve good performance. Therefore, reasonably adjusting the learning rate is an important means to optimize model training.

[0126] 2. Adaptive attenuation strategy:

[0127] (1) Initial learning rate setting:

[0128] The initial learning rate is set to 0.001. In the early stage of training, the model has not yet fully learned the characteristics and distribution of the data. A larger learning rate can enable the model to move quickly in the parameter space, quickly explore the roughly correct parameter direction, and accelerate the convergence of the model. This initial value is determined through experiments and experience. In the video content recognition task involved in the present invention, an initial learning rate of 0.001 can enable the model to effectively learn data features at the beginning, laying a foundation for subsequent optimization.

[0129] (2) Attenuation mode:

[0130] After every 100 epochs of training, the learning rate decays to 0.1 times of the original value. As the training progresses, the model gradually approaches the optimal solution. If a larger learning rate is continued to be used, the model may oscillate back and forth near the optimal solution and it is difficult to converge accurately. Through this exponential decay method, the learning rate will gradually decrease with the increase of training rounds. For example, after the first 100 epochs of training, the learning rate becomes 0.001×0.1=0.0001; after the second 100 epochs of training, the learning rate becomes 0.0001×0.1=0.00001, and so on. This decay method enables the model to learn quickly in the early stage of training, and to adjust parameters more finely in the later stage to improve the accuracy of the model.

[0131] 3. Total training rounds required:

[0132] The total number of training rounds is stipulated to be no less than 300 times. This is to ensure that the model has enough time to learn the complex features and patterns of the data. In the video content recognition task, the features of the data are relatively complex, and the model needs to go through multiple iterations to fully learn the various patterns in the video frames to accurately distinguish different video content categories. If the number of training rounds is too small, the model may not converge to a good solution, resulting in poor performance of the model and inability to accurately recognize the video content. By setting the number of training rounds to be no less than 300 times, the model is given sufficient training time, so that it can use appropriate learning rates to optimize parameters at different stages, thereby improving the accuracy and stability of the model in the video content recognition task.

[0133] In summary, the learning rate adaptive attenuation strategy and the total training round requirement in the iterative training phase of the present invention are carefully designed according to the characteristics of the video content recognition task, aiming to balance the training speed and convergence accuracy of the model so that the model can learn accurate feature representation on complex data, thereby achieving efficient and accurate video content recognition.

[0134] In this embodiment, the video surveillance system includes open kitchen monitoring, elevator safety monitoring or abnormal behavior detection scenarios in public places, and the positive sample set contains labeled data of standard operations, normal states or safe behaviors;

[0135] Furthermore, in the technical application scenarios of the present invention, the video surveillance system covers a variety of different fields, including open kitchen monitoring, elevator safety monitoring, and abnormal behavior detection scenarios in public places; these scenarios each have unique characteristics and requirements, and the uncertain positive and unlabeled learning method of the present invention can be applied to these diverse scenarios, and the corresponding monitoring purpose can be achieved by identifying the video content; at the same time, the labeled data contained in the positive sample set plays a key guiding role in the training and learning of the model; these aspects will be described in detail below;

[0136] 1. Open kitchen and stove monitoring scene:

[0137] (1) Scenario description:

[0138] The purpose of open kitchen monitoring is to ensure food safety in the catering industry, allowing consumers to intuitively see the operation status in the kitchen. In this scenario, cameras are installed in various key locations in the kitchen to record the chef's cooking process, food processing, and kitchen hygiene conditions in real time. The video monitoring system will continuously collect a large amount of video data, which contains rich information, including operation scenes that comply with food safety regulations, as well as some behavior scenes that do not comply with regulations.

[0139] (2) Positive sample set annotation:

[0140] In the open kitchen monitoring scenario, the positive sample set mainly includes the labeled data of standard operations; for example, the chef wears a mask, hat and gloves correctly to perform cooking operations; the washing, cutting and cooking of ingredients meet the hygiene standards and operating specifications; the kitchen cleaning work is completed on time and the hygiene conditions are good. These positive sample data provide a learning standard for the model, enabling the model to identify which operations meet the food safety requirements, so as to judge whether the current picture belongs to the standard operation in the subsequent video content recognition;

[0141] 2. Elevator safety monitoring scenario:

[0142] (1) Scenario description:

[0143] Elevator safety monitoring is related to people's life safety, and it is necessary to monitor the operation status of the elevator and the behavior of passengers in real time. The monitoring cameras are installed in the elevator car and the elevator machine room to record the operation process of the elevator, the entry and exit of passengers, and whether the elevator has any faults. There may be many situations during the operation of the elevator, such as normal operation, abnormal jitter, fault alarm, and uncivilized or dangerous behavior of passengers, all of which need to be identified and judged by the video monitoring system.

[0144] (2) Positive sample set annotation:

[0145] For elevator safety monitoring scenarios, the positive sample set contains labeled data in normal states; for example, the elevator runs smoothly, the doors open and close normally, and there is no abnormal noise or vibration; passengers ride the elevator in a civilized manner, without jumping in the elevator, leaning on the door, or other dangerous behaviors; the various indicator lights of the elevator are displayed normally, and the equipment operating parameters are within the normal range. These images are labeled as positive samples. By learning these positive sample data, the model can establish an understanding of the normal operating status of the elevator and the safety behavior of passengers, so as to accurately identify abnormal situations in actual monitoring.

[0146] 3. Abnormal behavior detection scenarios in public places:

[0147] (1) Scenario description:

[0148] Public places such as shopping malls, stations, and squares are crowded with people and complex situations, which are prone to various abnormal behaviors. Video surveillance systems deploy multiple cameras in these places to cover different areas and collect video data of people's activities in all directions. In these videos, there may be abnormal behaviors such as fighting, theft, and sudden fainting, as well as a large number of normal activities of people.

[0149] (2) Positive sample set annotation:

[0150] In the scenario of abnormal behavior detection in public places, the positive sample set contains the labeled data of safe behaviors. For example, video footage of people walking in an orderly manner, waiting in line, communicating normally, etc. are labeled as positive samples. These positive samples help the model learn the characteristic patterns of normal behaviors in public places. When the model detects behaviors that are significantly different from the positive sample characteristics in the real-time monitoring video, it can determine that abnormal behaviors may have occurred, and thus issue an alarm in time.

[0151] To sum up, the different scenarios of the video surveillance system involved in the present invention each have clear monitoring objectives and characteristics; the positive sample set accurately labels the data of standard operations, normal states or safe behaviors according to the specific requirements of each scenario, which provides an important training basis for the uncertain positive and unlabeled learning method, enabling the model to accurately identify the video content of different scenarios and realize effective monitoring and anomaly detection functions.

[0152] In this embodiment, the unlabeled sample set extracts temporal context features of video frames through a sliding window, and the model captures dynamic temporal information through a 3D convolutional layer or an LSTM module;

[0153] Furthermore, in the present invention, extracting temporal context features from unlabeled sample sets and capturing dynamic temporal information by the model are important steps to improve the video content recognition effect. The specific operation steps are described in detail below:

[0154] The unlabeled sample set extracts temporal context features of video frames through a sliding window:

[0155] 1. Data preparation:

[0156] First, it is necessary to obtain the video data corresponding to the unlabeled sample set; these video data are composed of unlabeled video frames extracted from the video surveillance system and converted into standard image data in the previous data preprocessing step;

[0157] 2. Determine the sliding window parameters:

[0158] Window size: Determine the number of video frames contained in the sliding window based on the characteristics of the video content and task requirements. For example, if you want to capture fast-paced action changes in a short period of time, the window size can be set smaller, such as 5-10 frames; if you focus on behavioral patterns over a longer period of time, the window size can be appropriately increased, such as 20-30 frames.

[0159] Step size: The step size determines the number of frames that the sliding window moves each time; a smaller step size can extract more dense timing information, but will increase the amount of calculation; a larger step size can reduce the amount of calculation, but may lose some detail information; generally speaking, the step size can be set to 1-5 frames;

[0160] 3. Sliding window operation:

[0161] According to the determined window size and step size, a sliding operation is performed on the video frame sequence; a group of video frames covered by each sliding window constitutes a time series sample; for example, assuming that the window size is 10 frames and the step size is 1 frame, then starting from the first frame, 10 consecutive frames are selected as the first time series sample; then the window moves backward by 1 frame, and the 2nd to 11th frames are selected as the second time series samples, and so on, until the window slides over the entire video frame sequence;

[0162] 4. Feature extraction:

[0163] For each video frame in the sliding window, extract its features; use a pre-trained convolutional neural network (such as the ResNet-50 mentioned in the step) to extract features from each video frame to obtain a feature vector for each frame; then concatenate these feature vectors in chronological order to form a feature representation of the time series sample, which contains the temporal context information between video frames;

[0164] The model captures dynamic timing information through 3D convolutional layers or LSTM modules:

[0165] Use 3D convolutional layers to capture dynamic timing information:

[0166] 1. Construct 3D convolution layer:

[0167] Add a 3D convolution layer to the model. Unlike traditional 2D convolution layers, 3D convolution layers perform convolution operations not only in the spatial dimension (height and width), but also in the temporal dimension, and can directly process video data containing temporal information. A 3D convolution layer contains multiple 3D convolution kernels, each of which performs convolution operations on the input temporal sample features in space and time.

[0168] 2. Convolution operation:

[0169] The time series sample features extracted by the sliding window are input into the 3D convolution layer; the 3D convolution kernel slides on the input data to calculate the convolution result; the convolution operation extracts the spatial and temporal feature patterns of the video data, such as the movement trajectory of the object, the change trend of the action, etc.

[0170] 3. Activation function and pooling operation:

[0171] After the 3D convolution layer, an activation function (such as ReLU) is usually added to introduce nonlinearity and enhance the expressiveness of the model; then a 3D pooling layer (such as 3D max pooling) can be used to downsample the convolution result to reduce the data dimension while retaining important feature information;

[0172] 4. Subsequent processing:

[0173] After the 3D convolution layer and related operations, the obtained features can be further input into subsequent network layers (such as fully connected layers) for classification or other tasks;

[0174] Use LSTM modules to capture dynamic timing information:

[0175] 1. Build LSTM module:

[0176] Add LSTM (Long Short-Term Memory) module to the model. LSTM is a special recurrent neural network that can effectively process sequence data and solve the gradient vanishing and gradient exploding problems of traditional recurrent neural networks when processing long sequences. LSTM module contains multiple LSTM units, each of which has an input gate, a forget gate, and an output gate, and can selectively memorize and forget information.

[0177] 2. Sequence input:

[0178] The time series sample features extracted by the sliding window are input into the LSTM module in chronological order; the LSTM unit will update its own state according to the current input and the hidden state of the previous moment, and output the hidden state of the current moment;

[0179] 3. Status updates and information delivery:

[0180] At each time step, the input gate of the LSTM unit determines how much of the current input information will be added to the unit state; the forget gate determines how much of the unit state at the previous moment will be retained; the output gate determines how much of the current unit state will be passed as output to the next layer; in this way, the LSTM module can capture long-term dependencies and dynamic timing information in the video frame sequence;

[0181] 4. Output processing:

[0182] The output of the last time step of the LSTM module can be used as the feature representation of the entire time series sample and input into the subsequent network layer for classification or other task processing;

[0183] Through the above steps, the temporal context features of the unlabeled sample set can be effectively extracted, and the model can capture the dynamic temporal information in the video data through the 3D convolutional layer or LSTM module, thereby improving the accuracy and performance of video content recognition.

[0184] In this embodiment, the method of the present invention further includes an online update mechanism after model deployment, which dynamically integrates newly annotated data with historical model parameters through incremental learning;

[0185] Specifically: In the video content recognition system constructed by the present invention, the online update mechanism after model deployment is crucial, and incremental learning is the core means to achieve this mechanism, which aims to dynamically integrate new annotated data with historical model parameters and continuously optimize model performance;

[0186] 1. Acquisition and organization of new annotation data:

[0187] Data source: After the model is actually deployed in various video surveillance scenarios (such as open kitchen monitoring, elevator safety monitoring, or abnormal behavior detection in public places), new video data will be continuously generated over time and as business needs change; part of this data will be manually annotated to form new annotated data; the annotators will annotate the video frames according to the standards of standardized operation, normal state, or safe behavior in each scenario to clarify their category information;

[0188] Data collation: Newly annotated data needs to be preprocessed and collated before entering the model update process. This includes unifying the format of the annotation information to ensure that it is consistent with the data format during model training. At the same time, the video frame data is preprocessed with the same standards as the training data, such as size normalization, color space conversion, etc., to ensure data consistency and availability.

[0189] 2. Principle of incremental learning algorithm:

[0190] Knowledge retention and updating: The core goal of the incremental learning algorithm is to avoid forgetting the knowledge learned by the historical model while using the newly annotated data to improve the model performance. It achieves this goal by comprehensively considering the historical model parameters and the newly annotated data. When the newly annotated data is input, the algorithm does not simply retrain the entire model, but adjusts the existing model parameters.

[0191] Parameter adjustment strategy: Based on optimization algorithms such as gradient descent, the incremental learning algorithm calculates the gradient of the newly annotated data on the current model; according to the gradient information of the new data and the protection strategy of the historical model parameters, the weight parameters of the model are adjusted; for example, different "consolidation weights" are set for the historical model parameters using technologies such as elastic weight consolidation (EWC), so that when updating parameters, those parameters that are critical to historical tasks are adjusted to a smaller extent, while the parameters related to the newly annotated data are updated more flexibly to balance the learning of new and old knowledge;

[0192] 3. Dynamic Fusion Process:

[0193] Data loading and fusion preparation: Load the sorted new annotated data into the incremental learning module; at the same time, obtain its parameter information from the historical model, including the weight matrix and bias vector of each layer; the incremental learning module will analyze these historical parameters and new data to determine the direction and magnitude of the update;

[0194] Model parameter update: Based on the incremental learning algorithm, the newly annotated data is gradually integrated into the model. In each update iteration, the model parameters are updated according to the gradient of the new data and the protection strategy of the historical parameters. For example, in the fully connected layer of the neural network, the weight parameters are updated according to the back-propagation gradient of the new data and the adjustment amount determined by the EWC and other strategies. This process is repeated until the newly annotated data is fully integrated into the model and the dynamic fusion of the model parameters is completed.

[0195] Model verification and evaluation: After completing a round of fusion update of newly annotated data and historical model parameters, the updated model needs to be verified and evaluated; use a part of the new data that did not participate in the update and the verification set in the historical data to test the performance of the model, such as the accuracy and recall rate in the video content recognition task; if the model performance meets expectations, the updated model will be deployed in actual applications; if the performance does not meet the standards, it may be necessary to adjust the parameters of the incremental learning algorithm or re-perform the data annotation and update process;

[0196] Through the above-mentioned online update mechanism and the dynamic fusion process achieved by incremental learning, the model of the present invention can continuously optimize its own performance as new annotated data is continuously generated, better adapt to complex and changeable video surveillance scenarios, and improve the accuracy and reliability of video content recognition.

[0197] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An uncertain unlabeled learning method for video content recognition, characterized by: The following steps are involved: Step S1, data preprocessing: extracting video frames from the video surveillance system, converting them into standard image data, and dividing them into a labeled positive sample set and an unlabeled sample set; Step S2, model preheating: training the initial model based on the improved non-negative positive unlabeled nnPU loss function, wherein the loss function prevents the model from being excessively negatively biased to unlabeled data by balancing the contribution of positive samples and unlabeled samples; Step S3, soft label generation: using the exponential moving average EMA strategy, dynamically integrating historical prediction information with the current model output, and generating smooth soft labels, the formula is: ,in, For the Samples at iteration Soft label, is the weight parameter, For the Samples at iteration Soft label, For the The model prediction results of the iteration; Step S4: Adaptive uncertainty learning: Uncertainty weight generation: By comparing the differences in soft labels at different training stages, the sample uncertainty score is calculated, and the uncertainty weight is generated based on the radial basis function kernel; Segment optimization: Unlabeled samples are divided into soft positive samples, soft negative samples, and uncertain samples according to the soft label threshold, and weighted loss functions are designed for optimization respectively; Step S5, iterative training: repeat steps S3 and S4, update soft labels and optimize model parameters until the model converges.

2. The uncertain positive and negative labeling learning method for video content recognition according to claim 1, characterized in that: The improved nnPU loss function in step S2 is defined as: , used to balance the gradient contribution of positive samples and unlabeled samples, where is the improved non-negative positive unlabeled nnPU loss function value, is the prior probability of the positive sample, is the number of positive samples, is the number of unlabeled samples, is the cross entropy loss function, For the model Positive samples The prediction results, For the model Unlabeled samples prediction results.

3. The uncertain positive and negative labeling learning method for video content recognition according to claim 1, characterized in that: The calculation formula of the uncertainty score in step S4 is: ,in, For sample The uncertainty score, For the Samples at iteration Soft label, For the Samples at iteration Soft label, and , .

4. The uncertain positive and negative labeling learning method for video content recognition according to claim 3 is characterized by: The uncertainty weight is generated by the radial basis function kernel, and the formula is: ,in, For sample The uncertainty weight of is the bandwidth parameter, ranging from 0.05 to 0.3, which is used to adjust the sensitivity of the uncertainty weight.

5. The uncertain positive and negative labeling learning method for video content recognition according to claim 1, characterized in that: The threshold value of the segmented optimization in step S4 is set as: Soft positive samples: , corresponding to high confidence positive samples; Soft negative samples: , corresponding to high confidence negative samples; Uncertain samples: .7, corresponding to low confidence samples, requires conservative optimization, where Representative samples Soft label.

6. The uncertain positive and negative labeling learning method for video content recognition according to claim 1, characterized in that: The total loss function of the piecewise optimization is: ,in, is the total loss function value of the segmented optimization, is the loss function value of the soft positive sample, is the loss function value of the soft negative sample, Hyperparameter, ranging from 0.3 to 0.7, is used to adjust the optimization intensity of uncertain samples. is the loss function value of uncertain samples; , used to strengthen soft positive sample learning, where is the number of samples in the soft positive sample set, For the model Soft positive samples The prediction results, For the Soft positive samples The uncertainty weight of , used to strengthen soft negative sample learning, where is the number of samples in the soft negative sample set; , used to limit the negative gradient of uncertain samples, where is the number of samples in the uncertain sample set, is the prior probability of the positive sample, is the number of positive samples, For the model Positive samples The prediction results, is the cross entropy loss function.

7. The uncertain positive and negative labeling learning method for video content recognition according to claim 1, characterized in that: The model uses ResNet-50 as the backbone network, the end is replaced by a binary classification fully connected layer, and spatial features are extracted through a global average pooling layer.

8. The uncertain positive and negative labeling learning method for video content recognition according to claim 1, characterized in that: The learning rate in the iterative training phase adopts an adaptive decay strategy, with an initial learning rate of 0.001, which decays to 0.1 times of the original value every 100 epochs of training, and the total number of training rounds is no less than 300 times.

Images