A monitoring data analysis system based on deep learning

Abstract

The present invention discloses a monitoring data analysis system based on deep learning, specifically related to the technical field of deep learning, including a preliminary recognition module, a performance verification module, a performance classification module, a labeling analysis module, a confidence level classification module, and an adoption module. The preliminary recognition module obtains the application information of the deep learning model and preliminarily judges performance hidden dangers; the performance verification module verifies the performance and obtains a comprehensive performance value; the performance classification module classifies the model quality based on the comprehensive performance value; the labeling analysis module performs incremental and re-learning labeling operations on medium and low-quality models; the confidence level classification module analyzes and obtains a comprehensive confidence level coefficient; the adoption module classifies the quality of the labeling operation based on the confidence level coefficient and adopts high-quality data; the present invention can effectively screen out high-quality labeling operations, eliminate low-quality labeled data, thereby significantly improving the overall quality of the training data and improving the performance of the deep learning model in monitoring data analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and more specifically, to a monitoring data analysis system based on deep learning. Background Art

[0002] With the wide application of monitoring technologies, monitoring points are arranged in scenarios such as office buildings, residential areas, shopping malls, and hospitals for round-the-clock monitoring, generating a large amount of video data. The massive monitoring video data needs to be processed and analyzed efficiently. Traditional monitoring data analysis methods usually rely on manual monitoring and simple image processing technologies, suffering from problems such as low efficiency, high false alarm rates, and slow response speeds. The rise of deep learning technologies has provided new solutions for monitoring data analysis. By constructing and training deep neural networks, automatic recognition of targets, behavior analysis, and anomaly detection in monitoring videos can be achieved.

[0003] However, the performance of deep learning models highly depends on the quality of training data. In practical applications, the acquisition of labeled data often relies on manual labeling, which inevitably introduces human errors and inconsistencies. Low-quality labeled data will lead to a decline in model performance and increase the risks of false alarms and missed detections. How to effectively evaluate and screen high-quality labeling operations to ensure the high quality of training data has become a key issue in improving the performance of monitoring data analysis systems. Therefore, a monitoring data analysis system based on deep learning is proposed herein. Summary of the Invention

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A monitoring data analysis system based on deep learning, comprising a preliminary recognition module, a performance verification module, a performance classification module, a labeling analysis module, a confidence classification module, and an adoption module;

[0006] The preliminary recognition module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the present stage, and then conduct preliminary analysis. Based on the results of the preliminary analysis, it is preliminarily determined whether there are potential hazards of insufficient performance when the deep learning model that has been trained at the present stage is applied to the target scenario;

[0007] The performance verification module is used to perform performance verification operations on the deep learning model that has been trained at the present stage and has potential hazards of insufficient performance to obtain a comprehensive performance value;

[0008] The performance classification module divides the deep learning model that has been trained at the present stage into a high-quality deep learning model, a medium-quality deep learning model, and a low-quality deep learning model based on the comprehensive performance value;

[0009] The annotation analysis module is used to perform annotation operations in an incremental learning manner for medium-quality deep learning models and perform annotation operations in a re-learning manner for low-quality deep learning models;

[0010] The confidence division module is used to perform quality analysis on the annotation operation process, obtain the annotation consistency quality index and the review feedback quality index respectively, and then comprehensively analyze the annotation consistency quality index and the review feedback quality index to obtain the comprehensive confidence coefficient and transmit it to the adoption module;

[0011] The adoption module is used to divide the annotation operation into high-quality operations and low-quality operations based on the comprehensive confidence coefficient and adopt the annotation data of high-quality operations.

[0012] In a preferred embodiment, the preliminary identification module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the present stage, and then the preliminary analysis refers to:

[0013] Under a fixed time window, such as one day, one week or one month, obtain the application information of the target scenario of the deep learning model that has been trained at the present stage, divide the fixed time window into multiple consecutive and equal sub-windows, and then obtain the anomaly detection rate of each sub-window respectively to obtain the time series X. Calculate the average value and standard deviation of the time series X respectively; the average value represents the overall level of the anomaly detection rate of the model within the entire fixed time window. A higher average value indicates that the model performs well in detecting abnormal events. It can be used to preliminarily evaluate the overall performance of the model and help understand the detection ability of the model in the target scenario. The standard deviation represents the degree of fluctuation of the anomaly detection rate. A smaller standard deviation indicates that the detection performance of the model is relatively stable in different sub-windows; a larger standard deviation indicates that there are large fluctuations in the detection performance. The standard deviation can help identify the performance of the model at different times.

[0014] The anomaly detection rate refers to the ratio of the total number of detected abnormal events to the total number of actually occurring abnormal events.

[0015] In a preferred embodiment, based on the results of the preliminary analysis, the preliminary judgment is made to determine whether there is a hidden danger of insufficient performance when the deep learning model that has been trained at the present stage is applied to the target scenario, which refers to:

[0016] Obtain the mean and standard deviation of the computed time series X, and then compare the mean of the time series X with the preset anomaly detection rate average parameter threshold in this target scenario, and compare the standard deviation of the time series X with the preset anomaly detection rate fluctuation parameter threshold in this target scenario. If the mean of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold in this target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold in this target scenario, a normal signal is generated. If the mean of the time series X is not less than or equal to the preset anomaly detection rate average parameter threshold in this target scenario and the standard deviation of the time series X is not less than or equal to the preset anomaly detection rate fluctuation parameter threshold in this target scenario, an abnormal signal is generated. When the abnormal signal is generated, it indicates that there is a potential problem with the performance of the deep learning model that has been trained and completed at the current stage when applied to the target scenario.

[0017] In a preferred embodiment, the performance verification operation on the deep learning model that has been trained and completed at the current stage and has a potential problem with performance refers to:

[0018] Collect a new set of validation data from the real scenario and ensure that the validation data has been accurately labeled. Use the deep learning model that has been trained and completed at the current stage and has a potential problem with performance to predict the validation data set to obtain the prediction results.

[0019] In a preferred embodiment, the acquisition logic of the comprehensive performance value is as follows:

[0020] Obtain the corresponding data of each preset performance metric according to the prediction results, and then calculate the mean of each performance metric and label it as PX i. By using the weighted average method, combine multiple performance metrics and allocate the weight of each performance metric according to the importance of the specific application scenario. The formula for calculating the comprehensive performance value is: CPS represents the comprehensive performance value, n is the total number of preset performance metrics, Wi represents the weight corresponding to the preset performance metric i, and the sum of all weights Wi is equal to one.

[0021] In a preferred embodiment, dividing the deep learning model that has been trained and completed at the current stage into a high-quality deep learning model, a medium-quality deep learning model, and a low-quality deep learning model based on the comprehensive performance value refers to:

[0022] Obtain the comprehensive performance value of the deep learning model that has completed training at the current stage. Compare the comprehensive performance value with a preset benchmark comprehensive performance threshold. Classify the deep learning models with a comprehensive performance value higher than the preset benchmark comprehensive performance threshold as high-quality deep learning models, classify the deep learning models with a comprehensive performance value lower than the product value of the preset benchmark comprehensive performance threshold and p as low-quality deep learning models, where p is the preset percentage coefficient in this target scenario, and classify the deep learning models with a comprehensive performance value not lower than the product value of the preset benchmark comprehensive performance threshold and p and not higher than the preset benchmark comprehensive performance threshold as medium-quality deep learning models.

[0023] In a preferred implementation, the acquisition logic of the annotation consistency quality index is as follows:

[0024] Collect the annotation results of the same dataset containing M samples from N annotators. For each sample, calculate the consistency among all annotators. Assume that the total number of annotation categories is C, and construct a consistency matrix, where each element represents the consistency of two annotators for a sample. For each annotator i, calculate the average value of its consistency with all other annotators. The calculation formula is: A i represents the average value of the consistency of annotator i with all other annotators, I represents the consistency of annotator i and j for all samples, and then sum the average values of the consistency of all annotators to obtain the overall consistency. The calculation formula is: A total represents the overall consistency;

[0025] Calculate the consistency coefficient Fleiss′Kappa. The calculation formula is: P o represents the observed proportion of consistency, that is, the proportion of the number of times the annotators actually reach an agreement in the total number of times, P e represents the expected proportion of consistency, that is, the probability of reaching an agreement assuming that the annotation is random; k represents the consistency coefficient;

[0026] P o The calculation formula is:

[0027] P e The calculation formula is:

[0028] M is the total number of samples, N is the total number of annotators, C is the total number of categories, n mc is the number of annotators who annotate sample m as category c;

[0029] Perform a weighted average on the consistency coefficient k and the overall consistency A total to obtain the annotation consistency quality index.

[0030] In a preferred embodiment, the acquisition logic of the audit feedback quality index is as follows:

[0031] Preset multiple indicators for evaluating the quality of audits and feedback, collect data corresponding to the multiple indicators during the audit and feedback process and construct the current audit feedback quality vector, obtain the standard audit feedback quality vector, calculate the similarity between the current audit feedback quality vector and the standard audit feedback quality vector, and the value of the audit feedback quality index is the same as the similarity value.

[0032] In a preferred embodiment, the acquisition logic of the comprehensive confidence coefficient is as follows:

[0033] Substitute the annotation consistency quality index and the audit feedback quality index into a pre-trained machine learning model for comprehensive analysis, and the output data of the machine learning model is the comprehensive confidence coefficient.

[0034] Technical effects and advantages of the present invention:

[0035] By calculating and analyzing the comprehensive confidence coefficient, the present invention can effectively screen out high-quality annotation operations and eliminate low-quality annotation data, thereby significantly improving the overall quality of training data and enhancing the performance of the deep learning model in monitoring data analysis. Using high-quality annotation data for incremental learning or retraining can improve the accuracy, robustness, and generalization ability of the deep learning model, making it perform better in object recognition, behavior analysis, and anomaly detection of monitoring videos.

[0036] The present invention uses a machine learning model for comprehensive evaluation, which can quickly and accurately identify high-quality annotation operations, reduce the workload of manual audits, improve the overall evaluation efficiency, and thus achieve real-time analysis and response of monitoring data. Through automated annotation quality evaluation and screening, the cost of re-annotation and model tuning caused by low-quality annotation data can be reduced, thereby reducing the total cost of the monitoring data analysis system. Moreover, high-quality annotation data can be systematically evaluated and adopted to ensure the stability and reliability of the deep learning model in different monitoring scenarios and enhance the overall performance of the monitoring data analysis system. Description of the Drawings

[0037] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with the accompanying drawings;

[0038] Figure 1 It is the schematic diagram of a monitoring data analysis system based on deep learning in the present invention. Detailed Embodiments

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] Embodiment 1

[0041] A monitoring data analysis system based on deep learning includes a preliminary recognition module, a performance verification module, a performance classification module, a labeling analysis module, a confidence level classification module, and an adoption module; the preliminary recognition module, the performance verification module, the performance classification module, the labeling analysis module, the confidence level classification module, and the adoption module are communicatively connected.

[0042] The preliminary recognition module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the present stage, and then perform a preliminary analysis. Based on the results of the preliminary analysis, it is preliminarily determined whether there is a potential risk of insufficient performance when the deep learning model that has been trained at the present stage is applied to the target scenario;

[0043] The performance verification module is used to perform a performance verification operation on the deep learning model that has been trained at the present stage and has a potential risk of insufficient performance, and obtain a comprehensive performance value;

[0044] The performance classification module divides the deep learning model that has been trained at the present stage into a high-quality deep learning model, a medium-quality deep learning model, and a low-quality deep learning model based on the comprehensive performance value;

[0045] The labeling analysis module is used to perform a labeling operation in an incremental learning manner for the medium-quality deep learning model and a re-learning manner for the low-quality deep learning model;

[0046] The confidence level classification module is used to perform a quality analysis on the labeling operation process, obtain a labeling consistency quality index and a review feedback quality index respectively, and then perform a comprehensive analysis on the labeling consistency quality index and the review feedback quality index to obtain a comprehensive confidence level coefficient and transmit it to the adoption module;

[0047] The adoption module is used to divide the labeling operation into a high-quality operation and a low-quality operation based on the comprehensive confidence level coefficient and adopt the labeling data of the high-quality operation.

[0048] The preliminary recognition module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the present stage, and then the preliminary analysis refers to:

[0049] Under a fixed time window, such as one day, one week, or one month, obtain the application information of the target scenario of the deep learning model that has completed training at the current stage. Divide this fixed time window into multiple consecutive and equal sub-windows, and then obtain the anomaly detection rate of each sub-window respectively to obtain the time series X. Calculate the average value and standard deviation of the time series X respectively; the average value represents the overall level of the anomaly detection rate of the model within the entire fixed time window. A higher average value indicates that the model performs well in detecting abnormal events. It can be used to preliminarily evaluate the overall performance of the model and help understand the detection ability of the model in the target scenario. The standard deviation represents the degree of fluctuation of the anomaly detection rate. A smaller standard deviation indicates that the detection performance of the model is relatively stable in different sub-windows; a larger standard deviation indicates that there are large fluctuations in the detection performance. The standard deviation can help identify the performance of the model at different times.

[0050] The anomaly detection rate refers to the ratio of the total number of detected abnormal events to the total number of actually occurring abnormal events.

[0051] Based on the results of the preliminary analysis, a preliminary judgment is made to determine whether there is a hidden danger of insufficient performance when the deep learning model that has completed training at the current stage is applied to the target scenario, which means:

[0052] Obtain and calculate the average value and standard deviation of the time series X, and then compare the average value of the time series X with the preset anomaly detection rate average parameter threshold in this target scenario, and compare the standard deviation of the time series X with the preset anomaly detection rate fluctuation parameter threshold in this target scenario. If it satisfies that the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold in this target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold in this target scenario, then generate a normal signal. If it does not satisfy that the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold in this target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold in this target scenario, then generate an abnormal signal. When the abnormal signal is generated, it indicates that there is a hidden danger of insufficient performance when the deep learning model that has completed training at the current stage is applied to the target scenario.

[0053] By comparing the average value of the time series X with the preset average parameter threshold of the anomaly detection rate, the overall detection performance of the model can be evaluated to determine whether it meets the expectations. If the average value is less than or equal to the preset average parameter threshold of the anomaly detection rate, it indicates that the detection ability of the model meets or exceeds the expectations; otherwise, it indicates that the detection ability of the model is insufficient, which helps to determine whether the model needs to be further optimized to improve the overall detection performance. By comparing the standard deviation of the time series X with the preset fluctuation parameter threshold of the anomaly detection rate, the stability of the model's detection performance can be evaluated. If the standard deviation is less than or equal to the preset fluctuation parameter threshold of the anomaly detection rate, it indicates that the detection performance of the model is relatively stable at different time periods; otherwise, it indicates that there are significant fluctuations in the model's detection performance. This helps to identify the performance fluctuations of the model at different time periods and determine whether further adjustments or improvements to the model are required. Generate normal or abnormal signals to facilitate real-time monitoring of the model's performance status. If an abnormal signal is generated, it indicates that there are potential performance deficiencies in the deep learning model trained at the current stage when applied to the target scenario, and measures need to be taken in a timely manner for optimization. Provide an automated performance monitoring method to help operation and maintenance personnel promptly discover and solve potential problems, ensuring the reliability and effectiveness of the deep learning model trained at the current stage in actual applications.

[0054] The performance verification operation for the deep learning model trained at the current stage with potential performance deficiencies refers to: collecting a new set of validation data from the real scenario and ensuring that the validation data is accurately labeled, and using the deep learning model trained at the current stage with potential performance deficiencies to predict the validation data set to obtain the prediction results; specifically: collecting validation data: collecting a set of unseen and representative validation data from the real scenario. Labeling the validation data: ensuring that the validation data is accurately labeled so that the prediction results of the model can be evaluated. Prediction results: using the model, that is, the deep learning model trained at the current stage with potential performance deficiencies, to predict the validation data set to obtain the prediction results.

[0055] The acquisition logic of the comprehensive performance value is as follows:

[0056] Based on the prediction results, obtain the corresponding data for each preset performance indicator, and then calculate the average value of each performance indicator and label it as PX i. By using the weighted average method, combine multiple performance indicators and allocate the weight of each performance indicator according to the importance of the specific application scenario. The formula for calculating the comprehensive performance value is: The CPS represents the comprehensive performance value, n is the total number of preset performance indicators, Wi represents the weight corresponding to the preset performance indicator i, and the sum of all weights Wi is equal to one. Each performance indicator such as accuracy, precision, recall, and F1 value, adjust the weights: According to specific application requirements, adjust the weights of each indicator to reflect the degree of attention to different performance indicators in the actual scenario. Application scenarios such as in campuses, property management, factories, etc. where intelligent security management is required. In different application scenarios, the weights of each performance indicator may be adjusted accordingly.

[0057] Dividing the deep learning models that have been trained at the current stage into high-quality deep learning models, medium-quality deep learning models, and low-quality deep learning models based on the comprehensive performance value means:

[0058] Obtain the comprehensive performance value of the deep learning models that have been trained at the current stage, compare the comprehensive performance value with the preset benchmark comprehensive performance threshold, divide the deep learning models with a comprehensive performance value higher than the preset benchmark comprehensive performance threshold into high-quality deep learning models, divide the deep learning models with a comprehensive performance value lower than the product value of the preset benchmark comprehensive performance threshold and p into low-quality deep learning models, where p is the preset percentage coefficient in this target scenario and is used to determine the demarcation point between medium-quality and low-quality models. Divide the deep learning models that are not lower than the product value of the preset benchmark comprehensive performance threshold and p and not higher than the preset benchmark comprehensive performance threshold into medium-quality deep learning models. The comprehensive performance value of high-quality models is higher than the preset benchmark value, indicating that these models perform excellently in the target scenario and have high practical value and reliability. The comprehensive performance value of medium-quality models is between that of low-quality and high-quality models, indicating that these models perform okay in the target scenario but still have room for improvement. The comprehensive performance value of low-quality models is lower than p times the preset benchmark value, indicating that these models perform poorly in the target scenario and have obvious performance deficiencies.

[0059] The annotation consistency quality index is used to measure the consistency among multiple annotators when annotating the same dataset. The acquisition logic of the annotation consistency quality index is as follows:

[0060] Collect annotation data: Collect the annotation results of the same dataset (including M samples) from multiple annotators (assuming there are N annotators); Construct a consistency matrix: For each sample, calculate the consistency among all annotators. Assume the total number of annotation categories is C. Construct an NxN consistency matrix, where each element represents the consistency between two annotators for a sample. Calculate the consistency of each sample: For each sample, calculate the consistency among all annotators. For a classification task, if the annotations are consistent, the consistency score is 1, otherwise it is 0. For each annotator i, calculate the average value of the consistency with all other annotators. The calculation formula is:

[0061] A i represents the average consistency of annotator i with all other annotators. I represents the consistency between annotators i and j for all samples. Then, the average consistencies of all annotators are summed to obtain the overall consistency. The calculation formula is as follows: A total represents the overall consistency;

[0062] Calculate the consistency coefficient Fleiss' Kappa. Fleiss' Kappa is a commonly used consistency coefficient for evaluating the consistency of multiple annotators in a multi-classification task. The calculation formula is as follows: P o represents the observed proportion of consistency, that is, the proportion of the number of times annotators actually reach an agreement to the total number of times. P e represents the expected proportion of consistency, that is, the probability of reaching an agreement assuming that the annotation is random; k represents the consistency coefficient;

[0063] For each sample, calculate the consistency of all annotators for each category and take the average. The calculation formula for P is as follows:

[0064] Calculate the expected consistency according to the annotation proportions of each category. P e The calculation formula is as follows:

[0065] M is the total number of samples, N is the total number of annotators, C is the total number of categories, and n mc is the number of annotators who annotate sample m as category c;

[0066] Perform a weighted average on the consistency coefficient k and the overall consistency A total The calculation formula is illustrated as: ACQI = q1 * k + q2 * A total ; q1 and q2 are both preset proportional coefficients used to adjust the contribution proportions of the consistency coefficient k and the overall consistency A total in the annotation consistency quality index, which can be adjusted according to the actual situation. ACQI is the annotation consistency quality index. The annotation consistency quality index is used to measure the degree of annotation consistency of different annotators for the same dataset. The larger the annotation consistency quality index, the higher the consistency of different annotators for the same data. This means that the quality of the annotated data is better, the understanding and judgment among annotators are more unified, and high consistency indicates that the data annotation is more reliable, reducing noise and errors, and can provide more accurate training data for the model, thereby improving the performance of the model.

[0067] The acquisition logic of the review feedback quality index is as follows:

[0068] Preset multiple metrics for evaluating the quality of review and feedback, such as the annotation error rate: the proportion of annotation errors found during the review; the number of feedbacks: the quantity of feedback given by the reviewers; the timeliness of feedback: the average time elapsed from submission to receipt of the feedback; the effectiveness of feedback: the proportion of feedback adopted and errors corrected by the annotators; the repeated error rate: the proportion of errors that reappear after feedback; collect data corresponding to the multiple metrics during the review and feedback process and construct the current review feedback quality vector, obtain the standard review feedback quality vector, calculate the similarity between the current review feedback quality vector and the standard review feedback quality vector, and the value of the review feedback quality index is the same as the similarity value. The review feedback quality index is used to measure the effectiveness of the review and feedback mechanism during the annotation process. The larger the review feedback quality index, the more effective the review and feedback mechanism is, the higher the quality of the feedback, and the annotation errors can be detected and corrected in a timely manner. High review feedback quality indicates a low error rate during the annotation process, high timeliness and effectiveness of feedback, and a significant improvement in the overall quality of the annotation data, which helps to improve the quality of the model training data.

[0069] Common similarity algorithms such as cosine similarity, Euclidean distance, and Manhattan distance can be used, and the selection method is judged as follows: Data dimension and characteristics: If the data dimension is high and the dimensions of each dimension are different, it is more appropriate to use cosine similarity. If the data dimension is low and the dimensions of each dimension are the same, Euclidean distance is a simple and effective choice. If there are outliers or noise in the data, Manhattan distance is more robust. Focus of the metrics: If the relative relationship and direction between the metrics are concerned, cosine similarity is the best choice. If the absolute differences between the metrics are concerned and the dimensions of each metric are the same, both Euclidean distance and Manhattan distance can be considered. Specific application requirements: According to the specific application scenario and requirements, select the similarity algorithm that best reflects the data characteristics.

[0070] The acquisition logic of the comprehensive confidence coefficient is as follows: The annotation consistency quality index and the review feedback quality index are jointly input into a pre-trained machine learning model for comprehensive analysis. The output data of the machine learning model is the comprehensive confidence coefficient. The machine learning model is not specifically limited here, and any model that can perform comprehensive analysis on the annotation consistency quality index and the review feedback quality index to generate the comprehensive confidence coefficient can be used. To implement the technical solution of the present invention, the present invention provides a specific implementation method: CCS = f1 * ACQI + f2 * RFQI; both f1 and f2 are preset proportionality coefficients used to adjust the contribution ratio of the annotation consistency quality index ACQI and the review feedback quality index RFQI in the comprehensive confidence coefficient CCS, and can be adjusted according to the actual situation. The comprehensive confidence coefficient is used to measure the overall performance and reliability of the model in the target scenario. The larger the comprehensive confidence coefficient, the better the overall performance of the model in the target scenario, including excellent performance in indicators such as accuracy, precision, recall rate, and F1 value. A high confidence coefficient indicates higher reliability of the model in practical applications, can complete tasks stably and efficiently, and is more suitable for critical tasks and important scenarios.

[0071] The adoption module is used to divide the annotation tasks into high-quality tasks and low-quality tasks based on the comprehensive confidence coefficient and adopt the annotation data of high-quality tasks, which means: The annotation tasks with a comprehensive confidence coefficient higher than the preset confidence threshold are divided into high-quality tasks, and the annotation tasks with a comprehensive confidence coefficient not higher than the preset confidence threshold are divided into low-quality tasks. The annotation data of high-quality tasks are adopted for incremental learning to improve the currently trained deep learning model, or a new deep learning model is constructed by re-learning.

[0072] By screening high-quality annotation data, the quality of the training dataset can be significantly improved, the influence of noise and mislabeling can be reduced, thereby improving the performance and accuracy of the deep learning model. Using high-quality annotation data for incremental learning or re-training the model can help the model better learn and generalize, and improve the performance of the model in practical applications. Concentrate resources on processing and adopting high-quality annotation data, and avoid wasting computing resources and time on low-quality data, thereby improving the annotation and training efficiency. By continuously adopting the data of high-quality tasks for incremental learning, continuous improvement and optimization of the model can be achieved, and the model can be kept in the latest and best state.

[0073] The performance of deep learning models highly depends on the quality of training data. Low-quality labeled data introduces noise, causing the model to learn incorrect information during training, thereby affecting the reliability and stability of the model. High-quality labeled data can help the model better capture the key features in the data, thus enhancing the model's generalization ability on unseen data and reducing overfitting. Dividing the annotation tasks into high-quality and low-quality can effectively clean the dataset and remove the noise brought by low-quality data. This is a crucial step in improving the training effect. In practical applications, the data and environment may change continuously. By continuously adopting high-quality labeled data for incremental learning, the model can dynamically adapt to the new environment and data, improving the practicality and longevity of the model.

[0074] The above formulas are all dimensionless and take their numerical values for calculation. The formulas are obtained by collecting a large amount of data and performing software simulations to get a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0075] It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0076] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0077] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A monitoring data analysis system based on deep learning, characterized in that, It includes a preliminary recognition module, a performance verification module, a performance classification module, a labeling analysis module, a confidence level classification module, and an adoption module; The preliminary recognition module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the current stage, and then conduct a preliminary analysis. Based on the results of the preliminary analysis, it preliminarily judges whether there is a potential performance deficiency when the deep learning model that has been trained at the current stage is applied to the target scenario; The performance verification module is used to perform a performance verification operation on the deep learning model that has been trained at the current stage and has a potential performance deficiency, and obtain a comprehensive performance value; The performance classification module divides the deep learning model that has been trained at the current stage into a high-quality deep learning model, a medium-quality deep learning model, and a low-quality deep learning model based on the comprehensive performance value; The labeling analysis module is used to perform a labeling operation in an incremental learning manner for the medium-quality deep learning model, and perform a labeling operation in a re-learning manner for the low-quality deep learning model; The confidence level classification module is used to perform a quality analysis on the labeling operation process, obtain a labeling consistency quality index and a review feedback quality index respectively, and then conduct a comprehensive analysis of the labeling consistency quality index and the review feedback quality index to obtain a comprehensive confidence level coefficient and transmit it to the adoption module; The adoption module is used to divide the labeling operation into a high-quality operation and a low-quality operation based on the comprehensive confidence level coefficient and adopt the labeling data of the high-quality operation.

2. The monitoring data analysis system based on deep learning according to claim 1, characterized in that The preliminary recognition module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the current stage, and then the preliminary analysis refers to: Under a fixed time window, obtain the application information of the target scenario of the deep learning model that has been trained at the current stage. Divide the fixed time window into multiple consecutive and equal sub-windows, and then obtain the anomaly detection rate of each sub-window respectively to obtain a time series X. Calculate the average value and standard deviation of the time series X respectively.

3. The monitoring data analysis system based on deep learning according to claim 2, wherein The anomaly detection rate refers to the ratio of the total number of detected anomaly events to the total number of actually occurring anomaly events.

4. The monitoring data analysis system based on deep learning according to claim 3, characterized in that Based on the results of the preliminary analysis, preliminarily judge whether there is a potential performance deficiency when the deep learning model that has been trained at the current stage is applied to the target scenario refers to: Obtain and calculate the average value and standard deviation of the time series X, and then compare the average value of the time series X with the preset anomaly detection rate average parameter threshold in this target scenario, and compare the standard deviation of the time series X with the preset anomaly detection rate fluctuation parameter threshold in this target scenario. If it satisfies that the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold in this target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold in this target scenario, then generate a normal signal. If it does not satisfy that the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold in this target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold in this target scenario, then generate an anomaly signal. When the anomaly signal is generated, it indicates that there is a potential performance deficiency when the deep learning model that has been trained at the current stage is applied to the target scenario.

5. The monitoring data analysis system based on deep learning according to claim 4, wherein, The performance verification operation for the currently trained deep learning model with potential performance deficiencies refers to: Collect a new set of validation data from the real scenario and ensure that the validation data is accurately labeled. Use the currently trained deep learning model with potential performance deficiencies to predict the validation data set and obtain the prediction results.

6. The monitoring data analysis system based on deep learning according to claim 5, wherein The acquisition logic of the comprehensive performance value is: Obtain the corresponding data of each preset performance index according to the prediction result, then calculate the average value of each performance index and mark it as PX i. By using the weighted average method, combine multiple performance indexes, and allocate the weight of each performance index according to the importance of the specific application scenario. The formula for calculating the comprehensive performance value is as follows: CPS represents the comprehensive performance value, n is the total number of preset performance indexes, Wi represents the weight corresponding to the preset performance index i, and the sum of all weights Wi is equal to one.

7. A monitoring data analysis system based on deep learning according to claim 6, characterized in that, Based on the comprehensive performance value, classifying the currently trained deep learning model into high-quality deep learning models, medium-quality deep learning models, and low-quality deep learning models refers to: Obtain the comprehensive performance value of the currently trained deep learning model, compare the comprehensive performance value with the preset benchmark comprehensive performance threshold. Classify the deep learning model with a comprehensive performance value higher than the preset benchmark comprehensive performance threshold as a high-quality deep learning model. Classify the deep learning model with a comprehensive performance value lower than the product value of the preset benchmark comprehensive performance threshold and p as a low-quality deep learning model, where p is the preset percentage coefficient in this target scenario. Classify the deep learning model with a comprehensive performance value not lower than the product value of the preset benchmark comprehensive performance threshold and p and not higher than the preset benchmark comprehensive performance threshold as a medium-quality deep learning model.

8. A monitoring data analysis system based on deep learning according to claim 7, characterized in that, The acquisition logic of the annotation consistency quality index is: Collect the annotation results of the same data set containing M samples from N annotators. For each sample, calculate the consistency among all annotators. Assume the total number of annotation categories is C, and construct a consistency matrix, where each element represents the consistency of two annotators for a sample. For each annotator i, calculate the average value of its consistency with all other annotators. The calculation formula is: A i represents the average consistency of annotator i with all other annotators, I ij represents the consistency between annotators i and j for all samples, and then the average consistencies of all annotators are summed to obtain the overall consistency. The calculation formula is: A total represents the overall consistency; Calculate the Fleiss' Kappa coefficient of consistency, and the calculation formula is: P o represents the observed proportion of agreement, and P e represents the expected proportion of agreement; k represents the coefficient of consistency; P o The calculation formula is as follows: P e The calculation formula is: Let M be the total number of samples, N be the total number of annotators, C be the total number of categories, and n mc be the number of annotators who annotate sample m as category c; Perform a weighted average on the consistency coefficient k and the overall consistency A total to obtain the annotation consistency quality index.

9. The monitoring data analysis system based on deep learning according to claim 8, characterized in that, The acquisition logic of the review feedback quality index is: Preset multiple indicators for evaluating the review and feedback quality. Collect the data corresponding to the multiple indicators from the review and feedback process and construct the current review feedback quality vector. Obtain the standard review feedback quality vector, calculate the similarity between the current review feedback quality vector and the standard review feedback quality vector. The value of the review feedback quality index is the same as the similarity value.

10. A monitoring data analysis system based on deep learning according to claim 9, characterized in that, The acquisition logic of the comprehensive confidence coefficient is: Substitute the annotation consistency quality index and the review feedback quality index into the pre-trained machine learning model for comprehensive analysis. The output data of the machine learning model is the comprehensive confidence coefficient.

Images