Monitoring data analysis system based on deep learning

By calculating the comprehensive confidence coefficient in the deep learning monitoring data analysis system, the false alarm and missed report problems caused by low-quality labeling data are solved, the accuracy and robustness of the model are improved, and the monitoring video is realized real-time analysis and response.

CN118968379AActive Publication Date: 2024-11-15BEIJING AIYOUMENG INFORMATION TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411011569.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-15
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

The performance of deep learning models in monitoring data analysis depends on high-quality labeled data. Low-quality labeled data leads to the risks of false alarms and missed reports. How to effectively evaluate and screen high-quality labeled operations has become the key to improving system performance.

Method used

A monitoring data analysis system based on deep learning was designed, including preliminary identification module, performance verification module, performance division module, labeling analysis module, confidence division module and adoption module. High-quality labeling operations are screened by calculating the comprehensive confidence coefficient. Do incremental learning or retrain the model.

Benefits of technology

It significantly improves the quality of training data, improves the accuracy and robustness of deep learning models, reduces the risks of false alarms and underreports, realizes real-time analysis and response of monitoring videos, and reduces system costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968379A_ABST
    Figure CN118968379A_ABST
Patent Text Reader

Abstract

The invention discloses a monitoring data analysis system based on deep learning, and particularly relates to the technical field of deep learning, which comprises a preliminary identification module, a performance verification module, a performance division module, a label analysis module, a confidence division module and an adoption module. The preliminary recognition module obtains application information of the deep learning model and preliminarily judges performance hidden dangers; the performance verification module verifies the performance and obtains a comprehensive performance value; the performance division module divides model quality based on the comprehensive performance value; the labeling analysis module is used for performing increment and re-learning labeling operation on the medium-quality model and the low-quality model; a confidence coefficient division module performs analysis to obtain a comprehensive confidence coefficient; the adoption module divides and marks operation quality based on the confidence coefficient and adopts high-quality data; according to the method, high-quality labeling operation can be effectively screened out, and low-quality labeling data can be eliminated, so that the overall quality of training data is remarkably improved, and the performance of a deep learning model in monitoring data analysis is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically, to a monitoring data analysis system based on deep learning. Background Art

[0002] With the widespread application of surveillance technology, surveillance points are deployed in office buildings, residential areas, shopping malls, hospitals and other scenes for 24-hour surveillance, generating a large amount of video data. Massive surveillance video data needs to be processed and analyzed efficiently. Traditional surveillance data analysis methods usually rely on manual monitoring and simple image processing technology, which has problems such as low efficiency, high false alarm rate and slow response speed. The rise of deep learning technology has provided a new solution for surveillance data analysis. By building and training deep neural networks, it can realize automatic recognition of targets in surveillance videos, behavior analysis and anomaly detection.

[0003] However, the performance of deep learning models is highly dependent on the quality of training data. In practical applications, the acquisition of labeled data often relies on manual labeling, which inevitably introduces human errors and inconsistencies. Low-quality labeled data will lead to degraded model performance and increase the risk of false positives and false negatives. How to effectively evaluate and screen high-quality labeling jobs and ensure the high quality of training data has become a key issue in improving the performance of monitoring data analysis systems. Therefore, a monitoring data analysis system based on deep learning is proposed here. Summary of the invention

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A monitoring data analysis system based on deep learning, including a preliminary identification module, a performance verification module, a performance division module, a labeling analysis module, a confidence division module, and an adoption module;

[0006] The preliminary identification module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the current stage, and then conduct a preliminary analysis. Based on the results of the preliminary analysis, it is preliminarily determined whether there is a hidden danger of insufficient performance when the deep learning model that has been trained at the current stage is applied in the target scenario;

[0007] The performance verification module is used to perform performance verification operations on the currently trained deep learning model that has the potential for insufficient performance, and obtain a comprehensive performance value;

[0008] The performance classification module classifies the deep learning models that have been trained at this stage into high-quality deep learning models, medium-quality deep learning models, and low-quality deep learning models based on the comprehensive performance values;

[0009] The annotation analysis module is used to perform annotation work in an incremental learning manner for medium-quality deep learning models, and to perform annotation work in a re-learning manner for low-quality deep learning models;

[0010] The confidence division module is used to analyze the quality of the annotation process, obtain the annotation consistency quality index and the review feedback quality index respectively, and then conduct a comprehensive analysis of the annotation consistency quality index and the review feedback quality index to obtain a comprehensive confidence coefficient and transmit it to the adoption module;

[0011] The adoption module is used to divide the labeling tasks into high-quality tasks and low-quality tasks based on the comprehensive confidence coefficient and adopt the labeling data of the high-quality tasks.

[0012] In a preferred embodiment, the preliminary identification module is used to obtain application information of the target scenario of the deep learning model that has been trained at the current stage, and then perform preliminary analysis, which means:

[0013] In a fixed time window, such as one day, one week, or one month, obtain the application information of the target scenario of the deep learning model that has been trained at this stage, divide the fixed time window into multiple continuous and equal sub-windows, and then obtain the anomaly detection rate of each sub-window to obtain the time series X, and calculate the average value and standard deviation of the time series X respectively; the average value represents the overall level of the anomaly detection rate of the model in the entire fixed time window. A higher average value indicates that the model performs well in detecting abnormal events. It can be used to preliminarily evaluate the overall performance of the model and help understand the detection ability of the model in the target scenario. The standard deviation indicates the degree of fluctuation of the anomaly detection rate. A smaller standard deviation indicates that the detection performance of the model in different sub-windows is relatively stable; a larger standard deviation indicates that the detection performance has large fluctuations. The standard deviation can help identify the performance of the model in different time periods.

[0014] The anomaly detection rate refers to the ratio of the total number of detected anomaly events to the total number of actual anomaly events.

[0015] In a preferred embodiment, the preliminary judgment based on the results of the preliminary analysis to determine whether the deep learning model currently trained has a hidden danger of insufficient performance when applied in the target scenario refers to:

[0016] The average value and standard deviation of the calculated time series X are obtained, and then the average value of the time series X is compared with the preset anomaly detection rate average parameter threshold under the target scenario, and the standard deviation of the time series X is compared with the preset anomaly detection rate fluctuation parameter threshold under the target scenario. If the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold under the target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold under the target scenario, a normal signal is generated. If the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold under the target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold under the target scenario is not satisfied, an abnormal signal is generated. When the abnormal signal is generated, it indicates that the deep learning model trained at the current stage has a hidden danger of insufficient performance when applied in the target scenario.

[0017] In a preferred embodiment, the performance verification operation of the currently trained deep learning model with the hidden danger of insufficient performance refers to:

[0018] Collect a new set of validation data sets from real scenarios and ensure that the validation data sets are accurately labeled. Use the currently trained deep learning model with potential performance deficiencies to predict the validation data sets and obtain prediction results.

[0019] In a preferred embodiment, the logic for obtaining the comprehensive performance value is:

[0020] According to the prediction results, the corresponding data of each preset performance indicator are obtained, and then the average value of each performance indicator is marked as PXi. Through the weighted average method, multiple performance indicators are combined together, and the weight of each performance indicator is assigned according to the importance of the specific application scenario. The formula for calculating the comprehensive performance value is: CPS represents the comprehensive performance value, n is the total number of preset performance indicators, Wi represents the weight corresponding to the preset performance indicator i, and the sum of all weights Wi is equal to one.

[0021] In a preferred embodiment, dividing the deep learning model currently trained into a high-quality deep learning model, a medium-quality deep learning model, and a low-quality deep learning model based on the comprehensive performance value means:

[0022] Obtain the comprehensive performance value of the deep learning model that has been trained at the current stage, compare the comprehensive performance value with the preset benchmark comprehensive performance threshold, classify the deep learning model with a value higher than the preset benchmark comprehensive performance threshold as a high-quality deep learning model, classify the deep learning model with a value lower than the product of the preset benchmark comprehensive performance threshold and p as a low-quality deep learning model, where p is a preset percentage coefficient under the target scenario, and classify the deep learning model with a value not lower than the product of the preset benchmark comprehensive performance threshold and p and not higher than the preset benchmark comprehensive performance threshold as a medium-quality deep learning model.

[0023] In a preferred embodiment, the logic for obtaining the label consistency quality index is:

[0024] Collect the labeling results of the same data set containing M samples from N labelers. For each sample, calculate the consistency between all labelers. Assuming that the total number of labeling categories is C, construct a consistency matrix, in which each element represents the consistency of two labelers on a sample. For each labeler i, calculate the average consistency between him and all other labelers. The calculation formula is: A i represents the average consistency between annotator i and all other annotators, I represents the consistency between annotators i and j for all samples, and then the average consistency of all annotators is summed up to get the overall consistency. The calculation formula is: A total Indicates overall consistency;

[0025] Calculate the consistency coefficient Fleiss′Kappa, the calculation formula is: P o It represents the observed consistency ratio, that is, the ratio of the number of times the annotators actually reach agreement to the total number of times, P e represents the expected consistency ratio, that is, the probability of reaching a consensus assuming that the annotations are random; k represents the consistency coefficient;

[0026] P o The calculation formula is:

[0027] P e The calculation formula is:

[0028] M is the total number of samples, N is the total number of labelers, C is the total number of categories, n mc is the number of labelers who label sample m as category c;

[0029] The consistency coefficient k and the overall consistency A total Perform weighted averaging to obtain the labeling consistency quality index.

[0030] In a preferred embodiment, the logic for obtaining the audit feedback quality index is:

[0031] Multiple indicators for evaluating the audit and feedback quality are preset, corresponding data of multiple indicators are collected from the audit and feedback process and the current audit feedback quality vector is constructed, the standard audit feedback quality vector is obtained, the similarity between the current audit feedback quality vector and the standard audit feedback quality vector is calculated, and the value of the audit feedback quality index is the same as the similarity value.

[0032] In a preferred embodiment, the logic for obtaining the comprehensive confidence coefficient is:

[0033] The annotation consistency quality index and the review feedback quality index are substituted into the pre-trained machine learning model for comprehensive analysis. The output data of the machine learning model is the comprehensive confidence coefficient.

[0034] Technical effects and advantages of the present invention:

[0035] By calculating and analyzing the comprehensive confidence coefficient, the present invention can effectively screen out high-quality annotation jobs and eliminate low-quality annotation data, thereby significantly improving the overall quality of training data and improving the performance of deep learning models in monitoring data analysis. Using high-quality annotated data for incremental learning or retraining can improve the accuracy, robustness, and generalization ability of deep learning models, making them perform better in target recognition, behavior analysis, and anomaly detection in surveillance videos.

[0036] The present invention uses a machine learning model for comprehensive evaluation, which can quickly and accurately identify high-quality labeling jobs, reduce the workload of manual review, and improve the overall evaluation efficiency, thereby realizing real-time analysis and response of monitoring data. Through automated labeling quality evaluation and screening, the cost of re-labeling and model tuning caused by low-quality labeled data can be reduced, thereby reducing the total cost of the monitoring data analysis system. In addition, high-quality labeled data can be systematically evaluated and adopted to ensure the stability and reliability of deep learning models in different monitoring scenarios, and enhance the overall performance of the monitoring data analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0038] Figure 1 This is a schematic diagram of a monitoring data analysis system based on deep learning in the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example 1

[0041] A monitoring data analysis system based on deep learning includes a preliminary identification module, a performance verification module, a performance partitioning module, a labeling analysis module, a confidence partitioning module, and an adoption module; the preliminary identification module, the performance verification module, the performance partitioning module, the labeling analysis module, the confidence partitioning module, and the adoption module are communicated with each other.

[0042] The preliminary identification module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the current stage, and then conduct a preliminary analysis. Based on the results of the preliminary analysis, it is preliminarily determined whether there is a hidden danger of insufficient performance when the deep learning model that has been trained at the current stage is applied in the target scenario;

[0043] The performance verification module is used to perform performance verification operations on the currently trained deep learning model that has the potential for insufficient performance, and obtain a comprehensive performance value;

[0044] The performance classification module classifies the deep learning models that have been trained at this stage into high-quality deep learning models, medium-quality deep learning models, and low-quality deep learning models based on the comprehensive performance values;

[0045] The annotation analysis module is used to perform annotation work in an incremental learning manner for medium-quality deep learning models, and to perform annotation work in a re-learning manner for low-quality deep learning models;

[0046] The confidence division module is used to analyze the quality of the annotation process, obtain the annotation consistency quality index and the review feedback quality index respectively, and then conduct a comprehensive analysis of the annotation consistency quality index and the review feedback quality index to obtain a comprehensive confidence coefficient and transmit it to the adoption module;

[0047] The adoption module is used to divide the labeling tasks into high-quality tasks and low-quality tasks based on the comprehensive confidence coefficient and adopt the labeling data of the high-quality tasks.

[0048] The preliminary identification module is used to obtain the application information of the target scenario of the deep learning model that has been trained at this stage, and then conduct preliminary analysis, which means:

[0049] In a fixed time window, such as one day, one week, or one month, obtain the application information of the target scenario of the deep learning model that has been trained at this stage, divide the fixed time window into multiple continuous and equal sub-windows, and then obtain the anomaly detection rate of each sub-window to obtain the time series X, and calculate the average value and standard deviation of the time series X respectively; the average value represents the overall level of the anomaly detection rate of the model in the entire fixed time window. A higher average value indicates that the model performs well in detecting abnormal events. It can be used to preliminarily evaluate the overall performance of the model and help understand the detection ability of the model in the target scenario. The standard deviation indicates the degree of fluctuation of the anomaly detection rate. A smaller standard deviation indicates that the detection performance of the model in different sub-windows is relatively stable; a larger standard deviation indicates that the detection performance has large fluctuations. The standard deviation can help identify the performance of the model in different time periods.

[0050] The anomaly detection rate refers to the ratio of the total number of detected anomaly events to the total number of actual anomaly events.

[0051] Based on the results of the preliminary analysis, the initial judgment determines whether the currently trained deep learning model has the potential risk of insufficient performance when applied in the target scenario. This refers to:

[0052] The average value and standard deviation of the calculated time series X are obtained, and then the average value of the time series X is compared with the preset anomaly detection rate average parameter threshold under the target scenario, and the standard deviation of the time series X is compared with the preset anomaly detection rate fluctuation parameter threshold under the target scenario. If the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold under the target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold under the target scenario, a normal signal is generated. If the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold under the target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold under the target scenario is not satisfied, an abnormal signal is generated. When the abnormal signal is generated, it indicates that the deep learning model trained at the current stage has a hidden danger of insufficient performance when applied in the target scenario.

[0053] By comparing the average value of the time series X with the preset anomaly detection rate average parameter threshold, you can evaluate whether the overall detection performance of the model meets expectations. If the average value is less than or equal to the preset anomaly detection rate average parameter threshold, it means that the detection capability of the model meets or exceeds expectations; otherwise, it means that the detection capability of the model is insufficient, which helps determine whether the model needs to be further optimized to improve the overall detection performance. By comparing the standard deviation of the time series X with the preset anomaly detection rate fluctuation parameter threshold, you can evaluate whether the detection performance of the model is stable. If the standard deviation is less than or equal to the preset anomaly detection rate fluctuation parameter threshold, it means that the detection performance of the model in different time periods is relatively stable; otherwise, it means that the detection performance of the model has large fluctuations. Helps identify the performance fluctuations of the model in different time periods and determine whether the model needs to be further adjusted or improved. Generates normal or abnormal signals to facilitate real-time monitoring of the performance status of the model. If an abnormal signal is generated, it means that the deep learning model that has been trained at this stage has a hidden danger of insufficient performance when applied in the target scenario, and timely measures need to be taken to optimize it. Provides automated performance monitoring methods to help operation and maintenance personnel promptly discover and solve potential problems and ensure the reliability and effectiveness of the deep learning model that has been trained at this stage in actual applications.

[0054] The performance verification operation of the currently trained deep learning model with the hidden danger of insufficient performance refers to: collecting a set of new verification data sets from real scenes, and ensuring that the verification data are accurately labeled, using the currently trained deep learning model with the hidden danger of insufficient performance to predict the verification data sets, and obtaining the prediction results; specifically: Collecting verification data: Collecting a set of unseen and representative verification data sets from real scenes. Labeling verification data: Ensure that the verification data is accurately labeled so that the prediction results of the model can be evaluated. Prediction results: Using the model, that is, the currently trained deep learning model with the hidden danger of insufficient performance, to predict the verification data sets, and obtain the prediction results.

[0055] The logic for obtaining the comprehensive performance value is:

[0056] According to the prediction results, the corresponding data of each preset performance indicator are obtained, and then the average value of each performance indicator is marked as PXi. Through the weighted average method, multiple performance indicators are combined together, and the weight of each performance indicator is assigned according to the importance of the specific application scenario. The formula for calculating the comprehensive performance value is: CPS represents the comprehensive performance value, n is the total number of preset performance indicators, Wi represents the weight corresponding to the preset performance indicator i, and the sum of all weights Wi is equal to one. Adjust the weights of various performance indicators such as accuracy, precision, recall, and F1 value: adjust the weights of various indicators according to specific application requirements to reflect the degree of attention paid to different performance indicators in actual scenarios. Application scenarios such as campuses, properties, factories, etc. require intelligent security management. In different application scenarios, the weight of each performance indicator may be adjusted accordingly.

[0057] Based on the comprehensive performance value, the deep learning models trained at this stage are divided into high-quality deep learning models, medium-quality deep learning models, and low-quality deep learning models, which means:

[0058] The comprehensive performance value of the deep learning model that has been trained at the current stage is obtained, and the comprehensive performance value is compared with the preset benchmark comprehensive performance threshold. The deep learning model with a value higher than the preset benchmark comprehensive performance threshold is classified as a high-quality deep learning model, and the deep learning model with a value lower than the product of the preset benchmark comprehensive performance threshold and p is classified as a low-quality deep learning model. p is a preset percentage coefficient in the target scenario, which is used to determine the dividing point between medium-quality and low-quality models. The deep learning model with a value not lower than the product of the preset benchmark comprehensive performance threshold and p and not higher than the preset benchmark comprehensive performance threshold is classified as a medium-quality deep learning model. The comprehensive performance value of the high-quality model is higher than the preset benchmark value, indicating that these models perform well in the target scenario and have high practical value and reliability. The comprehensive performance value of the medium-quality model is between the low-quality and high-quality models, indicating that these models perform well in the target scenario, but there is still room for improvement. The comprehensive performance value of the low-quality model is lower than p times the preset benchmark value, indicating that these models perform poorly in the target scenario and have obvious performance deficiencies.

[0059] The annotation consistency quality index is used to measure the consistency of multiple annotators when annotating the same data set. The logic for obtaining the annotation consistency quality index is as follows:

[0060] Collect labeled data: Collect the labeling results of the same data set (containing M samples) from multiple labelers (assuming there are N labelers); Construct a consistency matrix: For each sample, calculate the consistency between all labelers. Assume that the total number of labeling categories is C. Construct an NxN consistency matrix, in which each element represents the consistency of two labelers on a sample, and calculate the consistency of each sample: For each sample, calculate the consistency between all labelers. For classification tasks, if the labeling is consistent, the consistency score is 1, otherwise it is 0. For each labeler i, calculate the average consistency with all other labelers, and the calculation formula is:

[0061] A i represents the average consistency between annotator i and all other annotators, I represents the consistency between annotators i and j for all samples, and then the average consistency of all annotators is summed up to get the overall consistency. The calculation formula is: A total Indicates overall consistency;

[0062] Calculate the consistency coefficient Fleiss′Kappa. Fleiss′Kappa is a commonly used consistency coefficient used to evaluate the consistency of multiple labelers on multi-classification tasks. The calculation formula is: P o It represents the observed consistency ratio, that is, the ratio of the number of times the annotators actually reach agreement to the total number of times, P e represents the expected consistency ratio, that is, the probability of reaching a consensus assuming that the annotations are random; k represents the consistency coefficient;

[0063] For each sample, calculate the consistency of all annotators for each category and find the average. The calculation formula for P is:

[0064] The expected consistency is calculated based on the labeling ratio of each category, P e The calculation formula is:

[0065] M is the total number of samples, N is the total number of labelers, C is the total number of categories, n mc is the number of labelers who label sample m as category c;

[0066] The consistency coefficient k and the overall consistency A total Perform weighted averaging. The calculation formula is as follows: ACQI = q1*k + q2*A total q1 and q2 are preset proportional coefficients used to adjust the consistency coefficient k and the overall consistency A total The contribution ratio in the labeling consistency quality index can be adjusted according to the actual situation. ACQI is the labeling consistency quality index. The labeling consistency quality index is used to measure the consistency of labeling of the same data set by different labelers. The larger the labeling consistency quality index, the higher the consistency of the same data by different labelers. This means that the quality of the labeled data is better, and the understanding and judgment between labelers are more unified. High consistency indicates that the data labeling is more reliable, reducing noise and errors, and can provide more accurate training data for the model, thereby improving the performance of the model.

[0067] The logic for obtaining the audit feedback quality index is:

[0068] Multiple indicators for evaluating the quality of review and feedback are preset, such as the annotation error rate: the proportion of annotation errors found in the review; the number of feedbacks: the number of feedbacks given by the reviewers; the timeliness of feedback: the average time between submission and receipt of feedback; the effectiveness of feedback: the proportion of feedback adopted and corrected by the annotators; the repeated error rate: the proportion of errors that reappear after feedback; collect data corresponding to multiple indicators from the review and feedback process and construct the current review feedback quality vector, obtain the standard review feedback quality vector, calculate the similarity between the current review feedback quality vector and the standard review feedback quality vector, and the value of the review feedback quality index is the same as the similarity value. The review feedback quality index is used to measure the effectiveness of the review and feedback mechanism in the annotation process. The larger the review feedback quality index, the more effective the review and feedback mechanism, the higher the quality of feedback, and the timely discovery and correction of annotation errors. High review feedback quality indicates that the error rate in the annotation process is low, the feedback timeliness and effectiveness are high, and the overall quality of the annotation data has been significantly improved, which helps to improve the quality of model training data.

[0069] Common similarity algorithms such as cosine similarity, Euclidean distance, and Manhattan distance can be used. The selection method is determined as follows: Data dimension and characteristics: If the data dimension is high and the dimensions of each dimension are different, it is more appropriate to use cosine similarity. If the data dimension is low and the dimensions of each dimension are the same, Euclidean distance is a simple and effective choice. If there are outliers or noise in the data, Manhattan distance is more robust. Focus of indicators: If you are concerned about the relative relationship and direction between indicators, cosine similarity is the best choice. If you are concerned about the absolute difference of indicators and the dimensions of each indicator are the same, both Euclidean distance and Manhattan distance can be considered. Specific application requirements: According to the specific application scenarios and requirements, select the similarity algorithm that best reflects the characteristics of the data.

[0070] The logic for obtaining the comprehensive confidence coefficient is: the labeling consistency quality index and the audit feedback quality index are substituted into the pre-trained machine learning model for comprehensive analysis. The output data of the machine learning model is the comprehensive confidence coefficient. The machine learning model is not specifically limited here. Any model that can realize the comprehensive analysis of the labeling consistency quality index and the audit feedback quality index to generate the comprehensive confidence coefficient is acceptable. In order to realize the technical solution of the present invention, the present invention provides a specific implementation method: CCS = f1*ACQI+f2*RFQI; f1 and f2 are both preset proportional coefficients, which are used to adjust the contribution ratio of the labeling consistency quality index ACQI and the audit feedback quality index RFQI in the comprehensive confidence coefficient CCS. They can be adjusted according to actual conditions. The comprehensive confidence coefficient is used to measure the overall performance and reliability of the model in the target scenario. The larger the comprehensive confidence coefficient, the better the overall performance of the model in the target scenario, including excellent performance in indicators such as accuracy, precision, recall, and F1 value. A high confidence coefficient indicates that the model is more reliable in practical applications, can complete tasks stably and efficiently, and is more suitable for critical tasks and important scenarios.

[0071] The adoption module is used to divide the labeling jobs into high-quality jobs and low-quality jobs based on the comprehensive confidence coefficient and adopt the labeling data of the high-quality jobs, which means: the labeling jobs with a comprehensive confidence coefficient higher than the preset confidence threshold are divided into high-quality jobs, and the labeling jobs with a comprehensive confidence coefficient not higher than the preset confidence threshold are divided into low-quality jobs. The labeling data of the high-quality jobs are adopted for incremental learning to improve the deep learning model that has been trained at this stage, or to build a new deep learning model by re-learning.

[0072] By screening high-quality labeled data, the quality of training data sets can be significantly improved, the impact of noise and incorrect labeling can be reduced, and the performance and accuracy of deep learning models can be improved. Using high-quality labeled data for incremental learning or retraining models can help models learn and generalize better, and improve the performance of models in practical applications. Concentrate resources on processing and adopting high-quality labeled data to avoid wasting computing resources and time on low-quality data, thereby improving labeling and training efficiency. By continuously adopting high-quality job data for incremental learning, continuous improvement and optimization of the model can be achieved, keeping the model up to date and optimal.

[0073] The performance of deep learning models is highly dependent on the quality of training data. Low-quality labeled data will introduce noise, causing the model to learn incorrect information during training, which in turn affects the reliability and stability of the model. High-quality labeled data can help the model better capture the key features in the data, thereby improving the model's generalization ability on unseen data and reducing overfitting. Dividing the labeling work into high-quality and low-quality data can effectively clean the data set and remove the noise caused by low-quality data. This is a key step to improve training results. In actual applications, data and environments may change continuously. By continuously adopting high-quality labeled data for incremental learning, the model can dynamically adapt to new environments and data, improving the practicality and durability of the model.

[0074] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0075] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0076] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0078] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A monitoring data analysis system based on deep learning, characterized in that: It includes preliminary identification module, performance verification module, performance division module, annotation analysis module, confidence division module and adoption module; The preliminary identification module is used to obtain the application information of the target scenario of the deep learning model that has been trained at the current stage, and then conduct a preliminary analysis. Based on the results of the preliminary analysis, it is preliminarily determined whether there is a hidden danger of insufficient performance when the deep learning model that has been trained at the current stage is applied in the target scenario; The performance verification module is used to perform performance verification operations on the currently trained deep learning model that has the potential for insufficient performance, and obtain a comprehensive performance value; The performance classification module classifies the deep learning models that have been trained at this stage into high-quality deep learning models, medium-quality deep learning models, and low-quality deep learning models based on the comprehensive performance values; The annotation analysis module is used to perform annotation work in an incremental learning manner for medium-quality deep learning models, and to perform annotation work in a re-learning manner for low-quality deep learning models; The confidence division module is used to analyze the quality of the annotation process, obtain the annotation consistency quality index and the review feedback quality index respectively, and then conduct a comprehensive analysis of the annotation consistency quality index and the review feedback quality index to obtain a comprehensive confidence coefficient and transmit it to the adoption module; The adoption module is used to divide the labeling tasks into high-quality tasks and low-quality tasks based on the comprehensive confidence coefficient and adopt the labeling data of the high-quality tasks.

2. A monitoring data analysis system based on deep learning according to claim 1, characterized in that: The preliminary identification module is used to obtain the application information of the target scenario of the deep learning model that has been trained at this stage, and then conduct preliminary analysis, which means: In a fixed time window, the application information of the target scenario of the deep learning model that has been trained at this stage is obtained, and the fixed time window is divided into multiple continuous and equal sub-windows. Then, the anomaly detection rate of each sub-window is obtained respectively to obtain the time series X, and the mean and standard deviation of the time series X are calculated respectively.

3. A monitoring data analysis system based on deep learning according to claim 2, characterized in that: The anomaly detection rate refers to the ratio of the total number of detected anomaly events to the total number of actual anomaly events.

4. The monitoring data analysis system based on deep learning according to claim 3 is characterized in that: Based on the results of the preliminary analysis, the initial judgment determines whether the currently trained deep learning model has the potential risk of insufficient performance when applied in the target scenario. This refers to: The average value and standard deviation of the calculated time series X are obtained, and then the average value of the time series X is compared with the preset anomaly detection rate average parameter threshold under the target scenario, and the standard deviation of the time series X is compared with the preset anomaly detection rate fluctuation parameter threshold under the target scenario. If the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold under the target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold under the target scenario, a normal signal is generated. If the average value of the time series X is less than or equal to the preset anomaly detection rate average parameter threshold under the target scenario and the standard deviation of the time series X is less than or equal to the preset anomaly detection rate fluctuation parameter threshold under the target scenario is not satisfied, an abnormal signal is generated. When the abnormal signal is generated, it indicates that the deep learning model trained at the current stage has a hidden danger of insufficient performance when applied in the target scenario.

5. A monitoring data analysis system based on deep learning according to claim 4, characterized in that: The performance verification operation for the currently trained deep learning model with potential performance deficiencies refers to: Collect a new set of validation data sets from real scenarios and ensure that the validation data sets are accurately labeled. Use the currently trained deep learning model with potential performance deficiencies to predict the validation data sets and obtain prediction results.

6. The monitoring data analysis system based on deep learning according to claim 5, characterized in that: The logic for obtaining the comprehensive performance value is: According to the prediction results, the corresponding data of each preset performance indicator are obtained, and then the average value of each performance indicator is marked as PXi. Through the weighted average method, multiple performance indicators are combined together, and the weight of each performance indicator is assigned according to the importance of the specific application scenario. The formula for calculating the comprehensive performance value is: CPS represents the comprehensive performance value, n is the total number of preset performance indicators, Wi represents the weight corresponding to the preset performance indicator i, and the sum of all weights Wi is equal to one.

7. A monitoring data analysis system based on deep learning according to claim 6, characterized in that: Based on the comprehensive performance value, the deep learning models trained at this stage are divided into high-quality deep learning models, medium-quality deep learning models, and low-quality deep learning models, which means: Obtain the comprehensive performance value of the deep learning model that has been trained at the current stage, compare the comprehensive performance value with the preset benchmark comprehensive performance threshold, classify the deep learning model with a value higher than the preset benchmark comprehensive performance threshold as a high-quality deep learning model, classify the deep learning model with a value lower than the product of the preset benchmark comprehensive performance threshold and p as a low-quality deep learning model, where p is a preset percentage coefficient under the target scenario, and classify the deep learning model with a value not lower than the product of the preset benchmark comprehensive performance threshold and p and not higher than the preset benchmark comprehensive performance threshold as a medium-quality deep learning model.

8. The monitoring data analysis system based on deep learning according to claim 7, characterized in that: The logic for obtaining the annotation consistency quality index is: Collect the labeling results of the same data set containing M samples from N labelers. For each sample, calculate the consistency between all labelers. Assuming that the total number of labeling categories is C, construct a consistency matrix, in which each element represents the consistency of two labelers on a sample. For each labeler i, calculate the average consistency between him and all other labelers. The calculation formula is: A i It represents the average consistency between the annotator i and all other annotators, I ij represents the consistency of all samples between annotators i and j, and then the average consistency of all annotators is summed up to get the overall consistency. The calculation formula is: A total Indicates overall consistency; Calculate the consistency coefficient Fleiss′Kappa, the calculation formula is: P o represents the observed consistency ratio, P e represents the expected consistency ratio; k represents the consistency coefficient; P o The calculation formula is: P e The calculation formula is: M is the total number of samples, N is the total number of labelers, C is the total number of categories, n mc is the number of labelers who label sample m as category c; The consistency coefficient k and the overall consistency A total Perform weighted averaging to obtain the labeling consistency quality index.

9. A monitoring data analysis system based on deep learning according to claim 8, characterized in that: The logic for obtaining the audit feedback quality index is: Multiple indicators for evaluating the audit and feedback quality are preset, corresponding data of multiple indicators are collected from the audit and feedback process and the current audit feedback quality vector is constructed, the standard audit feedback quality vector is obtained, the similarity between the current audit feedback quality vector and the standard audit feedback quality vector is calculated, and the value of the audit feedback quality index is the same as the similarity value.

10. A monitoring data analysis system based on deep learning according to claim 9, characterized in that: The logic for obtaining the comprehensive confidence coefficient is: The annotation consistency quality index and the review feedback quality index are substituted into the pre-trained machine learning model for comprehensive analysis. The output data of the machine learning model is the comprehensive confidence coefficient.

Citation Information

Patent Citations

  • Incremental learning method and system based on small number of labeled samples

    CN112132179A

  • Online training method and device of deep learning model, equipment and storage medium

    CN114998679A

  • Intelligent monitoring management method and system

    CN118075427A

  • Methods of training deep learning model and predicting class and electronic device for performing the methods

    US20230177331A1

  • Data set construction method, mobile terminal and readable storage medium

    WO2019233297A1

Cited By

  • Intelligent data labeling method and system, electronic equipment and storage medium

    CN120375375A

  • Visual real-time data monitoring method and device based on MR technology

    CN121259253A