An audio-video data management system and method based on the Internet of Things

Abstract

The application discloses a kind of based on thing networking's audio-video data management system and method, it is related to thing networking data analysis technical field, and the system includes preprocessing module, feature engineering module, abnormality judging module and abnormality repair module;The preprocessing module is used to obtain audio and video data, carries out frame processing and extracts features;The feature engineering module is used to audio signal and video frame image artificial marking classification, constructs classifier and extracts important features, provides basis for abnormality judging;The abnormality judging module is used to determine clustering center, carries out clustering operation and judges whether audio signal and video frame image are abnormal, accurately identifies abnormal data;The abnormality repair module is used to construct time series model correction abnormal data and restore it.The application further proposes a kind of method for the implementation of system.The application can effectively improve the situation that part of abnormal frame in prior art exists influence on management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things data analysis, and particularly relates to an audio-video data management system and method based on Internet of Things. BACKGROUND

[0002] Through the Internet of Things technology, various audio-video collection devices such as cameras and microphones can be connected with the network to realize interconnection and intercommunication between devices. These devices can collect audio-video data in real time and transmit them to designated storage or processing centers for centralized management and analysis. By using big data processing technology and machine learning algorithms, massive audio-video data can be mined and feature extracted to discover hidden information and rules. Multi-dimensional data provides more abundant materials for audio-video data management, which helps to better understand user needs and preferences.

[0003] During the collection, transmission and storage of audio-video data, noise images, device problems, network problems and storage medium problems often cause noise or poor quality frames in audio and video signals in large-scale data, affecting the effectiveness of data and the accuracy of analysis. SUMMARY

[0004] The present application aims to provide an audio-video data management system and method based on Internet of Things to solve the problems in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme: an audio-video data management system based on Internet of Things, which comprises a preprocessing module, a feature engineering module, an anomaly judgment module and an anomaly correction module.

[0006] The preprocessing module is used to obtain audio and video data, perform frame processing and extract features to provide basic data preparation for subsequent analysis. The feature engineering module is used to manually label and classify audio signals and video frame images, build a classifier and extract important features to provide a basis for anomaly judgment. The anomaly judgment module is used to determine the cluster center, perform clustering operation and judge whether the audio signal and video frame image are abnormal to accurately identify abnormal data. The anomaly correction module is used to build a time series model to correct abnormal data and restore them.

[0007] The output end of the preprocessing module is connected with the input end of the feature engineering module. The output end of the feature engineering module is connected with the input end of the anomaly judgment module. The output end of the anomaly judgment module is connected with the input end of the anomaly correction module.

[0008] The preprocessing module comprises a data acquisition unit, a frame division unit and a feature unit.

[0009] The data acquisition unit is configured to acquire audio and video data; the frame division unit is configured to perform frame division processing on the acquired audio and video; and the feature unit is configured to extract features of the audio and video after frame division.

[0010] An output end of the data acquisition unit is connected to an input end of the frame division unit; an output end of the frame division unit is connected to an input end of the feature unit; and an output end of the feature unit is connected to an input end of the feature engineering module.

[0011] The feature engineering module comprises a label classification unit, a classifier construction unit and a feature extraction unit.

[0012] The label classification unit is configured to manually label the audio and video after frame division processing, classify the audio and video into subsets according to label types, and form a data set; the classifier construction unit is configured to construct a random forest classifier and train the random forest classifier to measure the importance of audio signal and video frame image features; and the feature extraction unit is configured to extract important features from the classified subsets.

[0013] An output end of the label classification unit is connected to an input end of the classifier construction unit; an output end of the classifier construction unit is connected to an input end of the feature extraction unit; and an output end of the feature extraction unit is connected to an input end of the anomaly judgment module.

[0014] The anomaly judgment module comprises a cluster center determination unit, a K-means clustering unit and an anomaly judgment unit.

[0015] The cluster center determination unit is configured to determine the mass center of a dense clustering area as an initial cluster center; the K-means clustering unit is configured to perform clustering, judge anomalies, and recompute the center and assign audio signals and video frame images; and the anomaly judgment unit is configured to set a threshold to judge anomalies of audio signals and video frame images.

[0016] An output end of the cluster center determination unit is connected to an input end of the K-means clustering unit; an output end of the K-means clustering unit is connected to an input end of the anomaly judgment unit; and an output end of the anomaly judgment unit is connected to an input end of the anomaly correction module.

[0017] The anomaly correction module comprises a time series model unit, a correction unit and a restoration unit.

[0018] The time sequence model unit is configured to construct a time sequence model for each audio signal feature, and train the model with the audio signal feature time sequence as input; the correction unit is configured to input the frame number corresponding to the frame determined as abnormal, and correct the abnormal value using the value predicted by the time sequence model; and the restoration unit is configured to restore the corrected audio signal into an audio frame and restore the corrected video frame image into a video frame.

[0019] The output end of the time sequence model unit is connected to the input end of the correction unit, and the output end of the correction unit is connected to the input end of the restoration unit.

[0020] A video and audio data management method based on the Internet of Things, the method comprising the following steps:

[0021] Step 1, obtaining audio and video from a device or a database and dividing them into frames, and representing the audio frame signal and the video frame image in the form of a feature value;

[0022] Step 2, manually labeling the audio signal and the video frame image to form a data set and classify them, and constructing a random forest classifier to extract important features;

[0023] Step 3, judging abnormal audio signals and video frame images according to the feature value distribution of the features extracted from the same classified data;

[0024] Step 4, marking the audio signal and the video frame image, constructing a time sequence model, correcting and restoring.

[0025] In Step 1, for audio, the audio is obtained by an audio recording device or from an audio database; the audio is divided into frames to obtain N audio signals; for each audio signal, the audio signal features are extracted and represented as [A1(i), A2(i), …, An(i)]. n (i)];

[0026] For video, the video is obtained by a video recording device or from a video database; the video is divided into frames to obtain M images; for each video frame image, the video frame image features are extracted and represented as [V1(j), V2(j), …, Vm(j)]. m (j)];

[0027] wherein M and N are positive integers, representing the number of audio signals and video frame images respectively; i∈{1,2,…,N} represents an audio frame sequence; j∈{1,2,…,M} represents a video frame sequence; n is a positive integer, representing the number of audio signal features, An(i) represents the nth audio signal feature of the ith audio frame; m is a positive integer, representing the number of video frame image features, Vm(j) represents the mth video frame image feature of the jth video frame. n m ​(j) represents the m-th video frame image feature of the j-th video frame.

[0028] In step Step2, each frame of audio signal is manually annotated, and the manually annotated type is represented as: [K1, K2, …, K u ]; The audio signal feature of each frame and its corresponding manually annotated type are obtained to form a data set;

[0029] Wherein, u is a positive integer, representing the number of audio signal types, K u represents the u-th audio signal type.

[0030] According to different types of manual annotation, the audio signal is classified to obtain a classification subset of different annotation types; a random forest classifier is constructed, and the number of decision trees is set to E; for each classification subset, the audio signal feature data in the subset is used as the input feature, and the corresponding manually annotated type is used as the target output to train the random forest classifier:

[0031] When training each decision tree, samples are randomly selected with replacement from the data set;

[0032] For each node split, instead of selecting the audio signal feature with the maximum purity from all audio signal features, k audio signal features are randomly selected from all audio signal features; the audio signal feature that maximizes the purity of the split node is selected as the split feature of the node; the node purity is measured by the Gini index;

[0033] For an audio signal feature A r , its importance in a decision tree is calculated: traverse each node in the tree, calculate the Gini index of the node without considering feature A r as a split feature, and obtain Gb(A r ) by weighted sum according to the proportion of the number of samples contained in the node to the total number of samples; traverse each node in the tree, calculate the Gini index of the split node when using feature A r as a split feature, and obtain Ga(A r ) by weighted sum according to the proportion of the number of samples contained in the node to the total number of samples; the importance of feature A r in this decision tree is represented as It(A r )=Gb(A r )-Ga(A r ); for feature A r , its importance If(A r ) in the entire random forest is obtained by averaging the importance of the feature in each decision tree: If(A r )=(1 / E)·Σ(It(A r )).

[0034] wherein r is a positive integer, r∈{1, 2, …, n}, representing the audio signal feature sequence; k is a positive integer, k < n, representing the number of selected audio signal features;

[0035] calculating the importance of each audio signal feature to each classification subset; for each classification subset, selecting the top a features with the largest importance value as the important audio signal features of the subset;

[0036] According to the above manner, the labeled video frame images are classified and the important video frame image features are extracted.

[0037] In step Step3, for a classification subset of different labels obtained by classifying the audio signals according to different types of artificial labeling, the selected important audio signal features are represented as: [F1, F2, …, F α ];

[0038] wherein F1, F2, …, F α ∈{A1, A2, …, A n};

[0039] In this subset, the distribution of different important audio signal features is obtained:

[0040] Calculate the Euclidean distance between each audio signal feature value and all other audio signal feature values to form a distance matrix; sort all distance values in the distance matrix and draw a distance value sorting graph; according to the trend of distance value change in the sorting graph, find the point where the distance value changes from gentle to sharply increasing, and select the corresponding distance value as an initial estimate of the neighborhood radius eps;

[0041] Set the minimum number of points minPts to a+1;

[0042] For each audio signal, calculate the number of data points contained in the eps neighborhood, and when the number of data points is greater than or equal to minPts, mark the audio signal as a core point; otherwise, mark it as a boundary point or a noise point; starting from the core point, form clusters by connecting density-reachable data points; density-reachable means that there is a chain of data points, each point in the chain is in the eps neighborhood of its predecessor, and the predecessor is a core point;

[0043] Using the DBSCAN algorithm, the data is divided into different clusters; for each identified dense cluster region, select the centroid of each cluster as the initial cluster center;

[0044] wherein the centroid represents the average value of all audio signal feature data vectors in the cluster;

[0045] Set the number of clusters to 2, select two of the initial cluster centers, and use the K-means clustering algorithm to divide all audio signals in the classification subset into two clusters;

[0046] After completing a cluster, for each cluster, the cluster center is recalculated: the average of all points after the first clustering is taken as the new two cluster centers; the K-means clustering algorithm is used again to assign different audio signals; the maximum number of iterations T is set, and when the number of iterations reaches T, the iteration is stopped; a threshold G is set, and the distance of each audio signal to the cluster center to which it belongs is calculated, and when the distance is greater than G, the audio signal is considered abnormal;

[0047] Abnormal judgment is performed on the audio signals corresponding to each classification subset;

[0048] According to the above method, abnormal judgment is performed on the video frame images corresponding to each classification subset.

[0049] In step Step4, the audio signals judged to be abnormal are marked, and for each audio signal feature, an audio signal database is constructed: [A v (1), A v (2), …, A v (N)];

[0050] Where v is a positive integer, v∈{1,2,…,n}, representing the audio signal feature sequence; A v (N) represents the vth audio signal feature of the Nth audio frame.

[0051] Further, a time series model LSTM of the audio signal feature A v is constructed, and A v (1), A v (2), …, A v (N) are input to the model for training to obtain a trained model; the frame number corresponding to the audio signal judged to be abnormal is input to obtain the audio signal feature A v predicted by the model as a correction of the abnormal value;

[0052] A time series model is constructed for each audio signal feature to correct the audio signal features judged to be abnormal; and the corrected audio signal features are restored to audio frames;

[0053] According to the above method, the video frame image features judged to be abnormal are corrected and restored to video frames.

[0054] Compared with the prior art, the present application has the beneficial effects that: the present application mines the most representative and key features for different annotation types, so that subsequent processing is more focused on important features related to specific tasks, improving processing efficiency and accuracy; the present application clusters audio signals and video frame images by using the DBSCAN algorithm combined with K-means, and performs anomaly detection by judging the density of the centroid and neighborhood, which can efficiently identify and correct abnormal data points; the present application repairs the time sequence of abnormal audio signal features by using LSTM, which not only corrects the abnormal data, but also smooths the data according to the prediction results of the model, thereby ensuring the quality and consistency of the audio and video data. BRIEF DESCRIPTION OF DRAWINGS

[0055] Fig. 1 A flowchart of the present application is shown in the figure.

[0056] Fig. 2 A flowchart of the present application is shown in the figure. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0058] Embodiment: as shown in the figure, the present application provides a technical solution, a video and audio data management system based on Internet of Things, which includes a preprocessing module, a feature engineering module, an anomaly judgment module and an anomaly correction module. Figs. 1-2

[0059] The preprocessing module is used to obtain audio and video data, perform frame processing and extract features, and provide basic data preparation for subsequent analysis; the feature engineering module is used to manually annotate and classify audio signals and video frame images, build a classifier and extract important features, and provide a basis for anomaly judgment; the anomaly judgment module is used to determine the cluster center, perform clustering operation and judge whether the audio signal and video frame image are abnormal, and accurately identify abnormal data; the anomaly correction module is used to build a time series model to correct abnormal data and restore it.

[0060] The output end of the preprocessing module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the anomaly judgment module; the output end of the anomaly judgment module is connected to the input end of the anomaly correction module. ​

[0061] The preprocessing module comprises a data acquisition unit, a frame division unit and a feature unit.

[0062] The data acquisition unit is configured to acquire audio and video data; the frame division unit is configured to perform frame division processing on the acquired audio and video; and the feature unit is configured to extract features of the audio and video after frame division.

[0063] An output end of the data acquisition unit is connected to an input end of the frame division unit; an output end of the frame division unit is connected to an input end of the feature unit; and an output end of the feature unit is connected to an input end of the feature engineering module.

[0064] The feature engineering module comprises a label classification unit, a classifier construction unit and a feature extraction unit.

[0065] The label classification unit is configured to manually label the audio and video after frame division processing, classify the audio and video into subsets according to the label types, and form a data set; the classifier construction unit is configured to construct a random forest classifier and train the random forest classifier to measure the importance of the audio signal and video frame image features; and the feature extraction unit is configured to extract important features from the classified subsets.

[0066] An output end of the label classification unit is connected to an input end of the classifier construction unit; an output end of the classifier construction unit is connected to an input end of the feature extraction unit; and an output end of the feature extraction unit is connected to an input end of the anomaly judgment module.

[0067] The anomaly judgment module comprises a cluster center determination unit, a K-means clustering unit and an anomaly judgment unit.

[0068] The cluster center determination unit is configured to determine the mass center of a dense clustering area as an initial cluster center; the K-means clustering unit is configured to perform clustering, judge anomalies, and recompute the center and assign audio signals and video frame images; and the anomaly judgment unit is configured to set a threshold to judge the audio signals and video frame images for anomalies.

[0069] An output end of the cluster center determination unit is connected to an input end of the K-means clustering unit; an output end of the K-means clustering unit is connected to an input end of the anomaly judgment unit; and an output end of the anomaly judgment unit is connected to an input end of the anomaly correction module.

[0070] The anomaly correction module comprises a time series model unit, a correction unit and a restoration unit.

[0071] The time sequence model unit is configured to construct a time sequence model for each audio signal feature, and train the model with the audio signal feature time sequence as input; the correction unit is configured to input the frame number corresponding to the frame determined as abnormal, and correct the abnormal value using the value predicted by the time sequence model; and the restoration unit is configured to restore the corrected audio signal into an audio frame and restore the corrected video frame image into a video frame.

[0072] The output end of the time sequence model unit is connected to the input end of the correction unit, and the output end of the correction unit is connected to the input end of the restoration unit.

[0073] A video and audio data management method based on the Internet of Things, the method comprising the following steps:

[0074] Step 1, obtaining audio and video from a device or a database and framing, representing the audio frame signal and the video frame image in the form of feature values;

[0075] Step 2, manually labeling the audio signal and the video frame image to form a data set and classify, constructing a random forest classifier to extract important features;

[0076] Step 3, judging the abnormal audio signal and the video frame image according to the feature value distribution for the features extracted from the same classified data;

[0077] Step 4, marking the audio signal and the video frame image, constructing a time sequence model, correcting and restoring.

[0078] In Step 1, for audio, the audio is obtained by an audio recording device or from an audio database; the audio is framed to obtain N audio signals; for each audio signal, the audio signal features are extracted and represented as: [A1(i), A2(i), …, An(i)]; n (i)];

[0079] For video, the video is obtained by a video recording device or from a video database; the video is framed to obtain M images; for each video frame image, the video frame image features are extracted and represented as: [V1(j), V2(j), …, Vm(j)]; m (j)];

[0080] wherein M and N are positive integers, representing the number of audio signals and video frame images respectively; i∈{1,2,…,N} represents an audio frame sequence; j∈{1,2,…,M} represents a video frame sequence; n is a positive integer, representing the number of audio signal features, An(i) represents the nth audio signal feature of the ith audio frame; m is a positive integer, representing the number of video frame image features, Vm(j) represents the mth video frame image feature of the jth video frame. n (i)]; m(j) represents the m-th video frame image feature of the j-th video frame.

[0081] In step Step2, each frame of audio signal is manually annotated, and the manually annotated type is represented as: [K1, K2, …, K u ]; the audio signal feature of each frame and its corresponding manually annotated type are obtained to form a data set;

[0082] wherein u is a positive integer, representing the number of audio signal types, K u represents the u-th audio signal type.

[0083] According to different types of manual annotation, the audio signals are classified to obtain classification subsets of different annotation types; a random forest classifier is constructed, and the number of decision trees is set to E; for each classification subset, the audio signal feature data in the subset is used as the input feature, and the corresponding manually annotated type is used as the target output to train the random forest classifier:

[0084] When training each decision tree, samples are randomly drawn with replacement from the data set;

[0085] For each node split, instead of selecting the audio signal feature with the maximum purity from all audio signal features, k audio signal features are randomly selected from all audio signal features; the audio signal feature that maximizes the purity of the split node is selected as the split feature of the node; the node purity is measured by the Gini index;

[0086] For an audio signal feature A r , its importance in a decision tree is calculated: traverse each node in the tree, calculate the Gini index of the node without considering feature A r as a split feature, and obtain Gb(A r ) by weighted summation according to the proportion of the number of samples contained in the node to the total number of samples; traverse each node in the tree, calculate the Gini index of the split node after using feature A r as a split feature, and obtain Ga(A r ) by weighted summation according to the proportion of the number of samples contained in the node to the total number of samples; the importance of feature A r in this decision tree is represented as It(A r )=Gb(A r )-Ga(A r ); for feature A r , its importance If(A r ) in the entire random forest is obtained by averaging the importance of the feature in each decision tree: If(A r )=(1 / E)·Σ(It(A r )).

[0087] wherein r is a positive integer, r∈{1,2,…,n}, representing the audio signal feature sequence; k is a positive integer, k<n, representing the number of selected audio signal features;

[0088] The importance of each audio signal feature to each classification subset is calculated; for each classification subset, the first a features with the largest importance values are selected as the important audio signal features of the subset;

[0089] According to the above manner, the labeled video frame images are classified and the important video frame image features are extracted.

[0090] In step Step3, for the audio signal classified according to different types of artificial labeling, the selected important audio signal features of a classification subset of a certain labeling type are represented as: [F1,F2,…,F α ];

[0091] wherein F1,F2,…,F α ∈{A1,A2,…,A n};

[0092] In the subset, the distribution of different important audio signal features is obtained:

[0093] The Euclidean distance between each audio signal important feature value and all other audio signal important feature values is calculated to form a distance matrix; all distance values in the distance matrix are sorted, and a sorted graph of distance values is drawn; according to the change trend of the distance values in the sorted graph, the point with the largest difference value where the distance value changes from gentle to sharp increase is found, and the corresponding distance value is selected as an initial estimate of the neighborhood radius eps;

[0094] The minimum number of points minPts is set to a+1;

[0095] For each audio signal, the number of data points contained in the eps neighborhood is calculated, and when the number of data points is greater than or equal to minPts, the audio signal is marked as a core point; otherwise, it is marked as a boundary point or a noise point; starting from the core point, the clustering is formed by connecting the density-reachable data points; density-reachable means that there is a data point chain, and each point on the chain is in the eps neighborhood of its predecessor, and the predecessor is a core point;

[0096] Using the DBSCAN algorithm, the data is divided into different clusters; for each identified dense cluster region, the centroid of each cluster is selected as the initial cluster center;

[0097] wherein the centroid represents the average value of all audio signal feature data vectors in the cluster;

[0098] Set the number of clusters to 2, select two of the initial cluster centers, and use the K-means clustering algorithm to divide all audio signals in the classification subset into two clusters;

[0099] After completing a cluster, for each cluster, the cluster center is recalculated: the average of all points after the first clustering is taken as the new two cluster centers; the K-means clustering algorithm is used again to assign different audio signals; the maximum number of iterations T is set, and when the number of iterations reaches T, the iteration is stopped; a threshold G is set, and the distance of each audio signal to the cluster center to which it belongs is calculated, and when the distance is greater than G, the audio signal is considered abnormal;

[0100] Abnormal judgment is performed on the audio signals corresponding to each classification subset;

[0101] According to the above method, abnormal judgment is performed on the video frame images corresponding to each classification subset.

[0102] In step Step4, the audio signals judged to be abnormal are marked, and for each audio signal feature, an audio signal database is constructed: [A v (1), A v (2), …, A v (N)];

[0103] Where v is a positive integer, v∈{1,2,…,n}, representing the audio signal feature sequence; A v (N) represents the vth audio signal feature of the Nth audio frame;

[0104] Further, a time series model LSTM of the audio signal feature A v is constructed, and A v (1), A v (2), …, A v (N) are input to train the model to obtain the trained model; the frame number corresponding to the audio signal judged to be abnormal is input to obtain the audio signal feature A v predicted by the model as a correction of the abnormal value;

[0105] A time series model is constructed for each audio signal feature to correct the audio signal features judged to be abnormal; and the corrected audio signal is restored to an audio frame;

[0106] According to the above method, the video frame image features judged to be abnormal are corrected and restored to video frames.

[0107] In this embodiment: an audio is obtained from an audio database:

[0108] Step 1, frame the audio, and get several frame audio signals; for each frame audio signal, extract the audio signal features, represented as: [zero-crossing rate, energy, spectral centroid, mel-frequency cepstral coefficient];

[0109] Step 2, manually label each frame of audio signal, and the manually labeled type is represented as: [happy, sad, angry, calm]; get the audio signal features and their corresponding manually labeled types for each frame and form a data set;

[0110] Classify the audio signals according to different types of manual labeling, and get classification subsets of different labeling types; build a random forest classifier and set the number of decision trees; for each classification subset, use the audio signal feature data in the subset as input features and the corresponding manually labeled type as target output to train the random forest classifier:

[0111] When training each decision tree, randomly sample from the data set with replacement;

[0112] For each node split, randomly select 2 audio signal features from all audio signal features; select the audio signal feature that maximizes the node purity after splitting as the splitting feature of the node; node purity is measured by Gini index;

[0113] For an audio signal feature spectral centroid, calculate its importance in a decision tree: traverse each node in the tree, calculate the Gini index of the node without considering the feature spectral centroid as the splitting feature, and weight the sum according to the proportion of the number of samples contained in the node to the total number of samples to get Gb(spectral centroid); traverse each node in the tree, calculate the Gini index of the node after splitting when using the feature spectral centroid as the splitting feature, and weight the sum according to the proportion of the number of samples contained in the node to the total number of samples to get Ga(spectral centroid); the importance of the feature spectral centroid in this decision tree is represented as It(spectral centroid)=Gb(spectral centroid)-Ga(spectral centroid); for the feature spectral centroid, its importance If(spectral centroid) in the entire random forest is obtained by averaging the importance of the feature in each decision tree: If(spectral centroid)=(1 / number of decision trees)·Σ(It(spectral centroid));

[0114] Calculate the importance of each audio signal feature for each classification subset; for each classification subset, select the top 3 features with the highest importance value as the important audio signal features of the subset;

[0115] Step Step3, the important audio signal features selected for classifying the audio signals according to different types of artificial labeling are represented as: [zero-crossing rate, spectral centroid, mel-frequency cepstral coefficient];

[0116] In this subset, the distribution of different important audio signal features is obtained:

[0117] The Euclidean distance between each audio signal important feature value and all other audio signal important feature values is calculated to form a distance matrix; all distance values in the distance matrix are sorted, and a sorted distance value graph is drawn; according to the trend of the distance values in the sorted graph, the point where the distance value changes from gentle to sharply increasing is found, and the corresponding distance value is selected as an initial estimate of the neighborhood radius eps;

[0118] The minimum number of points minPts is set to 4;

[0119] For each audio signal, the number of data points contained in the eps neighborhood is calculated, and when the number of data points is greater than or equal to minPts, the audio signal is marked as a core point; otherwise, it is marked as a boundary point or a noise point; starting from the core point, the clustering is formed by connecting the density-reachable data points; density-reachable means that there is a data point chain, and each point in the chain is in the eps neighborhood of its predecessor, and the predecessor is a core point;

[0120] Using the DBSCAN algorithm, the data is divided into different clusters; for each identified dense cluster region, the centroid of each cluster is selected as the initial cluster center;

[0121] The number of clusters is set to 2, two of the initial cluster centers are selected, and the K-means clustering algorithm is used to divide all audio signals in the classification subset into two clusters;

[0122] After completing a clustering, for each cluster, the cluster centers are recalculated: the average of all points after the first clustering is taken as the new two cluster centers; the K-means clustering algorithm is used again to assign different audio signals; the maximum number of iterations is set, and when the number of iterations reaches the maximum, the iteration is stopped; a threshold is set, and the distance of each audio signal to its cluster centroid is calculated, and when the distance is greater than the threshold, the audio signal is considered abnormal;

[0123] For each classification subset, the corresponding audio signal is judged for abnormality;

[0124] Step Step4, mark the audio signals judged to be abnormal; for each audio signal feature, an audio signal database is constructed;

[0125] Further construct an audio signal feature Av a time series model LSTM, trains the model with each frame feature value of the audio signal as input, and obtains a trained model; inputs the frame number corresponding to the audio signal determined as abnormal, and obtains the model-predicted feature of the audio signal as a correction to the abnormal value;

[0126] corrects the audio signal feature determined as abnormal by constructing a time series model for each audio signal feature; and restores the corrected audio signal feature to an audio frame.

[0127] It is apparent for those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as illustrative and not restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all changes falling within the meaning and range of equivalents of the elements of the claims. Any reference signs in the claims should not be considered as limiting the claims involved.

Claims

1. A method for managing audio and video data based on Internet of Things, characterized in that: The method comprises the following steps: Step 1: Obtain audio and video from equipment or database and frame, and represent audio frame signal and video frame image in the form of feature value; Step 2: Artificially mark audio signal and video frame image to form a data set and classify, and construct a random forest classifier to extract important features; Step 3: For the features extracted from the same classified data, judge abnormal audio signal and video frame image according to the feature value distribution; Step 4: Mark audio signal and video frame image, construct a time sequence model, correct and restore; In step Step1, for audio, audio recording device is taken or from the audio database; audio is framed, and N audio signals are obtained; for each audio signal, audio signal features are extracted, represented as: [A1(i), A2(i), …, A n (i)]; For video, recorded by a video recording device or obtained from a video database; frame the video, and obtain M images respectively; for each video frame image, extract the video frame image features, represented as: [V1(j), V2(j), …, V m (j)]; wherein M, N are positive integers, respectively representing the number of audio signals and video frame images; i∈{1, 2, …, N} represents an audio frame sequence; j∈{1, 2, …, M} represents a video frame sequence; n is a positive integer, representing the number of audio signal features, A n (i) represents the n-th audio signal feature of the i-th audio frame; m is a positive integer, representing the number of video frame image features, V m (j) represents the m-th video frame image feature of the j-th video frame. In step Step2, each frame of audio signal is manually labeled, and the manually labeled type is represented as: [K1, K2, …, K u ]; to obtain the audio signal feature of each frame and its corresponding manually labeled type and form a data set; wherein u is a positive integer, represents the number of audio signal types, K u represents the u-th audio signal type; Classify the audio signal according to different types of artificial marking to obtain classified subsets of different marking types; construct a random forest classifier, and set the number of decision trees as E; for each classified subset, use the audio signal feature data in the subset as input features, and use the corresponding artificial marking type as target output to train the random forest classifier: When training each decision tree, randomly sample from the data set with replacement; For the splitting of each node, randomly select k audio signal features from all audio signal features; select the audio signal feature that maximizes the purity of the node after splitting as the splitting feature of the node; the node purity is measured by the Gini index; For an audio signal feature A r , its importance in a decision tree is calculated: traverse each node in the tree, calculate the Gini index of the node without considering feature A r as the split feature, and obtain Gb(A r ) by weighted sum according to the proportion of the number of samples contained in the node in the total number of samples; traverse each node in the tree, calculate the Gini index of the split child node of the node using feature A r as the split feature, and obtain Ga(A r ) by weighted sum according to the proportion of the number of samples contained in the child node in the total number of samples; the importance of feature A r in the decision tree is represented as It(A r )=Gb(A r )-Ga(A r ); for feature A r , its importance If(A r ) in the entire random forest is obtained by averaging the importance of the feature in each decision tree: If(A r )=(1 / E)·Σ(It(A r )). Wherein, r is a positive integer, r∈{1,2,…,n}, representing the sequence of audio signal features; k is a positive integer, k<n, representing the number of selected audio signal features; Calculate the importance of each audio signal feature for each classified subset; for each classified subset, select the top alpha features with the largest importance value as the important audio signal features of the subset; Classify the marked video frame image and extract important video frame image features according to the above method; In step Step3, for classifying the audio signals according to different types of artificial labeling, a certain classification subset of different labeled types is obtained, and the selected important audio signal features are represented as: [F1, F2, …, F α ];wherein F1, F2, …, F α ∈{A1, A2, …, A n}; In the subset, obtain the distribution of different important audio signal features: Calculate the Euclidean distance between each audio signal important feature value and all other audio signal important feature values to form a distance matrix; sort all distance values in the distance matrix and draw a sorted distance value graph; according to the trend of distance value in the sorted graph, find the point where the distance value changes from gentle to sharply increasing, and select the corresponding distance value as an initial estimate of the neighborhood radius eps; Set the minimum number of points minPts to alpha+1; For each audio signal, calculate the number of data points contained in the eps neighborhood, and when the number of data points is greater than or equal to minPts, mark the audio signal as a core point; otherwise, mark it as a boundary point or a noise point; start from the core point, and form clusters by connecting density reachable data points; density reachable means that there is a chain of data points, and each point in the chain is in the eps neighborhood of its predecessor, and the predecessor is a core point; Use the DBSCAN algorithm to divide the data into different clusters; for each identified dense cluster region, select the centroid of each cluster as the initial cluster center; Wherein, the centroid represents the average value of all audio signal feature data vectors in the cluster; Set the number of clusters to 2, select two of the initial cluster centers, and use the K-means clustering algorithm to divide all audio signals in the classification subset into two clusters; After completing a clustering, for each cluster, the cluster center is recalculated: the average of all points after the first clustering is taken as the new two cluster centers; the K-means clustering algorithm is used again to assign different audio signals; the maximum number of iterations T is set, and the iteration is stopped when the number of iterations reaches T; a threshold G is set, and the distance of each audio signal to the cluster center to which it belongs is calculated, and when the distance is greater than G, the audio signal is considered abnormal; Abnormality judgment is performed on the audio signals corresponding to each classification subset; Abnormality judgment is performed on the video frame images corresponding to each classification subset according to the above method; In step Step4, the audio signals judged as abnormal are marked, and for each audio signal feature, an audio signal database is constructed: [A v (1), A v (2), …, A v (N)]; wherein v is a positive integer, v e {1, 2, …, n}, representing an audio signal feature sequence; A v (N) represents the vth audio signal feature of the Nth audio frame. Further construct a time series model LSTM of audio signal feature A v , train the model with A v (1), A v (2), …, A v (N) as input, obtain the trained model; input the frame number corresponding to the audio signal judged as abnormal, obtain the audio signal feature A v predicted by the model as the correction of the abnormal value; A time series model is constructed for each audio signal feature to correct the abnormal audio signal features; the corrected audio signal features are restored to audio frames; Abnormal video frame features are corrected according to the above method and restored to video frames.

2. An audio and video data management system based on the Internet of Things, applying the audio and video data management method based on the Internet of Things according to claim 1, characterized in that: The system comprises a preprocessing module, a feature engineering module, an abnormality judgment module, and an abnormality repair module; The preprocessing module is used to obtain audio and video data, perform frame processing and extract features, and provide basic data preparation for subsequent analysis; the feature engineering module is used to manually label and classify audio signals and video frame images, construct a classifier, and extract important features, which provide the basis for abnormality judgment; the abnormality judgment module is used to determine the cluster center, perform clustering operation, and judge whether the audio signal and video frame image are abnormal, and accurately identify abnormal data; the abnormality repair module is used to construct a time series model to correct abnormal data and restore it; The output end of the preprocessing module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the abnormality judgment module; the output end of the abnormality judgment module is connected to the input end of the abnormality repair module. 3.The video and audio data management system based on the Internet of Things according to claim 2, characterized in that: The preprocessing module comprises a data acquisition unit, a frame unit, and a feature unit; The data acquisition unit is used to acquire audio and video data; the frame unit is used to perform frame processing on the acquired audio and video; The feature unit is used to extract the features of the audio and video after frame processing; The output end of the data acquisition unit is connected to the input end of the frame unit; the output end of the frame unit is connected to the input end of the feature unit; the output end of the feature unit is connected to the input end of the feature engineering module. 4.The video and audio data management system based on the Internet of Things according to claim 3, characterized in that: The feature engineering module comprises a labeling classification unit, a classifier construction unit, and a feature extraction unit; The labeling classification unit is used to manually label the audio and video after frame processing, classify them into subsets according to the labeling type, and form a data set; the classifier construction unit is used to construct a random forest classifier and train it to measure the importance of audio signal and video frame image features; the feature extraction unit is used to extract important features from the classification subsets; An output end of the labeling classification unit is connected with an input end of the classifier construction unit; an output end of the classifier construction unit is connected with an input end of the feature extraction unit; and an output end of the feature extraction unit is connected with an input end of the anomaly judgment module. 5.The video and audio data management system based on the Internet of Things according to claim 4, characterized in that: The anomaly judgment module comprises a cluster center determination unit, a K-means clustering unit and an anomaly judgment unit. The cluster center determination unit is used for determining the center of a dense clustering area as an initial cluster center; the K-means clustering unit is used for clustering, judging anomaly, re-computing the center and assigning audio signals and video frame images; and the anomaly judgment unit is used for setting a threshold to judge the anomaly of the audio signals and the video frame images. An output end of the cluster center determination unit is connected with an input end of the K-means clustering unit; an output end of the K-means clustering unit is connected with an input end of the anomaly judgment unit; and an output end of the anomaly judgment unit is connected with an input end of the anomaly repair module. 6.The video and audio data management system based on the Internet of Things according to claim 5, characterized in that: The anomaly repair module comprises a time sequence model unit, a correction unit and a restoration unit. The time sequence model unit is used for constructing a time sequence model for each audio signal feature, and training the model by taking the time sequence of the audio signal feature as input; the correction unit is used for inputting the frame number corresponding to the frame judged as abnormal, and correcting the abnormal value by using the value predicted by the time sequence model; and the restoration unit is used for restoring the corrected audio signal into an audio frame and restoring the corrected video frame image into a video frame. An output end of the time sequence model unit is connected with an input end of the correction unit; and an output end of the correction unit is connected with an input end of the restoration unit.

Images