Audio and video data management system and method based on Internet of Things

Abstract

The invention discloses a video and audio data management system and method based on the Internet of Things, and relates to the technical field of Internet of Things data analysis, and the system comprises a preprocessing module, a feature engineering module, an abnormity judgment module and an abnormity restoration module. The preprocessing module is used for acquiring audio and video data, performing framing processing and extracting features; the feature engineering module is used for manually marking and classifying audio signals and video frame images, constructing a classifier, extracting important features and providing a basis for abnormality judgment; the abnormity judgment module is used for determining a clustering center, performing clustering operation, judging whether the audio signal and the video frame image are abnormal or not, and accurately identifying abnormal data; and the exception repairing module is used for constructing a time sequence model to correct and restore the exceptional data. The invention also provides a method for realizing the system. According to the invention, the situation that part of abnormal frames influence management in the prior art can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things data analysis, and in particular to an audio and video data management system and method based on the Internet of Things. Background Art

[0002] Through the Internet of Things technology, various audio and video acquisition devices such as cameras and microphones can be connected to the network to achieve interconnection between devices. These devices can collect audio and video data in real time and transmit it to a designated storage or processing center for centralized management and analysis. Using big data processing technology and machine learning algorithms, massive amounts of audio and video data can be mined and feature extracted to discover hidden information and patterns; multi-dimensional data provides richer materials for audio and video data management, which helps to more fully understand user needs and preferences.

[0003] In the process of collecting, transmitting and storing audio and video data, due to noisy images, equipment problems, network problems, storage media problems and other reasons, there will often be noise or poor quality frames in the audio and video signals in large-scale data, affecting the validity of the data and the accuracy of the analysis. Summary of the invention

[0004] The purpose of the present invention is to provide an audio and video data management system and method based on the Internet of Things to solve the problems raised in the prior art.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an audio and video data management system based on the Internet of Things, the system comprising a preprocessing module, a feature engineering module, an abnormality judgment module and an abnormality correction module; The preprocessing module is used to obtain audio and video data, perform frame processing and extract features, and provide basic data preparation for subsequent analysis; the feature engineering module is used to manually label and classify audio signals and video frame images, build a classifier and extract important features, and provide a basis for abnormal judgment; the abnormal judgment module is used to determine the cluster center, perform clustering operations and judge whether the audio signal and video frame image are abnormal, and accurately identify abnormal data; the abnormal correction module is used to construct a time series model to correct abnormal data and restore it; The output end of the preprocessing module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the abnormality judgment module; the output end of the abnormality judgment module is connected to the input end of the abnormality correction module.

[0006] The preprocessing module includes a data acquisition unit, a frame division unit and a feature unit; The data acquisition unit is used to acquire audio and video data; the framing unit is used to perform framing processing on the acquired audio and video; the feature unit is used to extract features of the audio and video after framing; The output end of the data acquisition unit is connected to the input end of the framing unit; the output end of the framing unit is connected to the input end of the feature unit; the output end of the feature unit is connected to the input end of the feature engineering module.

[0007] The feature engineering module includes a labeling classification unit, a classifier construction unit and a feature extraction unit; The annotation classification unit is used to manually annotate the audio and video after the frame processing, and classify them into subsets according to the annotation type to form a data set; the classifier construction unit is used to construct and train a random forest classifier to measure the importance of audio signal and video frame image features; the feature extraction unit is used to extract important features from the classification subset; The output end of the labeling classification unit is connected to the input end of the classifier construction unit; the output end of the classifier construction unit is connected to the input end of the feature extraction unit; the output end of the feature extraction unit is connected to the input end of the abnormality judgment module.

[0008] The abnormality judgment module includes a cluster center determination unit, a K-means clustering unit and an abnormality judgment unit; The cluster center determination unit is used to determine the centroid of the dense cluster area as the initial cluster center; the K-means clustering unit is used to perform clustering, judge abnormalities, and recalculate the center and distribute audio signals and video frame images; the abnormality judgment unit is used to set a threshold to judge abnormalities of audio signals and video frame images; The output end of the cluster center determination unit is connected to the input end of the K-means clustering unit; the output end of the K-means clustering unit is connected to the input end of the abnormality judgment unit; the output end of the abnormality judgment unit is connected to the input end of the abnormality correction module.

[0009] The anomaly correction module includes a time series model unit, a correction unit and a restoration unit; The time series model unit is used to construct a time series model for each audio signal feature, and train the model with the audio signal feature time series as input; the correction unit is used to input the frame number corresponding to the frame judged to be abnormal, and correct the abnormal value using the value predicted by the time series model; the restoration unit is used to restore the corrected audio signal to an audio frame, and restore the corrected video frame image to a video frame; The output end of the time series model unit is connected to the input end of the correction unit; the output end of the correction unit is connected to the input end of the restoration unit.

[0010] A method for managing audio and video data based on the Internet of Things, the method comprising the following steps: Step 1, obtain audio and video from the device or database and divide them into frames, and represent the audio frame signal and video frame image in the form of feature values; Step 2: Manually label audio signals and video frame images to form a data set and classify them, and build a random forest classifier to extract important features; Step 3: For the features extracted from the same classification data, abnormal audio signals and video frame images are judged according to the distribution of feature values; Step 4: Mark the audio signal and video frame images, build a time series model to correct and restore them.

[0011] In step 1, the audio is recorded by an audio recording device or obtained from an audio database; the audio is divided into frames to obtain N frames of audio signals; for each frame of the audio signal, the audio signal features are extracted, which are expressed as: [A1(i), A2(i),…, A n (i)]; For the video, it is recorded by a video recording device or obtained from a video database; the video is divided into frames to obtain M images respectively; for each video frame image, the video frame image features are extracted, which are expressed as: [V1(j), V2(j), …, V m (j)]; Where M and N are positive integers, representing the number of audio signals and video frame images respectively; i∈{1,2,…,N} represents an audio frame sequence; j∈{1,2,…,M} represents a video frame sequence; n is a positive integer representing the number of audio signal features, A n (i) represents the nth audio signal feature of the ith audio frame; m is a positive integer, representing the number of video frame image features, V m (j) represents the mth video frame image features of the jth video frame.

[0012] In step 2, each frame of audio signal is manually annotated, and the manual annotation type is expressed as: [K1, K2, …, K u ]; obtain the audio signal features of each frame and its corresponding manual annotation type and form a data set; Among them, u is a positive integer, indicating the number of types of audio signals, K u Indicates the u-th audio signal type; Classify the audio signals according to different types of manual annotations to obtain classification subsets with different annotation types; construct a random forest classifier and set the number of decision trees to E; for each classification subset, use the audio signal feature data in the subset as input features and the corresponding manual annotation type as the target output to train the random forest classifier: When training each decision tree, samples are randomly drawn from the dataset with replacement; For each node split, k audio signal features are randomly selected from all audio signal features; the audio signal feature that maximizes the purity of the node after splitting is selected as the split feature of the node; the node purity is measured by the Gini index; For an audio signal feature A r , calculate its importance in a decision tree: traverse each node in the tree and calculate the importance of r The Gini index of the node when used as a split feature is obtained by weighted summing the number of samples contained in the node according to the proportion of the total number of samples (Gb(A) r ); traverse each node in the tree and calculate the r When the node is used as the split feature, the Gini index of the child node after the node is split is weighted and summed according to the proportion of the number of samples contained in the child node to the total number of samples to obtain Ga(A r ); Feature A r The importance in this decision tree is represented by It(A r )=Gb(A r )-Ga(A r ); for feature A r , its importance in the entire random forest If(A r ) is obtained by averaging the importance of the feature in each decision tree: If(A r )=(1 / E)·Σ(It(A r )); Wherein, r is a positive integer, r∈{1,2,…,n}, representing the audio signal feature sequence; k is a positive integer, k<n, representing the number of selected audio signal features; Calculate the importance of each audio signal feature to each classification subset; for each classification subset, select the first α features with the largest importance value as the important audio signal features of the subset; According to the above method, the annotated video frame images are classified and important video frame image features are extracted.

[0013] In step 3, the audio signal is classified according to different types of manual annotation to obtain a classification subset of different annotation types, and the important audio signal features selected are expressed as: [F1, F2, ..., F α ]; Among them, F1, F2, …, F α ∈{A1,A2,…,A n}; In this subset, obtain the distribution of different important audio signal features: Calculate the Euclidean distance between each important eigenvalue of the audio signal and all other important eigenvalues ​​of the audio signal to form a distance matrix; sort all the distance values ​​in the distance matrix and draw a sorting diagram of the distance values; according to the changing trend of the distance values ​​in the sorting diagram, find the point where the distance value changes from a gentle to a sharp increase with the largest difference, and select the corresponding distance value as an initial estimate of the neighborhood radius eps; The minimum number of points minPts is set to α+1; For each audio signal, calculate the number of data points contained in its eps neighborhood. When the number of data points is greater than or equal to minPts, mark the audio signal as a core point; otherwise, mark it as a boundary point or a noise point. Starting from the core point, clusters are formed by connecting density-reachable data points. Density-reachable means that there is a chain of data points, each point in the chain is in the eps neighborhood of its predecessor point, and the predecessor point is a core point. Using the DBSCAN algorithm, the data was divided into different clusters; for each identified dense cluster area, the centroid of each cluster was selected as the initial cluster center; Among them, the centroid represents the average value of all audio signal feature data vectors in the cluster; Set the number of clusters to 2, select two of the initial cluster centers, and use the K-means clustering algorithm to divide all audio signals of the classification subset into two clusters; After completing a clustering, for each cluster, recalculate the cluster center: use the average value of all points after the first clustering as the new two cluster centers; use the K-means clustering algorithm again to distribute different audio signals; set the maximum number of iterations T, and stop the iteration when the number of iterations reaches T; set the threshold G, calculate the distance from each audio signal to the centroid of the cluster to which it belongs, and when the distance is greater than G, the audio signal is considered abnormal; Perform abnormality judgment on the audio signal corresponding to each classification subset; According to the above method, the video frame image corresponding to each classification subset is judged to be abnormal.

[0014] In step 4, the audio signals judged as abnormal are marked, and for each audio signal feature, an audio signal database is constructed: [A v (1),A v (2),…,A v (N)]; Where v is a positive integer, v∈{1,2,…,n}, representing the audio signal feature sequence; A v (N) represents the vth audio signal feature of the Nth audio frame; Further construct the audio signal feature A v The time series model LSTM, with A v (1),A v (2),…,A v (N) is used as input to train the model and obtain the trained model; the number of frames corresponding to the audio signal judged as abnormal is input to obtain the audio signal feature A predicted by the model v as a correction for outliers; A time series model is constructed for each audio signal feature to correct the audio signal features judged to be abnormal; the corrected audio signal features are restored to audio frames; According to the above method, the image features of the video frame determined to be abnormal are corrected and restored to the video frame.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention mines out the most representative and key features for different annotation types, so that subsequent processing can focus more on important features related to specific tasks, thereby improving processing efficiency and accuracy; the present invention clusters audio signals and video frame images by using the DBSCAN algorithm in combination with K-means, and performs anomaly detection by judging the density of the centroid and the neighborhood, so as to efficiently identify and correct abnormal data points; the present invention performs time series repair on abnormal audio signal features through LSTM, which can not only correct anomalies in the data, but also smooth the data according to the results of model prediction, thereby ensuring the quality and consistency of audio and video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of an audio and video data management system based on the Internet of Things of the present invention; Figure 2 The figure is a schematic diagram of the steps of an audio and video data management method based on the Internet of Things of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, an audio and video data management system based on the Internet of Things, the system includes a preprocessing module, a feature engineering module, an abnormality judgment module and an abnormality correction module; The preprocessing module is used to obtain audio and video data, perform frame processing and extract features, and provide basic data preparation for subsequent analysis; the feature engineering module is used to manually label and classify audio signals and video frame images, build a classifier and extract important features, and provide a basis for abnormal judgment; the abnormal judgment module is used to determine the cluster center, perform clustering operations and judge whether the audio signal and video frame image are abnormal, and accurately identify abnormal data; the abnormal correction module is used to construct a time series model to correct abnormal data and restore it; The output end of the preprocessing module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the abnormality judgment module; the output end of the abnormality judgment module is connected to the input end of the abnormality correction module.

[0019] The preprocessing module includes a data acquisition unit, a frame division unit and a feature unit; The data acquisition unit is used to acquire audio and video data; the framing unit is used to perform framing processing on the acquired audio and video; the feature unit is used to extract features of the audio and video after framing; The output end of the data acquisition unit is connected to the input end of the framing unit; the output end of the framing unit is connected to the input end of the feature unit; the output end of the feature unit is connected to the input end of the feature engineering module.

[0020] The feature engineering module includes a labeling classification unit, a classifier construction unit and a feature extraction unit; The annotation classification unit is used to manually annotate the audio and video after the frame processing, and classify them into subsets according to the annotation type to form a data set; the classifier construction unit is used to construct and train a random forest classifier to measure the importance of audio signal and video frame image features; the feature extraction unit is used to extract important features from the classification subset; The output end of the labeling classification unit is connected to the input end of the classifier construction unit; the output end of the classifier construction unit is connected to the input end of the feature extraction unit; the output end of the feature extraction unit is connected to the input end of the abnormality judgment module.

[0021] The abnormality judgment module includes a cluster center determination unit, a K-means clustering unit and an abnormality judgment unit; The cluster center determination unit is used to determine the centroid of the dense cluster area as the initial cluster center; the K-means clustering unit is used to perform clustering, judge abnormalities, and recalculate the center and distribute audio signals and video frame images; the abnormality judgment unit is used to set a threshold to judge abnormalities of audio signals and video frame images; The output end of the cluster center determination unit is connected to the input end of the K-means clustering unit; the output end of the K-means clustering unit is connected to the input end of the abnormality judgment unit; the output end of the abnormality judgment unit is connected to the input end of the abnormality correction module.

[0022] The anomaly correction module includes a time series model unit, a correction unit and a restoration unit; The time series model unit is used to construct a time series model for each audio signal feature, and train the model with the audio signal feature time series as input; the correction unit is used to input the frame number corresponding to the frame judged to be abnormal, and correct the abnormal value using the value predicted by the time series model; the restoration unit is used to restore the corrected audio signal to an audio frame, and restore the corrected video frame image to a video frame; The output end of the time series model unit is connected to the input end of the correction unit; the output end of the correction unit is connected to the input end of the restoration unit.

[0023] A method for managing audio and video data based on the Internet of Things, the method comprising the following steps: Step 1, obtain audio and video from the device or database and divide them into frames, and represent the audio frame signal and video frame image in the form of feature values; Step 2: Manually label audio signals and video frame images to form a data set and classify them, and build a random forest classifier to extract important features; Step 3: For the features extracted from the same classification data, abnormal audio signals and video frame images are judged according to the distribution of feature values; Step 4: Mark the audio signal and video frame images, build a time series model to correct and restore them.

[0024] In step 1, the audio is recorded by an audio recording device or obtained from an audio database; the audio is divided into frames to obtain N frames of audio signals; for each frame of the audio signal, the audio signal features are extracted, which are expressed as: [A1(i), A2(i),…, A n (i)]; For the video, it is recorded by a video recording device or obtained from a video database; the video is divided into frames to obtain M images respectively; for each video frame image, the video frame image features are extracted, which are expressed as: [V1(j), V2(j), …, V m (j)]; Where M and N are positive integers, representing the number of audio signals and video frame images respectively; i∈{1,2,…,N} represents an audio frame sequence; j∈{1,2,…,M} represents a video frame sequence; n is a positive integer representing the number of audio signal features, A n (i) represents the nth audio signal feature of the ith audio frame; m is a positive integer, representing the number of video frame image features, V m (j) represents the mth video frame image features of the jth video frame.

[0025] In step 2, each frame of audio signal is manually annotated, and the manual annotation type is expressed as: [K1, K2, …, K u ]; obtain the audio signal features of each frame and its corresponding manual annotation type and form a data set; Among them, u is a positive integer, indicating the number of types of audio signals, K u Indicates the u-th audio signal type; Classify the audio signals according to different types of manual annotations to obtain classification subsets with different annotation types; construct a random forest classifier and set the number of decision trees to E; for each classification subset, use the audio signal feature data in the subset as input features and the corresponding manual annotation type as the target output to train the random forest classifier: When training each decision tree, samples are randomly drawn from the dataset with replacement; For each node split, k audio signal features are randomly selected from all audio signal features; the audio signal feature that maximizes the purity of the node after splitting is selected as the split feature of the node; the node purity is measured by the Gini index; For an audio signal feature A r , calculate its importance in a decision tree: traverse each node in the tree and calculate the importance of rThe Gini index of the node when used as a split feature is obtained by weighted summing the number of samples contained in the node according to the proportion of the total number of samples (Gb(A) r ); traverse each node in the tree and calculate the r When the node is used as the split feature, the Gini index of the child node after the node is split is weighted and summed according to the proportion of the number of samples contained in the child node to the total number of samples to obtain Ga(A r ); Feature A r The importance in this decision tree is represented by It(A r )=Gb(A r )-Ga(A r ); for feature A r , its importance in the entire random forest If(A r ) is obtained by averaging the importance of the feature in each decision tree: If(A r )=(1 / E)·Σ(It(A r )); Wherein, r is a positive integer, r∈{1,2,…,n}, representing the audio signal feature sequence; k is a positive integer, k<n, representing the number of selected audio signal features; Calculate the importance of each audio signal feature to each classification subset; for each classification subset, select the first α features with the largest importance value as the important audio signal features of the subset; The annotated video frame images are classified and important video frame image features are extracted according to the above method.

[0026] In step 3, the audio signal is classified according to different types of manual annotation to obtain a classification subset of different annotation types, and the important audio signal features selected are expressed as: [F1, F2, ..., F α ]; Among them, F1, F2, …, F α ∈{A1,A2,…,A n}; In this subset, obtain the distribution of different important audio signal features: Calculate the Euclidean distance between each important eigenvalue of the audio signal and all other important eigenvalues ​​of the audio signal to form a distance matrix; sort all the distance values ​​in the distance matrix and draw a sorting diagram of the distance values; according to the changing trend of the distance values ​​in the sorting diagram, find the point where the distance value changes from a gentle to a sharp increase with the largest difference, and select the corresponding distance value as an initial estimate of the neighborhood radius eps; The minimum number of points minPts is set to α+1; For each audio signal, calculate the number of data points contained in its eps neighborhood. When the number of data points is greater than or equal to minPts, mark the audio signal as a core point; otherwise, mark it as a boundary point or a noise point. Starting from the core point, clusters are formed by connecting density-reachable data points. Density-reachable means that there is a chain of data points, each point in the chain is in the eps neighborhood of its predecessor point, and the predecessor point is a core point. Using the DBSCAN algorithm, the data was divided into different clusters; for each identified dense cluster area, the centroid of each cluster was selected as the initial cluster center; Among them, the centroid represents the average value of all audio signal feature data vectors in the cluster; Set the number of clusters to 2, select two of the initial cluster centers, and use the K-means clustering algorithm to divide all audio signals of the classification subset into two clusters; After completing a clustering, for each cluster, recalculate the cluster center: use the average value of all points after the first clustering as the new two cluster centers; use the K-means clustering algorithm again to distribute different audio signals; set the maximum number of iterations T, and stop the iteration when the number of iterations reaches T; set the threshold G, calculate the distance from each audio signal to the centroid of the cluster to which it belongs, and when the distance is greater than G, the audio signal is considered abnormal; Perform abnormality judgment on the audio signal corresponding to each classification subset; According to the above method, the video frame image corresponding to each classification subset is judged to be abnormal.

[0027] In step 4, the audio signals judged as abnormal are marked, and for each audio signal feature, an audio signal database is constructed: [A v (1),A v (2),…,A v (N)]; Where v is a positive integer, v∈{1,2,…,n}, representing the audio signal feature sequence; A v (N) represents the vth audio signal feature of the Nth audio frame; Further construct the audio signal feature A v The time series model LSTM, with A v (1),A v (2),…,A v (N) is used as input to train the model and obtain the trained model; the number of frames corresponding to the audio signal judged as abnormal is input to obtain the audio signal feature A predicted by the model v as a correction for outliers; A time series model is constructed for each audio signal feature to correct the audio signal features judged to be abnormal; the corrected audio signal is restored to an audio frame; According to the above method, the image features of the video frame determined to be abnormal are corrected and restored to the video frame.

[0028] In this embodiment: Get a piece of audio from the audio database: Step 1, the audio is divided into frames to obtain several frames of audio signals respectively; for each frame of the audio signal, the audio signal features are extracted, which are expressed as: [zero crossing rate, energy, spectrum centroid, Mel frequency cepstrum coefficient]; Step 2: manually annotate each frame of audio signal, where the manual annotation type is expressed as: [happy, sad, angry, calm]; obtain the audio signal features of each frame and its corresponding manual annotation type to form a data set; Classify the audio signals according to different types of manual annotations to obtain classification subsets with different annotation types; build a random forest classifier and set the number of decision trees; for each classification subset, use the audio signal feature data in the subset as input features and the corresponding manual annotation type as the target output to train the random forest classifier: When training each decision tree, samples are randomly drawn from the dataset with replacement; For each node split, two audio signal features are randomly selected from all audio signal features; the audio signal feature that maximizes the purity of the node after splitting is selected as the split feature of the node; the node purity is measured by the Gini index; For a characteristic spectrum centroid of an audio signal, calculate its importance in a decision tree: traverse each node in the tree, calculate the Gini index of the node when the characteristic spectrum centroid is not considered as a split feature, and perform weighted summation according to the proportion of the number of samples contained in the node to the total number of samples to obtain Gb (spectrum centroid); traverse each node in the tree, calculate the Gini index of the child nodes after the node is split when the characteristic spectrum centroid is used as a split feature, and perform weighted summation according to the proportion of the number of samples contained in the child nodes to the total number of samples to obtain Ga (spectrum centroid); The importance of the characteristic spectrum centroid in this decision tree is expressed as It (spectrum centroid) = Gb (spectrum centroid) - Ga (spectrum centroid); For the characteristic spectrum centroid, its importance If (spectrum centroid) in the entire random forest is obtained by averaging the importance of the feature in each decision tree: If (spectrum centroid) = (1 / number of decision trees) Σ(It (spectrum centroid)); Calculate the importance of each audio signal feature to each classification subset; for each classification subset, select the top three features with the largest importance values ​​as the important audio signal features of the subset; Step 3, classifying the audio signal according to different types of manual annotation to obtain a classification subset of different annotation types, and selecting important audio signal features, which are expressed as: [zero crossing rate, spectrum centroid, Mel frequency cepstrum coefficient]; In this subset, obtain the distribution of different important audio signal features: Calculate the Euclidean distance between each important eigenvalue of the audio signal and all other important eigenvalues ​​of the audio signal to form a distance matrix; sort all the distance values ​​in the distance matrix and draw a sorting diagram of the distance values; according to the changing trend of the distance values ​​in the sorting diagram, find the point where the distance value changes from a gentle to a sharp increase with the largest difference, and select the corresponding distance value as an initial estimate of the neighborhood radius eps; The minimum number of points minPts is set to 4; For each audio signal, calculate the number of data points contained in its eps neighborhood. When the number of data points is greater than or equal to minPts, mark the audio signal as a core point; otherwise, mark it as a boundary point or a noise point. Starting from the core point, clusters are formed by connecting density-reachable data points. Density-reachable means that there is a chain of data points, each point in the chain is in the eps neighborhood of its predecessor point, and the predecessor point is a core point. Using the DBSCAN algorithm, the data was divided into different clusters; for each identified dense cluster area, the centroid of each cluster was selected as the initial cluster center; Set the number of clusters to 2, select two of the initial cluster centers, and use the K-means clustering algorithm to divide all audio signals of the classification subset into two clusters; After completing a clustering, for each cluster, recalculate the cluster center: use the average value of all points after the first clustering as the new two cluster centers; use the K-means clustering algorithm again to distribute different audio signals; set the maximum number of iterations, and stop the iteration when the number of iterations reaches the maximum; set the threshold, calculate the distance from each audio signal to the centroid of the cluster to which it belongs, and when the distance is greater than the threshold, the audio signal is considered abnormal; Perform abnormality judgment on the audio signal corresponding to each classification subset; Step 4: Mark the audio signal judged to be abnormal, and for each audio signal feature, build an audio signal database; Further construct the audio signal feature A vThe time series model LSTM is trained by taking the feature value of each frame of the audio signal as input to obtain a trained model; the number of frames corresponding to the audio signal judged to be abnormal is input to obtain the audio signal feature predicted by the model as a correction for the abnormal value; A time series model is constructed for each audio signal feature to correct the audio signal features judged to be abnormal; and the corrected audio signal features are restored to audio frames.

[0029] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An audio and video data management system based on the Internet of Things, characterized by: The system includes a preprocessing module, a feature engineering module, an anomaly judgment module and an anomaly correction module; The preprocessing module is used to obtain audio and video data, perform frame processing and extract features, and provide basic data preparation for subsequent analysis; the feature engineering module is used to manually label and classify audio signals and video frame images, build a classifier and extract important features, and provide a basis for abnormal judgment; the abnormal judgment module is used to determine the cluster center, perform clustering operations and judge whether the audio signal and video frame image are abnormal, and accurately identify abnormal data; the abnormal correction module is used to construct a time series model to correct abnormal data and restore it; The output end of the preprocessing module is connected to the input end of the feature engineering module; the output end of the feature engineering module is connected to the input end of the abnormality judgment module; the output end of the abnormality judgment module is connected to the input end of the abnormality correction module.

2. The audio and video data management system based on the Internet of Things according to claim 1, characterized in that: The preprocessing module includes a data acquisition unit, a frame division unit and a feature unit; The data acquisition unit is used to acquire audio and video data; the framing unit is used to perform framing processing on the acquired audio and video; The feature unit is used to extract the features of the audio and video after framing; The output end of the data acquisition unit is connected to the input end of the framing unit; the output end of the framing unit is connected to the input end of the feature unit; the output end of the feature unit is connected to the input end of the feature engineering module.

3. The audio and video data management system based on the Internet of Things according to claim 2, characterized in that: The feature engineering module includes a labeling classification unit, a classifier construction unit and a feature extraction unit; The annotation classification unit is used to manually annotate the audio and video after the frame processing, and classify them into subsets according to the annotation type to form a data set; the classifier construction unit is used to construct and train a random forest classifier to measure the importance of audio signal and video frame image features; the feature extraction unit is used to extract important features from the classification subset; The output end of the labeling classification unit is connected to the input end of the classifier construction unit; the output end of the classifier construction unit is connected to the input end of the feature extraction unit; the output end of the feature extraction unit is connected to the input end of the abnormality judgment module.

4. The audio and video data management system based on the Internet of Things according to claim 3 is characterized by: The abnormality judgment module includes a cluster center determination unit, a K-means clustering unit and an abnormality judgment unit; The cluster center determination unit is used to determine the centroid of the dense cluster area as the initial cluster center; the K-means clustering unit is used to perform clustering, judge abnormalities, and recalculate the center and distribute audio signals and video frame images; the abnormality judgment unit is used to set a threshold to judge abnormalities of audio signals and video frame images; The output end of the cluster center determination unit is connected to the input end of the K-means clustering unit; the output end of the K-means clustering unit is connected to the input end of the abnormality judgment unit; the output end of the abnormality judgment unit is connected to the input end of the abnormality correction module.

5. The audio and video data management system based on the Internet of Things according to claim 4, characterized in that: The anomaly correction module includes a time series model unit, a correction unit and a restoration unit; The time series model unit is used to construct a time series model for each audio signal feature, and train the model with the audio signal feature time series as input; the correction unit is used to input the frame number corresponding to the frame judged to be abnormal, and correct the abnormal value using the value predicted by the time series model; the restoration unit is used to restore the corrected audio signal to an audio frame, and restore the corrected video frame image to a video frame; The output end of the time series model unit is connected to the input end of the correction unit; the output end of the correction unit is connected to the input end of the restoration unit.

6. A method for managing audio and video data based on the Internet of Things, characterized in that: The method comprises the following steps: Step 1, obtain audio and video from the device or database and divide them into frames, and represent the audio frame signal and video frame image in the form of feature values; Step 2: Manually label audio signals and video frame images to form a data set and classify them, and build a random forest classifier to extract important features; Step 3: For the features extracted from the same classification data, abnormal audio signals and video frame images are judged according to the distribution of feature values; Step 4: Mark the audio signal and video frame images, build a time series model to correct and restore them.

7. The method for managing audio and video data based on the Internet of Things according to claim 6, characterized in that: In step 1, the audio is recorded by an audio recording device or obtained from an audio database; the audio is divided into frames to obtain N frames of audio signals; for each frame of the audio signal, the audio signal features are extracted, which are expressed as: [A1(i), A2(i),…, A n (i)]; For the video, it is recorded by a video recording device or obtained from a video database; the video is divided into frames to obtain M images respectively; for each video frame image, the video frame image features are extracted, which are expressed as: [V1(j), V2(j), …, V m (j)]; Where M and N are positive integers, representing the number of audio signals and video frame images respectively; i∈{1,2,…,N} represents an audio frame sequence; j∈{1,2,…,M} represents a video frame sequence; n is a positive integer representing the number of audio signal features, A n (i) represents the nth audio signal feature of the ith audio frame; m is a positive integer, representing the number of video frame image features, V m (j) represents the mth video frame image features of the jth video frame.

8. The method for managing audio and video data based on the Internet of Things according to claim 7, characterized in that: In step 2, each frame of audio signal is manually annotated, and the manual annotation type is expressed as: [K1, K2, …, K u ]; obtain the audio signal features of each frame and its corresponding manual annotation type and form a data set; Among them, u is a positive integer, indicating the number of types of audio signals, K u Indicates the u-th audio signal type; Classify the audio signals according to different types of manual annotations to obtain classification subsets with different annotation types; construct a random forest classifier and set the number of decision trees to E; for each classification subset, use the audio signal feature data in the subset as input features and the corresponding manual annotation type as the target output to train the random forest classifier: When training each decision tree, samples are randomly drawn from the dataset with replacement; For each node split, k audio signal features are randomly selected from all audio signal features; the audio signal feature that maximizes the purity of the node after splitting is selected as the split feature of the node; the node purity is measured by the Gini index; For an audio signal feature A r , calculate its importance in a decision tree: traverse each node in the tree and calculate the importance of r The Gini index of the node when used as a split feature is obtained by weighted summing the number of samples contained in the node according to the proportion of the total number of samples (Gb(A) r ); traverse each node in the tree and calculate the r When the node is used as the split feature, the Gini index of the child node after the node is split is weighted and summed according to the proportion of the number of samples contained in the child node to the total number of samples to obtain Ga(A r ); Feature A r The importance in this decision tree is represented by It(A r )=Gb(A r )-Ga(A r ); for feature A r , its importance in the entire random forest If(A r ) is obtained by averaging the importance of the feature in each decision tree: If(A r )=(1 / E)·Σ(It(A r )); Wherein, r is a positive integer, r∈{1,2,…,n}, representing the audio signal feature sequence; k is a positive integer, k<n, representing the number of selected audio signal features; Calculate the importance of each audio signal feature to each classification subset; for each classification subset, select the first α features with the largest importance value as the important audio signal features of the subset; The annotated video frame images are classified and important video frame image features are extracted according to the above method.

9. The method for managing audio and video data based on the Internet of Things according to claim 8, characterized in that: In step 3, the audio signal is classified according to different types of manual annotation to obtain a classification subset of different annotation types, and the important audio signal features selected are expressed as: [F1, F2, ..., F α ]; Among them, F1, F2, …, F α ∈{A1,A2,…,A n }; In this subset, obtain the distribution of different important audio signal features: Calculate the Euclidean distance between each important eigenvalue of the audio signal and all other important eigenvalues ​​of the audio signal to form a distance matrix; sort all the distance values ​​in the distance matrix and draw a sorting diagram of the distance values; according to the changing trend of the distance values ​​in the sorting diagram, find the point where the distance value changes from a gentle to a sharp increase with the largest difference, and select the corresponding distance value as an initial estimate of the neighborhood radius eps; The minimum number of points minPts is set to α+1; For each audio signal, calculate the number of data points contained in its eps neighborhood. When the number of data points is greater than or equal to minPts, mark the audio signal as a core point; otherwise, mark it as a boundary point or a noise point. Starting from the core point, clusters are formed by connecting density-reachable data points. Density-reachable means that there is a chain of data points, each point in the chain is in the eps neighborhood of its predecessor point, and the predecessor point is a core point. Using the DBSCAN algorithm, the data was divided into different clusters; for each identified dense cluster area, the centroid of each cluster was selected as the initial cluster center; Among them, the centroid represents the average value of all audio signal feature data vectors in the cluster; Set the number of clusters to 2, select two of the initial cluster centers, and use the K-means clustering algorithm to divide all audio signals of the classification subset into two clusters; After completing a clustering, for each cluster, recalculate the cluster center: use the average value of all points after the first clustering as the new two cluster centers; use the K-means clustering algorithm again to distribute different audio signals; set the maximum number of iterations T, and stop the iteration when the number of iterations reaches T; set the threshold G, calculate the distance from each audio signal to the centroid of the cluster to which it belongs, and when the distance is greater than G, the audio signal is considered abnormal; Perform abnormality judgment on the audio signal corresponding to each classification subset; According to the above method, the video frame image corresponding to each classification subset is judged to be abnormal.

10. The method for managing audio and video data based on the Internet of Things according to claim 9, characterized in that: In step 4, the audio signals judged as abnormal are marked, and for each audio signal feature, an audio signal database is constructed: [A v (1),A v (2),…,A v (N)]; Where v is a positive integer, v∈{1,2,…,n}, representing the audio signal feature sequence; A v (N) represents the vth audio signal feature of the Nth audio frame; Further construct the audio signal feature A v The time series model LSTM, with A v (1),A v (2),…,A v (N) is used as input to train the model and obtain the trained model; the number of frames corresponding to the audio signal judged as abnormal is input to obtain the audio signal feature A predicted by the model v as a correction for outliers; A time series model is constructed for each audio signal feature to correct the audio signal features judged to be abnormal; the corrected audio signal features are restored to audio frames; According to the above method, the image features of the video frame determined to be abnormal are corrected and restored to the video frame.

Images