News event recognition system based on audio and video analysis of model

Abstract

The invention provides a news event recognition system based on audio and video analysis of a model. The news event recognition system comprises an audio and video acquisition module, an audio feature extraction module, a video feature extraction module, a feature fusion module, an event recognition model and a result output module. The system collects news audio and video data, extracts audio and video features, performs fusion to form a comprehensive feature vector, adopts an SVM algorithm to construct a model for classification and recognition, and outputs a result. According to the invention, automation and accuracy of news event analysis can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video analysis, and more specifically, to a news event recognition system based on audio and video analysis of a model. Background Art

[0002] Existing news event recognition systems mainly rely on natural language processing and machine learning technologies, including deep learning, unsupervised clustering and other methods. These technologies are used to process raw news texts, perform syntactic analysis and semantic role recognition, extract key information, and convert it into vector representation. Subsequently, structured data is formed through dimensionality reduction and clustering combined with manual annotation to build a news event database.

[0003] A semi-supervised learning method for identifying and classifying news about emergencies is disclosed in a public document with a publication number of CN112989841B, comprising the following steps: capturing the content of a microblog website as an initial news data set, obtaining a labeled data set D1 through manual labeling; performing a pre-filtering operation on unlabeled samples, and obtaining a disturbed unlabeled data set using a data enhancement method; using the labeled data set D1 to supervise the learning of a BERT-based multi-classifier model, and minimizing the empirical risk function, using the BERT-based multi-classifier model to predict the pre-filtered data set and the disturbed unlabeled data set, and obtaining a BERT-based multi-classifier model trained by semi-supervised learning to achieve the recognition and classification tasks of news about emergencies. The present invention can overcome the problem of overfitting of deep learning classifiers in the case of insufficient labeled data, and improve the accuracy of identifying and classifying Chinese news about emergencies.

[0004] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: the prior art is mostly used to analyze text-based news, and with the development of intelligent technologies such as artificial intelligence and electronic storytelling, it is increasingly common for people to obtain news information in the form of audio and video. At the same time, although existing systems are capable of processing and analyzing news events, they still need to be improved in terms of automation and accuracy, especially in real-time processing of large amounts of news data and accurate identification of event types. Summary of the invention

[0005] The present invention provides a news event recognition system based on audio and video analysis of a model, comprising: The audio and video acquisition module is used to collect news audio and video data and convert them into a digital format for subsequent processing; The audio feature extraction module performs frame processing on the collected audio data and extracts audio features; The video feature extraction module performs image segmentation on the video data and extracts image features; The feature fusion module fuses the audio feature vector with the video feature vector to form a comprehensive feature vector; The event recognition model module uses the support vector machine (SVM) algorithm to build an SVM model for news event recognition. The comprehensive feature vector is used as the input of the SVM model to classify and recognize news events. The result output module outputs and displays the news event recognition results obtained by the event recognition model module.

[0006] Furthermore, the audio feature extraction module also includes an audio feature dimensionality reduction submodule, which uses a principal component analysis (PCA) algorithm to perform dimensionality reduction processing on the extracted audio features. The specific steps include: Calculate the covariance matrix of the audio feature vector

[0007] in For the audio feature vectors, is the mean vector of the audio feature vector, is the number of audio feature vectors; Solving for the covariance matrix The eigenvalue of and the corresponding eigenvector ; Sort the eigenvectors by eigenvalue and select the first The eigenvectors corresponding to the eigenvalues ​​constitute the projection matrix ,in is the feature dimension after dimensionality reduction, which is determined according to the set contribution rate threshold. The contribution rate calculation formula is:

[0008] in is the contribution rate threshold; The audio feature vector Through the projection matrix Perform projection dimensionality reduction to obtain the audio feature vector after dimensionality reduction .

[0009] Furthermore, the contribution rate threshold The value range is .

[0010] Furthermore, the video feature extraction module also includes a video feature selection submodule, which uses an information gain algorithm to perform feature selection on the extracted video image features, and the specific steps include: For each video image feature , calculate its information gain :

[0011] in is the category information entropy, and the calculation formula is:

[0012] in, is a collection of news event categories. For Category The probability of appearing in the training sample set; Features The value set of Features The value is The probability of For the characteristics The value is Under the condition of category information entropy, the calculation formula is:

[0013] in, For the characteristics The value is Under the conditions of category Probability of occurrence; Sort the video image features according to the information gain and select the features with information gain greater than the set threshold. The features are used as the final video features for news event recognition.

[0014] Furthermore, the information gain threshold The value range is .

[0015] Furthermore, the event recognition model module also includes a model update submodule, which uses an incremental learning algorithm to update the SVM model when there is new labeled news event sample data. The specific steps include: Merge the new training sample set with the original training sample set to form a new total training sample set; Using the parameters of the original SVM model as initial values, retrain the SVM model on the new total training sample set and update the model parameters; The performance of the updated SVM model is evaluated. If the performance meets the requirements, the update is completed; If the performance does not meet the requirements, adjust the relevant parameters of the incremental learning algorithm and re-update the model.

[0016] Furthermore, the relevant parameters of the incremental learning algorithm include a learning rate, and the value range of the learning rate is .

[0017] Furthermore, in the feature fusion module, the fusion method of the comprehensive feature vector is weighted fusion, and the formula is:

[0018] in is the comprehensive feature vector, is the audio feature vector, is the video feature vector, is the weight of the audio feature vector, is the weight of the video feature vector, and .

[0019] Furthermore, in the event recognition model module, the training process of the SVM model includes: Collect training sample sets with labeled news event categories; Construct an SVM classifier, and its optimization objective function is:

[0020] in is the normal vector of the hyperplane, is the intercept of the hyperplane, is the slack variable, is the penalty parameter, is the number of training samples; The constraints are: , in For the The class labels of the training samples, To input the feature vector Mapping function to high-dimensional feature space; The radial basis kernel function (RBF) is used to map the low-dimensional comprehensive feature vector to a high-dimensional space. The RBF formula is:

[0021] in is the kernel function parameter, Representation sample and The Euclidean distance between By solving the above optimization problem, we can get the parameters of the SVM model: and .

[0022] Furthermore, in the audio feature extraction module, the calculation process of the Mel-frequency cepstral coefficient MFCC includes: The audio signal is pre-emphasized, and the formula is:

[0023] in is the first sampling points, is the pre-emphasized audio signal, is the pre-emphasis coefficient; The pre-emphasized audio signal is windowed using a Hamming window, and the window function formula is:

[0024] in is the sampling point number in the window, is the window length; Perform fast Fourier transform (FFT) on the windowed audio signal to obtain the spectrum of the audio signal ; Compute the Mel frequency filter bank, The frequency response formula of the filter is: if ,but ; if ,but ; if ,but ; if ,but ; in, It is The center frequency of the Mel frequency filter is given by the formula Calculated, , is the number of filter banks; The frequency spectrum of the audio signal Through the Mel frequency filter bank, the Mel spectrum is obtained ; Take the logarithm of the Mel spectrum to get the logarithmic Mel spectrum ; Finally, the logarithmic Mel spectrum is subjected to discrete cosine transform (DCT) to obtain the MFCC coefficients. , , is the number of MFCC coefficients.

[0025] The above-mentioned embodiments of the present invention have at least the following beneficial effects: the audio and video analysis system of the present invention can effectively extract key information from news audio and video by integrating audio and video feature extraction modules, and synthesize this information into a feature vector through a feature fusion module. This comprehensive feature vector can provide more comprehensive data support for the event recognition model, thereby improving the accuracy and efficiency of news event recognition. In addition, the system uses an event recognition model constructed by a support vector machine (SVM) algorithm to accurately classify news events, making the recognition results more reliable.

[0026] The system also includes a model update submodule, so that when there is new labeled news event sample data, the SVM model can be updated using an incremental learning algorithm. This incremental learning mechanism can maintain the continuous optimization and adaptability of the model without retraining from scratch, which can save computing resources and time. At the same time, the principal component analysis (PCA) algorithm is used to reduce the dimensionality of the audio features, which can reduce the dimension of the feature space and improve computing efficiency while retaining the most important information, which can further enhance the system's recognition ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which: Figure 1 A schematic diagram of the structure of a news event recognition system for audio and video analysis based on a model provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0029] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0030] It should be noted that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0031] Reference below Figure 1 , Figure 1 The structure diagram of the news event recognition system based on audio and video analysis provided by one embodiment of the present invention is as follows. Figure 1 As shown, a news event recognition system 100 based on audio and video analysis of a model includes: The audio and video acquisition module 101 is used to collect news audio and video data and convert them into a digital format for subsequent processing; The audio feature extraction module 102 performs frame processing on the collected audio data and extracts audio features; The video feature extraction module 103 performs image segmentation on the video data and extracts image features; A feature fusion module 104, which fuses the audio feature vector with the video feature vector to form a comprehensive feature vector; The event recognition model module 105 uses a support vector machine (SVM) algorithm to build an SVM model for news event recognition, uses the comprehensive feature vector as the input of the SVM model, and classifies and recognizes news events; The result output module 106 outputs and displays the news event recognition results obtained by the event recognition model module.

[0032] It should be noted that the system includes an audio and video acquisition module, whose function is to collect news audio and video data and convert it into a digital format for subsequent processing. Here, the audio and video acquisition module refers to a combination of hardware and software that can capture audio and video signals in analog or digital form and convert them into digital data that can be processed by a computer. The digital format refers to the binary representation of the analog signal converted through the analog-to-digital conversion (ADC) process.

[0033] Specifically, the audio and video acquisition module may include a high-definition camera and microphone, as well as a corresponding audio / video interface card, for capturing high-quality audio and video signals. The collected data is compressed and encoded through a specific codec and converted into a digital format suitable for storage and processing, such as MP3, MP4, etc. Parameter settings may include sampling rate, bit rate, and compression level, which can be adjusted according to the specific needs of news collection.

[0034] Preferably, the audio and video acquisition module can be equipped with automatic gain control and noise suppression functions to improve the acquisition quality. In addition, the system can also be configured to support the acquisition of multiple audio and video formats to adapt to different news collection environments and equipment. For example, the system can be configured to prioritize the acquisition of high-definition video and stereo audio to ensure the quality and compatibility of the data.

[0035] In some embodiments, the audio feature extraction module further includes an audio feature dimensionality reduction submodule, which uses a principal component analysis (PCA) algorithm to perform dimensionality reduction processing on the extracted audio features. The specific steps include: Calculate the covariance matrix of the audio feature vector

[0036] in For the audio feature vectors, is the mean vector of the audio feature vector, is the number of audio feature vectors; Solving for the covariance matrix The eigenvalue of and the corresponding eigenvector ; Sort the eigenvectors by eigenvalue and select the first The eigenvectors corresponding to the eigenvalues ​​constitute the projection matrix ,in is the feature dimension after dimensionality reduction, which is determined according to the set contribution rate threshold. The contribution rate calculation formula is:

[0037] in is the contribution rate threshold; The audio feature vector Through the projection matrix Perform projection dimensionality reduction to obtain the audio feature vector after dimensionality reduction .

[0038] It should be noted that the audio feature extraction module contains an audio feature dimensionality reduction submodule, which uses the principal component analysis (PCA) algorithm to reduce the dimensionality of the extracted audio features. Here, principal component analysis (PCA) is a statistical method that transforms a set of possibly related variables into a set of linearly unrelated variables, called principal components, through orthogonal transformation.

[0039] Specifically, the steps of the PCA algorithm include calculating the covariance matrix of the audio feature vector, solving the eigenvalues ​​and corresponding eigenvectors of the covariance matrix, sorting the eigenvectors according to the eigenvalues, and selecting the eigenvectors corresponding to the first few eigenvalues ​​to form a projection matrix. Among them, the number of selected eigenvectors is determined according to the set contribution rate threshold, and the contribution rate calculation formula is the ratio of the sum of the eigenvalues ​​to the sum of the total eigenvalues, and this ratio needs to be greater than or equal to the preset contribution rate threshold.

[0040] Preferably, the contribution rate threshold is in the range of [90%, 99%], which means that the feature vector after dimensionality reduction can retain at least 90% of the original data information. In this way, the system can reduce the computational complexity while retaining the most important audio feature information, thereby improving the efficiency and effect of subsequent processing.

[0041] In some embodiments, the contribution rate threshold The value range is .

[0042] It should be noted that the contribution rate threshold is set in the range of [90%, 99%], which means that in the process of audio feature dimensionality reduction, only when the selected feature vector can explain at least 90% to 99% of the information of the original data will it be retained. The contribution rate threshold is a measure used to determine how much information is sufficient to retain during the dimensionality reduction process to ensure that the performance of the model is not affected by information loss. This threshold ensures that the feature vector after dimensionality reduction can retain as much important information of the original data as possible.

[0043] Specifically, the contribution rate threshold is determined by calculating the cumulative contribution rate of all eigenvalues. In practice, this involves performing eigenvalue decomposition on the covariance matrix of the audio feature vector, then sorting the eigenvalues ​​according to their size, and accumulating these eigenvalues ​​until the cumulative contribution rate reaches or exceeds the set threshold. For example, if an audio feature vector has 10 eigenvalues, their cumulative contribution rates are: 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4,0.45, 0.5, 0.55. Then, when the cumulative contribution rate reaches 90%, it may be necessary to retain the eigenvectors corresponding to the first 5 eigenvalues, because the cumulative contribution rate of the first 5 eigenvalues ​​is 0.3, and reaching 90% may require retaining more eigenvectors.

[0044] Preferably, this threshold can be adjusted according to the specific application scenario and model performance requirements. In some scenarios that require higher computational efficiency, a lower contribution rate threshold, such as 90%, may be selected to reduce the dimension of the feature. In scenarios that require higher recognition accuracy, a higher threshold, such as 95% or 99%, may be selected to retain more information. In addition, the optimal contribution rate threshold can also be determined by methods such as cross-validation to balance the performance and computational efficiency of the model.

[0045] In some embodiments, the video feature extraction module further includes a video feature selection submodule, which uses an information gain algorithm to perform feature selection on the extracted video image features. The specific steps include: For each video image feature , calculate its information gain :

[0046] in is the category information entropy, and the calculation formula is:

[0047] in, is a collection of news event categories. For Category The probability of appearing in the training sample set; Features The value set of Features The value is The probability of For the characteristics The value is Under the condition of category information entropy, the calculation formula is:

[0048] in, For the characteristics The value is Under the conditions of category Probability of occurrence; Sort the video image features according to the information gain and select the features with information gain greater than the set threshold. The features are used as the final video features for news event recognition.

[0049] It should be noted that the video feature extraction module contains a video feature selection submodule, which uses the information gain algorithm to select the extracted video image features. Here, information gain is an indicator to measure the contribution of a feature to the classification problem, which describes the degree to which the feature reduces the uncertainty of the sample classification result. Specifically, the greater the information gain, the greater the contribution of the feature to the classification result.

[0050] Specifically, the steps of the information gain algorithm include calculating the information gain value of each video image feature and sorting the features according to these values. The calculation of information gain involves the concepts of category information entropy and conditional entropy. Category information entropy measures the uncertainty of the category, while conditional entropy measures the uncertainty of the category under the condition of a known feature value. Information gain is the difference between category information entropy and conditional entropy. During the feature selection process, the system will select features with information gain greater than the set threshold as the final video features for news event recognition.

[0051] Preferably, the value range of the information gain threshold is [0.05, 0.2]. This means that only when the information gain value of a feature exceeds 0.05 but does not exceed 0.2, the feature will be selected as a valid video feature. This threshold range can be adjusted according to the requirements for feature selection accuracy in practical applications. For example, when selecting video features, if it is found that the information gain values ​​of most features are concentrated in a small interval, the threshold range can be appropriately adjusted to more accurately select the features that are most helpful for news event recognition.

[0052] In some embodiments, the information gain threshold The value range is .

[0053] It should be noted that the value range of the information gain threshold is set to [0.05, 0.2]. This is a parameter used to screen out video features that have significant contributions to news event recognition during the video feature selection process. The information gain threshold means that during the feature selection process, only when the information gain value of a feature exceeds this threshold will the feature be retained for subsequent event recognition. The setting of this threshold helps to exclude features that contribute less to the classification results, thereby improving the recognition efficiency and accuracy of the model.

[0054] Specifically, the information gain threshold is a parameter used to measure the contribution of a feature to the classification result. In actual operation, the system calculates the information gain value of each video image feature and then compares it with this threshold. If the information gain value of a feature is greater than 0.05 but not more than 0.2, then the feature is considered to have a significant impact on the classification and will be selected as part of the final feature set. The setting of this threshold needs to comprehensively consider the accuracy of feature selection and computational efficiency to ensure that the final feature set can effectively represent the original data, but is not too large to cause low computational efficiency.

[0055] Preferably, the specific value of the information gain threshold can be adjusted according to the actual application scenario and data characteristics. For example, in some scenarios that require higher accuracy in feature selection, a smaller threshold, such as 0.03 or 0.04, may be selected to retain more features; while in scenarios that require higher computational efficiency, a larger threshold, such as 0.25 or 0.3, may be selected to reduce the number of features. In addition, the optimal information gain threshold can be determined by methods such as cross-validation to balance the performance and computational efficiency of the model.

[0056] In some embodiments, the event recognition model module further includes a model updating submodule. When there is new labeled news event sample data, the SVM model is updated using an incremental learning algorithm. The specific steps include: Merge the new training sample set with the original training sample set to form a new total training sample set; Using the parameters of the original SVM model as initial values, retrain the SVM model on the new total training sample set and update the model parameters; The performance of the updated SVM model is evaluated. If the performance meets the requirements, the update is completed; If the performance does not meet the requirements, adjust the relevant parameters of the incremental learning algorithm and re-update the model.

[0057] It should be noted that the event recognition model module contains a model update submodule, which is used to update the SVM model using an incremental learning algorithm when new labeled news event sample data is available. The incremental learning algorithm is a method that allows the learning algorithm to adjust and optimize the original model instead of starting from scratch when new data arrives.

[0058] Specifically, the update steps of the incremental learning algorithm include merging the new training sample set with the original training sample set to form a new total training sample set; using the parameters of the original SVM model as the initial values, retraining the SVM model on the new total training sample set and updating the model parameters; evaluating the performance of the updated SVM model. If the performance meets the requirements, the update is completed; if the performance does not meet the requirements, adjust the relevant parameters of the incremental learning algorithm and re-update the model. In this process, the relevant parameters include the learning rate, and the value range of the learning rate is , the learning rate determines how fast the model learns and how much it adjusts on new data.

[0059] Preferably, the relevant parameters of the incremental learning algorithm, such as the learning rate, can be dynamically adjusted according to the performance of the model on the new data. For example, if the model performs poorly on the new data, the learning rate can be appropriately increased to speed up the adaptation of the model; conversely, if the model has adapted well to the new data, the learning rate can be appropriately reduced to avoid performance fluctuations caused by over-adjustment. In addition, other parameters, such as regularization parameters, can also be considered to control the complexity of the model and prevent overfitting.

[0060] In some embodiments, the parameters of the incremental learning algorithm include a learning rate, and the range of the learning rate is .

[0061] It should be noted that the relevant parameters of the incremental learning algorithm include the learning rate, and the value range of the learning rate is Here, the learning rate is an important parameter in incremental learning, which determines the magnitude of the model's weight update when faced with new data. A learning rate that is too high may cause the model to be unstable during training, while a learning rate that is too low may cause the model to update too slowly and have difficulty adapting to new data.

[0062] Specifically, the learning rate setting needs to be adjusted according to the actual data characteristics and model performance. In actual operation, you can select an initial value from the preset learning rate range, such as 0.05, and then gradually adjust it according to the performance of the model on the validation set. If the model performance is not good enough, you can increase the learning rate appropriately to speed up the learning speed; if the model forgets old knowledge too quickly, you can reduce the learning rate appropriately to maintain the stability of the model.

[0063] Preferably, in addition to the learning rate, other parameters can also be considered for adjustment, such as the decay rate, which can help control the decrease of the learning rate over time, thereby allowing faster learning in the early stages of training, and gradually slowing down the learning speed in the later stages of training to refine the model weights. In addition, a momentum term can be introduced, which can help the model maintain a certain directionality when updating weights and avoid oscillations on the optimization path. The introduction and adjustment of these parameters can further improve the performance and adaptability of the incremental learning algorithm.

[0064] In some embodiments, in the feature fusion module, the fusion method of the comprehensive feature vector is weighted fusion, and the formula is:

[0065] in is the comprehensive feature vector, is the audio feature vector, is the video feature vector, is the weight of the audio feature vector, is the weight of the video feature vector, and .

[0066] It should be noted that the fusion method of the comprehensive feature vector in the feature fusion module is weighted fusion, and the formula is: ,in is the audio feature vector, is the video feature vector, is the weight of the audio feature vector, is the weight of the video feature vector, and Here, weighted fusion is a feature combination method that assigns different weights to different features to reflect their importance in the final decision.

[0067] Specifically, the weights in the weighted fusion process and The weights can be set based on the contribution of audio and video features to news event recognition. These weights can be determined through experimental adjustment, based on prior knowledge, or using some optimization algorithm (such as gradient descent). The choice of weights will affect the performance of the model, so they need to be carefully adjusted to ensure that the model can effectively learn from audio and video data.

[0068] Preferably, the specific implementation of weighted fusion may include the use of cross-validation to determine the best weight configuration. In addition, other feature fusion methods may also be explored, such as simple feature concatenation or more complex machine learning algorithms, such as decision-level fusion, where the model first makes independent predictions for audio and video features and then combines these predictions to make a final decision. These alternatives can be selected and adjusted based on the actual application scenario and system performance requirements.

[0069] In some embodiments, in the event recognition model module, the training process of the SVM model includes: Collect training sample sets with labeled news event categories; Construct an SVM classifier, and its optimization objective function is:

[0070] in is the normal vector of the hyperplane, is the intercept of the hyperplane, is the slack variable, is the penalty parameter, is the number of training samples; The constraints are: , ,in For the The class labels of the training samples, To input the feature vector Mapping function to high-dimensional feature space; The radial basis kernel function (RBF) is used to map the low-dimensional comprehensive feature vector to a high-dimensional space. The RBF formula is: in is the kernel function parameter, Representation sample and The Euclidean distance between By solving the above optimization problem, we can get the parameters of the SVM model: and .

[0071] It should be noted that the training process of the SVM model includes collecting training sample sets with labeled news event categories and constructing an SVM classifier, whose optimization objective function is:

[0072] in is the normal vector of the hyperplane, is the intercept of the hyperplane, is the slack variable, is the penalty parameter, is the number of training samples; the constraints are: , ,in For the The class labels of the training samples, To input the feature vector Mapping function to high-dimensional feature space.

[0073] Specifically, during the training process of the SVM model, the optimization objective function and determines the location and direction of the decision boundary, and Used to handle situations that do not completely conform to the hard margin support vectors. Controls the model's penalty for misclassification. The constraint ensures that the model can correctly classify the training samples, while at the same time Some samples are allowed to violate this rule to obtain better generalization ability. Mapping data in a low-dimensional feature space to a high-dimensional space makes it easier to find a hyperplane that separates different categories in the high-dimensional space.

[0074] Preferably, the training process of the SVM model can use a radial basis kernel function (RBF) as a kernel function, and its formula is: ,in is the kernel function parameter, Representation sample and By choosing the appropriate The RBF kernel function can effectively handle nonlinear problems. In addition, other kernel functions, such as the polynomial kernel or the Sigmoid kernel, can also be considered to adapt to different types of data distributions and problem characteristics.

[0075] In some embodiments, in the audio feature extraction module, the calculation process of the Mel-frequency cepstral coefficient MFCC includes: The audio signal is pre-emphasized, and the formula is:

[0076] in is the first sampling points, is the pre-emphasized audio signal, is the pre-emphasis coefficient; The pre-emphasized audio signal is windowed using a Hamming window, and the window function formula is:

[0077] in is the sampling point number in the window, is the window length; Perform fast Fourier transform (FFT) on the windowed audio signal to obtain the spectrum of the audio signal ; Compute the Mel frequency filter bank, The frequency response formula of the filter is: if ,but ; if ,but ; if ,but ; if ,but ; in, It is The center frequency of the Mel frequency filter is given by the formula Calculated, , is the number of filter banks; The frequency spectrum of the audio signal Through the Mel frequency filter bank, the Mel spectrum is obtained ; Take the logarithm of the Mel spectrum to get the logarithmic Mel spectrum ; Finally, the logarithmic Mel spectrum is subjected to discrete cosine transform (DCT) to obtain the MFCC coefficients. , , is the number of MFCC coefficients.

[0078] It should be noted that the calculation process of the Mel Frequency Cepstral Coefficient (MFCC) in the audio feature extraction module includes pre-emphasis processing of the audio signal, and the formula is:

[0079] in is the first sampling points, is the pre-emphasized audio signal, Here, pre-emphasis processing is a technology that enhances the high-frequency part of the signal, which is usually used in speech signal processing to highlight the characteristics of the high-frequency part, because these parts usually contain important speech information.

[0080] Specifically, the pre-emphasis coefficient Usually set between 0.9 and 1, this value can be adjusted according to the specific audio characteristics and processing requirements. After pre-emphasis processing, the audio signal is windowed, usually using a Hamming window, and the window function formula is:

[0081] in is the sampling point number in the window, The purpose of windowing is to reduce the discontinuity caused by the edge of the signal, making the signal smoother and thus improving the accuracy of the subsequent Fourier transform.

[0082] Preferably, in addition to pre-emphasis and windowing, the calculation of the Mel frequency cepstral coefficient (MFCC) also includes fast Fourier transform (FFT) to obtain the spectrum of the audio signal, calculate the response of the Mel frequency filter bank, and take the logarithm of the Mel spectrum and perform discrete cosine transform (DCT). These steps work together to convert the audio signal into a series of coefficients reflecting its spectral envelope, which can effectively represent the characteristics of the speech signal. In practical applications, the number of MFCC coefficients and the parameters of the Mel filter bank can be adjusted as needed to adapt to different speech recognition tasks.

[0083] The above-mentioned embodiments of the present invention have the following beneficial effects: the system described in the present invention can improve the accuracy and efficiency of news event recognition by integrating audio and video feature extraction, feature fusion and application of support vector machine (SVM) algorithm. The audio and video acquisition module can convert news audio and video data into digital format, providing a basis for subsequent processing; the audio feature extraction module can reduce the dimension through the principal component analysis (PCA) algorithm, which can reduce the computational complexity while retaining key information; the video feature extraction module can enhance the feature recognition ability of the model by selecting the most useful features through the information gain algorithm. The feature fusion module adopts a weighted fusion method, which can balance the contribution of audio and video features and further improve the recognition effect.

[0084] In addition, the event recognition model module uses the SVM algorithm combined with the radial basis kernel function (RBF) for feature space mapping, which can handle nonlinear problems and improve classification performance. The model update submodule supports incremental learning, allowing the system to adapt to new data without retraining, maintaining the timeliness and accuracy of the model. Through the application of these technologies, the system can not only achieve rapid and accurate recognition of news events, but also adapt to the ever-changing data environment and continuously optimize the recognition results.

[0085] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0086] The above descriptions are only some preferred embodiments of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A news event recognition system based on audio and video analysis of a model, characterized in that: Includes the following modules: The audio and video acquisition module is used to collect news audio and video data and convert them into a digital format for subsequent processing; The audio feature extraction module performs frame processing on the collected audio data and extracts audio features; The video feature extraction module performs image segmentation on the video data and extracts image features; The feature fusion module fuses the audio feature vector with the video feature vector to form a comprehensive feature vector; The event recognition model module uses the support vector machine (SVM) algorithm to build an SVM model for news event recognition. The comprehensive feature vector is used as the input of the SVM model to classify and recognize news events. The result output module outputs and displays the news event recognition results obtained by the event recognition model module.

2. The system according to claim 1, characterized in that The audio feature extraction module also includes an audio feature dimensionality reduction submodule, which uses a principal component analysis (PCA) algorithm to perform dimensionality reduction processing on the extracted audio features. The specific steps include: Calculate the covariance matrix of the audio feature vector using the formula: in For the audio feature vectors, is the mean vector of the audio feature vector, is the number of audio feature vectors; Solving for the covariance matrix The eigenvalue of and the corresponding eigenvector ; Sort the eigenvectors by eigenvalue and select the first The eigenvectors corresponding to the eigenvalues ​​constitute the projection matrix ,in is the feature dimension after dimensionality reduction, which is determined according to the set contribution rate threshold. The contribution rate calculation formula is: in, is the contribution rate threshold; The audio feature vector Through the projection matrix Perform projection dimensionality reduction to obtain the audio feature vector after dimensionality reduction .

3. The system according to claim 2, characterized in that The contribution rate threshold The value range is .

4. The system according to claim 1, characterized in that The video feature extraction module also includes a video feature selection submodule, which uses an information gain algorithm to perform feature selection on the extracted video image features. The specific steps include: For each video image feature , calculate its information gain : in is the category information entropy, and the calculation formula is: in, is a collection of news event categories. For Category The probability of appearing in the training sample set; Features The value set of Features The value is The probability of For the characteristics The value is Under the condition of category information entropy, the calculation formula is: in, For the characteristics The value is Under the conditions of category Probability of occurrence; Sort the video image features according to the information gain and select the features with information gain greater than the set threshold. The features are used as the final video features for news event recognition.

5. The system according to claim 4, characterized in that The information gain threshold The value range is .

6. The system according to claim 1, characterized in that The event recognition model module also includes a model update submodule. When there is new labeled news event sample data, the SVM model is updated using an incremental learning algorithm. The specific steps include: Merge the new training sample set with the original training sample set to form a new total training sample set; Using the parameters of the original SVM model as initial values, retrain the SVM model on the new total training sample set and update the model parameters; The performance of the updated SVM model is evaluated. If the performance meets the requirements, the update is completed; If the performance does not meet the requirements, adjust the relevant parameters of the incremental learning algorithm and re-update the model.

7. The system according to claim 6, characterized in that The relevant parameters of the incremental learning algorithm include the learning rate, and the value range of the learning rate is .

8. The system according to claim 1, characterized in that In the feature fusion module, the fusion method of the comprehensive feature vector is weighted fusion, and the formula is: in, is the comprehensive feature vector, is the audio feature vector, is the video feature vector, is the weight of the audio feature vector, is the video feature vector, and .

9. The system according to claim 1, characterized in that In the event recognition model module, the training process of the SVM model includes: Collect training sample sets with labeled news event categories; Construct an SVM classifier, and its optimization objective function is: in is the normal vector of the hyperplane, is the intercept of the hyperplane, is the slack variable, is the penalty parameter, is the number of training samples; The constraints are: , ,in For the The class labels of the training samples, To input the feature vector Mapping function mapped to high-dimensional feature space; radial basis kernel function RBF is used to map the low-dimensional comprehensive feature vector to high-dimensional space. The RBF formula is: in is the kernel function parameter, Representation sample and The Euclidean distance between By solving the above optimization problem, we can get the parameters of the SVM model: and .

10. The system according to claim 1, characterized in that In the audio feature extraction module, the calculation process of the Mel frequency cepstral coefficient MFCC includes: The audio signal is pre-emphasized, and the formula is: in, is the first sampling points, is the pre-emphasized audio signal, is the pre-emphasis coefficient; the pre-emphasized audio signal is windowed using a Hamming window, and the window function formula is: in is the sampling point number in the window, is the window length; perform fast Fourier transform (FFT) on the windowed audio signal to obtain the spectrum of the audio signal ; Compute the Mel frequency filter bank, The frequency response formula of the filter is: if ,but ; if ,but ; if ,but ; if ,but ; in, It is The center frequency of the Mel frequency filter is given by the formula Calculated, , is the number of filter banks; The frequency spectrum of the audio signal Through the Mel frequency filter bank, we get the Mel spectrum ; Take the logarithm of the Mel spectrum to get the logarithmic Mel spectrum ; Finally, the logarithmic Mel spectrum is subjected to discrete cosine transform DCT to obtain the MFCC coefficients , , is the number of MFCC coefficients.

Images