A method and system for identifying specific audio information

By enhancing the speech and image information in public data and multi-dimensional analysis, the problem of high accuracy and cost of bad behavior recognition systems in the prior art is solved, and efficient and accurate bad content recognition is achieved.

CN118658456BActive Publication Date: 2024-11-22INSTITUTE OF NETWORK TECHNOLOGY (YANTAI) +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411146603.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-22
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

The accuracy of existing bad behavior or bad speech recognition systems is limited by the dependence of speech recognition technology, which is prone to misjudgment and high cost, and cannot fully understand human emotions and context, which limits its application effect.

Method used

By obtaining voice and image information in public data, pre-emphasis, windowing and endpoint detection are enhanced, combined with voice and image recognition models, analyzing whether there are explicit or implicit malicious behaviors, and improving recognition accuracy and efficiency.

Benefits of technology

It improves the accuracy and efficiency of audio recognition, and can quickly identify bad content, such as violence, gambling, and malicious speech. Combined with multi-dimensional judgment, it improves the accuracy of the recognition and the reliability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118658456B_ABST
    Figure CN118658456B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for identifying audio specific information, which belongs to the field of information identification and solves the problem of identifying bad behavior or bad language. The specific method is as follows: step S1: acquiring image information and voice information; step S2: respectively enhancing the image information and voice information to obtain the image to be analyzed and the voice to be analyzed; step S3: constructing a behavior recognition model and a voice recognition model, and analyzing the image to be analyzed and the voice to be analyzed to determine whether there is malicious behavior in public data; if so, saving the public data in which explicit malicious behavior or implicit malicious behavior occurs; if not, continuing monitoring; the present invention obtains, analyzes and processes the public data of a target area, identifies bad behavior or bad language in the public data, and reduces the difficulty of managing public areas.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention discloses a method and a system for identifying specific audio information, and relates to the field of information identification. Background Art

[0002] The existing methods or systems for identifying bad behavior or bad speech have the following deficiencies:

[0003] Limited accuracy: Although speech recognition technology is constantly improving, its accuracy is still affected by many factors, such as the quality of the sound sample, emotions, background noise, and changes in sound over time. In the existing recognition system for bad behavior or bad speech, if the speech recognition module cannot accurately recognize the speech content, it may lead to false positives or false negatives, which will not only reduce the reliability of the system but also bring unnecessary troubles to managers.

[0004] Risk of misjudgment: Existing recognition systems for bad behavior or bad speech rely heavily on the accuracy and reliability of speech recognition technology. If speech recognition technology fails or its performance degrades, the entire system may not work properly or false alarms or missed alarms may occur. In addition, speech recognition technology cannot fully understand human emotions and context, and may make misjudgments when processing certain complex or ambiguous speech content. This risk of misjudgment limits the application effect of speech recognition technology in bad behavior or bad speech recognition systems to a certain extent.

[0005] High technical cost: High-quality speech recognition technology requires a large amount of data and computing power support, so the development and use of speech recognition technology requires a high cost investment; in the existing recognition system for bad behavior or bad speech, in order to ensure the accuracy and reliability of the system, it is often necessary to use advanced speech recognition algorithms and hardware equipment; the procurement and maintenance costs of these devices are high and the technical threshold is relatively high, which hinders the application and implementation of the technology. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention aims to provide a method and system for identifying audio specific information, aiming to solve the problem of identifying bad behavior or language.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solution: A method for identifying audio specific information includes:

[0008] Step S1: acquiring public data of a target area; extracting the voice part of the public data to obtain voice information; extracting the image part of the public data to obtain image information;

[0009] Step S2: performing pre-emphasis, windowing and endpoint detection enhancement processing on the voice information to obtain the voice to be analyzed; performing enhancement processing on the image information to obtain the image to be analyzed;

[0010] Step S3: Acquire specific image information and specific voice information as an image test set and a voice test set; use the voice test set and the image test set to build a voice recognition model and a behavior recognition model; use the voice recognition model and the behavior recognition model to analyze the voice to be analyzed and the image to be analyzed, and determine whether there are explicit malicious behaviors and implicit malicious behaviors in the public data; if so, save the public data where the explicit malicious behaviors or implicit malicious behaviors appear; if not, continue monitoring.

[0011] Furthermore, the specific steps of step S1 are:

[0012] Step S11: Acquire public data of the target area;

[0013] Step S12: extracting the voice part in the public data to obtain voice information;

[0014] Step S121: import the moviepy library; use the absolute path of the public data as a parameter of the cv2.VideoCapture() method, use the VideoFileClip() method to read the video file in the public data, and receive it with the variable video;

[0015] Step S122: extract the audio attribute of the variable video, obtain the voice in the video file, and receive it with the variable adio, that is, adio = audio.audio;

[0016] Step S123: Use write_audiofile() method to save the audio file, i.e., adio.write_audiofile(...);

[0017] The ... in adios.write_audiofile(...) indicates the parameters of write_audiofile(), which is used to set the saving format of the audio file.

[0018] Step S124: Summarize the audio files saved in step S133 to obtain voice information; use the close() method to release the variables video and adio, that is:

[0019] video.close();

[0020] adio.close();

[0021] Step S13: extracting the image part from the public data to obtain image information;

[0022] Import OpenCV library; use cv2.VideoCapture() method and read() method to process the image part in the public data and obtain image information;

[0023] Step S14: Summarize the image information and voice information and proceed to step S2.

[0024] Furthermore, the specific steps of step S2 are as follows:

[0025] Step S21: pre-emphasize the voice information to obtain sound information V1;

[0026] Step S211: Select the characteristic (1-aZ -1 ) as a pre-emphasis filter; where a represents the pre-emphasis coefficient; Z represents the sine wave amplitude;

[0027] Step S212: using a pre-emphasis filter to process the voice information, remove low-frequency noise in the voice information, and obtain sound information V1;

[0028] Step S213: Suppose the frequency function of the voice information before pre-emphasis is S(n); wherein n represents the timestamp of the voice information, and S(n) represents the frequency corresponding to the nth timestamp of the voice information; Suppose the frequency function of the sound information V1 is ;

[0029] S(n) and The relationship is as follows:

[0030] ;

[0031] Step S213: performing windowing processing on the voice information V1 to obtain sound information V2;

[0032] Step S214: performing endpoint detection processing on the voice information V2 to obtain the voice to be analyzed;

[0033] Step S22: Enhance the image information to obtain an image set to be analyzed, and proceed to step S3.

[0034] Furthermore, the specific steps of step S213 are as follows:

[0035] Step S2131: Use the MATLAB Signal Processing Toolbox library function to extract the main lobe and side lobe of the sound information V1; the main lobe represents the largest radiation beam on the sound wave radiation pattern, which is the area where the sound wave energy is most concentrated; the side lobe is the small radiation beam on both sides of the main lobe in the sound wave radiation pattern;

[0036] Step S2132: Set the sampling rate of the sound information V1 to 8KHz; With 256 timestamps as one frame, the sampling points of frame shift are 64 timestamps;

[0037] For the lth frame of sound information x in the sound information V1 l , the relationship between the phase of the sound information of the lth frame after the frame shift and the original phase is as follows:

[0038] x l (i) = x[(l-1)*m+i];

[0039] Wherein, i is a natural number;

[0040] x l (i) represents the phase of the i-th timestamp of the l-th frame after the frame shift, and x[(l-1)*m+i] represents the original phase of the [(l-1)*m+i]-th timestamp, that is, when x in the sound information V1 is l After frame shift occurs, x l The phase of the i-th timestamp in becomes the corresponding phase of the [(l-1)*m+i]-th timestamp in the original phase;

[0041] F represents the frame length, F=256; M represents the sampling period, M=64;

[0042] Step S2133: Multiply it with the window function w(n) to get the sound information V2;

[0043] The function expression of the sound information V2 is denoted as Sw;

[0044] Sw= *w(n);

[0045] Window function w(n) selection: rectangular window and Hamming window;

[0046] When the window function is a rectangular window, the expression of w(n) is:

[0047] ;

[0048] When the window function is a Hamming window, the expression of w(n) is:

[0049] ;

[0050] Step S2134: setting the use scenarios of the rectangular window and the Hamming window;

[0051] Scenario 1: When the sampling point is in the main lobe, the window function uses a rectangular window; when the sampling point is in the side lobe, the window function uses a Hamming window;

[0052] Scenario 2: When the sampling point does not belong to Scenario 1, a rectangular window is used uniformly.

[0053] Furthermore, the specific steps of step S214 are as follows:

[0054] Step S2141: using MATLAB library functions to distinguish between unvoiced and voiced sounds in the sound information V2; wherein, unvoiced sounds mean: in the sound information V2, the time domain waveform spectrum has a small amplitude, low energy, and no periodic sound segment; voiced sounds mean: in the sound information V2, the time domain waveform has a large amplitude, high energy, and a quasi-periodic sound segment;

[0055] Step S2142: Use the short-time average energy to detect the unvoiced sound. The calculation formula of the short-time average energy E(l) is as follows:

[0056] ; Use the short-time zero-crossing rate to detect voiced sounds. The calculation formula of the short-time zero-crossing rate ZCR(l) is as follows:

[0057] ;

[0058] Among them, F, l, i and x in the E(l) calculation formula and ZCR(l) calculation formula are l (i) corresponds to the meaning of F, l, i and x in step S212 l (i) has the same meaning as x; l (i+1) represents the phase of the (i+1)th timestamp of the lth frame;

[0059] Step S243: Summarize the speech obtained in step S2142 as the speech to be analyzed.

[0060] Furthermore, the specific steps of step S3 are as follows:

[0061] Step S31: obtaining specific image information and specific voice information as an image test set and a voice test set;

[0062] Step S32: constructing a speech recognition model according to the speech test set;

[0063] Step S321: import the SpeechRecognition library; use the Recognizer class in the SpeechRecognition library to perform speech recognition on the speech test set and the speech to be analyzed, and obtain the detection segment set and the to-be-detected segment set;

[0064] Step S322: Using the HMM model as the basic model of the speech recognition model;

[0065] Define a regular expression in the fully connected layer of the HMM model:

[0066] print(normalized_phrase1 == normalized_phrase2); where normalized_phrase1 represents a word segment in the detection word segment set, and normalized_phrase2 represents a word segment in the to-be-detected word segment set;

[0067] Step S323: defining the output of the speech recognition model;

[0068] If the regular expression is true, then the output is that there is bad language; if the regular expression is not true, then no output;

[0069] Step S33: constructing a behavior recognition model based on the image test set;

[0070] Step S34: Substitute the analyzed image and the speech to be analyzed into the behavior recognition model and the speech recognition model, and check the model output;

[0071] If the behavior recognition model has output, then there is overt malicious behavior in the target area;

[0072] If the behavior recognition model has no output, but the speech recognition model has output, then there is hidden malicious behavior in the target area;

[0073] If both the behavior recognition model and the speech recognition model have no output, monitoring is performed.

[0074] Furthermore, the specific steps of step S33 are as follows:

[0075] Step S331: constructing a behavior recognition model using the basic model;

[0076] The activation function selects the softmax function;

[0077] Step S332: define the BN operation: constrain the underlying eigenvalues ​​of the basic model to a distribution with a mean equal to 0 and a variance equal to 1;

[0078] For any input B{x1, x2, ... m The process of performing BN operation is as follows:

[0079] Calculate the mean μB of the elements in set B, ;

[0080] Calculate the variance σB of the elements in set B 2 , ; Standardize the elements in set B and get ~ ;

[0081] Calculate x1~x m get ~ of The calculation formula is as follows:

[0082] ; Wherein, input B refers to the image in the image test set; i represents 1 to m, and m represents the pixel value of the image in input B;

[0083] Step S333: Create two pooling layers, Pool 1 and Pool 2, for the behavior recognition model;

[0084] Create 5 convolutional layers, starting with the convolutional layer Con1 and the subsequent convolutional layers Conv2 to Conv5;

[0085] Create a connection layer Concat; create two fully connected layers Fc6~Fc8;

[0086] Step S334: upload the image of the image test set as a primary input to the Con1 layer and the Pool1 layer, perform bottom-level feature extraction, and obtain extracted feature 1;

[0087] Take the extracted feature 1 as the secondary input and upload it to the Conv2 layer and the Pool2 layer to obtain the extracted feature 2;

[0088] Backup extracted feature 2 as the underlying feature;

[0089] Step S335: repeat step S324, upload the extracted feature 2 to the Conv3 layer for processing, and obtain the extracted feature 3; upload the extracted feature 3 to the Conv4 layer for processing, and obtain the extracted feature 4; upload the extracted feature 4 to the Conv5 layer for processing, and obtain the extracted feature 5 as the high-level feature;

[0090] Step S336: perform BN operation on the bottom-level features and high-level features respectively to obtain fused features; use the fused features as the input of the fully connected layer fc6;

[0091] Step S337: Take the model input of the behavior recognition model as x, the model output as y, and define the loss function J:

[0092] ;

[0093] Where N represents the number of iterations, i.e., batch-size;

[0094] x i and i , respectively represent the model input and model output when the behavior recognition model iterates i times; w represents the weight, and f(w, xi) represents the weight of the model input xi when the behavior recognition model iterates i times;

[0095] R(w) represents the regularization term, and λ represents the weight of the control regularization term;

[0096] L represents the cross entropy loss function;

[0097] Step S338: define the output of the behavior recognition model; if the image to be analyzed is associated with the image test set, then output that there is bad behavior; if the image to be analyzed is not associated with the image test set, then no output is given.

[0098] A system for identifying audio specific information comprises:

[0099] Audio acquisition module: used to acquire public data of the target area; extract the image part in the public data to obtain image information; extract the voice part in the public data to obtain voice information;

[0100] Audio enhancement module: used to perform denoising, enhancement and binarization on image information to obtain the image to be analyzed; perform pre-emphasis, windowing and endpoint detection on voice information to obtain the voice to be analyzed;

[0101] Audio analysis module: used to obtain specific image information and specific voice information as image test sets and voice test sets; use the voice test sets and image test sets to build behavior recognition models and voice recognition models; use the behavior recognition models and voice recognition models to analyze the voice and image to be analyzed, and determine whether there are explicit malicious behaviors and implicit malicious behaviors in the public data;

[0102] If it exists, save the public data showing explicit malicious behavior or implicit malicious behavior;

[0103] If not present, continue monitoring;

[0104] User interaction module: used to summarize the public data saved by the audio analysis module and feed it back to the user; periodically destroy the public data obtained by the audio acquisition module.

[0105] Compared with the prior art, the present invention has the following beneficial effects:

[0106] Improve the efficiency of network content analysis: Through automated processing, the present invention can quickly scan and analyze voice content on the network, greatly improving the recognition efficiency and accuracy of audio.

[0107] Accurately identify inappropriate content: The present invention is based on machine learning and can quickly identify various inappropriate content, such as violence, gambling, malicious speech, etc., through data training. It will also combine the user's words and deeds to make comprehensive judgments from multiple dimensions to improve the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0109] Figure 1 It is a schematic diagram of the method of the present invention;

[0110] Figure 2 It is a schematic diagram of the system of the present invention;

[0111] Figure 3 This is a schematic diagram of data processing of the present invention. DETAILED DESCRIPTION

[0112] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0113] Embodiment 1

[0114] See also Figure 1 and Figure 3 , a method for identifying audio specific information includes:

[0115] Step S1: acquiring public data of a target area; extracting the voice part of the public data to obtain voice information; extracting the image part of the public data to obtain image information;

[0116] It should be noted that the "target area" in the present invention refers to the area where the present invention is used to perform "audio specific information recognition"; "public data" refers to open source data obtained from various media websites or video websites.

[0117] Specific steps of step S1:

[0118] Step S11: Obtaining public data (surveillance video) of the target area;

[0119] Step S12: extracting the image part from the public data to obtain image information;

[0120] Step S121: Import the OpenCV library; (OpenCV is an open source computer vision and machine learning software library)

[0121] Use the absolute path of the public data as the parameter of the cv2.VideoCapture() method and use the cv2.VideoCapture() method to read the video file in the public data;

[0122] Step S122: Use the read() method to read the image in the video file frame by frame; the read() method returns two values: one is a Boolean value ("True" or "False"), indicating whether the frame is successfully read; the other is the frame, that is, the image itself;

[0123] Step S123: Determine whether the read() method is True;

[0124] If yes, use cv2.imwrite method to save the frame output by read() method as an image file; the image file format options are: JPEG, PNG, TIFF, etc. (users or relevant technicians can determine the image file saving format according to actual needs)

[0125] If not, no processing;

[0126] Step S124: Summarize the image files saved in step S123 to obtain image information;

[0127] Step S13: extracting the voice part in the public data to obtain voice information;

[0128] Step S131: Import the moviepy library; (moviepy is a Python library for video editing)

[0129] Use the absolute path of the public data as the parameter of the cv2.VideoCapture() method, use the VideoFileClip() method to read the video file in the public data, and receive it with the variable video;

[0130] Step S132: extract the audio attribute of the variable video, obtain the voice in the video file, and receive it with the variable adio, that is, adio = audio.audio;

[0131] Step S133: Use the write_audiofile() method to save the audio file, i.e., adio.write_audiofile(...); wherein, the ... in adio.write_audiofile(...) represents the parameter of write_audiofile(), which is used to set the saving format of the audio file; the saving formats of the audio file include: WAV, FLAC, APE, etc. (the user or relevant technical personnel can determine the saving format of the audio file according to actual needs)

[0132] Step S134: Summarize the audio files saved in step S133 to obtain voice information; use the close() method to release the variables video and adio, that is:

[0133] video.close();

[0134] adio.close().

[0135] Step S2: performing pre-emphasis, windowing and endpoint detection enhancement processing on the speech information to obtain the speech to be analyzed; performing denoising, enhancement and binarization enhancement processing on the image information to obtain the image to be analyzed;

[0136] The specific steps of step S2 are as follows:

[0137] Step S21: performing denoising, enhancement and binarization enhancement processing on the image information to obtain an image set to be analyzed;

[0138] Step S211: Obtain the frame rate of the image information, denoted as fr; perform enhancement processing on the first frame of image information; (hereinafter, "the first frame of image information" is referred to as "image 1")

[0139] Step S212: obtaining the resolution of image 1, denoted as res;

[0140] Calculate the kernel order of image 1, denoted by J; J = log pa (res); where pa represents the setting parameter, and the value of pa is 4; J is a positive integer, rounded down; (image denoising)

[0141] It should be noted that the function of the kernel order J is to determine the kernel size of the Gaussian filter; pa is generally 4; users or relevant technical personnel can decide the value of pa according to actual needs.

[0142] Step S213: define a J×J minimum image scanning unit, use a Gaussian filter to process image 1, remove noise from image 1, and obtain image A1;

[0143] Step S214: construct a histogram of the image A1 with the pixel brightness value (usually between 0 and 255, corresponding to the pixel value range of an 8-bit grayscale image) as the abscissa and the frequency of occurrence of the pixel brightness value in the image (i.e., the number of pixels) as the ordinate;

[0144] Import the OpenCV library; use the cv2.equalizeHist histogram equalization function in the OpenCV library to process image A1 to obtain image A2; (image enhancement)

[0145] Step S215: Use the cv2.adaptiveThreshold() function in the OpenCV library to process the image A2 and obtain an adaptive threshold;

[0146] Use the cv2.threshold() function in the OpenCV library to binarize image A2 according to the adaptive threshold to obtain image 1 to be analyzed; (image binarization)

[0147] Step S216: repeating the same step of enhancing the first frame of image information to process the second to fr frames of image information;

[0148] Step S217: Summarize the images obtained in steps S211 to S216 as images to be analyzed;

[0149] Step S22: performing pre-emphasis, windowing and endpoint detection enhancement processing on the voice information to obtain the voice to be analyzed;

[0150] Step S221: Select the characteristic (1-aZ -1 ) as a pre-emphasis filter; a represents the pre-emphasis coefficient, the value range of a is 0.9 to 1, and a is generally 0.9375; Z represents the amplitude of the sine wave; (users or relevant technical personnel can decide the value of a according to actual needs)

[0151] Use a pre-emphasis filter to process the voice information, remove the low-frequency noise in the voice information, and obtain the sound information V1;

[0152] Suppose the frequency function of the voice information before pre-emphasis is S(n); where n represents the timestamp of the voice information, and S(n) represents the frequency corresponding to the nth timestamp of the voice information; Suppose the frequency function of the sound information V1 is ;

[0153] S(n) and The relationship is as follows:

[0154] ; (pre-emphasis)

[0155] Step S222: Use the MATLAB Signal Processing Toolbox library function to extract the main lobe and side lobe of the sound information V1; the main lobe represents the largest radiation beam on the sound wave radiation pattern, which is the area where the sound wave energy is most concentrated; the side lobe is the small radiation beam on both sides of the main lobe in the sound wave radiation pattern; (the "sound wave radiation pattern" in the present invention means the "sound wave radiation pattern of the sound information V1"; the sound wave radiation pattern can be obtained by the MATLAB library function)

[0156] Set the sampling rate of the sound information V1 to 8KHz (8×1024 Hz); for (sound information V1) With 256 timestamps as one frame, the sampling points of frame shift are 64 timestamps;

[0157] For the lth frame of sound information x in the sound information V1 lThe relationship between "the phase of the sound information of the lth frame after the frame shift" and "the original phase (the phase of the sound information V1 without the frame shift, i.e. the sound information V1 in step S221)" is as follows: (where l is a natural number, and l is only used for sample demonstration and has no special meaning)

[0158] x l (i) = x[(l-1)*m+i];

[0159] Where i is a natural number with no special meaning, and the value range of i is 0 to (F-1);

[0160] x l (i) represents the phase of the i-th timestamp of the l-th frame after the frame shift of (sound information V1), and x[(l-1)*m+i] represents the original phase of the [(l-1)*m+i]-th timestamp, that is, (in step S222) when x in the sound information V1 l After frame shift occurs, x l The phase of the i-th timestamp in becomes the corresponding phase of the [(l-1)*m+i]-th timestamp in the original phase (in step S221);

[0161] F represents the frame length, F=256; M represents the sampling period, M=64;

[0162] Will Multiply it with the window function w(n) to get the sound information V2 (windowed speech information);

[0163] The function expression of the sound information V2 is denoted as Sw;

[0164] Sw= *w(n);

[0165] Window function w(n) selection: rectangular window and Hamming window (users or relevant technical personnel can decide the window function according to actual needs);

[0166] When the window function is a rectangular window, the expression of w(n) is:

[0167] ;

[0168] When the window function is a Hamming window, the expression of w(n) is:

[0169] ;

[0170] Scenario 1: When the sampling point is in the main lobe, the window function uses a rectangular window; when the sampling point is in the side lobe, the window function uses a Hamming window;

[0171] Scenario 2: When the sampling point does not belong to Scenario 1, a rectangular window is used uniformly; (windowing)

[0172] Step S223: using MATLAB library functions to distinguish “unvoiced sound” and “voiced sound” in the sound information V2; wherein “unvoiced sound” means (in the sound information V2) a sound segment with a small amplitude of the time domain waveform spectrum, low energy, and no periodicity; and “voiced sound” means (in the sound information V2) a sound segment with a large amplitude of the time domain waveform spectrum, high energy, and quasi-periodicity;

[0173] The short-time average energy is used to detect the unvoiced sound. The calculation formula of the short-time average energy E(l) is as follows:

[0174] ; Use the short-time zero-crossing rate to detect voiced sounds. The calculation formula of the short-time zero-crossing rate ZCR(l) is as follows:

[0175] ;

[0176] Among them, F, l, i and x in the E(l) calculation formula and ZCR(l) calculation formula are l (i), the corresponding meanings are the same as F, l, i and x in step S222 l (i) has the same meaning as x; l (i+1) represents the phase of the (i+1)th timestamp of the lth frame. (Endpoint detection, the purpose of endpoint detection is to improve the accuracy of speech recognition methods or systems (methods or systems that convert "language" into "text"))

[0177] Step S224: Summarize the speech obtained in steps S221 to S223 to obtain the speech to be analyzed.

[0178] Step S3: obtaining specific image information and specific voice information as an image test set and a voice test set; using the voice test set and the image test set to construct a (bad) voice recognition model and a (bad) behavior recognition model; using the (bad) voice recognition model and the (bad) behavior recognition model to analyze the voice to be analyzed and the image to be analyzed, and determining whether there is explicit malicious behavior and implicit malicious behavior in the public data;

[0179] If so, save the public data (surveillance video) showing explicit or implicit malicious behavior;

[0180] If not present, continue monitoring;

[0181] The specific steps of step S3 are as follows:

[0182] Step S31: obtaining specific image information and specific voice information as an image test set and a voice test set;

[0183] It should be noted that the "specific image information" in the present invention refers to images of bad behaviors such as violence, gambling, and alcoholism; "specific voice information" refers to bad words such as cursing, ridicule, and slander;

[0184] Step S32: constructing a (bad) behavior recognition model based on the image test set;

[0185] Step S321: using the torchvision.models module in the PyTorch library, importing the AlexNet model, the SqueezeNet model, and the GoogLeNet model as the basic models of the (bad) behavior recognition model;

[0186] Select "softmax function" for the activation function.

[0187] Step S322: define a BN operation (a Batch Normalization operation that removes linear changes); constrain the underlying eigenvalues ​​of the basic model to a distribution with a mean of 0 and a variance of 1;

[0188] For any input B{x1, x2, ... m The process of performing BN operation is as follows: (input B represents the picture)

[0189] Calculate the mean μB of the elements in set B, ;

[0190] Calculate the variance σB of the elements in set B 2 , ; Standardize the elements in set B and get ~ ;

[0191] Calculate x1~x m get ~ of The calculation formula is as follows:

[0192] ; Wherein, input B refers to the image in the image test set (set B); i represents 1 to m, and m represents the pixel value of the image in input B;

[0193] Step S323: Create two pooling layers, Pool1 and Pool2, for the (bad) behavior recognition model;

[0194] Create 5 convolutional layers, starting with the convolutional layer Con1 and the subsequent convolutional layers Conv2 to Conv5;

[0195] Create a connection layer Concat; create two fully connected layers Fc6~Fc8;

[0196] Step S324: upload the image of the image test set as a primary input to the Con1 layer and the Pool1 layer, perform bottom-level feature extraction, and obtain extracted feature 1;

[0197] Take the extracted feature 1 as the secondary input and upload it to the Conv2 layer and the Pool2 layer to obtain the extracted feature 2;

[0198] Backup extracted feature 2 as the underlying feature;

[0199] Step S325: repeat step S324, upload the extracted feature 2 to the Conv3 layer for processing, and obtain the extracted feature 3; upload the extracted feature 3 to the Conv4 layer for processing, and obtain the extracted feature 4; upload the extracted feature 4 to the Conv5 layer for processing, and obtain the extracted feature 5 as the high-level feature;

[0200] Step S326: perform BN operation on the bottom-level features and high-level features respectively to obtain fused features; use the fused features as the input of the fully connected layer fc6;

[0201] Step S327: Take the model input of the (bad) behavior recognition model as x (image test set), the model output as y, and define the loss function J:

[0202] ;

[0203] Where N represents the number of iterations (for training the (bad) behavior recognition model), i.e., batch-size;

[0204] x i and i , respectively represent the model input and model output when the (bad) behavior recognition model is iterated i times; w represents the weight, and f(w, xi) represents the weight of the model input xi when the (bad) behavior recognition model is iterated i times;

[0205] R(w) represents the regularization term, and λ represents the weight of the control regularization term;

[0206] L represents the cross entropy loss function;

[0207] Step S328: defining the output of the (bad) behavior recognition model;

[0208] If the image to be analyzed is associated with the image test set (i.e., there is an intersection), the output has bad behavior; if the image to be analyzed is not associated with the image test set (i.e., there is no intersection), no output;

[0209] Step S33: constructing a (bad) speech recognition model according to the speech test set;

[0210] Step S331: import the SpeechRecognition library; use the Recognizer class in the SpeechRecognition library to perform speech recognition on the speech test set and the speech to be analyzed (convert "speech" to "text"), and obtain the detection segment set and the to-be-detected segment set;

[0211] Step S332: Using the HMM model as the basic model of the (bad) speech recognition model;

[0212] Define a regular expression in the fully connected layer of the HMM model:

[0213] print(normalized_phrase1 == normalized_phrase2); where normalized_phrase1 represents the words in the detection word set (speech test set), and normalized_phrase2 represents the words in the to-be-detected word set (speech to be analyzed);

[0214] Step S333: defining the output of the (bad) speech recognition model;

[0215] If the regular expression is true, then the output is that there is bad language; if the regular expression is not true, then no output;

[0216] Step S34: Substitute the analyzed image and the speech to be analyzed into the (bad) behavior recognition model and the (bad) speech recognition model, and check the model output;

[0217] If the (bad) behavior recognition model has output, then there is overt malicious behavior in the target area;

[0218] If the (bad) behavior recognition model has no output, but the (bad) speech recognition model has output, then there is hidden malicious behavior in the target area;

[0219] If both the (bad) behavior recognition model and the (bad) speech recognition model have no output, monitoring is performed.

[0220] It should be noted that explicit malicious behavior refers to behavior that involves physical conflict and verbal conflict or only physical conflict, while implicit malicious behavior refers to behavior that involves verbal conflict but not physical conflict;

[0221] A method for identifying audio specific information also includes step S4: periodically destroying the public data obtained in step S1, and the destruction period is: 3 weeks (users or relevant technical personnel can adjust the destruction period according to actual needs).

[0222] Embodiment 2

[0223] See also Figure 2A system for identifying audio specific information includes: an audio acquisition module, a feature extraction module, an audio analysis module, a user interaction module, a database and a server; wherein the audio acquisition module, the feature extraction module, the audio analysis module and the user interaction module are connected to the database and the server respectively.

[0224] Audio acquisition module: used to acquire public data of the target area; extract the voice part in the public data to obtain voice information; extract the image part in the public data to obtain image information;

[0225] Audio enhancement module: used to perform pre-emphasis, windowing and endpoint detection on voice information to obtain the voice to be analyzed; perform denoising, enhancement and binarization on image information to obtain the image to be analyzed;

[0226] Audio analysis module: used to obtain specific image information and specific voice information as image test sets and voice test sets; use the voice test sets and image test sets to build (bad) voice recognition models and (bad) behavior recognition models; use the (bad) voice recognition models and (bad) behavior recognition models to analyze the voice and image to be analyzed, and determine whether there is explicit malicious behavior and implicit malicious behavior;

[0227] If it exists, save the public data showing explicit malicious behavior or implicit malicious behavior;

[0228] If not present, continue monitoring;

[0229] User interaction module: used to aggregate public data and provide feedback to users; periodically destroy the public data obtained by the audio acquisition module, with a destruction cycle of 3 weeks.

[0230] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technical personnel in this field according to actual conditions. For example, if there are weight coefficients and proportional coefficients, their set sizes are to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. Regarding the size of the weight coefficient and the proportional coefficient, it is sufficient as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0231] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for identifying audio specific information, characterized in that: Identification methods include: Step S1: acquiring public data of a target area; extracting the voice part of the public data to obtain voice information; extracting the image part of the public data to obtain image information; Step S2: performing pre-emphasis, windowing and endpoint detection enhancement processing on the voice information to obtain the voice to be analyzed; performing enhancement processing on the image information to obtain the image to be analyzed; Step S3: obtaining specific image information and specific voice information as an image test set and a voice test set; using the voice test set and the image test set, constructing a voice recognition model and a behavior recognition model; using the voice recognition model and the behavior recognition model, analyzing the voice to be analyzed and the image to be analyzed, and determining whether there are explicit malicious behaviors and implicit malicious behaviors in the public data; if there are, saving the public data in which the explicit malicious behaviors or implicit malicious behaviors appear; if not, continuing monitoring; The specific steps of step S2 are as follows: Step S21: pre-emphasize the voice information to obtain sound information V1; Step S211: Select the characteristic (1-aZ -1 ) as a pre-emphasis filter; where a represents the pre-emphasis coefficient; Z represents the sine wave amplitude; Step S212: using a pre-emphasis filter to process the voice information, remove low-frequency noise in the voice information, and obtain sound information V1; Step S213: Suppose the frequency function of the voice information before pre-emphasis is S(n); wherein n represents the timestamp of the voice information, and S(n) represents the frequency corresponding to the nth timestamp of the voice information; Suppose the frequency function of the sound information V1 is ; S(n) and The relationship is as follows: ; Step S213: performing windowing processing on the voice information V1 to obtain sound information V2; Step S214: performing endpoint detection processing on the voice information V2 to obtain the voice to be analyzed; Step S22: Enhance the image information to obtain an image set to be analyzed, and proceed to step S3; The specific steps of step S213 are as follows: Step S2131: Use the MATLAB Signal Processing Toolbox library function to extract the main lobe and side lobe of the sound information V1; the main lobe represents the largest radiation beam on the sound wave radiation pattern, which is the area where the sound wave energy is most concentrated; the side lobe is the small radiation beam on both sides of the main lobe in the sound wave radiation pattern; Step S2132: Set the sampling rate of the sound information V1 to 8KHz; With 256 timestamps as one frame, the sampling points of frame shift are 64 timestamps; For the lth frame of sound information x in the sound information V1 l , the relationship between the phase of the sound information of the lth frame after the frame shift and the original phase is as follows: x l (i)=x[(l-1)*m+i]; Wherein, i is a natural number; x l (i) represents the phase of the i-th timestamp of the l-th frame after the frame shift, and x[(l-1)*m+i] represents the original phase of the [(l-1)*m+i]-th timestamp, that is, when x in the sound information V1 is l After frame shift occurs, x l The phase of the i-th timestamp in becomes the corresponding phase of the [(l-1)*m+i]-th timestamp in the original phase; F represents the frame length, F=256; M represents the sampling period, M=64; Step S2133: Multiply it with the window function w(n) to get the sound information V2; The function expression of the sound information V2 is denoted as Sw; Sw= *in(n); Window function w(n) selection: rectangular window and Hamming window; When the window function is a rectangular window, the expression of w(n) is: ; When the window function is a Hamming window, the expression of w(n) is: ; Step S2134: setting the use scenarios of the rectangular window and the Hamming window; Scenario 1: When the sampling point is in the main lobe, the window function uses a rectangular window; when the sampling point is in the side lobe, the window function uses a Hamming window; Scenario 2: When the sampling point does not belong to scenario 1, a rectangular window is used uniformly; The specific steps of step S214 are as follows: Step S2141: using MATLAB library functions to distinguish between unvoiced and voiced sounds in the sound information V2; wherein, unvoiced sounds mean: in the sound information V2, the time domain waveform spectrum has a small amplitude, low energy, and no periodic sound segment; voiced sounds mean: in the sound information V2, the time domain waveform has a large amplitude, high energy, and a quasi-periodic sound segment; Step S2142: Use the short-time average energy to detect the unvoiced sound. The calculation formula of the short-time average energy E(l) is as follows: ; Use the short-time zero-crossing rate to detect voiced sounds. The calculation formula of the short-time zero-crossing rate ZCR(l) is as follows: ; Among them, F, l, i and x in the E(l) calculation formula and ZCR(l) calculation formula are l (i), the corresponding meanings are the same as F, l, i and x in step S2132 l (i) has the same meaning as x; l (i+1) represents the phase of the (i+1)th timestamp of the lth frame; Step S2143: Summarize the speech obtained in step S2142 as the speech to be analyzed.

2. The method for identifying audio specific information according to claim 1, characterized in that: The specific steps of step S1 are: Step S11: Acquire public data of the target area; Step S12: extracting the voice part from the public data to obtain voice information; Step S121: import the moviepy library; use the absolute path of the public data as a parameter of the cv2.VideoCapture() method, use the VideoFileClip() method to read the video file in the public data, and receive it with the variable video; Step S122: extract the audio attribute of the variable video, obtain the voice in the video file, and receive it with the variable adio, that is, adio = audio.audio; Step S123: Use write_audiofile() method to save the audio file, i.e., adio.write_audiofile(...); The ... in adios.write_audiofile(...) indicates the parameters of write_audiofile(), which is used to set the saving format of the audio file. Step S124: Summarize the audio files saved in step S133 to obtain voice information; use the close() method to release the variables video and adio, that is: video.close(); adio.close(); Step S13: extracting the image part from the public data to obtain image information; Import OpenCV library; use cv2.VideoCapture() method and read() method to process the image part in the public data and obtain image information; Step S14: Summarize the image information and voice information and proceed to step S2.

3. The method for identifying audio specific information according to claim 1, characterized in that: The specific steps of step S3 are as follows: Step S31: acquiring specific image information and specific voice information as an image test set and a voice test set; Step S32: constructing a speech recognition model according to the speech test set; Step S321: import the SpeechRecognition library; use the Recognizer class in the SpeechRecognition library to perform speech recognition on the speech test set and the speech to be analyzed, and obtain the detection segment set and the to-be-detected segment set; Step S322: Using the HMM model as the basic model of the speech recognition model; Define a regular expression in the fully connected layer of the HMM model: print(normalized_phrase1 == normalized_phrase2); where normalized_phrase1 represents a word segment in the detection word segment set, and normalized_phrase2 represents a word segment in the to-be-detected word segment set; Step S323: defining the output of the speech recognition model; If the regular expression is true, then the output is that there is bad language; if the regular expression is not true, then no output; Step S33: constructing a behavior recognition model based on the image test set; Step S34: Substitute the analyzed image and the speech to be analyzed into the behavior recognition model and the speech recognition model, and check the model output; If the behavior recognition model has output, then there is overt malicious behavior in the target area; If the behavior recognition model has no output, but the speech recognition model has output, then there is hidden malicious behavior in the target area; If both the behavior recognition model and the speech recognition model have no output, monitoring is performed.

4. The method for identifying audio specific information according to claim 3, characterized in that: The specific steps of step S33 are as follows: Step S331: constructing a behavior recognition model using the basic model; The activation function selects the softmax function; Step S332: define the BN operation: constrain the underlying eigenvalues ​​of the basic model to a distribution with a mean equal to 0 and a variance equal to 1; For any input B{x1, x2, ... m The process of performing BN operation is as follows: Calculate the mean μB of the elements in set B, ; Calculate the variance σB of the elements in set B 2 , ; Standardize the elements in set B and get ~ ; Calculate x1~x m get ~ of The calculation formula is as follows: ; Wherein, input B refers to the image in the image test set; i represents 1 to m, and m represents the pixel value of the image in input B; Step S333: Create two pooling layers, Pool 1 and Pool 2, for the behavior recognition model; Create 5 convolutional layers, starting with the convolutional layer Con1 and the subsequent convolutional layers Conv2 to Conv5; Create a connection layer Concat; create two fully connected layers Fc6~Fc8; Step S334: upload the image of the image test set as a primary input to the Con1 layer and the Pool1 layer, perform bottom-level feature extraction, and obtain extracted feature 1; Take the extracted feature 1 as the secondary input and upload it to the Conv2 layer and the Pool2 layer to obtain the extracted feature 2; Backup extracted feature 2 as the underlying feature; Step S335: repeat step S324, upload the extracted feature 2 to the Conv3 layer for processing, and obtain the extracted feature 3; upload the extracted feature 3 to the Conv4 layer for processing, and obtain the extracted feature 4; upload the extracted feature 4 to the Conv5 layer for processing, and obtain the extracted feature 5 as the high-level feature; Step S336: perform BN operation on the bottom-level features and high-level features respectively to obtain fused features; use the fused features as the input of the fully connected layer fc6; Step S337: Take the model input of the behavior recognition model as x, the model output as y, and define the loss function J: ; Where N represents the number of iterations, i.e., batch-size; x i and i , respectively represent the model input and model output when the behavior recognition model iterates i times; w represents the weight, and f(w, xi) represents the weight of the model input xi when the behavior recognition model iterates i times; R(w) represents the regularization term, and λ represents the weight of the control regularization term; L represents the cross entropy loss function; Step S338: define the output of the behavior recognition model; if the image to be analyzed is associated with the image test set, then output that there is bad behavior; if the image to be analyzed is not associated with the image test set, then no output is given.

5. A system for identifying audio specific information, applicable to any one of the methods for identifying audio specific information in claims 1-4, characterized in that: The identification system comprises: Audio acquisition module: used to acquire public data of the target area; extract the voice part in the public data to obtain voice information; extract the image part in the public data to obtain image information; Audio enhancement module: used to perform pre-emphasis, windowing and endpoint detection on voice information to obtain the voice to be analyzed; perform denoising, enhancement and binarization on image information to obtain the image to be analyzed; Audio analysis module: used to obtain specific image information and specific voice information as image test sets and voice test sets; using the voice test sets and image test sets, construct a voice recognition model and a behavior recognition model; using the voice recognition model and the behavior recognition model, analyze the voice and image to be analyzed to determine whether there is explicit malicious behavior and implicit malicious behavior; If it exists, save the public data showing explicit malicious behavior or implicit malicious behavior; If not present, continue monitoring; User interaction module: used to summarize the public data saved by the audio analysis module and feed it back to the user; periodically destroy the public data obtained by the audio acquisition module.