A method, apparatus, device, and readable storage medium for detecting audio anomalies.
By combining image and audio data detection in a video conferencing system, audio anomalies can be quickly identified and resolved, improving the efficiency of audio anomaly detection and reducing the time spent locating problems during meetings.
Patent Information
- Application Number
- CN202211667016.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-12-22
AI Technical Summary
When a video conferencing system encounters an audio anomaly, multiple meeting participants need to gather information to locate the problem, resulting in low efficiency in audio anomaly detection.
By acquiring image and audio data from the video conferencing venue, we can perform motion feature detection and audio detection, and combine the detection results to determine whether there are any abnormalities in the target device, including the speaking status of the person, the silence status of the venue, and noise detection.
It improves the efficiency of audio anomaly detection, enabling quick location and resolution of audio anomalies and reducing delays in the meeting process.
Smart Images

Figure CN116935459B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to an audio anomaly detection method, apparatus, device, and readable storage medium. Background Technology
[0002] With the increasing popularity of remote collaboration and online learning, video conferencing has become an important communication tool in our lives and work. However, in actual use, when video conferencing systems encounter audio anomalies, multiple participants typically need to gather information to locate the problem. This approach is limited by the fact that participants need to have basic audio problem-solving skills, and it takes a considerable amount of time to locate and resolve the issue, thus slowing down the meeting process and resulting in low efficiency in audio anomaly detection. Summary of the Invention
[0003] This invention provides an audio anomaly detection method, apparatus, device, and readable storage medium. In practical applications, when a video conferencing system encounters an audio anomaly, multiple meeting participants generally need to summarize information to locate the problem, resulting in low efficiency in audio anomaly detection.
[0004] To address the above problems, the embodiments of the present invention adopt the following technical solutions:
[0005] In a first aspect, embodiments of the present invention provide an audio anomaly detection method, comprising:
[0006] Acquire image and audio data of a target location, wherein the target location is a video conferencing location, and the image and audio data are the image and audio data corresponding to the target device in the video conferencing location;
[0007] Based on the image data, human motion features are detected to obtain a first detection result, which is used to indicate whether the person corresponding to the target device is in a speaking state.
[0008] Based on the audio data, audio detection is performed to obtain a second detection result, which is used to indicate whether the target location is in a silent state.
[0009] Based on the first detection result and the second detection result, it is determined whether the target device is abnormal.
[0010] Optionally, the step of detecting human motion features based on the image data to obtain a first detection result includes:
[0011] Based on the image data, facial key point detection is performed to determine the key points of the mouth of the person corresponding to the target device;
[0012] Based on the key points, the opening and closing state of the mouth is detected to determine the character's speaking state;
[0013] Determine whether the person corresponding to the target device is in a speaking state.
[0014] Optionally, the step of performing audio detection based on the audio data to determine the second detection result includes:
[0015] Based on the audio data, calculate the average amplitude of the audio data;
[0016] If the average amplitude is greater than or equal to a preset value, it is determined that the target location is not in a silent state;
[0017] If the average amplitude is less than a preset value, the target location is determined to be in a silent state.
[0018] Optionally, determining whether the target device is abnormal based on the first detection result and the second detection result includes:
[0019] Based on the first detection result and the second detection result, it is determined whether the person corresponding to the target device is in a speaking state and whether the target location is in a silent state.
[0020] If the person is not speaking and the target location is in a silent state, the target device is determined to be normal.
[0021] When the person is not speaking and the target location is not in a silent state, noise detection is performed on the audio data to determine whether the target device is abnormal;
[0022] When the person is speaking and the target location is in a silent state, the distance between the person and the microphone is calculated based on the image data to determine whether there is any abnormality in the target device, where the microphone is the voice input terminal of the target device;
[0023] When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine whether the target device is abnormal.
[0024] Optionally, when the person is not speaking and the target location is not in a silent state, performing noise detection on the audio data to determine whether the target device is abnormal includes:
[0025] If the person is not speaking and the target location is not muted, scene detection is performed on the target location to determine the scene of the target location;
[0026] Noise detection is performed based on the audio data to determine noise characteristics;
[0027] Determine whether the scene and the noise characteristics match to determine if the target device is abnormal.
[0028] Optionally, when the person is speaking and the target location is in a silent state, calculating the distance between the person and the microphone based on the image data to determine whether the target device is abnormal includes:
[0029] When the person is speaking and the target location is in a silent state, the microphone position coordinates and the mouth position coordinates of the person are determined based on the image data.
[0030] Calculate the distance between the microphone and the mouth based on the microphone position coordinates and the mouth position coordinates;
[0031] If the distance is less than a preset distance, it is determined that the target device is malfunctioning.
[0032] If the distance is greater than or equal to a preset distance, the target device is determined to be normal.
[0033] Optionally, when the person is speaking and the target location is not muted, performing activity detection based on the audio data to determine whether the target device is abnormal includes:
[0034] When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine the probability of speech presence in the audio data.
[0035] If the probability of the voice being present is greater than a preset probability value, the target device is determined to be normal.
[0036] If the probability of the voice being present is less than or equal to a preset probability value, the distance between the microphone and the mouth is calculated to determine whether the target device is abnormal.
[0037] Secondly, embodiments of the present invention provide an audio anomaly detection device, the device comprising:
[0038] The acquisition module is used to acquire image data and audio data of a target location, wherein the target location is a video conferencing location, and the image data and audio data are image data and audio data corresponding to the target device in the video conferencing location;
[0039] The first detection module is used to perform human motion feature detection based on the image data to obtain a first detection result, which is used to indicate whether the person corresponding to the target device is in a speaking state.
[0040] The second detection module is used to perform audio detection based on the audio data and obtain a second detection result, which is used to indicate whether the target location is in a silent state.
[0041] The determination module is used to determine whether the target device has any abnormalities based on the first detection result and the second detection result.
[0042] Thirdly, embodiments of the present invention provide an audio anomaly detection device, the device comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor;
[0043] The processor is configured to read a program from memory to implement the steps described in the first aspect of the method.
[0044] Fourthly, embodiments of the present invention provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0045] In this embodiment of the invention, the audio anomaly detection method can detect the image data and audio data of the video conferencing venue separately to determine whether the person corresponding to the target device is speaking and whether the target venue is in a silent state. This allows for a comprehensive analysis of whether the target device is abnormal, pinpointing the specific cause of the audio anomaly, and providing specific solutions in conjunction with the image data, thus making the audio anomaly detection highly efficient. Attached Figure Description
[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is one of the flowcharts of an audio anomaly detection method provided in an embodiment of the present invention;
[0048] Figure 2 This is one of the functional block diagrams of an audio anomaly detection method provided in an embodiment of the present invention;
[0049] Figure 3 This is the second functional module diagram of an audio anomaly detection method provided in an embodiment of the present invention;
[0050] Figure 4 The third functional module diagram of an audio anomaly detection method provided in this embodiment of the invention;
[0051] Figure 5 A second flowchart of an audio anomaly detection method provided in an embodiment of the present invention;
[0052] Figure 6 A flowchart of an audio anomaly detection method provided in an embodiment of the present invention is shown in Part 3.
[0053] Figure 7 This is a structural diagram of an audio anomaly detection device provided in an embodiment of the present invention;
[0054] Figure 8 This is a structural diagram of an audio anomaly detection device provided in an embodiment of the present invention. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0056] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.
[0057] Please see Figure 1 , Figure 1 A flowchart of an audio anomaly detection method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:
[0058] Step 101: Obtain image data and audio data of the target location, wherein the target location is a video conferencing location, and the image data and audio data are the image data and audio data corresponding to the target device in the video conferencing location.
[0059] Specifically, the target location can be a video conferencing venue, which can be a venue where both the microphone and camera are turned on. The image data can be image data corresponding to the target device in the video conferencing venue over a period of time. The audio data can be audio data corresponding to the target device in the video conferencing venue over a period of time. The target device can be audio equipment used in the video conferencing venue, etc.
[0060] Step 102: Perform character motion feature detection based on the image data to obtain a first detection result. The first detection result is used to indicate whether the character corresponding to the target device is in a speaking state.
[0061] Specifically, the character motion feature detection can be lip movement recognition, which detects the mouth movements of the person corresponding to the target device to obtain a first detection result; whether the person corresponding to the target device is in a speaking state can be determined by whether the person's mouth is in a speaking state.
[0062] Step 103: Perform audio detection based on the audio data to obtain a second detection result, which is used to indicate whether the target location is in a silent state.
[0063] Specifically, the audio detection can be to perform silence detection on the audio data, determine the silence state of the audio data, and thus obtain a second detection result. Whether the target location is in a silent state can be determined by whether there is any sound in the target location, including human voices and noise.
[0064] Step 104: Based on the first detection result and the second detection result, determine whether the target device is abnormal.
[0065] Specifically, determining whether the target device is abnormal can mean that the target device is normal and the user is operating it improperly, or that the target device is abnormal and needs to be inspected, repaired, or replaced by the user. The user is the person using the target device.
[0066] Steps 102 and 103 are not sequential; they can be executed simultaneously or sequentially.
[0067] In this embodiment of the invention, the audio anomaly detection method can detect the image data and audio data of the video conferencing venue separately to determine whether the person corresponding to the target device is speaking and whether the target venue is in a silent state. This allows for a comprehensive analysis of whether the target device is abnormal, pinpointing the specific cause of the audio anomaly, and providing specific solutions in conjunction with the image data, thus making the audio anomaly detection highly efficient.
[0068] Optionally, the step of detecting human motion features based on the image data to obtain a first detection result includes:
[0069] Based on the image data, facial key point detection is performed to determine the key points of the mouth of the person corresponding to the target device;
[0070] Based on the key points, the opening and closing state of the mouth is detected to determine the character's speaking state;
[0071] Determine whether the person corresponding to the target device is in a speaking state.
[0072] Specifically, the facial key point detection can be based on the image data to detect key points of the person's mouth, and then detect the opening and closing state of the mouth to determine the person's speaking state.
[0073] In this embodiment of the invention, the audio anomaly detection method can detect whether the person is speaking by analyzing the image data, thereby obtaining the behavioral information of the participants in the video conference. This allows for a comprehensive analysis of the causes of audio anomalies by combining the audio data, resulting in higher efficiency in resolving audio anomaly problems in video conferences.
[0074] Optionally, the step of performing audio detection based on the audio data to determine the second detection result includes calculating the average amplitude of the audio data based on the audio data;
[0075] If the average amplitude is greater than or equal to a preset value, it is determined that the target location is not in a silent state;
[0076] If the average amplitude is less than a preset value, the target location is determined to be in a silent state.
[0077] Specifically, the average amplitude can be calculated by measuring the volume of the audio data. In this embodiment of the invention, the audio anomaly detection method can detect whether the target location is in a silent state by detecting the volume of the audio data, thereby combining image data to conduct a comprehensive analysis of the cause of audio anomalies, making the resolution of audio anomaly problems in video conferencing more efficient.
[0078] Optionally, determining whether the target device is abnormal based on the first detection result and the second detection result includes:
[0079] Based on the first detection result and the second detection result, it is determined whether the person corresponding to the target device is in a speaking state and whether the target location is in a silent state.
[0080] If the person is not speaking and the target location is in a silent state, the target device is determined to be normal.
[0081] When the person is not speaking and the target location is not in a silent state, noise detection is performed on the audio data to determine whether the target device is abnormal;
[0082] When the person is speaking and the target location is in a silent state, the distance between the person and the microphone is calculated based on the image data to determine whether there is any abnormality in the target device, where the microphone is the voice input terminal of the target device;
[0083] When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine whether the target device is abnormal.
[0084] Specifically, determining the speaking state S1 of the person corresponding to the target device and the silent state S2 of the target location involves four possibilities:
[0085] When S1 is 0 and S2 is 1, it is determined that the audio data is normal, that is, the target device is normal;
[0086] When S1 is 0 and S2 is 0, it is determined that there is obvious noise in the audio data, and further noise detection is performed on the audio data.
[0087] When S1 is 1 and S2 is 1, no valid audio data is collected, and the image data is further detected.
[0088] When S1 is 1 and S2 is 0, the audio data is currently normal. Further activity detection is performed on the audio data.
[0089] In this embodiment of the invention, the audio anomaly detection method can preliminarily determine whether the target device has an anomaly based on the first detection result and the second detection result, and further analyze the audio data and the image data to determine the specific cause of the audio anomaly, thereby making the audio anomaly detection highly efficient.
[0090] Optionally, when the person is not speaking and the target location is not in a silent state, performing noise detection on the audio data to determine whether the target device is abnormal includes:
[0091] If the person is not speaking and the target location is not muted, scene detection is performed on the target location to determine the scene of the target location;
[0092] Noise detection is performed based on the audio data to determine noise characteristics;
[0093] Determine whether the scene and the noise characteristics match to determine if the target device is abnormal.
[0094] Specifically, the scene detection can use a convolutional neural network algorithm to detect the meeting scene using a trained model. That is, image samples of different people and scenes are used as input and scene type is used as output to build a convolutional neural network model, thereby training a model that can detect the background scene type of people.
[0095] The noise detection can be achieved by extracting audio features using Mel-frequency cepstral coefficients (MFCC). MFCC is a set of feature vectors obtained by encoding the physical information of speech (spectral envelope and details). The audio feature vectors are then used as input and the corresponding noise type is used as output for training, thereby training a noise type detection model.
[0096] The step of determining whether the scene and the noise characteristics match, and whether the target device is abnormal, can be as follows: if the noise matches the scene, it is determined that the ambient noise of the current meeting is high but the target device is normal; if the noise does not match the scene, it is determined that the current noise is caused by the target device, and the user needs to check or replace the audio device.
[0097] In this embodiment of the invention, the audio anomaly detection method can accurately detect audio anomaly problems by performing noise detection on the audio data and determining whether the scene and the noise features match, thereby providing users with specific solutions and making the resolution of audio anomaly problems more efficient.
[0098] Optionally, when the person is speaking and the target location is in a silent state, calculating the distance between the person and the microphone based on the image data to determine whether the target device is abnormal includes:
[0099] When the person is speaking and the target location is in a silent state, the microphone position coordinates and the mouth position coordinates of the person are determined based on the image data.
[0100] Calculate the distance between the microphone and the mouth based on the microphone position coordinates and the mouth position coordinates;
[0101] If the distance is less than a preset distance, it is determined that the target device is malfunctioning.
[0102] If the distance is greater than or equal to a preset distance, the target device is determined to be normal.
[0103] Specifically, determining the microphone position coordinates can be done by detecting the microphone position in the image using a target detection algorithm and calculating the distance from the microphone to the person's mouth. The target detection algorithm can be a Haar feature extraction algorithm, R-CNN, or other algorithms. If the distance is greater than or equal to a preset distance, determining that the target device is normal can be done by initially assuming that the audio abnormality is caused by the user being too far away from the microphone, suggesting that the user move closer to the microphone, and ending the audio abnormality detection.
[0104] In this embodiment of the invention, the audio anomaly detection method can determine whether the distance between the user and the microphone is normal by calculating the distance between the microphone and the mouth, thereby analyzing the cause of the audio anomaly and making the accuracy of audio anomaly detection high.
[0105] Optionally, when the person is speaking and the target location is not muted, performing activity detection based on the audio data to determine whether the target device is abnormal includes:
[0106] When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine the probability of speech presence in the audio data.
[0107] If the probability of the voice being present is greater than a preset probability value, the target device is determined to be normal.
[0108] If the probability of the voice being present is less than or equal to a preset probability value, the distance between the microphone and the mouth is calculated to determine whether the target device is abnormal.
[0109] Specifically, the activity detection can be performed using the VAD algorithm; the probability of speech presence can be the proportion of human voice in the audio data; if the probability of speech presence is less than or equal to a preset probability value, it may indicate that the quality of the speech portion in the audio is poor, requiring calculation of the distance between the microphone and the mouth, and further analysis of the cause of the audio anomaly.
[0110] In this embodiment of the invention, the audio anomaly detection method can determine whether the speech quality in the audio is normal by performing activity detection on the audio data, thereby further analyzing the cause of the audio anomaly and making the accuracy of audio anomaly detection higher.
[0111] Please see Figure 2 , Figure 2 This is one of the functional block diagrams of an audio anomaly detection method provided in an embodiment of the present invention, such as... Figure 2 As shown, the audio anomaly detection method includes image detection, audio detection, and audio anomaly analysis and processing modules.
[0112] Figure 3 This is a second functional block diagram of an audio anomaly detection method provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the image detection module includes three sub-modules: speech detection, microphone recognition, and scene detection. Speech detection, also known as lip movement recognition, can detect key points of the mouth on the face through facial key point detection, and then detect the opening and closing state of the mouth and the specific location of the mouth. Microphone recognition also uses common object detection methods to identify whether there is a microphone in the current video image and the specific location of the microphone. Scene detection uses convolutional neural networks to detect the real scene in which the meeting participants are located based on the background information of the video image.
[0113] Figure 4 This is the third functional module diagram of an audio anomaly detection method provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the audio detection module contains three sub-modules: silence detection, speech activity detection, and noise detection. Silence detection checks whether the volume of the audio data is too low; speech activity detection checks whether there is human voice in the audio data; and noise detection includes calculating the volume of the noise and the time-frequency domain characteristics of the noise.
[0114] The audio anomaly analysis and processing module analyzes the detection results from both the image and audio detection modules. For example, if the image detects that a participant is speaking, but the audio detection shows no speech, an audio anomaly can be determined. This module uses pre-defined analysis and processing logic to analyze the image and audio detection results to obtain the audio anomaly detection result. Figure 5 This is a second flowchart of an audio anomaly detection method provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the logic of the audio anomaly analysis and processing module includes the following steps:
[0115] The first step is to perform speech detection on the image. If a meeting participant is detected speaking, proceed to step two; otherwise, proceed to step four.
[0116] The second step is to perform silence detection on the image and process the results accordingly:
[0117] If the current audio is detected to be muted, proceed to step three to calculate the microphone distance;
[0118] If the current audio is not detected to be silent, then a speech activity detection is performed; if the speech activity detection detects speech, then the current audio is considered normal; otherwise, proceed to step three.
[0119] The third step is to calculate the distance between the microphone and the mouth. If the distance is too far, it is recommended that the user move closer to the microphone and end the test. Otherwise, it is considered that the audio equipment is abnormal and the user needs to check or replace the audio equipment.
[0120] The fourth step is to perform a silence detection on the image. If the current audio is detected to be silent, it is considered that the current audio is normal; otherwise, proceed to the fifth step to further analyze the current noise.
[0121] The fifth step involves performing scene detection and noise detection on the video images and audio data respectively. Then, based on the detection results, including scene type and noise characteristics, a comprehensive analysis is conducted to determine whether the noise matches the scene. If the noise matches the scene, it is determined that the ambient noise of the current meeting is high but the audio equipment is working properly. If the noise does not match the scene, it is determined that the current noise is caused by the audio equipment, and the user needs to check or replace the audio equipment.
[0122] Please see Figure 6 , Figure 6 The third flowchart of an audio anomaly detection method provided in this embodiment of the invention is as follows: Figure 6 As shown:
[0123] The first step is to acquire images of the video conference over a period of time for speech detection. Facial key point detection is used to obtain key points of the mouth, and then the opening and closing state of the mouth is detected to obtain the speech state S1 and the mouth position coordinates P1(x1,y1).
[0124] The second step is to acquire the audio data of the video conference within a certain period of time for silence detection. The silence state S2 is detected by judging whether the average audio amplitude is less than a certain value. Based on the results of step 1, there are 4 possible scenarios:
[0125] When S1 is 0 and S2 is 1, it indicates that the current audio data is normal and the audio anomaly detection is exited.
[0126] When S1 is 0 and S2 is 0, it is determined that there is obvious noise in the current audio data, and step five is executed to further analyze the noise information.
[0127] When S1 is 1 and S2 is 0, the audio data is currently normal. Step 3 will be executed to further test the audio data.
[0128] When S1 is 1 and S2 is 1, no valid audio data is collected. Step four is then performed to further detect the image.
[0129] The third step involves performing speech activity detection on the audio data to determine the probability of speech presence. If the probability is greater than a certain value, the audio data is considered normal, and the audio anomaly detection process is terminated. If the probability of speech presence is less than a certain value, the speech portion of the audio is considered to be of poor quality, and step four is required to further pinpoint the cause.
[0130] The fourth step is to detect the microphone position P2(x2,y2) in the image using the object detection algorithm and calculate the distance from microphone P2 to the speaker's mouth P1. If the distance is less than a certain value, the audio device is determined to be abnormal. If the distance is greater than a certain value, it is initially considered that the distance between the user and the microphone is too far. The user is advised to move closer to the microphone, and the audio abnormality detection is terminated.
[0131] The fifth step is to detect the scene type of the people and background in the image. This can be done using a convolutional neural network algorithm, which utilizes a trained model to detect the meeting scene.
[0132] The sixth step is to perform noise detection on the audio data. Based on the detected noise characteristics and the meeting scene detected in step five, a comprehensive analysis is conducted to determine whether the noise is the original noise of the meeting scene or noise caused by a malfunction of the audio equipment.
[0133] This invention provides an audio anomaly detection device 700, such as... Figure 7 As shown, the device includes:
[0134] The acquisition module 701 is used to acquire image data and audio data of a target location, wherein the target location is a video conferencing location, and the image data and audio data are image data and audio data corresponding to the target device in the video conferencing location;
[0135] The first detection module 702 is used to perform human motion feature detection based on the image data to obtain a first detection result, wherein the first detection result is used to indicate whether the person corresponding to the target device is in a speaking state.
[0136] The second detection module 703 is used to perform audio detection based on the audio data and obtain a second detection result, which is used to indicate whether the target location is in a silent state.
[0137] The determination module 704 is used to determine whether the target device has an anomaly based on the first detection result and the second detection result.
[0138] Optionally, the first detection module 702 includes:
[0139] The first determining unit is used to perform facial key point detection based on the image data to determine the key points of the mouth of the person corresponding to the target device;
[0140] The second determining unit is used to determine the speaking state of the character based on the detection of the mouth opening and closing state at the key points.
[0141] The third determining unit is used to determine whether the person corresponding to the target device is in a speaking state.
[0142] Optionally, the second detection module 703 includes:
[0143] A calculation unit is used to calculate the average amplitude of the audio data based on the audio data;
[0144] The fourth determining unit is used to determine that the target location is not in a silent state when the average amplitude is greater than or equal to a preset value; and to determine that the target location is in a silent state when the average amplitude is less than a preset value.
[0145] Optionally, the determining module 704 includes:
[0146] The fifth determining unit is used to determine, based on the first detection result and the second detection result, whether the person corresponding to the target device is in a speaking state and whether the target location is in a silent state.
[0147] If the person is not speaking and the target location is in a silent state, the target device is determined to be normal.
[0148] If the person is not speaking and the target location is not in a silent state, 5 performs noise detection on the audio data to determine if the target device is abnormal;
[0149] When the person is speaking and the target location is in a silent state, the distance between the person and the microphone is calculated based on the image data to determine whether there is any abnormality in the target device, where the microphone is the voice input terminal of the target device;
[0150] When the person is speaking and the target location is not muted, activity detection is performed based on the audio data to determine whether the target device is abnormal.
[0151] Optionally, the fifth determining unit is configured to: perform scene detection on the target location and determine the scene of the target location when the person is not speaking and the target location is not in a silent state;
[0152] Noise detection is performed based on the audio data to determine noise characteristics;
[0153] 5. And determine whether the scene and the noise characteristics match, and determine whether the target device is abnormal.
[0154] Optionally, the fifth determining unit is used to:
[0155] When the person is speaking and the target location is in a silent state, the microphone position coordinates and the mouth position coordinates of the person are determined based on the image data; the distance between the microphone and the mouth is calculated based on the microphone position coordinates and the mouth position coordinates.
[0156] If the distance is less than a preset distance, it is determined that the target device is malfunctioning.
[0157] If the distance is greater than or equal to a preset distance, the target device is determined to be normal.
[0158] Optionally, the fifth determining unit is used to:
[0159] When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine the probability of speech presence in the audio data.
[0160] If the probability of the voice being present is greater than a preset probability value, the target device is determined to be normal.
[0161] If the probability of the voice being present is less than or equal to a preset probability value, the distance between the microphone and the mouth is calculated to determine whether the target device is abnormal.
[0162] This invention provides an audio anomaly detection device, such as... Figure 8 As shown, the resource scheduling device includes: a transceiver 801, a memory 802, a processor 800, and a program stored in the memory and executable on the processor.
[0163] The transceiver 801 is used to acquire image data and audio data of a target location, wherein the target location is a video conferencing location, and the image data and audio data are the image data and audio data corresponding to the target device in the video conferencing location;
[0164] Processor 800 reads the program from memory 802 and performs the following steps:
[0165] Based on the image data, human motion features are detected to obtain a first detection result, which is used to indicate whether the person corresponding to the target device is in a speaking state.
[0166] Based on the audio data, audio detection is performed to obtain a second detection result, which is used to indicate whether the target location is in a silent state.
[0167] Based on the first detection result and the second detection result, it is determined whether the target device is abnormal.
[0168] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 800) and memory (memory 802). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 801 can be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. The processor 800 is responsible for managing the bus architecture and general processing, and the memory 802 can store data used by the processor 800 during operation.
[0169] Optionally, the processor 800 is further configured to:
[0170] Based on the image data, facial key point detection is performed to determine the key points of the mouth of the person corresponding to the target device;
[0171] Based on the key points, the opening and closing state of the mouth is detected to determine the character's speaking state;
[0172] Determine whether the person corresponding to the target device is in a speaking state.
[0173] Optionally, the processor 800 is further configured to:
[0174] Based on the audio data, calculate the average amplitude of the audio data;
[0175] If the average amplitude is greater than or equal to a preset value, it is determined that the target location is not in a silent state;
[0176] If the average amplitude is less than a preset value, the target location is determined to be in a silent state.
[0177] Optionally, the processor 800 is further configured to:
[0178] Based on the first detection result and the second detection result, it is determined whether the person corresponding to the target device is in a speaking state and whether the target location is in a silent state.
[0179] If the person is not speaking and the target location is in a silent state, the target device is determined to be normal.
[0180] When the person is not speaking and the target location is not in a silent state, noise detection is performed on the audio data to determine whether the target device is abnormal;
[0181] When the person is speaking and the target location is in a silent state, the distance between the person and the microphone is calculated based on the image data to determine whether there is any abnormality in the target device, where the microphone is the voice input terminal of the target device;
[0182] When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine whether the target device is abnormal.
[0183] Optionally, the processor 800 is further configured to:
[0184] If the person is not speaking and the target location is not muted, scene detection is performed on the target location to determine the scene of the target location;
[0185] Noise detection is performed based on the audio data to determine noise characteristics;
[0186] Determine whether the scene and the noise characteristics match to determine if the target device is abnormal.
[0187] Optionally, the processor 800 is further configured to:
[0188] When the person is speaking and the target location is in a silent state, the microphone position coordinates and the mouth position coordinates of the person are determined based on the image data.
[0189] Calculate the distance between the microphone and the mouth based on the microphone position coordinates and the mouth position coordinates;
[0190] If the distance is less than a preset distance, it is determined that the target device is malfunctioning.
[0191] If the distance is greater than or equal to a preset distance, the target device is determined to be normal.
[0192] Optionally, the processor 800 is further configured to:
[0193] When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine the probability of speech presence in the audio data.
[0194] If the probability of the voice being present is greater than a preset probability value, the target device is determined to be normal.
[0195] If the probability of the voice being present is less than or equal to a preset probability value, the distance between the microphone and the mouth is calculated to determine whether the target device is abnormal.
[0196] This invention also provides a readable storage medium for storing a program that can be executed by a processor to implement various processes of the embodiments of the above-described method. The implementation principles and technical effects are similar, and to avoid repetition, this embodiment will not be described again here.
[0197] In the several embodiments provided by this invention, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.
[0198] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.
[0199] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0200] The above description is a preferred embodiment of the present invention. It should be noted that the scope of protection of the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An audio anomaly detection method, characterized by, include: Acquire image and audio data of a target location, wherein the target location is a video conferencing location, and the image and audio data are the image and audio data corresponding to the target device in the video conferencing location; Based on the image data, human motion features are detected to obtain a first detection result, which is used to indicate whether the person corresponding to the target device is in a speaking state. Based on the audio data, audio detection is performed to obtain a second detection result, which is used to indicate whether the target location is in a silent state. Based on the first detection result and the second detection result, determine whether the target device is abnormal; The step of determining whether the target device is abnormal based on the first detection result and the second detection result includes: Based on the first detection result and the second detection result, it is determined whether the person corresponding to the target device is in a speaking state and whether the target location is in a silent state. If the person is not speaking and the target location is in a silent state, the target device is determined to be normal. When the person is not speaking and the target location is not in a silent state, noise detection is performed on the audio data to determine whether the target device is abnormal; When the person is speaking and the target location is in a silent state, the distance between the person and the microphone is calculated based on the image data to determine whether there is any abnormality in the target device, where the microphone is the voice input terminal of the target device; When the person is speaking and the target location is not in a silent state, activity detection is performed based on the audio data to determine whether there is any abnormality in the target device; The step of performing noise detection on the audio data to determine whether the target device is malfunctioning, when the person is not speaking and the target location is not silent, includes: If the person is not speaking and the target location is not muted, scene detection is performed on the target location to determine the scene of the target location; Noise detection is performed based on the audio data to determine noise characteristics; Determine whether the scene and the noise characteristics match to determine if the target device is abnormal; If the scene and the noise characteristics match, the target device is determined to be normal; if the scene and the noise characteristics do not match, the noise is determined to be caused by the target device, and the user needs to check or replace the audio device.
2. The method of claim 1, wherein, The step of detecting human motion features based on the image data to obtain a first detection result includes: Based on the image data, facial key point detection is performed to determine the key points of the mouth of the person corresponding to the target device; Based on the key points, the opening and closing state of the mouth is detected to determine the character's speaking state; Determine whether the person corresponding to the target device is in a speaking state.
3. The method according to claim 1, characterized in that, The step of performing audio detection based on the audio data to determine the second detection result includes: Based on the audio data, calculate the average amplitude of the audio data; If the average amplitude is greater than or equal to a preset value, it is determined that the target location is not in a silent state; If the average amplitude is less than a preset value, the target location is determined to be in a silent state.
4. The method according to claim 1, characterized in that, When the person is speaking and the target location is silent, calculating the distance between the person and the microphone based on the image data to determine if the target device is malfunctioning includes: When the person is speaking and the target location is in a silent state, the microphone position coordinates and the mouth position coordinates of the person are determined based on the image data. Calculate the distance between the microphone and the mouth based on the microphone position coordinates and the mouth position coordinates; If the distance is less than a preset distance, it is determined that the target device is malfunctioning. If the distance is greater than or equal to a preset distance, the target device is determined to be normal.
5. The method according to claim 1, characterized in that, When the person is speaking and the target location is not muted, the activity detection based on the audio data is performed to determine whether the target device is abnormal, including: When the person is speaking and the target location is not silent, activity detection is performed based on the audio data to determine the probability of speech presence in the audio data. If the probability of the voice being present is greater than a preset probability value, the target device is determined to be normal. If the probability of the voice being present is less than or equal to a preset probability value, the distance between the microphone and the mouth is calculated to determine whether the target device is abnormal.
6. An audio anomaly detection device, characterized in that, The device includes: The acquisition module is used to acquire image data and audio data of a target location, wherein the target location is a video conferencing location, and the image data and audio data are image data and audio data corresponding to the target device in the video conferencing location; The first detection module is used to perform human motion feature detection based on the image data to obtain a first detection result, which is used to indicate whether the person corresponding to the target device is in a speaking state. The second detection module is used to perform audio detection based on the audio data and obtain a second detection result, which is used to indicate whether the target location is in a silent state. The determination module is used to determine whether the target device has an anomaly based on the first detection result and the second detection result; The determining module includes: The fifth determining unit is used to determine, based on the first detection result and the second detection result, whether the person corresponding to the target device is in a speaking state and whether the target location is in a silent state. If the person is not speaking and the target location is in a silent state, the target device is determined to be normal. When the person is not speaking and the target location is not in a silent state, noise detection is performed on the audio data to determine whether the target device is abnormal; When the person is speaking and the target location is in a silent state, the distance between the person and the microphone is calculated based on the image data to determine whether there is any abnormality in the target device, where the microphone is the voice input terminal of the target device; When the person is speaking and the target location is not in a silent state, activity detection is performed based on the audio data to determine whether there is any abnormality in the target device; The fifth determining unit is used to: perform scene detection on the target location and determine the scene of the target location when the person is not speaking and the target location is not in a silent state; Noise detection is performed based on the audio data to determine noise characteristics; And determine whether the scene and the noise features match, and determine whether the target device is abnormal; If the scene and the noise characteristics match, the target device is determined to be normal; if the scene and the noise characteristics do not match, the noise is determined to be caused by the target device, and the user needs to check or replace the audio device.
7. An audio anomaly detection device, the device comprising: A transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that, The processor is configured to read a program from memory to implement the steps of the method as described in any one of claims 1 to 5.
8. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Abnormal event monitoring device and method combining videos and audios
CN111091073A
Conference audio and video automatic detection method and system
CN111866497A
Audio signal abnormity monitoring method, device, equipment, medium and program product
CN114627897A
