Acoustic event detection method, device, equipment and storage medium

Abstract

The embodiments of the present application provide an acoustic event detection method, apparatus, device and storage medium to improve the accuracy of acoustic event detection. The method includes: obtaining audio data to be detected; extracting the first acoustic features of a target audio segment from the audio data to be detected, wherein the target audio segment includes multiple audio frames; determining the probability information of the first acoustic features of multiple batches corresponding to acoustic events; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; determining the probability information of the target acoustic event in the target audio segment based on the probability information of the first acoustic features of the multiple batches corresponding to the acoustic event; and then the embodiments of the present application improve the accuracy of sound event detection by utilizing the time sequence information of multiple audio frames contained in the audio data to be detected to detect acoustic events.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and device, an electronic device, and a storage medium. Background Art

[0002] Acoustic event detection is the process of detecting specific acoustic events from sound signals. For example, given a piece of speech data, it can identify gunshots, shouts, instrument percussion, whistles, etc.

[0003] With the continuous development of multimedia and network technologies, the application of acoustic event detection is becoming more and more extensive. For example, it can detect voice clips containing terrorism and violence from massive amounts of voice data. Another example is to locate the highlights of live sports games by detecting audience cheers, applause, whistles, etc.

[0004] Traditional acoustic event detection methods typically focus on feature extraction and modeling. Their drawbacks include poor robustness to noise and channel interference, and a significant reduction in event detection accuracy in the presence of noise, particularly high levels of noise. In recent years, deep learning techniques, such as CNN (Convolutional Neural Networks) and CNN-RNN (Recurrent Neural Networks), have also been applied to acoustic event detection. However, these approaches still suffer from low event detection accuracy. Summary of the Invention

[0005] The embodiments of the present application provide an acoustic event detection method to improve the accuracy of acoustic event detection.

[0006] Correspondingly, the embodiments of the present application also provide an acoustic event detection device, an electronic device, and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, an embodiment of the present application discloses an acoustic event detection method, which includes: obtaining audio data to be detected; extracting the first acoustic features of a target audio segment from the audio data to be detected, wherein the target audio segment includes multiple audio frames; determining the probability information of the first acoustic features of multiple batches corresponding to acoustic events; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; and determining the probability information of the target acoustic event in the target audio segment based on the probability information of the first acoustic features of the multiple batches corresponding to the acoustic events.

[0008] Optionally, extracting the first acoustic feature of the target audio segment from the audio data to be detected includes: extracting the first acoustic feature of the audio data to be detected; using a window with a preset time length to slide on the first acoustic feature of the audio data to be detected according to a set sliding step size, and intercepting the first acoustic feature of the target audio segment.

[0009] Optionally, determining the probability information that the first acoustic features of multiple batches correspond to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting the probability information that the first acoustic features of multiple batches correspond to acoustic events.

[0010] Optionally, the acoustic event detector outputs the probability information of the first acoustic features of the multiple batches corresponding to the acoustic events, including: the acoustic event detector divides the first acoustic features of the multiple audio frames into multiple batches of first acoustic features according to a preset timing relationship, and determines and outputs the probability information of the first acoustic features of the multiple batches corresponding to the acoustic events.

[0011] Optionally, dividing the first acoustic features of the multiple audio frames into multiple batches of first acoustic features according to a preset timing relationship includes: dividing the first acoustic features of the multiple audio frames into multiple batches of first acoustic features by performing time pooling on the first acoustic features of the multiple audio frames according to a preset timing relationship.

[0012] Optionally, inputting the first acoustic feature into the acoustic event detector includes: determining second acoustic features corresponding to the multiple audio frames respectively based on the first acoustic features of the multiple audio frames; and inputting the second acoustic feature into the acoustic event detector.

[0013] Optionally, the second acoustic features corresponding to the multiple audio frames are determined based on the first acoustic features of the multiple audio frames, including: selecting the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames; splicing the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain the third acoustic feature corresponding to the target audio frame; and determining the second acoustic feature corresponding to the target video frame based on the third acoustic feature corresponding to the target audio frame.

[0014] Optionally, the determining of the second acoustic feature corresponding to the target video frame based on the third acoustic feature corresponding to the target audio frame includes: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame; and / or performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame; and / or regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0015] Optionally, the probability information of the acoustic event includes the probability information of the target acoustic event, and determining the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches includes: averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment; or, selecting the first acoustic features of multiple target batches from the first acoustic features of the multiple batches, averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtaining the probability information of the target acoustic event in the target audio segment; or, score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches after score normalization; or, determining the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches using a discriminative method.

[0016] Optionally, the acoustic event detector includes multiple first preset networks, time pooling layers, second preset networks and third preset networks; the first preset network includes: a fully connected layer, a rectified linear unit, a random dropout layer and a feedforward sequence memory network; the second preset network includes: a fully connected layer, a rectified linear unit and a random dropout layer, and the third preset network includes: a fully connected layer, a rectified linear unit and an activation function.

[0017] An embodiment of the present application also provides a training method for an acoustic event detector, the method comprising: collecting audio data of multiple target acoustic events and audio data of multiple other acoustic events; for the audio data of a target acoustic event, dividing the audio data of the target acoustic event into multiple audio segments; and using the multiple audio segments as positive samples and the audio data of multiple other acoustic events as negative samples to train the acoustic event detector.

[0018] Optionally, in the process of training the acoustic event detector, binary cross entropy is used as a loss function.

[0019] Optionally, the positive examples are not less than the negative examples.

[0020] An embodiment of the present application also provides a monitoring method based on acoustic event detection, which is applied to a monitoring system, wherein the monitoring system includes an audio acquisition device, and the method includes: obtaining audio data collected by the audio acquisition device; extracting the first acoustic features of a target audio segment from the audio data, wherein the target audio segment includes multiple audio frames; determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; determining the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches; and generating prompt information based on the probability information of the target acoustic event in the target audio segment.

[0021] An embodiment of the present application also provides a monitoring method based on acoustic event detection, which is applied to a monitoring system, wherein the monitoring system includes an audio acquisition device and an image acquisition device, and the method includes: obtaining audio data collected by the audio acquisition device; extracting the first acoustic feature of a target audio segment from the audio data, wherein the target audio segment includes multiple audio frames; determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; determining the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches; obtaining image data collected by the image acquisition device when determining that an abnormality has occurred based on the probability information of the target acoustic event in the target audio segment; confirming the abnormality based on the target image segment corresponding to the target audio segment in the image data; and generating prompt information when confirming that an abnormality has occurred based on the target image segment.

[0022] An embodiment of the present application also provides a monitoring method based on acoustic event detection, which is applied to a vehicle-mounted monitoring system, wherein the vehicle-mounted monitoring system includes an audio acquisition device, an image acquisition device, and a vehicle information acquisition device, and the method includes: obtaining audio data collected by the audio acquisition device; extracting a first acoustic feature of a target audio segment from the audio data, wherein the target audio segment includes multiple audio frames; determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch include first acoustic features of multiple audio frames with a preset time sequence relationship; determining probability information of a target acoustic event in the target audio segment based on the probability information of acoustic events corresponding to the first acoustic features of the multiple batches; when determining that an abnormality occurs based on the probability information of the target acoustic event in the target audio segment, obtaining image data collected by the image acquisition device and vehicle information collected by the vehicle information acquisition device; determining the abnormality type of the abnormality based on the image data and vehicle information; the abnormality type includes an internal vehicle abnormality type or an external vehicle abnormality type; and generating corresponding prompt information based on the abnormality type of the abnormality.

[0023] An embodiment of the present application also provides a method for identifying illegal clips based on acoustic event detection, the method comprising: obtaining audio data to be detected; extracting the first acoustic features of a target audio clip from the audio data to be detected, the target audio clip comprising multiple audio frames; determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch comprise the first acoustic features of multiple audio frames having a preset time sequence relationship; determining probability information of a target acoustic event in the target audio clip based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; and judging whether the target audio clip is an illegal clip based on the probability information of the illegal acoustic event in the target audio clip.

[0024] An embodiment of the present application also provides a segment search method based on acoustic event detection, the method comprising: receiving query information; determining the audio data to be queried and the acoustic event to be queried corresponding to the query information; extracting the first acoustic feature of the target audio segment from the audio data to be queried, the target audio segment including multiple audio frames; determining probability information of the first acoustic features of multiple batches corresponding to the acoustic event; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; based on the probability information of the first acoustic features of the multiple batches corresponding to the acoustic event, determining the probability information of the acoustic event to be queried in the target audio segment; based on the probability information of the acoustic event to be queried in the target audio segment, returning the target audio segment.

[0025] An embodiment of the present application also provides an anomaly analysis method based on acoustic event detection, the method comprising: obtaining audio data collected by an audio acquisition device; extracting a first acoustic feature of a target audio segment from the audio data, the target audio segment comprising multiple audio frames; determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch comprise first acoustic features of multiple audio frames having a preset time sequence relationship; determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; determining location information corresponding to the target acoustic event when an anomaly occurs based on the probability information of the target acoustic event in the target audio segment; calling other information acquisition devices corresponding to the location information to collect environmental information, and performing anomaly analysis based on the environmental information.

[0026] An embodiment of the present application also provides an acoustic event detection device, which includes: a first data acquisition module for acquiring audio data to be detected; a first extraction module for extracting first acoustic features of a target audio segment from the audio data to be detected, wherein the target audio segment includes multiple audio frames; a first detection module for determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch include first acoustic features of multiple audio frames with a preset time sequence relationship; and a first probability determination module for determining probability information of a target acoustic event in the target audio segment based on the probability information of acoustic events corresponding to the first acoustic features of the multiple batches.

[0027] Optionally, the first extraction module is used to extract the first acoustic feature of the audio data to be detected; a window with a preset length is used to slide on the first acoustic feature of the audio data to be detected according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0028] Optionally, the first detection module includes: an input submodule for inputting the first acoustic feature into an acoustic event detector, and a probability detection submodule for calling the acoustic event detector to output probability information of acoustic events corresponding to multiple batches of the first acoustic features.

[0029] Optionally, the probability detection submodule is used to call the acoustic event detector to divide the first acoustic features of the multiple audio frames into multiple batches of first acoustic features according to a preset time sequence relationship, and to determine and output the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches respectively.

[0030] Optionally, the probability detection submodule is used to divide the first acoustic features of the multiple audio frames into multiple batches of first acoustic features by performing time pooling on the first acoustic features of the multiple audio frames according to a preset time sequence relationship.

[0031] Optionally, the input submodule includes: a feature determination unit for determining second acoustic features corresponding to the multiple audio frames based on the first acoustic features of the multiple audio frames; and a feature input unit for inputting the second acoustic features into the acoustic event detector.

[0032] Optionally, the feature determination unit is used to select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames; splice the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame; and determine the second acoustic feature corresponding to the target video frame based on the third acoustic feature corresponding to the target audio frame.

[0033] Optionally, the feature determination unit is used to determine the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame; and / or perform differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; use the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame; and / or regularize the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and use the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0034] Optionally, the probability information of the acoustic event includes the probability information of the target acoustic event, and the first probability determination module is used to average the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment; or, select the first acoustic features of multiple target batches from the first acoustic features of the multiple batches, average the probability information of the target acoustic event corresponding to the first acoustic features of the multiple target batches, and obtain the probability information of the target acoustic event in the target audio segment; or, score-normalize the probability information of the acoustic event corresponding to the first acoustic features of the multiple batches, and determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic event corresponding to the first acoustic features of the multiple batches after score normalization; or, use an identification device to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic event corresponding to the first acoustic features of the multiple batches.

[0035] Optionally, the acoustic event detector includes multiple first preset networks, time pooling layers, second preset networks and third preset networks; the first preset network includes: a fully connected layer, a rectified linear unit, a random dropout layer and a feedforward sequence memory network; the second preset network includes: a fully connected layer, a rectified linear unit and a random dropout layer, and the third preset network includes: a fully connected layer, a rectified linear unit and an activation function.

[0036] An embodiment of the present application also provides a training device for an acoustic event detector, which includes: a data collection module for collecting audio data of multiple target acoustic events and audio data of multiple other acoustic events; a data expansion module for dividing the audio data of a target acoustic event into multiple audio segments; and a training module for training the acoustic event detector using the multiple audio segments as positive samples and the audio data of multiple other acoustic events as negative samples.

[0037] Optionally, in the process of training the acoustic event detector, binary cross entropy is used as a loss function.

[0038] Optionally, the positive examples are not less than the negative examples.

[0039] An embodiment of the present application also provides a monitoring device based on acoustic event detection, which is applied to a monitoring system, wherein the monitoring system includes an audio acquisition device, and the device includes: a second data acquisition module, used to acquire audio data collected by the audio acquisition device; a second extraction module, used to extract the first acoustic features of a target audio segment from the audio data, wherein the target audio segment includes multiple audio frames; a second detection module, used to determine probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; a second probability determination module, used to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the first acoustic features of the multiple batches corresponding to the acoustic event; and a first prompt module, used to generate prompt information based on the probability information of the target acoustic event in the target audio segment.

[0040] The present application also provides a monitoring device based on acoustic event detection, which is applied to a monitoring system. The monitoring system includes an audio acquisition device and an image acquisition device. The device includes:

[0041] a third data acquisition module for acquiring audio data collected by the audio acquisition device; a third extraction module for extracting the first acoustic features of a target audio segment from the audio data, wherein the target audio segment includes multiple audio frames; a third detection module for determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; a third probability determination module for determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a fourth data acquisition module for acquiring image data collected by the image acquisition device when determining that an abnormality has occurred based on the probability information of the target acoustic event in the target audio segment; an abnormality confirmation module for confirming an abnormality based on a target image segment corresponding to the target audio segment in the image data; and a second prompt module for generating prompt information when confirming that an abnormality has occurred based on the target image segment.

[0042] An embodiment of the present application further provides a monitoring device based on acoustic event detection, which is applied to an in-vehicle monitoring system, wherein the in-vehicle monitoring system includes an audio acquisition device, an image acquisition device, and a vehicle information acquisition device. The device includes: a fifth data acquisition module, configured to acquire audio data acquired by the audio acquisition device; a fourth extraction module, configured to extract first acoustic features of a target audio segment from the audio data, wherein the target audio segment includes multiple audio frames; and a fourth detection module, configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship. A fourth probability determination module is used to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic event corresponding to the first acoustic features of the multiple batches; a sixth data acquisition module is used to obtain the image data collected by the image acquisition device and the vehicle information collected by the vehicle information acquisition device when determining the occurrence of an abnormality based on the probability information of the target acoustic event in the target audio segment; an abnormality type determination module is used to determine the abnormality type of the abnormality based on the image data and the vehicle information; the abnormality type includes an internal vehicle abnormality type or an external vehicle abnormality type; a third prompt module is used to generate corresponding prompt information based on the abnormality type of the abnormality.

[0043] An embodiment of the present application also provides a device for identifying illegal clips based on acoustic event detection, the device comprising: a seventh data acquisition module for acquiring audio data to be detected; a fifth extraction module for extracting the first acoustic features of a target audio clip from the audio data to be detected, the target audio clip comprising multiple audio frames; a fifth detection module for determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch comprise the first acoustic features of multiple audio frames having a preset time sequence relationship; a fifth probability determination module for determining probability information of a target acoustic event in the target audio clip based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; and a violation judgment module for judging whether the target audio clip is an illegal clip based on the probability information of the illegal acoustic event in the target audio clip.

[0044] An embodiment of the present application also provides a fragment search device based on acoustic event detection, the device comprising: a receiving module for receiving query information; a data determination module for determining the audio data to be queried and the acoustic event to be queried corresponding to the query information; a sixth extraction module for extracting the first acoustic feature of the target audio fragment from the audio data to be queried, the target audio fragment comprising multiple audio frames; a sixth detection module for determining probability information of the first acoustic features of multiple batches corresponding to the acoustic event; wherein the first acoustic features of a batch comprise the first acoustic features of multiple audio frames having a preset time sequence relationship; a sixth probability determination module for determining the probability information of the acoustic event to be queried in the target audio fragment based on the probability information of the first acoustic features of the multiple batches corresponding to the acoustic event; and a return module for returning the target audio fragment based on the probability information of the acoustic event to be queried in the target audio fragment.

[0045] An embodiment of the present application also provides an anomaly analysis device based on acoustic event detection, the device comprising: an eighth data acquisition module for acquiring audio data collected by an audio collection device; a seventh extraction module for extracting the first acoustic features of a target audio segment from the audio data, the target audio segment comprising multiple audio frames; a seventh detection module for determining probability information of acoustic events corresponding to the first acoustic features of multiple batches; wherein the first acoustic features of a batch comprise the first acoustic features of multiple audio frames having a preset time sequence relationship; a seventh probability determination module for determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a position determination module for determining position information corresponding to the target acoustic event when an anomaly occurs based on the probability information of the target acoustic event in the target audio segment; and an analysis module for calling other information collection devices corresponding to the position information to collect environmental information, and performing anomaly analysis based on the environmental information.

[0046] An embodiment of the present application further discloses an electronic device, comprising: a processor; and a memory on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0047] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0048] Compared with the prior art, the embodiments of the present application have the following advantages:

[0049] In an embodiment of the present application, after obtaining the audio data to be detected, the first acoustic features of the target audio segment can be extracted from the audio data to be detected, and the probability information of the acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of the acoustic events corresponding to the multiple batches, the probability information of the target acoustic events in the target audio segment can be determined; wherein, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset timing relationship, that is, the embodiment of the present application utilizes the timing information of multiple audio frames contained in the audio data to be detected to perform acoustic event detection; thereby, the accuracy of sound event detection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1A This is a schematic diagram of acoustic event detection data processing according to an embodiment of the present application;

[0051] Figure 1B is a flowchart of the steps of an embodiment of an acoustic event detection method of the present application;

[0052] Figure 1C is another schematic diagram of acoustic event detection data processing according to an embodiment of the present application;

[0053] Figure 1D is a schematic structural diagram of an acoustic event detector according to an embodiment of the present application;

[0054] Figure 2 is a flowchart of steps of an embodiment of a training method for an acoustic event detector of the present application;

[0055] Figure 3 This is a flowchart of the steps of an embodiment of a monitoring method based on acoustic event detection of the present application;

[0056] Figure 4 is a flowchart of another embodiment of a monitoring method based on acoustic event detection of the present application;

[0057] Figure 5 This is a flowchart of another embodiment of a monitoring method based on acoustic event detection of the present application;

[0058] Figure 6 This is a flowchart of an embodiment of a method for identifying illegal segments based on acoustic event detection in the present application;

[0059] Figure 7 This is a flowchart of an embodiment of a segment search method based on acoustic event detection of the present application;

[0060] Figure 8 This is a flowchart of an embodiment of a method for adding a marker based on acoustic event detection of the present application;

[0061] Figure 9 This is a flowchart of an embodiment of an abnormality analysis method based on acoustic event detection of the present application;

[0062] Figure 10 A structural block diagram of an embodiment of an acoustic event detection device of the present application;

[0063] Figure 11 A structural block diagram of an optional embodiment of an acoustic event detection device of the present application;

[0064] Figure 12 is a structural block diagram of a training embodiment of an acoustic event detector of the present application;

[0065] Figure 13 This is a structural block diagram of an embodiment of a monitoring device based on acoustic event detection of the present application;

[0066] Figure 14 is a structural block diagram of another embodiment of a monitoring device based on acoustic event detection of the present application;

[0067] Figure 15 This is a structural block diagram of another embodiment of a monitoring device based on acoustic event detection of the present application;

[0068] Figure 16 This is a structural block diagram of an embodiment of a device for identifying illegal segments based on acoustic event detection according to the present application;

[0069] Figure 17 This is a structural block diagram of an embodiment of a segment search device based on acoustic event detection of the present application;

[0070] Figure 18 This is a structural block diagram of an embodiment of an abnormality analysis device based on acoustic event detection of the present application;

[0071] Figure 19 It is a structural diagram of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0073] The acoustic event detection method provided in the embodiment of the present application can detect various acoustic events, such as horns, collisions, explosions, cries for help, fighting, quarrels, cheers, applause, whistles, etc., and the embodiment of the present application does not limit this.

[0074] The embodiments of the present application can detect acoustic events in online audio data in real time, and can also detect acoustic events in offline audio data, and the embodiments of the present application are not limited to this.

[0075] The embodiment of the present application can use the time sequence information of multiple audio frames contained in the audio data to be detected to detect acoustic events, so as to improve the accuracy of sound event detection. Figure 1A , showing a schematic diagram of an acoustic event detection data processing process of the present application. After obtaining the audio data to be detected, the first acoustic features of the target audio segment can be extracted from the audio data to be detected, and the probability information of the first acoustic features of multiple batches corresponding to the acoustic events can be determined, wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship; then based on the probability information of the first acoustic features of each batch corresponding to the acoustic event output by the acoustic detector, the acoustic event detection result of the target audio segment is determined, and the detection result is the probability information of the target acoustic event in the target audio segment.

[0076] Reference Figure 1B , shows a step flow chart of an embodiment of an acoustic event detection method of the present application.

[0077] Step 102: Acquire audio data to be detected.

[0078] In an embodiment of the present application, when acoustic event detection needs to be performed on certain audio data, the audio data can be acquired, and the audio data that needs to be subjected to acoustic event detection can be determined as audio data to be detected.

[0079] The audio data to be detected may include multiple audio frames.

[0080] Step 104: Extract a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes multiple audio frames.

[0081] Among them, one method may be to first perform feature extraction on the audio data to be detected to extract the first acoustic feature of the audio data to be detected, and then extract the first acoustic feature of the target audio segment from the first acoustic feature of the audio data to be detected.

[0082] Correspondingly, step 104 may include the following sub-steps S1042 to S1044:

[0083] Sub-step S1042: extracting the first acoustic feature of the audio data to be detected.

[0084] Sub-step S1044: Use a window with a preset time length to slide on the first acoustic feature of the audio data to be detected according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0085] Feature extraction can be performed on each audio frame in the audio data to be detected, respectively, to extract the first acoustic features of each audio frame in the detected audio data. The first acoustic features of each audio frame can include multiple dimensions, and the number of dimensions of the first acoustic features corresponding to each audio frame can be the same; by extracting the features of the audio data to be detected, A*B first acoustic features can be obtained; A can refer to the number of audio frames included in the audio data to be detected, and B can refer to the dimension of the first acoustic features of each audio frame, and both A and B are positive integers.

[0086] The first acoustic feature may be a time domain feature or a frequency domain feature, which is not limited in this embodiment of the present application.

[0087] In an embodiment of the present application, a time window can be set, and the window length of the time window can be set to a preset duration; and a sliding step length can be set for the time window to slide on the first acoustic feature of the audio data to be detected. The sliding step length can be set as needed, such as a preset duration, and this embodiment of the present application does not limit this. Then, the time window can be used to slide on the first acoustic feature of the audio data to be detected according to the sliding step length; each time it slides according to the sliding step length, the first acoustic feature of a target audio segment can be obtained, and the length of the target audio segment is the preset duration. The preset duration can be less than or equal to the duration of the audio data to be detected; the preset duration can be determined based on the duration corresponding to the target acoustic event, such as the average duration of the target acoustic event. The target acoustic event can refer to the acoustic event to be detected, such as when a whistle needs to be detected, the target acoustic event can refer to a whistle; for example, when a fighting sound needs to be detected, the target acoustic event can refer to a fighting sound; and for example, when a collision sound needs to be detected, the target acoustic event can refer to a collision sound; and so on.

[0088] After obtaining the first acoustic feature of each target audio segment, steps 106 to 108 may be executed to detect acoustic events for the target audio segment.

[0089] Another method may be to first extract a target audio segment from the audio data to be detected, and then perform feature extraction on the target audio segment to extract the first acoustic feature of the target audio segment. The method for extracting the target audio segment from the audio data to be detected is similar to the method for extracting the first acoustic feature of the target audio segment from the first acoustic feature of the audio data to be detected, and will not be further described here.

[0090] Step 106: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames having a preset time sequence relationship.

[0091] In this embodiment of the present application, first acoustic features with a preset temporal relationship can be divided into batches of first acoustic features based on the temporal relationship of multiple audio frames corresponding to the target audio segment. Multiple batches of first acoustic features can then be obtained. Probability information of acoustic events corresponding to the first acoustic features of each batch can then be determined.

[0092] The preset timing relationship can be set as required, such as adjacent timing, timing interval setting duration, etc., and the embodiment of the present application does not limit this. The set duration can be set as required, such as the frame length corresponding to an audio frame, etc., and the embodiment of the present application does not limit this.

[0093] If the preset timing relationship is temporally adjacent, the first acoustic features of multiple adjacent audio frames can be divided into a batch. For example, the target audio segment includes 10 audio frames of audio data to be detected, and these 10 audio frames are: the 20th audio frame to the 29th audio frame. The 5 adjacent audio frames can be divided into a batch to obtain the first acoustic features of 2 batches; the first acoustic features of these 2 batches are: the first acoustic feature of batch 1 (the first acoustic feature of the 20th audio frame to the first acoustic feature of the 24th audio frame), and the first acoustic feature of batch 2 (the first acoustic feature of the 25th audio frame to the first acoustic feature of the 29th audio frame). Correspondingly, the first acoustic features of a batch can include the first acoustic features of multiple adjacent audio frames. Of course, other division methods can also be included, and the embodiments of the present application are not limited to this.

[0094] If the preset timing relationship is a timing interval setting duration, the first acoustic features of multiple audio frames that are sequentially spaced by the set duration can be divided into a batch. For example, the target audio segment includes 12 audio frames of audio data to be detected, and these 12 audio frames are: the 45th audio frame to the 56th audio frame. If the set duration is the frame length of an audio frame, the first acoustic features of two audio frames separated by the set duration can be divided into the first acoustic features of a batch, and three batches of first acoustic features are obtained; the first acoustic features of these three batches are: the first acoustic features of batch 1 (the first acoustic features of the 45th audio frame and the first acoustic features of the 47th audio frame), the first acoustic features of batch 2 (the first acoustic features of the 49th audio frame and the first acoustic features of the 51st audio frame), and the first acoustic features of batch 3 (the first acoustic features of the 53rd audio frame and the first acoustic features of the 55th audio frame). Correspondingly, the first acoustic features of a batch can include the first acoustic features of multiple audio frames that are sequentially spaced by the set duration. Of course, other division methods may also be included, and the embodiments of the present application do not limit this.

[0095] The embodiment of the present application can perform acoustic event detection based on an acoustic event detector by utilizing the timing information of multiple audio frames contained in the audio data to be detected, so as to improve the accuracy of sound event detection.

[0096] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0097] Among them, you can refer to Figure 1C , showing another schematic diagram of the acoustic event detection data processing process of the present application. After obtaining the audio data to be detected, the first acoustic feature of the target audio segment can be extracted from the audio data to be detected, and the first acoustic feature can be input into the acoustic event detector. The acoustic event detector can output multiple batches of probability information of the first acoustic features corresponding to the acoustic events; and then based on the probability information of the first acoustic features of each batch corresponding to the acoustic events output by the acoustic detector, the acoustic event detection result of the target audio segment is determined.

[0098] In an optional embodiment of the present application, the acoustic event detector may include a deep neural network, which may include multiple first preset networks, a time pooling layer, a second preset network, and a third preset network. The first preset network, the time pooling layer, the second preset network, and the third preset network are connected in sequence; multiple first preset networks are connected in sequence, and the number of first preset networks can be set as needed, which is not limited in this embodiment of the present application.

[0099] The first preset network may include a network with a feature extraction function, and the second preset network and the third preset network may both include networks with a classification function.

[0100] As an example of this application, please refer to Figure 1D , shows a schematic diagram of the structure of an acoustic event detector of the present application. Among them, the first preset network may include: a fully connected layer, a rectified linear unit, a random dropout layer and a feedforward sequence memory network; the second preset network includes: a fully connected layer, a rectified linear unit and a random dropout layer, and the third preset network includes: a fully connected layer, a rectified linear unit and an activation function. Among them, Figure 1D 3 first preset networks are shown in FIG. Compared with the prior art using CNN or CNN-RNN, the Figure 1D The acoustic event detector shown in can reduce the delay of acoustic event detection and improve the acoustic event detection rate.

[0101] In an embodiment of the present application, the first acoustic feature can be input into an acoustic event detector, which processes the input first acoustic feature and outputs multiple batches of probability information of acoustic events corresponding to the first acoustic feature. The probability information of acoustic events corresponding to the first acoustic feature of a batch may include probability information of multiple acoustic events, and the probability information of the multiple acoustic events may include probability information of a target acoustic event and probability information of other acoustic events, and the other acoustic events may refer to acoustic events other than the target acoustic event.

[0102] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S1062 to S1064:

[0103] Sub-step S1062: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0104] Sub-step S1064: input the second acoustic feature into the acoustic event detector.

[0105] Based on the first acoustic feature, a second acoustic feature for input to the acoustic event detector is determined; the second acoustic feature can then be input into the acoustic detector as the input of the acoustic event detector.

[0106] The sub-step S1062 may include the following sub-steps S10622 to S10626:

[0107] Sub-step S10622: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0108] In this embodiment of the present application, the first acoustic feature of one audio frame can be selected from the first acoustic features of multiple audio frames included in the target audio segment as the first acoustic feature of the target audio frame. Then, sub-steps S10624 to S10626 are executed to determine the second acoustic feature of the target audio frame that is input into the acoustic event detector. Furthermore, according to sub-steps S10624 to S10626, the second acoustic feature of each audio frame in the target audio segment that is input into the acoustic event detector can be determined.

[0109] Sub-step S10624: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0110] In an optional embodiment of the present application, the first acoustic feature of the target audio frame can be expanded to add contextual information of the target video frame, and then further utilize the timing information of each audio frame in the target audio segment to further improve the accuracy of acoustic event detection.

[0111] The first acoustic feature of the target audio frame can be spliced ​​with the first acoustic features of the N preceding audio frames to obtain the third acoustic feature corresponding to the target audio frame; the first acoustic feature of the target audio frame can also be spliced ​​with the first acoustic features of the M succeeding audio frames to obtain the third acoustic feature corresponding to the target audio frame; the first acoustic feature of the target audio frame can also be spliced ​​with the first acoustic features of the N preceding audio frames and the first acoustic features of the M succeeding audio frames to obtain the third acoustic feature corresponding to the target audio frame; this embodiment of the application does not impose any restrictions on this. M and N are positive integers, and the values ​​of M and N can be set as required, such as M can be equal to N, which is not restricted in this embodiment of the application.

[0112] For example, assuming that the target audio segment includes 200 frames, M=N=5; if the target audio frame is the 11th audio frame of the target audio segment, the first acoustic features of the 6th to 11th audio frames can be spliced ​​together to obtain the third acoustic feature corresponding to the 11th audio frame, and the third acoustic feature includes the third acoustic features of 6 audio frames. The third acoustic features of the 11th to 16th audio frames can also be spliced ​​together to obtain the third acoustic feature corresponding to the 11th audio frame, and the third acoustic feature includes the third acoustic features of 6 audio frames. The third acoustic features of the 6th to 16th audio frames can also be spliced ​​together to obtain the third acoustic feature corresponding to the 11th audio frame, and the third acoustic feature includes the third acoustic features of 11 audio frames.

[0113] Sub-step S10626: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0114] In the embodiment of the present application, an implementation of sub-step S10626 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0115] In an embodiment of the present application, another implementation of sub-step S10626 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature corresponding to the target audio frame and the multiple fourth acoustic features as the second acoustic feature corresponding to the target video frame. The third acoustic feature corresponding to the target audio frame includes the third acoustic features of multiple consecutive audio frames, and the third acoustic features of each of the three audio frames in the target audio frame may be differentially processed. In one example, the third acoustic features of two adjacent audio frames in the third acoustic feature may be differentially processed to obtain a first differential acoustic feature; then, the two adjacent first differential acoustic features may be differentially processed again to obtain a second differential acoustic feature; and then, the first differential acoustic feature and the second differential acoustic feature may be used as the fourth acoustic feature. In the process of differential processing the third acoustic features of two adjacent audio frames in the third acoustic feature, the third acoustic features of the dimensions corresponding to the two adjacent audio frames in the third acoustic feature may be differentially processed. In addition, the present application does not limit the number of times the third acoustic feature corresponding to the target audio frame is differentiated, nor does it limit the differentiation method (such as forward differentiation, backward differentiation, etc.); and does not limit the interval between the two audio frames in which the third acoustic feature corresponding to the target audio frame is differentiated.

[0116] In an embodiment of the present application, another implementation method of sub-step S10626 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame. Among them, one regularization method may be mean square error regularization, and then the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame may be used as the second acoustic feature corresponding to the target video frame input to the acoustic event detector; or the fifth acoustic feature corresponding to the target video frame may be used as the second acoustic feature corresponding to the target video frame input to the acoustic event detector; this embodiment of the present application does not limit this.

[0117] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0118] In an optional embodiment of the present application, the acoustic event detector outputting multiple batches of probability information of first acoustic features corresponding to acoustic events may include: the acoustic event detector dividing the first acoustic features into multiple batches of first acoustic features according to a preset temporal relationship, and determining and outputting the probability information of the first acoustic features corresponding to acoustic events for each of the multiple batches. This embodiment of the present application does not limit the number of audio frames whose acoustic features are divided into a batch.

[0119] The dividing the first acoustic features of the plurality of audio frames into a plurality of batches of first acoustic features according to a preset temporal relationship includes: dividing the first acoustic features of the plurality of audio frames into a plurality of batches of first acoustic features by performing temporal pooling on the first acoustic features of the plurality of audio frames according to the preset temporal relationship. The temporal pooling layer of the acoustic event detector may be used to perform temporal pooling on the first acoustic features of the plurality of audio frames according to the preset temporal relationship.

[0120] Based on the above-mentioned use of the second acoustic feature as the input of the acoustic event detector, the step of performing time pooling on the first acoustic features of the multiple audio frames according to a preset temporal relationship to divide the first acoustic features of the multiple audio frames into multiple batches of first acoustic features may include the following sub-steps S22-S24:

[0121] Sub-step S22: extracting features from the second acoustic features corresponding to the multiple audio frames to obtain sixth acoustic features of the multiple audio frames.

[0122] Among them, the second acoustic features of each audio frame in the target audio segment can be input into multiple first preset networks of the acoustic event detector in sequence, and the multiple first preset networks of the acoustic event detector can respectively extract features of the second acoustic features of each audio frame in the target audio segment to determine the sixth acoustic features of each audio frame in the target audio segment; and then the last first preset network outputs the sixth acoustic features corresponding to each audio frame.

[0123] Sub-step S24: By performing time pooling on the sixth acoustic features of the multiple audio frames according to a preset timing relationship, the sixth acoustic features of the multiple audio frames are divided into multiple batches of sixth acoustic features, and a batch of sixth acoustic features includes the sixth acoustic features of multiple audio frames with a preset timing relationship.

[0124] In an optional embodiment of the present application, the output of the last first preset network of the acoustic event detector can be used as the input of the time pooling layer; then the time pooling layer divides the sixth acoustic features of each audio frame in the target audio segment into multiple batches of sixth acoustic features according to a preset timing relationship.

[0125] In one example, the preset timing relationship is temporal adjacency, and the time pooling layer can divide the sixth acoustic features of X consecutive audio frames in each audio frame in the target audio segment into a batch of sixth acoustic features; X is a positive integer, and the value of X can be set as required, and the embodiment of the present application is not limited to this. For example, the target audio segment includes 200 audio frames. If X=10, the sixth acoustic features of the 200 audio frames can be divided into 20 batches, and each batch includes the sixth acoustic features of 10 consecutive audio frames. The sixth acoustic features of the 200 audio frames can also be divided into 40 batches, and each batch includes the sixth acoustic features of 10 consecutive audio frames; two adjacent batches contain the same sixth acoustic features of 5 audio frames. Of course, other division methods can also be included, and the embodiment of the present application is not limited to this.

[0126] After the acoustic event detector divides the sixth acoustic features of multiple audio frames into multiple batches of sixth acoustic features, it can process each batch of sixth acoustic features to determine the probability information of each batch of sixth acoustic features corresponding to an acoustic event. Specifically, the sixth acoustic features of each batch output by the temporal pooling layer can be sequentially input into the second preset network and the third preset network for calculation; the third preset network outputs the probability information of each batch of sixth acoustic features corresponding to an acoustic event.

[0127] As an example of the present application, the activation function can be a Sigmoid function, which can make the probability information of the target acoustic event more prominent, and thus can accurately detect the target acoustic event from the audio data where multiple acoustic events overlap, thereby enhancing the practicality of the acoustic event detector.

[0128] Step 108: Determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0129] After the acoustic event detector outputs multiple batches of probability information of acoustic events corresponding to the first acoustic features, the probability information of acoustic events corresponding to the first acoustic features of these multiple batches can be post-processed to determine the acoustic event detection results of the target audio segment, that is, to determine the probability information of the target acoustic event in the target audio segment.

[0130] Subsequently, the probability information of the target acoustic event in the target audio segment can be compared with the first probability threshold to determine whether the target acoustic event exists in the target audio segment. When the probability information of the target acoustic event in the target audio segment is greater than or equal to the first probability threshold, it can be determined that the target acoustic event exists in the target audio segment. When the probability information of the target acoustic event in the target audio segment is less than the first probability threshold, it can be determined that the target acoustic event does not exist in the target audio segment. Among them, the first probability threshold can be set as required, and the embodiment of the present application does not limit this.

[0131] In an example of the present application, an implementation of step 108 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0132] In one example of the present application, another implementation of step 108 may include: selecting first acoustic features of multiple target batches from the multiple batches of first acoustic features, averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtaining the probability information of the target acoustic event in the target audio segment. Among them, the first acoustic features of the batch that contributes most to determining the probability information of the target acoustic event can be selected from the multiple batches of first acoustic features as the first acoustic features of the target batch. Among them, it can be determined by pre-training which batches of first acoustic features of the multiple batches contribute most to determining the probability information of the target acoustic event.

[0133] In one example of the present application, another implementation of step 108 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches after score normalization. The score normalization may be mean square error normalization. In one example, the probability information of the target acoustic events corresponding to the first acoustic features of the multiple batches after score normalization may be averaged to obtain the probability information of the target acoustic event in the target audio segment; of course, other methods may also be used, and the embodiments of the present application do not limit this.

[0134] In one example of the present application, another implementation of step 108 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the first acoustic features of the multiple batches corresponding to the acoustic event. In one example, a support vector machine method can be used to classify the batches; then, based on the probability information of the first acoustic features of each classified batch corresponding to the target acoustic event, the probability information of the target acoustic event in the target audio segment is determined. For example, weight information corresponding to each category after classification is determined, and based on the weight information of each category, the probability information of the first acoustic features of the batches included in each category corresponding to the target acoustic event is weighted to determine the probability information of the target acoustic event in the target audio segment. For another example, a target category that contributes most to the determination of the probability information of the target acoustic event in the target audio segment is selected from the multiple categories; and the probability information of the target acoustic event in the target audio segment is determined based on the probability information of the batches included in the target category corresponding to the target acoustic event. This embodiment of the present application is not limited to this.

[0135] In summary, in an embodiment of the present application, after obtaining the audio data to be detected, the first acoustic features of the target audio segment can be extracted from the audio data to be detected, and the probability information of the acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of the acoustic events corresponding to the multiple batches, the probability information of the target acoustic events in the target audio segment is determined; wherein, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset timing relationship, that is, the embodiment of the present application utilizes the timing information of multiple audio frames contained in the audio data to be detected to perform acoustic event detection; thereby, the accuracy of sound event detection can be improved.

[0136] The acoustic event detector described in the above embodiment may be a trained acoustic event detector. The following describes the training process of the acoustic event detector.

[0137] Reference Figure 2 , shows a flowchart of the steps of an embodiment of a training method for an acoustic event detector of the present application.

[0138] Step 202: Collect audio data of multiple target acoustic events and audio data of multiple other acoustic events.

[0139] In an embodiment of the present application, audio data of multiple target acoustic events and audio data of multiple other acoustic events can be collected as training data; the training data can then be used to train an acoustic event detector. The audio data of the target acoustic event can refer to audio data that includes the target acoustic event, and the audio data of the other acoustic events can refer to audio data that does not include the target acoustic event.

[0140] Step 204 : For audio data of a target acoustic event, divide the audio data of the target acoustic event into multiple segments.

[0141] In actual practice, the audio data of the target acoustic event is less in resource than the audio data of other acoustic events. In order to ensure the accuracy of the acoustic event detector in detecting the target acoustic event, the audio data of each target acoustic event can be divided into multiple audio segments, thereby enriching the audio data of the target acoustic event. In one implementation method, a time window can be used to slide on the audio data of the target acoustic event, and multiple audio segments can be intercepted from the audio data of the target acoustic event. The window length of the time window and the corresponding sliding step size can be set as required, and the embodiment of the present application does not impose any restrictions on this.

[0142] In an embodiment of the present application, the number of audio segments obtained from the audio data of multiple target acoustic events can be balanced with the number of audio data of other acoustic events; wherein, maintaining balance can mean keeping the number of audio segments no less than the number of other acoustic events.

[0143] Step 206: Use the multiple audio clips as positive samples and the audio data of multiple other acoustic events as negative samples to train the acoustic event detector.

[0144] Then, each audio clip can be used as a positive sample, and the audio data of each other acoustic event can be used as a negative sample; and the positive samples and negative samples are used to train the acoustic event detector, wherein the positive samples are not less than the negative samples.

[0145] In one example of the present application, in the process of training the acoustic event detector, binary cross entropy can be used as a loss function so that in the process of using the acoustic event detector to detect acoustic events, the acoustic event detector can accurately detect the target acoustic event from the audio data where multiple acoustic events overlap, thereby increasing the practicality of the acoustic event detector.

[0146] Based on the above embodiments, the embodiments of the present application also provide a monitoring method based on acoustic event detection, which is applied to security monitoring in various scenarios, such as traffic safety monitoring, home care safety monitoring, and public place safety monitoring.

[0147] In an optional embodiment of the present application, the monitoring method based on acoustic event detection can be executed by a monitoring system to perform security monitoring of various scenes. The monitoring system can include an audio acquisition device.

[0148] Reference Figure 3 , shows a step flow chart of an embodiment of a monitoring method based on acoustic event detection of the present application.

[0149] Step 302: Acquire audio data collected by an audio collection device.

[0150] In an embodiment of the present application, the audio acquisition device in the monitoring system can collect audio data so that the monitoring system can perform security monitoring based on the audio data collected by the audio acquisition device.

[0151] For example, in a traffic safety monitoring scenario, the various audio acquisition devices in a monitoring system used to monitor traffic safety can collect audio data in real time, such as audio acquisition devices used to monitor roads and audio acquisition devices in public transportation facilities. Another example is a home care safety monitoring scenario where the various audio acquisition devices in a monitoring system used to monitor home care safety can collect audio data in real time, such as audio acquisition devices in homes or hospitals. Another example is a public space safety monitoring scenario where audio acquisition devices in public spaces can collect audio data in real time.

[0152] Correspondingly, during the monitoring process, the monitoring system may obtain audio data collected by the audio collection device; the audio data may include multiple audio frames.

[0153] Step 304: Extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes multiple audio frames.

[0154] One implementation of step 304 may include the following sub-steps S3042 to S3044:

[0155] Sub-step S3042: extracting the first acoustic feature of the audio data.

[0156] Sub-step S3044: Use a window with a preset duration to slide on the first acoustic feature of the audio data according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0157] Among them, sub-steps S3042 to S3044 are similar to the above-mentioned sub-steps S1042 to S1044 and will not be repeated here.

[0158] Another implementation of step 304 may be to first capture a target audio segment from the audio data, and then perform feature extraction on the target audio segment to extract a first acoustic feature of the target audio segment.

[0159] Step 306: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include first acoustic features of multiple audio frames having a preset time sequence relationship.

[0160] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0161] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S3062 to S3064:

[0162] Sub-step S3062: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0163] Sub-step S3064: input the second acoustic feature into the acoustic event detector.

[0164] Among them, the sub-steps S3062 to S3064 are similar to the above-mentioned sub-steps S1064 to S1064, and will not be repeated here.

[0165] The sub-step S3062 may include the following sub-steps S30622 to S30626:

[0166] Sub-step S30622: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0167] Sub-step S30624: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0168] Sub-step S30626: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0169] Among them, the sub-steps S30622 to S30626 and the above-mentioned sub-steps S10622 to S10626 are not repeated here.

[0170] In the embodiment of the present application, an implementation of sub-step S30626 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0171] In an embodiment of the present application, another implementation method of sub-step S30626 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0172] In an embodiment of the present application, another implementation method of sub-step S30626 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0173] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0174] Step 308: Determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0175] The target acoustic event includes at least one of the following: a horn, a collision, an explosion, a cry for help, a fighting sound, and a quarrel; the target acoustic event may also include other acoustic events in a security monitoring scenario, which are not listed here one by one.

[0176] In an example of the present application, an implementation of step 308 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0177] In an example of the present application, another implementation method of step 308 may include: selecting the first acoustic features of multiple target batches from the first acoustic features of the multiple batches, averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtaining the probability information of the target acoustic event in the target audio segment.

[0178] In an example of the present application, another implementation method of step 308 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0179] In an example of the present application, another implementation of step 308 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0180] Here, step 304 is similar to step 108 above and will not be described again here.

[0181] Step 310: Generate prompt information based on the probability information of the target acoustic event in the target audio segment.

[0182] In one embodiment of the present application, after determining the probability information of the target acoustic event in the target audio segment, the probability information of the target acoustic event can be compared with the second probability threshold. If the probability information of the target acoustic event is greater than or equal to the second probability threshold, it can be determined that the target acoustic event exists in the target audio segment. At this time, prompt information such as a short message, the target audio segment, an alarm audio, etc. can be generated; and then sent to the corresponding terminal device to prompt the user. If the probability information of the target acoustic event is less than the second probability threshold, it can be determined that the target acoustic event does not exist in the target audio segment. At this time, the acoustic event in the next target audio segment can be detected. Among them, the second probability threshold can be set as needed, and the embodiment of the present application does not limit this.

[0183] In one embodiment of the present application, multiple probability threshold ranges can be set, and the information type of prompt information corresponding to each probability threshold range can be set. The probability information of the target acoustic event can then be compared with each probability threshold range to determine the probability threshold range to which the probability information of the target acoustic event belongs; then, prompt information of the information type corresponding to the probability threshold range to which the probability information of the target acoustic event belongs can be generated.

[0184] In summary, in the embodiment of the present application, after obtaining the audio data collected by the audio acquisition device, the first acoustic features of the target audio segment can be extracted from the audio data, and the probability information of the acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, the probability information of the target acoustic events in the target audio segment can be determined; subsequently, prompt information can be generated based on the probability information of the target acoustic events in the target audio segment; and thus, the detection and monitoring of various target acoustic events in the security monitoring scene can be achieved. Among them, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to detect acoustic events, which can improve the accuracy of acoustic event detection, thereby effectively performing security monitoring.

[0185] Based on the above embodiments, the embodiments of the present application also provide a monitoring method based on acoustic event detection, which is applied to security monitoring in various scenarios, such as traffic safety monitoring, home care safety monitoring, and public place safety monitoring.

[0186] In an optional embodiment of the present application, the monitoring method based on acoustic event detection can be executed by a monitoring system to perform security monitoring of various scenarios. The monitoring system can include an audio acquisition device and an image acquisition device, and the monitoring system can combine audio data collected by the audio acquisition device with image data collected by the image acquisition device to perform security monitoring.

[0187] Reference Figure 4 , shows a step flow chart of another embodiment of a monitoring method based on acoustic event detection of the present application.

[0188] Step 402: Acquire audio data collected by an audio collection device.

[0189] Step 404: Extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes multiple audio frames.

[0190] One implementation of step 404 may include the following sub-steps S4042 to S4044:

[0191] Sub-step S4042: extracting the first acoustic feature of the audio data.

[0192] Sub-step S4044: Use a window with a preset duration to slide on the first acoustic feature of the audio data according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0193] Among them, sub-steps S4042 to S4044 are similar to the above-mentioned sub-steps S1042 to S1044 and will not be repeated here.

[0194] One way of performing step 404 may be to first capture a target audio segment from the audio data, and then perform feature extraction on the target audio segment to extract the first acoustic feature of the target audio segment.

[0195] Step 406: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include first acoustic features of multiple audio frames having a preset time sequence relationship.

[0196] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0197] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S4062 to S4064:

[0198] Sub-step S4062: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0199] Sub-step S4064: input the second acoustic feature into the acoustic event detector.

[0200] Among them, the sub-steps S4062 to S4064 are similar to the above-mentioned sub-steps S1064 to S1064, and will not be repeated here.

[0201] The sub-step S4062 may include the following sub-steps S40622 to S40626:

[0202] Sub-step S40622: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0203] Sub-step S40624: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0204] Sub-step S40626: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0205] Among them, the sub-steps S40622 to S40626 and the above-mentioned sub-steps S10622 to S10626 are not repeated here.

[0206] In the embodiment of the present application, an implementation of sub-step S40626 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0207] In an embodiment of the present application, another implementation method of sub-step S40626 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0208] In an embodiment of the present application, another implementation method of sub-step S40626 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0209] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0210] Step 408: Determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0211] The target acoustic event includes at least one of the following: a horn, a collision, an explosion, a cry for help, a fighting sound, and a quarrel; the target acoustic event may also include other acoustic events in the security monitoring scene, which are not listed here one by one.

[0212] In an example of the present application, an implementation of step 408 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0213] In an example of the present application, another implementation method of step 408 may include: selecting the first acoustic features of multiple target batches from the first acoustic features of the multiple batches, averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtaining the probability information of the target acoustic event in the target audio segment.

[0214] In an example of the present application, another implementation method of step 408 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0215] In an example of the present application, another implementation of step 408 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0216] Here, step 404 is similar to step 108 above and will not be described again here.

[0217] Step 410: When it is determined that an abnormality occurs based on the probability information of the target acoustic event in the target audio segment, image data captured by the image capture device is acquired.

[0218] Step 412: Perform anomaly confirmation based on the target image segment corresponding to the target audio segment in the image data.

[0219] In one embodiment of the present application, after determining the probability information of the target acoustic event in the target audio segment, the probability information of the target acoustic event can be compared with a third probability threshold. If the probability information of the target acoustic event is greater than or equal to the third probability threshold, it can be determined that the target acoustic event exists in the target audio segment, and at this time, it can be determined that an abnormality has occurred. If the probability information of the target acoustic event is less than the second probability threshold, it can be determined that the target acoustic event does not exist in the target audio segment, and at this time, it can be determined that no abnormality has occurred. Among them, the third probability threshold can be set as needed, and the embodiment of the present application does not limit this.

[0220] In an embodiment of the present application, when it is determined that an abnormality has occurred, image data captured by the image capture device can be obtained; then, the image data can be analyzed to confirm the abnormality, thereby reducing false alarms and minimizing disturbances to users.

[0221] In particular, based on the timestamp of the target audio segment, the image data may be searched for an image segment with the same timestamp as the target audio segment, and the image segment may be determined as the target image segment. Alternatively, after searching the image data for an image segment with the same timestamp as the target audio segment, the target image segment may be formed by combining the image segment with an image segment of a first set duration before the image segment, or combining the image segment with an image segment of a second set duration after the image segment, or combining the image segment with an image segment of a first set duration before the image segment and an image segment of a second set duration after the image segment; this embodiment of the present application is not limited to this.

[0222] Then, the target image segment can be analyzed to confirm whether an abnormality has occurred.

[0223] Step 414: When an abnormality is confirmed based on the target image segment, a prompt message is generated.

[0224] When an abnormality is confirmed based on the target image segment, a prompt message may be generated; this is similar to the above step 310 and will not be described in detail here.

[0225] In summary, in an embodiment of the present application, after acquiring audio data collected by an audio acquisition device, first acoustic features of a target audio segment can be extracted from the audio data, and probability information of acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of acoustic events corresponding to the first acoustic features of the multiple batches, probability information of a target acoustic event in the target audio segment can be determined; subsequently, when an abnormality is determined to have occurred based on the probability information of the target acoustic event in the target audio segment, image data collected by the image acquisition device is acquired, and then, based on the target image segment corresponding to the target audio segment in the image data, an abnormality is confirmed, and when an abnormality is confirmed to have occurred based on the target image segment, a prompt message is generated; thereby, by combining audio and image, various target acoustic events in a security monitoring scene can be detected and monitored, thereby reducing false alarms and user disturbances. In particular, a batch of first acoustic features includes first acoustic features of multiple audio frames with a preset time sequence relationship. That is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data for acoustic event detection, which can improve the accuracy of acoustic event detection, thereby effectively performing security monitoring.

[0226] Based on the above embodiments, the embodiments of the present application further provide a monitoring method based on acoustic event detection, which is applied to vehicle safety monitoring.

[0227] In an optional embodiment of the present application, the acoustic event detection-based monitoring method can be executed by an on-board monitoring system to perform safety monitoring of the vehicle. The on-board monitoring system can include an audio acquisition device, an image acquisition device, and a vehicle information acquisition device. The on-board monitoring system can combine audio data collected by the audio acquisition device, image data collected by the image acquisition device, and vehicle information collected by the vehicle information acquisition device to perform safety monitoring.

[0228] Reference Figure 5 , shows a step flow chart of another embodiment of a monitoring method based on acoustic event detection of the present application.

[0229] Step 502: Acquire audio data collected by an audio collection device.

[0230] In an embodiment of the present application, the audio acquisition device in the vehicle-mounted monitoring system may include audio acquisition devices arranged in the vehicle, such as a microphone array, etc.; these audio acquisition devices can collect audio data so that the vehicle-mounted monitoring system can perform security monitoring based on the audio data collected by the audio acquisition devices.

[0231] Step 504: Extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes multiple audio frames.

[0232] One implementation of step 504 may include the following sub-steps S5042 to S5044:

[0233] Sub-step S5042: extracting the first acoustic feature of the audio data.

[0234] Sub-step S5044: Use a window with a preset duration to slide on the first acoustic feature of the audio data according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0235] Among them, sub-steps S5042 to S5044 are similar to the above-mentioned sub-steps S1042 to S1044 and will not be repeated here.

[0236] One way of performing step 504 may be to first extract a target audio segment from the audio data, and then perform feature extraction on the target audio segment to extract the first acoustic feature of the target audio segment.

[0237] Step 506: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames having a preset time sequence relationship.

[0238] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0239] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S5062 to S5064:

[0240] Sub-step S5062: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0241] Sub-step S5064: input the second acoustic feature into the acoustic event detector.

[0242] Among them, the sub-steps S5062 to S5064 are similar to the above-mentioned sub-steps S1064 to S1064, and will not be repeated here.

[0243] The sub-step S4062 may include the following sub-steps S40622 to S40626:

[0244] Sub-step S50622: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0245] Sub-step S50624: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0246] Sub-step S50626: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0247] Among them, the sub-steps S50622 to S50626 and the above-mentioned sub-steps S10622 to S10626 are not repeated here.

[0248] In the embodiment of the present application, an implementation of sub-step S50626 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0249] In an embodiment of the present application, another implementation method of sub-step S50626 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0250] In an embodiment of the present application, another implementation method of sub-step S50626 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0251] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0252] Step 508: Determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0253] The target acoustic event includes at least one of the following: honking, collision, explosion, cry for help, fighting, quarreling, crying, knocking, etc.; the target acoustic event may also include other acoustic events in the vehicle monitoring scene, which are not listed here one by one.

[0254] In an example of the present application, an implementation of step 508 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0255] In an example of the present application, another implementation method of step 508 may include: selecting multiple first acoustic features of target batches from the multiple batches of first acoustic features, averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtaining the probability information of the target acoustic event in the target audio segment.

[0256] In an example of the present application, another implementation method of step 508 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0257] In an example of the present application, another implementation of step 508 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0258] Here, step 508 is similar to step 108 above and will not be described again here.

[0259] Step 510: When it is determined that an abnormality has occurred based on the probability information of the target acoustic event in the target audio segment, image data collected by the image acquisition device in the monitoring system and vehicle information collected by the vehicle information acquisition device are acquired.

[0260] Step 512: Determine the abnormality type of the abnormality based on the image data and the vehicle information; the abnormality type includes an internal abnormality type of the vehicle or an external abnormality type of the vehicle.

[0261] In one embodiment of the present application, after determining the probability information of the target acoustic event in the target audio segment, the probability information of the target acoustic event can be compared with a fourth probability threshold. If the probability information of the target acoustic event is greater than or equal to the fourth probability threshold, it can be determined that the target acoustic event exists in the target audio segment, and at this time, it can be determined that an abnormality has occurred. If the probability information of the target acoustic event is less than the fourth probability threshold, it can be determined that the target acoustic event does not exist in the target audio segment, and at this time, it can be determined that no abnormality has occurred. Among them, the fourth probability threshold can be set as needed, and the embodiment of the present application does not limit this.

[0262] In an embodiment of the present application, an image acquisition device is deployed in a vehicle. The image acquisition device may include an image acquisition device deployed inside the vehicle and an image acquisition device deployed outside the vehicle; accordingly, the image data inside the vehicle and the image data outside the vehicle can be collected separately. In addition, various vehicle information collection devices are deployed in the vehicle, such as various sensors such as lock sensors and seat pressure sensors, etc., which can collect vehicle information. The vehicle information may include information related to the vehicle, such as status information of various vehicle devices such as air conditioning, seats, lights, locks, etc., as well as vehicle location information, etc., which are not limited in the embodiment of the present application.

[0263] In an embodiment of the present application, when it is determined that an abnormality has occurred, image data collected by an image acquisition device and vehicle information collected by a vehicle information acquisition device can be obtained; then, the abnormality type of the abnormality can be determined based on the image data and the vehicle information; the abnormality type includes an internal abnormality type of the vehicle or an external abnormality type of the vehicle, so as to prompt the user in a timely manner.

[0264] For example, if the probability information of the target acoustic event in the target audio segment is the probability information of crying and is greater than the fourth probability threshold, an abnormality may be determined to have occurred. After acquiring the image data and vehicle information, if the presence of a child is determined in the vehicle based on the interior image data, and the presence of a user is determined in the vehicle based on the pressure value of the seat pressure sensor in the vehicle information, the abnormality type of the abnormality may be determined to be a vehicle interior abnormality type.

[0265] For another example, if the probability information of the target acoustic event in the target audio segment is the probability information of a collision sound and is greater than a fourth probability threshold, an abnormality may be determined to have occurred. After acquiring image data and vehicle information, if the vehicle interior is determined to be normal based on the interior image data, if the vehicle exterior is determined to have rear-ended a preceding vehicle based on the exterior image data, and if damage is determined based on the vehicle information, the abnormality type may be determined to be a vehicle exterior abnormality type.

[0266] Step 514: Generate corresponding prompt information according to the exception type of the exception.

[0267] After generating corresponding prompt information based on the abnormality type, the prompt information can be sent to the corresponding user's terminal device. For example, if the abnormality type is an in-vehicle abnormality type, the prompt information can be sent to the vehicle owner's terminal device. For another example, if the abnormality type is an external abnormality type, the prompt information can be sent to the terminal devices of each vehicle owner, the terminal devices of relevant departments, the terminal devices of insurance companies, etc. This allows other monitoring systems to combine data collected by the vehicle monitoring system for security monitoring, which is not limited in this embodiment of the present application.

[0268] In summary, the embodiments of the present application can be applied to a vehicle-mounted monitoring system, which includes an audio acquisition device, an image acquisition device, and a vehicle information acquisition device. The vehicle-mounted monitoring system can obtain audio data collected by the audio acquisition device, and extract the first acoustic feature of the target audio segment from the audio data, and then determine the probability information of the acoustic event corresponding to the first acoustic feature of multiple batches; then, based on the probability information of the acoustic events corresponding to the multiple batches, determine the probability information of the target acoustic event in the target audio segment; subsequently, when determining that an abnormality has occurred based on the probability information of the target acoustic event in the target audio segment, image data collected by the image acquisition device in the monitoring system and vehicle information collected by the vehicle information acquisition device can be obtained, and based on the image data and vehicle information, the abnormality type of the abnormality can be determined; the abnormality type includes an internal vehicle abnormality type or an external vehicle abnormality type, and then a corresponding prompt information is generated based on the abnormality type of the abnormality; thereby, by combining audio, image, and vehicle information, various target acoustic events in the vehicle monitoring scene can be detected and monitored, thereby reducing false alarms and reducing disturbance to users. Among them, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship. That is to say, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to detect acoustic events, which can improve the accuracy of acoustic event detection and thus effectively monitor vehicle safety.

[0269] On the basis of the above embodiments, the embodiments of the present application further provide a method for identifying illegal segments based on acoustic event detection, which is applied to identifying illegal segments in various audio and video playback platforms.

[0270] Reference Figure 6 , shows a flowchart of the steps of an embodiment of a method for identifying illegal fragments based on acoustic event detection of the present application.

[0271] Step 602: Acquire audio data to be detected.

[0272] In an embodiment of the present application, when it is necessary to detect whether there are any illegal clips in the audio / video played on a certain platform, such as an audio / video platform, the audio data played on the audio / video platform can be obtained as the audio data to be detected. The audio data to be detected can be audio data played independently by the platform, such as songs, audio books, etc.; it can also be audio data corresponding to the video data played on the platform. The illegal clips can refer to clips that violate corresponding laws and regulations, such as pornographic clips, terrorism clips, explosion-related clips, etc.

[0273] The audio data to be detected may include multiple audio frames.

[0274] Step 604: Extract a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes multiple audio frames.

[0275] One implementation of step 604 may include the following sub-steps S6042 to S6044:

[0276] Sub-step S6042: extracting the first acoustic feature of the audio data to be detected.

[0277] Sub-step S6044: Use a window with a preset duration to slide on the first acoustic feature of the audio data according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0278] Among them, sub-steps S6042 to S6044 are similar to the above-mentioned sub-steps S1042 to S1044 and will not be repeated here.

[0279] One way of performing step 604 may be to first capture a target audio segment from the audio data, and then perform feature extraction on the target audio segment to extract the first acoustic feature of the target audio segment.

[0280] Step 606: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include first acoustic features of multiple audio frames having a preset time sequence relationship.

[0281] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0282] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S6062 to S6064:

[0283] Sub-step S6062: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0284] Sub-step S6064: input the second acoustic feature into the acoustic event detector.

[0285] Among them, the sub-steps S6062 to S6064 are similar to the above-mentioned sub-steps S1064 to S1064, and will not be repeated here.

[0286] The sub-step S6062 may include the following sub-steps S60622 to S60626:

[0287] Sub-step S60622: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0288] Sub-step S60624: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0289] Sub-step S60626: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0290] Among them, the sub-steps S60622 to S60626 and the above-mentioned sub-steps S10622 to S10626 are not repeated here.

[0291] In the embodiment of the present application, an implementation of sub-step S60626 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0292] In an embodiment of the present application, another implementation method of sub-step S60626 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0293] In an embodiment of the present application, another implementation method of sub-step S60626 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0294] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0295] Step 608: Determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0296] In an example of the present application, an implementation of step 608 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0297] In an example of the present application, another implementation method of step 608 may include: selecting multiple target batches of first acoustic features from the multiple batches of first acoustic features, averaging the probability information of the target acoustic events corresponding to the multiple target batches of first acoustic features, and obtaining the probability information of the target acoustic event in the target audio segment.

[0298] In an example of the present application, another implementation method of step 608 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0299] In an example of the present application, another implementation of step 608 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0300] Among them, step 608 is similar to the above-mentioned step 108 and will not be repeated here.

[0301] Step 610: Determine whether the target audio segment is an illegal segment based on the probability information of the illegal acoustic events in the target audio segment.

[0302] Among them, the probability information of the illegal acoustic event in the target audio segment can be compared with the fifth probability threshold; if the probability information of the illegal acoustic event in the target audio segment is greater than or equal to the fifth probability threshold, it can be determined that the target audio segment contains an illegal acoustic event, and at this time, it can be determined that the target audio segment is an illegal segment. If the probability information of the illegal acoustic event in the target audio segment is less than the fifth probability threshold, it can be determined that the target audio segment does not contain an illegal acoustic event, and at this time, it can be determined that the target audio segment is not an illegal segment. Among them, the fifth probability threshold can be set as needed, and the embodiment of the present application does not limit this.

[0303] In addition, if the audio data to be detected is audio data corresponding to video data, then the target audio segment has a corresponding video segment. In this case, the video segment corresponding to the target audio segment can also be determined as an illegal segment.

[0304] In summary, in an embodiment of the present application, after obtaining the audio data played in the audio and video platform, the first acoustic features of the target audio segment can be extracted from the audio data to be detected, and the probability information of the acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, the probability information of the target acoustic events in the target audio segment is determined; and then, based on the probability information of the illegal acoustic events in the target audio segment, it is judged whether the target audio segment is an illegal segment; wherein, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to be detected to perform acoustic event detection, which can improve the accuracy of acoustic event detection, thereby effectively identifying illegal segments.

[0305] On the basis of the above embodiments, the embodiments of the present application further provide a segment search method based on acoustic event detection, which is applied to searching for audio and video segments played on various audio and video playback platforms.

[0306] Reference Figure 7 , shows a step flow chart of an embodiment of a fragment search method based on acoustic event detection of the present application.

[0307] Step 702: Receive query information.

[0308] User input query information can be obtained; wherein, the query information can be input by the user on the audio and video platform, or input on other platforms, and the embodiment of the present application does not limit this.

[0309] Step 704: Determine the audio data to be queried and the acoustic events to be queried corresponding to the query information.

[0310] The acoustic event to be queried includes at least one of the following: cheers, applause, whistles, fighting sounds, and quarreling sounds; of course, other acoustic events may also be included, which is not limited in the embodiment of the present application.

[0311] The query information can then be subjected to natural language processing to determine the audio data and acoustic events to be queried corresponding to the query information. The audio data to be queried corresponding to the query information can be independent audio data such as songs, audiobooks, etc., or audio data corresponding to video data.

[0312] For example, if the user enters the query information "fighting scenes in the movie AAA," the audio data corresponding to the movie "AAA" can be determined as the audio data to be queried, and the sounds of fighting can be determined as the acoustic events to be queried. For another example, if the user enters the query information "highlights in the movie XXX," the audio data corresponding to the movie "XXX" can be determined as the audio data to be queried. Furthermore, cheering, applause, whistles, fighting sounds, and quarreling sounds can all be determined as acoustic events to be queried.

[0313] Step 706: Extract a first acoustic feature of a target audio segment from the audio data to be queried, where the target audio segment includes multiple audio frames.

[0314] One implementation of step 704 may include the following sub-steps S7042 to S7044:

[0315] Sub-step S7042: extracting the first acoustic feature of the audio data to be queried.

[0316] Sub-step S7044: Use a window with a preset duration to slide on the first acoustic feature of the audio data according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0317] Among them, sub-steps S7042 to S7044 are similar to the above-mentioned sub-steps S1042 to S1044 and will not be repeated here.

[0318] Another implementation of step 704 may be to first intercept a target audio segment from the audio data to be queried, and then perform feature extraction on the target audio segment to extract the first acoustic feature of the target audio segment.

[0319] Step 708: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames having a preset time sequence relationship.

[0320] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0321] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S7082 to S7084:

[0322] Sub-step S7082: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0323] Sub-step S7084: input the second acoustic feature into the acoustic event detector.

[0324] Among them, the sub-steps S7082 to S7084 are similar to the above-mentioned sub-steps S1064 to S1064, and will not be repeated here.

[0325] The sub-step S7082 may include the following sub-steps S70822 to S70826:

[0326] Sub-step S70822: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0327] Sub-step S70824: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0328] Sub-step S70826: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0329] Among them, the sub-steps S70822 to S70826 and the above-mentioned sub-steps S10622 to S10626 are not repeated here.

[0330] In the embodiment of the present application, an implementation of sub-step S70826 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0331] In an embodiment of the present application, another implementation method of sub-step S70826 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0332] In an embodiment of the present application, another implementation method of sub-step S70826 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0333] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0334] Step 710: Determine the probability information of the acoustic event to be queried in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0335] In an example of the present application, an implementation of step 710 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0336] In an example of the present application, another implementation method of step 710 may include: selecting multiple first acoustic features of target batches from the multiple batches of first acoustic features, averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtaining the probability information of the target acoustic event in the target audio segment.

[0337] In an example of the present application, another implementation method of step 710 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0338] In an example of the present application, another implementation of step 710 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0339] Among them, step 710 is similar to the above-mentioned step 108 and will not be repeated here.

[0340] Step 712: Return the target audio segment based on the probability information of the acoustic event to be queried in the target audio segment.

[0341] Among them, the probability information of the acoustic event to be queried in the target audio segment can be compared with the sixth probability threshold; if the probability information of the acoustic event to be queried in the target audio segment is greater than or equal to the sixth probability threshold, the target audio segment can be returned to the user. Of course, when the audio data to be queried is the audio data corresponding to the video data, it is determined that there is a corresponding video segment for the target audio segment. At this time, the target audio segment and the video segment corresponding to the target audio segment can be returned to the user. If the probability information of the acoustic event to be queried in the target audio segment is less than the sixth probability threshold, the target audio segment can be filtered. Among them, the sixth probability threshold can be set as needed, and the embodiment of the present application does not limit this.

[0342] In summary, after receiving the query information, the embodiment of the present application can determine the audio data to be queried and the acoustic event to be queried corresponding to the query information; then extract the first acoustic feature of the target audio segment from the audio data to be queried, and determine the probability information of the acoustic event corresponding to the first acoustic features of multiple batches; then determine the probability information of the acoustic event to be queried in the target audio segment based on the probability information of the acoustic event corresponding to the first acoustic features of the multiple batches, and return the target audio segment based on the probability information of the acoustic event to be queried in the target audio segment; wherein, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to be queried to perform acoustic event detection, which can improve the accuracy of acoustic event detection, thereby searching for segments that better meet user needs, improving the search quality, and also improving user experience.

[0343] Of course, it is also possible to add identifiers to the target acoustic event segments in each audio data in advance; then, after receiving the user's query information, the target acoustic event segments can be quickly searched directly based on the identifiers, which can improve the search efficiency.

[0344] Reference Figure 8 , shows a step flow chart of an embodiment of a method for adding a marker based on acoustic event detection of the present application.

[0345] Step 802: Acquire audio data played on the audio and video platform.

[0346] Among them, step 802 is similar to the above step 702 and will not be repeated here.

[0347] Step 804: Extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes multiple audio frames.

[0348] One implementation of step 804 may include the following sub-steps S8042 to S8044:

[0349] Sub-step S8042: extracting the first acoustic feature of the audio data.

[0350] Sub-step S8044: Use a window with a preset duration to slide on the first acoustic feature of the audio data according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0351] Among them, sub-steps S8042 to S8044 are similar to the above-mentioned sub-steps S1042 to S1044 and will not be repeated here.

[0352] One way of performing step 804 may be to first capture a target audio segment from the audio data, and then perform feature extraction on the target audio segment to extract the first acoustic feature of the target audio segment.

[0353] Step 806: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames having a preset time sequence relationship.

[0354] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0355] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S8062 to S8064:

[0356] Sub-step S8062: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0357] Sub-step S8064: input the second acoustic feature into the acoustic event detector.

[0358] Among them, the sub-steps S8062 to S8064 are similar to the above-mentioned sub-steps S1064 to S1064, and will not be repeated here.

[0359] The sub-step S8062 may include the following sub-steps S80622 to S80626:

[0360] Sub-step S80622: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0361] Sub-step S80624: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0362] Sub-step S80626: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0363] Among them, the sub-steps S80622 to S80626 and the above-mentioned sub-steps S10622 to S10626 are not repeated here.

[0364] In the embodiment of the present application, an implementation of sub-step S80626 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0365] In an embodiment of the present application, another implementation method of sub-step S80626 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0366] In an embodiment of the present application, another implementation method of sub-step S80626 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0367] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0368] Step 808: Determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0369] The target acoustic event includes at least one of the following: cheers, applause, whistles, fighting sounds, and quarreling sounds; of course, other acoustic events may also be included, which is not limited in this application.

[0370] In an example of the present application, an implementation of step 808 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0371] In an example of the present application, another implementation method of step 808 may include: selecting multiple target batches of first acoustic features from the multiple batches of first acoustic features, averaging the probability information of the target acoustic events corresponding to the multiple target batches of first acoustic features, and obtaining the probability information of the target acoustic event in the target audio segment.

[0372] In an example of the present application, another implementation method of step 808 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0373] In an example of the present application, another implementation of step 808 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0374] Among them, step 808 is similar to the above-mentioned step 108 and will not be repeated here.

[0375] Step 810: Add a corresponding acoustic event identifier to the target audio segment based on the probability information of the target acoustic event in the target audio segment.

[0376] Among them, the probability information of the target acoustic event in the target audio segment can be compared with the seventh probability threshold; if the probability information of the target acoustic event in the target audio segment is greater than or equal to the seventh probability threshold, it can be determined that the target acoustic event exists in the target audio segment, and the corresponding acoustic event identifier can be added to the target audio segment. Among them, different target acoustic events can correspond to different acoustic event identifiers; for example, when the target acoustic event in the predicted time segment is applause, an applause identifier can be added to the preset time segment; when the target acoustic event in the predicted time segment is a fighting sound, a fighting identifier can be added to the preset time segment; and so on.

[0377] If the probability information of the target acoustic event in the target audio segment is less than the seventh probability threshold, it can be determined that the target acoustic event does not exist in the target audio segment, and in this case, there is no need to add an acoustic event identifier to the target audio segment. The seventh probability threshold can be set as required and is not limited in this embodiment of the present application.

[0378] Subsequently, after receiving the user's query information, the audio data to be queried and the acoustic event to be queried corresponding to the query information can be determined; then, the audio clip that matches the acoustic event identifier corresponding to the acoustic event to be queried can be searched from the audio data to be queried, and then the searched audio clip and / or the video clip corresponding to the audio clip can be returned to the user.

[0379] On the basis of the above embodiments, the embodiments of the present application further provide an anomaly analysis method based on acoustic event detection, which can call corresponding information collection equipment to collect information and analyze the anomaly when an anomaly is detected based on an acoustic event.

[0380] Reference Figure 9 , shows a step flow chart of an embodiment of an abnormality analysis method based on acoustic event detection of the present application.

[0381] Step 902: Acquire audio data collected by an audio collection device.

[0382] Step 904: Extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes multiple audio frames.

[0383] One implementation of step 904 may include the following sub-steps S9042 to S9044:

[0384] Sub-step S9042: extracting the first acoustic feature of the audio data.

[0385] Sub-step S9044: Use a window with a preset duration to slide on the first acoustic feature of the audio data according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0386] Among them, sub-steps S9042 to S9044 are similar to the above-mentioned sub-steps S1042 to S1044 and will not be repeated here.

[0387] One way of performing step 904 may be to first capture a target audio segment from the audio data, and then perform feature extraction on the target audio segment to extract the first acoustic feature of the target audio segment.

[0388] Step 906: Determine probability information of acoustic events corresponding to the first acoustic features of the multiple batches; wherein the first acoustic features of a batch include the first acoustic features of multiple audio frames having a preset time sequence relationship.

[0389] The determining of probability information of multiple batches of first acoustic features corresponding to acoustic events includes: inputting the first acoustic features into an acoustic event detector, and the acoustic event detector outputting probability information of multiple batches of first acoustic features corresponding to acoustic events.

[0390] The step of inputting the first acoustic feature into the acoustic event detector may include the following sub-steps S9062 to S9064:

[0391] Sub-step S9062: Based on the first acoustic features of the multiple audio frames, determine the second acoustic features corresponding to the multiple audio frames respectively.

[0392] Sub-step S9064: input the second acoustic feature into the acoustic event detector.

[0393] Among them, the sub-steps S9062 to S9064 are similar to the above-mentioned sub-steps S1064 to S1064, and will not be repeated here.

[0394] The sub-step S9062 may include the following sub-steps S90622 to S90626:

[0395] Sub-step S90622: Select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames.

[0396] Sub-step S90624: concatenate the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame.

[0397] Sub-step S90626: Based on the third acoustic feature corresponding to the target audio frame, determine the second acoustic feature of the target video frame that is input to the acoustic event detector.

[0398] Among them, the sub-steps S90622 to S90626 and the above-mentioned sub-steps S10622 to S10626 are not repeated here.

[0399] In the embodiment of the present application, an implementation of sub-step S90626 may include: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0400] In an embodiment of the present application, another implementation method of sub-step S90626 may include: performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; and using the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0401] In an embodiment of the present application, another implementation method of sub-step S90626 may include: regularizing the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and using the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0402] It should be noted that the embodiment of the present application can adopt any one of the above methods to determine the second acoustic feature corresponding to the target video frame; it can also adopt any two or three of the above methods to determine the second acoustic feature corresponding to the target video frame; the embodiment of the present application does not limit this.

[0403] Step 908: Determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0404] In an example of the present application, an implementation of step 908 may include: averaging the probability information of the target acoustic event corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment.

[0405] In an example of the present application, another implementation method of step 908 may include: selecting the first acoustic features of multiple target batches from the first acoustic features of the multiple batches, averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtaining the probability information of the target acoustic event in the target audio segment.

[0406] In an example of the present application, another implementation method of step 908 may include: score-normalizing the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determining the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0407] In an example of the present application, another implementation of step 908 may include: using a discriminative method to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0408] Step 910: When it is determined that an abnormality has occurred based on the probability information of the target acoustic event in the target audio segment, position information corresponding to the target acoustic event is determined.

[0409] Step 912: Call other information collection devices corresponding to the location information to collect environmental information, and perform abnormality analysis based on the environmental information.

[0410] The probability information of the target acoustic event in the target audio segment can be compared with an eighth probability threshold; if the probability information of the target acoustic event in the target audio segment is greater than or equal to the eighth probability threshold, it can be determined that an abnormality has occurred. If the probability information of the target acoustic event in the target audio segment is less than the eighth probability threshold, it can be determined that no abnormality has occurred. The eighth probability threshold can be set as required and is not limited in this embodiment of the present application.

[0411] When it is determined that an abnormality has occurred, the location information corresponding to the occurrence of the target acoustic event can be determined; wherein the audio acquisition device that collects the target audio segment can be determined, and based on the location information of the audio acquisition device that collects the target audio segment, the location information corresponding to the occurrence of the target acoustic event can be determined. Then, other information acquisition devices corresponding to the location information (such as image acquisition devices, speed acquisition devices, smoke acquisition devices, etc.) are called to collect environmental information; then, an abnormality analysis is performed based on the environmental information to further determine whether an abnormality has occurred. And after determining that an abnormality has occurred, the type of abnormality, such as rear-end collision, fire, gas leak, etc., is analyzed to facilitate subsequent corresponding processing, which is not limited in the embodiments of the present application.

[0412] In summary, after acquiring the audio data collected by the audio acquisition device, the embodiment of the present application can extract the first acoustic feature of the target audio segment from the audio data, the target audio segment includes multiple audio frames, and determine the probability information of the acoustic event corresponding to the first acoustic feature of multiple batches; then, based on the probability information of the acoustic event corresponding to the first acoustic feature of the multiple batches, determine the probability information of the target acoustic event in the target audio segment, and then determine the location information corresponding to the target acoustic event based on the probability information of the target acoustic event in the target audio segment when an abnormality occurs, and call other information acquisition devices corresponding to the location information to collect environmental information, and perform abnormality analysis based on the environmental information; after detecting the abnormality based on the acoustic event, analyze the abnormality in combination with other information, so as to better monitor and analyze the abnormalities in each scene. Among them, the first acoustic feature of a batch includes the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to detect acoustic events, which can improve the accuracy of acoustic event detection, thereby effectively detecting abnormalities.

[0413] In addition, the embodiments of the present application can also be applied to other scenarios that require target acoustic event detection, such as acoustic event detection in unmanned driving scenarios to build an unmanned driving auditory perception system; and gavel sound detection in court, etc., which are not limited by the embodiments of the present application.

[0414] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0415] On the basis of the above embodiments, this embodiment further provides an acoustic event detection device, which is applied to electronic devices such as terminal devices and servers.

[0416] Reference Figure 10 , shows a structural block diagram of an embodiment of an acoustic event detection device of the present application, which may specifically include the following modules:

[0417] A first data acquisition module 1002 is used to acquire audio data to be detected;

[0418] A first extraction module 1004 is configured to extract a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes a plurality of audio frames;

[0419] A first detection module 1006 is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship;

[0420] The first probability determination module 1008 is configured to determine the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0421] Reference Figure 11 , shows a structural block diagram of an optional embodiment of an acoustic event detection device of the present application, which may specifically include the following modules:

[0422] In an optional embodiment of the present application, the first extraction module 1004 is used to extract the first acoustic feature of the audio data to be detected; a window with a preset time length is used to slide on the first acoustic feature of the audio data to be detected according to a set sliding step size to obtain the first acoustic feature of the target audio segment.

[0423] In an optional embodiment of the present application, the first detection module 1006 includes:

[0424] The input submodule 10062 is configured to input the first acoustic feature into the acoustic event detector.

[0425] The probability detection submodule 10064 is configured to call the acoustic event detector to output multiple batches of probability information of acoustic events corresponding to the first acoustic features.

[0426] In an optional embodiment of the present application,

[0427] The probability detection submodule 10064 is used to call the acoustic event detector to divide the first acoustic features of the multiple audio frames into multiple batches of first acoustic features according to a preset time sequence relationship, and to determine and output the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches respectively.

[0428] In an optional embodiment of the present application, the probability detection submodule 10064 is used to divide the first acoustic features of the multiple audio frames into multiple batches of first acoustic features by performing time pooling on the first acoustic features of the multiple audio frames according to a preset timing relationship.

[0429] In an optional embodiment of the present application, the input submodule 10062 includes:

[0430] a feature determining unit 100622, configured to determine second acoustic features corresponding to the plurality of audio frames based on the first acoustic features of the plurality of audio frames;

[0431] The feature input unit 100624 is used to input the second acoustic feature into the acoustic event detector.

[0432] In an optional embodiment of the present application, the feature determination unit 100622 is used to select the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames; splice the first acoustic feature of the target audio frame with the first acoustic features of the first N audio frames of the target audio frame and / or the first acoustic features of the last M audio frames of the target audio frame to obtain a third acoustic feature corresponding to the target audio frame; and determine the second acoustic feature corresponding to the target video frame based on the third acoustic feature corresponding to the target audio frame.

[0433] In an optional embodiment of the present application,

[0434] The feature determination unit 100622 is used to determine the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame; and / or, perform differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; use the third acoustic feature and multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame; and / or, regularize the third acoustic feature corresponding to the target audio frame to obtain the fifth acoustic feature corresponding to the target audio frame, and use the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target video frame.

[0435] In an optional embodiment of the present application, the probability information of the acoustic event includes probability information of the target acoustic event.

[0436] The first probability determination module 1008 is used to average the probability information of the target acoustic events corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic events in the target audio segment; or, select the first acoustic features of multiple target batches from the first acoustic features of the multiple batches, average the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtain the probability information of the target acoustic events in the target audio segment; or, score-normalize the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determine the probability information of the target acoustic events in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches after score normalization; or, use an identification device to determine the probability information of the target acoustic events in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

[0437] In an optional embodiment of the present application, the acoustic event detector includes multiple first preset networks, time pooling layers, second preset networks and third preset networks; the first preset network includes: a fully connected layer, a rectified linear unit, a random dropout layer and a feedforward sequence memory network; the second preset network includes: a fully connected layer, a rectified linear unit and a random dropout layer, and the third preset network includes: a fully connected layer, a rectified linear unit and an activation function.

[0438] In summary, in an embodiment of the present application, after obtaining the audio data to be detected, the first acoustic features of the target audio segment can be extracted from the audio data to be detected, and the probability information of the acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of the acoustic events corresponding to the multiple batches, the probability information of the target acoustic events in the target audio segment is determined; wherein, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset timing relationship, that is, the embodiment of the present application utilizes the timing information of multiple audio frames contained in the audio data to be detected to perform acoustic event detection; thereby, the accuracy of sound event detection can be improved.

[0439] Based on the above embodiment, this embodiment further provides a training device for an acoustic event detector, which is applied to electronic devices such as terminal devices and servers.

[0440] Reference Figure 12 , shows a structural block diagram of an embodiment of a training device for an acoustic event detector of the present application, which may specifically include the following modules:

[0441] A data collection module 1202 is configured to collect audio data of a plurality of target acoustic events and audio data of a plurality of other acoustic events;

[0442] The data expansion module 1204 is configured to divide the audio data of a target acoustic event into multiple audio segments;

[0443] The training module 1206 is configured to train the acoustic event detector by using the multiple audio clips as positive samples and audio data of multiple other acoustic events as negative samples.

[0444] In an optional embodiment of the present application, binary cross entropy is used as a loss function during the training of the acoustic event detector.

[0445] In an optional embodiment of the present application, the positive examples are no less than the negative examples.

[0446] On the basis of the above embodiment, this embodiment further provides a monitoring device based on acoustic event detection, which is applied to a monitoring system. The monitoring system includes an audio acquisition device.

[0447] Reference Figure 13 , shows a structural block diagram of an embodiment of a monitoring device based on acoustic event detection of the present application, which may specifically include the following modules:

[0448] A second data acquisition module 1302 is used to acquire audio data collected by the audio collection device;

[0449] A second extraction module 1304 is configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames;

[0450] A second detection module 1306 is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events, wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship;

[0451] a second probability determination module 1308, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches;

[0452] The first prompt module 1310 is configured to generate prompt information based on probability information of a target acoustic event in the target audio segment.

[0453] In summary, in the embodiment of the present application, after obtaining the audio data collected by the audio acquisition device, the first acoustic features of the target audio segment can be extracted from the audio data, and the probability information of the acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, the probability information of the target acoustic events in the target audio segment can be determined; subsequently, prompt information can be generated based on the probability information of the target acoustic events in the target audio segment; and thus, the detection and monitoring of various target acoustic events in the security monitoring scene can be achieved. Among them, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to detect acoustic events, which can improve the accuracy of acoustic event detection, thereby effectively performing security monitoring.

[0454] On the basis of the above embodiment, this embodiment further provides a monitoring device based on acoustic event detection, which is applied to a monitoring system. The monitoring system includes an audio acquisition device and an image acquisition device.

[0455] Reference Figure 14 , shows a structural block diagram of another embodiment of a monitoring device based on acoustic event detection of the present application, which may specifically include the following modules:

[0456] A third data acquisition module 1402 is used to acquire audio data collected by the audio collection device;

[0457] A third extraction module 1404 is configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames;

[0458] A third detection module 1406 is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship;

[0459] a third probability determination module 1408, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches;

[0460] a fourth data acquisition module 1410, configured to acquire image data captured by the image acquisition device when determining that an abnormality has occurred based on probability information of a target acoustic event in the target audio segment;

[0461] Anomaly confirmation module 1412, configured to perform anomaly confirmation based on the target image segment corresponding to the target audio segment in the image data;

[0462] The second prompt module 1414 is configured to generate prompt information when an abnormality is confirmed based on the target image segment.

[0463] In summary, in an embodiment of the present application, after acquiring audio data collected by an audio acquisition device, first acoustic features of a target audio segment can be extracted from the audio data, and probability information of acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of acoustic events corresponding to the first acoustic features of the multiple batches, probability information of a target acoustic event in the target audio segment can be determined; subsequently, when an abnormality is determined to have occurred based on the probability information of the target acoustic event in the target audio segment, image data collected by the image acquisition device is acquired, and then, based on the target image segment corresponding to the target audio segment in the image data, an abnormality is confirmed, and when an abnormality is confirmed to have occurred based on the target image segment, a prompt message is generated; thereby, by combining audio and image, various target acoustic events in a security monitoring scene can be detected and monitored, thereby reducing false alarms and user disturbances. In particular, a batch of first acoustic features includes first acoustic features of multiple audio frames with a preset time sequence relationship. That is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data for acoustic event detection, which can improve the accuracy of acoustic event detection, thereby effectively performing security monitoring.

[0464] On the basis of the above embodiment, this embodiment further provides a monitoring device based on acoustic event detection, which is applied to a vehicle-mounted monitoring system. The vehicle-mounted monitoring system includes an audio acquisition device, an image acquisition device, and a vehicle information acquisition device.

[0465] Reference Figure 15, shows a structural block diagram of another embodiment of a monitoring device based on acoustic event detection of the present application, which may specifically include the following modules:

[0466] A fifth data acquisition module 1502 is configured to acquire audio data collected by the audio collection device;

[0467] a fourth extraction module 1504, configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames;

[0468] A fourth detection module 1506 is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events, wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship;

[0469] a fourth probability determination module 1508, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches;

[0470] a sixth data acquisition module 1510 for acquiring image data acquired by the image acquisition device and vehicle information acquired by the vehicle information acquisition device when determining that an abnormality has occurred based on the probability information of the target acoustic event in the target audio segment;

[0471] Anomaly type determination module 1512 is configured to determine an anomaly type of the anomaly based on the image data and vehicle information; the anomaly type includes an internal vehicle anomaly type or an external vehicle anomaly type;

[0472] The third prompt module 1514 is configured to generate corresponding prompt information according to the exception type of the exception.

[0473] In summary, the embodiments of the present application can be applied to a vehicle-mounted monitoring system, which includes an audio acquisition device, an image acquisition device, and a vehicle information acquisition device. The vehicle-mounted monitoring system can obtain audio data collected by the audio acquisition device, and extract the first acoustic feature of the target audio segment from the audio data, and then determine the probability information of the acoustic event corresponding to the first acoustic feature of multiple batches; then, based on the probability information of the acoustic events corresponding to the multiple batches, determine the probability information of the target acoustic event in the target audio segment; subsequently, when determining that an abnormality has occurred based on the probability information of the target acoustic event in the target audio segment, image data collected by the image acquisition device in the monitoring system and vehicle information collected by the vehicle information acquisition device can be obtained, and based on the image data and vehicle information, the abnormality type of the abnormality can be determined; the abnormality type includes an internal vehicle abnormality type or an external vehicle abnormality type, and then a corresponding prompt information is generated based on the abnormality type of the abnormality; thereby, by combining audio, image, and vehicle information, various target acoustic events in the vehicle monitoring scene can be detected and monitored, thereby reducing false alarms and reducing disturbance to users. Among them, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship. That is to say, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to detect acoustic events, which can improve the accuracy of acoustic event detection and thus effectively monitor vehicle safety.

[0474] Based on the above embodiment, this embodiment further provides a device for identifying illegal segments based on acoustic event detection.

[0475] Reference Figure 16 , shows a structural block diagram of an embodiment of a device for identifying illegal segments based on acoustic event detection of the present application, which may specifically include the following modules:

[0476] The seventh data acquisition module 1602 is used to acquire audio data to be detected;

[0477] a fifth extraction module 1604, configured to extract a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes a plurality of audio frames;

[0478] A fifth detection module 1606 is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events, wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship;

[0479] a fifth probability determination module 1608, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches;

[0480] The violation determination module 1610 is configured to determine whether the target audio segment is a violation segment based on probability information of the violation acoustic events in the target audio segment.

[0481] In summary, in an embodiment of the present application, after obtaining the audio data played in the audio and video platform, the first acoustic features of the target audio segment can be extracted from the audio data to be detected, and the probability information of the acoustic events corresponding to the first acoustic features of multiple batches can be determined; then, based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, the probability information of the target acoustic events in the target audio segment is determined; and then, based on the probability information of the illegal acoustic events in the target audio segment, it is judged whether the target audio segment is an illegal segment; wherein, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to be detected to perform acoustic event detection, which can improve the accuracy of acoustic event detection, thereby effectively identifying illegal segments.

[0482] Based on the above embodiment, this embodiment further provides a segment search device based on acoustic event detection.

[0483] Reference Figure 17 , shows a structural block diagram of an embodiment of a segment search device based on acoustic event detection of the present application, which may specifically include the following modules:

[0484] Receiving module 1702, used for receiving query information;

[0485] The data determination module 1704 is configured to determine the audio data to be queried and the acoustic event to be queried corresponding to the query information;

[0486] a sixth extraction module 1706, configured to extract a first acoustic feature of a target audio segment from the audio data to be queried, where the target audio segment includes a plurality of audio frames;

[0487] A sixth detection module 1708 is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events, wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship;

[0488] a sixth probability determination module 1710, configured to determine probability information of an acoustic event to be queried in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches;

[0489] The returning module 1712 is configured to return the target audio segment based on the probability information of the acoustic event to be queried in the target audio segment.

[0490] In summary, after receiving the query information, the embodiment of the present application can determine the audio data to be queried and the acoustic event to be queried corresponding to the query information; then extract the first acoustic feature of the target audio segment from the audio data to be queried, and determine the probability information of the acoustic event corresponding to the first acoustic features of multiple batches; then determine the probability information of the acoustic event to be queried in the target audio segment based on the probability information of the acoustic event corresponding to the first acoustic features of the multiple batches, and return the target audio segment based on the probability information of the acoustic event to be queried in the target audio segment; wherein, the first acoustic features of a batch include the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to be queried to perform acoustic event detection, which can improve the accuracy of acoustic event detection, thereby searching for segments that better meet user needs, improving the search quality, and also improving user experience.

[0491] Based on the above embodiment, this embodiment further provides an abnormality analysis device based on acoustic event detection.

[0492] Reference Figure 18 , shows a structural block diagram of an embodiment of an abnormality analysis device based on acoustic event detection of the present application, which may specifically include the following modules:

[0493] An eighth data acquisition module 1802 is configured to acquire audio data collected by an audio collection device;

[0494] a seventh extraction module 1804, configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames;

[0495] A seventh detection module 1806 is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events, wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship;

[0496] a seventh probability determination module 1808, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches;

[0497] a position determination module 1810 for determining position information corresponding to a target acoustic event when an abnormality is determined based on probability information of the target acoustic event in the target audio segment;

[0498] The analysis module 1812 is used to call other information collection devices corresponding to the location information to collect environmental information and perform abnormality analysis based on the environmental information.

[0499] In summary, after acquiring the audio data collected by the audio acquisition device, the embodiment of the present application can extract the first acoustic feature of the target audio segment from the audio data, the target audio segment includes multiple audio frames, and determine the probability information of the acoustic event corresponding to the first acoustic feature of multiple batches; then, based on the probability information of the acoustic event corresponding to the first acoustic feature of the multiple batches, determine the probability information of the target acoustic event in the target audio segment, and then determine the location information corresponding to the target acoustic event based on the probability information of the target acoustic event in the target audio segment when an abnormality occurs, and call other information acquisition devices corresponding to the location information to collect environmental information, and perform abnormality analysis based on the environmental information; after detecting the abnormality based on the acoustic event, analyze the abnormality in combination with other information, so as to better monitor and analyze the abnormalities in each scene. Among them, the first acoustic feature of a batch includes the first acoustic features of multiple audio frames with a preset time sequence relationship, that is, the embodiment of the present application utilizes the time sequence information of multiple audio frames contained in the audio data to detect acoustic events, which can improve the accuracy of acoustic event detection, thereby effectively detecting abnormalities.

[0500] An embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0501] The present application provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In the present application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0502] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 19 An exemplary apparatus 1900 that can be used to implement various embodiments described herein is schematically illustrated.

[0503] For one embodiment, Figure 19An exemplary apparatus 1900 is shown having one or more processors 1902, a control module (chip set) 1904 coupled to at least one of the processor(s) 1902, a memory 1906 coupled to the control module 1904, a non-volatile memory (NVM) / storage device 1908 coupled to the control module 1904, one or more input / output devices 1910 coupled to the control module 1904, and a network interface 1912 coupled to the control module 1904.

[0504] The processor 1902 may include one or more single-core or multi-core processors, and the processor 1902 may include any combination of general-purpose processors or dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 1900 can function as a terminal device, server (cluster), or other device described in the embodiments of the present application.

[0505] In some embodiments, the apparatus 1900 may include one or more computer-readable media (e.g., memory 1906 or NVM / storage 1908) having instructions 1914 and one or more processors 1902 configured in conjunction with the one or more computer-readable media to execute the instructions 1914 to implement a module to perform the actions described in the present disclosure.

[0506] For one embodiment, the control module 1904 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1902 and / or any suitable device or component in communication with the control module 1904 .

[0507] The control module 1904 may include a memory controller module to provide an interface to the memory 1906. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0508] The memory 1906 can be used, for example, to load and store data and / or instructions 1914 for the device 1900. For one embodiment, the memory 1906 can include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1906 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0509] For one embodiment, the control module 1904 may include one or more input / output controllers to provide an interface to the NVM / storage device 1908 and the input / output device(s) 1910 .

[0510] For example, NVM / storage 1908 may be used to store data and / or instructions 1914. NVM / storage 1908 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0511] NVM / storage device 1908 may include storage resources that are physically part of the device on which apparatus 1900 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1908 may be accessible over a network via input / output device(s) 1910.

[0512] (One or more) input / output devices 1910 may provide an interface for apparatus 1900 to communicate with any other appropriate device. Input / output devices 1910 may include communication components, audio components, sensor components, etc. Network interface 1912 may provide an interface for apparatus 1900 to communicate via one or more networks. Apparatus 1900 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0513] For one embodiment, at least one of the processor(s) 1902 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1904. For one embodiment, at least one of the processor(s) 1902 may be packaged together with the logic of one or more controllers of the control module 1904 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1902 may be integrated on the same die with the logic of one or more controllers of the control module 1904. For one embodiment, at least one of the processor(s) 1902 may be integrated on the same die with the logic of one or more controllers of the control module 1904 to form a system-on-chip (SoC).

[0514] In various embodiments, apparatus 1900 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, apparatus 1900 may have more or fewer components and / or a different architecture. For example, in some embodiments, apparatus 1900 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0515] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in the memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0516] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0517] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0518] The present application embodiment is described with reference to the flow chart and / or block diagram of the method, terminal device (system), and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flow chart and / or block diagram and the combination of the process and / or box in the flow chart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for realizing the function specified in one process or multiple processes and / or one box or multiple boxes of the flow chart.

[0519] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0520] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0521] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0522] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0523] The above is a detailed introduction to an acoustic event detection method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting acoustic events, characterized in that: The method comprises: Get the audio data to be detected; Extracting a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes a plurality of audio frames; Determining probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset time sequence relationship; Determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; The step of extracting the first acoustic feature of the target audio segment from the audio data to be detected includes: Extracting a first acoustic feature of the audio data to be detected; A window with a preset duration is used to slide on the first acoustic feature of the audio data to be detected according to a set sliding step, so as to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

2. The method according to claim 1, characterized in that Determining the probability information of the first acoustic features of the plurality of batches corresponding to the acoustic events includes: The first acoustic feature is input into an acoustic event detector, and the acoustic event detector outputs multiple batches of probability information of acoustic events corresponding to the first acoustic feature.

3. The method according to claim 2, characterized in that The acoustic event detector outputs multiple batches of probability information of the first acoustic features corresponding to the acoustic events, including: The acoustic event detector divides the first acoustic features of the multiple audio frames into multiple batches of first acoustic features according to a preset time sequence relationship, and determines and outputs probability information of the first acoustic features of the multiple batches corresponding to acoustic events.

4. The method according to claim 3, characterized in that The dividing the first acoustic features of the plurality of audio frames into a plurality of batches of first acoustic features according to a preset time sequence relationship includes: By performing time pooling on the first acoustic features of the multiple audio frames according to a preset temporal relationship, the first acoustic features of the multiple audio frames are divided into multiple batches of first acoustic features.

5. The method according to claim 2, characterized in that Inputting the first acoustic feature into an acoustic event detector comprises: Determining second acoustic features corresponding to the multiple audio frames respectively based on the first acoustic features of the multiple audio frames; The second acoustic feature is input to the acoustic event detector.

6. The method according to claim 5, characterized in that The determining, based on the first acoustic features of the multiple audio frames, respectively the second acoustic features corresponding to the multiple audio frames includes: Selecting the first acoustic feature of the target audio frame from the first acoustic features of the multiple audio frames; concatenating the first acoustic feature of the target audio frame with the first acoustic features of the N audio frames preceding the target audio frame and / or the first acoustic features of the M audio frames following the target audio frame to obtain a third acoustic feature corresponding to the target audio frame; Based on the third acoustic feature corresponding to the target audio frame, a second acoustic feature corresponding to the target audio frame is determined.

7. The method according to claim 6, characterized in that The determining, based on the third acoustic feature corresponding to the target audio frame, the second acoustic feature corresponding to the target audio frame includes: determining the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target audio frame; and / or, performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain multiple fourth acoustic features corresponding to the target audio frame; using the third acoustic feature and the multiple fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target audio frame; and / or, The third acoustic feature corresponding to the target audio frame is regularized to obtain a fifth acoustic feature corresponding to the target audio frame, and the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame are used as the second acoustic feature corresponding to the target audio frame.

8. The method according to claim 1, characterized in that The probability information of the acoustic event includes probability information of a target acoustic event, and determining the probability information of the target acoustic event in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches includes: Averaging the probability information of the target acoustic events corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic event in the target audio segment; or, Selecting multiple target batches of first acoustic features from the multiple batches of first acoustic features, averaging the probability information of target acoustic events corresponding to the multiple target batches of first acoustic features, and obtaining the probability information of the target acoustic event in the target audio segment; or Score-normalize the probability information of acoustic events corresponding to the first acoustic features of the multiple batches, and determine the probability information of the target acoustic event in the target audio segment based on the probability information after the score normalization of the acoustic events corresponding to the first acoustic features of the multiple batches; or The probability information of the target acoustic event in the target audio segment is determined by using a discriminative method based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

9. The method according to any one of claims 1 to 8, characterized in that: The acoustic event detector includes a plurality of first preset networks, a time pooling layer, a second preset network and a third preset network; The first preset network includes: a fully connected layer, a rectified linear unit, a random dropout layer and a feedforward sequence memory network; the second preset network includes: a fully connected layer, a rectified linear unit and a random dropout layer, and the third preset network includes: a fully connected layer, a rectified linear unit and an activation function.

10. The method according to claim 2, characterized in that The acoustic event detector is trained by: collecting audio data of a plurality of target acoustic events and audio data of a plurality of other acoustic events; For audio data of a target acoustic event, dividing the audio data of the target acoustic event into multiple audio segments; The acoustic event detector is trained by using the multiple audio clips as positive samples and audio data of multiple other acoustic events as negative samples.

11. The method according to claim 10, characterized in that In the process of training the acoustic event detector, binary cross entropy is used as the loss function.

12. The method according to claim 10, characterized in that The positive examples are no less than the negative examples.

13. A monitoring method based on acoustic event detection, characterized in that: Applied in a monitoring system, the monitoring system includes an audio acquisition device, and the method includes: Acquiring audio data collected by the audio collection device; Extracting a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; Determining probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset time sequence relationship; Determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; generating prompt information based on probability information of a target acoustic event in the target audio segment; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

14. A monitoring method based on acoustic event detection, characterized in that: Applied in a monitoring system, the monitoring system includes an audio acquisition device and an image acquisition device, and the method includes: Acquiring audio data collected by the audio collection device; Extracting a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; Determining probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset time sequence relationship; Determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; When determining that an abnormality occurs based on probability information of a target acoustic event in the target audio segment, acquiring image data captured by the image capture device; performing abnormality confirmation based on a target image segment corresponding to the target audio segment in the image data; When an abnormality is confirmed based on the target image segment, a prompt message is generated; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

15. A monitoring method based on acoustic event detection, characterized in that: Applied to a vehicle-mounted monitoring system, the vehicle-mounted monitoring system includes an audio acquisition device, an image acquisition device, and a vehicle information acquisition device, and the method includes: Acquiring audio data collected by the audio collection device; Extracting a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; Determining probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset time sequence relationship; Determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; When determining that an abnormality occurs based on probability information of a target acoustic event in the target audio segment, acquiring image data captured by the image capture device and vehicle information captured by the vehicle information capture device; Determining the abnormality type of the abnormality based on the image data and the vehicle information; the abnormality type includes an internal abnormality type of the vehicle or an external abnormality type of the vehicle; Generate corresponding prompt information according to the exception type of the exception; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

16. A method for identifying illegal segments based on acoustic event detection, characterized in that: The method includes: Get the audio data to be detected; Extracting a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes a plurality of audio frames; Determining probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset time sequence relationship; Determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; Determining whether the target audio segment is a violation segment based on probability information of the violation acoustic event in the target audio segment; The step of extracting the first acoustic feature of the target audio segment from the audio data to be detected includes: Extracting a first acoustic feature of the audio data to be detected; A window with a preset duration is used to slide on the first acoustic feature of the audio data to be detected according to a set sliding step, so as to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

17. A fragment search method based on acoustic event detection, characterized in that: The method includes: Receive query information; Determining the audio data to be queried and the acoustic event to be queried corresponding to the query information; Extracting a first acoustic feature of a target audio segment from the audio data to be queried, where the target audio segment includes a plurality of audio frames; Determining probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset time sequence relationship; Determining probability information of an acoustic event to be queried in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; Returning the target audio segment based on the probability information of the acoustic event to be queried in the target audio segment; The step of extracting the first acoustic feature of the target audio segment from the audio data to be queried includes: Extracting a first acoustic feature of the audio data to be queried; A window with a preset duration is used to slide on the first acoustic feature of the audio data to be queried according to a set sliding step, so as to intercept the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the acoustic event to be queried.

18. An abnormality analysis method based on acoustic event detection, characterized in that: The method includes: Obtain audio data collected by the audio collection device; Extracting a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; Determining probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset time sequence relationship; Determining probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; When determining that an abnormality occurs based on probability information of a target acoustic event in the target audio segment, determining position information corresponding to the target acoustic event; Calling other information collection devices corresponding to the location information to collect environmental information, and performing abnormality analysis based on the environmental information; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

19. An acoustic event detection device, characterized in that: The device comprises: A first data acquisition module, configured to acquire audio data to be detected; A first extraction module is configured to extract a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes a plurality of audio frames; A first detection module is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship; a first probability determination module, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; Among them, the first extraction module is used to extract the first acoustic feature of the audio data to be detected; use a window with a preset duration to slide on the first acoustic feature of the audio data to be detected according to a set sliding step size, and obtain the first acoustic feature of the target audio segment; wherein, the preset duration is determined according to the duration corresponding to the target acoustic event.

20. The device according to claim 19, characterized in that The first detection module includes: an input submodule, configured to input the first acoustic feature into an acoustic event detector, The probability detection submodule is used to call the acoustic event detector to output multiple batches of probability information of acoustic events corresponding to the first acoustic features.

21. The device according to claim 20, characterized in that The probability detection submodule is used to call the acoustic event detector to divide the first acoustic features of the multiple audio frames into multiple batches of first acoustic features according to a preset time sequence relationship, and to determine and output the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches respectively.

22. The device according to claim 21, characterized in that The probability detection submodule is configured to divide the first acoustic features of the multiple audio frames into multiple batches of first acoustic features by performing time pooling on the first acoustic features of the multiple audio frames according to a preset time sequence relationship.

23. The device according to claim 20, characterized in that The input submodule includes: a feature determining unit, configured to determine second acoustic features corresponding to the plurality of audio frames respectively based on the first acoustic features of the plurality of audio frames; A feature input unit is used to input the second acoustic feature into the acoustic event detector.

24. The device according to claim 23, characterized in that The feature determination unit is configured to select a first acoustic feature of a target audio frame from the first acoustic features of the multiple audio frames; and concatenate the first acoustic feature of the target audio frame with the first acoustic features of the N audio frames preceding the target audio frame and / or the first acoustic features of the M audio frames following the target audio frame to obtain a third acoustic feature corresponding to the target audio frame. Based on the third acoustic feature corresponding to the target audio frame, a second acoustic feature corresponding to the target audio frame is determined.

25. The device according to claim 24, characterized in that The feature determining unit is configured to determine the third acoustic feature corresponding to the target audio frame as the second acoustic feature corresponding to the target audio frame; and / or performing differential processing on the third acoustic feature corresponding to the target audio frame to obtain a plurality of fourth acoustic features corresponding to the target audio frame; using the third acoustic feature and the plurality of fourth acoustic features corresponding to the target audio frame as the second acoustic feature corresponding to the target audio frame; And / or, the third acoustic feature corresponding to the target audio frame is regularized to obtain the fifth acoustic feature corresponding to the target audio frame, and the third acoustic feature and the fifth acoustic feature corresponding to the target audio frame are used as the second acoustic feature corresponding to the target audio frame.

26. The device according to claim 19, characterized in that The probability information of the acoustic event includes probability information of the target acoustic event, The first probability determination module is used to average the probability information of the target acoustic events corresponding to the first acoustic features of the multiple batches to obtain the probability information of the target acoustic events in the target audio segment; or, select the first acoustic features of multiple target batches from the first acoustic features of the multiple batches, average the probability information of the target acoustic events corresponding to the first acoustic features of the multiple target batches, and obtain the probability information of the target acoustic events in the target audio segment; or, score-normalize the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches, and determine the probability information of the target acoustic events in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches after score normalization; or, use an identification device to determine the probability information of the target acoustic events in the target audio segment based on the probability information of the acoustic events corresponding to the first acoustic features of the multiple batches.

27. The device according to any one of claims 19 to 26, characterized in that: The acoustic event detector includes a plurality of first preset networks, a time pooling layer, a second preset network and a third preset network; The first preset network includes: a fully connected layer, a rectified linear unit, a random dropout layer and a feedforward sequence memory network; the second preset network includes: a fully connected layer, a rectified linear unit and a random dropout layer, and the third preset network includes: a fully connected layer, a rectified linear unit and an activation function.

28. The device according to claim 20, characterized in that The acoustic event detector is trained using the following modules: a data collection module, configured to collect audio data of a plurality of target acoustic events and audio data of a plurality of other acoustic events; a data expansion module, configured to divide the audio data of a target acoustic event into a plurality of audio segments; A training module is configured to train the acoustic event detector by using the multiple audio clips as positive samples and audio data of multiple other acoustic events as negative samples.

29. The device according to claim 28, characterized in that In the process of training the acoustic event detector, binary cross entropy is used as the loss function.

30. The device according to claim 28, wherein The positive examples are no less than the negative examples.

31. A monitoring device based on acoustic event detection, characterized in that: Applied in a monitoring system, the monitoring system includes an audio acquisition device, and the device includes: A second data acquisition module is used to acquire the audio data collected by the audio acquisition device; a second extraction module, configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; A second detection module is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship; a second probability determination module, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a first prompt module, configured to generate prompt information based on probability information of a target acoustic event in the target audio segment; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

32. A monitoring device based on acoustic event detection, characterized in that: Applied in a monitoring system comprising an audio acquisition device and an image acquisition device, the device comprises: A third data acquisition module is used to acquire the audio data collected by the audio acquisition device; a third extraction module, configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; A third detection module is configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship; a third probability determination module, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a fourth data acquisition module, configured to acquire image data captured by the image acquisition device when determining that an abnormality has occurred based on probability information of a target acoustic event in the target audio segment; an abnormality confirmation module, configured to perform abnormality confirmation based on a target image segment corresponding to the target audio segment in the image data; A second prompt module is configured to generate prompt information when an abnormality is confirmed based on the target image segment; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

33. A monitoring device based on acoustic event detection, characterized in that: Applied to a vehicle-mounted monitoring system, the vehicle-mounted monitoring system includes an audio acquisition device, an image acquisition device, and a vehicle information acquisition device, and the device includes: A fifth data acquisition module, configured to acquire audio data collected by the audio acquisition device; a fourth extraction module, configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; a fourth detection module, configured to determine probability information of a plurality of batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of a plurality of audio frames having a preset temporal relationship; a fourth probability determination module, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a sixth data acquisition module, configured to acquire image data acquired by the image acquisition device and vehicle information acquired by the vehicle information acquisition device when determining that an abnormality has occurred based on probability information of a target acoustic event in the target audio segment; An abnormality type determination module, configured to determine the abnormality type of the abnormality based on the image data and vehicle information; the abnormality type includes an internal abnormality type of the vehicle or an external abnormality type of the vehicle; A third prompt module is used to generate corresponding prompt information according to the abnormality type of the abnormality; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

34. A device for identifying illegal segments based on acoustic event detection, characterized in that: The device comprises: A seventh data acquisition module, configured to acquire audio data to be detected; a fifth extraction module, configured to extract a first acoustic feature of a target audio segment from the audio data to be detected, where the target audio segment includes a plurality of audio frames; a fifth detection module, configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship; a fifth probability determination module, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a violation judgment module, configured to judge whether the target audio segment is a violation segment based on probability information of a violation acoustic event in the target audio segment; The step of extracting the first acoustic feature of the target audio segment from the audio data to be detected includes: Extracting a first acoustic feature of the audio data to be detected; A window with a preset duration is used to slide on the first acoustic feature of the audio data to be detected according to a set sliding step, so as to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

35. A fragment search device based on acoustic event detection, characterized in that: The device comprises: A receiving module, configured to receive query information; A data determination module, configured to determine the audio data to be queried and the acoustic event to be queried corresponding to the query information; a sixth extraction module, configured to extract a first acoustic feature of a target audio segment from the audio data to be queried, where the target audio segment includes a plurality of audio frames; a sixth detection module, configured to determine probability information of a plurality of batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of a plurality of audio frames having a preset temporal relationship; a sixth probability determination module, configured to determine probability information of an acoustic event to be queried in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a returning module, configured to return the target audio segment based on probability information of the acoustic event to be queried in the target audio segment; The step of extracting the first acoustic feature of the target audio segment from the audio data to be queried includes: Extracting a first acoustic feature of the audio data to be queried; A window with a preset duration is used to slide on the first acoustic feature of the audio data to be queried according to a set sliding step, so as to intercept the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the acoustic event to be queried.

36. An abnormality analysis device based on acoustic event detection, characterized in that: The device comprises: An eighth data acquisition module, configured to acquire audio data collected by an audio acquisition device; a seventh extraction module, configured to extract a first acoustic feature of a target audio segment from the audio data, where the target audio segment includes a plurality of audio frames; a seventh detection module, configured to determine probability information of multiple batches of first acoustic features corresponding to acoustic events; wherein a batch of first acoustic features includes first acoustic features of multiple audio frames having a preset temporal relationship; a seventh probability determination module, configured to determine probability information of a target acoustic event in the target audio segment based on probability information of acoustic events corresponding to the first acoustic features of the multiple batches; a position determination module, configured to determine position information corresponding to the target acoustic event when an abnormality is determined based on probability information of the target acoustic event in the target audio segment; An analysis module, configured to call other information collection devices corresponding to the location information to collect environmental information, and perform abnormality analysis based on the environmental information; The step of extracting the first acoustic feature of the target audio segment from the audio data includes: extracting a first acoustic feature of the audio data; A window with a preset duration is used to slide on the first acoustic feature of the audio data according to a set sliding step to obtain the first acoustic feature of the target audio segment; wherein the preset duration is determined according to the duration corresponding to the target acoustic event.

37. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to perform the method according to any one of claims 1 to 18.

38. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method of any one of claims 1-18.

Images