Bullying incident detection method and system based on acoustic event recognition and emotion recognition

Abstract

The present application provides a bullying incident detection method and system based on acoustic event recognition and emotion recognition. On the one hand, the bullying incident detection method can capture sound signals in every corner of the campus through multiple acoustic sensors deployed in the target campus, thereby achieving full coverage of the target campus from the perspective of data collection, and adopts an edge computing processing method to only process the collected voice data in the edge computing device, without sending or displaying any collected voice data, thereby improving data processing efficiency while effectively protecting the personal privacy of campus teachers and students; on the other hand, by analyzing voice data from multiple dimensions such as emotion recognition, bullying sound event detection, and voice recognition, it can be comprehensively determined whether a bullying incident is detected at the voice data collection location (i.e., the deployment location of the acoustic sensor), thereby effectively improving the accuracy of the bullying incident detection results.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular, to a method and system for detecting bullying incidents based on acoustic event recognition and emotion recognition. Background Art

[0002] Existing methods for detecting bullying incidents usually adopt a method of combining voice data with image data. When bullying behavior is identified in the image data or verbal bullying is identified in the voice data, it is determined that a bullying incident has been detected, and the persons involved in the image data are identified as persons involved in the bullying incident, so that subsequent investigations and handling of the bullying incident can be carried out against the above-mentioned persons involved.

[0003] However, since bullying incidents on campus often occur in private areas that are not covered by surveillance, such as dormitories and toilets, the above-mentioned existing bullying incident detection methods that rely on image data are unable to efficiently and accurately detect bullying incidents that may occur on campus. Summary of the invention

[0004] In view of this, the present application provides a bullying incident detection method and system based on acoustic event recognition and emotion recognition. On the one hand, by deploying multiple acoustic sensors in the target campus, sound signals in every corner of the campus can be captured, thereby achieving full coverage of the target campus from the perspective of data collection, and adopting the edge computing processing method, only processing the collected voice data in the edge computing device, and not sending and displaying any collected voice data, thereby improving the data processing efficiency while effectively protecting the personal privacy of campus teachers and students; on the other hand, by analyzing voice data from multiple dimensions such as emotion recognition, bullying sound event detection, and voice recognition, it can be comprehensively determined whether a bullying incident is detected at the voice data collection location (i.e., the deployment location of the acoustic sensor), thereby effectively improving the accuracy of the bullying incident detection results.

[0005] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0006] In a first aspect, an embodiment of the present application provides a bullying incident detection method based on acoustic event recognition and emotion recognition, which is applied to an edge computing device, wherein the edge computing device and a plurality of acoustic sensors deployed in a target campus perform data transmission according to a preset data transmission protocol, and the bullying incident detection method includes:

[0007] Receive the voice data collected by each of the acoustic sensors, and pre-process each of the received voice data to obtain target audio data corresponding to each of the voice data; wherein each voice information in the target audio data carries a speaker label corresponding to the voice information;

[0008] For each of the target audio data, when it is detected that there are preset keywords related to the bullying incident in the target audio data, emotion recognition is performed on the target audio data to obtain an emotion recognition result corresponding to the target audio data;

[0009] For each of the target audio data, respectively calculate the speech similarity between the target audio data and the acoustic event templates of multiple bullying events, and obtain the highest value of the speech similarity in the calculation results as the bullying acoustic event detection result corresponding to the target audio data;

[0010] For each of the target audio data, determining whether the target audio data contains voice information related to the bullying incident, and obtaining a determination result as a voice recognition result of the target audio data;

[0011] For each of the target audio data, based on the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data, the bullying event detection result corresponding to the deployment location of the target acoustic sensor is comprehensively determined; wherein the target acoustic sensor represents the acoustic sensor corresponding to the target audio data.

[0012] In a second aspect, an embodiment of the present application provides a bullying incident detection system based on acoustic event recognition and emotion recognition, the bullying incident detection system comprising: an edge computing device and a plurality of acoustic sensors deployed in a target campus; wherein the edge computing device and the plurality of acoustic sensors perform data transmission according to a preset data transmission protocol, and the edge computing device is used to:

[0013] Receive the voice data collected by each of the acoustic sensors, and pre-process each of the received voice data to obtain target audio data corresponding to each of the voice data; wherein each voice information in the target audio data carries a speaker label corresponding to the voice information;

[0014] For each of the target audio data, when it is detected that there are preset keywords related to the bullying incident in the target audio data, emotion recognition is performed on the target audio data to obtain an emotion recognition result corresponding to the target audio data;

[0015] For each of the target audio data, respectively calculate the speech similarity between the target audio data and the acoustic event templates of multiple bullying events, and obtain the highest value of the speech similarity in the calculation results as the bullying acoustic event detection result corresponding to the target audio data;

[0016] For each of the target audio data, determining whether the target audio data contains voice information related to the bullying incident, and obtaining a determination result as a voice recognition result of the target audio data;

[0017] For each of the target audio data, based on the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data, the bullying event detection result corresponding to the deployment location of the target acoustic sensor is comprehensively determined; wherein the target acoustic sensor represents the acoustic sensor corresponding to the target audio data.

[0018] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned bullying incident detection method based on acoustic event recognition and emotion recognition when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned bullying incident detection method based on acoustic event recognition and emotion recognition are executed.

[0020] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0021] The embodiments of the present application provide a method and system for detecting bullying incidents based on acoustic event recognition and emotion recognition. On the one hand, multiple acoustic sensors deployed in the target campus can capture sound signals from every corner of the campus, thereby achieving full coverage of the target campus from the perspective of data collection. In addition, the edge computing processing method is adopted to process the collected voice data only in the edge computing device, and no collected voice data is sent or displayed. This improves data processing efficiency while effectively protecting the personal privacy of teachers and students on campus. On the other hand, by analyzing voice data from multiple dimensions such as emotion recognition, bullying sound event detection, and voice recognition, it is possible to comprehensively determine whether a bullying incident is detected at the voice data collection location (i.e., the deployment location of the acoustic sensor), thereby effectively improving the accuracy of the bullying incident detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A bullying incident detection system based on acoustic event recognition and emotion recognition provided by an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram of a process of a bullying incident detection method based on acoustic event recognition and emotion recognition provided by an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram of a flow chart of a method for preprocessing speech data provided in an embodiment of the present application is shown;

[0026] Figure 4 A schematic diagram of a process for performing emotion recognition on target audio data provided by an embodiment of the present application is shown;

[0027] Figure 5 A flow chart of a method for controlling the target acoustic sensor to play intervention voice information for a bullying incident according to a preset intervention voice template provided in an embodiment of the present application is shown;

[0028] Figure 6 A flow chart of a method for determining persons involved in a bullying incident through voiceprint comparison provided in an embodiment of the present application is shown;

[0029] Figure 7 A schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0031] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0032] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0033] Here, in the embodiment of the present application, it is taken as an example that the place where bullying incident detection is required is the target campus. Figure 1 A bullying incident detection system based on acoustic event recognition and emotion recognition provided by an embodiment of the present application is shown. Figure 1 As shown, the bullying incident detection system includes: an edge computing device 100 and multiple acoustic sensors 101 deployed in a target campus (i.e., a detection site for bullying incidents); wherein, the multiple acoustic sensors 101 deployed in the target campus can capture sound signals from every corner of the campus (i.e., real-time collection of voice data), thereby achieving full coverage of the target campus from a data collection perspective, and the edge computing device 100 can be connected to the same local area network as the multiple acoustic sensors 101, so that the edge computing device 100 can perform data transmission with each acoustic sensor 101 in the connected local area network according to a preset data transmission protocol, receive the voice data sent by each acoustic sensor 101 in real time, and perform parallel processing on the voice data sent by each acoustic sensor 101, thereby detecting whether a bullying incident occurs at the deployment location of each acoustic sensor 101 based on the voice data sent by each acoustic sensor 101.

[0034] It should be noted that, in order to achieve full coverage of the target campus, the above-mentioned multiple acoustic sensors 101 can be deployed and installed in various corners of the target campus. The embodiment of the present application does not impose any limitation on the specific deployment location of each acoustic sensor 101 in the target campus.

[0035] It should be noted that, in order to improve the data collection quality of voice data, a high-sensitivity microphone array can be preferably used as the above-mentioned acoustic sensor 101. The embodiment of the present application does not impose any limitation on the specific device type of the above-mentioned acoustic sensor 101.

[0036] It should be noted that since the edge computing device 100 and each acoustic sensor 101 in the target campus are connected to the same local area network, when detecting bullying incidents on the voice data collected and sent by each acoustic sensor 101, the edge computing device 100 can perform edge computing on each received voice data (equivalent to the edge computing device 100 being close to the data source of the voice data), and provide bullying incident detection services for the target campus nearby, which can effectively reduce the time delay generated during data transmission and improve the detection efficiency of bullying incidents; in addition, since the embodiment of the present application only processes the collected voice data in the edge computing device 100, and does not send and display any collected voice data, it can improve the data processing efficiency while also effectively protecting the personal privacy of teachers and students on campus.

[0037] It should be noted that the above-mentioned target campus can be a primary school campus, a middle school campus or a university campus, etc. The embodiment of the present application does not impose any limitation on the specific campus type of the above-mentioned target campus.

[0038] To facilitate understanding of the embodiments of the present application, a bullying incident detection method and system based on acoustic event recognition and emotion recognition provided in the embodiments of the present application are introduced in detail below.

[0039] Reference Figure 2 As shown, Figure 2 The flowchart of a bullying incident detection method based on acoustic event recognition and emotion recognition provided by an embodiment of the present application is shown, wherein the bullying incident detection method can be applied to Figure 1 In the edge computing device 100 shown, the edge computing device 100 and multiple acoustic sensors 101 deployed in the target campus perform data transmission according to a preset data transmission protocol. The bullying incident detection method includes steps S201-S205; specifically:

[0040] S201, receiving the voice data collected by each of the acoustic sensors, and preprocessing each of the received voice data to obtain target audio data corresponding to each of the voice data.

[0041] Here, refer to Figure 1 In the bullying incident detection system shown, the edge computing device 100 can receive the voice data sent by each acoustic sensor 101 in real time, and process the voice data sent by each acoustic sensor 101 in parallel, that is, when executing step S201, the edge computing device 100 can pre-process each received voice data in parallel, wherein the edge computing device 100 pre-processes each voice data in the same way.

[0042] Specifically, in the prior art solution (i.e., the method of combining voice data with image data), voice data is usually directly identified and analyzed in bullying scenarios. However, considering that there may be multiple noise sources in the campus environment (such as noisy crowds, machine operation, etc.), these noises will interfere with the clarity of the voice data. At the same time, in campus bullying incidents, multiple participants may speak at the same time, which will cause voice signal overlap and increase the difficulty of recognition. Therefore, compared with the above-mentioned prior art solution, the embodiment of the present application can first perform preprocessing on each received voice data before identifying and analyzing the voice data according to the method shown in subsequent steps S202-S204, thereby solving the problem of poor voice data quality in campus bullying scenarios through the newly added preprocessing method and improving the accuracy of subsequent voice data recognition and analysis.

[0043] It should be noted that the preprocessing methods that can be performed on each voice data in step S201 include but are not limited to: effective sound detection, voice noise reduction, human voice and background sound separation, speaker separation, voice enhancement, etc. The embodiments of this application do not impose any limitations on the specific preprocessing methods that can be performed in the above steps.

[0044] In an embodiment of the present application, by executing the above step S201, the preprocessing result of each voice data can be obtained as the target audio data corresponding to the voice data; wherein, the target audio data can be composed of multiple voice information, and after the "speaker separation" processing in the above preprocessing, multiple voice information belonging to the same speaker can be separated from the target audio data, so that the speaker corresponding to each voice information in the target audio data can be marked, so that each voice information in the target audio data carries the speaker label corresponding to the voice information.

[0045] S202, for each of the target audio data, when it is detected that there are preset keywords related to the bullying incident in the target audio data, emotion recognition is performed on the target audio data to obtain an emotion recognition result corresponding to the target audio data.

[0046] Here, considering that the existing technical solutions for detecting bullying behavior are to conduct real-time recognition and analysis of data transmitted by the semantic recognition module and the image recognition module, the two independent technical judgments have limitations. Among them, relying only on preset keywords for recognition in the voice data analysis part may lead to insufficient recognition of new or implicit bullying behaviors, and in scenes with insufficient light or dense crowds, the image recognition module may find it difficult to accurately identify bullying behaviors.

[0047] Based on this, in order to overcome the above-mentioned defects, in an embodiment of the present application, for each target audio data obtained after preprocessing in the above step S201, the edge computing device can execute the three different types of speech recognition analysis methods shown in subsequent steps S202-S204 on the target audio data (i.e., any target audio data after preprocessing) in parallel, thereby analyzing the speech data from multiple dimensions such as emotion recognition (step S202), bullying sound event detection (step S203), and speech recognition (step S204), and it can be comprehensively determined whether a bullying incident is detected at the collection location of the speech data (i.e., the deployment location of the acoustic sensor), thereby effectively improving the accuracy of the bullying incident detection results.

[0048] Specifically, for each target audio data, keyword detection can be performed on the target audio data before emotion recognition is performed on the target audio data. After detecting the presence of preset keywords related to the bullying incident in the target audio data, emotion recognition can be performed on the target audio data. For the target audio data in which the above preset keywords are not detected, it can be determined that the target audio data does not need to undergo subsequent emotion recognition steps, which is conducive to improving the efficiency of bullying incident detection and reducing the data processing burden of edge computing devices.

[0049] It should be noted that the above-mentioned preset keywords represent keywords related to bullying incidents. The embodiment of the present application does not impose any limitation on the specific vocabulary content corresponding to the above-mentioned preset keywords and the specific number of the above-mentioned preset keywords.

[0050] Specifically, if it is determined that the above-mentioned preset keywords exist in a target audio data, then when performing emotion recognition on the target audio data, the target audio data can be classified and predicted based on a pre-trained speech emotion recognition model, and the probability that the target audio data belongs to each preset emotion category can be output as the emotion recognition result corresponding to the target audio data; wherein the above-mentioned preset emotion categories can include at least: positive emotions unrelated to bullying incidents and negative emotions related to bullying incidents.

[0051] It should be noted that compared with the existing technical solutions that only have a semantic recognition module for voice data, in actual applications, due to the diversity and complexity of school bullying behaviors, such as verbal bullying, social bullying, cyber bullying and other highly concealed means, semantic recognition technology may not be able to accurately identify all types of bullying behaviors. Therefore, the embodiment of the present application adds voice emotion analysis on the basis of semantic recognition, and recognizes voice emotions to assist in judging speaking scenes, which can effectively distinguish between daily fights and bullying scenes, thereby improving the accuracy of bullying incident detection results.

[0052] S203, for each of the target audio data, respectively calculate the speech similarity between the target audio data and the acoustic event templates of multiple bullying events, and obtain the highest value of the speech similarity in the calculation results as the bullying acoustic event detection result corresponding to the target audio data.

[0053] Here, multiple acoustic event templates of bullying incidents may be pre-stored in the edge computing device; wherein, the higher the voice similarity between the target audio data and the acoustic event template, the higher the possibility of a bullying incident occurring at the data collection location of the target audio data (that is, the deployment location of the acoustic sensor).

[0054] Specifically, for each target audio data, the edge computing device can respectively calculate the speech similarity between the target audio data and the above-mentioned multiple pre-stored acoustic event templates, and select the highest value of the speech similarity as the bullying sound event detection result corresponding to the target audio data (which is also equivalent to the highest probability of a bullying incident occurring at the data collection location of the target audio data).

[0055] It should be noted that each of the above-mentioned acoustic event templates may correspond to a piece of sample speech data occurring in a bullying scenario. The embodiment of the present application does not impose any limitation on the specific template content and the specific number of templates of the above-mentioned acoustic event templates.

[0056] S204, for each of the target audio data, determine whether the target audio data contains voice information related to the bullying incident, and obtain a determination result as a voice recognition result of the target audio data.

[0057] Specifically, for each target audio data, when judging whether the target audio data contains voice information related to the bullying incident, as an optional embodiment, the target audio data can be first converted from a voice format to a text format to obtain target text data corresponding to the target audio data (equivalent to text data that semantically matches the target audio data); then the target text data is input into a pre-trained text classification prediction model, and the probability that the target text data belongs to each preset text category is output as the text classification prediction result corresponding to the target audio data, and the above text classification prediction result is used as the speech recognition result corresponding to the target audio data; wherein the above preset text categories may at least include: a first text category indicating that text information related to the bullying incident is contained, and a second text category indicating that text information related to the bullying incident is not contained.

[0058] For example, taking the first and second text categories as the preset text categories, after obtaining the target text data a corresponding to the target audio data A through speech-to-text processing, the target text data a is input into a pre-trained text classification prediction model. If the output shows that the probability that the target text data a belongs to the first text category is 70% (i.e., the probability that the target text data a contains text information related to the bullying incident is 70%), and the probability that the target text data a belongs to the second text category is 30% (i.e., the probability that the target text data a does not contain text information related to the bullying incident is 30%), then the text classification prediction result corresponding to the target text data a can be "the probability that the target text data a belongs to the first text category is 70%", or "the probability that the target text data a belongs to the first text category is 70%, and the probability that the target text data a belongs to the second text category is 30%" (equivalent to only ensuring that the text classification prediction result includes: the probability that the target text data contains text information related to the bullying incident).

[0059] Here, before executing steps S201-S205, multiple conversation text information may be collected in advance as training data, wherein each conversation text information carries a true label, and the true label is used to mark whether the conversation text information contains text information related to the bullying incident (for example, if the conversation text information contains text information related to the bullying incident, the true label of the conversation text information may be marked as 1; if the conversation text information does not contain text information related to the bullying incident, the true label of the conversation text information may be marked as 0), and the above training data is input into the original text classification prediction model, and text classification prediction is performed on each input conversation text information through the original text classification prediction model, and a text classification prediction result of each conversation text information (that is, the probability that the conversation text information contains text information related to the bullying incident and the probability that the conversation text information does not contain text information related to the bullying incident) is output, and the original text classification prediction model is trained according to the classification loss between the text classification prediction result of each conversation text information and the true label of the conversation text information until the original text classification prediction model converges, and the converged original text classification prediction model is obtained as the trained text classification prediction model.

[0060] It should be noted that the above-mentioned text classification prediction model can be a BERT (Bidirectional Encoder Representations from Transformers, a deep bidirectional language representation model based on the Transformer architecture) model, or it can be a text classification model of other structures such as a Word2Vec model; the embodiment of the present application does not impose any limitation on the specific model structure of the above-mentioned text classification prediction model.

[0061] S205, for each target audio data, comprehensively determine the bullying event detection result corresponding to the deployment position of the target acoustic sensor according to the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data.

[0062] Here, the above-mentioned target acoustic sensor represents the acoustic sensor corresponding to the target audio data, that is, for each piece of target audio data, it is possible to comprehensively judge whether a bullying incident is detected at the collection position of the target audio data (that is, the deployment position of the above-mentioned target acoustic sensor) from multiple dimensions such as emotion recognition, bullying sound event detection and speech recognition based on the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data.

[0063] In the embodiment of the present application, when executing step S205, the comprehensive determination method that can be selected is not unique. In practical applications, the specific determination method suitable for the target campus can be flexibly configured and selected according to the actual bullying incident detection needs of the target campus.

[0064] Specifically, as an optional embodiment, the above step S205 may be performed according to the method shown in the following steps a1-a3:

[0065] Step a1: when the emotion recognition result indicates that the target audio data belongs to a negative emotion, determine whether the bullying sound event detection result is higher than a preset voice similarity threshold.

[0066] Here, referring to the relevant instructions in the aforementioned steps S201-S204, it can be known that the emotion recognition result corresponding to the target audio data includes the probability that the target audio data belongs to negative emotions related to a bullying incident. When the probability value is higher than the preset emotion judgment threshold, it can be determined that the target audio data belongs to negative emotions, and it is necessary to further judge the bullying sound event detection result and the speech recognition result of the target audio data; when the probability value is less than or equal to the preset emotion judgment threshold, it can be determined that no bullying incident is detected at the deployment position of the target acoustic sensor, and thus the bullying sound event detection result and the speech recognition result of the target audio data will no longer be judged.

[0067] Step a2: If it is determined that the bullying sound event detection result is higher than the preset voice similarity threshold, continue to determine whether the voice recognition result indicates that the target audio data contains voice information related to the bullying event.

[0068] Here, referring to the relevant instructions at the aforementioned steps S201-S204, it can be known that the bullying acoustic event detection result corresponding to the target audio data includes the highest voice similarity between the target audio data and the acoustic event template of the bullying event. When the voice similarity is higher than the preset voice similarity threshold, it can be determined that the voice recognition result of the target audio data needs to be further judged; when the voice similarity is less than or equal to the preset voice similarity threshold, it can be determined that no bullying incident is detected at the deployment location of the target acoustic sensor, and thus the voice recognition result of the target audio data is no longer judged.

[0069] Step a3: If it is determined that the speech recognition result indicates that the target audio data contains speech information related to a bullying incident, it is determined that a bullying incident is detected at the deployment location of the target acoustic sensor.

[0070] Here, referring to the relevant instructions at the aforementioned steps S201-S204, it can be known that the speech recognition result corresponding to the target audio data includes a probability value that the target text data (i.e., the text data corresponding to the target audio data) belongs to the first text category (i.e., the first text category indicating that it contains text information related to the bullying incident). When the probability value is higher than a preset threshold, it can be determined that the speech recognition result indicates that the target audio data contains speech information related to the bullying incident; when the probability value is less than or equal to the preset threshold, it can be determined that the speech recognition result indicates that the target audio data does not contain speech information related to the bullying incident.

[0071] Specifically, as another optional embodiment, the above step S205 may also be performed according to the method shown in the following steps b1-b2:

[0072] Step b1, according to the weight coefficients corresponding to the emotion recognition detection dimension, the bullying sound event detection dimension and the speech recognition detection dimension respectively, weighted summation is performed on the emotion recognition result, the bullying sound event detection result and the speech recognition result, and the weighted summation result is obtained as the target probability of detecting a bullying incident at the deployment position of the target acoustic sensor.

[0073] It should be noted that the weight coefficients corresponding to the emotion recognition detection dimension, the bullying sound event detection dimension and the speech recognition detection dimension can be flexibly configured according to the actual detection needs of the target campus for bullying incidents. The embodiments of this application do not impose any restrictions on the specific values ​​of the weight coefficients corresponding to the emotion recognition detection dimension, the bullying sound event detection dimension and the speech recognition detection dimension.

[0074] Specifically, referring to the relevant instructions at the aforementioned steps S201-S204, it can be known that the emotion recognition result includes the probability that the target audio data belongs to the negative emotion related to the bullying incident, the bullying sound event detection result includes the highest voice similarity between the target audio data and the acoustic event template of the bullying incident, and the voice recognition result includes the probability value that the target text data (i.e., the text data corresponding to the target audio data) belongs to the first text category (i.e., the first text category indicating that it contains text information related to the bullying incident). Therefore, when performing weighted summation, the probability that the above-mentioned target audio data belongs to the negative emotion related to the bullying incident can be multiplied by the weight coefficient of the emotion recognition detection dimension, the above-mentioned voice similarity can be multiplied by the weight coefficient of the bullying sound event detection dimension, and the probability value that the above-mentioned target text data belongs to the first text category can be multiplied by the weight coefficient of the voice recognition detection dimension, so as to calculate the sum of the three products as the target probability of detecting a bullying incident at the deployment location of the target acoustic sensor.

[0075] Step b2: If the target probability is higher than a preset probability threshold, it is determined that a bullying incident is detected at the deployment location of the target acoustic sensor.

[0076] Here, if the above target probability is less than or equal to a preset probability threshold, it can be determined that no bullying incident is detected at the deployment location of the target acoustic sensor.

[0077] The following is an exemplary description of each step in the above bullying incident detection method provided in the embodiment of the present application:

[0078] For the specific implementation of step S201, the specific preprocessing method performed includes: speech noise reduction, human voice and background sound separation, and speaker separation. In an optional implementation, refer to Figure 3 As shown, Figure 3 A flow chart of a method for preprocessing speech data provided in an embodiment of the present application is shown, wherein the method comprises steps S301-S303; specifically:

[0079] S301, for each of the voice data, perform noise reduction processing on the voice data to obtain noise-reduced voice data.

[0080] Specifically, when executing step S301, the specific noise reduction processing method that can be used is not unique; for example, a filter can be used to filter the speech data to remove noise data within a specific frequency range in the speech data; an online audio editing tool can also be used to process the speech data online to remove the noise data contained in the speech data; an acoustic model that can distinguish between speech and noise can also be used to remove the noise data contained in the speech data; the embodiment of the present application does not impose any limitation on the specific noise reduction processing method used in step S301.

[0081] S302, performing human voice separation from background sound on the noise-reduced speech data to obtain target human voice data separated from the noise-reduced speech data.

[0082] Specifically, when executing step S302, the denoised speech data can be imported into an audio editing tool, and based on the voice separation function configured in the audio editing tool, the voice data after denoising is separated from the background sound to obtain the separated target voice data (i.e., the pure voice data in the denoised speech data); the voice data after denoising can also be separated from the background sound by a pre-trained sound separation model, or with the help of AI (Artificial Intelligence) technology. The specific voice and background sound separation method used in step S302 is not limited in any way in the embodiments of the present application.

[0083] S303, performing speaker separation on the target human voice data, and marking the speaker corresponding to each voice information in the target human voice data according to the speaker separation result, to obtain the target audio data corresponding to the voice data.

[0084] Specifically, when executing step S302, as an optional embodiment, the target voice data can be input into a pre-trained speaker separation model, and the speaker separation model is used to identify the vocal characteristics of different speakers in the target voice data, so as to determine the speaker corresponding to each voice information in the target voice data, and mark the speaker corresponding to each voice information, and then output the target audio data carrying different speaker labels.

[0085] Specifically, when executing step S302, as another optional embodiment, the speaker separation of the above-mentioned target voice data can be performed by using AI technology or using audio editing software (such as Audacity) configured with a speaker separation function, and target audio data carrying different speaker labels can also be obtained.

[0086] It should be noted that, when performing speaker separation on the above-mentioned target voice data in the preprocessing stage, it is only necessary to ensure that it can be determined whether the speaker to which each voice information in the target voice data belongs is the same, without the need to accurately determine the specific identity information of each speaker; for example, taking three different speakers ABC appearing in the target voice data as an example, it is only necessary to determine which speaker among ABC each voice information in the target voice data belongs to, without the need to determine which teacher or student in the target campus speaker A, speaker B and speaker C specifically correspond to.

[0087] For the specific implementation of step S202, in an optional implementation, refer to Figure 4 As shown, Figure 4 A flow chart of a method for performing emotion recognition on target audio data provided by an embodiment of the present application is shown, wherein the method comprises steps S401-S403; specifically:

[0088] S401, according to the speaker label carried by each voice information in the target audio data, determine the speech information corresponding to each speaker and the context information corresponding to the speech information in the target audio data from the target audio data.

[0089] Here, the specific method for determining the speaker label corresponding to each voice information in the target audio data can refer to the preprocessing method shown in the aforementioned steps S301-S303, and the repeated parts will not be repeated here.

[0090] As an exemplary explanation, taking the target audio data containing three speaker labels A, B, and C as an example, the voice information carrying speaker label A can be determined from the target audio data as the speech information corresponding to speaker A, and the context information corresponding to the speech information can be determined from the target audio data; correspondingly, for speakers B and C, the voice information carrying speaker label B can also be determined from the target audio data as the speech information corresponding to speaker B and the context information corresponding to the speech information, and the voice information carrying speaker label C can be determined from the target audio data as the speech information corresponding to speaker C and the context information corresponding to the speech information.

[0091] S402, for each of the speech information, the speech information and the context information corresponding to the speech information are input into a pre-trained speech emotion recognition model, and the emotion classification prediction result of the speech information is output.

[0092] Here, for the speech information corresponding to each speaker, the speech information and the context information corresponding to the speech information are input into a pre-trained speech emotion recognition model. The speech emotion recognition model can perform emotion classification prediction on the speech information based on understanding the above context information (equivalent to using the context information as known reference information), and output the probability that the above speech information belongs to each preset emotion category as the emotion classification prediction result of the speech information; wherein the above preset emotion categories can at least include: positive emotions unrelated to bullying incidents and negative emotions related to bullying incidents.

[0093] For example, taking the above-mentioned positive emotions and negative emotions as the above-mentioned preset emotion categories as an example, the speech information corresponding to speaker A and the context information corresponding to the speech information are input into a pre-trained speech emotion recognition model. If the output shows that the probability that the speech information corresponding to speaker A belongs to the above-mentioned positive emotion is 30%, and the probability that it belongs to the above-mentioned negative emotion is 70%, then the emotion classification prediction result of the speech information corresponding to speaker A can be "the probability that the speech information belongs to the above-mentioned negative emotion is 70%", or it can be "the probability that the speech information belongs to the above-mentioned negative emotion is 70%, and the probability that it belongs to the above-mentioned positive emotion is 30%" (equivalent to only ensuring that the emotion classification prediction result only includes the probability that the speech information belongs to the negative emotion).

[0094] S403: Using the emotion classification prediction result of each speech information as the emotion recognition result corresponding to the target audio data.

[0095] Specifically, when the target audio data contains only a single speaker, based on the emotion classification prediction result of the speaker's speech information, the probability that the speaker's speech information belongs to a negative emotion can be directly used as the emotion recognition result corresponding to the target audio data.

[0096] Specifically, when the target audio data contains multiple speakers, the probability that each speaker's speech information belongs to a negative emotion can be determined based on the emotion classification prediction result of each speaker's speech information. At this time, as an optional embodiment, the maximum probability of belonging to a negative emotion can be determined from the probabilities that the speech information of multiple speakers belongs to a negative emotion as the emotion recognition result corresponding to the target audio data; as another optional embodiment, the average value of the probabilities that the speech information of multiple speakers belongs to a negative emotion can also be calculated, and the calculated average value can be used as the emotion recognition result corresponding to the target audio data.

[0097] Here, before executing steps S201-S205, multiple conversational speech data can be collected in advance, and the speech information of a speaker and the context information of the speech information can be separated from each conversational speech data as a group of training samples, wherein each group of training samples carries a true label, and the true label is used to mark whether the speech information in the group of training samples belongs to positive emotion or negative emotion. The group of training samples is input into the original speech emotion recognition model, and the original speech emotion recognition model is used to perform emotion classification prediction on each group of input training samples, and the emotion classification prediction result of the speech information in each group of training samples (that is, the probability that the speech information belongs to negative emotion) is output. According to the classification loss between the emotion classification prediction result of the speech information in each group of training samples and the true label of the speech information, the original speech emotion recognition model is trained until the original speech emotion recognition model converges, and the converged original speech emotion recognition model is obtained as the trained speech emotion recognition model mentioned above.

[0098] It should be noted that the above-mentioned speech emotion recognition model can be a discrete form of emotion description model or a continuous form of emotion description model. The embodiments of the present application do not impose any limitation on the specific model structure of the above-mentioned speech emotion recognition model.

[0099] On the basis of the bullying incident detection method shown in the above steps S201-S205, after executing the above step S205, if it is determined that a bullying incident is detected at the deployment location of the target acoustic sensor according to the bullying incident detection result obtained in step S205, the edge computing device can automatically send a bullying incident alarm message carrying the target location coordinates (i.e., the coordinates corresponding to the deployment location of the target acoustic sensor) to the management terminal of the target campus, so that the management personnel on the side of the above management terminal can promptly investigate and track the currently detected bullying incident after receiving the above bullying incident alarm message.

[0100] In addition to sending the bullying incident alarm information as mentioned above, considering that the management personnel need a certain amount of time to go to the location of the bullying incident (i.e., the deployment location of the target acoustic sensor) after receiving the alarm information, in order to minimize the harm caused by the bullying incident to the victim, the embodiment of the present application is based on a system framework in which the edge computing device and the target acoustic sensor can communicate (i.e., transmit data according to a preset data transmission protocol), and also proposes the pre- and in-process intervention method shown in the following step c1, so as to timely intervene in the bullying incident that is about to occur (before the incident) and during the occurrence (in the incident), specifically:

[0101] Step c1: when it is determined according to the bullying incident detection result that a bullying incident is detected at the deployment position of the target acoustic sensor, the target acoustic sensor is controlled to play intervention voice information for the bullying incident according to a preset intervention voice template.

[0102] Here, based on the ability to communicate between the edge computing device and the target acoustic sensor, the edge computing device can directly control the target acoustic sensor to play intervention voice information for the bullying incident according to a preset intervention voice template when it is determined that a bullying incident is detected at the deployment location of the above-mentioned target acoustic sensor according to the bullying incident detection result obtained in step S205, so as to promptly stop the bullying incident that is about to occur or is occurring at the deployment location of the target acoustic sensor. Compared with the above-mentioned method of sending bullying incident alarm information, the pre- and in-process intervention method shown in step c1 can intervene in the currently detected bullying incident more promptly, which is conducive to reducing the harm suffered by the victims of the bullying incident.

[0103] It should be noted that the specific template content of the above-mentioned preset intervention voice playback template can be flexibly configured according to the bullying incident intervention needs of the target campus. It is only necessary to ensure that the above-mentioned intervention voice information played according to the above-mentioned preset intervention voice playback template can play a certain intervention role in the bullying incident.

[0104] Here, when executing step c1, in addition to a preset intervention voice playback template according to a general configuration, as another optional embodiment, a specific preset intervention voice playback template applicable to each event severity may be configured according to the event severity corresponding to the detected bullying event.

[0105] In the above optional embodiment, if Figure 5 As shown, Figure 5 A flow chart of a method for controlling the target acoustic sensor to play intervention voice information for a bullying incident according to a preset intervention voice template provided in an embodiment of the present application is shown, wherein the method includes steps S501-S503; specifically:

[0106] S501, feature matching is performed on the target audio data with multiple event sample voices at different event levels, and a target event sample voice having the highest feature matching degree with the target audio data is determined from the multiple event sample voices.

[0107] Here, the event severity corresponding to each event level and the event sample speech corresponding to each event severity can be predetermined according to different event severities.

[0108] Specifically, taking the event levels including: a first event level indicating a lighter event severity, a second event level indicating an event severity of medium level, and a third event level indicating a higher event severity as an example, a variety of voice data containing only human voices can be pre-collected as event sample voices under the above-mentioned first event level, a variety of voice data when physical conflicts occur can be pre-collected as event sample voices under the above-mentioned second event level, and a variety of voice data when fighting with different types of weapons such as sticks can be pre-collected as event sample voices under the above-mentioned third event level.

[0109] S502: According to the target event level corresponding to the target event sample voice, determine a preset intervention voice template matching the target event level from a plurality of pre-stored preset intervention voice templates as a target intervention voice template.

[0110] Here, different event levels correspond to different event severities, and different event severities correspond to different preset intervention voice templates (for example, the higher the event severity, the higher the intensity of the language warning prompt that can be provided by the corresponding preset intervention voice template). Therefore, after determining the target event level that matches the target audio data, the preset intervention voice template that matches the target event level can be determined as the target intervention voice template from multiple pre-stored preset intervention voice templates based on the event severity corresponding to the target event level.

[0111] S503: Control the target acoustic sensor to play intervention voice information on the bullying incident according to the target intervention voice template.

[0112] Specifically, taking the case where the target event level is the first event level with a lighter severity as an example, the edge computing device can determine the first preset intervention voice template that matches the first event level with a lighter severity from multiple pre-stored preset intervention voice templates (that is, the target intervention voice template determined at this time is the first preset intervention voice template), and then send the first preset intervention voice template to the template acoustic sensor to control the target acoustic sensor to play the intervention voice information for the bullying incident according to the first preset intervention voice template.

[0113] In the embodiment of the present application, considering that when the detection location of the bullying incident has been determined to be the target campus, based on the characteristics of the relatively fixed permanent population in the target campus, as an optional embodiment, voiceprint collection can also be carried out on the permanent campus population such as teachers and students in the target campus, and a voiceprint database associated with the target campus can be established, so that when it is determined that a bullying incident has been detected, the specific identity information of each speaker can be determined from the voiceprint database by voiceprint comparison based on the speaker label carried in the target audio data, so as to facilitate the subsequent tracking and processing of the bullying incident.

[0114] In the above optional embodiment, if Figure 6 As shown, Figure 6 A flow chart of a method for determining persons involved in a bullying incident by voiceprint comparison provided in an embodiment of the present application is shown, wherein the method includes steps S601-S603; specifically:

[0115] S601, when it is determined according to the bullying incident detection result that a bullying incident is detected at the deployment position of the target acoustic sensor, based on the speaker label carried by each voice information in the target audio data, determine the speech information corresponding to each speaker from the target audio data.

[0116] Here, in step S601, the specific implementation method of determining the speech information corresponding to each speaker from the target audio data can refer to the specific implementation method of the aforementioned step S401, and the repeated parts will not be repeated here.

[0117] S602: For each of the speakers, voiceprint comparison is performed with speech information corresponding to the speaker according to the voiceprint database associated with the target campus to determine the identity information of the speaker in the target campus.

[0118] Here, the permanent campus population such as teachers and students in the target campus can be targeted in advance, and each of the above permanent campus population can be used as a voiceprint collector. After the voiceprint is collected for each voiceprint collector, the identity information of each voiceprint collector and the voiceprint features corresponding to the voiceprint collector can be stored in the voiceprint database associated with the target campus.

[0119] S603, determining the information of persons involved in the bullying incident according to the identity information of each speaker.

[0120] Specifically, based on the voiceprint database associated with the target campus, for each speaker's speech information, the speech information of the speaker can be matched with the voiceprint features corresponding to each voiceprint collector stored in the voiceprint database through voiceprint comparison, and the identity information of the voiceprint collector with a feature matching degree higher than the preset matching threshold is determined as the identity information of the speaker in the target campus. In this way, the specific identity information of each speaker involved in the target audio data can be determined, so as to facilitate the subsequent tracking and processing of bullying incidents.

[0121] It should be noted that based on the voiceprint database and the characteristics of the speaker labels carried in the target audio data, in the embodiment of the present application, as an optional embodiment, in the edge computing device, according to the voiceprint characteristics of each speaker, the target audio data of the same speaker in the multiple target audio data collected at different time nodes can be associated and aggregated in the time dimension, so as to facilitate the subsequent tracking and processing of the bullying incidents, so as to more conveniently track the continuous development line of the bullying incidents in the time dimension, thereby reducing the burden of data collection when tracking and processing the bullying incidents.

[0122] Based on the same inventive concept, the present application also provides a bullying incident detection system corresponding to the above-mentioned bullying incident detection method based on acoustic event recognition and emotion recognition. Since the principle of solving the problem by the bullying incident detection system in the embodiment of the present application is similar to the above-mentioned bullying incident detection method in the embodiment of the present application, the implementation of the bullying incident detection system can refer to the implementation of the above-mentioned bullying incident detection method, and the repeated parts will not be repeated.

[0123] like Figure 1 As shown, the bullying event detection system based on acoustic event recognition and emotion recognition includes: an edge computing device 100 and multiple acoustic sensors 101 deployed in the target campus; wherein the edge computing device 100 and the multiple acoustic sensors 101 perform data transmission according to a preset data transmission protocol, and the edge computing device 100 is used to:

[0124] Receive the voice data collected by each of the acoustic sensors 101, and pre-process each of the received voice data to obtain target audio data corresponding to each of the voice data; wherein each voice information in the target audio data carries a speaker label corresponding to the voice information;

[0125] For each of the target audio data, when it is detected that there are preset keywords related to the bullying incident in the target audio data, emotion recognition is performed on the target audio data to obtain an emotion recognition result corresponding to the target audio data;

[0126] For each of the target audio data, respectively calculate the speech similarity between the target audio data and the acoustic event templates of multiple bullying events, and obtain the highest value of the speech similarity in the calculation results as the bullying acoustic event detection result corresponding to the target audio data;

[0127] For each of the target audio data, determining whether the target audio data contains voice information related to the bullying incident, and obtaining a determination result as a voice recognition result of the target audio data;

[0128] For each of the target audio data, based on the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data, the bullying event detection result corresponding to the deployment location of the target acoustic sensor is comprehensively determined; wherein the target acoustic sensor represents the acoustic sensor 101 corresponding to the target audio data.

[0129] In an optional implementation, when preprocessing each of the received voice data, the edge computing device 100 is used to:

[0130] For each of the voice data, performing noise reduction processing on the voice data to obtain noise-reduced voice data;

[0131] Performing human voice and background sound separation on the de-noised speech data to obtain target human voice data separated from the de-noised speech data;

[0132] Speaker separation is performed on the target vocal data, and based on the speaker separation result, the speaker corresponding to each voice information in the target vocal data is marked to obtain the target audio data corresponding to the voice data.

[0133] In an optional implementation, when performing emotion recognition on the target audio data, the edge computing device 100 is used to:

[0134] According to the speaker label carried by each voice information in the target audio data, determining the speech information corresponding to each speaker and the context information corresponding to the speech information in the target audio data from the target audio data;

[0135] For each of the speech information, the speech information and the context information corresponding to the speech information are input into a pre-trained speech emotion recognition model, and the emotion classification prediction result of the speech information is output;

[0136] The emotion classification prediction result of each speech information is used as the emotion recognition result corresponding to the target audio data.

[0137] In an optional implementation, when determining whether the target audio data contains voice information related to the bullying incident, the edge computing device 100 is used to:

[0138] Convert the target audio data from a voice format to a text format to obtain target text data corresponding to the target audio data;

[0139] The target text data is input into a pre-trained text classification prediction model, and a text classification prediction result corresponding to the target text data is output, and the text classification prediction result is used as a speech recognition result corresponding to the target audio data; wherein the text classification prediction result includes: the probability that the target text data contains text information related to the bullying incident.

[0140] In an optional implementation, when comprehensively determining the bullying event detection result corresponding to the deployment location of the target acoustic sensor based on the emotion recognition result, the bullying sound event detection result, and the speech recognition result corresponding to the target audio data, the edge computing device 100 is used to:

[0141] When the emotion recognition result indicates that the target audio data belongs to a negative emotion, determining whether the bullying sound event detection result is higher than a preset voice similarity threshold;

[0142] If it is determined that the bullying sound event detection result is higher than the preset voice similarity threshold, then continue to determine whether the voice recognition result indicates that the target audio data contains voice information related to the bullying event;

[0143] If it is determined that the speech recognition result indicates that the target audio data contains speech information related to a bullying incident, it is determined that a bullying incident is detected at the deployment location of the target acoustic sensor.

[0144] In an optional implementation, when comprehensively determining the bullying event detection result corresponding to the deployment location of the target acoustic sensor based on the emotion recognition result, the bullying sound event detection result, and the speech recognition result corresponding to the target audio data, the edge computing device 100 is used to:

[0145] According to the weight coefficients corresponding to the emotion recognition detection dimension, the bullying sound event detection dimension and the speech recognition detection dimension respectively, the emotion recognition result, the bullying sound event detection result and the speech recognition result are weighted and summed to obtain the weighted sum result as the target probability of detecting a bullying event at the deployment position of the target acoustic sensor;

[0146] If the target probability is higher than a preset probability threshold, it is determined that a bullying incident is detected at the deployment location of the target acoustic sensor.

[0147] In an optional implementation, the edge computing device 100 is further used for:

[0148] When it is determined according to the bullying incident detection result that a bullying incident is detected at the deployment position of the target acoustic sensor, the target acoustic sensor is controlled to play intervention voice information for the bullying incident according to a preset intervention voice template.

[0149] In an optional implementation, when controlling the target acoustic sensor to play the intervention voice information for the bullying incident according to the preset intervention voice template, the edge computing device 100 is used to:

[0150] Performing feature matching on the target audio data with multiple event sample voices at different event levels, and determining, from the multiple event sample voices, the target event sample voice with the highest degree of feature matching with the target audio data;

[0151] According to the target event level corresponding to the target event sample voice, determining a preset intervention voice template matching the target event level from a plurality of pre-stored preset intervention voice templates as a target intervention voice template;

[0152] The target acoustic sensor is controlled to play intervention voice information for the bullying incident according to the target intervention voice template.

[0153] In an optional implementation, the edge computing device 100 is further used for:

[0154] When it is determined according to the bullying incident detection result that a bullying incident is detected at the deployment position of the target acoustic sensor, determining speech information corresponding to each speaker from the target audio data according to the speaker label carried by each voice information in the target audio data;

[0155] For each of the speakers, voiceprint comparison is performed on the speech information corresponding to the speaker according to the voiceprint database associated with the target campus to determine the identity information of the speaker in the target campus;

[0156] According to the identity information of each speaker, the information of persons involved in the bullying incident is determined.

[0157] Based on the same inventive concept, the present application also provides an electronic device corresponding to the above-mentioned bullying incident detection method based on acoustic event recognition and emotion recognition. Since the principle of solving the problem by the electronic device in the embodiment of the present application is similar to the above-mentioned bullying incident detection method in the embodiment of the present application, the implementation of the electronic device can refer to the implementation of the above-mentioned bullying incident detection method, and the repeated parts will not be repeated.

[0158] like Figure 7As shown, an embodiment of the present application provides an electronic device 700 for executing the bullying incident detection method based on acoustic event recognition and emotion recognition in the present application, and the device includes a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor 702, wherein the memory 701 and the processor 702 are communicatively connected via a bus, and the processor 702 implements the steps of the bullying incident detection method based on acoustic event recognition and emotion recognition when executing the computer program.

[0159] Specifically, the above-mentioned memory 701 and processor 702 can be general-purpose memory and processor, which are not specifically limited here. When the processor 702 runs the computer program stored in the memory 701, it can execute the above-mentioned bullying event detection method based on acoustic event recognition and emotion recognition.

[0160] Corresponding to the bullying incident detection method based on acoustic event recognition and emotion recognition in the present application, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned bullying incident detection method based on acoustic event recognition and emotion recognition are executed.

[0161] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned bullying event detection method based on acoustic event recognition and emotion recognition can be executed.

[0162] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0165] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0166] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0167] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A bullying incident detection method based on acoustic event recognition and emotion recognition, characterized in that: Applied to an edge computing device, wherein the edge computing device and a plurality of acoustic sensors deployed in a target campus perform data transmission according to a preset data transmission protocol, and the bullying incident detection method includes: Receive the voice data collected by each of the acoustic sensors, and pre-process each of the received voice data to obtain target audio data corresponding to each of the voice data; wherein each voice information in the target audio data carries a speaker label corresponding to the voice information; For each of the target audio data, when it is detected that there are preset keywords related to the bullying incident in the target audio data, emotion recognition is performed on the target audio data to obtain an emotion recognition result corresponding to the target audio data; For each of the target audio data, respectively calculate the speech similarity between the target audio data and the acoustic event templates of multiple bullying events, and obtain the highest value of the speech similarity in the calculation results as the bullying acoustic event detection result corresponding to the target audio data; For each of the target audio data, determining whether the target audio data contains voice information related to the bullying incident, and obtaining a determination result as a voice recognition result of the target audio data; For each of the target audio data, comprehensively determine the bullying event detection result corresponding to the deployment position of the target acoustic sensor according to the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data; wherein the target acoustic sensor represents the acoustic sensor corresponding to the target audio data; Wherein, the preprocessing of each received voice data comprises: For each of the voice data, performing noise reduction processing on the voice data to obtain noise-reduced voice data; Performing human voice and background sound separation on the de-noised speech data to obtain target human voice data separated from the de-noised speech data; Performing speaker separation on the target human voice data, and marking the speaker corresponding to each voice information in the target human voice data according to the speaker separation result, to obtain the target audio data corresponding to the voice data; The step of performing emotion recognition on the target audio data includes: According to the speaker label carried by each voice information in the target audio data, determining the speech information corresponding to each speaker and the context information corresponding to the speech information in the target audio data from the target audio data; For each of the speech information, the speech information and the context information corresponding to the speech information are input into a pre-trained speech emotion recognition model, and the emotion classification prediction result of the speech information is output; The emotion classification prediction result of each speech information is used as the emotion recognition result corresponding to the target audio data; Wherein, the bullying incident detection method further includes: When it is determined according to the bullying incident detection result that a bullying incident is detected at the deployment position of the target acoustic sensor, the target acoustic sensor is controlled to play intervention voice information for the bullying incident according to a preset intervention voice template.

2. The method for detecting bullying incidents according to claim 1, characterized in that: The determining whether the target audio data contains voice information related to the bullying incident includes: Convert the target audio data from a voice format to a text format to obtain target text data corresponding to the target audio data; The target text data is input into a pre-trained text classification prediction model, and a text classification prediction result corresponding to the target text data is output, and the text classification prediction result is used as a speech recognition result corresponding to the target audio data; wherein the text classification prediction result includes: the probability that the target text data contains text information related to the bullying incident.

3. The method for detecting bullying incidents according to claim 1, characterized in that: The step of comprehensively determining the bullying event detection result corresponding to the deployment location of the target acoustic sensor according to the emotion recognition result, the bullying sound event detection result, and the speech recognition result corresponding to the target audio data includes: When the emotion recognition result indicates that the target audio data belongs to a negative emotion, determining whether the bullying sound event detection result is higher than a preset voice similarity threshold; If it is determined that the bullying sound event detection result is higher than the preset voice similarity threshold, then continue to determine whether the voice recognition result indicates that the target audio data contains voice information related to the bullying event; If it is determined that the speech recognition result indicates that the target audio data contains speech information related to a bullying incident, it is determined that a bullying incident is detected at the deployment location of the target acoustic sensor.

4. The method for detecting bullying incidents according to claim 1, characterized in that: The method of comprehensively determining the bullying event detection result corresponding to the deployment position of the target acoustic sensor according to the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data further includes: According to the weight coefficients corresponding to the emotion recognition detection dimension, the bullying sound event detection dimension and the speech recognition detection dimension respectively, the emotion recognition result, the bullying sound event detection result and the speech recognition result are weighted and summed to obtain the weighted sum result as the target probability of detecting a bullying event at the deployment position of the target acoustic sensor; If the target probability is higher than a preset probability threshold, it is determined that a bullying incident is detected at the deployment location of the target acoustic sensor.

5. The method for detecting bullying incidents according to claim 1, characterized in that: The controlling the target acoustic sensor to play the intervention voice information for the bullying incident according to the preset intervention voice template includes: Performing feature matching on the target audio data with multiple event sample voices at different event levels, and determining, from the multiple event sample voices, the target event sample voice with the highest degree of feature matching with the target audio data; According to the target event level corresponding to the target event sample voice, determining a preset intervention voice template matching the target event level from a plurality of pre-stored preset intervention voice templates as a target intervention voice template; The target acoustic sensor is controlled to play intervention voice information for the bullying incident according to the target intervention voice template.

6. The method for detecting bullying incidents according to claim 1, characterized in that: The bullying incident detection method further includes: When it is determined according to the bullying incident detection result that a bullying incident is detected at the deployment position of the target acoustic sensor, determining speech information corresponding to each speaker from the target audio data according to the speaker label carried by each voice information in the target audio data; For each of the speakers, voiceprint comparison is performed on the speech information corresponding to the speaker according to the voiceprint database associated with the target campus to determine the identity information of the speaker in the target campus; According to the identity information of each speaker, the information of persons involved in the bullying incident is determined.

7. A bullying incident detection system based on acoustic event recognition and emotion recognition, characterized in that: The bullying incident detection system includes: an edge computing device and a plurality of acoustic sensors deployed in a target campus; wherein the edge computing device and the plurality of acoustic sensors perform data transmission according to a preset data transmission protocol, and the edge computing device is used to: Receive the voice data collected by each of the acoustic sensors, and pre-process each of the received voice data to obtain target audio data corresponding to each of the voice data; wherein each voice information in the target audio data carries a speaker label corresponding to the voice information; For each of the target audio data, when it is detected that there are preset keywords related to the bullying incident in the target audio data, emotion recognition is performed on the target audio data to obtain an emotion recognition result corresponding to the target audio data; For each of the target audio data, respectively calculate the speech similarity between the target audio data and the acoustic event templates of multiple bullying events, and obtain the highest value of the speech similarity in the calculation results as the bullying acoustic event detection result corresponding to the target audio data; For each of the target audio data, determining whether the target audio data contains voice information related to the bullying incident, and obtaining a determination result as a voice recognition result of the target audio data; For each of the target audio data, comprehensively determine the bullying event detection result corresponding to the deployment position of the target acoustic sensor according to the emotion recognition result, the bullying sound event detection result and the speech recognition result corresponding to the target audio data; wherein the target acoustic sensor represents the acoustic sensor corresponding to the target audio data; Wherein, when preprocessing each of the received voice data, the edge computing device is used to: For each of the voice data, performing noise reduction processing on the voice data to obtain noise-reduced voice data; Performing human voice and background sound separation on the de-noised speech data to obtain target human voice data separated from the de-noised speech data; Performing speaker separation on the target human voice data, and marking the speaker corresponding to each voice information in the target human voice data according to the speaker separation result, to obtain the target audio data corresponding to the voice data; Wherein, when performing emotion recognition on the target audio data, the edge computing device is used to: According to the speaker label carried by each voice information in the target audio data, determining the speech information corresponding to each speaker and the context information corresponding to the speech information in the target audio data from the target audio data; For each of the speech information, the speech information and the context information corresponding to the speech information are input into a pre-trained speech emotion recognition model, and the emotion classification prediction result of the speech information is output; The emotion classification prediction result of each speech information is used as the emotion recognition result corresponding to the target audio data; Wherein, the edge computing device is also used for: When it is determined according to the bullying incident detection result that a bullying incident is detected at the deployment position of the target acoustic sensor, the target acoustic sensor is controlled to play intervention voice information for the bullying incident according to a preset intervention voice template.

Images