A behavior detection method, system, device, equipment and storage medium

Abstract

The embodiments of the present application provide a behavior detection method, system, device, equipment and storage medium, which relate to the field of speech recognition technology. The specific implementation scheme is: receiving data to be detected; wherein, the data to be detected is: the front-end device identifies bullying keywords on the first feature to be detected, and uploads it when the keyword recognition result indicates that the audio to be detected contains bullying keywords, the first feature to be detected is: the front-end device extracts frequency domain features from the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected; use a pre-trained large audio model to identify the intent of the data to be detected, and obtain a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected. It can be seen that through this scheme, bullying behavior can be effectively detected.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a behavior detection method, system, device, equipment and storage medium. Background Art

[0002] With the continuous development of the education industry, campus safety issues have received increasing attention. Since bullying often occurs in hidden corners of the campus or areas that security cameras cannot cover, it is particularly important to effectively warn of bullying to improve campus safety.

[0003] Currently, audio is usually captured by deploying multiple sound sensors on campus, and a neural network model is deployed on the device to identify bullying keywords in the audio, and an alarm is triggered when bullying keywords are identified in the audio.

[0004] However, determining whether there is bullying behavior only by whether the audio contains bullying keywords will lead to a high probability of false positives. Therefore, how to effectively detect bullying behavior has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a behavior detection method, system, device, equipment and storage medium to effectively detect bullying behavior. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a behavior detection method, which is applied to a central terminal device, and the method includes:

[0007] Receive data to be detected; wherein the data to be detected is: the front-end device identifies bullying keywords on the first feature to be detected, and uploads it when the keyword identification result indicates that the audio to be detected contains bullying keywords, the first feature to be detected is: the front-end device extracts frequency domain features from the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected;

[0008] The pre-trained audio model is used to perform intent recognition on the data to be detected, and a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected is obtained.

[0009] Optionally, the large audio model includes a large language model and a first conversion model, wherein the first conversion model includes a semantic feature extraction network for extracting context information;

[0010] The pre-trained audio model is used to perform intent recognition on the data to be detected, and a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected is obtained, including:

[0011] Using the semantic feature extraction network to extract features from the data to be detected, to obtain a second feature to be detected;

[0012] The designated input feature obtained based on the second feature to be detected is input into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected.

[0013] Optionally, the audio large model further includes a second conversion model, wherein the second conversion model includes an emotion feature extraction network for extracting emotion information;

[0014] Before inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the method further includes:

[0015] Using the emotion feature extraction network to extract features from the data to be detected, to obtain a third feature to be detected;

[0016] Based on feature fusion of the second feature to be detected and the third feature to be detected, a specified input feature is obtained.

[0017] Optionally, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, and the second conversion model further includes a first fusion network and a second fusion network;

[0018] Before inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the method further includes:

[0019] Inputting the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature;

[0020] Based on the feature fusion of the second feature to be detected and the third feature to be detected, a specified input feature is obtained, including:

[0021] Using the second fusion network to perform feature fusion on the first fusion feature and the third feature to be detected to obtain a second fusion feature;

[0022] Based on feature fusion of the second feature to be detected and the second fused feature, a specified input feature is obtained.

[0023] Optionally, the second fusion network is a cross attention network;

[0024] Using the second fusion network to perform feature fusion on the first fusion feature and the third feature to be detected to obtain a second fusion feature, including:

[0025] The first fused feature is mapped to obtain the query Q and value V in the input data of the cross-attention network, and the third feature to be detected is mapped to obtain the key K in the input data of the cross-attention network, and the obtained query Q, value V and key K are input into the cross-attention network for feature fusion to obtain the second fused feature.

[0026] Optionally, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes a first fusion network;

[0027] Before inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the method further includes:

[0028] Inputting the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature;

[0029] Based on feature fusion of the second feature to be detected and the first fused feature, a specified input feature is obtained.

[0030] Optionally, the data to be detected is the first feature to be detected.

[0031] Optionally, the first fusion network is an attention pooling network, and the semantic feature extraction layer is a Transformer Block.

[0032] Optionally, the pre-trained audio large model is an initial audio large model obtained by at least model pre-training; and the model pre-training process of the second conversion model includes:

[0033] Obtain a first sample audio and a first label representing an emotion category of the first sample audio;

[0034] Extracting frequency domain features of the first sample audio as first sample features;

[0035] Inputting the first sample feature into an initial classification model to obtain an emotion classification result of the first sample audio; wherein the initial classification model includes an initial second conversion model and a classification network, and output data of the initial second conversion model is used as input data of the classification network;

[0036] Determining a first model loss value based on a difference between the obtained emotion classification result and the first label;

[0037] Based on the first model loss value, the model parameters of the initial classification model are adjusted until a first convergence condition is reached.

[0038] Optionally, the pre-trained audio model is an initial audio model that is obtained by model pre-training and then fine-tuning the model; the model fine-tuning process of the audio model after model pre-training includes:

[0039] Acquire a second sample audio, and a second label indicating whether bullying behavior exists in the collection environment of the second sample audio;

[0040] Extracting frequency domain features of the second sample audio as second sample features;

[0041] Inputting the second sample feature into the audio large model after model pre-training to obtain a detection result indicating whether bullying behavior exists in the collection environment of the second sample audio;

[0042] Determine a second model loss value based on a difference between the obtained detection result and the second label;

[0043] Based on the second model loss value, the specified model parameters of the large audio model after model pre-training are adjusted until the second convergence condition is reached.

[0044] Optionally, the designated model parameters are model parameters other than the model parameters of the semantic feature extraction network, the model parameters of the emotion feature extraction network, and the model parameters of the large language model.

[0045] Optionally, the method further comprises:

[0046] When it is determined that bullying behavior exists in the collection environment of the audio to be detected, an alarm is issued in a first alarm mode.

[0047] In a second aspect, an embodiment of the present application provides a behavior detection method, which is applied to a front-end device, and the method includes:

[0048] Collecting the audio to be detected in the current environment, and extracting the frequency domain features of the audio to be detected as the first feature to be detected;

[0049] Performing bullying keyword recognition on the first feature to be detected to obtain a keyword recognition result indicating whether the audio to be detected contains the bullying keyword;

[0050] If the keyword recognition result indicates that the audio to be detected contains bullying keywords, the data to be detected is sent to the central device, so that after the central device receives the data to be detected, it uses a pre-trained audio model to perform intent recognition on the data to be detected, and obtains a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected.

[0051] Optionally, performing bullying keyword recognition on the first feature to be detected to obtain a keyword recognition result indicating whether the audio to be detected contains the bullying keyword includes:

[0052] The first feature to be detected is input into a pre-trained keyword recognition network to obtain a keyword recognition result indicating whether the audio to be detected contains a bullying keyword.

[0053] Optionally, the method further comprises:

[0054] If the keyword recognition result indicates that the audio to be detected contains bullying keywords, an alarm is issued in the second alarm mode.

[0055] In a third aspect, an embodiment of the present application provides a behavior detection system, the system comprising a central terminal device and a front-end device;

[0056] The central terminal device is used to execute any one of the above behavior detection methods applied to the central terminal device;

[0057] The front-end device is used to execute any one of the above-mentioned behavior detection methods applied to the front-end device.

[0058] Optionally, there are multiple front-end devices, and the multiple front-end devices are deployed at different locations;

[0059] Each front-end device is specifically configured to send the data to be detected and its own location information to the central device when the keyword recognition result indicates that the audio to be detected contains bullying keywords;

[0060] The central terminal device is also used to, when it is determined based on the data to be detected sent by any front-end device that bullying behavior exists, issue an alarm in a first alarm mode according to the location information of the front-end device.

[0061] In a fourth aspect, an embodiment of the present application provides a behavior detection device, which is applied to a central terminal device, and the device includes:

[0062] A receiving module, used for receiving data to be detected; wherein the data to be detected is: the front-end device performs bullying keyword recognition on the first feature to be detected, and is uploaded when the keyword recognition result indicates that the audio to be detected contains the bullying keyword, the first feature to be detected is: the front-end device extracts frequency domain features from the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected;

[0063] The intention recognition module is used to use a pre-trained audio model to perform intention recognition on the data to be detected, and obtain a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected.

[0064] Optionally, the large audio model includes a large language model and a first conversion model, wherein the first conversion model includes a semantic feature extraction network for extracting context information;

[0065] The intention recognition module includes:

[0066] A feature extraction submodule, used for extracting features from the data to be detected using the semantic feature extraction network to obtain a second feature to be detected;

[0067] The model processing submodule is used to input the specified input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected.

[0068] Optionally, the audio large model further includes a second conversion model, wherein the second conversion model includes an emotion feature extraction network for extracting emotion information;

[0069] The device also includes:

[0070] an emotion feature extraction module, configured to extract features from the data to be detected using the emotion feature extraction network to obtain a third feature to be detected before the model processing submodule inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected;

[0071] The first feature fusion module is used to obtain a specified input feature based on feature fusion of the second feature to be detected and the third feature to be detected.

[0072] Optionally, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, and the second conversion model further includes a first fusion network and a second fusion network;

[0073] The device also includes:

[0074] A first fusion processing module is configured to input the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature before the model processing submodule inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected;

[0075] The first feature fusion module includes:

[0076] A first fusion submodule is used to use the second fusion network to perform feature fusion on the first fusion feature and the third feature to be detected to obtain a second fusion feature;

[0077] The second fusion submodule is used to obtain a specified input feature based on feature fusion of the second feature to be detected and the second fused feature.

[0078] Optionally, the second fusion network is a cross attention network;

[0079] The first fusion submodule is specifically used for:

[0080] The first fused feature is mapped to obtain the query Q and value V in the input data of the cross-attention network, and the third feature to be detected is mapped to obtain the key K in the input data of the cross-attention network, and the obtained query Q, value V and key K are input into the cross-attention network for feature fusion to obtain the second fused feature.

[0081] Optionally, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes a first fusion network;

[0082] The device also includes:

[0083] A second fusion processing module is configured to input the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature before the model processing submodule inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected;

[0084] The second feature fusion module is used to obtain a specified input feature based on feature fusion of the second feature to be detected and the first fused feature.

[0085] Optionally, the data to be detected is the first feature to be detected.

[0086] Optionally, the first fusion network is an attention pooling network, and the semantic feature extraction layer is a Transformer Block.

[0087] Optionally, the pre-trained audio large model is an initial audio large model obtained by at least model pre-training; and the model pre-training process of the second conversion model includes:

[0088] Obtain a first sample audio and a first label representing an emotion category of the first sample audio;

[0089] Extracting frequency domain features of the first sample audio as first sample features;

[0090] Inputting the first sample feature into an initial classification model to obtain an emotion classification result of the first sample audio; wherein the initial classification model includes an initial second conversion model and a classification network, and output data of the initial second conversion model is used as input data of the classification network;

[0091] Determining a first model loss value based on a difference between the obtained emotion classification result and the first label;

[0092] Based on the first model loss value, the model parameters of the initial classification model are adjusted until a first convergence condition is reached.

[0093] Optionally, the pre-trained audio model is an initial audio model that is obtained by model pre-training and then fine-tuning the model; the model fine-tuning process of the audio model after model pre-training includes:

[0094] Acquire a second sample audio, and a second label indicating whether bullying behavior exists in the collection environment of the second sample audio;

[0095] Extracting frequency domain features of the second sample audio as second sample features;

[0096] Inputting the second sample feature into the audio large model after model pre-training to obtain a detection result indicating whether bullying behavior exists in the collection environment of the second sample audio;

[0097] Determine a second model loss value based on a difference between the obtained detection result and the second label;

[0098] Based on the second model loss value, the specified model parameters of the large audio model after model pre-training are adjusted until the second convergence condition is reached.

[0099] Optionally, the designated model parameters are model parameters other than the model parameters of the semantic feature extraction network, the model parameters of the emotion feature extraction network, and the model parameters of the large language model.

[0100] Optionally, the device further comprises:

[0101] The first alarm module is used to issue an alarm in a first alarm mode when it is determined that bullying behavior exists in the collection environment of the audio to be detected.

[0102] In a fifth aspect, an embodiment of the present application provides a behavior detection device, which is applied to a front-end device, and the device includes:

[0103] A collection module, used for collecting the audio to be detected in the current environment, and extracting the frequency domain feature of the audio to be detected as the first feature to be detected;

[0104] A keyword recognition module, used to perform bullying keyword recognition on the first feature to be detected, and obtain a keyword recognition result indicating whether the audio to be detected contains bullying keywords;

[0105] A sending module is used to send the data to be detected to the central device if the keyword recognition result indicates that the audio to be detected contains bullying keywords, so that after the central device receives the data to be detected, it uses a pre-trained audio model to perform intent recognition on the data to be detected, and obtains a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected.

[0106] Optionally, the keyword identification module is specifically used to:

[0107] The first feature to be detected is input into a pre-trained keyword recognition network to obtain a keyword recognition result indicating whether the audio to be detected contains a bullying keyword.

[0108] Optionally, the device further comprises:

[0109] The second alarm module is used to issue an alarm in a second alarm mode if the keyword recognition result indicates that the audio to be detected contains bullying keywords.

[0110] In a sixth aspect, an embodiment of the present application provides an electronic device, including:

[0111] Memory, used to store computer programs;

[0112] The processor is used to implement any of the above-mentioned behavior detection methods when executing the program stored in the memory.

[0113] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the behavior detection method described in any one of the above items is implemented.

[0114] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any of the above-mentioned behavior detection methods.

[0115] Beneficial effects of the embodiments of the present application:

[0116] In the solution provided by the embodiment of the present application, since the data to be detected is uploaded by the front-end device after the front-end device has identified bullying keywords for the first detection feature, and the keyword identification result indicates that the audio to be detected contains bullying keywords, therefore, the data to be detected uploaded by the front-end device is data that may contain bullying behavior after preliminary detection by the front-end device. After receiving the data to be detected uploaded by the front-end, the central-end device uses a pre-trained large audio model to perform intent recognition on the data to be detected, and can further analyze the true intention expressed by the audio to be detected, thereby more accurately determining whether there is bullying behavior in the collection environment of the audio to be detected. It can be seen that this solution can effectively detect bullying behavior through the cooperation between the front-end device and the central-end device.

[0117] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0119] Figure 1 A flow chart of a behavior detection method provided in an embodiment of the present application;

[0120] Figure 2 A flowchart for implementing step S102 provided in an embodiment of the present application;

[0121] Figure 3 A flowchart of a model pre-training process of a second conversion model provided in an embodiment of the present application;

[0122] Figure 4 A flowchart of a model fine-tuning process of a large audio model provided in an embodiment of the present application;

[0123] Figure 5 A flowchart of another behavior detection method provided in an embodiment of the present application;

[0124] Figure 6 A schematic diagram of the structure of a behavior detection system provided in an embodiment of the present application;

[0125] Figure 7 A flowchart of a specific example of a behavior detection method provided in an embodiment of the present application;

[0126] Figure 8 A schematic diagram of the model structure of a large audio model provided in an embodiment of the present application;

[0127] Fig. 9 A schematic diagram of the structure of a behavior detection device provided in an embodiment of the present application;

[0128] Fig.10 A schematic diagram of the structure of another behavior detection device provided in an embodiment of the present application;

[0129] Fig.11 A block diagram of an electronic device for implementing the behavior detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0130] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0131] Next, a behavior detection method provided in an embodiment of the present application is introduced.

[0132] A behavior detection method provided in an embodiment of the present application requires the cooperation of a central device and a front-end device. In actual applications, the front-end device collects the audio to be detected in the current environment, and when it detects that the audio to be detected contains bullying keywords, it sends the data to be detected to the central device. After receiving the data to be detected, the central device detects whether there is bullying behavior in the collection environment of the audio to be detected. Exemplarily, the front-end device can be various electronic devices with audio collection and voice recognition functions, such as an anti-bullying voice alarm; the central device can be a cloud server, a server cluster, and so on.

[0133] Specifically, since the front-end device and the central-end device need to cooperate to complete the process, corresponding computer programs need to exist in both the front-end device and the central-end device to process the audio or data to be detected.

[0134] Among them, a behavior detection method provided in an embodiment of the present application is applied to a central terminal device, including:

[0135] Receive data to be detected; wherein the data to be detected is: the front-end device identifies bullying keywords on the first feature to be detected, and uploads it when the keyword identification result indicates that the audio to be detected contains bullying keywords, the first feature to be detected is: the front-end device extracts frequency domain features of the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected;

[0136] The pre-trained audio model is used to perform intent recognition on the data to be tested, and a detection result is obtained to indicate whether bullying behavior exists in the collection environment of the audio to be tested.

[0137] In the solution provided by the embodiment of the present application, since the data to be detected is uploaded by the front-end device after the front-end device has identified bullying keywords for the first detection feature, and the keyword identification result indicates that the audio to be detected contains bullying keywords, therefore, the data to be detected uploaded by the front-end device is data that may contain bullying behavior after preliminary detection by the front-end device. After receiving the data to be detected uploaded by the front-end, the central-end device uses a pre-trained large audio model to perform intent recognition on the data to be detected, and can further analyze the true intention expressed by the audio to be detected, thereby more accurately determining whether there is bullying behavior in the collection environment of the audio to be detected. It can be seen that this solution can effectively detect bullying behavior through the cooperation between the front-end device and the central-end device.

[0138] The behavior detection method provided in the embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0139] like Figure 1 The behavior detection method provided in the embodiment of the present application is applied to a central terminal device, and includes steps S101-S102:

[0140] S101, receiving data to be detected; wherein the data to be detected is: the front-end device performs bullying keyword recognition on the first feature to be detected, and is uploaded when the keyword recognition result indicates that the audio to be detected contains the bullying keyword, the first feature to be detected is: the front-end device performs frequency domain feature extraction on the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected;

[0141] For example, in practical applications, in order to effectively warn of campus bullying, front-end devices can be deployed in hidden corners of the campus or areas that cannot be covered by security cameras, so that the front-end devices can collect sounds in the current environment and obtain audio to be detected. For example, the front-end device can collect sounds in the current environment in real time, and divide the collected audio according to a preset time length to obtain each audio to be detected. The preset time length can be 10s, 20s, and so on.

[0142] After the front-end device collects the audio to be detected, it extracts the frequency domain features of the audio to be detected to obtain the first feature to be detected. It can be understood that the frequency domain feature extraction of the audio to be detected is to convert the audio to be detected from the time domain to the frequency domain, so as to analyze the frequency components of the audio to be detected and provide a basis for subsequent speech recognition. Exemplarily, the method of extracting the frequency domain features of the audio to be detected can be to use transformation functions such as FFT (Fast Fourier Transform) and MFCC (Mel-scale Frequency Cepstral Coefficients) to convert the audio to be detected from the time domain to the frequency domain to obtain the first feature to be detected.

[0143] After the front-end device obtains the first feature to be detected, it identifies bullying keywords for the first feature to be detected, and when the keyword identification result indicates that the audio to be detected contains bullying keywords, it sends the data to be detected to the central device. The data to be detected is used to characterize the audio to be detected. In one implementation, the data to be detected is the first feature to be detected obtained by extracting frequency domain features of the audio to be detected. In this way, when the front-end device sends the data to be detected to the central device, the problem of privacy leakage caused by direct transmission of audio can be avoided, and the transmission bandwidth can be greatly reduced. It should be noted that the embodiments of the present application do not limit the data to be detected that characterizes the audio to be detected. For example, in another implementation, the data obtained after homomorphic encryption processing of the audio to be detected can also be used as the data to be detected, or, without considering privacy, the data to be detected can also be the audio to be detected itself, which is reasonable.

[0144] For example, bullying keywords may include "help", "fight", "beat to death", "murder", etc. It should be noted that in actual applications, bullying keywords may be set by relevant technicians based on experience, and the embodiments of the present application are not limited thereto.

[0145] It is understandable that when the audio to be detected contains bullying keywords, bullying behavior may or may not exist in the collection environment of the audio to be detected. For example, if the audio to be detected is "It's windy and thundering outside. If I go home at this time, I will probably be struck by lightning", then although the audio to be detected contains the bullying keyword "beat to death", there is no bullying behavior in the collection environment of the audio to be detected; if the audio to be detected is "Please, stop beating, people are going to be beaten to death", then there is bullying behavior in the collection environment of the audio to be detected.

[0146] Therefore, in order to more accurately determine whether there is bullying in the collection environment of the audio to be detected, after the front-end device detects that the audio to be detected contains bullying keywords, it sends the data to be detected to the central device. The central device receives the data to be detected and further analyzes the data to be detected to determine whether there is bullying.

[0147] In addition, if the keyword recognition result indicates that the audio to be detected contains bullying keywords, the front-end device can also alarm in the second alarm mode. Exemplarily, the second alarm mode can be to send an alarm prompt message containing its own location information to the central end device, or directly play an alarm prompt voice, for example, "Please stop bullying", etc. In this way, bullying behavior in the campus scene can be warned in a timely manner.

[0148] S102, using a pre-trained audio large model to perform intent recognition on the data to be detected, and obtaining a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected.

[0149] In this embodiment, the pre-trained audio large model can be SenseVoice (a multi-language recognition and emotion recognition model), or an audio large model including an LLM (Large Language Model) model, and so on. In order to make the solution layout clear, the specific structure and training method of the audio large model are described in the following embodiments, which will not be repeated here. It can be understood that the audio large model has a strong ability to understand intent, and it can accurately analyze the true intention represented by the data to be tested, so as to accurately determine whether there is bullying behavior in the collection environment of the audio to be tested.

[0150] It is understandable that the data to be detected is uploaded by the front-end device when it detects that the audio to be detected contains bullying keywords. After the front-end device recognizes the audio to be detected, it can preliminarily determine that bullying behavior may exist in the collection environment of the audio to be detected. Therefore, in order to more accurately determine whether there is bullying behavior in the collection environment of the audio to be detected, after receiving the data to be detected, the central end device uses the pre-trained audio large model to perform intent recognition on the data to be detected, so as to understand the intent of the data to be detected and obtain accurate detection results.

[0151] In addition, it is understandable that since the front-end device only needs to identify bullying keywords, there is no need to deploy a large audio model in the front-end device. Compared with directly performing intent recognition in the front-end device, this solves the problem of limited recognition ability of intent recognition in the front-end device due to limited resources of the front-end device, which in turn leads to low accuracy of the detection results.

[0152] Since the front-end device only needs to identify bullying keywords, the front-end device can quickly determine the audio to be detected that indicates possible bullying behavior from the collected audio. Since the front-end device sends the data to be detected to the central device only when it detects that the audio to be detected contains bullying keywords, that is, the front-end device performs an initial screening of the collected audio, and only sends it to the central device for intent recognition when the audio to be detected indicates possible bullying behavior, then, compared to directly sending all the audio to be detected to the central device for detection, the workload of the central device is reduced, thereby improving the detection efficiency. Therefore, through the cooperation between the front-end device and the central device, it is possible to accurately determine whether there is a detection result of bullying behavior in the collection environment of the audio to be detected.

[0153] In addition, when the central terminal device determines that there is bullying behavior in the collection environment of the audio to be detected, it can also alarm in the first alarm mode. Exemplarily, the first alarm mode can be to send an alarm prompt information to the security system of the campus where the audio to be detected is collected, or output an alarm prompt sound, etc. In this way, bullying behavior in the campus scene can be effectively handled.

[0154] In the solution provided by the embodiment of the present application, since the data to be detected is uploaded by the front-end device after the front-end device has identified bullying keywords for the first detection feature, and the keyword identification result indicates that the audio to be detected contains bullying keywords, therefore, the data to be detected uploaded by the front-end device is data that may contain bullying behavior after preliminary detection by the front-end device. After receiving the data to be detected uploaded by the front-end, the central-end device uses a pre-trained large audio model to perform intent recognition on the data to be detected, and can further analyze the true intention expressed by the audio to be detected, thereby more accurately determining whether there is bullying behavior in the collection environment of the audio to be detected. It can be seen that this solution can effectively detect bullying behavior through the cooperation between the front-end device and the central-end device.

[0155] Optionally, in another embodiment of the present application, the large audio model includes a large language model and a first conversion model, and the first conversion model includes a semantic feature extraction network for extracting context information;

[0156] Accordingly, in this embodiment, if Figure 2 As shown, in the above step S102, the pre-trained audio model is used to perform intent recognition on the data to be detected, and a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected is obtained, including steps S201-S202:

[0157] S201, extracting features from the data to be detected using a semantic feature extraction network to obtain a second feature to be detected;

[0158] S202, inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected.

[0159] In this embodiment, the semantic feature extraction network can be a neural network composed of multiple serially connected Transformer Blocks or a RNN (Recurrent Neural Network), which is reasonable.

[0160] It can be understood that by inputting the data to be detected into the semantic feature extraction network, the data to be detected can be converted into a token sequence, that is, the data to be detected is tokenized, thereby obtaining data suitable for processing by a large language model. In addition, by using the semantic feature extraction network to extract features from the data to be detected, the contextual information in the data to be detected can be captured. After obtaining the second feature to be detected, the second feature to be detected can be mapped to the text space to obtain a specified input feature, and the specified input feature can be input into the large language model, and the large language model can be used to perform intent recognition on the specified input feature, so that the large language model can more accurately perform intent recognition based on the contextual information captured from the data to be detected, thereby obtaining a more accurate detection result. It can be seen that through this solution, the accuracy of the detection results can be improved.

[0161] In addition, it is understandable that since the large language model is a deep learning model trained using a large amount of text data, and the second feature to be detected is an audio feature obtained by processing the audio to be detected, in order to use the large language model for intent recognition, it is necessary to convert the second detection feature into a semantic feature and use it as the model input of the large language model. That is, the designated input feature obtained based on the second feature to be detected is a semantic feature. Exemplarily, the designated input feature obtained based on the second feature to be detected can be a designated input feature obtained by converting the second detection feature into a semantic feature using an adapter layer. Among them, the Adapter layer is composed of multiple fully connected layers, which can convert audio features into semantic features.

[0162] Optionally, in another embodiment of the present application, Figure 2 On the basis of the illustrated embodiment, the audio large model further includes a second conversion model, and the second conversion model includes an emotion feature extraction network for extracting emotion information;

[0163] Accordingly, in this embodiment, before inputting the designated input feature obtained based on the second feature to be detected into the large language model in step S202 to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the above-mentioned behavior detection method further includes steps A1-A2:

[0164] A1, using the emotion feature extraction network to extract features from the data to be detected, and obtaining the third feature to be detected;

[0165] A2, obtaining a specified input feature based on feature fusion of the second feature to be detected and the third feature to be detected.

[0166] In this embodiment, the emotion feature extraction network can be obtained by supervised training of an initial deep neural network using emotion labels, and the deep neural network can be CNN (Convolutional Neural Networks), or a neural network based on a transformer structure, etc. Exemplarily, in one implementation, the emotion feature extraction network can be SER experts (Speech Emotion Recognition Expert System). It should be noted that the embodiment of the present application does not limit the network structure of the emotion feature extraction network.

[0167] It can be understood that after the initial deep neural network is supervised and trained using the emotion labels, the obtained emotion feature extraction network performs feature extraction on the data to be detected and can capture the emotion information in the data to be detected.

[0168] After obtaining the third feature to be detected, the specified input feature is obtained based on the feature fusion of the second feature to be detected and the third feature to be detected. In practical applications, based on the feature fusion of the second feature to be detected and the third feature to be detected, the second feature to be detected and the third feature to be detected can be converted into semantic features and then feature fusion is performed. Exemplarily, the feature fusion method can be splicing, or weighted fusion, which is reasonable.

[0169] It can be understood that by inputting the specified input features obtained by feature fusion of the second feature to be detected and the third feature to be detected into the large language model, the large language model can combine the contextual information and emotional information captured from the data to be detected to perform intent recognition, thereby obtaining more accurate detection results.

[0170] Optionally, in one implementation, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, and the second conversion model further includes a first fusion network and a second fusion network;

[0171] Accordingly, in this implementation, before inputting the designated input feature obtained based on the second feature to be detected into the large language model in step S202 to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the behavior detection method further includes:

[0172] Inputting the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature;

[0173] Accordingly, in the above step A2, based on feature fusion of the second feature to be detected and the third feature to be detected, the specified input feature is obtained, including steps A21-A22:

[0174] A21, using a second fusion network to fuse the first fusion feature and the third feature to be detected to obtain a second fusion feature;

[0175] A22, obtaining a specified input feature based on feature fusion of the second feature to be detected and the second fused feature.

[0176] In this implementation, the designated input feature obtained based on the second feature to be detected in the above step S202 is obtained based on feature fusion of the second feature to be detected and the second fusion feature.

[0177] In one implementation, the second number is smaller than the first number. In this case, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series. The data to be detected is input into the semantic feature extraction network. The output features of the first second number of semantic feature extraction layers in the semantic feature extraction network are shallow features in the data to be detected, that is, focusing on the physical properties of the data to be detected, such as pitch, timbre, length, etc. These shallow features are crucial for understanding the audio content.

[0178] Exemplarily, if the first number is 12 and the second number is 3, the output features of the 1st to 3rd layers in the semantic feature extraction network can be input into the first fusion network to fuse the output features output by different semantic feature extraction layers in the semantic feature extraction network.

[0179] It should be noted that the embodiment of the present application does not limit the above-mentioned second number. For example, in another implementation, the second number may be equal to the first number.

[0180] Since the deeper the network, the larger the receptive field, the output features of the first second number of semantic feature extraction layers are fused to obtain the feature representation of the global features + local features of the data to be detected. This can enhance the expressiveness of the model and improve the generalization ability of the model.

[0181] After obtaining the first fusion feature, the second fusion network is used to fuse the first fusion feature and the third feature to be detected, that is, to fuse the features output by multiple different extraction layers of the semantic feature extraction network and the output features of the emotion feature extraction network. In this way, the obtained second fusion feature can more accurately express the emotional content in the data to be detected.

[0182] Exemplarily, the second conversion model may also include a conversion layer for converting the second fused feature into a tokenized semantic feature, such as a proj (engineering) layer, which may be composed of a fully connected layer for mapping audio features into tokenized semantic features. In one implementation, the designated input feature is obtained by converting the second fused feature into a semantic feature and concatenating the semantic feature obtained by converting the second feature to be detected. It can be understood that the feature obtained by feature fusion based on the second feature to be detected and the second fused feature is used as the designated input feature and input into the large language model, which can further improve the accuracy of the detection result output by the large language model.

[0183] Exemplarily, in one implementation, the first fusion network is an attention pooling network (Attentivepooling), and the semantic feature extraction layer is a Transformer Block.

[0184] In this implementation, the output features of the first second number of Transformer Blocks are input into the attention pooling network, and the attention pooling network is used to assign different weights to the output features of different Transformer Blocks, and the output features of the first second number of Transformer Blocks are weighted averaged according to the assigned weights to obtain the first fusion feature. It can be understood that the semantic feature extraction layer is a Transformer Block, which can model global context information by applying the self-attention mechanism, so as to better realize the effective capture of context information.

[0185] It should be noted that the embodiments of the present application do not limit the specific types of the first fusion network and the semantic feature extraction layer. For example, the first fusion network can also be an average pooling network, the semantic feature extraction layer can also be an LSTM (Long Short-Term Memory), and so on.

[0186] Optionally, in one implementation, the second fusion network is a cross attention network (Crossattention);

[0187] Accordingly, in this implementation, the first fusion feature and the third feature to be detected are fused using the second fusion network to obtain the second fusion feature, including:

[0188] The first fused feature is mapped to obtain Q and V in the input data of the cross-attention network, and the third feature to be detected is mapped to obtain K in the input data of the cross-attention network, and the obtained Q, V and K are input into the cross-attention network for feature fusion to obtain the second fused feature.

[0189] In this implementation, in order to use the cross-attention network to perform feature fusion on the first fused feature and the third feature to be detected, it is necessary to convert the first fused feature and the third feature to be detected into a form suitable for attention mechanism calculation, that is, map the first fused feature to Q and V, and map the third feature to be detected to K, and obtain Q, K and V as input data of the cross-attention network.

[0190] Exemplarily, the first fused feature may be processed using a preset first transformation function to map the first fused feature to Q and V, and the third feature to be detected may be processed using a preset second transformation function to map the third feature to be detected to K. The first transformation function and the second transformation function may be preset linear variation functions, and in practical applications, the first transformation function and the second transformation function may be set by relevant technical personnel.

[0191] It can be understood that by inputting Q, K, and V obtained based on the first fusion feature and the third feature to be detected into the cross-attention network, the cross-attention network can extract the hidden layer representation between the context information and the emotional information in the data to be detected, that is, it can learn how to effectively combine the context information and the emotional information to generate a more representative feature representation. In this way, the designated input features determined by the second fusion feature obtained based on the cross-attention network are used as input data of the large language model, which can further enhance the large language model's ability to process emotional information and context information, thereby further improving the accuracy of the detection results output by the large language model.

[0192] Optionally, in another embodiment of the present application, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes the first fusion network;

[0193] Accordingly, in this embodiment, before inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected, the behavior detection method further includes:

[0194] Inputting the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature;

[0195] Based on the feature fusion of the second feature to be detected and the first fusion feature, a specified input feature is obtained.

[0196] In this embodiment, the output features of the first second number of semantic feature extraction layers are fused to obtain a feature representation of the global features + local features of the data to be detected. The second feature to be detected is a feature output by the semantic feature extraction network, which is a deep feature in the data to be detected, that is, more abstract semantic information in the data to be detected. Exemplarily, based on the feature fusion of the second feature to be detected and the first fused feature, the feature fusion can be performed after both the second feature to be detected and the first fused feature are converted into semantic features.

[0197] It can be understood that, based on the specified input features obtained by fusion of the first fusion feature and the second feature to be detected, the large language model is input, so that the large language model can comprehensively utilize information at different levels, thereby improving the accuracy of intent recognition by the large language model and the generalization ability of the model.

[0198] It should be noted that the semantic feature extraction network and the first fusion network in this embodiment are the same as those in the above embodiment and will not be described in detail here.

[0199] Optionally, in another embodiment of the present application, the pre-trained audio model is an initial audio model obtained by at least model pre-training; Figure 3 As shown, the model pre-training process of the second conversion model includes steps S301-S305:

[0200] S301, obtaining a first sample audio and a first label representing an emotion category of the first sample audio;

[0201] S302, extracting frequency domain features of the first sample audio as first sample features;

[0202] S303, inputting the first sample feature into an initial classification model to obtain an emotion classification result of the first sample audio; wherein the initial classification model includes an initial second conversion model and a classification network, and output data of the initial second conversion model is used as input data of the classification network;

[0203] S304, determining a first model loss value based on a difference between the obtained emotion classification result and the first label;

[0204] S305: Based on the first model loss value, adjust the model parameters of the initial classification model until a first convergence condition is reached.

[0205] In this embodiment, the first label of each first sample audio can be obtained by manually or machine-labeling the emotion category of the first sample audio, that is, the first label corresponding to each first sample audio is used as the true value of the emotion category of the first sample audio during the model pre-training process. Exemplarily, emotion categories may include happiness, sadness, anger, fear, and so on. It should be noted that the first sample audio can be audio obtained from an open source audio dataset, or audio collected in real time by an audio acquisition device, which is reasonable.

[0206] After obtaining the first sample audio, the frequency domain features of the first sample audio are extracted as the first sample features. The method of extracting the frequency domain features of the first sample audio can refer to the method of extracting the frequency domain features of the audio to be detected by the front-end device in step S101 above, which will not be repeated here.

[0207] After the initial second conversion model, a classification network is connected to use the output data of the second conversion model as the input data of the classification network. The initial second conversion model and the classification network constitute an initial classification model. Exemplarily, the initial second conversion model can be a neural network model based on a transformer structure, or a neural network model composed of multiple convolutional layers and fully connected layers, which is reasonable.

[0208] The first sample feature is input into the initial classification model to obtain the emotion classification result output by the classification model. Exemplarily, the first model loss value is determined based on the difference between the obtained emotion classification result and the first label by using a loss function such as a Euclidean distance loss function or a cosine distance loss function to calculate the difference between the emotion classification result corresponding to each first sample audio and its corresponding first label, and then taking the sum or mean of the differences as the first model loss value.

[0209] After obtaining the first model loss value, the various model parameters in the initial classification model are adjusted by minimizing the back propagation method of the first model loss value until the first convergence condition is reached. Exemplarily, the first convergence condition may be that the first model loss value is less than a first preset threshold. The first preset threshold is set by relevant technical personnel based on experience. It is understandable that after the classification model converges, the second conversion model in the classification model is the second conversion model that has been pre-trained by the model.

[0210] In addition, the large language model in the audio large model can be an open source, pre-trained large language model obtained from the Internet.

[0211] It can be seen that since the second conversion model that has undergone model pre-training is obtained by supervised training of the classification model, the second conversion model that has undergone model pre-training can improve the ability to extract emotional information.

[0212] Optionally, in another embodiment of the present application, the pre-trained audio model is an initial audio model that is obtained by pre-training the model and then fine-tuning the model; Figure 4 As shown, the model fine-tuning process of the audio model after model pre-training includes steps S401-S405:

[0213] S401, obtaining a second sample audio and a second label indicating whether there is bullying behavior in the collection environment of the second sample audio;

[0214] S402, extracting frequency domain features of the second sample audio as second sample features;

[0215] S403, inputting the second sample feature into the audio large model after model pre-training to obtain a detection result indicating whether bullying behavior exists in the collection environment of the second sample audio;

[0216] S404, determining a second model loss value based on a difference between the obtained detection result and the second label;

[0217] S405: Based on the second model loss value, the specified model parameters of the large audio model after model pre-training are adjusted until the second convergence condition is reached.

[0218] In this embodiment, the second label of each second sample audio can be obtained by manually or machine-marking whether there is bullying behavior in the collection environment of the second sample audio, that is, the second label corresponding to each second sample audio is used as the true value of whether there is bullying behavior in the collection environment of the second sample audio during the model fine-tuning process. It should be noted that the second sample audio can be audio obtained from an open source audio dataset, or audio collected in real time by an audio collection device, which is reasonable.

[0219] After obtaining the second sample audio, the frequency domain features of the second sample audio are extracted as the second sample features. The method of extracting the frequency domain features of the second sample audio can refer to the method of extracting the frequency domain features of the audio to be detected by the front-end device in step S101 above, which will not be repeated here.

[0220] Exemplarily, the method for determining the second model loss value based on the difference between the obtained detection result and the second label can be: using a loss function such as a Euclidean distance loss function and a cosine distance loss function to calculate the difference between the detection result corresponding to each second sample feature and the corresponding second label, and then using the sum or mean of the differences as the second model loss value.

[0221] After obtaining the second model loss value, the specified model parameters of the audio model after model pre-training are adjusted by minimizing the back propagation of the second model loss value until the second convergence condition is reached. Exemplarily, the second convergence condition may be that the second model loss value is less than a second preset threshold. The second preset threshold is set by relevant technical personnel based on experience, and the second preset threshold may be the same as or different from the first preset threshold.

[0222] Exemplarily, the specified model parameters of the audio large model after model pre-training may be model parameters in the audio large model other than the model parameters of the large language model, and the embodiments of the present application do not limit the specified model parameters. Optionally, in one implementation, the specified model parameters are model parameters other than the model parameters of the semantic feature extraction network, the model parameters of the emotional feature extraction network, and the model parameters of the large language model.

[0223] It can be understood that in the process of fine-tuning the model of the large audio model, by locking the model parameters of the semantic feature extraction network, the model parameters of the emotional feature extraction network, and the model parameters of the large language model, and adjusting other model parameters, the speed of model fine-tuning can be accelerated and the model accuracy of the large audio model can be further improved.

[0224] It can be seen that through this solution, fine-tuning the large audio model after model pre-training can further improve the model accuracy of the large audio model.

[0225] Corresponding to the above-mentioned behavior detection method provided from the central end device side, the embodiment of the present application also provides a behavior detection method applied to the front-end device, such as Figure 5 As shown, steps S501-S503 are included:

[0226] S501, collecting audio to be detected in the current environment, and extracting frequency domain features of the audio to be detected as first features to be detected;

[0227] It should be noted that the content of step S501 has been explained in the above step S101 and will not be repeated here.

[0228] S502, performing bullying keyword recognition on the first feature to be detected to obtain a keyword recognition result indicating whether the audio to be detected contains the bullying keyword;

[0229] For example, in practical applications, a neural network model for identifying keywords can be used to identify bullying keywords.

[0230] Optionally, in one implementation, bullying keyword recognition is performed on the first feature to be detected to obtain a keyword recognition result indicating whether the audio to be detected contains the bullying keyword, including:

[0231] The first feature to be detected is input into a pre-trained keyword recognition network to obtain a keyword recognition result indicating whether the audio to be detected contains bullying keywords.

[0232] In this implementation, the keyword recognition network is a classification network. In one implementation, the keyword recognition network is a binary classification network, which is used to predict the probability value of the audio to be detected containing bullying keywords, and then compare the predicted probability value with the preset classification threshold. If the probability value is greater than the preset classification threshold, a keyword recognition result containing bullying keywords in the audio to be detected is obtained, otherwise a keyword recognition result that does not contain bullying keywords in the audio to be detected is obtained. It can be understood that by adjusting the classification threshold, the detection rate of bullying keywords identified by the keyword recognition network can be adjusted.

[0233] Exemplarily, the pre-trained keyword recognition network can be trained in the following manner: obtain a third sample audio and a third label characterizing whether the third sample audio contains bullying keywords; extract the frequency domain features of the third sample audio as the third sample features; input the third sample features into the initial keyword recognition network to obtain a recognition result indicating whether the third sample audio contains bullying keywords; determine a third model loss value based on the difference between the obtained recognition result and the third label; based on the third model loss value, adjust the model parameters of the initial keyword recognition network until the third convergence condition is reached.

[0234] The third label of each third sample audio can be obtained by manually or machine-marking whether the third sample audio contains bullying keywords, that is, the third label corresponding to each third sample audio is used as the true value of whether the third sample audio contains bullying keywords during the training process. Among them, the third sample audio can be audio obtained from an open source audio dataset, or audio collected in real time by an audio collection device, which is reasonable.

[0235] After obtaining the third sample audio, the frequency domain features of the third sample audio are extracted as the third sample features. The method of extracting the frequency domain features of the third sample audio can refer to the method of extracting the frequency domain features of the audio to be detected by the front-end device in step S101 above, which will not be repeated here.

[0236] Exemplarily, the method for determining the third model loss value based on the difference between the obtained recognition result and the third label can be: using loss functions such as the Euclidean distance loss function and the cosine distance loss function to calculate the difference between the recognition result corresponding to each third sample feature and its corresponding third label, and then taking the sum or mean of the differences as the third model loss value.

[0237] After obtaining the third model loss value, the model parameters of the initial keyword recognition network are adjusted by minimizing the back propagation method of the third model loss value until the third convergence condition is reached. Exemplarily, the third convergence condition may be that the third model loss value is less than a third preset threshold. The third preset threshold is set by relevant technical personnel based on experience, and the second preset threshold may be the same as or different from the first preset threshold or the second preset threshold.

[0238] It is understandable that by using the third sample audio and the third label to perform supervised training on the initial keyword recognition network, the obtained keyword recognition network can accurately identify whether the input first feature to be detected contains bullying keywords. It should be noted that the embodiment of the present application does not limit the network structure of the keyword recognition network. For example, the keyword recognition network can be a neural network based on a transformer structure, or CNN, etc.

[0239] In addition, since the keyword recognition network only needs to identify whether the first feature to be detected contains bullying keywords, it is usually a small model with fewer model parameters, that is, the number of model parameters is in the millions. In this way, it can run in real time in front-end devices with limited resources.

[0240] S503, if the keyword recognition result indicates that the audio to be detected contains bullying keywords, the data to be detected is sent to the central device, so that after the central device receives the data to be detected, it uses the pre-trained audio model to perform intent recognition on the data to be detected, and obtains a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected.

[0241] It should be noted that the content of step S503 has been explained in the above step S102 and will not be repeated here.

[0242] It can be seen that through this solution, after the front-end device identifies bullying keywords for the first detection feature, if the keyword recognition result indicates that the audio to be detected contains bullying keywords, it indicates that the audio to be detected may contain bullying behavior after preliminary detection by the front-end device. At this time, the data to be detected is sent to the central device, so that the central device uses the pre-trained audio model to perform intent recognition on the data to be detected after receiving the data to be detected, so as to further analyze the true intention expressed by the audio to be detected, so as to more accurately determine whether there is bullying behavior in the collection environment of the audio to be detected. It can be seen that this solution can effectively detect bullying behavior through the cooperation between the front-end device and the central device.

[0243] Corresponding to the above method embodiment, the present application embodiment also provides a behavior detection system, such as Figure 6 As shown, it includes a central terminal device 610 and a front-end device 620;

[0244] A central terminal device 610, configured to execute any one of the above behavior detection methods applied to a central terminal device;

[0245] The front-end device 620 is used to execute any of the above-mentioned behavior detection methods applied to the front-end device.

[0246] Optionally, in one implementation, there are multiple front-end devices, and the deployment locations of the multiple front-end devices are different;

[0247] Each front-end device is specifically configured to send the data to be detected and its own location information to the central device when the keyword recognition result indicates that the audio to be detected contains the bullying keyword;

[0248] The central end device is also used to, when it is determined based on the data to be detected sent by any front-end device that bullying behavior exists, issue an alarm in a first alarm mode according to the location information of the front-end device.

[0249] For example, in actual applications, multiple front-end devices can be deployed in hidden corners of the campus or areas that cannot be covered by security cameras, and the deployment locations of the multiple front-end devices are different.

[0250] When each front-end device sends the data to be detected to the central device, it also sends its own location information, so that when the central device determines that bullying behavior exists based on the data to be detected sent by any front-end device, it can use the location information of the front-end device to determine the location where the bullying behavior occurs, thereby being able to effectively sound an alarm to stop the bullying behavior.

[0251] Exemplarily, the way in which the central device alarms in the first alarm mode based on the location information of the front-end device may be to determine the location where the bullying behavior occurs based on the location information of the front-end device, thereby sending an alarm prompt information to the security system of the campus where the bullying behavior occurs, and the alarm prompt information sent may carry the location information of the front-end device, so that after seeing the alarm prompt information, the security personnel can rush to the location where the bullying behavior occurs in time to stop the bullying behavior, thereby improving campus safety.

[0252] Exemplarily, in one implementation, the front-end device is also used to sound an alarm in a second alarm mode when the keyword recognition result indicates that the audio to be detected contains bullying keywords. Exemplarily, the second alarm mode may be to send an alarm prompt message to the central device, or to directly play an alarm prompt voice. By using the front-end device and the central device to sound an alarm separately, a hierarchical alarm can be implemented, so that the front-end device can respond quickly locally. At the same time, for complex scenarios, the central device can further understand the intention of the audio to be detected, and can filter out false alarms from the front-end device.

[0253] It can be seen that bullying behavior can be effectively detected through this solution.

[0254] In order to better understand the content of the embodiments of the present application, a specific example is described below.

[0255] In this example, a small model for keyword recognition is deployed in the front-end device (corresponding to the keyword recognition network mentioned above), and a large model for intent recognition is deployed in the central device (corresponding to the large audio model mentioned above). The front-end device and the central device cooperate with each other to detect bullying behavior, and hierarchical alarms can be achieved through the collaborative processing of large and small models. The model parameters of the small model are in the millions, and the model parameters of the large model are in the billions. Figure 7 As shown, the process of the front-end device and the central device cooperating with each other to detect bullying behavior includes steps S701-S705:

[0256] S701, the front-end device collects sound to obtain the audio to be detected;

[0257] S702, the front-end device uses the deployed small model to perform keyword recognition on the audio to be detected, and obtains a keyword recognition result;

[0258] S703: If the keyword recognition result indicates that the audio to be detected contains bullying keywords, the front-end device issues a primary alarm (corresponding to the alarm in the second alarm mode described above), and uploads the frequency domain features of the audio to be detected and the alarm information to the central device;

[0259] S704, the central end device receives the frequency domain features of the audio to be detected, uses the large model to perform intent recognition, and obtains a detection result;

[0260] S705, if the detection result indicates that bullying behavior exists in the collection environment of the audio to be detected, the central end device issues a final alarm based on the alarm information (corresponding to the alarm in the first alarm mode described above).

[0261] It can be seen that through the above two-level filtering solution, the front-end device only needs to use a small model for keyword recognition, such as a model with 1.5M parameters, which can run in real time on ARM (embedded processor). In this way, it can quickly identify whether bullying keywords appear on the local device side, such as "help" and "fight", and these bullying keywords can support user customization.

[0262] The front-end device can adjust the detection rate of the small model for bullying keywords by adjusting the classification threshold. The front-end device allows false alarms to occur. When the front-end device detects bullying keywords, it uploads the collected frequency domain features and alarm information of the audio to be detected to the central device to use the large model deployed in the central device for intent recognition. Since the front-end device does not need to upload the audio itself, but uploads the frequency domain features of the audio to be detected, this can avoid privacy leakage and greatly reduce bandwidth. After receiving the alarm information and frequency domain features, the central end will send them to the large model for analysis. This example uses a model with 0.3B (300 million) parameters to perform intent analysis by combining the capabilities of the large language model to further determine whether the bullying words in the audio to be detected are really in a bullying environment, rather than a false alarm. For example, the child is joking or is actually a homophone of the bullying keyword. This can greatly improve the accuracy of bullying behavior detection results and reduce the probability of false alarms.

[0263] The big model deployed in the central device is an audio big model based on the EAT (Experience Awareness Theory) framework. The audio big model has a strong ability to understand intent and can accurately judge whether bullying occurs in campus scenarios. For example, when the bullying keyword "beat to death" appears, it can further analyze the true intention. If the audio content is "It's windy and thundering outside, and I'm afraid I'll be struck by thunder if I go home at this time", then the appearance of "beat to death" does not mean bullying; if the audio content is "Please, stop beating, people are going to be beaten to death", then "beat to death" is a typical bullying scene and needs to be reported in time. At the same time, the audio big model also has other capabilities such as emotion recognition, gender recognition, age group classification, and abnormal sound recognition, which can enrich the scene description and enable the central device to issue an alarm more accurately.

[0264] In this example, the model structure of the audio model is as follows Figure 8 As shown, the frequency domain features of the audio to be detected are respectively input into a semantic feature extraction network composed of multiple layers of Transformer Blocks connected in series, and are input into an emotional feature extraction network. Among them, the output of the last layer of the semantic feature extraction network is converted into the first semantic feature through the Adapter layer. The outputs of the first second number of Transformer Blocks connected in series in the semantic feature extraction network are respectively input into the attention pooling network, the output of the attention pooling network is mapped to Q and V, and the output of the emotional feature extraction network is mapped to K, and are jointly input into the cross attention network. The output of the cross attention network is mapped to the second semantic feature through the fully connected layer. The first semantic feature and the second semantic feature are jointly used as input data of the large language model and input into the large language model. The large language model outputs the detection result of whether there is bullying behavior in the collection environment of the audio to be detected.

[0265] It can be seen that this solution extracts emotional information by introducing an emotional feature extraction network, and extracts shallow features from the semantic feature extraction network. The shallow features and the emotional information extracted by the emotional feature extraction network are fused using a cross-attention network, which can improve the latent representation of the large model in terms of emotional information and contextual information. By fusing the new features output by the cross-attention network with the output features of the semantic feature extraction network, the model performance of the large audio model can be greatly improved, so that the large audio model has the ability to recognize emotions. Through the coordination of large and small models, hierarchical alarms can be achieved, giving this solution the advantage of local rapid response. At the same time, for complex scenarios, the central device can perform intent analysis through the capabilities of the large audio model and filter out false alarms caused by the local small model. At the same time, the data transmitted to the central end is the audio features of the audio, which can avoid privacy leakage caused by direct transmission of audio data and reduce bandwidth.

[0266] Corresponding to the above method embodiment provided from the central terminal device side, the present application embodiment provides a behavior detection device, which is applied to the central terminal device, such as Fig. 9 As shown, the device comprises:

[0267] The receiving module 910 is used to receive data to be detected; wherein the data to be detected is: the front-end device performs bullying keyword recognition on the first feature to be detected, and is uploaded when the keyword recognition result indicates that the audio to be detected contains the bullying keyword, the first feature to be detected is: the front-end device extracts frequency domain features from the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected;

[0268] The intention recognition module 920 is used to use a pre-trained audio model to perform intention recognition on the data to be detected, and obtain a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected.

[0269] Optionally, the large audio model includes a large language model and a first conversion model, wherein the first conversion model includes a semantic feature extraction network for extracting context information;

[0270] The intention recognition module 920 includes:

[0271] A feature extraction submodule, used for extracting features from the data to be detected using the semantic feature extraction network to obtain a second feature to be detected;

[0272] The model processing submodule is used to input the specified input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected.

[0273] Optionally, the audio large model further includes a second conversion model, wherein the second conversion model includes an emotion feature extraction network for extracting emotion information;

[0274] The device also includes:

[0275] an emotion feature extraction module, configured to extract features from the data to be detected using the emotion feature extraction network to obtain a third feature to be detected before the model processing submodule inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected;

[0276] The first feature fusion module is used to obtain a specified input feature based on feature fusion of the second feature to be detected and the third feature to be detected.

[0277] Optionally, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, and the second conversion model further includes a first fusion network and a second fusion network;

[0278] The device also includes:

[0279] A first fusion processing module is configured to input the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature before the model processing submodule inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected;

[0280] The first feature fusion module includes:

[0281] A first fusion submodule is used to use the second fusion network to perform feature fusion on the first fusion feature and the third feature to be detected to obtain a second fusion feature;

[0282] The second fusion submodule is used to obtain a specified input feature based on feature fusion of the second feature to be detected and the second fused feature.

[0283] Optionally, the second fusion network is a cross attention network;

[0284] The first fusion submodule is specifically used for:

[0285] The first fused feature is mapped to obtain the query Q and value V in the input data of the cross-attention network, and the third feature to be detected is mapped to obtain the key K in the input data of the cross-attention network, and the obtained query Q, value V and key K are input into the cross-attention network for feature fusion to obtain the second fused feature.

[0286] Optionally, the semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes a first fusion network;

[0287] The device also includes:

[0288] A second fusion processing module is configured to input the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature before the model processing submodule inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected;

[0289] The second feature fusion module is used to obtain a specified input feature based on feature fusion of the second feature to be detected and the first fused feature.

[0290] Optionally, the data to be detected is the first feature to be detected.

[0291] Optionally, the first fusion network is an attention pooling network, and the semantic feature extraction layer is a Transformer Block.

[0292] Optionally, the pre-trained audio large model is an initial audio large model obtained by at least model pre-training; the model pre-training process of the second conversion model includes:

[0293] Obtain a first sample audio and a first label representing an emotion category of the first sample audio;

[0294] Extracting frequency domain features of the first sample audio as first sample features;

[0295] Inputting the first sample feature into an initial classification model to obtain an emotion classification result of the first sample audio; wherein the initial classification model includes an initial second conversion model and a classification network, and output data of the initial second conversion model is used as input data of the classification network;

[0296] Determining a first model loss value based on a difference between the obtained emotion classification result and the first label;

[0297] Based on the first model loss value, the model parameters of the initial classification model are adjusted until a first convergence condition is reached.

[0298] Optionally, the pre-trained audio model is an initial audio model that is obtained by model pre-training and then fine-tuning the model; the model fine-tuning process of the audio model after model pre-training includes:

[0299] Acquire a second sample audio, and a second label indicating whether bullying behavior exists in the collection environment of the second sample audio;

[0300] Extracting frequency domain features of the second sample audio as second sample features;

[0301] Inputting the second sample feature into the audio large model after model pre-training to obtain a detection result indicating whether bullying behavior exists in the collection environment of the second sample audio;

[0302] Determine a second model loss value based on a difference between the obtained detection result and the second label;

[0303] Based on the second model loss value, the specified model parameters of the large audio model after model pre-training are adjusted until the second convergence condition is reached.

[0304] Optionally, the designated model parameters are model parameters other than the model parameters of the semantic feature extraction network, the model parameters of the emotion feature extraction network, and the model parameters of the large language model.

[0305] Optionally, the device further comprises:

[0306] The first alarm module is used to issue an alarm in a first alarm mode when it is determined that bullying behavior exists in the collection environment of the audio to be detected.

[0307] Corresponding to the above-mentioned method embodiment provided from the front-end device side, the present application embodiment provides a behavior detection device, which is applied to the front-end device, such as Fig.10 As shown, the device comprises:

[0308] The acquisition module 1010 is used to acquire the audio to be detected in the current environment, and extract the frequency domain feature of the audio to be detected as the first feature to be detected;

[0309] A keyword recognition module 1020, configured to perform bullying keyword recognition on the first feature to be detected, and obtain a keyword recognition result indicating whether the audio to be detected contains the bullying keyword;

[0310] The sending module 1030 is used to send the data to be detected to the central device if the keyword recognition result indicates that the audio to be detected contains bullying keywords, so that after the central device receives the data to be detected, it uses a pre-trained audio model to perform intent recognition on the data to be detected, and obtains a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected.

[0311] Optionally, the keyword identification module 1020 is specifically configured to:

[0312] The first feature to be detected is input into a pre-trained keyword recognition network to obtain a keyword recognition result indicating whether the audio to be detected contains a bullying keyword.

[0313] Optionally, the device further comprises:

[0314] The second alarm module is used to issue an alarm in a second alarm mode if the keyword recognition result indicates that the audio to be detected contains bullying keywords.

[0315] In the technical solution of the present application, operations such as acquisition, storage, use, processing, transmission, provision and disclosure of audio are all carried out with the user's authorization.

[0316] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0317] Memory 1101, used for storing computer programs;

[0318] The processor 1102 is used to implement the steps of any of the above-mentioned behavior detection methods when executing the program stored in the memory 1101.

[0319] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0320] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0321] The communication interface is used for communication between the above electronic device and other devices.

[0322] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0323] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0324] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned behavior detection methods are implemented.

[0325] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any behavior detection method in the above embodiments.

[0326] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0327] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0328] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0329] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A behavior detection method, characterized in that: Applied to a central terminal device, the method comprises: Receive data to be detected; wherein the data to be detected is: the front-end device identifies bullying keywords on the first feature to be detected, and uploads it when the keyword identification result indicates that the audio to be detected contains bullying keywords, the first feature to be detected is: the front-end device extracts frequency domain features of the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected; the data to be detected is the first feature to be detected; Using a pre-trained audio model to perform intent recognition on the data to be detected, and obtaining a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected; The audio large model includes a large language model and a first conversion model, wherein the first conversion model includes a semantic feature extraction network for extracting context information; The pre-trained audio model is used to perform intent recognition on the data to be detected, and a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected is obtained, including: Using the semantic feature extraction network to extract features from the data to be detected, to obtain a second feature to be detected; Inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected; The semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes a first fusion network; Before inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the method further includes: Inputting the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature; Based on feature fusion of the second feature to be detected and the first fused feature, a specified input feature is obtained.

2. The method according to claim 1, characterized in that The audio large model also includes a second conversion model, wherein the second conversion model includes an emotion feature extraction network for extracting emotion information; Before inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the method further includes: Using the emotion feature extraction network to extract features from the data to be detected, to obtain a third feature to be detected; Based on feature fusion of the second feature to be detected and the third feature to be detected, a specified input feature is obtained.

3. The method according to claim 2, characterized in that The semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, and the second conversion model also includes a first fusion network and a second fusion network; Before inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the method further includes: Inputting the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature; Based on the feature fusion of the second feature to be detected and the third feature to be detected, a specified input feature is obtained, including: Using the second fusion network to perform feature fusion on the first fusion feature and the third feature to be detected to obtain a second fusion feature; Based on feature fusion of the second feature to be detected and the second fused feature, a specified input feature is obtained.

4. The method according to claim 3, characterized in that The second fusion network is a cross attention network; Using the second fusion network to perform feature fusion on the first fusion feature and the third feature to be detected to obtain a second fusion feature, including: The first fused feature is mapped to obtain the query Q and value V in the input data of the cross-attention network, and the third feature to be detected is mapped to obtain the key K in the input data of the cross-attention network, and the obtained query Q, value V and key K are input into the cross-attention network for feature fusion to obtain the second fused feature.

5. The method according to claim 1 or 3, characterized in that: The first fusion network is an attention pooling network, and the semantic feature extraction layer is a transformer layer Transformer Block.

6. The method according to claim 2, characterized in that The pre-trained audio large model is an initial audio large model obtained by at least model pre-training; The model pre-training process of the second conversion model includes: Obtain a first sample audio and a first label representing an emotion category of the first sample audio; Extracting frequency domain features of the first sample audio as first sample features; Inputting the first sample feature into an initial classification model to obtain an emotion classification result of the first sample audio; wherein the initial classification model includes an initial second conversion model and a classification network, and output data of the initial second conversion model is used as input data of the classification network; Determining a first model loss value based on a difference between the obtained emotion classification result and the first label; Based on the first model loss value, the model parameters of the initial classification model are adjusted until a first convergence condition is reached.

7. The method according to claim 2, characterized in that The pre-trained audio model is obtained by fine-tuning the initial audio model after model pre-training; the model fine-tuning process of the audio model after model pre-training includes: Acquire a second sample audio, and a second label indicating whether bullying behavior exists in the collection environment of the second sample audio; Extracting frequency domain features of the second sample audio as second sample features; Inputting the second sample feature into the audio large model after model pre-training to obtain a detection result indicating whether bullying behavior exists in the collection environment of the second sample audio; Determine a second model loss value based on a difference between the obtained detection result and the second label; Based on the second model loss value, the specified model parameters of the large audio model after model pre-training are adjusted until the second convergence condition is reached.

8. The method according to claim 7, characterized in that The designated model parameters are model parameters other than the model parameters of the semantic feature extraction network, the model parameters of the emotion feature extraction network, and the model parameters of the large language model.

9. The method according to claim 1, characterized in that: The method further comprises: When it is determined that bullying behavior exists in the collection environment of the audio to be detected, an alarm is issued in a first alarm mode.

10. A behavior detection method, characterized in that: Applied to a front-end device, the method comprises: Collecting the audio to be detected in the current environment, and extracting the frequency domain features of the audio to be detected as the first feature to be detected; Performing bullying keyword recognition on the first feature to be detected to obtain a keyword recognition result indicating whether the audio to be detected contains the bullying keyword; If the keyword recognition result indicates that the audio to be detected contains bullying keywords, the data to be detected is sent to the central end device, so that after receiving the data to be detected, the central end device uses a pre-trained audio model to perform intent recognition on the data to be detected, and obtains a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected; the data to be detected is the first feature to be detected; The audio large model includes a large language model and a first conversion model, wherein the first conversion model includes a semantic feature extraction network for extracting context information; The step of the central end device using a pre-trained audio large model to perform intent recognition on the data to be detected to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected includes: Using the semantic feature extraction network to extract features from the data to be detected, to obtain a second feature to be detected; Inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected; The semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes a first fusion network; Before the central-end device inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the central-end device inputs the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature; and obtains the designated input feature based on feature fusion of the second feature to be detected and the first fusion feature.

11. The method according to claim 10, characterized in that Performing bullying keyword recognition on the first feature to be detected to obtain a keyword recognition result indicating whether the audio to be detected contains the bullying keyword includes: The first feature to be detected is input into a pre-trained keyword recognition network to obtain a keyword recognition result indicating whether the audio to be detected contains a bullying keyword.

12. The method according to claim 10, characterized in that The method further comprises: If the keyword recognition result indicates that the audio to be detected contains bullying keywords, an alarm is issued in the second alarm mode.

13. A behavior detection system, characterized in that: The system includes a central terminal device and a front-end device; The central terminal device is used to execute the behavior detection method described in any one of claims 1 to 9 above; The front-end device is used to execute the behavior detection method described in any one of claims 10-12 above.

14. The system according to claim 13, characterized in that There are multiple front-end devices, and the deployment locations of the multiple front-end devices are different; Each front-end device is specifically configured to send the data to be detected and its own location information to the central device when the keyword recognition result indicates that the audio to be detected contains bullying keywords; The central terminal device is also used to, when it is determined based on the data to be detected sent by any front-end device that bullying behavior exists, issue an alarm in a first alarm mode according to the location information of the front-end device.

15. A behavior detection device, characterized in that: Applied to a central terminal device, the device comprises: A receiving module, used for receiving data to be detected; wherein the data to be detected is: the front-end device performs bullying keyword recognition on the first feature to be detected, and is uploaded when the keyword recognition result indicates that the audio to be detected contains the bullying keyword, the first feature to be detected is: the front-end device performs frequency domain feature extraction on the audio to be detected collected in the current environment, and the data to be detected is used to characterize the audio to be detected; the data to be detected is the first feature to be detected; An intention recognition module, used to use a pre-trained audio model to perform intention recognition on the data to be detected, and obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected; The audio large model includes a large language model and a first conversion model, wherein the first conversion model includes a semantic feature extraction network for extracting context information; The intention recognition module includes: A feature extraction submodule, used for extracting features from the data to be detected using the semantic feature extraction network to obtain a second feature to be detected; A model processing submodule, used for inputting the specified input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected; The semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes a first fusion network; The device also includes: A second fusion processing module is configured to input the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature before the model processing submodule inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected; The second feature fusion module is used to obtain a specified input feature based on feature fusion of the second feature to be detected and the first fused feature.

16. A behavior detection device, characterized in that: Applied to front-end equipment, the device comprises: A collection module, used for collecting the audio to be detected in the current environment, and extracting the frequency domain feature of the audio to be detected as the first feature to be detected; A keyword recognition module, used to perform bullying keyword recognition on the first feature to be detected, and obtain a keyword recognition result indicating whether the audio to be detected contains bullying keywords; A sending module, for sending data to be detected representing the audio to be detected to a central device if the keyword recognition result indicates that the audio to be detected contains a bullying keyword, so that after the central device receives the data to be detected, it uses a pre-trained audio large model to perform intent recognition on the data to be detected, and obtains a detection result indicating whether there is bullying behavior in the collection environment of the audio to be detected; the data to be detected is the first feature to be detected; the audio large model includes a large language model and a first conversion model, and the first conversion model includes a semantic feature extraction network for extracting context information; The step of the central end device using a pre-trained audio large model to perform intent recognition on the data to be detected to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected includes: Using the semantic feature extraction network to extract features from the data to be detected, to obtain a second feature to be detected; Inputting the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected; The semantic feature extraction network includes a first number of semantic feature extraction layers connected in series, the audio large model also includes a second conversion model, and the second conversion model includes a first fusion network; Before the central-end device inputs the designated input feature obtained based on the second feature to be detected into the large language model to obtain a detection result indicating whether bullying behavior exists in the collection environment of the audio to be detected, the central-end device inputs the output features of the first second number of semantic feature extraction layers in the semantic feature extraction network into the first fusion network to obtain a first fusion feature; and obtains the designated input feature based on feature fusion of the second feature to be detected and the first fusion feature.

17. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method described in any one of claims 1 to 9, or the method described in any one of claims 10 to 12, when executing a program stored in a memory.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 9 or the method described in any one of claims 10 to 12 is implemented.

19. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 9, or the method according to any one of claims 10 to 12.

Images